Reading ethical frameworks metrics without fooling yourself

Every number in this area is ambiguous in the same way: it can move because conduct changed, or because reporting changed, and the number itself cannot tell you which.

That is not a reason to give up on measurement. It is a reason to stop reading single figures and start reading pairs: one number that moves for a good reason paired with one that would move for the bad reason, so the pair tells you what neither can alone.

What to take away

Every figure in this area has two opposite readings, so read them in pairs: one number that moves for a good reason beside one that would move for the bad reason.
A rise in reports is the most misread number here. It can mean the channel started working or that conduct got worse, and the figure alone cannot tell you which.
Attaching a target to any of these makes it unreliable within a cycle, because suppressing reports is cheaper than improving conduct.
Before concluding that behavior changed, rule out a change in definition, in population, and in visibility, in that order.

The ambiguity, concretely

Suppose reports of concerns rose sharply this quarter.

That is either the best news you will get this year (people now believe the channel works, and problems are surfacing while they are still small), or the worst, meaning conduct deteriorated sharply. The figure is identical in both cases.

The same ambiguity runs through almost everything measurable here. Closed investigations can mean resolution or paperwork; zero incidents can mean a clean operation or a broken detection system; fast case closure can mean efficiency or superficiality.

High policy attestation rates mean only that a system sent reminders. The preference for evidence that a program changes behavior over evidence that it exists is the same one set out in the US Department of Justice's Evaluation of Corporate Compliance Programs. That is the guidance for prosecutors evaluating compliance programs most US programs are judged against.

This is why ethics dashboards tend to be so uninformative: they are built from figures that each have two opposite readings, presented as though they had one.

Reading in pairs

What it might mean

Reports received: More trust, or more trouble
Reports up in one function: That team is worse, or that team is the only one with a working route
Cases closed quickly: Efficient, or superficial
Substantiation rate high: Good filtering, or a channel only used when someone has proof
Substantiation rate low: Frivolous use, or investigations that do not find things
Zero retaliation complaints: It does not happen, or complaining twice is unthinkable
Training completion high: Nothing

Pair it with

Reports received: Where they came from, and how serious they were
Reports up in one function: Report rate per head by function
Cases closed quickly: Proportion reopened, and reporter follow-up at six months
Substantiation rate high: Proportion of reports arriving before harm occurred
Substantiation rate low: Who investigates, and whether they are independent of the area
Zero retaliation complaints: Retention and progression of people who reported, at twelve months
Training completion high: Whether people can describe what to do, in their own words

The second column of that table is the only honest way to present these figures to a board. A report that gives the number without the alternative reading is inviting a conclusion the data does not support.

A worked example. Suppose reports rise from 40 in one quarter to 70 in the next. Read alone, that total says either trouble or trust. Now split the increase by function.

Imagine 26 of the 30 extra reports come from one site that had filed almost nothing before, after a new manager there began referring cases to the channel. The total rose by three quarters while the underlying conduct, as far as these numbers show, barely moved.

Four ways ethics measurement goes wrong

Measuring effort as though it were effect. Sessions delivered, policies published, audits performed, suppliers signed up. These are inputs. They tell you what you spent, which is worth knowing, and nothing about whether anything changed.

Attaching a target. Any measure here becomes unreliable the moment someone's bonus depends on it. Incident counts fall when reporting is discouraged, faster and more cheaply than when conduct improves. This is not a hypothetical risk; it is the predictable response of a rational person to the incentive you created.

Aggregating away the signal. Ethics problems are local. A company-level figure with a stable trend routinely conceals one function where something is badly wrong. Look at distribution, not central tendency, and look at the outlier before the average.

Confusing detection with prevalence. Improving detection increases measured incidents. An organization that reads that as failure will stop improving detection. This single dynamic explains a large share of why programs stop working after year two.

Measures that carry real information

Not because they are unambiguous, none are, but because they are hard to fake and close to what you care about.

Time to first substantive responseon a raised concern, and the proportion receiving one at all. Slow or absent responses kill a channel more reliably than retaliation does.
Where in the sequence a problem was caught.Before anyone was affected, after but internally, or by an outsider. Movement toward the front is the strongest available evidence that anything is working.
Recurrence of the same issue in the same place.Recurrence means the last fix addressed a symptom.
How many decisions went against the commercial recommendation,and at what level. If none did, no procedure has any force.
Exception volume.Waivers granted, approvals outside policy, overrides. Rising exceptions mean a rule that no longer matches the work. Zero exceptions usually means people stopped asking.
Progression of people who raised concerns,compared with peers, at twelve and twenty-four months. Uncomfortable and highly informative. In some jurisdictions this is also a legal question rather than only a management one, as the UK's whistleblowing framework sets out.
What people say when asked in their own words.Ask a sample to describe what they would do in a specific situation. It resists gaming, because there is no form to fill in.

Interpreting a moving number

Before drawing a conclusion from any change, work through four alternatives in order:

Four checks before concluding

Did the definition change?

Yes

Category, form or team change

Did the population change?

Definition.Did the definition change? A category widened, a rule was rewritten, or a threshold moved, so the same conduct now lands in the figure.
Population.Did the population change? More staff, a new site, or a contractor group brought into scope moves the count while the rate per head stays flat.
Visibility.Did visibility change? A new channel, a training push, or a manager who encourages reporting means more gets seen without conduct moving.
Conduct.Only after the first three are ruled out does a change in behavior become the working explanation.

Most reporting on this subject starts at four. Starting at one is slower and produces conclusions that survive the next quarter.

What to put in front of a board

A short, honest package beats a dashboard:

Two or three figures with both readings stated.
The distribution behind each, not just the total.
One thing that got worse, with what you are doing about it. A report where everything improved is not being believed by anyone experienced.
One case, anonymized, described in enough detail to be real.
What you cannot measure and are not pretending to.

That last item does more for the credibility of the rest than any amount of additional data. The organizations that are actually good at this tend to be conspicuously modest about what their numbers establish.

Related reading: codes of conduct covers the rules they reflect, ethical frameworks framework the procedure they measure, ethics training programs why completion rates tell you nothing, and corporate responsibility the wider reporting problem.

Common questions

If reports go up, is that good or bad?

Unknowable from the number alone. Pair it with the source and the severity; a rise concentrated in one function that previously filed nothing usually means the channel started working.

Should any of these carry a target?

No. Pay tied to a count makes the count cheap to manage down.

What belongs in front of a board?

The package above: two or three figures with both readings, the distribution behind each, and the one item that got worse.