The Same Bug, Four Times, Three of Them Mine

A validator that reports a verdict without reporting its coverage is asserting something it did not measure. I found that in four systems before I admitted it was one problem.

Most validation tooling has two states: it passed, or it failed. Everything that was not actually evaluated has to be forced into one of them — and it is wrong in both directions.

I did not work that out from first principles. I worked it out by shipping the same defect four times.

One: a run that learned nothing, reported as healthy

In trainproof, a training run whose loss was exactly 0.0 on every step returned PASS.

Every loss-shape check in that tool is guarded against dividing by zero. A curve that is identically zero trips every guard, so all of them skipped — silently. And then the report listed those same skipped checks as having run. A run that learned nothing passed, accompanied by a list of the checks that had cleared it.

That is the whole problem in one artifact. The tool had no way to say I could not judge this, so the absence of a judgement rendered as a favourable one.

Two: the same tool, one loop earlier

The fix for that was a report field listing which checks ran and which did not. Good. Then a nastier version turned up somewhere else entirely.

trainproof doctor walks a directory twice — once to discover candidate logs, once to judge them. The judging pass reported anything it could not read. The discovery pass had except Exception: pass.

So a file that raised while being found never became a candidate, and never appeared in the report at all. Plainly visible on disk. Absent from the output. Indistinguishable from a file that passed.

Same failure as the first, one loop earlier than I had been looking. That is when I stopped treating it as a bug.

Three: not my system

I wrote the first two up publicly. Someone working on infrastructure compliance replied with the identical shape from a domain I know nothing about.

A compliance framework document is mostly prose. Most of it describes things no generated artifact can satisfy or violate — staff training, review procedures, who signs what. Only a fraction maps to something a machine can check.

The trap is reporting against the framework's name. Do that and everything unevaluated looks identical to everything that passed, and the ninety per cent that was never in scope disappears from the output entirely.

That exchange is why this became a library rather than another trainproof feature. One person hitting a shape twice is a habit. Two people hitting it in unrelated domains is a primitive.

Four: found while I was busy being pleased with myself

Then I ran an evaluation harness of my own over a retrieval experiment, and found three instances in a single afternoon.

It recorded model refusals under a failure type asserting an ordering the model had never given. It scored ten refusals as correct, because the expected phrase appeared inside the sentence explaining what could not be determined. And it missed eight correct answers because its negative pattern required a comma — so "No. The voice speaks at 3600, before the whisper" matched nothing and was marked wrong.

One absent value. Wrong in both directions. Inside the instrument I was using to judge my own hypothesis.

What the states actually are

Three instead of two — checked, could-not-check, never-in-scope — and eight once you ask two more questions of each gap: who can fix it, and can it ever change.

The owner column is the point. Two gaps sharing a bucket hand the reader a to-do they cannot action — "accept the gap" and "fix your parser" are opposite instructions, and only one of them will ever change on its own. A test enforces that no two states share both an owner and a remediation.

Then I attacked it, and it did not survive

Before publishing anything I tried to break the design. Twice. Both attacks landed.

The API attack found five holes, and two of them were the library committing its own thesis error one level up.

Declare every target out of scope and the evaluable denominator went to zero — and it exited 0. A report that judged nothing read as green. Worse: I had written a test asserting that as correct behaviour, so the bug was encoded as intent.

And a target that never became a record was invisible. The report claimed perfect coverage. That is failure two, from the top of this article, unprevented by the library written because of it.

Also: duplicate targets counted twice in silence; failing_verdicts defaulted to {"FAIL"}, so a compliance tool emitting NON_COMPLIANT exited 0 on real failures; and frozen=True protected the binding but not the dict behind it.

The taxonomy attack was the useful one. Twenty-four realistic cases across ML training, compliance, RAG evaluation, CI/CD and production monitoring. Twelve fit exactly one state. Five fit none. Four fit two.

Two of the five needed new states. A waived control is in scope, is applicable, and was deliberately not evaluated by a named person — filing it under "not requested" is false, and it removes the control from the denominator, which is exactly where an accepted risk goes to hide. And a check skipped because a prerequisite failed is not a broken checker; calling it one sends someone to debug a parser that works.

The other three needed rules, not states. One matters more than the rest:

Unreachable is not unhealthy. A health check that could not observe its target has made no determination. Reporting that as a failing verdict is this library's own error wearing a different costume.

That one was a single commit from shipping as a feature.

What it cannot do, largest first

A misconfigured check reports CHECKED. If your threshold is wrong, the check runs, produces a meaningless number, and this library faithfully records a clean determination. It accounts for what your checks report. It cannot know whether a check is meaningful, and nothing in it mitigates that.

It has no notion of history, so it will not notice a flake. It has no sub-target granularity, so a checker that sampled ten per cent of a target reports CHECKED for all of it. Evidence freshness is invisible. And expected — the fix for the vanished-target hole — is only as good as the target list you declare.

Why this is a schema and not a feature

Every instinct I had said to fix it in trainproof and move on. I had already done that twice, and the second fix did not prevent the third or fourth instance, because the bug is not in any of those tools. It is in the shape of the report they all produce.

The library is small on purpose — an enum, a record, a reporter that derives its counts rather than storing them. The value is in the vocabulary being right, and in having been proven wrong twice before anyone else saw it.

github.com/Mormolykos/notchecked — MIT, no dependencies. The ownership axis and the four-state split are mine; the fixed reason vocabulary, counts-from-rows and the permanence split came from the compliance side of that conversation, and are credited in the README.