Loss Curves Lie: Building a Deterministic Linter for ML Training Runs

A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't.

I kept losing GPU hours to runs that were already dead. A fine-tune that had quietly gone to NaN somewhere after step 300 and trained on garbage the rest of the night. A run that was training at learning-rate-zero the whole time and updating nothing. A dataset with a few hundred broken rows I only discovered at the end. Every time, the pattern was the same: the run looked like it was working, and nothing told me otherwise until the hours were already spent.

So I built a linter for training runs — trainproof — and then I spent days trying to prove it wrong.

pip install trainproof trainproof doctor .   # zero-config: discovers and judges every training log it finds

Repo: github.com/Mormolykos/trainproof (MIT)

The philosophy: no ML judging ML

The most important decision in trainproof is what it isn't. There's no model scoring your run. No "87%-confidence this looks unhealthy." Every verdict is a deterministic rule that either fires or it doesn't, and every finding cites the exact numbers behind it. When it can't be sure, it says so instead of guessing.

That constraint is the whole point. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms. trainproof's engine is IO-free and doesn't even import torch — it reads plain logs and applies rules. You can read every rule and know exactly why it fired.

Validating the rules: controlled fault injection

You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One base setup — a Qwen2.5-3B QLoRA on an RTX 5080 — run six ways:

• healthy — the control.
• lr_hot — learning rate cranked 100× too high.
• lr_zero — learning rate at zero.
• fp16_nan — an fp16 overflow that NaNs the loss.
• bad_labels — the dataset's labels shuffled into pure noise.
• overfit — 64 training samples and many epochs: pure memorisation, with a held-out eval set to prove it.

Three seeds each: 18 runs, real logs, all shipped in the repo so you can reproduce every verdict. A nineteenth log ships beside them with no injected fault at all — a 9.8-hour segment of a Coqui XTTS fine-tune. I originally described that run as one that "diverged on its own". Two audits later I had to withdraw that; see the correction below.

Four of the five failures got caught fast. lr_hot spiked the gradient norm to 2,650× the median — flagged in seconds. The NaN and the flatline were trivial. But one config beat the tool completely, and it's the one that taught me the most.

The one that fooled it: shuffled labels

bad_labels was pure garbage — a dataset that cannot be learned, because the labels no longer correspond to the inputs. And that run reduced its loss by 62%.

On its own loss curve it looked like textbook-healthy training: a clean downward slope. It was learning absolutely nothing useful — just memorizing the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve, it is indistinguishable from a real run. No loss-only rule, looking at that run in isolation, can catch it.

That's not a gap I papered over — I wrote it straight into the README. It's also why trainproof has a compare mode: the failure becomes obvious the moment you put the run next to a known-good baseline and look at the relative floor it reaches. The corrupted run's loss floor sits in a different regime. One run in isolation lies; two runs side by side tell the truth.

What trainproof actually does

It covers a run's whole life, and every stage exits with a status code so it drops straight into CI:

• Before a single GPU-second — preflight. Lints the dataset and tokenizer: malformed JSONL (with the line number), empty rows, duplicates, missing eos_token, pad == eos, over-length samples. Catch the broken dataset before you rent the GPU.
• During training — the guardian. A one-line Hugging Face callback. It warns by default; flip on stop_on_fail and it will abort a doomed run itself. In one guarded demo, it killed a diverging run at step 20 of 300 scheduled steps — so 280/300 (≈93%) of the scheduled steps never executed. (That's a statement about that run, not a promise about your GPU bill — but a diverging run you stop at step 20 is a diverging run you didn't pay to finish.)
• After — epoch and doctor. Read the finished log and classify it: diverged, flatlined, NaN'd, spiked, overfitting.
• Versus a baseline — compare. The relative-floor rules that catch the shuffled-labels case.

It reads the logs you already produce — Hugging Face trainer_state.json, Coqui, plain JSONL/CSV — so there's nothing to instrument.

Try it

pip install trainproof

• Repo (with all 18 fault-injection logs, plus the one that broke by itself): https://github.com/Mormolykos/trainproof
• MIT licensed.

If you fine-tune, you've had at least one of these failures. I'd genuinely like to know which failure mode has burned your GPU hours — if a deterministic check would have caught it, tell me on the repo and it goes in, with credit.

Update: what came after

Since this article was written, trainproof has grown from the experiment above into a fuller tool. The release timeline at the top of this page is generated from GitHub Releases and PyPI, so it is always current — this section covers what changed in kind, not which version you are on:

• trainproof doctor . — point it at a directory; it discovers every training log, judges them all, and prints a triage-sorted summary. The six configurations above are now one command.
• Guardian telemetry — the Hugging Face callback measures wall-clock step time live; deterministic rules catch step-time cliffs and dataloader-bound runs, and watch --stall-timeout warns when the log stops growing.
• Stable rule IDs (TP-DIVERGE, TP-DEAD-RUN, …) documented in RULES.md, an honest PASS that lists which checks ran and which were skipped, and --json output built for CI pipelines and AI coding agents.
• A log nobody injected a fault into. Injected faults are cleaner than real ones. That is the known limitation of any fault-injection study, and it is why this one does not stand alone. The gallery now ships examples/real_world/xtts_diverged: a 9.8-hour segment of a Coqui XTTS fine-tune from my own work, with nobody touching it. This entry originally called it "a fine-tune that diverged on its own" — a claim I withdrew in 0.22.0 and unpick in the correction below. All three seeds of every configuration ship too — 18 runs at seeds 42/43/44, where before only one seed per configuration was committed — so "three seeds out of three" became a claim you can check rather than one you have to take on trust.

Update: the linter failed its own test

This article argues that a training run can look healthy while learning nothing. Auditing trainproof before a release, I found it doing exactly that — to itself.

The question was mundane. What happens to a run whose loss is exactly 0.0 on every step? That is a real failure with a boring cause: if every target label is masked to -100, cross-entropy returns zero and the model learns nothing at all. trainproof's answer was PASS.

Every loss-shape check — flat curve, divergence, no-improvement — was guarded by a > 0 test, put there to avoid dividing by zero. On an all-zero curve all three guards bailed out, silently. The run reached a passing verdict, and that verdict then named those same three checks as having run.

That is worse than missing the failure. A tool that reports "I checked this" when it did not is not a weak alarm, it is a false one — and it is the same shape as the bad_labels run above: something that looks like evidence of health and is nothing of the kind. The same audit found the hole in three more places, including a baseline comparison that rated a zero-loss run as favourable, because a loss floor of zero beats every baseline there is.

v0.12.0 is the fix, and it encodes one sentence: a skipped check is not a passed check. A PASS now names every check that ran and every check that did not, each with its reason, as structured data rather than prose. Judging a log that has no evaluation set now says so out loud:

Ran: dead-run, divergence, flat-loss, grad-spike, lr, zero-grad, zero-loss. Skipped: loader (no loader_time/step_time pair in the log); overfit (no eval_loss in the log - this run has no generalisation signal at all); step-time (no step_time column in the log).

You can read that and know what the pass is worth. Nothing about how trainproof judges a run changed — all 38 locked verdicts are byte-identical to the previous release. What changed is that it stopped overstating what it had looked at.

Update: two audits, and the claim I had to withdraw

Twice on this page I told you about a real training run that "diverged on its own" — the one uninjected failure in a gallery of deliberate ones, the log that made the fault-injection study stand on something real. Before the 0.22.0 release I put trainproof through two independent forensic audits, one of them working blind, and that claim did not survive.

The log I shipped is a prefix. It stops at step 72,900 of a run that went to 125,039. I had truncated it, published the verdict computed from the truncation, and cited the trainer's own bookkeeping as corroboration — the last BEST MODEL line in the prefix is best_model_49880.pth, which does sit a long way behind the end of the prefix.

The rest of the run was still on disk. In the complete log the trainer promotes best_model_124700.pth — step 124,700 of 125,039, 99.7% of the way through — and all six retained held-out evaluations improve, the last one included, 4.8813 down to 2.5894. The bookkeeping I cited as corroboration was corroborating a sentence about the first 58% of a run.

I want to be exact about what replaces it, because the temptation is to overcorrect. What the retained evidence supports is narrower than either claim: the per-micro-batch training display loss ended above its own minimum. That is a property of that one series. It is not evidence the model got worse — and it is not evidence the model got better, because no audio, MOS or perceptual evaluation of that run was kept. Held-out loss improved. That is all that was measured, and it is all I will say.

Here is the part that matters more than my mistake. trainproof still returns FAIL on that run, and 0.22.0 does not fix it. TP-DIVERGE reads one training series. That run's event file carries a per-micro-batch loss with 1,251 points that ends 1.88× above its minimum, and an epoch-aggregated loss with 5 points that ends at its minimum. The reader resolves competing series by density, so the 1,251-point series wins and the 5-point series is discarded before any rule sees it. An improving held-out curve sitting in the same file cannot overrule the survivor.

Changing the threshold until this particular run went green would have been the worst available option — tuning a rule against a single example until it produces the answer I now prefer. So the run still FAILs, the limitation is written into the README, the rules document and the release's own machine-readable record, and it is the top item for the next architecture rather than something quietly smoothed over.

The audits found more, and the same principle decided each one: remove a claim the tool could not support, or make an advertised behaviour real — never strengthen a claim to close a gap.

• A missing sentencepiece install made trainproof tokenizer exit 1 with a FAIL. That is the tool reporting its own missing package as a fault in your tokenizer. It is now NOT-CHECKED and exit 2.
• The HuggingFace callback read the first batches of labels at on_train_begin, which means opening a fresh iterator over your training dataloader. With a map-style loader and a random sampler, that consumes generator state and can change the batch order of the run it is supposed to be observing. A measurement tool perturbing its own experiment. It is now opt-in.
• Four evidence strings described something other than what was measured — including a zero-learning-rate finding that reported 100.0% of steps have lr=0 for a run where every value was -1e-4.

No rule in trainproof is calibrated, and 0.22.0 does not change that either. There is no false-positive rate for any check, because nothing here has been run against a population where the answer is independently known. The gallery is a set of faults I injected on purpose, which makes it a regression suite, not a sample. What the tool offers is determinism and evidence: the same log gives the same verdict, every finding prints the numbers behind it, and a check that could not run says NOT-CHECKED instead of PASS.

Read a finding's evidence as the measurement and its message as an interpretation that may exceed it. That sentence is in the README now because the audits showed it needed to be.

Release 0.22.0 is archived at 10.5281/zenodo.22882964.