Loss Curves Lie: Building a Deterministic Linter for ML Training Runs

A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't.

I kept losing GPU hours to runs that were already dead. A fine-tune that had quietly gone to NaN somewhere after step 300 and trained on garbage the rest of the night. A run that was training at learning-rate-zero the whole time and updating nothing. A dataset with a few hundred broken rows I only discovered at the end. Every time, the pattern was the same: the run looked like it was working, and nothing told me otherwise until the hours were already spent.

So I built a linter for training runs — trainproof — and then I spent days trying to prove it wrong.

pip install trainproof trainproof doctor .   # zero-config: discovers and judges every training log it finds

Repo: github.com/Mormolykos/trainproof (MIT)

The philosophy: no ML judging ML

The most important decision in trainproof is what it isn't. There's no model scoring your run. No "87%-confidence this looks unhealthy." Every verdict is a deterministic rule that either fires or it doesn't, and every finding cites the exact numbers behind it. When it can't be sure, it says so instead of guessing.

That constraint is the whole point. A reliability tool that hallucinates is worse than no tool, because now you don't trust your own alarms. trainproof's engine is IO-free and doesn't even import torch — it reads plain logs and applies rules. You can read every rule and know exactly why it fired.

Validating the rules: controlled fault injection

You validate a detector by feeding it faults whose answer you already know, then measuring what it catches. One base setup — a Qwen2.5-3B QLoRA on an RTX 5080 — run six ways:

• healthy — the control.
• lr_hot — learning rate cranked 100× too high.
• lr_zero — learning rate at zero.
• fp16_nan — an fp16 overflow that NaNs the loss.
• bad_labels — the dataset's labels shuffled into pure noise.
• overfit — 64 training samples and many epochs: pure memorisation, with a held-out eval set to prove it.

Three seeds each: 18 runs, real logs, all shipped in the repo so you can reproduce every verdict. A nineteenth log ships beside them with no injected fault at all — a 9.8-hour Coqui XTTS fine-tune that diverged on its own.

Four of the five failures got caught fast. lr_hot spiked the gradient norm to 2,650× the median — flagged in seconds. The NaN and the flatline were trivial. But one config beat the tool completely, and it's the one that taught me the most.

The one that fooled it: shuffled labels

bad_labels was pure garbage — a dataset that cannot be learned, because the labels no longer correspond to the inputs. And that run reduced its loss by 62%.

On its own loss curve it looked like textbook-healthy training: a clean downward slope. It was learning absolutely nothing useful — just memorizing the statistics of noise, which any sufficiently large network will happily do. From a single run's loss curve, it is indistinguishable from a real run. No loss-only rule, looking at that run in isolation, can catch it.

That's not a gap I papered over — I wrote it straight into the README. It's also why trainproof has a compare mode: the failure becomes obvious the moment you put the run next to a known-good baseline and look at the relative floor it reaches. The corrupted run's loss floor sits in a different regime. One run in isolation lies; two runs side by side tell the truth.

What trainproof actually does

It covers a run's whole life, and every stage exits with a status code so it drops straight into CI:

• Before a single GPU-second — preflight. Lints the dataset and tokenizer: malformed JSONL (with the line number), empty rows, duplicates, missing eos_token, pad == eos, over-length samples. Catch the broken dataset before you rent the GPU.
• During training — the guardian. A one-line Hugging Face callback. It warns by default; flip on stop_on_fail and it will abort a doomed run itself. In one guarded demo, it killed a diverging run at step 20 of 300 scheduled steps — so 280/300 (≈93%) of the scheduled steps never executed. (That's a statement about that run, not a promise about your GPU bill — but a diverging run you stop at step 20 is a diverging run you didn't pay to finish.)
• After — epoch and doctor. Read the finished log and classify it: diverged, flatlined, NaN'd, spiked, overfitting.
• Versus a baseline — compare. The relative-floor rules that catch the shuffled-labels case.

It reads the logs you already produce — Hugging Face trainer_state.json, Coqui, plain JSONL/CSV — so there's nothing to instrument.

Try it

pip install trainproof

• Repo (with all 18 fault-injection logs, plus the one that broke by itself): https://github.com/Mormolykos/trainproof
• MIT licensed.

If you fine-tune, you've had at least one of these failures. I'd genuinely like to know which failure mode has burned your GPU hours — if a deterministic check would have caught it, tell me on the repo and it goes in, with credit.

Update: what came after

Since this article was written, trainproof has grown from the experiment above into a fuller tool. The release timeline at the top of this page is generated from GitHub Releases and PyPI, so it is always current — this section covers what changed in kind, not which version you are on:

• trainproof doctor . — point it at a directory; it discovers every training log, judges them all, and prints a triage-sorted summary. The six configurations above are now one command.
• Guardian telemetry — the Hugging Face callback measures wall-clock step time live; deterministic rules catch step-time cliffs and dataloader-bound runs, and watch --stall-timeout warns when the log stops growing.
• Stable rule IDs (TP-DIVERGE, TP-DEAD-RUN, …) documented in RULES.md, an honest PASS that lists which checks ran and which were skipped, and --json output built for CI pipelines and AI coding agents.
• A failure nobody injected. Injected faults are cleaner than real ones. That is the known limitation of any fault-injection study, and it is why this one does not stand alone. The gallery now ships examples/real_world/xtts_diverged: a 9.8-hour Coqui XTTS fine-tune that diverged on its own, in my own work, with nobody touching it. All three seeds of every configuration ship too — 18 runs at seeds 42/43/44, where before only one seed per configuration was committed — so "three seeds out of three" became a claim you can check rather than one you have to take on trust.

Update: the linter failed its own test

This article argues that a training run can look healthy while learning nothing. Auditing trainproof before a release, I found it doing exactly that — to itself.

The question was mundane. What happens to a run whose loss is exactly 0.0 on every step? That is a real failure with a boring cause: if every target label is masked to -100, cross-entropy returns zero and the model learns nothing at all. trainproof's answer was PASS.

Every loss-shape check — flat curve, divergence, no-improvement — was guarded by a > 0 test, put there to avoid dividing by zero. On an all-zero curve all three guards bailed out, silently. The run reached a passing verdict, and that verdict then named those same three checks as having run.

That is worse than missing the failure. A tool that reports "I checked this" when it did not is not a weak alarm, it is a false one — and it is the same shape as the bad_labels run above: something that looks like evidence of health and is nothing of the kind. The same audit found the hole in three more places, including a baseline comparison that rated a zero-loss run as favourable, because a loss floor of zero beats every baseline there is.

v0.12.0 is the fix, and it encodes one sentence: a skipped check is not a passed check. A PASS now names every check that ran and every check that did not, each with its reason, as structured data rather than prose. Judging a log that has no evaluation set now says so out loud:

Ran: dead-run, divergence, flat-loss, grad-spike, lr, zero-grad, zero-loss. Skipped: loader (no loader_time/step_time pair in the log); overfit (no eval_loss in the log - this run has no generalisation signal at all); step-time (no step_time column in the log).

You can read that and know what the pass is worth. Nothing about how trainproof judges a run changed — all 38 locked verdicts are byte-identical to the previous release. What changed is that it stopped overstating what it had looked at.