All Articles
Every article published here, newest first — 27 in total. Each one covers something built and what it turned out to be wrong about.
Genuine Human Speech, Called Synthetic. Nothing Was Generated.
September 2026Thirteen frozen synthetic-speech detectors, declared and hashed before any was scored. Hold the speaker, words, performance, room and microphone fixed on genuine studio recordings, apply an ordinary transformation that generates no speech, and measure the paired change. A neural-codec encode-decode moved 9 of 13 detectors past Holm correction, three of them fully prospective at 47 of 47 utterances with rank-biserial of exactly -1.000. Griffin-Lim, a 1984 algorithm with no neural network, moved 12 of 13, so the phenomenon is about reconstruction history rather than codecs. The codec-trained and ASVspoof-era families fail on opposite inputs, so choosing a detector is choosing which false positive to accept. No mechanism is identified, no listening test was run, and thirteen of the author's own claims were withdrawn after three rounds of independent adversarial audit.
Three of Twenty Decoders Actually Stream. My Quality Metric Was Beaten by an Algorithm from 1984.
September 2026Twenty neural audio decoders, five pre-registered gates, identical audio. Only three reproduce full-context output from partial input with load-bearing state - the causal FocalCodec configurations, at state ratios of 3,530 to 36,740 against a threshold of 10 - while fifteen are stateless chunking and the non-causal negative controls sit at exactly 1. The sole pre-registered cross-arm quality metric was topped by Griffin-Lim, zero trained parameters, in six states of six; the obvious explanation was tested and refuted. A second measurement sharing no implementation, speaker-embedding retention across three separately calibrated encoders, found a near-zero median additional change for the same three arms when streamed - offline-streamed cosine 0.999866 to 0.999946 - where thirteen others lose -0.078 to -0.742 on the separate retention delta. Five independent audits found three defects that changed conclusions, and a fifth round after publication found the advertised reproduction commands did not run. Corrected in v1.0.1; no measurement changed. No listening test was run.
I Wrote the Kernel. Then I Measured It, and at My Model's Shape It Was 1.67x Slower Than Doing Nothing.
September 2026A hand-written Triton LayerNorm against PyTorch eager and torch.compile on one RTX 5080, at six batch sizes. Above roughly 4,096 rows the kernel wins on time and wins harder on memory - 1.36x faster at 65,536 rows using half the peak memory, 258.5 MB against 512.5 MB. Below that crossover it loses, and this model trains at 128 rows, where it is 1.67x slower than doing nothing. Nothing at that shape is bandwidth-bound: the cost is kernel launch, so the right optimisation is fewer launches, not a better kernel. Includes the two defects the harness found in itself and one cell too unstable to quote.
The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.
September 2026IBM's Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them, so every gate count it reports is one draw from an unmeasured distribution. On bv_n140 over 400 seeds per version across 21 processes, the real change between Qiskit 1.4.3 and 2.0.0 is +5.37% [+4.27, +6.50] — yet the suite's own three-run protocol calls it a >=10% regression 24.4% of the time [19.5, 31.1]. A disjoint 200-seed sample across ten fresh processes reproduces it at 22.6%. Twenty runs per version, about forty hours of compute, still leaves 3.74%. A pre-registered replication over 39 circuits is reported in full, along with four claims withdrawn on the record and six defects an independent audit found in the measuring apparatus itself.
I Abandoned a Study Because the Index Already Answered It. Then I Checked the Index.
September 2026A code-intelligence tool indexes your repository and answers questions about it. The obvious objection to benchmarking a language model on graph reachability is that nobody would ask a model — the index answers it exactly, in microseconds. That objection stopped a study I had already built the apparatus for. Before dropping it I ran the index. Its knowledge graph has no function-level nodes at all, so the question I was told it answered perfectly is one it does not answer. Two bugs fell out on the way, both now filed and both confirmed by the maintainer at source level. One of his answers is the part worth keeping: a field that looks like evidence for an edge carries no guarantee, and a wrong reason looks identical to a right one.
I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.
August 2026Hierarchical retrieval came third of five on a 240,767-word corpus, behind a much simpler baseline. I blamed chapter selection. A stage-level ablation over 80 questions and 400 scored rows says chapter selection cost 27 questions and neighbour expansion cost 32. Disabling that one stage raised context recall from 0.3365 to 0.4771, exactly level with the baseline it was losing to, while the hit@k columns did not move at all. Oracle-chapter routing reaches 0.7844, which bounds how much is left in the router.
Fourteen Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.
August 2026Eight speakers each recorded the same 1,360 sentences in six phonation states, so the words are held constant and only the delivery changes. On one frozen list of 11,935 trials, equal error rate ran from 0.047 to 0.233 — produced by nothing but the choice of encoder, with 29 of 91 pairwise comparisons surviving Holm-Bonferroni correction. CAMPPlus is last for every speaker and every condition, and the mechanism is impostor-score compression: it places different people at cosine 0.28-0.34 where others place them near 0.034. Parameter count does not order the panel. Ten claims were retracted along the way, including one asserted with an instrument whose artifact budget exceeded the effect.
My Headline Was +32 Points. Then I Audited My Own Scorer.
August 2026Thirty-eight questions ask whether one event precedes another when the two are narrated in different documents and share no character, place or causal link. No passage states either relation. Given the passages, every model scored 0 of 38 and refused almost every time — correctly. Given the identical facts as a structured chronology, an 8B model scored 28 of 38, matching a 14B model given prose. Then I audited the scorer that produced that and my headline lost half its size: +32 points became +15.8. The equivalence survived; the gap did not. Corrected in place, not retracted.
Two of My Instruments Were Wrong, in Opposite Directions
August 2026I audited 38 live pages and fixed some ordinary defects. The two things worth writing down both came from my own tooling being broken. One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. One date parser had thrown an exception on every crawl-statistics read since it was written, so the report that answers "is the crawler arriving at all" had never once returned a number. Repairing it showed one search engine crawling this host 15–36 pages a day and holding 34, while the other holds none. Baseline recorded, prediction made, re-measure dated.
Requesting Indexing Did Nothing, and I Have the Control Group
August 2026One of my hosts had 0 of 38 URLs in Google while four sibling hosts on the same domain, the same IPs and the same certificate indexed normally. The standard advice is to submit the URLs, so I submitted them against a held-out control: treatment 32%, control 33%. Fifteen causes eliminated with measurements, two of them my own hypotheses withdrawn — and then the impressions report showed the pages had been indexed and served all along, and lost. Locus established, mechanism unknown.
The Less the Text Said, the More Sure It Got
August 2026On the standard benchmark for contradictory biomedical claims, 77.1% of the flagged pairs never state what a verdict would need. Seven of eight model runs asserted contradiction MORE often on exactly those pairs, and instructing the model to be conservative raised its refusal rate by forty points without changing the direction at all. 5,824 judgements, all local, no API key, with an audit script that recomputes every number.
My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read
August 2026My portfolio agent holds 1,003 chunks and answers from six of them — 0.598% — and the answer reads identically to one built from reading all 1,003. That is fine for “he mentions Rust” and false for “he never mentions Rust”, from the same six chunks. Faithfulness cannot catch it: an answer perfectly grounded in what was retrieved can still be wrong about what was not. The control is the shape of the claim, not a coverage threshold.
Panagiotis (Panos) Gkilis - Machine Learning Engineer
August 2026Production ML systems, MLOps, speech and language AI. A 730M-parameter TTS model trained from scratch and served behind a live API, a grounded RAG agent behind a 32-case adversarial gate, an OpenAI-compatible model gateway, MCP tooling, seven Zenodo DOIs and four open-source libraries. Everything deployed, measured, and open to inspection.
A Hundred Per Cent Success Rate and One Usable Answer
August 2026I removed the backpressure from my own model gateway to see what would break. Thirty-two callers, thirty-two HTTP 200s, and thirty-one arrived after the caller had given up. Then I made a backend go slow instead of dying, and watched failover log eight successes without ever calling the healthy backend. Both faults produced a clean log.
The Same Bug, Four Times, Three of Them Mine
August 2026A validator that reports a verdict without reporting its coverage is asserting something it did not measure. I found that in four systems before I admitted it was one problem.
The Size Where a Model Stops Saying "I Don’t Know"
August 2026Five local models, one fact their source text does not contain. Scaling recovered none of it. Compiling the fact worked — above a threshold, with a regression in the middle where the model stopped abstaining and started agreeing.
Aether: Real-Time Tracking and Collision Screening in Rust
August 2026A two-thousand-line component that turns a live, unreliable sensor feed into a collision picture in under a millisecond per cycle, built to be embedded inside a larger system. What it is, what a system like it is used for, why the problem is hard, and one measured investigation: a fault that caused 16.1% of valid observations to be rejected, cut to 1.9% by an architectural change.
Ten Tests, Written Before the Code. The One That Failed Taught Me the Most
August 2026I wrote a finite element solver from nothing and pointed it at an answer known since 1898. It landed on 3.00002. Then a test I was sure would pass didn't, two explanations for why both collapsed, and a number the internet repeats confidently failed to reproduce.
How Little Your Voice Has to Change Before a Machine Stops Recognising It
August 2026Four speakers, eight matched sentences, three encoder architectures. One speaker moved his pitch by less than a semitone and lost a quarter of his identity score. Includes a control that could have killed the result, a prior-art audit that killed my headline claim, and a pre-registered prediction I withdrew because two defensible ways of cleaning the data gave opposite answers.
The Canon Is the Database
August 2026A fictional universe outgrew my context window. So I stopped treating it as a pile of documents and started treating it as a persistent, machine-readable knowledge system that generation has to be validated against.
Geometric Optics Lies: Reproducing a 1976 Black-Hole Calculation from Scratch
August 2026The standard shortcut overstates photon emission from a black hole by 4.16x. Six validation gates, two analytic limits the code was never tuned to, and one number I had to retract.
Why Corrupted Training Data Doesn't Show Up as High Loss
August 2026Everyone assumes bad samples announce themselves. Two failures from my own work - one reproducible, one from production - say the opposite: noise is learnable, so it hides.
One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.
August 2026My text-to-speech model trained for weeks and never learned to stop talking. The cause was two constants that happened to be equal - and no loss curve, anywhere, could have told me.
Six Things Photopea's Scripting API Does That Nobody Wrote Down
August 2026Undocumented behaviours in Photopea's Live Messaging API - measured, with the workaround for each. If your automation reports success and changes nothing, one of these is probably why.
I Use AI to Write My Books. I Wrote the First One by Hand.
August 2026Six years, six books, one fantasy world, and a stack of tooling built to stop the machine from drifting. What "AI-assisted" actually looks like when a human directs the whole thing.
Your TTS Model Sounds Great — Until It Says "GPUB"
July 2026ttsproof: automated failure-mode QA for text-to-speech, backed by a published 390-sample study.
Loss Curves Lie: Building a Deterministic Linter for ML Training Runs
July 2026A run learning pure noise reduced its loss by 62% and looked textbook-healthy. trainproof catches the training failures that can be caught — and is honest about the ones that can't.