Research
Thirteen studies. Some reproduce a result with a known answer to find out what a fresh implementation gets wrong; some ablate a system the author built to find out which stage is actually costing him; some are audits of the instruments themselves, which is where several of the more uncomfortable findings came from.
What they have in common is the reporting standard rather than the subject. Each one states what was measured, over how many trials, with the correction applied where multiple comparisons were made — and each one records the claims that did not survive. Several headline results on this page are smaller than the first version of themselves, and two were withdrawn outright. Those are left visible on purpose: a page that only shows the claims that held is not a research record, it is a highlight reel.
13 articles in this section · all 25 articles, newest first →
Speech and training runs
What a model does to a voice, and what a loss curve fails to say about the run that produced it.
How Little Your Voice Has to Change Before a Machine Stops Recognising It
August 2026Four speakers, eight matched sentences, three encoder architectures. One speaker moved his pitch by less than a semitone and lost a quarter of his identity score. Includes a control that could have killed the result and a pre-registered prediction withdrawn because two defensible ways of cleaning the data gave opposite answers. Superseded in scale by the fourteen-encoder benchmark; the two studies are separate and their numbers are never combined.
One Integer Deleted the Stop Token From My Loss. The Curve Never Noticed.
August 2026A text-to-speech model trained for weeks and never learned to stop talking. Two constants happened to be equal, the stop token fell out of the loss, and no loss curve anywhere could have shown it.
Why Corrupted Training Data Doesn't Show Up as High Loss
August 2026Bad samples are assumed to announce themselves as high loss. Two failures from production say the opposite: noise is learnable, so it hides.
Retrieval and agents
Systems whose correctness depends on what they did not read. Every study here measures the part that is missing rather than the part that was returned.
I Assumed My Retriever Failed at Stage One. The Bigger Failure Was at Stage Three.
August 2026A stage-level ablation over 80 questions and 400 scored rows. The stage blamed for the loss was not the stage responsible: disabling neighbour expansion raised context recall from 0.3365 to 0.4771, level with the baseline it was losing to, while hit@k did not move at all.
My Agent Answers From 0.6% of Its Corpus and Reports It Like a Full Read
August 2026An agent holding 1,003 chunks answers from six of them, and the answer reads exactly like one built from reading all 1,003. Faithfulness cannot catch it: an answer perfectly grounded in what was retrieved can still be wrong about what was not.
My Headline Was +32 Points. Then I Audited My Own Scorer.
August 2026Thirty-eight questions whose answer no passage states. Every model scored 0 of 38 on the prose and correctly refused; an 8B model given the same facts as a structured chronology scored 28 of 38. Then an audit of the scorer cut the headline from +32 points to +15.8 — corrected in place, not retracted.
I Abandoned a Study Because the Index Already Answered It. Then I Checked the Index.
September 2026A study abandoned because a code-intelligence index was said to answer the question exactly. Checking the index first found it has no function-level nodes at all, so it does not answer that question. Two bugs filed, both confirmed by the maintainer at source level.
Instruments, search and indexing
Measurements of the measuring apparatus. Four cases where the instrument, not the system under test, was the thing that was wrong.
Two of My Instruments Were Wrong, in Opposite Directions
August 2026An audit of 38 live pages where the two findings worth keeping both came from the tooling being broken: one script reported fixed pages as broken, and one date parser had thrown on every crawl-statistics read since it was written, so the report answering "is the crawler arriving at all" had never once returned a number.
Requesting Indexing Did Nothing, and I Have the Control Group
August 2026One host with 0 of 38 URLs in Google while four sibling hosts on the same domain, IPs and certificate indexed normally. The standard advice — submit the URLs — was run against a held-out control: treatment 32%, control 33%. Fifteen causes eliminated with measurements, two of them the author’s own hypotheses withdrawn.
The Compiler Got 5% Slower. The Benchmark Called It a 10% Regression a Quarter of the Time.
September 2026A benchmark suite pins the seeds that build its circuits and passes no seed to the code that compiles them. The real change between two compiler versions is +5.37% [+4.27, +6.50], yet the suite’s own three-run protocol calls it a regression of 10% or more 24.4% of the time. A pre-registered replication over 39 circuits, four claims withdrawn on the record, and six defects an independent audit found in the measuring apparatus itself.
Six Things Photopea's Scripting API Does That Nobody Wrote Down
August 2026Six undocumented behaviours in a scripting API, each measured with the workaround for it. If your automation reports success and changes nothing, one of these is probably why.
Physics reproductions
Two results with a known answer, rebuilt from nothing to find out what the rebuild would get wrong.
Ten Tests, Written Before the Code. The One That Failed Taught Me the Most
August 2026A finite element solver written from scratch and pointed at an answer known since 1898. It landed on 3.00002. Ten tests were written before the code; the one that failed taught the most, and a number the internet repeats confidently failed to reproduce.
Geometric Optics Lies: Reproducing a 1976 Black-Hole Calculation from Scratch
August 2026A 1976 black-hole emission calculation reproduced from scratch. The standard geometric-optics shortcut overstates photon emission by 4.16x. Six validation gates, two analytic limits the code was never tuned to, and one number retracted.