Benchmarks
Five controlled comparisons. Each one holds everything constant except the single thing being compared — the encoder, the model size, the kernel — and reports the spread rather than a winner, because the spread is usually the part that changes a decision.
Where several things are compared at once the p-values are corrected for it, and where a result depends on an arbitrary choice — which clips were drawn, which batch size the model actually runs at — that dependence is stated instead of averaged away. Three of these panels reversed the author’s own prediction, and one of them is most useful for the cell it refuses to quote.
6 articles in this section · all 28 articles, newest first →
Speech
One panel, held constant in the words spoken so that only the delivery and the model change.
Fourteen Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.
August 2026Eight speakers each recorded the same 1,360 sentences in six phonation states, so the words are held constant and only the delivery changes. Over one frozen list of 11,935 trials, equal error rate ran from 0.047 to 0.233 — produced by nothing but the choice of encoder, with 29 of 91 pairwise comparisons surviving Holm-Bonferroni correction. CAMPPlus is last for every speaker and every condition, and the mechanism is impostor-score compression. Parameter count does not order the panel. Ten claims were retracted along the way.
Genuine Human Speech, Called Synthetic. Nothing Was Generated.
September 2026Thirteen frozen synthetic-speech detectors, declared and hashed before any was scored. Genuine studio recordings, with the speaker, words, performance, room and microphone held fixed, put through transformations that generate no speech at all. A codec encode-decode moved 9 of 13 detectors past Holm correction, three of them fully prospective at 47 of 47 utterances. Griffin-Lim — 1984, no neural network — moved 12 of 13, so this is about reconstruction history rather than codecs. The two detector families fail on opposite inputs, so there is no best one to pick. No mechanism identified, no listening test, and thirteen of the author’s own claims withdrawn after three audit rounds.
Three of Twenty Decoders Actually Stream. My Quality Metric Was Beaten by an Algorithm from 1984.
September 2026Twenty neural audio decoders through five pre-registered gates on identical audio. Only three reproduce full-context output from partial input with load-bearing state; fifteen are stateless chunking, and the non-causal negative controls sit at exactly 1. The sole pre-registered quality metric was then beaten by Griffin-Lim — zero trained parameters — in six states of six, which the pre-registration had declared in advance would be a finding about the metric. A second measurement sharing no implementation found a near-zero median additional change for the same three arms when streamed, with no detection threshold claimed. Corrected after publication, when a fifth audit round found the advertised reproduction commands did not run. No listening test was run, so nothing here says which decoder sounds best.
Models and language
What changes when the model gets bigger, and what does not change at all.
The Size Where a Model Stops Saying "I Don’t Know"
August 2026Five local models against one fact their source text does not contain. Scaling recovered none of it. Compiling the fact worked — above a threshold, with a regression in the middle where the model stopped abstaining and started agreeing.
The Less the Text Said, the More Sure It Got
August 2026On the standard benchmark for contradictory biomedical claims, 77.1% of the flagged pairs never state what a verdict would need. Seven of eight model runs asserted contradiction more often on exactly those pairs, and instructing the model to be conservative raised its refusal rate by forty points without changing the direction at all. 5,824 judgements, all local, with an audit script that recomputes every number.
Compute
A comparison whose useful answer was the crossover point, not the winner.
I Wrote the Kernel. Then I Measured It, and at My Model's Shape It Was 1.67x Slower Than Doing Nothing.
September 2026A hand-written Triton LayerNorm against PyTorch eager and torch.compile on one GPU, at six batch sizes. Above roughly 4,096 rows the kernel wins on time and wins harder on memory — 1.36x faster at 65,536 rows using half the peak memory. Below that crossover it loses, and the model in question trains at 128 rows, where it is 1.67x slower than doing nothing. Includes the two defects the harness found in itself and one cell too unstable to quote.