The Size Where a Model Stops Saying "I Don't Know"

Five local models, one fact their source text does not contain. Scaling recovered none of it. Compiling the fact worked — above a threshold, with a regression in the middle where the model stopped abstaining and started agreeing.

In The Canon Is the Database I argued that a fictional universe should be treated as a persistent, machine-readable state rather than a pile of documents, and that generation should be validated against it. That was a thesis. This is the measurement, including the four places where my own measuring apparatus was wrong.

The question

A saga narrated out of order contains events whose relative timing no sentence ever states. Book 1 places a voice in a cave chamber. Book 5 places a prisoner in a cell beneath a keep. They share no character, no place, no causal link, and no passage in the corpus mentions both. A reader who has been told the chronology knows which came first. A reader with only the text does not, and neither does a retriever — semantic similarity has nothing to grip.

So: does compiling those relations into explicit state change what a model can do with the same corpus, and if so, how much model do you need before it helps?

Setup

A 260,204-word corpus, frozen as a hashed snapshot, every passage carrying character offsets that re-slice out of the source byte for byte. Questions generated automatically from pairs of events narrated in different documents, in the form "When X happened, had Y already happened?"

Three conditions, pre-registered before anything ran. Passages only — the passages the state's own evidence spans point at, handed over directly, so retrieval failure is separated from reasoning failure. Retrieved — top-k from a retriever. Compiled state — the same evidence plus a short block giving each event's position on a story-order scale.

Five model sizes from one family: Qwen3 at 0.6B, 1.7B, 4B, 8B and 14B, all local, all through the same harness with the same system prompt.

Scaling recovers nothing

Given the source passages, correct negative answers — the subset where guessing cannot help, because saying "yes" to everything scores zero on them:

0.6B  0/12    1.7B  0/12    4B  0/12    8B  0/12    14B  0/12

Twenty-three times the parameters buys nothing. That is not a surprise once stated plainly — the relation is not in the input, so there is nothing to get better at — but it is worth measuring rather than assuming, and the models mostly said so themselves. Handed the passages, they returned NOT ESTABLISHED and named what was missing, 32 and 31 times out of 34 at the two larger sizes. The baseline's own refusals are the cleanest available evidence that the fact is genuinely absent rather than merely hard to extract.

With the compiled chronology

0.6B   0/12   | 1.7B   3/12   ### 4B     0/12   |            <- regression 8B    12/12   ############ 14B   10/12   ##########

The curve is real and it is not monotonic. I predicted, in writing, that it would rise and saturate. It does neither cleanly.

The 4B is the one to worry about

At 4B both conditions return identical numbers: 12 of 34 overall, 12 of 12 on questions whose answer is "yes", 0 of 12 on questions whose answer is "no", and zero refusals. It answers "yes" to all thirty-four regardless of what you hand it. Supplying the compiled chronology changes nothing at all.

Its neighbours refuse 32 and 31 times out of 34 without the state. The 4B refuses nothing.

There is a size band in which a model loses the ability to abstain before it gains the ability to use structure. A 1.7B tells you it does not know. A 4B tells you "yes". If you are running a small local model over compiled context, that is the failure you will not see in an accuracy number, because on a key with a majority class it looks like competence.

Whether this belongs to the 4B checkpoint or to the size band is not established by one family and one task. I record it as an observation and do not explain it.

Then I corrupted the state, and learned less than I hoped

An obvious worry about all of the above: maybe the models were not using the supplied positions at all. Maybe they were guessing from book numbers, or from having read the published novels during training, or from a bare preference for answering "yes".

So I permuted the story positions so the implied ordering was reversed, and changed nothing else. Both larger models followed the false ordering — 18 of 24 and 17 of 24. Their answers tracked the numbers I gave them, wherever those numbers pointed.

That is a mechanism check, and it does its job: it rules out narration order, memorisation and a yes-prior as the source of the correct answers. The models are computing from the supplied integers.

It is not evidence that models blindly trust external state, and I initially wrote it up as though it were. The reason it cannot be is the premise of the whole experiment: no passage states the relation. So a permuted chronology contradicts nothing the model could read. The corrupted state was undetectable by construction — the models had no signal to catch it with, and the permuted block is internally consistent besides. There was no conflict to miss.

The honest bound is narrower than the one I wanted:

The compiled chronology supplies a fact the corpus does not contain, and the model transcribes it. Nothing here demonstrates temporal reasoning, and nothing here demonstrates whether a model would notice if the state were wrong in a way it could actually check.

That second question is the one worth answering, and it needs a different question set — event pairs whose ordering the passages do establish, so that a corrupted state genuinely contradicts readable evidence. Every memory and state framework I have read assumes the compiled state is correct. I have not found one that reports what happens when it is not, and after today I am not in a position to be smug about that.

Four ways my own measurement was wrong

This is the part that decides whether any of the above is worth reading, so it goes in the body rather than a footnote.

1. The answer key had no negative class. The question generator deduplicated candidate event pairs on an unordered key while reading events in story order, so every pair was first reached with the earlier event as the subject — always the "yes" branch. The "No. It had not happened yet." branch was present in the source and structurally unreachable. The original key was 14 yes, 5 simultaneous, 0 no. A model answering "yes" unconditionally scored 73.7% on it. I would have published that.

2. The scorer credited refusals as correct. It searched the whole answer for had already — which appears inside "it is not possible to determine whether this event had already occurred". Ten refusals scored as correct answers. All ten were in baseline conditions; none in a state condition.

3. The scorer required a comma. Its negative pattern matched the literal "no,". An answer opening "No. The voice speaks at position 3600, before the whisper crosses the table" matched nothing and was marked wrong. Eight of those, all in the state condition.

4. Refusals were logged as wrong orderings. The failure taxonomy recorded every NOT ESTABLISHED as wrong_order_across_books. The model never gave an order.

All four are the same missing thing: no representation for "declined". A refusal has to become either a right answer or a wrong one, and it is misfiled in both directions. I have written about this before in the context of a linter that cannot tell "your run failed" from "I could not read your log". It turns out I had built the same bug into the evaluator I was using to judge my own hypothesis.

Every one of the four corrections moved the numbers against the hypothesis I wanted to hold. It held anyway. That is the only reason I am willing to write any of this down.

What I got wrong on purpose, and on record

Each phase was pre-registered with a prediction before it ran, so the misses are checkable rather than remembered.

Limits

One corpus, one author, one snapshot. n = 34 after deduplication, which caps every accuracy claim and is the binding constraint on all of it. One model family, one quantisation tier — a threshold found in Qwen3 is a fact about Qwen3 until shown elsewhere. The story positions were assigned by a human from author testimony, so this is a claim about what an explicit chronology enables, never about extracting one automatically.

And the scramble result means this is not a temporal reasoning benchmark. It measures the capacity required to transcribe a supplied relation correctly, which is a smaller and more honest thing.

If you run a state or memory layer in front of a small model, the question I would most like answered by someone other than me: what does your system do when the compiled state is wrong? Mine believed it.