I Abandoned a Study Because the Index Already Answered It. Then I Checked the Index.

A code-intelligence tool indexes your repository and answers questions about it. The obvious objection to benchmarking a language model on graph reachability is that nobody would ask a model — the index answers it exactly, in microseconds. That objection was strong enough to stop a study I had already built the apparatus for. Before dropping it I ran the index. Its knowledge graph turned out to have no function-level nodes at all, so the question I was told it answered perfectly is one it does not answer. Two bugs fell out on the way. Both are now filed, both were confirmed by the maintainer at source level, and one of his answers is the part worth keeping.

This is a note about checking a premise, and about a specific failure mode that has nothing to do with the tool it was found in: a system can hand you a correct answer attached to a reason that is fabricated, and give you no way to tell.

The premise that stopped the study

I had built a benchmark apparatus to ask whether a language model, handed a code graph as text, can determine that one function is reachable from another. Gold computed by traversal, contamination controls, a permuted-graph control, a closed-book positive control. No model had been called yet.

Then the objection arrived, and it was a good one. Nobody asks a language model whether X is reachable from Y. A code index answers that exactly and instantly. Whatever number the study produced, positive or negative, it would change nobody's decision — because the capability is already solved by a lookup.

That is a study-killing objection and I accepted it. But it rests on a factual claim about what an index does, and I had not checked the claim. So before dropping the work I installed octocode 0.23.1, indexed eighteen small Python files, and asked its knowledge graph a reachability question.

The graph has no function-level nodes

Loaded GraphRAG knowledge graph with 18 nodes and 4 relationships Node Types:          - file: 18 nodes Relationship Types:  - calls: 4 relationships

Eighteen nodes for eighteen files. Every node is a file. There is no node for a function, and therefore no calls edge between two functions.

So "is dispatch reachable from main" is not answered slowly or approximately by this graph. It is outside what the graph represents. The control file in my corpus containing the textbook chain a → b → c → d — four functions, no dynamism, deliberately trivial — returns no relationships at all, because every edge in it is intra-file and a file-node graph has nowhere to put those.

The premise that killed the study was false for the tool it was about. That does not automatically revive the study, and I will come back to why. It does mean the objection has to be re-argued rather than assumed.

The first bug: a function that calls itself resolves to someone else

While mapping what the graph did represent, an edge appeared that could not be right. Two files, no imports between them, connected by calls.

The minimal reproduction is five lines:

mkdir demo && cd demo
printf 'def target():\n    return 1\n\ndef main():\n    target()\n' > a.py
printf 'def target():\n    return 2\n' > b.py
octocode config --graphrag-enabled true
octocode index --no-git
octocode graphrag get-relationships --node-id a.py

This reports calls → b (b.py): a calls target. But a.py defines target() three lines above the call site and never imports or mentions b.py. The call binds locally. The edge points somewhere else.

My first write-up said the trigger was a name collision. An independent audit of my own work showed I had never varied the thing that actually controls it. Same package, one variable changed:

The local definition is never the answer. With one competing definer, that file wins. With two or more, the resolution abstains and the valid local call is dropped entirely. Package structure changes nothing — I had briefly believed it did, and that was wrong too.

Muvon, octocode's maintainer, confirmed it and named the line: select_scoped_targets in src/indexer/graphrag/relationships.rs filters the source file out of the candidate list in both passes, so a file can never resolve against itself. Suppressing the self-edge at the call site is correct. Falling through to the cross-file pass when the symbol already binds locally is not.

The second bug, and the part that generalises

In a small package, two files each import a third. api.py imports both core.base and utils.auth. auth.py imports only core.base. Nothing imports in the other direction.

api.py  → auth.py   "Imports ..core.base from app\utils\auth.py"
auth.py → api.py    "Imports app.core.base from app\handlers\api.py"

Two problems sit on top of each other.

The reverse edge does not exist in the source. auth.py → api.py is not an import in any form. The two files share an import of base, and that appears to have produced a link between the importers.

And the descriptions name the wrong files. The first says auth.py imports ..core.base, but auth.py contains no relative import at all. The second says api.py imports app.core.base, but api.py's import of base is the relative form. Each string attributes the other file's syntax to the file it names. The maintainer's diagnosis: the description is built as format!("Imports {} from {}", import, target_id), pairing the source file's import string with the target's name.

Now the part I actually care about. api.py → auth.py is a genuine import. That edge is correct. Its stated reason is not — the reason names a module that has nothing to do with why those two files are connected.

So the output contains a true fact justified by a false one, and nothing distinguishes that from a true fact justified by a true one. Score the endpoints and it passes. Read the reasons and it does not.

What the maintainer said, which is the most useful sentence in this whole exercise

I asked him directly whether the explanation string is meant to be authoritative evidence for the edge or a descriptive label. His answer:

Treat that string as a descriptive label. Nothing in the code makes it authoritative, so an agent reasoning from it is reasoning from a promise we never made.

That is a maintainer stating plainly that a field which looks like evidence is not evidence, and that the guarantee a consumer might infer was never offered. He is right, and he is right in a way that indicts the consumer rather than the tool.

It generalises immediately. Retrieval systems return a passage and a relevance score. Agents return an answer and a citation. Knowledge graphs return an edge and a justification. In every case the second field is easy to display, easy to log, easy to feed into a downstream model — and in every case it may carry no promise whatsoever. The failure is not that the reason is wrong. It is that a wrong reason and a right one look identical.

Both findings are filed: issue #85 and issue #86.

Why both bugs were in Python, which the maintainer volunteered

Python and markdown are the only two language modules in octocode's tree with no test file beside them. Python is the least-exercised parser they ship.

I would not have known that, and it changes how the findings should be read. They are not evidence that the graph is broadly unreliable. They are evidence about the least-covered parser in a multi-language tool, found by pointing adversarial input at it. That is a smaller claim and it is the true one. He gave it up unprompted, against his own interest, which is worth recording.

Three things I got wrong on the way here

I claimed the collision alone was the trigger. Every variant in my ladder had exactly one competing file, so the count was never varied and the claim was wider than the evidence. Corrected after an independent audit of my own harness.

I claimed the counts were a floor on what any static indexer misses. Not earned. Tree-sitter is a parsing library; an application built on it can add any resolution it likes. The floor holds for the resolution strategy I tested and transfers to nothing else without measurement.

A second agent auditing my harness introduced a defect while fixing another one — it gave the tracer a node name the static reader could never produce, and an ordinary list comprehension began reporting two missed edges. Code every analyser on earth handles correctly. Caught by a third pass, which is the entire argument for having one.

What this does not show

Not that octocode's file-level graph is inaccurate in general. Eighteen single-file cases with almost no imports is a deeply unfair corpus for a graph whose nodes are files, and its accuracy on a real repository is unmeasured.

Not that the tool is bad at its job. Its job is helping an agent find code — semantic search, structural AST search, signatures, LSP. I exercised two of those surfaces and can say nothing about the rest.

Not that the abandoned study is revived. The premise that killed it was false for this tool, but the study had a second, independent problem: on real Python a static call graph misses dynamic dispatch, decorators, callbacks and getattr, so a model that genuinely understands the code disagrees with the parser and is marked wrong — biasing a model-size comparison toward the result I had predicted. That one is still fatal and no amount of checking premises fixes it.

The thing I would take from this

The objection that stopped the study was correct in form and false in fact, and it cost nothing to check. One install, one index, one query.

I nearly did not run it. The argument was good, it came from someone who knew the domain, and accepting it felt like discipline rather than laziness. The cheapest measurement in this entire exercise was the one I almost skipped because a plausible sentence had already answered it.