Two of My Instruments Were Wrong, in Opposite Directions

On 28 August 2026 I audited 38 live pages and fixed some ordinary defects. The two things worth writing down both came from my own tooling being broken. One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. One date parser had thrown an exception on every crawl-statistics read since it was written, so the report that answers “is the crawler arriving at all” had never once returned a number. Repairing it showed one search engine crawling this host 15–36 pages a day and holding 34, while the other holds none. I am not going to tell you why.

This is a note about measurement rather than about search engines. The setting is an indexing problem on one of my hosts, but the transferable part is that I spent a day acting on numbers produced by instruments I had never checked, and both of them were wrong — one in each direction. One made a working fix look broken. One made a working crawler invisible.

Crawling and indexing are not the same thing

Everything below depends on this distinction, and most confused writing on the subject collapses it into one word.

Crawling is retrieval. A crawler requests a URL and receives bytes. That is all it means. A page can be crawled every day for a year and appear in no search result ever.

Indexing is selection. Having retrieved the page, the engine decides whether to keep it in a form it will serve. That is a judgement, each engine makes it independently, and none of them will tell you the reasoning.

So there are at least three states worth separating: never fetched, fetched and declined, and fetched and kept. “My pages aren’t indexed” describes the middle two identically, and their causes have nothing in common. Knowing which one you are in is the first useful thing, and a crawl-statistics report is what tells you.

The audit, kept in proportion

On 28 August 2026 I fetched every URL in the sitemap — 38 pages on the affected host, 20 on this notebook — over plain HTTP with no JavaScript executed, and produced a number per page rather than a verdict. That rule exists because an earlier audit on this estate returned “all clean” for months while the homepage was handing crawlers a fraction of its navigation in a form they could use.

It found real defects. Five pages had zero followable links: a crawler arriving there could go nowhere. The worst had two navigation controls, both <button onclick> — usable with a mouse, invisible to a crawler. Three more bound their only navigation to a JavaScript click handler, which has the same effect.

On this notebook site, one change to the head template took the robots directive, og:url, og:site_name and Twitter card metadata from 0 of 20 pages to 20 of 20.

This is housekeeping. It is worth doing and it is not a finding, and I am reporting it mainly because it is the baseline for the prediction at the end.

Instrument one: a false negative that looked exactly like a failed fix

After deploying, I re-ran the audit against the live site. It reported that three of the pages I had just fixed still had zero followable links.

They did not. I fetched one by hand and the anchor was plainly in the served HTML.

The bug was mine. The script counted links by matching href values that started with / or http. The anchors I had added were relative — href="./" — so they matched nothing and counted as zero.

What makes this worth writing down is the shape of the failure. A measurement that silently under-reports is indistinguishable from the defect it is supposed to detect. Every downstream step would have been reasonable and wrong: conclude the deploy failed, go hunting for a caching problem that did not exist, possibly “fix” three pages that were already correct and introduce a real defect doing it. Nothing in the output looked like an error. It looked like bad news.

The only thing that caught it was retrieving the page and reading the markup. The script now counts every followable href, excluding only fragments and non-navigation schemes, and the corrected run is the one in the table above.

Instrument two: a report that had never returned a number

The second failure was older and much more expensive.

Bing’s webmaster API serialises dates in a Microsoft-era format — /Date(1784190745000)/, epoch milliseconds wrapped in a string. My parser handled that correctly. But one endpoint, and as far as I can tell only one, returns a second variant carrying a UTC offset: /Date(1786345200000-0700)/.

My pattern captured “digits and minus signs”. It swallowed the offset into the number, and the integer conversion raised an exception every time.

Because that variant appears in exactly one report, the failure was invisible everywhere else. Every other Bing call worked. And the one report it broke was crawl statistics — which my own notes describe as the nearest available substitute for origin access logs, and the first thing to check when asking whether a crawler is showing up.

It had never once returned a value. I had been reasoning about crawler behaviour for weeks with the most direct instrument for it throwing an exception on every call. The fix is one line; the test asserts that both offset signs resolve to the same instant, because the value before the offset is already UTC.

What the repaired report showed

Bing crawls this host every day, 15 to 36 pages, and holds 34 of roughly 38 in its index. Google holds none of them.

I had a weaker version of this already — the same host shows impressions in Bing, and I had a static count of 32 indexed. Turning a single number into a daily series changes how emphatic it is. This is not a host a major crawler visits occasionally. It is one a major crawler works through continuously, and has done throughout the period I spent trying to work out why crawlers were not engaging with it.

Sorting the evidence

Separating these categories is the only reason the rest is worth anything.

Verified directly from the live sites (2026-08-28): every row of the 38-page audit, fetched over HTTP with no JavaScript, and re-verified after deployment. The defect counts before and after. The head-template metadata going from 0 of 20 to 20 of 20.

Measured from Bing’s API (2026-08-24 to 2026-08-28): the daily crawl volume and the index count. These are one engine’s figures about its own behaviour. They are good evidence about that engine and no evidence at all about any other.

Corrected tooling: two bugs, both mine, both fixed on 2026-08-28, one now covered by a regression test. The numbers reported here are the post-correction ones. The pre-correction numbers were wrong in a way that read as a genuine finding, which is the entire point of the article.

Observation: two major engines, given the same host, the same bytes, the same permissive robots.txt and the same sitemap, have reached opposite conclusions. One indexes nearly everything. One indexes nothing.

Unresolved: why. I do not know. I did not learn it on 28 August, and I am not going to name a mechanism to round the story off. Index selection is independent between engines, so an asymmetry is permitted and is not by itself evidence of a fault anywhere.

What the asymmetry does do is retire a family of explanations. A page that a large crawler retrieves daily and chooses to keep is demonstrably reachable, parseable, renderable and acceptable to a serious indexing pipeline. Whatever remains sits inside one engine’s selection, and I have no owner-side instrument that reads it.

I want to be explicit, because the genre invites the opposite reading: nothing here fixed the zero. It was zero before the audit and it was zero after. This is not a recovery story.

The prediction, recorded before the outcome

The obvious next question is whether the fixes helped. I cannot answer that today — the changes went live on 28 August, and there is no after to compare against.

So instead of waiting and then claiming a prediction I never wrote down, here it is in advance.

Baseline, 2026-08-28. Google: 0 pages indexed on the affected host. Bing: 34 held, 15–36 crawled per day. Audit defects as in the table above, all at zero.

I will re-measure on 2026-09-15, with the same script and the same API calls, and publish the result whichever way it goes.

What I expect: no material change in either engine. The defects I fixed were real, but none of them plausibly explains a host-wide zero, and Bing was already crawling this host thoroughly while every one of them was still present — which is fairly direct evidence that they were not what stood in the way. I also ran the standard remedies against a held-out control group earlier in this investigation and measured no effect.

If the numbers move, my model is wrong and that is the more interesting outcome. If they do not, I have a dated negative result instead of a vague impression. Either is better than the version where I check first and decide afterwards what I always thought.

What I would take from this

• Check instruments against reality before trusting their output. Two of mine were wrong on the same day, in opposite directions. Both would have survived indefinitely if I had only read what they printed.
• A silent nothing is worse than a loud failure. My date parser raised an exception where the absence of a number looked like the absence of data. Make a missing measurement say missing — never zero, and never an empty report.
• Separate crawled from indexed in your own vocabulary first. If you cannot say which state a URL is in, advice cannot help you, because the fixes for “never fetched” and “fetched and declined” have almost nothing in common.
• Count links on the page, not links to it. A page with no outgoing anchors is a dead end however many things point at it, and a click handler is invisible to the thing you are trying to persuade.
• Fetch it and read it. Every real finding here came from the actual bytes. Every error came from believing a summary.
• Write the prediction down first. It costs one paragraph and it is the difference between a result and a story.

The honest summary of the day: I improved a site that was already technically sound, repaired two instruments that had been quietly lying to me, and confirmed an asymmetry I can describe precisely and cannot explain at all.