The Number We Got Wrong, and How Long It Took Us to Notice

We published a perfect forecasting record for eight days. An audit of our own files found a prediction marked unresolved and counted as a win — and a much larger problem underneath it.

Every claim a research firm makes about itself has the same weakness: the firm keeps the scoreboard. We have been publishing ours for weeks — a record of predictions registered before the data was pulled, scored either way, misses included. It was the strongest line we had, and it was on a live page on this site.

This week we rebuilt it from source. It was wrong twice, and the second way was worse than the first.

This piece is for anyone weighing a vendor’s track record — ours or anyone else’s. The question is not whether the number is impressive. It is whether the number was derived from something you can check, by someone who would have published it either way.

The small half first. In October we ran a demand study on a recording artist and registered four predictions before pulling anything. One of them — whether a song or lyric term would outscore her name — could not be measured, because the instrument had been specified in a way that removed the term from its own count. The calibration record says so plainly: not measurable, unresolved.

A summary written the same day, in a different file, rolled the running record forward and counted that prediction as a win. Fifteen for fifteen became seventeen for seventeen. The correct figure was sixteen for sixteen.

Nobody recomputed it, because it arrived already computed. Over the following eight days that extra win traveled into our research design standard, into our publishing standard, onto a live study page, and into two articles drafted this week. Every document that repeated it cited a real source. The source had itself copied the figure.

Correcting one win would have been easy. The reason it took a rebuild is what the audit found next.

Our method separates predictions by where they came from: a prediction grounded in an earlier measurement with the same instrument is one thing, and a prediction grounded in something we merely believed is another. That split is the entire claim. Without it, a track record is just a batting average.

When we swept every project from the window, we found that 26 of roughly 33 had registered their predictions with a probability band, a falsifier and a result — and no basis label at all. Those predictions could never have entered the record, because the record is organized by a field they do not have.

So the published figure was not only one too high. It described about a fifth of the work and read as though it described all of it.

Rebuilt from the 48 predictions that carry an explicit basis: predictions grounded in a measurement held 14 of 19 times, or 72%. Predictions grounded in belief held 18 of 33, or 52%. The split survives. It is about twenty points. It is an edge, not a guarantee, and it is a great deal less impressive than the thing it replaces. Measured predictions do miss. Beliefs land about half the time, which is roughly what you would expect from beliefs.

Two things about the corrected figure are worse than they look, and both are on the ledger page rather than buried here.

One project supplies more than half the sample. A single brief registered 26 of the 48 predictions. Remove it and the gap widens to 86% against 42%. Read it alone and the gap is 64% against 62% — no measurable edge at all. That project registered 22 predictions in 73 minutes. Whether registering predictions at that speed degrades the care taken over labeling them is an open question on the page, not an explanation we have earned.

And one prediction held on a band it could not have failed. We predicted a search figure would land within 10% of the prior year, on an instrument whose own resolution at that value is about 10%. It held. It could not have done anything else informative. That is recorded as a hit with the defect stated beside it, because quietly dropping an inconvenient row is the thing this whole exercise exists to prevent.

We are publishing the weaker number for the same reason we published the stronger one. A perfect record is the least believable kind — it invites the suspicion that the scoring did the work. A record with its own audit attached is harder to dismiss, because the embarrassing parts are in it.

The corrected ledger is live, with all 48 predictions, their bands, their bases and their outcomes: every call we registered, including the ones we got wrong.