Eighteen Ways to Ask One Question

In September I argued that an AI visibility score is mostly a property of the tracker. The load-bearing evidence was somebody else’s preprint. This is my own, and it agrees on the direction, disagrees on the shape, and turned up a failure nobody has named.

Six weeks ago I published a piece arguing that the number on an AI visibility dashboard is mostly a property of the tracker rather than of your business. The argument rested on a study by Will Jack and colleagues — roughly 6,000 paraphrase trials against 6,000 same-prompt controls, showing that rephrasing a buying question in a second natural way produced a largely different set of recommended brands.

At the bottom of that piece, under the honest-limits heading, I wrote that the load-bearing source was a preprint. That was true and it was uncomfortable. A measurement argument that depends entirely on somebody else’s measurement is a book report.

So this week I built the smaller version of it on my own instrument and ran it. Three questions, six phrasings each, eighteen measurements, about eleven cents. What follows agrees with the preprint on the direction, disagrees with it on the shape, and found one thing that is not in it at all.

First, the part that looked stable

The seeds were not invented. On October 21 I ran fifteen questions a grocery buyer or distributor actually asks, sourced from retailer supplier pages, broker line-review guides and published buyer interviews. Twelve of the fifteen were led by a site almost nobody has heard of: a marketing agency, two software tools built for packaged-goods brands, a broker, a founder community.

Three of those incumbents became the seeds here, chosen because each already held first position on a real question. There was something to displace.

Asked again, in the same words, each of the three held. That is what stability looks like before you test it.

Then I changed the words

Six variations per seed, with the kinds of variation fixed before the test ran so that a failure could be traced to a type of rewording rather than to one awkward sentence: the exact original, a synonym swap, a grammatical reframe, added first-person context, a stripped-down short form, and the colloquial version a person would actually type.

This is where my result parts company with the preprint. Jack et al. found a gradient: a cosmetic rewording cost less than adding a buyer’s constraint. I found no gradient at all. Swapping one content word for a synonym did no better than writing the question the way a tired founder types it. The exact question behaves like a cliff edge, not a slope.

There is a second reading in that table and it matters more than the headline. Count presence rather than position and the picture improves sharply: the incumbent was still cited somewhere in the answer on an average of 4.67 of 6 rewordings, against 2.67 of 6 for first place.

Being in the answer survives rewording. Being the answer does not. The preprint measures overlap between recommended sets, which collapses those two into one number. Separating them changes what a content program is for.

And then one question left the category

One of the three seeds asked how grocery buyers use total distribution points — the measure of how widely a brand is already sold, weighted by the size of the stores selling it. Five phrasings returned grocery sources.

The sixth returned hp.com in first position.

"What is TDP and why does my buyer keep bringing it up?"

The colloquial variant. TDP in computing is Thermal Design Power.

Three letters mean two things. The assistant chose the meaning with more written behind it, answered confidently about processor heat, and never came back to groceries. Nothing errored. The row looks exactly like the other seventeen.

That collision was written into the test before it ran — the row carries a note, recorded at build time, saying the acronym means something else in computing and that a computing result there should be read as a collision rather than as a finding about grocery. It collided. Which is why it is a result rather than an embarrassment.

Here is the part that should bother anyone buying one of these reports. The phrasing that failed was the one a real person would use. The five that held were mine: careful, complete, industry named, written by somebody who already knew the answer he wanted. The sixth is how a founder types after a meeting where a term came up three times and they did not want to admit they did not know it.

A tidy question list is a biased question list. It is weighted toward the phrasings that behave. The messy ones are the likeliest to leave your category, and the least likely to be on anyone’s list.

Where this agrees with September, and where it does not

It agrees that wording moves the score more than the business does. That was the September argument and it survives first-party contact.

It disagrees on the shape. The preprint found decay by semantic distance. I found a cliff. If that holds up, “how far did you rephrase it” is the wrong question and “did you use the exact words” is the right one.

It disagrees on stability under repetition. Jack et al. report same-prompt rerun overlap of roughly 0.50 to 0.61 on OpenAI and Anthropic models. Asking an identical question three times on Perplexity, I get the top seven positions identical three times out of three, with movement only at position eight. That is almost certainly a platform difference rather than a contradiction, and it means a single reading on one platform is more trustworthy than the preprint’s baseline would suggest, while a single reading on another may not be.

And it adds a failure mode the preprint does not contain. Paraphrase brittleness is a question of degree. A domain collision is categorical: the answer is not worse, it is about a different industry.

Where this argument is weak

Three seeds. The variation table rests on three observations per row, which is enough to see a cliff and not enough to measure its height. One platform, one day, one generation per row, on the documented budget-tier model rather than the one a paying buyer is served.

The incumbents are other people’s pages. Our own answer pages went live the day before and have no post-publication reading, so whether a new page behaves like an established one under rewording is untested here.

And the seed whose colloquial variant collided is the weakest of the three on every other measure. Excluding that row it still scores worst, so the finding survives the exclusion — but the headline average would be wrong to quote without saying so.

Finally: none of this says how many people ask any of these questions. No instrument I trust measures question volume inside an assistant, and the one that claims to is derived from Google’s related-questions box, which inherits the exact blind spot it would be used to test.