Why AI Brand Recommendations Change From Run to Run

There Is No Single Permanent Recommendation List

When an AI assistant recommends three vendors today and a different three tomorrow, it is tempting to assume that one brand gained or lost a hidden ranking. That is usually the wrong mental model. The system generates an answer for the question, context, tools, and information available in that particular run. A shortlist is an output, not a stable league table stored somewhere behind the interface.

Some variation is fundamental to generative models. OpenAI describes model behavior as inherently non-deterministic in its API material, which means repeated generations can differ even when inputs look similar. Consumer products add more moving parts, including conversation history, personalization, live search, and product updates. The API documentation illustrates the underlying issue; it does not disclose the exact settings used by ChatGPT or any other consumer assistant.

The Six Variables That Commonly Change The Answer

A changed recommendation can come from several layers at once. Before treating it as a market signal, identify which layers your test actually held constant.

• Prompt wording: “best accounting software” invites a different judgment from “best accounting software for a five-person UK consultancy that needs multi-currency invoicing.”
• Conversation context: a brand named earlier in the chat, attached files, project instructions, or account preferences can shape the next answer.
• Personalization: ChatGPT memory can inform responses and search-query rewrites, while Claude offers profile and project preferences that provide additional context.
• Tool state: a search-enabled answer can draw on current web results, while an answer without search may rely on different available context.
• Retrieval path: ChatGPT Search may rewrite a request into one or more targeted queries, and Google documents query fan-out for its generative search features. Different retrieval paths can expose different sources.
• Generation and product changes: sampling can change which qualifying option is expressed, while model, index, interface, and ranking-system updates can change behavior over longer periods.

Build A Test That Controls What You Can

A useful monitoring run starts with a written protocol. Keep the exact prompt, language, country context, account state, search mode, and model or product label where visible. Run each prompt in a new conversation so a brand mentioned in one answer cannot leak into the next test. If the product offers a temporary or non-personalized chat, document whether you used it rather than mixing those results with ordinary account sessions.

Repeat each prompt several times in the same measurement window. Three to five runs will not make the system deterministic, but they expose whether one result was an outlier. Save the complete answer, not only the brand list: ordering, rationale, caveats, citations, and the date are all part of the evidence.

• Freeze the prompt text and punctuation for the baseline series.
• Separate search-enabled and non-search runs instead of blending them.
• Use the same locale, language, and buyer constraints in every comparison.
• Record visible model or product labels because platforms change over time.
• Keep retries as new observations; do not replace an inconvenient first result.

A Worked Example: One Buyer, Three Reasonable Shortlists

Imagine the baseline prompt is: “Which payroll tools should a 25-person UK agency consider?” One run may emphasize ease of setup and recommend three mainstream options. A second may infer that the agency employs contractors and add a tool known for international payments. A third may use current search results and prioritize vendors whose UK payroll documentation is easier to verify. Those answers can differ without directly contradicting one another because the prompt left important selection criteria unstated.

Now add stable constraints: monthly budget, required pension integration, number of contractors, and whether managed payroll is needed. The list may become more consistent because the decision space is narrower. That does not make the new list objectively correct; it makes the evaluation question clearer. Prompt specificity improves comparability only when the constraints reflect a real buyer rather than conditions invented to force a preferred brand into the answer.

Measure A Pattern, Not A Screenshot

A single appearance is anecdotal. A monitoring series turns the answers into descriptive metrics. Keep branded and unbranded prompts separate, and calculate each metric within a defined prompt set and time window. Otherwise a new batch of easier prompts can create an apparent improvement that has nothing to do with visibility.

• Mention rate: the share of eligible runs in which the brand appears at all.
• First-mention rate: the share of ordered answers in which the brand appears first; use only when the answer genuinely presents an order.
• Shortlist stability: how often the same set of brands recurs across repeated runs.
• Rationale consistency: whether the same strengths, weaknesses, and use cases are attached to the brand.
• Source recurrence: which cited pages or domains appear repeatedly in search-grounded answers.
• Volatility: how widely these values move among repetitions within the same window.

How To Tell Noise From A Meaningful Change

Treat movement as more credible when it persists across multiple measurement windows, appears in several related prompts, and has a plausible external explanation. A new product launch, corrected pricing page, major independent review, or platform update gives you something to investigate. One missing mention in one retry does not.

Use a simple change log beside the results. Record changes to your site, third-party coverage, product positioning, prompt set, platform settings, and visible model versions. When a metric moves, the log prevents a neat but unsupported story from replacing evidence. If the protocol changed, start a new baseline rather than comparing unlike periods.

What This Testing Cannot Prove

Controlled prompting estimates what happened in your chosen test environment. It does not reveal how often real buyers ask the question, what every user sees, how a provider's internal systems rank sources, or whether a mention caused a purchase. Personalized users can receive different responses, web results change, and platform operators can update products without preserving your baseline conditions.

Use AI recommendation tracking as one research signal alongside customer interviews, pipeline data, search demand, referrals, and direct traffic. The honest conclusion is usually probabilistic: a brand appeared more often in this prompt set under this protocol. That is useful. Claiming a universal AI ranking from the same evidence is not.