How do AI assistants get information about my product?

Through three separate channels: training data (crawlers like GPTBot and ClaudeBot collect content for model training), search retrieval (OAI-SearchBot, Claude-SearchBot, and PerplexityBot fetch pages to ground live answers), and user-triggered fetches (ChatGPT-User and Claude-User load a page when someone asks about it directly). Google's AI Overviews use the normal Google Search index. Each channel is controlled separately in robots.txt — you can allow search retrieval while opting out of training.

Why do AI assistants give wrong answers about products?

Usually because the accurate, current information wasn't retrievable when the answer was generated — so the model falls back on stale training data or third-party sources. Research from Columbia's Tow Center found AI search tools answered confidently rather than declining even when wrong. If your pricing changed or a feature shipped last month and there's no crawlable, dated page saying so, the assistant literally cannot know.

Does llms.txt improve AI search visibility?

The evidence says no, not for AI search. Ahrefs' June 2026 study of 137,000 domains found 97% of published llms.txt files received zero requests, and AI search retrieval bots accounted for about 1% of the fetches that did occur. Google has said its AI features don't use such files. Where llms.txt shows real traction is AI coding agents — Claude Code fetched llms.txt more than any AI search bot — so it's worth publishing if developers use AI tools alongside your product.

Does fresh content really get cited more by AI assistants?

There's solid correlational evidence. Ahrefs analyzed about 17 million citations across ChatGPT, Perplexity, Gemini, Copilot, and AI Overviews and found AI assistants cite content roughly 25.7% fresher than what ranks in organic search — ChatGPT's cited pages were on average 458 days newer. A regularly updated changelog is inherently the freshest page about your product, which makes it a natural citation target.

How do I check what AI assistants say about my product?

Ask them, the way a prospect would: 'What does [product] cost?', 'Does [product] integrate with X?', 'What's new in [product]?' — in ChatGPT with search enabled, Claude, Perplexity, and Google's AI Mode. Note which sources each answer cites, then fix wrong information at the cited source. Repeat monthly; answers shift as models and indexes update.

Can I stop AI companies from training on my content but still appear in AI search?

Yes — that's exactly what the separate user agents are for. Blocking GPTBot in robots.txt opts you out of OpenAI's model training while OAI-SearchBot keeps you eligible for ChatGPT search results; Anthropic's ClaudeBot (training) and Claude-SearchBot (search) split the same way. One caution: OpenAI notes robots.txt changes take about 24 hours to propagate, and blocking the search bots will remove you from AI search answers.

Accurate AI Answers About Your Product

Prospects used to google you. Now they ask ChatGPT, Claude, or Perplexity — and whatever the assistant says, confidently, becomes their first impression of your product. This is a fact-checked guide to making those answers accurate: what the crawlers actually do, what the studies actually show, and what’s mostly hype.

Here’s the uncomfortable finding to start with. In 2025, Columbia’s Tow Center for Digital Journalism tested eight AI search tools on 1,600 retrieval queries. More than 60% of answers were wrong — and the tools almost never said “I don’t know.” ChatGPT misidentified 134 of 200 test items while signaling uncertainty only 15 times. Premium tiers gave more definitive wrong answers, not fewer.

That study tested news attribution, not SaaS products — there’s no equivalent published measurement of how often assistants get product facts wrong, and we won’t pretend there is. But the mechanism it exposed applies directly: when good sources aren’t retrievable, AI assistants answer anyway. If your current pricing, your new feature, or your deprecation notice isn’t on a crawlable, dated page, the assistant fills the gap with stale training data or whatever a Reddit thread said in 2023.

You can’t control what a model says. You can control whether the truth is available, fresh, and easy to retrieve at answer time. That’s the whole game. Here’s how it works in mid-2026, step by step.

First, Understand the Three Kinds of AI Traffic

Every major AI vendor now separates its crawlers by purpose, and the distinction matters because you control each independently:

Training crawlers collect content for model training: OpenAI’s GPTBot, Anthropic’s ClaudeBot. Blocking them affects future model knowledge, not live answers.

Search/retrieval crawlers feed live AI search answers: OpenAI’s OAI-SearchBot (“sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers,” per OpenAI’s own docs), Anthropic’s Claude-SearchBot, and PerplexityBot. Blocking these removes you from AI answers — the one place you probably want to be.

User-triggered fetchers load a page because a human asked about it right now: ChatGPT-User, Claude-User, Perplexity-User. OpenAI notes these aren’t crawlers in the usual sense — “because these actions are initiated by a user, robots.txt rules may not apply.” Perplexity says the same outright.

Google is the exception: AI Overviews and AI Mode have no separate crawler. They’re grounded on the ordinary Google Search index via “query fan-out” — multiple related searches issued behind one question. Google’s stated eligibility bar is refreshingly boring: “a page must be indexed and eligible to be shown in Google Search with a snippet… There are no additional technical requirements.” (Google-Extended, the training opt-out, is a robots.txt token, not a bot.)

ChatGPT search, for the record, is built on a fine-tuned GPT-4o using third-party search providers OpenAI names as Bing and Shopify, plus direct partner content. Classic Bing SEO quietly matters again.

Step 1: Stop Blocking the Bots You Want

This is the most common failure, and it’s usually silent. Since July 2025, Cloudflare has blocked AI training crawlers by default for new domains, and new default rules distinguishing Search, Agent, and Training bots take effect for new zones in September 2026. Other WAFs and bot-management products ship similar presets. It is entirely possible — common, even — to be invisible to ChatGPT search because of a setting nobody on your team chose deliberately.

The audit takes ten minutes:

Read your own robots.txt. Check for blanket rules that catch OAI-SearchBot, Claude-SearchBot, ClaudeBot, or PerplexityBot.
Check your CDN/WAF bot settings against the vendor lists — OpenAI, Anthropic, and Perplexity all publish their user agents and IP ranges precisely so you can allowlist retrieval while blocking abuse.
Decide the training question separately. Allowing search retrieval while blocking training bots is a legitimate, supported configuration — see the FAQ below.
Grep your server logs for the retrieval bots. If OAI-SearchBot and Claude-SearchBot never appear, they’re being stopped somewhere.

Step 2: Be Indexed, Be Snippet-Eligible, Skip the Magic Files

Google published an AI-optimization guide in 2026 with a blunt mythbusting section: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” Same for special schema: structured data “isn’t required for generative AI search… there’s no special schema.org markup you need to add.”

Translation: for the Google side of AI visibility, the work is ordinary, unglamorous SEO — indexed pages, sensible titles, snippets enabled. If your changelog already ranks, you’re most of the way there for AI Overviews.

One genuinely new note from that same guide: browser-using agents read pages by “analyzing visual renderings, inspecting the DOM structure, and interpreting the accessibility tree.” Semantic HTML — real headings, real lists, alt text — now serves a second constituency beyond accessibility.

Step 3: Keep a Fresh, Dated, Canonical “What Changed” Page

The strongest data point in AI citation research is freshness. Ahrefs analyzed ~17 million citations across ChatGPT, Perplexity, Gemini, Copilot, and AI Overviews: AI assistants cite content on average 25.7% fresher than what ranks in organic search — the median AI-cited page was about a year newer, and ChatGPT skewed freshest at 458 days newer. (Correlation, not proven causation, and the authors warn against faking update dates. Noted.)

Think about which page on your site is updated most often, carries a real date on every entry, and states plainly what’s true about your product now. It’s your changelog. A well-maintained changelog is structurally everything this data favors: perpetually fresh, dated per entry, factual, and specific. It’s also the only page that can answer the question assistants get about every product: “what’s new in X?”

The requirements to make it retrieval-friendly are the same ones that make it work for AI coding tools: stable URLs per entry, dates in the visible text, server-rendered HTML (retrieval bots don’t reliably execute your JavaScript), and ideally a Markdown version at a stable URL — Markdown survives the extraction pipeline better than div soup.

Step 4: Write So You Can Be Quoted

The one peer-reviewed study on influencing generative-engine answers — “GEO: Generative Engine Optimization” (Aggarwal et al., KDD 2024) — tested content modifications against a benchmark of generative engines and found visibility improvements of up to 40%. What worked: adding citations, quotations, and statistics. What didn’t: keyword stuffing.

Honest caveats before you act on that: the experiments ran on 2023-era engines, used the authors’ own visibility metrics, and “up to 40%” is a benchmark best case, not a forecast for your site. But the direction is intuitive and costless to follow — an assistant assembling an answer needs specific, attributable statements. “ReleasePad’s Pro plan is $35/month per product, with no per-seat fees” can be lifted into an answer verbatim. “Flexible pricing for teams of every size” cannot. Audit your pricing page, homepage, and docs for sentences a machine could quote without interpretation.

Step 5: Fix the Sources Assistants Actually Cite

Profound analyzed 680 million AI citations from August 2024 to June 2025. Within each platform’s ten most-cited sources, Wikipedia dominates ChatGPT (47.9% of top-ten citations) and Reddit dominates Perplexity (46.7%), with Reddit also leading Google AI Overviews’ top sources. Assistants don’t just read your website — they triangulate from the sources they trust.

So the monitoring loop is: ask the assistants what a prospect would ask (“what does X cost,” “does X do Y,” “best tools for Z”), note which sources get cited, and correct at the source. Wrong pricing in a comparison blog post that Perplexity loves? That post is now your problem. Outdated Wikipedia claim? Fix it (with citations, per Wikipedia’s rules). A Reddit thread as the top answer about your category? That’s a reason to have an accurate, current presence in those conversations — not spam, just the correct numbers where the assistants are reading. And make your own pages the easiest primary source to verify against: specific, dated, quotable.

The llms.txt Reality Check

You’ll be told the fix for all of this is publishing an llms.txt file. Here’s the honest status as of July 2026.

The proposal (Jeremy Howard, September 2024) is a curated Markdown index at /llms.txt telling AI systems where your important content lives. Real companies publish one — Stripe, Zapier, Vercel, Cloudflare, Supabase, Anthropic’s docs site, Cursor’s docs. We publish one too, and wrote the guide on it.

But measure the other side. Ahrefs’ June 2026 log-analysis of 137,000 domains — the largest study to date — found 97% of published llms.txt files received zero requests at all. Of the fetches that did happen, AI search retrieval bots (OAI-SearchBot, PerplexityBot, Claude’s search crawler) accounted for roughly 1%. Google’s John Mueller compared the file to the keywords meta tag back in 2025, and Google’s official guidance (above) says its AI features don’t use such files. No major AI platform has announced support.

The study found one audience that does fetch it, though: AI coding agents. Claude Code out-fetched every AI search bot in the dataset. Mueller’s own 2026 refinement matches: llms.txt is useful as “a temporary crutch” for AI tools parsing developer documentation. So the honest recommendation: if developers use AI tools with your product — reading your docs, integrating your API — publish llms.txt for that audience, and it costs you an hour. Just don’t expect it to move ChatGPT search visibility, because the best available data says it won’t.

Set Expectations, Then Do the Work

Two numbers to hold simultaneously. Cloudflare’s crawl-to-refer data shows AI platforms crawl vastly more than they send back — as of mid-2025, roughly 1,100 OpenAI crawls per referred human visit, and tens of thousands for Anthropic. AI visibility is not a traffic firehose. But the referrals that do come are growing fast (Similarweb measured ChatGPT referrals to news sites growing 25x year over year) and arrive unusually late in the decision — the person already asked the question and got your name in the answer.

The checklist, in priority order:

Retrieval bots reachable — robots.txt, WAF, and CDN checked against vendor IP lists
Key pages indexed in Google and Bing, snippets enabled
Changelog current, dated, server-rendered, one URL per entry
Pricing and feature pages written in quotable, specific sentences
Monthly spot-check of ChatGPT, Claude, Perplexity, and AI Mode answers about your product, fixing errors at the cited source
llms.txt published — filed under “cheap, serves coding agents,” not “AI SEO strategy”

Most of this list is the same discipline it’s always been: publish the truth about your product, somewhere crawlable, and keep it current. The assistants are new; the work, mostly, isn’t. A changelog tool that publishes machine-readable output automates the part of this you’d otherwise forget — because the freshest page about your product only helps if it actually stays fresh.

How to Make AI Assistants Give Accurate Answers About Your Product