A small yoga studio owner asks ChatGPT where to find beginner yoga in her city. It confidently lists three studios. Hers isn't one of them, even though her site works and she's on page one of Google. Most of the time AI search misses a site for one of five technical reasons: robots.txt or your host's security keeps the crawlers out, JavaScript draws the text, the page isn't in Google's index, or the crawler keeps hitting broken links. You can check all five in a couple of minutes.
Why does this matter now? More people get their answer without visiting any site. Pew Research Center found in 2025 that when Google shows an AI summary, people click a regular link in 8% of visits, versus 15% without one. If you're not inside the answer, you're close to invisible. For more ways a site quietly loses customers, see our piece on seven silent website leaks.
How AI finds websites in the first place
AI search works a lot like regular search. A crawler (a program that visits sites and reads pages) collects text, and then the model builds an answer from it and adds a link. If the crawler couldn't read your page, the model doesn't know it exists.
The catch is that every AI company runs several crawlers, and they do different jobs. Some collect text to train models. Some collect it for search. Others visit a page only when a specific user asks. That's where the first problem starts.
Reason 1. Your robots.txt blocks the wrong crawler
Robots.txt (a small file at the root of your site with rules for crawlers) got edited on a lot of sites in 2023 and 2024, when "blocking AI" was the trend. Many people blocked everything. But the crawlers are very different:
- OpenAI: GPTBot collects text for training, OAI-SearchBot for ChatGPT search. OpenAI's docs say sites that block OAI-SearchBot won't appear in ChatGPT search answers. Blocking GPTBot doesn't change search visibility on its own.
- Anthropic (Claude): ClaudeBot is for training, Claude-SearchBot for search. Blocking the second, the company says, may reduce your visibility in Claude's search answers.
- Perplexity: PerplexityBot is what gets your site shown and linked in Perplexity results. The company says it isn't used to train AI models.
- Google: there's no separate crawler for AI Overviews. They're built on the regular Googlebot index. Google's docs say blocking Google-Extended doesn't affect your inclusion in Search; it covers training and Gemini answers.
So here's the rule of thumb. If you don't want models trained on your writing, block the training crawlers. If you want to show up in answers, keep the search crawlers open. You can check your file and build a correct one with the AI robots.txt generator.
Sometimes robots.txt is fine and the crawlers still can't get in.
Reason 2. Your host's security blocks the crawlers
Many sites sit behind a security service such as Cloudflare. On 1 July 2025 Cloudflare changed how it treats AI crawlers. Every new domain is now asked at sign-up whether to allow them, and the starting setting is to block. Cloudflare called it Content Independence Day.
The reasoning makes sense. By Cloudflare's estimate, getting traffic from OpenAI is 750 times harder than it was from the Google of old, and from Anthropic 30,000 times harder. But if you connected a domain after July 2025 and don't remember what you picked, check your settings. You may have shut out AI search without noticing. Cloudflare says you can allow access at any time.
Reason 3. The text only shows up after JavaScript runs
Many modern sites send the browser an empty shell and let a script fill in the text. People don't notice, since the page appears in a second. AI crawlers don't run scripts, though. In December 2024 Vercel, a large web hosting platform, looked at hundreds of millions of requests from GPTBot, ClaudeBot and PerplexityBot. Its finding: none of the major AI crawlers execute JavaScript. They download the script files but never run them.
Picture mailing a friend a flat-pack box that says "assemble yourself," when they can only read finished letters. They see an empty box. That's how a crawler sees a site built entirely in the browser: a title, and no text.
How to check: open your page, press Ctrl+U (view source) and search for a sentence from your main text. If it's there, crawlers can see it. If not, a script is drawing it, and you'll need a developer to send the page fully built from the server.
Reason 4. The page isn't in Google's index
For Google's AI Overviews the rule is simple. Google's docs say a page must be indexed and eligible to appear in Search with a snippet to show up as a supporting link. There are no extra requirements.
So anything that blocks regular search blocks AI too: a noindex tag (an instruction not to index the page) left over from development, a nosnippet tag (don't show a description), or a section closed off in robots.txt. You can check whether Google sees a page in Search Console, or just search for site:yoursite.com.
Reason 5. Crawlers keep hitting broken links
In the same Vercel study, about 34% of requests from OpenAI's and Anthropic's crawlers landed on 404 error pages, meaning addresses that don't exist. Crawlers follow old links, renamed pages and sitemap entries that were deleted long ago. Each of those visits is a missed chance to read a real page.
The fix: keep your sitemap up to date, redirect old addresses to new ones, and scan for broken links once a month.
What about llms.txt?
llms.txt is a file proposed in 2024 that gives AI models a compact summary of your site. Lots of guides call it step one. But Google says plainly that you don't need any new machine-readable files, AI text files or special markup to appear in AI Overviews or AI Mode.
It won't hurt, and you can make one in a minute with the llms.txt generator. Just treat it as an extra, not a substitute for the five fixes above. No llms.txt will rescue a site the crawler can't read.
How to check everything at once
Checking all five by hand takes a while. Awe Check does it in about a minute, for free. It opens your site like a crawler would, reads your robots.txt, checks whether text is visible without JavaScript, and looks for noindex tags and broken links. A dedicated AI search section in the report shows whether ChatGPT, Claude and Perplexity can see you. The AI search visibility guide explains what to do about each finding.
Once crawlers can reach your site, there's one step left: give them something worth quoting. AI tools prefer short paragraphs where a question and its answer sit side by side, like "How much is a first class? $10, mats included." If you're new to these assistants, our beginner's guide to ChatGPT, Claude and Gemini is a good start.
Remember the yoga studio that didn't make the list? Start where she would. Open your site, press Ctrl+U and look for the first sentence of your home page in the code. It takes a minute and tells you right away whether AI can see you.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Claude's crawlers
- Perplexity: PerplexityBot and Perplexity-User
- Google: AI features and your website
- Google: common crawlers and Google-Extended
- Cloudflare: Content Independence Day (2025)
- Cloudflare press release on AI crawler control (2025)
- Vercel: The rise of the AI crawler (2024)
- Pew Research Center: clicks when Google shows AI summaries (2025)
