subinternal Talk to us

ResearchGEO, crawlers

Is your site keeping AI out?

9.1% of the sites we read turn every crawler away, and 4% of the rest turn away AI crawlers by name. Some of that is a choice. A lot of it is a line nobody remembers adding.

How many sites keep AI out?

About one in eleven. Of the 316,711 sites mttr read, 28,914 (9.1%) turn crawlers away entirely, through robots.txt or their server, so no search engine or assistant can read them.1 Of the 287,797 that are open, another 4.2% let search engines in but turn AI crawlers away by name in robots.txt.

Blocking AI crawlers is a reasonable choice for a publisher protecting its writing. For a dentist or a roofer who wants to be named when someone asks an assistant for one nearby, it usually isn’t a choice at all, just a line a theme, a plugin or a hosting setting added.

Which crawler does what?

Most AI companies now run separate crawlers for separate jobs, and each one can be allowed or blocked on its own. The split that matters is search against training: one fetches your pages so an assistant can answer with them and link to you, the other collects text to train future models.

CompanyFor search answersFor trainingWhen a person asks
OpenAIOAI-SearchBotGPTBotChatGPT-User
AnthropicClaude-SearchBotClaudeBotClaude-User
PerplexityPerplexityBotNot used for training, by its own accountPerplexity-User
GoogleGooglebotGoogle-Extended, for Gemini and grounding

OpenAI says each of its settings is independent: blocking GPTBot doesn’t keep you out of ChatGPT search, which uses OAI-SearchBot.2 Google says Google-Extended doesn’t affect a site’s inclusion or ranking in Google Search, which Googlebot handles.3 Anthropic separates training (ClaudeBot) from search (Claude-SearchBot) and user requests (Claude-User).4 Perplexity says PerplexityBot surfaces sites in its search and isn’t used to train models.5 The crawlers that fetch a page because a person asked for it can behave differently around robots.txt; check each company’s own page.

Should you block AI crawlers?

Block training if you want to, and allow search. If you want assistants to recommend you, the search crawlers have to be able to read you. If you’d rather your writing didn’t train future models, block the training crawlers by name. The two are separate, so you don’t have to choose one answer for both.

How do you check your own site?

  1. Read your robots.txt. It’s at your domain followed by /robots.txt. Look for User-agent lines naming the crawlers above, followed by Disallow: /. Those lines are the whole rule.6
  2. Check your hosting and CDN settings. Some offer a switch that blocks AI crawlers, and it can be on without anyone deciding to turn it on.
  3. Check the words arrive without script. 36% of open sites have a page whose words only appear once a script runs. Many crawlers don’t run scripts, so a page can be allowed and still read as empty.
  4. Run a check. The free AI visibility check shows whether mttr found your site open, and what it could read.

Do you need an llms.txt?

It helps, but it isn’t what decides whether you’re read. An llms.txt is a short file that outlines a site for language models: what it is, and which pages matter.7 78% of sites don’t have one. It’s quick to add and worth doing once the basics are right, but no llms.txt makes up for a site the crawlers can’t get into.

References

  1. subinternal Research, “How 316,711 websites do with AI search,” October 2026. subinternal.com
  2. OpenAI, “Overview of OpenAI crawlers.” developers.openai.com
  3. Google Search Central, “Google’s common crawlers,” on Google‑Extended. developers.google.com
  4. Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?” support.claude.com
  5. Perplexity, “Perplexity crawlers.” docs.perplexity.ai
  6. IETF, “RFC 9309: Robots Exclusion Protocol,” September 2022. rfc-editor.org
  7. “The /llms.txt file,” a proposal for a file that outlines a site for language models. llmstxt.org