How many sites keep AI out?
9.1% of the sites we read keep crawlers out entirely: 28,914 of 316,711. Their robots.txt or their server turns crawlers away, so search engines and AI assistants can't read them at all. Of the 287,797 that are open, another 4.2% let search engines in but turn AI crawlers away by name in robots.txt.
That second group is worth a look. Blocking AI crawlers is a reasonable choice for a publisher protecting its writing. For a dentist or a roofer who wants to be named when someone asks an assistant for one nearby, it may not have been a choice at all, just a line a theme or a plugin added.
How well can AI read a site?
The middle open site scores 78 of 100. 12.3% score 90 or more, and 1.6% score under 50. A score is four stages weighted together: access (can it be fetched and read), understand (does each page say what it is), trust (does it show who stands behind it), and act (is there a clear next step).
| Score | Sites | Share |
|---|---|---|
| Under 50 | 4,554 | 1.6% |
| 50 to 69 | 82,197 | 28.6% |
| 70 to 79 | 74,226 | 25.8% |
| 80 to 89 | 91,475 | 31.8% |
| 90 and up | 35,345 | 12.3% |
Most sites clear the basics. They load, they can be indexed, and their words arrive in the HTML. The points are lost further along, in whether the pages answer anything.
What do most sites miss?
Answering, more than anything. Of the 99,109 sites with guides or articles, 99.2% have one that doesn't answer right under its first question, and 90% of sites with content pages have headings that never ask the question a reader brought. An assistant lifts the sentence under a question. When that sentence is a preamble, it lifts nothing, or it lifts someone else's.
Each row is a rule from mttr's rulebook, the sites it applies to, and the share with at least one page failing it.
| Rule | Stage | Sites it applies to | Failing |
|---|---|---|---|
| Answer the first question right under it | Understand | 99,109 | 99.2% |
| Phrase headings as questions | Understand | 124,110 | 89.7% |
| Fix description lengths | Understand | 287,797 | 90.8% |
| Add an llms.txt | Access | 287,797 | 77.8% |
| Add a tap‑to‑call link | Act | 250,260 | 69.7% |
| Add structured data | Understand | 287,797 | 63.5% |
| Match structured data to the page | Understand | 276,945 | 53.4% |
| Show sources on guides | Trust | 99,109 | 51.1% |
| Add a way to get in touch | Act | 287,797 | 45.4% |
| Show when guides were updated | Trust | 99,109 | 37.2% |
| Put the text in the HTML | Access | 276,945 | 36.4% |
| Fix pages that fail to load | Access | 287,797 | 22.8% |
| Make near-copy pages distinct | Trust | 287,797 | 19.4% |
| Allow pages to be indexed | Access | 287,797 | 14.0% |
Two findings stand out for anyone who sells locally. 70% of sites that show a phone number have a page where it can't be tapped to call, and 45% of all sites have a page with no way to get in touch. Those are the pages an assistant sends someone to.
Which lines of work do best?
Trades and home services lead with a middle score of 85; news and publishers trail at 75. mttr places a site in a line of work by what its pages mean, and has placed 78,248 of the 287,797 open sites so far; the rest are in the bottom row only. The last three columns are the share of sites with a page whose words arrive only by script, a page missing structured data, and a page with no way to get in touch.
| Line of work | Sites | Middle score | Script only | No schema | No contact |
|---|---|---|---|---|---|
| Trades and home services | 43,411 | 85 | 21% | 47% | 25% |
| Clinics and dentists | 515 | 84 | 23% | 52% | 30% |
| Real estate | 737 | 83 | 21% | 55% | 26% |
| Online shops | 3,059 | 82 | 37% | 62% | 34% |
| Auto repair and dealers | 1,688 | 82 | 20% | 55% | 31% |
| Lending, insurance and finance | 782 | 82 | 23% | 56% | 29% |
| Cities and public services | 215 | 82 | 32% | 66% | 22% |
| Hotels and travel | 3,358 | 80 | 19% | 61% | 35% |
| Manufacturers and suppliers | 2,550 | 80 | 30% | 63% | 28% |
| Law firms | 1,881 | 80 | 24% | 57% | 33% |
| Restaurants and cafes | 2,928 | 79 | 39% | 51% | 38% |
| Schools and courses | 751 | 79 | 24% | 66% | 34% |
| Software and SaaS | 6,332 | 78 | 24% | 67% | 54% |
| Agencies and studios | 5,188 | 77 | 33% | 61% | 45% |
| Nonprofits and charities | 2,537 | 77 | 32% | 62% | 34% |
| News and publishers | 2,316 | 75 | 34% | 65% | 44% |
| Every open site | 287,797 | 78 | 36% | 63% | 45% |
Trades sites score well and still miss the basics a homeowner needs: 31% of them have a page with a phone number that can't be tapped. Law firms and finance sites write guides, and nearly every one misses the answer under the question.
What to fix first
In order, since each one stops the ones after it from counting:
- Let the crawlers in. Check robots.txt for lines that turn away GPTBot, ClaudeBot, PerplexityBot or Google-Extended, and keep only the ones you meant.
- Answer under the question. Phrase the heading as the question, and make the first sentence under it the answer, with the number or the name in it.
- Say what each page is. Structured data that matches the page: the business, its address and hours, the service, the article and its date.
- Make the next step tappable. A phone number that dials, and a way to get in touch on every page someone might land on.
To see where your own site stands, run the free AI visibility check. It shows the score, the place among these sites, and the three fixes worth the most.
What a score does not say
A score says the pages are well formed for search engines and AI assistants. It doesn't say an assistant cites the site, and nothing in this post measures citations. A well formed page is one an assistant can read, understand and trust; whether it names the business also depends on what other sites say and on what the person asked.
How we counted
Every figure is from mttr's catalog, counted on October 30, 2026: each site at its latest read, drafts left out. A read takes a site's first 100 pages, fetched as a crawler would, with robots.txt obeyed, and checks each page against mttr's rulebook.1 A site counts as failing a rule when at least one of its pages fails it, among the sites the rule applies to; guide rules apply only to sites with guides. Sites closed to crawlers are counted in the first section and left out of every other figure. Lines of work are placed by meaning, from each site's own pages.2
Questions
How many websites block AI crawlers?
Of the 316,711 sites mttr read, 28,914 (9.1%) keep crawlers out entirely. Of the rest, 4.2% let search engines in but turn AI crawlers away in robots.txt.
What do most websites get wrong for AI search?
Answering. Of the 99,109 sites with guides, 99.2% have one that does not answer right under its first question, and 51% have one that shows no sources.
Does a high score mean AI assistants recommend a site?
No. A score says pages are well formed for search engines and AI assistants. It does not say an assistant cites the site.
References
- mttr, the rulebook: every rule, its stage, its weight and its fix. mttr.si/rulebook
- mttr, methodology: how reads, scores, places and lines of work are worked out. mttr.si/methodology
- IETF, "RFC 9309: Robots Exclusion Protocol," September 2022. rfc-editor.org
- Google Search Central, "Introduction to structured data markup in Google Search." developers.google.com
- "The /llms.txt file," a proposal for a file that outlines a site for language models. llmstxt.org