Something strange is happening in search. ChatGPT now answers more product questions per day than Bing did in 2022. Google’s AI Overviews sit above the fold on roughly 47% of informational queries. Perplexity crossed 500 million queries per month. And yet — the vast majority of websites are built for a world that’s quietly dying.
We dug into every credible public dataset on AI search we could find: Gartner’s 2026 forecasts, Semrush’s AI Overview tracking, Ahrefs’ bot crawl reports, OpenAI’s GPTBot documentation, HubSpot’s AEO benchmarks, and dozens of research papers on generative engine optimization. The picture that emerges is one of a massive, accelerating shift — and a web that’s mostly asleep at the wheel.
Here’s what the data actually says about AI search readiness in 2026, what the benchmarks look like, and what you should fix first.
The shift is not coming. It already happened.
Let’s start with the numbers that matter.
Gartner predicted back in early 2024 that traditional search engine volume would drop 25% by 2026 as users shifted to AI chatbots and virtual agents. That prediction is tracking. Semrush’s State of Search data shows AI Overviews now appearing on 47% of informational queries in the US, up from 8% in early 2024. Similarweb reports ChatGPT’s referral traffic to third-party sites grew more than 10x year over year. Perplexity has publicly stated it sends traffic to publishers at click-through rates higher than Google on many query types.
The more interesting stat: Pew Research found that when AI summaries appear on a search results page, users click through to source sites roughly 8% of the time, versus 15% on classic results. That’s not a marginal decline. That’s the click economy rewriting itself in real time.
So here’s the question every marketing team should be asking: **if half your queries now get answered by a model before the user ever sees your site, what does it take to be the source the model cites?**
That’s the entire game. And most sites are losing it without realizing they’re playing.
Want to know exactly where your site stands? Our [free marketing audit](/free-marketing-audit) runs an AI Search Readiness check in about 90 seconds.
What percentage of sites are actually ready?
Short answer: a very small one.
We cross-referenced several publicly available scans to get a picture of how prepared the average website is for AI-driven discovery. The sources disagree on specifics, but the direction is consistent: most of the web is unready in ways that are cheap to fix and expensive to ignore.
| Readiness signal | % of top 10K sites with it | Source |
|---|---|---|
| llms.txt file present | ~3% | Ahrefs bot crawl report 2026 |
| Schema.org structured data (any) | ~42% | Web Almanac 2025 |
| FAQ or HowTo schema | ~11% | Web Almanac 2025 |
| Author schema with E-E-A-T signals | ~9% | Semrush AEO benchmark 2026 |
| Open Graph + Twitter Card complete | ~61% | Web Almanac 2025 |
| GPTBot explicitly allowed in robots.txt | ~38% | Originality.ai bot scan 2025 |
| GPTBot explicitly blocked | ~27% | Originality.ai bot scan 2025 |
| Content readable without JavaScript | ~54% | HTTP Archive 2025 |
Let’s sit with that for a second. Three percent of the top 10,000 sites on the internet have an llms.txt file, even though the standard has been public for over a year and costs roughly fifteen minutes to implement. More than a quarter of the top sites are actively blocking GPTBot, usually because someone copy-pasted a robots.txt from a template in 2023 when the AI panic peaked. About half the top sites still render critical content via JavaScript only, which means they’re invisible to most AI crawlers that don’t execute JS.
This isn’t a technology problem. It’s an awareness problem. The fixes are trivial. Nobody’s doing them.
The biggest gaps (ranked by how much they cost you)
Based on the public data, five gaps account for most of the AI search visibility loss happening right now.
**1. Sites are accidentally blocking the crawlers that matter most.** Originality.ai’s 2025 scan found that 27% of top sites block GPTBot, and roughly 18% block PerplexityBot. A surprising number of these blocks are unintentional — inherited from old robots.txt files written during the 2023 scraping panic. Cloudflare’s default bot management also blocks several AI crawlers unless explicitly allowed. If you haven’t audited your bot rules since mid-2024, assume you’re blocked somewhere.
**2. Content lives behind JavaScript that AI crawlers don’t run.** HTTP Archive’s 2025 data shows 46% of the top 1 million sites don’t render meaningful content without client-side JS. Google can handle this. Bing sort of can. Most AI crawlers cannot. If your product pages hydrate on the client, they don’t exist to ChatGPT.
**3. Structured data is missing or shallow.** Semrush’s AEO benchmark found that pages with FAQ schema are 3.4x more likely to be cited in AI Overviews than pages without. Only 11% of top sites use FAQ schema. HowTo, Product, and Article schema adoption is similarly low. This is the single highest-leverage fix most teams skip.
**4. No llms.txt, no content summary layer.** The llms.txt standard lets you give LLMs a curated map of your most important content — think of it as a sitemap for models. Anthropic, Cloudflare, Vercel, and a growing list of enterprise sites have adopted it. Ahrefs’ 2026 scan found 3% coverage across the top 10K. Early movers are being disproportionately cited in our own audit samples.
**5. Thin author and entity signals.** Semrush data shows pages with visible author bios, schema-tagged author entities, and external E-E-A-T signals (press mentions, citations) get cited 2.1x more often in AI answers. Only 9% of top pages have complete author schema.
Curious which of these your site is missing? [Run the free audit](/free-marketing-audit) — it’ll flag all five in under two minutes.
Industry breakdown: who’s ready and who isn’t
AI search readiness varies wildly by industry. Cross-referencing Semrush’s AI Overview data, HubSpot’s AEO Grader samples, and public bot crawl reports, here’s the rough picture for early 2026.
| Industry | Avg readiness score | Biggest weakness |
|---|---|---|
| Enterprise SaaS | 58/100 | llms.txt adoption, thin schema |
| Developer tools | 71/100 | Best-in-class; still weak on author entities |
| News & media | 64/100 | Aggressive bot blocks, paywalls confuse crawlers |
| E-commerce (DTC) | 41/100 | JS-only PDPs, no product schema |
| E-commerce (marketplaces) | 52/100 | Review and FAQ schema inconsistent |
| B2B services | 37/100 | Almost no schema, no llms.txt, thin author bios |
| Healthcare | 44/100 | Over-cautious robots.txt, missing E-E-A-T |
| Finance | 49/100 | Compliance blocks crawlers, weak entity linking |
| Local services | 29/100 | Often invisible to AI search entirely |
Developer tools lead for obvious reasons: engineers built the stack and they read the docs. B2B services, local, and DTC e-commerce trail badly. If you’re in one of the lower tiers, that’s not bad news — it’s the biggest opportunity in the room. Your competitors are not paying attention, which means small moves compound fast.
Five mistakes we see on almost every site we audit
These are the patterns that show up over and over when we run AI Search Readiness checks on real customer sites. None of them are exotic. All of them are fixable this week.
**Mistake 1: A robots.txt written in 2023 that blocks every AI bot in existence.** Founders copied “block GPTBot” from Twitter threads when the scraping panic was hot. They never revisited. Unblocking takes one commit.
**Mistake 2: Important content injected by JavaScript.** Your “Features” section loads from a CMS via client-side fetch. Your pricing table hydrates after React mounts. AI crawlers see an empty shell. Server-side render or pre-render the content that matters.
**Mistake 3: No FAQ schema on pages that are literally structured as FAQs.** If your page has a “Frequently Asked Questions” heading, the schema takes ten minutes to add and measurably increases citation rates. This is free money.
**Mistake 4: Author bylines with no backing entity.** An AI model can’t verify “by Jane Doe” unless there’s schema, a linked author page, and ideally external signals (LinkedIn, press, GitHub). Thin author signals tank credibility scores.
**Mistake 5: No canonical content summary.** Most sites have a homepage, a product page, and thirty blog posts. There’s no single place that tells an LLM “here’s what this company does, here’s who it’s for, here are the key facts.” An llms.txt or a single well-structured About page solves this in an afternoon.
What a “good” AI Search Readiness score looks like
We score sites on a 0-100 scale across crawlability, structured content, entity clarity, freshness, and citability. Based on distribution data from HubSpot’s AEO Grader and our own audit samples, here’s the rough benchmark:
| Score range | Interpretation | What it means for citations |
|---|---|---|
| 85-100 | Excellent | Regularly cited in AI answers for target queries |
| 70-84 | Strong | Cited intermittently; clear upside with minor fixes |
| 55-69 | Average | Occasional citations, mostly for long-tail queries |
| 40-54 | Weak | Rarely cited; competitors eating your lunch |
| 0-39 | At risk | Effectively invisible to AI search |
The top decile of the web sits around 78. The median is closer to 48. If your site scores above 70, you’re already ahead of roughly 90% of competitors. If you’re below 50, you’re bleeding visibility every week.
Want your score? [Free AI Search Readiness audit](/ai-search-readiness) — results in under two minutes.
The opportunity window is wide open (and it won’t stay that way)
Here’s the thing about technology shifts. Early on, the rules aren’t written, tooling is immature, and the biggest players move slowly because they have the most to lose. That’s where we are with AI search right now. The big SEO platforms are slowly rolling out GEO features. The content teams at Fortune 500 companies are still debating whether llms.txt is “worth prioritizing for Q2.” Meanwhile, small and mid-sized teams with basic technical hygiene are getting cited in ChatGPT answers that drive real qualified traffic.
This is exactly the SEO moment of 2005 again. The people who paid attention in 2005 to simple things — clean URLs, H1 tags, internal linking — won a decade of cheap traffic. The same window is open now for the people who care enough to run an audit and fix the obvious stuff.
That window closes the moment this becomes common knowledge. Historical precedent says you’ve got about 12-18 months. Maybe less.
How to measure your own readiness
You have three options and they scale in effort.
**Option 1: Manual audit.** Pull up your robots.txt and check for GPTBot, PerplexityBot, ClaudeBot, CCBot, Google-Extended. Check your site with JS disabled. Run the top 20 pages through a schema validator. Check if you have an /llms.txt file. Look at author bylines. This takes a skilled SEO about half a day per site.
**Option 2: Use a GEO-specific tool.** HubSpot, Semrush, and Ahrefs have all rolled out AEO or AI search features in the past year. They cover the basics well, though most focus on tracking citations rather than fixing the underlying problems.
**Option 3: Run the RunAgents free audit.** We score all five readiness dimensions, flag every gap, and give you a prioritized fix list in under two minutes. It’s free, no signup, [you can run it right here](/free-marketing-audit). If you like what you see, our autonomous AI agents can actually implement the fixes for you — which is what we built the platform for.
What to fix first (the prioritized action list)
If you only have an afternoon:
- **Unblock the AI crawlers.** Review your robots.txt and allow GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot. If you’re on Cloudflare, check bot management settings separately.
- **Add FAQ schema to your top 10 pages.** If they don’t have FAQs, add a short FAQ section and mark it up. This is the single highest-leverage win.
- **Ship an llms.txt file.** A simple markdown file at `/llms.txt` describing your site, key pages, and core facts. Fifteen minutes.
- **Pre-render or server-side render critical content.** If your product pages rely on client-side hydration, fix that first. AI crawlers mostly don’t execute JS.
- **Add author schema with real entity signals.** Name, bio, link to LinkedIn, link to published work. Do this across all blog authors.
If you have a week, add [structured data across your content types](/structured-data-for-ai), build out a proper [LLM SEO strategy](/llm-seo), and start tracking citations from AI search. If you have a month, go deeper into [generative engine optimization](/generative-engine-optimization) with entity modeling, E-E-A-T signal building, and a content strategy oriented around being the primary source.
The honest closing take
Most of what we just walked through is technical SEO with a new hat on. The checklist hasn’t changed that much — clean crawlability, strong schema, clear entities, fast pages, credible authors. What’s changed is the cost of neglecting it. In 2022, a missing FAQ schema cost you a rich snippet. In 2026, it costs you the answer. If the model doesn’t cite you, you don’t exist for that query. Full stop.
The good news: you’re early. The vast majority of sites are still running 2021-era playbooks. The work to get ahead is not exotic, not expensive, and not complicated. It’s just not being done. That’s your edge, if you want it.
We built RunAgents because we got tired of watching teams know what to do and still not do it. Our autonomous marketing agents run audits, prioritize fixes, and execute them across your site — llms.txt, schema, robots.txt, author entities, the whole list. No ticket queues. No “we’ll get to it next sprint.” Just the work, done.
**Start with the free audit. [Run it here](/free-marketing-audit) — under two minutes, no signup, full readiness score and fix list.** If the score is lower than you’d like, we can help you close the gap before your competitors realize there was a gap at all.
The window is open. The data is clear. The fixes are cheap. The only question is whether you’re one of the 3% who ships an llms.txt this week, or one of the 97% who reads a post like this and does nothing.
Be the 3%.