Cloudflare Blocks AI Crawlers by Default on Ad-Supported Sites: What Changes Today
As of today Cloudflare is enforcing a setting almost nobody changed by hand: on pages that show ads, AI crawlers are blocked by default. The awkward part is that Googlebot can end up on the blocked side too.
What Changes Today at Cloudflare
The date has been known since 1 July 2026, when Cloudflare announced the measure alongside the second edition of its Content Independence Day. This 15 September the new defaults stop being optional for domains that nobody has touched by hand.
It is worth being precise: this is not a blackout of AI across the internet. It is a default that flips from allowed to blocked, and only on one kind of page, the ones that show ads.
Search Yes, Training and Agents No
Search crawlers still pass through unchanged. The ones left out are those that download content to train models and those that browse on behalf of an AI agent.
Read also
The Three Categories: Search, Agent and Training
Cloudflare sorts bots into three groups. Search feeds traditional search. Training pulls content to train models. Agent acts on behalf of an assistant browsing for the user. The new default allows the first and blocks the other two on ad-supported pages.
Who It Applies To: New Domains, New Sites and Untouched Free Accounts
The scope is narrower than the headline suggests. It covers new domains added to Cloudflare, new sites an existing customer adds to their account, and free-plan accounts whose settings were left untouched. Until 15 September, any customer could opt out of those defaults; from today, whoever did not is keeping them.
What a Mixed-Use Crawler Is and Why It Is the Whole Problem
The confusing part is not the block itself, but how incoming traffic gets classified.
One User Agent Doing Three Jobs
A mixed-use crawler blends three jobs under a single identity: indexing for traditional search, fetching pages for AI agents, and scraping content for training. Cloudflare evaluates them by behaviour, and when two rules overlap, the most restrictive one wins.
The Number: More Than a Third of Crawling Is Mixed-Use
Cloudflare reported in July 2026 that more than 36% of crawler activity comes from mixed-use bots, and that AI crawlers sit around 20% of verified bot traffic. The company also measures how many visits each bot returns: Google returns a visit every few crawls, while some training crawlers need tens of thousands of pages per referred visit.
Googlebot: Search, AI Overviews and AI Mode Under One Identity
Here is the knot. Crawling for Search, AI Overviews and AI Mode shares one user agent: Googlebot. For Cloudflare's classification, that makes it a mixed-use crawler.
The Side Effect: Googlebot Can Get Blocked
This comes from a report by SEO specialists rather than an official confirmation from Google. But the technical explanation rests on Cloudflare's own documentation.
The Most Restrictive Rule Wins
If two rules point at the same bot, the stricter one prevails. Since Googlebot is mixed-use, any setting that blocks AI training also reaches mixed-use crawlers. That includes the legacy block-AI-bots option many people switched on months ago and no longer remember having enabled.
How to Check: 403s in Your Logs and the Sitemap in Search Console
The check is quick and takes minutes: look at the bot settings in the Cloudflare dashboard, search your server logs for 403 responses served to Googlebot, and confirm in Search Console that the sitemap downloads and URL inspection responds. A drop in impressions with no content change is another warning sign.
What Happens to the Legacy "Block AI Bots" Option
Anyone who enabled the blanket AI bot block is still bound by it. The difference from the new defaults is that the old option did not tell categories apart: it blocked everything Cloudflare identified as AI, without separating search from training.
Pay Per Crawl Becomes Pay Per Use
The same July announcement carried an economic twist that points the opposite way from blocking.
Paid When Your Content Shows Up in an Answer, Not When It Is Fetched
The old Pay Per Crawl charged when a bot downloaded the page. The new Pay Per Use charges when the content creates value, that is, when it shows up inside an AI answer. The shift moves billing from access to outcome.
x402, You.com and Ceramic: The First Partners
Settlement runs on the x402 protocol, and the first announced partners are You.com and Ceramic.ai, with a Monetization Gateway on the waiting list.
The Other Side of the Trade: Losing Visibility in AI Answers
Blocking is not free in either direction. You protect your content and you keep your search traffic, but you also shut yourself out of AI-generated answers, which resolve more and more queries without a click.
How to Check Your Settings in Two Minutes
The Cloudflare Dashboard: Block AI Bots and AI Crawl Control
In the dashboard, bot settings live under the security section. Both the classic AI bot block and the more granular per-category control sit there.
Before You Touch Anything: What Gets Blocked, What Stays Allowed
Before flipping a switch, decide it on purpose: if you live off ads and not off content, blocking training is reasonable; if your traffic depends on showing up in AI answers, it may not be. What is not reasonable is leaving it by inertia without knowing what is active.
Mind the Proxy: DNS-Only Domains Are Not Affected
Bot policies act at Cloudflare's network edge, so they only affect traffic that runs through its proxy. If your domain is set to DNS-only mode, the new defaults do not touch you.
What It Means When Cloudflare Blocks AI Crawlers for a Small Blog
For a blog or a small store running ads, the practical effect is a shift in the balance: less training crawling and lower bandwidth cost, in exchange for less presence in generated answers. The sensible move is to review the settings once, decide them deliberately and look again only when the rules change.
Conclusion
The news is not that AI has stopped reading the internet, but that a default stopped being neutral. Check your bot settings, confirm Googlebot is not getting 403s and decide with data whether blocking or charging suits you better. For the background on how we got here, see the analysis of the toll for AI bots, the Node.js HTTP APIs inside Cloudflare Workers and a walkthrough of how Google's algorithms work and their impact on your searches.
