If you run sites behind Cloudflare, there’s a date you probably haven’t circled: September 15, 2026. That’s when Cloudflare’s new AI bot defaults start applying, and the way it now classifies crawlers means a rule you set to block AI training could quietly take Googlebot down with it.
Here’s the part that keeps me up at night on client sites: this happens at the network layer, before robots.txt is ever read, and it doesn’t throw an obvious error in Google Search Console. Your crawl coverage degrades silently. Let’s walk through exactly what changed and how to audit it this week.
The September 15 deadline most Cloudflare site owners have missed
On July 1, 2026, Cloudflare replaced its single AI-or-not toggle with three independently configurable categories and announced new defaults. On September 15, 2026, those defaults switch on: Training and Agent bots get blocked by default on pages that display ads, while Search stays allowed by default.
The scope matters, so read this carefully. The new blocking defaults apply automatically to new domains and new sites added by existing customers. If you’re a current customer with existing zones, the defaults don’t silently flip for you—but that cuts both ways. It means whatever you (or a WordPress plugin, or a previous agency) configured months ago is still live, and it may already be catching search crawlers you never meant to touch.
Cloudflare frames this as urgent structural work. CEO Matthew Prince put it bluntly: “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.” Treat that as his stated position, not an audited statistic—but it tells you the direction of travel. The defaults are getting more aggressive, not less.
The three-tier taxonomy: Search, Agent, Training
The whole thing rests on three behavioral definitions. They’re worth quoting exactly, because the categories are named after intent, not after companies—and that’s where the trouble starts.
- Search—“any behavior that collects or indexes your content, so it can answer questions about it later.”
- Agent—“automated behavior that is acting, usually in real time, on a person’s behalf, to get something done right now.”
- Training—“a crawler taking your content to train or fine-tune models. The key is that your data is permanently absorbed.”
Cloudflare’s own bot reference is more granular than the three tiers suggest—it uses four buckets. GPTBot, ClaudeBot, CCBot, and Google-CloudVertexBot sit in AI Crawlers (training). OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Applebot sit in AI Search. ChatGPT-User, Claude-User, and Perplexity-User sit in AI Assistant. And Googlebot and BingBot get their own bucket: Traditional Search Engines.
Read those buckets and the risk jumps out. “Block AI bots” is not one switch—it’s a decision that can sweep across four different categories of crawler, several of which you almost certainly want to keep.
The mixed-purpose trap: why Googlebot sits in a dangerous middle
This is the crux of the article, so I’ll state it plainly. Cloudflare classifies Googlebot, BingBot, and Applebot as mixed-purpose bots—crawlers that do both search indexing and training data collection in a single user agent. A WAF rule or Bot Fight Mode setting aimed at the Training category can inadvertently suppress these search crawlers at the network layer, silently degrading crawl coverage without any change to robots.txt and without triggering an obvious error in Search Console.
In Cloudflare’s own words, multi-purpose crawlers that combine Search with Training “will be blocked by customers who have selected to block Training”—and the post names names: “Googlebot, Applebot, and BingBot will be blocked” under those conditions. The Register described the same publisher bind: because Googlebot fuses search indexing with content harvesting for AI training, publishers have tended to grit their teeth and accept the bot, “because they fear blocking could mean they disappear from Google Search results.”
How one rule silently suppresses your search crawl
To understand why this is so easy to miss, you have to understand where the block happens. A Cloudflare WAF or Bot Fight Mode block fires at the network edge, before the crawler ever reaches your origin server. That means it executes before robots.txt is fetched or read.
Search Engine Journal put the practical consequence well: “a Cloudflare block operates at the network level, making it harder to bypass than a simple robots.txt line that Google can ignore.” Practitioners describe the same architecture—“WAF rules are enforced before robots.txt is even accessed, so a WAF block overrides any robots.txt Allow directive.”
So picture the sequence. Googlebot requests a page. Cloudflare inspects the request at the edge, matches it against your Training-block rule, and returns a challenge or a 403. Googlebot never reaches your server. Your carefully-tuned robots.txt that says Allow: / for Googlebot? Never consulted. It was overruled one layer up, and nothing about that overrule looks like a robots.txt problem.
Two independent blocking layers that can contradict each other
It helps to think of Cloudflare WAF and robots.txt as two entirely separate enforcement layers that don’t talk to each other. Changing one does nothing to the other, and they can flatly contradict each other with no reconciliation step and no error signal.
robots.txt is, by Cloudflare’s own admission, an honor system: “Adding an entry to your robots.txt file is the first step to telling AI bots not to crawl you. But robots.txt is an honor system. Nothing forces bots to follow it.” That’s the entire reason WAF-level enforcement and AI Crawl Control exist as a separate, harder layer. As one analysis of Cloudflare’s Content Signals put it: “Signals ≠ blocks”—some companies may ignore signals, so you combine them with WAF or Bot Management to actually enforce anything, up to returning a 402 Payment Required.
Now hold both facts at once. Your robots.txt can say “Googlebot, come on in” while your WAF says “block the Training category”—and because Googlebot is mixed-purpose, the WAF wins. Two layers, one saying allow, one saying block, and the block executes first. No contradiction gets flagged anywhere. That’s the trap in a sentence.
The five-step September audit
Here’s the check I’m running on every client site before September 15. One caveat up front: dashboard paths move around, so treat the specific navigation labels below as accurate as of July 2026 and adjust if Cloudflare has reshuffled the menus.
- Super Bot Fight Mode. Find its settings and check whether “Verified bots” are being allowed. If SBFM is on and challenging definitely-good bots, you have a candidate culprit. Note that the mode itself can’t be reconfigured through WAF expressions—more on the fix below.
- AI Crawl Control (formerly the AI bot blocking toggle). Look at which of the categories—Search, Agent, Training—you’re blocking, and remember that blocking Training catches mixed-purpose Googlebot, BingBot, and Applebot.
- WAF Custom Rules. Read every rule that references bots, user agents, AI, or crawlers. Look for broad expressions like a generic “AI bot” category block that would sweep in the search buckets.
- Rate limiting rules. Aggressive rate limits keyed to crawler-like behavior can throttle Googlebot into partial crawls even without an outright block.
- Confirm with a live fetch. Use the URL Inspection tool in Search Console’s live-test mode on several real pages, not just the homepage. You’re looking for whether Google can actually fetch the rendered page right now.
For context on scale: Cloudflare has said Super Bot Fight Mode blocks an average of 336 million requests per day for self-service customers (as of that feature’s announcement). It is a blunt, high-volume instrument. When it catches something it shouldn’t, you’ll want the fix in the next section rather than trying to “turn it down.”
The correct configuration: block Training without touching Search
The goal is surgical: block the pure-training crawlers, leave the mixed-purpose search crawlers and the AI search crawlers alone. Work from Cloudflare’s own bot reference rather than a generic category toggle.
Block the training-only bots explicitly by user agent—GPTBot, ClaudeBot, CCBot, Bytespider, Google-CloudVertexBot, Meta-ExternalAgent, Amazonbot. Do not use a blanket “block all AI” category action, because that’s what pulls in the search buckets.
# WAF Custom Rule — block training crawlers only
# (Illustrative expression; confirm exact field names in your dashboard)
(http.user_agent contains “GPTBot”) or
(http.user_agent contains “ClaudeBot”) or
(http.user_agent contains “CCBot”) or
(http.user_agent contains “Bytespider”) or
(http.user_agent contains “Google-CloudVertexBot”)
# Action: BlockIf Super Bot Fight Mode is the thing catching Googlebot, you can’t fix it by editing SBFM—it can’t be reconfigured via WAF custom rules. The documented remedy is a higher-priority skip rule. In Cloudflare’s words: “in the new Custom Rules, there’s only one action called ‘skip’. Rules that ‘skip’ traffic can skip the rest of your custom rules (just like an ‘allow’ rule would) and other Cloudflare services.” Put a skip rule for verified search crawlers above your blocking rules and above SBFM’s reach.
The GEO and AEO dimension: blocking Agent bots can cost you citations
Now the part most SEO checklists miss. Blocking the Training category is a defensible business decision—you may genuinely not want your content permanently absorbed into someone’s model. But the September defaults also block the Agent category, and that’s a different trade-off entirely.
OAI-SearchBot, Claude-SearchBot, and PerplexityBot are the crawlers that feed live retrieval in ChatGPT Search, Claude, and Perplexity. They live in the AI Search bucket, not Training. Block them—deliberately or through an overly broad rule—and you remove your site’s chance of showing up in AI-generated answers and citations. Anthropic is explicit that blocking Claude-SearchBot “prevents Anthropic from indexing your content for this search optimization system” and that your “visibility and accuracy in user search results” may drop as a result.
So think in three tiers, not two. Block pure training crawlers if that’s your policy. Keep the AI search crawlers open if you care about being cited in AI answers—which, in 2026, most of our clients do. A practical allow/block split from one practitioner guide: allow OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot, and BingBot; block GPTBot, CCBot, Google-Extended, and Applebot-Extended.
The hard part is that you can’t see the cost of blocking Agent bots in any traditional analytics view—a citation that never happened leaves no trace. This is exactly the gap we built Cited to close: it tracks how ChatGPT, Claude, Perplexity, Gemini, and Google AI Overviews actually describe your brand over time, so if a Cloudflare change quietly cuts you out of AI answers, you see the drop instead of guessing. Even if you never touch the tool, the principle stands: measure AI visibility deliberately, because your firewall config now influences it.
Action items for this week
Two things to do on every client site running Cloudflare, plus the one metric that tells you whether you already have a problem.
- Audit the WAF and bot settings using the five-step checklist above. Do it before September 15, and re-check any new zones after that date, since new sites inherit the aggressive defaults automatically.
- Split your bot policy into three tiers—block training-only crawlers by user agent, keep mixed-purpose search crawlers open with a priority skip rule, and make a conscious decision about AI search crawlers based on whether you want AI citations.
The GSC metric to watch: in URL Inspection, the combination “Crawl allowed? Yes” with “Page fetch failed.” That contradiction is the diagnostic fingerprint of firewall-level blocking. It means robots.txt returned 200 and didn’t disallow the page, but the actual fetch was blocked upstream—which points at Cloudflare, not at your directives. Watch for it alongside “Blocked due to access forbidden (403)” and, in Cloudflare’s logs, HTTP 1020 Access Denied responses.
The reason this is worth a dedicated audit rather than a wait-and-see: the damage is progressive, not dramatic. As one technical SEO team described it, once Cloudflare starts blocking or challenging Google, “it will slowly begin deindexing your pages one by one.” There’s no single alarm. Rankings just erode while everything on your side looks fine. The fix is to confirm your recovery too—look for 403 errors falling and 200 responses rising in your logs after you adjust the rules.
If you’d rather not comb through WAF rules on a dozen client zones yourself, that’s precisely the kind of thing our AI Visibility Audit checks—including whether your Cloudflare configuration is quietly costing you crawl coverage or AI citations. Either way: look before September 15. This is a 30-minute check that prevents a three-month recovery.