Cloudflare's new defaults may be hiding your website from AI
Last checked:
Cloudflare is a service that sits in front of websites to speed them up and keep attackers out. If a web designer or hosting company set your site up, it may run through Cloudflare without you ever having opened its dashboard.
Over the last eighteen months Cloudflare has changed how it treats AI crawlers, the programs that ChatGPT, Claude, Perplexity and others use to read the web. Those changes are good news for publishers who want control. But a setting that blocks too much can also keep a local business out of AI answers it would like to be in.
What changed, in order
September 2024: a one-click block. Cloudflare says it introduced "the option to block AI crawlers in a single click" that month, and that more than one million customers had turned it on by mid-2025 [1].
July 2025: a choice at sign-up. Cloudflare announced that new domains would be asked at sign-up whether they want to allow AI crawlers, and described itself as blocking AI crawlers that take content "without permission or compensation, by default" [1][2].
July to September 2026: three separate switches. Cloudflare replaced its single "Block AI bots" switch with separate settings for three kinds of AI traffic [3][5]:
- Search: crawlers that index your content "so they can answer questions about it later".
- Agent: automated activity "acting in real time on a person's behalf", such as an assistant fetching your page while it answers someone.
- Training: crawlers that take content "to train or fine-tune a model".
From September 15, 2026, Cloudflare says new domains get one of two presets. Sites that are not monetized with ads allow all three. Sites that show ads keep Search allowed, ask AI companies not to train on their content, and block agents on pages with ads [4].
Why a local business should care
For most local businesses, being found is the point. You want an assistant that is asked for a plumber, a dentist or a florist in your town to be able to read your site.
Four details in Cloudflare's own write-up are worth knowing:
- Old settings carry over. Cloudflare says that for most existing domains "your current settings carry over on their own". A site that had the old block switched on was moved to blocking agents on pages with ads and asking for no training [4]. If someone switched that on for you years ago, it may still be shaping who reads your site.
- "Block" now reaches search engines too. Cloudflare says its Block and "Block on pages with ads" actions "now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot", and that Block "will stop Applebot, Bingbot, and Googlebot from reaching your site", for search as well [4]. Apple says Applebot powers search in Siri, Spotlight and Safari [6].
- There is now a way to say no to training and yes to search. Cloudflare's September 15, 2026 setting, Disallow AI Training, "works by publishing a preference in robots.txt". Cloudflare says Googlebot and Applebot "can keep crawling your site for search" under it, and Bingbot keeps crawling for search too, though Cloudflare notes the no-training preference does not reach Bing through robots.txt until Microsoft's support for it launches, which Microsoft targets for early 2027 [4].
- Agents are how assistants read in the moment. When an assistant fetches a page to answer a question right now, that is agent traffic. Blocking agents can stop that fetch.
The crawler makers say the same thing from their side. OpenAI says sites that opt out of its search crawler "will not be shown in ChatGPT search answers" [7]. Perplexity recommends allowing PerplexityBot "to ensure your site appears in search results" [8]. Anthropic says blocking its search crawler "may reduce your site's visibility and accuracy in user search results" [9].
How to check your own site
If you do not know whether your site uses Cloudflare, ask whoever built or hosts it. If it does:
- Log in to Cloudflare and open your domain.
- Go to Security, then Settings, and find AI Crawler and Agent access policies [3].
- Look at the three rows. Keep Search on Allow. To keep your content out of AI training, choose Disallow AI Training rather than Block: it asks for no training and keeps search crawlers coming. Decide on Agent for yourself.
- Avoid Block on Search unless you mean it, since that can also keep out Google, Bing and Apple [4].
- Save, and note the date you changed it.
Cloudflare's settings are yours to choose. Some owners will want to keep AI training out, and that is a fair choice. The point is to make it on purpose, not to inherit it from a setting nobody remembers turning on.
Where Echobat fits
Echobat's scan reads your site the way a crawler does and tells you if the door is shut, whether by robots.txt or by a firewall answering crawlers with an error. It shows exactly what came back, so you or your web person can fix the right thing.
Scan your site free and see what a crawler sees.
See what a machine sees. Echobat reads your site the way these crawlers do and shows the evidence behind each point.
Scan your site freeSources
- "Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model", Cloudflare press release, July 1, 2025. https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ (accessed November 9, 2026)
- "Content Independence Day: no AI crawl without compensation!", The Cloudflare Blog, July 1, 2025. https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ (accessed November 9, 2026)
- "AI Crawler and Agent access policies", Cloudflare bot solutions docs. https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/ (accessed November 9, 2026)
- "Have it both ways: stay discoverable in search while disallowing AI training", The Cloudflare Blog, September 15, 2026. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/ (accessed November 9, 2026)
- "New options to manage AI traffic", Cloudflare changelog, July 1, 2026. https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/ (accessed November 9, 2026)
- "About Applebot", Apple Support. https://support.apple.com/en-us/119829 (accessed November 9, 2026)
- "Overview of OpenAI crawlers", OpenAI. https://developers.openai.com/api/docs/bots (accessed November 9, 2026)
- "Perplexity crawlers", Perplexity. https://docs.perplexity.ai/guides/bots (accessed November 9, 2026)
- "Does Anthropic crawl data from the web, and how can site owners block the crawler?", Claude Help Center, Anthropic. https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (accessed November 9, 2026)