Cloudflare lets websites refuse AI training while staying visible in search

Cloudflare introduced Disallow AI Training on 15 September 2026, giving website owners a way to refuse model training while keeping supported search crawlers on their sites. It combines published preferences with selective crawler blocking, rather than treating every AI-related request alike.

The setting preserves a route to search discovery, with support that varies by operator. It does not guarantee indexing, rankings or referrals.

Why one crawler can create two different decisions

Consider a retailer publishing detailed buying guides. It wants people searching for product advice to find those pages, but it may decline to have the same material used to train a general-purpose model.

When one crawler collects content for both purposes, refusing its requests can also remove its access for search. The publisher needs a way to distinguish the permitted use from the refused use after the page has been collected.

Separate operator controls already existed. Google-Extended and Applebot-Extended let publishers restrict specified training uses without blocking the main search crawler. Cloudflare's change brings that distinction into a common set of traffic controls.

Cloudflare also introduced an Accountable designation. Its criteria cover training choices, summary opt-outs, visibility into content use and assurances about traditional search ranking. Operators can qualify through existing capabilities or commitments with specified timelines. That last distinction matters when checking what works today.

Cloudflare AI training controls: choose Disallow carefully

Cloudflare separates Search, Training and Agent controls. Within Training, these settings have different consequences:

Training setting	What happens	Effect on search access
Allow	Training traffic is allowed, subject to other rules	No training-related access block is added
Disallow AI Training	Publishes no-training preferences, keeps Accountable mixed-use crawlers and blocks other training crawlers	Retains supported mixed-use search access
Block	Blocks training crawlers, including mixed-use crawlers	Can also block Googlebot, Applebot and Bingbot
Block on pages with ads	Applies the block on pages detected as serving ads	Can restrict mixed-use search crawling on those pages

For search access without training, choose Disallow AI Training rather than Training: Block. Stronger access blocking can remove the discovery route you wanted to keep.

Bing still needs separate attention. Cloudflare says domain-level no-training support through robots.txt is targeted for early 2027. Until it launches, Disallow AI Training does not automatically convey that preference to Bing.

Microsoft's documented NOARCHIVE option excludes content from its generative-model training and Bing Chat answers while retaining ordinary search availability. That broader effect on answers matters if the business also wants AI referrals; review Microsoft's instructions before adopting it.

New-domain recommendations also differ by business model. Cloudflare recommends training refusal for ad-supported sites, alongside search access and agent blocking on ad-carrying pages. For other new sites, all three categories are offered as allowed. There is no universal default refusing training for every domain.

Bot Preference Sync connects the settings to robots.txt

Bot Preference Sync generates managed robots.txt directives from the site's Search, Agent and Training choices. It adds managed material before an existing file, reducing the need to maintain each supported crawler's rules individually. It does not translate every custom security rule into a complete content-use policy.

The current Cloudflare setup instructions are:

1. Open the domain's Security Settings and filter by Bot traffic.
2. Select Configure AI bot policies.
3. Set Training to Disallow.
4. Enable Bot Preference Sync and save.

For the goal discussed here, keep Search allowed and choose the Agent policy separately. After saving, inspect the robots.txt response actually served by the site, check that existing path restrictions remain and review overlapping security rules.

Publishing a refusal and enforcing access are separate mechanisms. A robots.txt file communicates preferences to compliant crawlers; it does not itself stop requests. Cloudflare's Disallow configuration combines those preferences with edge blocking for training crawlers, excluding supported mixed-use crawlers from that block.

Our guide to what robots.txt does and does not decide explains the wider distinction between crawler rules, technical access and permission.

Content Signals separate training from answer-time use

Cloudflare's Content Signals Policy, introduced in September 2025, describes three uses of collected content:

Signal	Use it describes
search	Building an index and returning links and excerpts; excludes AI-generated search summaries
ai-input	Supplying content to an AI model when answering a query, including retrieval and grounding
ai-train	Training or fine-tuning models

These usage signals are distinct from the dashboard's traffic categories. Under the policy, an omitted signal neither grants nor restricts that use through Content Signals. A missing ai-input field therefore does not silently mean yes.

For a publisher, the commercial questions also differ. A model learning from an article and an assistant retrieving that article to answer today's question involve different uses. Refusing the first does not, by itself, settle the second.

Google illustrates why the exact control matters

Google supports this operator-specific robots.txt rule:

User-agent: Google-Extended
Disallow: /

Google-Extended is a control token, rather than a separate crawler identity to look for in server logs. Its documented scope includes future Gemini model training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect Google Search inclusion or act as a Search ranking signal. It should not be confused with blocking Googlebot.

Google also has a separate Search Console control for participation in generative AI Search features, including AI Overviews, AI Mode and AI Overviews in Discover. Google confirmed worldwide rollout by 31 August 2026 and says the choice is not a ranking signal outside those generative features.

A business seeking visibility in AI answers should therefore review answer inclusion separately from training. Neither choice substitutes for checking the actual search and referral outcomes after a configuration change.

The intended use should stay with the dataset

The same distinction matters to teams collecting web data.

Suppose a company checks public product pages each morning for prices and availability. Later, another team proposes feeding the collected descriptions into a retrieval assistant. A third team wants to fine-tune a model on them.

Those are three project decisions, even if the source URLs are identical. A review supporting price monitoring should not silently become approval for every later use. Equally, a no-training signal does not settle the access rules or permitted scope of an unrelated business collector.

Keep the source policy, checked date and intended use alongside the collection configuration. Review them when the dataset gains a new purpose, rather than assuming the original collection decision covers it.

For a suitable source, build and validate a sitemap in the Web Scraper browser extension, inspect representative records, then schedule the tested configuration in Cloud. This provides a repeatable collection workflow while the team retains responsibility for its chosen sources and uses.