Robots.txt AI training opt-out: what the proposal changes

A proposal published on 30 October 2026 would let websites express an AI-training preference to several providers through one shared robots.txt group. That could reduce the work of maintaining a separate opt-out for every company.

The mechanism remains an individual Internet-Draft. Providers have announced different rollout schedules, with different consequences for training and AI answers. Website owners should review those differences before replacing controls they currently rely on.

How the robots.txt AI training opt-out would work

The proposed whole-site preference uses this group:

User-Agent: __usage_ai-training
Disallow: /

The shared policy token __usage_ai-training identifies a content use. The syntax is unchanged. To favour training on /press-kit/, a publisher could add:

Allow: /press-kit/

That longer path takes precedence over /, leaving the refusal in place elsewhere. Both settings express usage preferences; an Allow is not a blanket licence. [1][2]

The illustrative path-resolution table below uses the training group in the complete example that follows. All paths belong to https://example.com/.

Path	Matching rule	Expressed preference
/	Disallow: /	Against training
/guides/bottles	Disallow: /	Against training
/press-kit/	Allow: /press-kit/, the longest match	In favour of training
/press-kit/logo.png	Allow: /press-kit/, the longest match	In favour of training
/press-kit	Disallow: /	Against training
/press-kitchen/	Disallow: /	Against training

/press-kit lacks the trailing slash required to match the /press-kit/ prefix, so the broader refusal wins. /press-kitchen/ is a different prefix and also falls under the refusal. If an Allow and a Disallow rule are equivalent, the Allow rule should be used. [8]

Kevin Kelley of Anthropic submitted the proposal as an individual Internet-Draft. The IETF document tracker explicitly says it is not endorsed by the IETF. The proposal remains work in progress. [1]

Kelley's announcement also explains a deliberate compatibility choice: User-Agent: * does not count as the shared training preference. Without an explicit __usage_ai-training group, a file expresses no preference through this mechanism. Existing crawler rules retain their meaning. [2]

This complete illustrative robots.txt for https://example.com/ combines ordinary crawl restrictions, existing named controls and the new preference. It represents a publisher choosing both a press-kit exception and an explicit Google exception.

# Illustrative robots.txt for https://example.com/
User-agent: *
Disallow: /account/
Disallow: /checkout/

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Allow: /

User-agent: __usage_ai-training
Allow: /press-kit/
Disallow: /

Sitemap: https://example.com/sitemap.xml

Line or group	What it does
User-agent: *	Supplies ordinary crawl rules excluding /account/ and /checkout/ for crawlers with no matching named group. It no longer applies to a crawler that has its own matching group. [8]
User-agent: GPTBot	Retains an illustrative existing named crawl restriction. Its Disallow: / applies to every path, including the press kit; the wildcard group's rules are not added to it. [8]
User-agent: Google-Extended with Allow: /	Makes the Google-specific choice explicit. Under Google's announcement as described above, Google does not apply its shared-refusal fallback when Google-Extended is explicitly specified. Google-Extended covers both its stated training and grounding uses, so this is a Google exception to the broader training refusal.
User-agent: __usage_ai-training	Expresses a training refusal except for /press-kit/. The Allow line comes first for compatibility with the first-match parser demonstrated below.
The training group in a legacy parser	A parser that does not recognise the policy token treats it as an unmatched user-agent group, so it has no effect on ordinary crawling. [8]
Sitemap: https://example.com/sitemap.xml	Declares a sitemap outside every group. Its position after the training rules does not make it part of that group. [8]

User-agent matching is case-insensitive. If several groups match the same user-agent, their rules are combined before resolving the path; the wildcard group is a fallback, not an extra set of rules added to a named group. [8]

Which providers have announced support?

These are the published positions checked on 3 November 2026. They describe implementation plans, rather than a completed rollout across all services.

Provider	Announced position	What happens to existing controls
Google	Plans support in early 2027	Google-Extended continues alongside the shared signal
OpenAI	Aims to implement in the coming months	Existing robots.txt crawler controls continue
Microsoft	Starts honouring new preferences from 1 February 2027, as it crawls	A transition replaces the training and grounding meanings of NoArchive and NoCache
Anthropic	Kelley says implementation has started and will take time across services	Existing crawler controls remain

Sources: provider announcements, Google's crawler documentation and Kelley's mailing-list post. [2][3][5][6]

Before deployment, check each provider's instructions for current availability and precedence between shared and company-specific settings.

Change calendar

Date	Provider	What to check
30 October 2026	Proposal publication and provider announcements	Record the draft being reviewed and the existing controls before planning a change.
Coming months; no exact date stated	OpenAI	Confirm implementation is available before relying on the shared signal; retain existing crawler controls.
Early 2027	Google	Confirm an explicit Google-Extended group exists if Gemini grounding should continue, and review the accompanying Google training exception.
1 February 2027	Microsoft	Check the new training and grounding choices as Microsoft begins honouring them through crawling.
1 March 2027	Microsoft	Replace reliance on NoArchive and NoCache for these uses; review the page-level grounding setting and temporary grounding preference in Bing Webmaster Tools.
Ongoing implementation; no completion date stated	Anthropic	Confirm coverage across the relevant services and retain existing crawler controls while support develops.

Training and grounding need separate attention

The AI Preferences vocabulary draft defines training as using content to modify a generative model's learned parameters. Fine-tuning fits that description. Retrieving a page to supply context for an answer is a different operation. The training category also excludes models used exclusively for non-generative tasks. [7]

A publisher might want its buying guides cited in an assistant's answers while refusing their use to train a future model. Whether the shared signal supports that combination depends on the provider's implementation.

Google maps the shared refusal to Google-Extended

Google plans to interpret the shared training opt-out as a Google-Extended disallow when Google-Extended is not explicitly specified. Its announcement preserves the ability to make a company-specific choice. [4]

Google-Extended currently covers future Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect inclusion in Google Search or act as a Search ranking signal. [5]

Google's planned fallback can therefore affect grounding as well as training. A publisher that wants Gemini answer visibility should review its explicit Google-Extended setting before treating the common token as an isolated training switch.

Microsoft separates the choices and changes its older controls

Microsoft has announced independent training and grounding preferences. A page-level training preference overrides the robots.txt setting.

From 1 March 2027, Microsoft will stop reading NoArchive and NoCache as training or grounding preferences. It also says detecting either new preference on a site causes it to disregard remaining legacy preferences there.

Sites relying on the older controls should review both choices before adding the shared training rule. Microsoft's transition instructions include page-level grounding settings and a temporary grounding preference in Bing Webmaster Tools. Publishing only a training choice could leave the intended grounding restriction unexpressed in the new system. [6]

The illustrative publisher decisions below are alternatives. The complete example above selects a press-kit exception and a Google exception; it does not represent a refusal covering every provider and path.

Intended outcome	What to publish	Why
Refuse training everywhere	A User-agent: __usage_ai-training group with Disallow: /, without the press-kit Allow exception; review explicit provider-specific choices too.	Expresses a refusal for every path through the shared mechanism. Its effect still depends on provider support and any applicable company-specific setting.
Refuse training but keep Gemini grounding	The shared refusal plus an explicit Google-Extended group with Allow: /, accepting the Google exception.	Google's shared-refusal fallback does not apply when Google-Extended is explicit. That allow covers the stated Google training uses as well as grounding, so this combination does not refuse Google's covered training use.
Refuse training except the press kit	In the training group, put Allow: /press-kit/ before Disallow: /.	The longer press-kit path expresses a preference in favour of training there while retaining the refusal elsewhere.
Keep grounding restricted after Microsoft's transition on a site using NoArchive or NoCache today	Publish Microsoft's page-level grounding setting and record the temporary grounding preference in Bing Webmaster Tools; review the training choice separately.	The older controls stop expressing those preferences after the transition. Adding a new training preference can also cause remaining legacy settings to be disregarded.

Check access and usage separately

The proposed training token concerns how content is used. It is not a crawling directive; ordinary crawler groups continue to govern fetching. A consumer may choose to avoid retrieving content it cannot use, but the draft does not require every request to stop. [1]

RFC 9309 also states that robots rules are not access authorisation. Publishing a preference does not enforce a network block, supply a licence or settle the legal position of a collection project. Our guide to robots.txt and web scraping explains the broader protocol. [8]

Before changing a publisher's configuration:

1. Record the desired outcome for training and answer-time use, including any provider-specific exceptions.
2. Check the provider's rollout and transition guidance against existing settings.
3. Preserve the current configuration, make the change and inspect the robots.txt response actually served by the site, including any CDN-managed additions.
4. Test representative paths and retain the checked date and configuration with the change record.

Save the served file and test representative paths

This illustrative shell example checks https://example.com/robots.txt. It saves the body, response headers and SHA-256 hash in files carrying the check date; the command prints the HTTP status separately.

checked_on=$(date -u +%F)
prefix="example-com-robots-${checked_on}"

curl --silent --show-error \
  --dump-header "${prefix}.headers" \
  --output "${prefix}.txt" \
  --write-out 'HTTP %{http_code}\n' \
  'https://example.com/robots.txt' &&
sha256sum "${prefix}.txt" > "${prefix}.sha256"

Read the status before interpreting the body. Under RFC 9309, a 4xx response such as 404 allows crawlers to access any resource under the protocol; a 404 supplies no robots.txt preferences. If the file is unreachable because of a 5xx error, crawlers must assume complete disallow. Keep that distinction with the snapshot so a retrieval failure is not mistaken for a published training refusal. [8]

RFC 9309 requires crawlers to parse at least 500 KiB. Parsing beyond that size is not guaranteed, so an oversized file expanded by a CDN can lose groups near the end in parsers that stop at that limit. Check the complete served file, not just the part edited at the origin. [8]

Python's standard-library urllib.robotparser needs particular care here: it applies the first matching rule in file order, not the longest match. This illustrative Python example uses synthetic rules and tests the same six example.com paths as the earlier table.

from urllib.robotparser import RobotFileParser

paths = [
    "/",
    "/guides/bottles",
    "/press-kit/",
    "/press-kit/logo.png",
    "/press-kit",
    "/press-kitchen/",
]

rule_orders = {
    "allow_first": ["Allow: /press-kit/", "Disallow: /"],
    "disallow_first": ["Disallow: /", "Allow: /press-kit/"],
}

for label, rules in rule_orders.items():
    parser = RobotFileParser()
    parser.parse(["User-agent: __usage_ai-training", *rules])
    results = [
        parser.can_fetch(
            "__usage_ai-training", "https://example.com" + path
        )
        for path in paths
    ]
    print(label, results)

The illustrative output follows the order of the path list:

allow_first [False, False, True, True, False, False]
disallow_first [False, False, False, False, False, False]

Both press-kit results flip when the lines are swapped. An RFC 9309 parser returns True for those two paths in either order because their Allow rule is the longest match. Write the shown Allow exception before the broader Disallow so first-match and longest-match parsers agree on these paths, and treat an RFC 9309 parser as the reference when results differ. [8]

This test checks rule resolution in the supplied synthetic group. It does not establish whether a provider has implemented the preference or how that provider uses collected content.

Fetching the new file confirms what the site serves. It cannot demonstrate how a provider subsequently uses the content.

Keep the intended use with the dataset

Consider a company collecting public product listings to monitor availability. Another team later wants the descriptions for a retrieval assistant. A third proposes fine-tuning a generative model on them. The source URLs may be identical, but the intended uses have changed.

Proposed use	Review focus
Recurring product monitoring	Applicable crawler rules, access conditions and the collection scope
Retrieving descriptions into generated answers	Whether source terms, permissions and usage preferences cover that answer-time use
Fine-tuning a generative model	The training preference and rights or agreement supporting the use

This checklist is an operational recommendation; the proposal does not prescribe it. A decision supporting the monitoring project should not silently cover later model training. Equally, a training refusal does not grant permission for the other uses.

Keep the source URL, retrieval time, policy snapshot and reviewed purpose in a source register that downstream users can find. Record excluded uses and an owner who can reassess a new proposal. For a combined dataset, retain that context per source so one source's conditions are not accidentally applied to every row.

This synthetic source register records three proposed uses of https://example.com/products/. Its dates, owners and review outcomes are illustrative. The illustrative SHA-256 calculation uses the complete example robots.txt above, including its final newline; it is not evidence of a live website check. The product path lies outside the press-kit exception.

Source	Robots.txt checked (date and hash)	Shared training preference found	Path exceptions	Reviewed purpose	Excluded uses	Owner	Next review
https://example.com/products/	2026-11-03; SHA-256 f443c189c33918bfefcdf29d6e9657ecd8bc688d337ea4052487211849b449fc	Refusal: Disallow: /	/press-kit/ favours training	Availability monitoring only	Training; other unreviewed uses	Data operations owner	When the hash changes or a new purpose is proposed
https://example.com/products/	2026-11-03; SHA-256 f443c189c33918bfefcdf29d6e9657ecd8bc688d337ea4052487211849b449fc	Refusal: Disallow: /	/press-kit/ favours training	Retrieval assistant; review pending	Assistant use until reviewed; training	Assistant project owner	Before the first assistant use
https://example.com/products/	2026-11-03; SHA-256 f443c189c33918bfefcdf29d6e9657ecd8bc688d337ea4052487211849b449fc	Refusal: Disallow: /	/press-kit/ favours training	Fine-tuning. Not approved; seek licence	Fine-tuning on the product listings	ML project owner	After a licence is agreed, before training

In Web Scraper Cloud, + Add time scraped records the retrieval time on every exported record, and a Virtual column can carry a source label derived from scraped columns into the same export. The policy snapshot and hash come from the curl step above and live in the register, not in the scraper.

If the required use is refused or remains unclear, seek an agreed feed, API, licence or direct permission before adding the material to that workflow.

Automate a tested collection scope

Once the source and intended use have been reviewed, build and test a sitemap in the Web Scraper browser extension. Inspect an actual sample export, then schedule the tested sitemap in Web Scraper Cloud. The validation guide covers representative records and limited test runs.

The Sitemap.xml Link selector's Discover from robots.txt option reads the Sitemap: lines declared in the same file that would carry the __usage_ai-training group. Those lines sit outside every group, so adding the training group does not change which sitemaps are discovered when the Sitemap declarations remain unchanged. Discovery is a natural moment to take the robots.txt snapshot for the source register. [8]

In a synthetic scheduling example for https://example.com/, a Daily schedule at 09:00 in the selected Europe/Riga scheduler time zone is paired with a downstream script that re-hashes robots.txt daily and flags the source for review when the hash changes, before the next export is used for a new purpose.

Automatic export sends the complete dataset after each scraping job finishes, with a separate export per job. Apply the register check downstream, in a script or spreadsheet, to the relevant job's export.

When the dataset gains a new purpose, revisit the source review even if the sitemap itself needs no changes.