United Kingdom version. Explains robots.txt against UK copyright law: the only text and data mining exception (CDPA s.29A) is for non-commercial research, and the government has dropped an opt-out exception as its preferred plan. US version
AI Crawler robots.txt Generator (UK)
Write robots.txt rules for GPTBot, ClaudeBot and Google-Extended, and see where robots.txt sits in UK copyright law, where text and data mining is research-only.
The interactive tool loads here when JavaScript runs.
About this toolWhat is the AI Crawler robots.txt Generator?
For a UK publisher, shop or membership site, robots.txt does a technical job: it tells well-behaved crawlers which paths to leave alone. Under the Copyright, Designs and Patents Act 1988 it is not the legal switch it has become in the EU, because the UK's text and data analysis exception in section 29A covers only computational analysis for non-commercial research.
The policy picture moved in March 2026. The government's report on copyright and AI, published under the Data (Use and Access) Act, says a broad copyright exception with an opt-out is no longer its preferred way forward and that it will not reform copyright law until it is confident the changes will work. Treat this as open policy and check the GOV.UK page before relying on it.
So the practical choice is the same one the tool offers: block training crawlers, keep AI search and user fetches, or block everything. Pick a preset, adjust bots one by one, and paste your existing file so the tool merges the AI rules without disturbing your other directives.
- Documented AI crawlers, fetchers and control tokens with purpose and doc links
- Three presets plus per-bot Allow, Block or Some paths
- Merge into an existing robots.txt with a line diff
- Copies * rules into allowed AI groups (RFC 9309 group matching)
- Date the bot list was last checked shown in the tool
- Runs in the browser, nothing is uploaded
Updated
How to use the AI Crawler robots.txt Generator
- Pick a preset. Block training, allow AI search is a common middle ground for publishers who want citations but not training use.
- Adjust individual bots. Filter the table by purpose to see training crawlers, control tokens, search indexers and user fetchers.
- For bots set to Some paths, list the paths they must not read.
- Paste your existing robots.txt and leave "Copy my * Disallow rules" on so allowed AI bots still respect private paths.
- Review the diff, then copy or download the new robots.txt and publish it at the root of your domain.
Worked examples
UK news site: block training, keep AI search
Input
Preset "Block training, allow AI search"; existing robots.txt with User-agent: * Disallow: /subscribers/
Output
8 agents get Disallow: / (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, MistralAI-Training); 17 search and user agents get Disallow: /subscribers/
The search and user agents keep reading public articles, so the site can still be cited in AI answers, but they inherit the subscriber block because the tool copies the * rules into their group.
UK online shop: keep checkout private
Input
Same preset, no existing robots.txt; set OAI-SearchBot to Some paths with /checkout/ and /account/
Output
OAI-SearchBot gets its own group with Disallow: /checkout/ and Disallow: /account/; the other search and user agents keep Allow: /
Product pages stay visible to ChatGPT search while basket and account pages are excluded. Repeat for other search agents if you want the same rule everywhere.
Background and insight
Most AI companies now run several agents with different jobs. OpenAI uses GPTBot to collect training data, OAI-SearchBot to index pages for ChatGPT search and ChatGPT-User when a person asks ChatGPT to open a page. Anthropic splits the same way with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity, Meta, Amazon and Mistral publish similar sets. Google-Extended and Applebot-Extended are different again: they are tokens read from robots.txt, not crawlers, and control whether content already fetched by Googlebot or Applebot may be used for Gemini or Apple model training. Blocking them does not affect search results.
Under RFC 9309 a crawler obeys the group that names its user agent and ignores the * group, so an AI group that says Allow: / also overrides your general Disallow rules unless you copy them in; this tool can do that for you. robots.txt is voluntary: operators of user-triggered fetchers, including OpenAI, Perplexity, Google and Meta, say these may not follow it because a person requested the page, and some scrapers ignore it entirely. Use server, CDN or firewall rules when you need a hard block.
Common use cases
News and content publishers
Opt out of model training while staying eligible for AI search citations.
SaaS documentation
Allow every AI agent so assistants can answer questions about your product accurately.
Private or paywalled areas
Let AI agents read marketing pages but keep them out of account and checkout paths.
Auditing an inherited robots.txt
Find out which AI bots an old file blocks and replace stale or retired tokens.
Tips and limitations
- Put an AI and data mining clause in your website terms as well. Under UK law, robots.txt alone is a crawl instruction, and your terms are where you state the licence you do or do not grant.
- Blocking Google-Extended does not affect Google Search crawling or ranking; it is a control token for Gemini use.
- Keep a dated copy of each robots.txt you publish. If the UK adopts a rights-reservation model later, a record of when you reserved could help.
- If you also serve EU readers from a .uk domain, EU law treats a machine-readable reservation as an opt-out from the commercial mining exception, so the same file does more work there.
Sources and further reading
- Copyright, Designs and Patents Act 1988, section 29A, legislation.gov.uk. Copies for text and data analysis for non-commercial research; contract terms restricting it are unenforceable (s.29A(5)).
- Report and impact assessment on Copyright and Artificial Intelligence, DSIT, DCMS and Intellectual Property Office. March 2026: an exception with opt-out is no longer the preferred way forward.
- Copyright and Artificial Intelligence (consultation), Intellectual Property Office. The December 2024 consultation and its options.
- RFC 9309: Robots Exclusion Protocol, IETF. Group matching rules the tool follows.
Frequently asked questions
Is robots.txt a legal opt-out from AI training in the UK?
Not under current UK law. The CDPA has no general commercial text and data mining exception to opt out of: section 29A allows copies for computational analysis only for non-commercial research. robots.txt is a technical signal that major crawlers say they follow.
What did the UK government decide on AI and copyright?
Its March 2026 report says a broad exception with an opt-out is no longer the preferred way forward. It plans to gather more evidence and engage on other approaches, and says it will not reform copyright law until it is confident changes meet its objectives.
Can my website terms override the s.29A research exception?
No. Section 29A(5) says a contract term that tries to prevent or restrict copying allowed by the section is unenforceable. Terms can still set out what you license for commercial uses.
How do I block GPTBot without losing ChatGPT search traffic?
Block GPTBot and leave OAI-SearchBot allowed. The "Block training, allow AI search" preset does this for OpenAI, Anthropic, Meta, Amazon, Apple, Google and Mistral in one step.
Does blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended does not change Google Search. Blocking Googlebot would, so the tool never touches it; only the AI-specific tokens and crawlers are listed.
Is there an in-depth guide to the AI Crawler robots.txt Generator?
Yes. “How to Make Your Website Visible to AI Assistants and AI Search” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.