AI Crawler robots.txt Generator

Choose which AI crawlers may read your site, from GPTBot and ClaudeBot to PerplexityBot and Google-Extended, and merge the rules into your robots.txt with a diff.

The interactive tool loads here when JavaScript runs.

About this tool

What is the AI Crawler robots.txt Generator?

Choose a policy for each AI crawler, fetcher and control token, and this generator writes the robots.txt groups and merges them into your existing file. Start from a preset: allow all AI, block model training while allowing AI search and user-triggered fetches, or block every AI agent. Then set any bot to Allow, Block or Some paths.

Paste your current robots.txt and the tool keeps your non-AI rules, removes AI user agents from old groups, adds a clearly marked AI block, moves Sitemap lines to the end and shows a line diff of what changed. Each bot in the table shows its operator, purpose, whether the operator says it follows robots.txt and a link to the operator's own documentation.

  • Documented AI crawlers, fetchers and control tokens with purpose and doc links
  • Three presets plus per-bot Allow, Block or Some paths
  • Merge into an existing robots.txt with a line diff
  • Copies * rules into allowed AI groups (RFC 9309 group matching)
  • Date the bot list was last checked shown in the tool
  • Runs in the browser, nothing is uploaded
Read the guide: How to Make Your Website Visible to AI Assistants and AI Search

Updated

How to use the AI Crawler robots.txt Generator

  1. Pick a preset. Block training, allow AI search is a common middle ground for publishers who want citations but not training use.
  2. Adjust individual bots. Filter the table by purpose to see training crawlers, control tokens, search indexers and user fetchers.
  3. For bots set to Some paths, list the paths they must not read.
  4. Paste your existing robots.txt and leave "Copy my * Disallow rules" on so allowed AI bots still respect private paths.
  5. Review the diff, then copy or download the new robots.txt and publish it at the root of your domain.

Worked examples

Block training, keep AI search

Input

Preset Block training, allow AI search, existing file blocking /admin/ for *

Output

GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent and others Disallow: /; OAI-SearchBot, Claude-SearchBot, PerplexityBot and user fetchers get Disallow: /admin/

Search and user agents keep access to public pages but still respect the private path.

Merging an old file

Input

Existing file with a GPTBot group and a CCBot group

Output

Both old groups removed, replaced by the generated block; other rules untouched

Leaving old AI groups in place would give a bot two groups whose rules are combined, which is easy to get wrong.

Background and insight

Most AI companies now run several agents with different jobs. OpenAI uses GPTBot to collect training data, OAI-SearchBot to index pages for ChatGPT search and ChatGPT-User when a person asks ChatGPT to open a page. Anthropic splits the same way with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity, Meta, Amazon and Mistral publish similar sets. Google-Extended and Applebot-Extended are different again: they are tokens read from robots.txt, not crawlers, and control whether content already fetched by Googlebot or Applebot may be used for Gemini or Apple model training. Blocking them does not affect search results.

Under RFC 9309 a crawler obeys the group that names its user agent and ignores the * group, so an AI group that says Allow: / also overrides your general Disallow rules unless you copy them in; this tool can do that for you. robots.txt is voluntary: operators of user-triggered fetchers, including OpenAI, Perplexity, Google and Meta, say these may not follow it because a person requested the page, and some scrapers ignore it entirely. Use server, CDN or firewall rules when you need a hard block.

Common use cases

  • News and content publishers

    Opt out of model training while staying eligible for AI search citations.

  • SaaS documentation

    Allow every AI agent so assistants can answer questions about your product accurately.

  • Private or paywalled areas

    Let AI agents read marketing pages but keep them out of account and checkout paths.

  • Auditing an inherited robots.txt

    Find out which AI bots an old file blocks and replace stale or retired tokens.

Tips and limitations

  • Blocking GPTBot does not remove you from ChatGPT search; that is OAI-SearchBot. The same split applies to Claude.
  • Crawlers cache robots.txt. Amazon says it may take up to 30 days to see a change.
  • Tokens such as anthropic-ai and Claude-Web are no longer on Anthropic's list; Bytespider and cohere-ai have no official documentation, so they are left out here.
  • Verify suspicious traffic with the IP ranges or reverse DNS that operators publish; user agent strings are easy to fake.
  • For general crawl rules, sitemaps and search engine bots, use the separate robots.txt Generator.

Frequently asked questions

How do I block GPTBot but stay in ChatGPT search?

Disallow GPTBot and allow OAI-SearchBot. OpenAI uses GPTBot for training and OAI-SearchBot for search, and treats them separately.

What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?

ClaudeBot collects training data, Claude-SearchBot indexes pages for Claude's search results and Claude-User fetches a page when a Claude user asks.

Is Google-Extended a crawler?

No. It is a token read from robots.txt. Blocking it stops content being used for Gemini training and grounding but does not change Google Search crawling or ranking.

Do AI crawlers obey robots.txt?

The major training and search crawlers say they do. Several user-triggered fetchers, such as ChatGPT-User, Perplexity-User and Google-Agent, say robots.txt may not apply because a person asked for the page.

Will blocking AI crawlers remove my content from existing models?

No. robots.txt affects future crawling. Content already collected may remain in training data; check each operator's opt-out options.

Why does my AI group need the same Disallow rules as *?

A crawler uses only the most specific group that names it. If GPTBot has its own group, it ignores the rules under *.

How do I block bots that ignore robots.txt?

Use server, CDN or web application firewall rules that match the user agent and, where published, the operator's IP ranges.

Is there an in-depth guide to the AI Crawler robots.txt Generator?

Yes. “How to Make Your Website Visible to AI Assistants and AI Search” covers the background, how it works, common pitfalls and worked examples. Read the full guide.

Related Tools You May Like

Free, in-browser tools that pair well with this one.