Ireland version. Explains robots.txt as a machine-readable reservation under section 53B of the Copyright and Related Rights Act 2000 (S.I. 567 of 2021) and the EU AI Act duties on model providers. US version

AI Crawler robots.txt Generator (Ireland and EU)

Write a machine-readable text and data mining opt-out for GPTBot, ClaudeBot and Google-Extended under section 53B of the Irish Copyright Act and the EU AI Act.

The interactive tool loads here when JavaScript runs.

About this tool

What is the AI Crawler robots.txt Generator?

In Ireland, commercial text and data mining of lawfully accessible content is allowed unless the rights holder has reserved it. Section 53B of the Copyright and Related Rights Act 2000, inserted by S.I. No. 567 of 2021, says the reservation must be machine-readable for content made publicly available online, including metadata and the terms and conditions of a website or service. A robots.txt rule that blocks an AI training crawler is one such machine-readable signal.

The other half of the picture sits with the AI companies. Article 53(1)(c) of the EU AI Act requires providers of general-purpose AI models to have a policy to identify and comply with reservations made under Article 4(3) of the Copyright Directive, and that duty has applied since 2 August 2025. Signatories of the EU's General-Purpose AI Code of Practice also commit to using crawlers that read and follow robots.txt as specified in RFC 9309.

So for an Irish publisher, shop or membership site, the file this tool writes does legal work as well as technical work. Pick a preset, adjust each bot, paste your current robots.txt to merge the rules, and pair it with a matching reservation in your website terms.

  • Documented AI crawlers, fetchers and control tokens with purpose and doc links
  • Three presets plus per-bot Allow, Block or Some paths
  • Merge into an existing robots.txt with a line diff
  • Copies * rules into allowed AI groups (RFC 9309 group matching)
  • Date the bot list was last checked shown in the tool
  • Runs in the browser, nothing is uploaded
Read the guide: How to Make Your Website Visible to AI Assistants and AI Search

Updated

How to use the AI Crawler robots.txt Generator

  1. Pick a preset. Block training, allow AI search is a common middle ground for publishers who want citations but not training use.
  2. Adjust individual bots. Filter the table by purpose to see training crawlers, control tokens, search indexers and user fetchers.
  3. For bots set to Some paths, list the paths they must not read.
  4. Paste your existing robots.txt and leave "Copy my * Disallow rules" on so allowed AI bots still respect private paths.
  5. Review the diff, then copy or download the new robots.txt and publish it at the root of your domain.

Worked examples

Irish news publisher: reserve training, stay in AI search

Input

Preset "Block training, allow AI search"; existing file with User-agent: * Disallow: /subscribers/

Output

8 agents get Disallow: / (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Amazonbot, MistralAI-Training); 17 search and user agents get Disallow: /subscribers/

The training crawlers and the Google and Apple control tokens carry the reservation; AI search agents can still cite public articles but inherit the subscriber block.

Irish e-commerce site: keep account pages out of AI search

Input

Same preset, no existing robots.txt; OAI-SearchBot set to Some paths with /checkout/ and /account/

Output

OAI-SearchBot gets Disallow: /checkout/ and Disallow: /account/; other search and user agents keep Allow: /

Product pages remain available to ChatGPT search, while the reservation for training stays on GPTBot and the other training crawlers.

Background and insight

Most AI companies now run several agents with different jobs. OpenAI uses GPTBot to collect training data, OAI-SearchBot to index pages for ChatGPT search and ChatGPT-User when a person asks ChatGPT to open a page. Anthropic splits the same way with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity, Meta, Amazon and Mistral publish similar sets. Google-Extended and Applebot-Extended are different again: they are tokens read from robots.txt, not crawlers, and control whether content already fetched by Googlebot or Applebot may be used for Gemini or Apple model training. Blocking them does not affect search results.

Under RFC 9309 a crawler obeys the group that names its user agent and ignores the * group, so an AI group that says Allow: / also overrides your general Disallow rules unless you copy them in; this tool can do that for you. robots.txt is voluntary: operators of user-triggered fetchers, including OpenAI, Perplexity, Google and Meta, say these may not follow it because a person requested the page, and some scrapers ignore it entirely. Use server, CDN or firewall rules when you need a hard block.

Common use cases

  • News and content publishers

    Opt out of model training while staying eligible for AI search citations.

  • SaaS documentation

    Allow every AI agent so assistants can answer questions about your product accurately.

  • Private or paywalled areas

    Let AI agents read marketing pages but keep them out of account and checkout paths.

  • Auditing an inherited robots.txt

    Find out which AI bots an old file blocks and replace stale or retired tokens.

Tips and limitations

  • Add a text and data mining reservation to your website terms too. Section 53B names the terms and conditions of a website as a machine-readable place for it, and terms cover crawlers that never read robots.txt.
  • Date and keep each version of robots.txt you publish, so you can show when a reservation was in place.
  • The Code of Practice encourages signatories that run search engines to make sure honouring a training reservation does not directly harm your search indexing.
  • User-triggered fetchers such as ChatGPT-User act for a person who asked for a page; several operators say robots.txt may not apply to them. Use server or CDN rules if you need a hard block.
  • Text and data mining for research falls under a separate exception (section 53A), which a section 53B reservation does not switch off.

Sources and further reading

  1. S.I. No. 567/2021: European Union (Copyright and Related Rights in the Digital Single Market) Regulations 2021, Irish Statute Book (Office of the Attorney General). Regulation 4 inserts section 53B: machine-readable reservation for content online, including metadata and website terms.
  2. Directive (EU) 2019/790 on copyright in the Digital Single Market, EUR-Lex. Article 4(3): reservation by machine-readable means for content made publicly available online.
  3. Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex. Article 53(1)(c) and recital 106; Chapter V applies from 2 August 2025 (Article 113).
  4. The General-Purpose AI Code of Practice, European Commission. Copyright chapter, Measure 1.3: crawlers that follow robots.txt (RFC 9309).

Frequently asked questions

Does robots.txt count as an opt-out from AI training in Ireland?

It is a machine-readable means, which is what section 53B(3) requires for content made publicly available online, and the EU AI Act expects model providers to identify and respect such reservations. Whether a particular file is enough in a dispute is for the courts, so pair it with a reservation in your terms.

What does section 53B of the Copyright and Related Rights Act say?

It lets a person with lawful access reproduce a work for text and data mining unless the author has expressly reserved that use in an appropriate manner. For content online, the reservation must be machine-readable, including metadata and website terms and conditions.

Do AI companies have to respect my reservation?

Providers of general-purpose AI models placed on the EU market must, under Article 53(1)(c) of the AI Act, put in place a policy to identify and comply with Article 4(3) reservations. Recital 106 says this applies regardless of where the training takes place.

Will blocking AI training crawlers hurt my Google ranking?

Blocking Google-Extended does not change Google Search crawling. The GPAI Code of Practice also encourages signatories that run search engines to make sure honouring a training reservation does not directly harm indexing.

Is robots.txt enough on its own?

It covers crawlers that read it, which Code of Practice signatories commit to. It does not reach content already collected, scrapers that ignore it, or copies of your work on other sites. A reservation in your terms and server-side blocks fill some of those gaps.

Does the opt-out stop academic research mining?

No. Research mining has its own exception: section 53A in Irish law, and Article 3 of the Directive for research organisations and cultural heritage institutions doing scientific research. The 53B reservation applies to the general exception only, and Article 4(4) of the Directive says Article 4 does not affect Article 3.

Is there an in-depth guide to the AI Crawler robots.txt Generator?

Yes. “How to Make Your Website Visible to AI Assistants and AI Search” covers the background, how it works, common pitfalls and worked examples. Read the full guide.