How to block AI crawlers without disappearing from AI search

What is the difference between training crawlers and AI search crawlers?

Training crawlers collect content to build AI models, while search crawlers and user-triggered fetchers retrieve pages so an AI product can answer a question and link to you. Blocking the first group stops your text being used for training. Blocking the second group can remove you from AI answers.

The vendors now publish separate names for each job, so you can split the decision. A blanket “block all AI” rule is the blunt option. The rest of this guide shows the precise one. The background explainer covers how these bots sit alongside search engines and bad bots.

Which AI bots exist, and what does each one do?

The table below lists the main named crawlers, each checked against its vendor’s own documentation.

Vendor	User-agent or token	What it does	Suggested setting
OpenAI	GPTBot	Crawls content that may be used to train OpenAI’s models	Block if you object to training
OpenAI	OAI-SearchBot	Makes sites appear in ChatGPT search features	Allow
OpenAI	ChatGPT-User	Visits a page when a user asks ChatGPT to; robots.txt rules may not apply	Allow
Anthropic	ClaudeBot	Collects web content that may contribute to training	Block if you object to training
Anthropic	Claude-SearchBot	Crawls to improve the quality of Claude’s search results	Allow
Anthropic	Claude-User	Retrieves pages when a Claude user asks a question	Allow
Google	Google-Extended	A robots.txt token controlling use for Gemini training and grounding; no effect on Google Search	Your choice
Perplexity	PerplexityBot	Search crawler, not used for training	Allow
Perplexity	Perplexity-User	Fetches a page when a person asks; generally ignores robots.txt	Allow
Apple	Applebot-Extended	Does not crawl; controls whether Applebot-crawled content trains Apple’s models	Your choice
Common Crawl	CCBot	Builds an open web archive used by researchers and others	Your choice

Sources: OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl.

Two details matter. Google-Extended has no user agent of its own: Google says crawling uses its existing agents and the token only acts as a control. Applebot-Extended also does not crawl, so blocking it leaves Apple’s search crawling untouched.

What should your robots.txt say?

Your robots.txt should disallow the training bots and leave the search bots alone. Add the snippet below to the robots.txt file at the root of each domain and subdomain, since Anthropic notes the rules apply per subdomain.

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Allow AI search and user-requested fetchers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-User
Allow: /

Keep your existing rules for Googlebot and Bingbot untouched. This file is a defensive starting point. If you want your content in no AI product at all, you can disallow the search names too, but you will lose any citation traffic.

OpenAI says its search systems can take about 24 hours to react to a robots.txt change, and Perplexity gives the same window. Do not expect instant results.

How do you enforce the rules at your CDN?

A CDN rule enforces what robots.txt only requests. Cloudflare’s AI Crawl Control shows which AI services crawl your site, lets you allow or block individual crawlers, and tracks which crawlers follow your robots.txt. Cloudflare says it is available on all plans.

Cloudflare’s older Block AI bots setting targets verified bots classified as AI training crawlers, and unverified bots that behave like them. By default it leaves out mixed-purpose crawlers that do both training and search. The same page says the setting was due for deprecation on 15 September 2026, and names “Configure AI bot policies” under Security Settings as the replacement. It offers separate Search, Agent and Training presets, each with three choices: block on all pages, block on pages with ads, or allow.

For most small sites, set Training to block, and leave Search and Agent on allow. Check the dashboard after saving, because an old “block all bots” or bot-fight setting elsewhere can override your choice.

How do you check that it worked?

You can confirm the result in four steps.

1. Open yourdomain.com/robots.txt in a browser and confirm the new rules are live.
2. Look at your CDN’s AI crawler report after a few days. Training crawlers should show blocks, search crawlers should show allowed requests.
3. Search your own server logs for GPTBot and ClaudeBot. Requests should stop or return blocked responses.
4. Ask an AI search product about your business a week later, to confirm you still appear.

Unnamed scrapers will not appear in these lists and will not obey robots.txt. For those, use the rate limits and bot checks in the guide to stopping automated login attacks, which apply to scrapers too. A WAF is the usual home for those rules.

Should you block, allow or limit?

Your choice depends on what your site earns from visibility. A publisher who sells content may block training and accept a loss of AI citations. A local service business usually gains from being named in AI answers and has little to protect from training.

There is also a middle path. Anthropic says it supports the non-standard Crawl-delay extension in robots.txt, so you can ask ClaudeBot to slow down instead of stopping, for example with a Crawl-delay: 1 line. Anthropic also advises against blocking by IP address, because that stops the crawler from reading your robots.txt at all, and the opt-out may not hold.

Review the choice twice a year. Vendors add and rename crawlers, and Cloudflare’s own settings changed in 2026. Reread the vendor pages in the sources list above when you do.