Should I block GPTBot?

It is a business decision, not a technical default. Blocking removes your content from ChatGPT-era answers and future training; allowing trades content for visibility in AI surfaces. For most businesses that want to be found and cited, allowing search/agent crawlers is the pragmatic choice.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls use of your content for Gemini training and grounding; normal Googlebot indexing and ranking are unaffected.

Do AI crawlers respect robots.txt?

The major named crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) publicly commit to respecting robots.txt. Unnamed scrapers may not - robots.txt is a policy signal, not an enforcement mechanism.

AI crawlers explained - and what to put in robots.txt

A handful of named crawlers now decide whether your site exists in AI answers. Each serves a different purpose - training, live search, or agent browsing - and each can be allowed or refused independently in robots.txt. Here is the practical map.

The crawlers that matter

User-agent	Operator	What it feeds
`GPTBot`	OpenAI	Model training and ChatGPT browsing corpus
`OAI-SearchBot`	OpenAI	ChatGPT search results (live citations)
`ClaudeBot`	Anthropic	Claude training and retrieval
`Claude-User`	Anthropic	Fetches made on behalf of a Claude user/agent
`PerplexityBot`	Perplexity	Perplexity answer engine index (cited sources)
`Google-Extended`	Google	Gemini training/grounding - separate from Googlebot ranking
`CCBot`	Common Crawl	Open web corpus used by many research/AI projects
`Bytespider`	ByteDance	Model training

Decide deliberately, not by default

Three coherent policies - pick one on purpose:

1. Open (be findable in AI surfaces)

# AI crawlers welcome
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

2. Search yes, training no

# Allow live AI search/citation, refuse training corpora
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

3. Closed

User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /

Whatever you choose: keep a Sitemap: line in robots.txt, and remember an empty or missing robots.txt means "everything allowed" to compliant crawlers - a decision by omission.

How this affects your audit

Our 30-check audit reads your robots.txt and reports whether AI crawlers have explicit rules - silence scores lower than a deliberate policy, whichever direction you pick, because agents and their operators reward clarity.

Check your robots.txt and full readiness now

Frequently asked

Should I block GPTBot?: It is a business decision, not a technical default. Blocking removes your content from ChatGPT-era answers and future training; allowing trades content for visibility in AI surfaces. For most businesses that want to be found and cited, allowing search/agent crawlers is the pragmatic choice.
Does blocking Google-Extended hurt my Google rankings?: No. Google-Extended only controls use of your content for Gemini training and grounding; normal Googlebot indexing and ranking are unaffected.
Do AI crawlers respect robots.txt?: The major named crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot) publicly commit to respecting robots.txt. Unnamed scrapers may not - robots.txt is a policy signal, not an enforcement mechanism.