Trust and visibility ยท 2026-09-22

Managing GPTBot, PerplexityBot, and ClaudeBot: a robots.txt guide

Covers AI crawlers beyond OAI-SearchBot (GPTBot, PerplexityBot, ClaudeBot, Google-Extended) separately, and explains why blocking the training crawler does not block the search crawler.

Rule-list diagram showing training-focused crawlers blocked and search-focused crawlers allowed within the same robots.txt file.

There is no single 'AI bot'

OAI-SearchBot, covered in the OAI-SearchBot ChatGPT Search visibility article, is a search-only crawler and a separate user agent from OpenAI's training-data crawler, GPTBot. The same distinction holds for other providers: Anthropic's ClaudeBot, Perplexity's PerplexityBot, and Google's Google-Extended tag are each managed by their own robots.txt rule.

Robots.txt has no 'AI bots' category. A `User-agent: *` group applies to every crawler that has no more specific group of its own, so it covers AI crawlers, but also search crawlers such as Googlebot and Bingbot. If you only want to block particular AI crawlers, you have to name them in their own user-agent groups.

Blocking the training crawler doesn't affect search visibility

A site not wanting its content used for model training and wanting to be cited as a source in products like ChatGPT Search, Perplexity, or Bing Copilot are not contradictory; these are separated by different user agents. Blocking GPTBot while allowing OAI-SearchBot means refusing use of content for model training while still being able to appear as a source in search results.

Blocking every user agent that contains 'GPT' or 'AI' wholesale, without making this distinction, can cause a site to unintentionally drop out of search visibility as well.

  • GPTBot (OpenAI, training) and OAI-SearchBot (OpenAI, search) are separate user agents.
  • ClaudeBot (Anthropic) and PerplexityBot (Perplexity) are managed by their own rules.
  • Google-Extended is a permission tag separate from Google's general search crawler.
  • Blocking one category does not automatically block another; write each line separately.

Review rule changes periodically

Providers can add new user-agent names or change an existing one's behavior; rather than writing robots.txt once and forgetting it, periodically checking providers' official crawler documentation keeps the rules current.

Frequently asked questions

If I block GPTBot, will I also disappear from ChatGPT Search?

No, they are separate user agents. GPTBot collects training data, while OAI-SearchBot crawls content for search results. Blocking GPTBot while allowing OAI-SearchBot lets you manage the two independently.

Can I block every AI crawler with a single rule?

A single `User-agent: *` group covers every crawler that has no group of its own, but it blocks search engine crawlers as well as AI crawlers. A crawler that is named in its own group ignores the wildcard group and follows only its own. To block AI crawlers while keeping search visibility, name each of them in a separate group.

Related posts