Trust and compliance · 2026-06-04

OAI-SearchBot ChatGPT Search visibility: robots.txt and referral measurement

Manage OAI-SearchBot ChatGPT Search visibility with robots.txt, noindex, GPTBot separation, UTM referral measurement, and LLMTR public-content boundaries.

Why OAI-SearchBot and GPTBot should be separated

OpenAI search and training crawlers should not be treated as the same thing. OAI-SearchBot relates to discovery and citation in ChatGPT Search, while GPTBot should be evaluated separately for training preferences.

That distinction matters for LLMTR. Public blog, docs, and model pages can be discoverable, but user data, private dashboard surfaces, API keys, and internal systems must remain protected independently of crawler policy.

Handle search and training crawlers separately.
robots.txt communicates public discovery preferences.
Security belongs in auth, authorization, and network boundaries.
Private user content should not be exposed to public crawlers.

How robots.txt, noindex, and llms.txt fit together

robots.txt is the first signal for crawler access, noindex is for public pages that should not be indexed, and llms.txt is a helpful map of important public content.

These tools help manage visibility, but they do not guarantee it. For LLMTR, the goal is to make useful public pages understandable while preserving security in code and infrastructure.

Public blog and docs can remain discoverable.
Private application pages are not crawler targets.
llms.txt is not a special ranking mechanism.
Pages that need noindex should not be hidden from the crawler that must read the tag.

Measuring ChatGPT referrals

OpenAI publisher guidance says ChatGPT Search referral traffic can be tracked in analytics systems. For LLMTR, this helps connect AI Search visibility to product-funnel movement.

Referral measurement is not enough by itself. Also check whether the same session transitions to docs, model catalog, pricing, or signup pages.

Check ChatGPT referral sources in GA4 and server logs.
Segment UTM parameters separately.
Measure blog-to-docs and blog-to-model transitions.
Do not confuse crawler hits with user referral traffic.

The LLMTR public-content boundary

When targeting ChatGPT Search visibility, public content scope should be explicit. Blog posts, docs, model catalog pages, and policy pages can be source candidates; dashboard, user data, and secret-bearing surfaces are not public.

This boundary should be enforced through route guards, auth, secure headers, server logging, and data-retention policy, not just robots.txt.

Do not publish API keys or secrets.
Do not expose user prompts or response bodies publicly.
Protect dashboard routes with auth.
Crawler policy is not a substitute for the security model.

Review OAI-SearchBot and ChatGPT Search visibility

Safely review ChatGPT Search discovery, crawler preferences, and referral measurement for LLMTR public content.

List public surfaces. Separate blog, docs, model catalog, pricing, and policy pages as public discovery surfaces.
Check crawler preferences. Verify OAI-SearchBot, GPTBot, and other AI crawler rules according to their purpose.
Connect referral measurement. Match ChatGPT UTM and referrer traffic with GA4, server logs, and product transitions.
Verify the security boundary. Confirm private dashboard, API keys, user data, and internal routes are not exposed to public crawlers.

Frequently asked questions

Is allowing OAI-SearchBot the same as allowing GPTBot?

No. OAI-SearchBot relates to ChatGPT Search discovery, while GPTBot relates to training preferences. Treat each user-agent as a separate policy.

Is llms.txt required for ChatGPT Search visibility?

No. llms.txt is a helpful public content map. ChatGPT Search visibility also depends on crawler access, page quality, relevance, and citation behavior.

How should ChatGPT referral traffic be reported?

Use GA4, server logs, and product analytics together, then track whether ChatGPT-sourced sessions move to docs, model pages, pricing, and signup.