Which AI Crawlers Should You Allow? A robots.txt Guide to GPTBot, ClaudeBot and PerplexityBot

By the alphaa team — we crawl and scan thousands of business websites, and misconfigured robots rules are one of the most common reasons a business is invisible to AI engines.

Short answer: allow the retrieval agents that fetch pages to answer live questions — OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User — because blocking them makes you ineligible to be cited in AI answers. The training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) are a separate, genuinely optional decision about whether your content may be used to train future models. Most businesses that want to be recommended by AI should allow both; businesses with proprietary content sometimes block training while keeping retrieval open.

Why there are two kinds of AI bot

An AI crawler is an automated client that requests your pages and identifies itself with a user-agent string. The critical distinction, and the one that trips people up, is what the fetch is for:

• Training crawlers collect large volumes of text to build or update a model. The content they gather may end up in the model's frozen knowledge, months later, after the next training run.
• Retrieval agents fetch pages in real time, either to maintain a search index the assistant queries, or because a user just asked a question and the assistant is going to read your page and cite it in the answer. This is the traffic that produces citations today.

These use different user agents, so you can treat them differently. That is the whole point of the design. The failure mode we see repeatedly: someone reads a "block AI scrapers" blog post, pastes a blanket rule, and unknowingly removes their business from the retrieval layer that AI assistants use to answer "who is the best plumber near me?"

The user agents that actually matter

Grouped by operator, with what each one is for:

Each operator publishes its own list, and the lists change — OpenAI documents its bots at platform.openai.com/docs/bots, Anthropic in its crawler support article, Perplexity at docs.perplexity.ai/guides/bots, and Google at developers.google.com. Check the source rather than trusting a copied list, including this one.

The Google-Extended misunderstanding

This one is worth stating precisely, because it is the most common piece of bad advice in circulation. Google-Extended is not a crawler. It is a product token that controls whether content Google has already crawled may be used to train Gemini models and to ground Gemini Apps and the Vertex AI API. Google's documentation states plainly that it does not affect a site's inclusion in Google Search and is not a ranking signal.

The practical consequence: blocking Google-Extended does not remove you from AI Overviews. AI Overviews are part of Search and are served from the regular Google index via Googlebot. If you want to limit how your content appears there, the levers are snippet controls like nosnippet, max-snippet, and data-nosnippet — and those also reduce your normal search snippets, which is usually a bad trade. For how AI Overviews actually pick businesses, see AI Overviews for local businesses.

How to configure robots.txt, step by step

Here is the workflow we run when auditing a site.

Step 1: read what you currently serve

Open https://yourdomain.com/robots.txt in a browser. Do not trust the file in your repo or your plugin's settings screen — trust what the server actually returns. Plugins, CDNs, and host-level rules all silently override each other, and we regularly find a live file that nobody on the team wrote.

Step 2: look for the blanket block

The rule that does the damage looks like one of these. If you see either, an AI engine that honors robots rules will not read any page on your site:

User-agent: *
Disallow: /

# or, more subtly:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Disallow: /

The second block is the one people paste deliberately, then forget. It blocks PerplexityBot, which is the index behind every Perplexity citation — so the business opts itself out of Perplexity answers entirely while still hoping to show up in them.

Step 3: write the rules you actually mean

For a local business or SaaS that wants to be found and recommended, the honest default is to let everything in. Public marketing content is meant to be read:

# Allow all crawlers, including AI retrieval and training
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you genuinely do not want your content training future models but still want to be cited today, split the decision by user agent:

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Stay eligible for citations in AI answers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Two mechanics to get right. Robots.txt matching uses the most specific user-agent group, and a bot only obeys the one group that matches it — so a bot named in its own group ignores the User-agent: * block entirely. And robots.txt controls crawling, not what a model already learned in a previous training run; blocking today does not unlearn yesterday.

Step 4: verify with your server logs

This is the step almost everyone skips, and it is the only one that proves anything. Filter your access logs or CDN analytics for the user-agent strings above over the last 30 days. You are looking for two things: which AI bots are actually visiting, and what status codes they get. A crawler receiving 403s from your WAF is being blocked regardless of what robots.txt says — bot-protection rules on Cloudflare and similar services frequently catch AI crawlers by default, and that block is invisible in robots.txt.

Honest limitations

• robots.txt is voluntary. It is a request, not enforcement. Major operators state that they honor it, but bad actors do not, and the only hard control is blocking at the server or WAF level.
• Allowing crawlers does not make you get cited. Access is necessary, not sufficient. It removes a blocker; it does not create the reviews, structured data, and consistent entity signals that decide whether an engine names you.
• Bot names change. Operators add and rename agents. A rule list written in 2024 is already partly stale. Re-check the official docs every few months.
• Nobody can guarantee inclusion in an AI answer. Engines vary their outputs by phrasing, user, and model version. You control the inputs, not the output.

Frequently asked questions

Will blocking GPTBot hurt my visibility in ChatGPT?

Not directly, if you leave OAI-SearchBot and ChatGPT-User allowed. GPTBot is the training crawler; the other two handle search and live fetching, which is what produces citations in a ChatGPT answer today. Blocking all three does remove you.

Do I need an llms.txt file as well as robots.txt?

They do different jobs. robots.txt grants or denies access; llms.txt offers a clean summary of who you are and where your key pages live. llms.txt is an emerging convention with partial adoption, so treat it as a helpful extra rather than a replacement.

Should a small local business bother with any of this?

Yes, but as a five-minute check rather than a project. Confirm your robots.txt is not blocking AI bots, confirm your firewall is not either, then spend your remaining effort on reviews, accurate business details, and answer-shaped content. Access is the floor, not the strategy.

The bottom line

Treat crawler access as two separate decisions. Retrieval agents should almost always be allowed — they are the mechanism by which AI assistants find and cite you right now. Training crawlers are a real choice with real trade-offs, and blocking them is defensible. What is never defensible is a blanket block pasted without knowing which bots it catches. Read your live robots.txt, check your logs for 403s, then go fix the signals that actually earn the recommendation.

Run a free AI visibility scan →

Last updated July 27, 2026.