Robots.txt for AI Crawlers: How to Allow or Block GPTBot, ClaudeBot, and PerplexityBot in 2026

A robots.txt for AI crawlers is the same file you already publish, with rules addressed to a new set of user agents. The mistake that costs the most is treating those agents as one switch, because each provider now runs separate crawlers for training, for search, and for fetches a person triggered.

OpenAI states the point in its crawler documentation: "Each setting is independent of the others." A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of model training.

Google draws the same line from the other direction. Its list of common crawlers, last updated 14 July 2026, states that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Key Takeaways

• There is no single AI crawler: OpenAI documents four user agents and Anthropic three. Google lists several relevant controls, but Google-Extended is a robots.txt token, not a separate crawler. Perplexity documents two agents.
• Blocking GPTBot does not remove you from ChatGPT: GPTBot is the training crawler. OAI-SearchBot is the one that surfaces your site in ChatGPT's search features.
• You cannot opt out of AI Overviews and stay in Google Search: Google documents Googlebot directives as the control for AI features, because AI is built into Search itself.
• A robots.txt allow means nothing if your firewall blocks the same agent: Perplexity publishes WAF setup guidance for Cloudflare and AWS precisely because this is the common failure.
• IP blocking backfires: Anthropic states that blocking its IP addresses impedes its ability to read your robots.txt, so it cannot guarantee the opt-out you wanted.

What AI Crawlers Are — and Why Your Robots.txt Suddenly Matters

An AI crawler is a bot that fetches your pages for training, search, or a user-directed request. The providers have split them by job, and the split is the whole story.

Three jobs, three kinds of agent. Training crawlers collect text that may enter a foundation model. Search crawlers gather content that assistants can use when answering questions. User-triggered fetchers visit a page because someone asked a question about it.

Robots.txt matters now because it is the only control most providers offer. Anthropic's guidance for site owners states that opting out "requires modifying the robots.txt file." There is no dashboard and no account.

The file is also the cheapest thing to get wrong. One line can remove you from an engine your buyers use. The access layer sits upstream of everything else, including schema markup for AI search, so a page an engine cannot read cannot be described, ranked, or cited.

Why the Old Mental Model Breaks

Under classic SEO, one crawler served one search engine and one directive governed it. That one-to-one mapping is gone.

OpenAI alone publishes four agents with four different effects on your visibility. Writing User-agent: * and Disallow: / still works, and it now costs far more than it used to.

The Full List of AI Crawler User-Agents: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and More

Every row below is taken from the provider's own documentation, read on 16 September 2026. The robots.txt token is what you write in the file.

Two rows carry most of the confusion. A GPTBot robots.txt rule governs training and nothing else, and Google-Extended is a control token with no crawler behind it. Google states that Google-Extended "doesn't have a separate HTTP request user agent string."

Perplexity's split runs the other way from OpenAI's. Perplexity's crawler documentation states that PerplexityBot "is not used to crawl content for AI foundation models." Blocking it protects nothing from training and removes you from Perplexity's results.

Robots.txt Guide: Syntax, Directives, and the Mistakes That Break It

This robots.txt guide covers what the AI-crawler decision needs, not how to author a file from scratch. One rule decides more AI opt-outs than any other: directives apply only to the host, protocol, and port that serve the file, so a block on your root domain leaves your docs subdomain wide open.

A group is a user-agent line plus its rules. Agents match one token each, and the most specific matching group wins.

text
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /

The longest matching path wins, and order is irrelevant. Google's robots.txt specification page states that crawlers "use the most specific rule based on the length of the rule path", and that conflicting rules of equal length resolve to the least restrictive one. A longer matching path beats a shorter one, whether the rule says Allow or Disallow. In the example below, Allow: /docs/ overrides Disallow: / for that directory.

text
User-agent: ClaudeBot
Disallow: /
Allow: /docs/

The Four Mistakes That Break the File

Mistake one: a blanket wildcard you forgot about. A User-agent: * group with Disallow: / blocks agents that have no more specific matching group. A separate group for a named agent can override that wildcard, but only if the agent matches the name you wrote.

Mistake two: expecting an instant effect. OpenAI states that for search results "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Perplexity says changes may take "up to 24 hours" to reflect, and Google caches robots.txt for up to 24 hours of its own.

Mistake three: blocking the file itself. If a crawler cannot fetch robots.txt, it cannot read your rules. Anthropic warns that IP blocking "impedes our ability to read your robots.txt file."

Mistake four: assuming Crawl-delay works everywhere. Anthropic supports it and calls it a non-standard extension. Google's specification page lists the fields it supports and states that "other fields such as crawl-delay aren't supported", so a Crawl-delay line aimed at Googlebot does nothing.

User-agent: ClaudeBot
Crawl-delay: 1

How to Block AI Crawlers (and When You Actually Should)

The question of how to block AI crawlers has a clean answer and a messy one. The clean answer is the syntax. The messy answer is which of the three jobs you actually object to.

Blocking training is a defensible business decision. Publishers with licensing revenue, sites with proprietary research, and anyone whose content is the product have a real argument. The providers built separate tokens so you could make it.

text
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /

That file blocks the documented training crawlers at OpenAI and Anthropic, and uses Google-Extended to opt out of the Gemini uses it controls. It leaves the search crawlers unblocked. ChatGPT search, Claude’s search, Perplexity, and Google Search can still access you through their separate agents. It is the configuration most teams think they are writing when they "block AI".

The Google-Extended line carries one cost worth knowing about. Google documents that token as governing Gemini training and grounding, which it defines as "providing content from the Google Search index to the model at prompt time." Disallowing it keeps you in Google Search and takes you out of Gemini's grounded answers.

Blocking search retrieval is a different decision and a much larger one. Disallowing OAI-SearchBot, Claude-SearchBot, and PerplexityBot removes you from the answers your buyers read. Perplexity's own documentation recommends allowing PerplexityBot to appear in results.

What a Block Cannot Do

A blanket block does not stop a user-directed fetch. OpenAI states that because ChatGPT-User actions are initiated by a person, "robots.txt rules may not apply." Perplexity is blunter: Perplexity-User "generally ignores robots.txt rules."

A block does not retroactively remove anything. Anthropic's wording is about "future materials", not about text already in a training set. Nothing in robots.txt reaches backwards.

How to Allow AI Crawlers to Maximize Citations

The permissive configuration is short, and the work is in confirming that nothing downstream contradicts it.

text
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xml

Google's own advice names the infrastructure explicitly. Its guidance on AI features lists "ensuring that crawling is allowed in robots.txt, and by any CDN or hosting infrastructure" as an SEO fundamental that still applies.

The firewall is where allow rules go to die. Perplexity publishes step-by-step WAF configuration for Cloudflare and AWS, and publishes IP ranges for both its agents, because a permissive robots.txt plus a default bot rule is a silent block.

Allowing access is the prerequisite, not the tactic. To appear in Google’s AI Overviews, a page must also be indexed and eligible for a snippet. Elsewhere, access gives an engine the chance to read you, not a citation. What you do with the page still matters.

Verify Both Agents and Addresses

Every provider publishes its IP ranges as JSON: OpenAI per agent, Anthropic in one file, Perplexity per agent, Google in a common-crawlers object. Match on user agent and address together, which is what Perplexity recommends and what stops a spoofed agent string walking through your WAF rule.

Llms.txt vs. Robots.txt: Which File Controls What

The llms.txt vs robots.txt question is a category difference, not a feature comparison. One file is an access directive, the other is a proposed index.

Robots.txt is enforced by convention that providers document in writing. Every automatic crawler in the table above has published a statement about honoring it, which is what makes it a control. The two user-triggered fetchers are the documented exceptions.

llms.txt asks rather than instructs. It has no standing to permit or refuse anything, so it cannot substitute for a robots.txt rule. What the specification proposes, and which engines have said anything about reading one, belongs to the markdown index for AI crawlers.

Testing and Auditing Your AI Crawler Access

Reading your own file is not a test. The file can be correct while the request still fails three layers later.

Fetch your robots.txt as each agent. Send the published user-agent string from outside your network and confirm a 200 and the expected body. A WAF that challenges unknown agents fails here, not in the file.

Read your server logs by user agent. The agents in the table above identify themselves, and their published IP ranges let you confirm the request was genuine. If an expected crawler is absent from your logs, check whether it tried to reach you before treating the gap as a block. Allowing an agent does not guarantee that it will visit.

Check the CDN rules separately. Bot-management defaults change on vendor schedules, and nobody tells the SEO team. This is the layer Google names alongside robots.txt in its own guidance.

Access is step one of a wider check, and running the audit by hand covers the prompt set and what to record per answer. Once the agents are reading you, measuring the result is a separate discipline, and what a citation record contains sets the baseline you compare against.

What Klarivo Checks Before Anything Else

Crawler access is the first thing worth confirming, because every other investment in answer engine optimization assumes an engine can read the page. A blocked agent makes good content invisible and the reporting looks identical to content that simply was not chosen.

Klarivo Monitor tracks ChatGPT, Claude, Perplexity, Gemini, and Grok independently, refreshed on a schedule the client controls. Its "Top 10 Citation Sources" surface reports the domains the engines drew on for a category, which is where an access problem shows up as a pattern rather than a guess. When four engines cite you and one never does, the gap is a question you can take straight to that provider's user agent.

Crawler access is cheap to check and expensive to ignore. A blocked agent will not read the next page you publish, however good that page is. Book a Klarivo discovery call to walk through your crawler access and what the engines currently read.

Frequently Asked Questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot crawls for foundation-model training. OAI-SearchBot is the agent that surfaces sites in ChatGPT's search features, and OpenAI states the two settings are independent. A site that disallows GPTBot alone stays eligible for ChatGPT search.

Can I stay in Google Search but opt out of AI Overviews?

Not through robots.txt. Google states that AI is built into Search, which is why Googlebot directives are the control for how sites are crawled for Search. To limit what is shown rather than whether you appear, Google points to the nosnippet, data-nosnippet, and max-snippet controls.

Do I need a separate robots.txt for each subdomain?

Yes. Google's specification states that the rules apply only to the host, protocol, and port serving the file, so a rule on the root domain does not cover docs.example.com. Anthropic asks site owners to repeat the block "for every subdomain that you wish to opt out from."

Should I block AI crawlers by IP address instead?

No. Anthropic states that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file." Use IP ranges to verify a crawler is genuine, not to block it.

How long does a robots.txt change take to have an effect?

OpenAI documents roughly 24 hours for its search systems to adjust, and Perplexity documents up to 24 hours as well. Google's crawl cadence varies by page, so a change there can take days to months to propagate.