aiseocheck.hu / Articles / Guide

AI Bots in robots.txt: GPTBot, ClaudeBot, PerplexityBot and More

A practical guide to naming AI user-agents in robots.txt to ensure your content reaches modern AI search engines.

By Zoltán Varga2026-11-054 min read

Which AI User-Agents Exist

Modern AI search engines deploy distinct crawlers separate from traditional search bots. These agents include GPTBot for OpenAI, ClaudeBot for Anthropic, and PerplexityBot for Perplexity AI. Other notable agents include Google-Extended for Google's AI training, Applebot-Extended for Apple Intelligence, and CCBot for Common Crawl data collection.

You must identify these specific names to grant or deny access correctly. A generic User-agent: * rule often fails to address the unique requirements of these new entities. Some bots, like ChatGPT-User, operate differently than standard indexing crawlers. Knowing the exact string is essential for precise control over your content visibility.

Source audits reveal that omitting these names leaves sites vulnerable to unintended blocking or uncontrolled scraping. The list expands as new models launch their own indexing infrastructure. You should verify the current names against official documentation from each provider. As of March 2026, the core list remains stable but requires regular monitoring for updates.

• GPTBot (OpenAI)
• ClaudeBot and Claude-Web (Anthropic)
• PerplexityBot (Perplexity AI)
• Google-Extended (Google)
• Applebot-Extended (Apple)
• CCBot (Common Crawl)
• Bytespider (ByteDance)

Allow and Disallow Syntax

The Robots Exclusion Protocol uses simple text rules to manage crawler behavior. You place the User-agent name on one line and the action on the next. To allow access, write User-agent: GPTBot followed by Allow: /. To block access, replace Allow with Disallow: /.

Specific rules always override general ones in the file structure. If you define a rule for GPTBot later in the file, it takes precedence over a generic User-agent: * rule defined earlier. This order matters significantly when you want to permit a specific bot while restricting others.

You can also restrict access to specific directories rather than the whole site. Use Allow: /public/ and Disallow: /private/ to create granular permissions. This approach lets you share public content while protecting sensitive data from AI ingestion. Always test these paths to ensure they match your actual URL structure.

• Place specific bot rules before generic wildcard rules.
• Use Allow: / to permit full site access.
• Use Disallow: / to block full site access.
• Target specific folders with path-based rules.

Wildcards and Inheritance Traps

The wildcard User-agent: * applies to any bot not explicitly named above it. Many site owners rely on this single line to manage all traffic. This creates a trap when new AI bots arrive that do not recognize the generic rule or require specific handling. They may default to blocked status if the site has strict policies.

Conflicting rules within the same file cause unpredictable results. If you disallow everything for the wildcard but then allow a specific bot, the specific rule wins. However, if you list the specific bot after a broad disallow without an explicit allow, the bot remains blocked. Precision in ordering prevents these logic errors.

Some crawlers ignore the wildcard entirely and only read their specific agent lines. If you only have a User-agent: * rule, these bots might skip your site or assume a default deny policy. Explicitly naming GPTBot, ClaudeBot, and others ensures they process your instructions correctly regardless of their internal parsing logic.

• Order specific rules before generic wildcards.
• Avoid mixing broad disallows with specific allows without clear intent.
• Check that every new AI bot has its own named section.

Content-Signal Lines and Header Conflicts

Your robots.txt file is not the only signal controlling AI access. HTTP response headers like X-Content-Signal can override file-level permissions. A site might allow GPTBot in robots.txt but send a noai header with every page request. Sophisticated crawlers prioritize these headers and will refuse to index the content.

This contradiction creates a critical failure point for AI visibility. You might see your robots.txt permitting access while your analytics show zero AI impressions. The header acts as a hard stop that the file cannot override. You must align both the file and the server configuration to ensure consistent signals.

Recent audits found sites with robots.txt allowing all AI bots but headers blocking them. This mismatch renders the file useless for those specific crawlers. Check your server logs and headers alongside your robots.txt file. Consistency between the two layers is required for successful indexing.

• HTTP headers take precedence over robots.txt rules.
• X-Content-Signal: noai blocks AI indexing regardless of file settings.
• Align file permissions with server header configurations.

Validating the File

Testing your robots.txt file is essential before publishing changes. Use online validators or command-line tools to fetch the file and check for syntax errors. Ensure the file returns a 200 status code and contains plain text. Any HTML error page served instead of the text file breaks the protocol.

Verify that the sitemap reference points to a valid XML file. A broken sitemap link reduces the efficiency of crawler discovery. Check that the file ends with a newline character, as some parsers fail otherwise. Review the last modified date to ensure recent updates are recognized.

Monitor your server logs for crawler activity after making changes. Look for requests from GPTBot, ClaudeBot, and PerplexityBot to confirm they are fetching pages. If you see 403 errors, check your firewall or security plugins. These tools sometimes block automated agents even when robots.txt permits them.

• Validate syntax with online robots.txt testers.
• Ensure the file returns plain text, not HTML.
• Check server logs for successful bot requests.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.