aiseocheck.hu / Articles / Guide

Testing Real Bot Access: Why robots.txt Text Is Not Enough

Reading rules is not the same as verifying access; manual testing reveals silent blocks that automated tools miss.

By Zoltán Varga2026-11-054 min read

Reading Rules Versus Testing Access

Many automated audit tools stop their analysis after reading the text of your robots.txt file. They check if the file exists and if it contains rules allowing or blocking specific user agents. This approach assumes that the text on the server matches the actual behavior of the web server. It fails to detect when server configurations override these text instructions with hard blocks.

A critical audit case revealed this gap clearly. An automated tool scored a website 70 out of 100 and labeled it as being in the top 6% of performance. The tool read the robots.txt file, saw permission granted, and marked the site as accessible. However, a manual check sent a request to the same URL pretending to be GPTBot. The server responded with a 403 Forbidden status code, denying access entirely.

This discrepancy creates a false sense of security for site owners. The automated report suggests the site is ready for AI indexing, while the reality is that major models cannot read the content. The tool analyzed the rules but did not execute the action. You must distinguish between the permission slip written in a text file and the actual gatekeeping performed by the server software.

Requesting as a Bot User-Agent

You can simulate a crawler by modifying the User-Agent header in your HTTP request. Standard browsers send a string identifying themselves as Chrome or Firefox. You change this string to match known AI bot identifiers. Common targets include GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. The server reads this header to decide which rules to apply.

Use command-line tools or scriptable environments to send these requests. A simple curl command allows you to specify the user agent and capture the full response. You do not need to build a full crawler that follows links. You only need to hit the specific URLs you want to verify. This targeted approach isolates the access issue from other crawling behaviors.

Do not rely on third-party tools that claim to test this for you unless they explicitly state they perform live HTTP requests. Some tools only parse the static text of the robots.txt file again. Verify the methodology of any tool you use. If the tool does not show a status code from a live connection, it cannot detect a 403 block. Manual testing remains the most reliable method for validation.

Status Codes to Look For

When you send a request as a bot, the server returns a numeric status code. A 200 code means the request succeeded and the content is available. This is the desired outcome for AI visibility. A 403 code means the server understood the request but refuses to authorize it. This is the specific error that blocks AI crawlers despite open robots.txt rules.

Other codes indicate different problems. A 404 code means the page does not exist at that address. A 500 code indicates a server-side error that prevents the page from loading. These errors also stop AI indexing but stem from broken infrastructure rather than intentional blocking. You must record the exact code for every URL you test to diagnose the root cause accurately.

Focus your testing on high-value pages like your homepage, service descriptions, and key blog posts. A single 403 on a main page can prevent the entire site from being indexed by some models. Check multiple bot user-agents because servers sometimes block specific bots while allowing others. A site might allow Googlebot but reject GPTBot, creating a partial visibility problem.

Response Headers That Matter

Beyond the status code, inspect the response headers sent by the server. These headers can contradict the robots.txt file. A robots.txt might say "Allow," but an HTTP header like X-Robots-Tag or X-Content-Signal might say "noai" or "notraining." Sophisticated crawlers prioritize these headers over the text file rules.

In one documented case, a site had a robots.txt file explicitly allowing all AI traffic. The file contained signals indicating search and training were permitted. However, the HTTP response headers on every page included directives blocking AI input and training. This conflict meant that advanced bots respecting the headers would ignore the robots.txt permission and refuse to index the content.

Check for headers that control caching, content type, and security policies. A mismatch in content types can also cause parsing failures. Ensure your server sends the correct content type for text files like llms.txt. If a text file returns an HTML content type, parsers may fail to read it correctly. Consistency between your configuration files and server headers is essential for reliable access.

Logging Results for Comparison

Record your findings in a structured format to compare against automated reports. Create a table listing the URL, the user agent used, the status code received, and the relevant headers. This log serves as your ground truth. It allows you to prove to stakeholders that the automated tool missed a critical access failure.

Compare your manual log with the output of any audit software you use. Look for discrepancies where the tool reports success but your log shows a 403 or 500 error. These gaps highlight the limitations of the software. They also identify the specific areas where your manual intervention adds value. This comparison justifies the need for deeper technical audits.

Update your logs regularly as server configurations change. A fix applied today might break tomorrow due to a software update or a new security rule. Periodic re-testing ensures that your site remains accessible to AI crawlers over time. Treat this as an ongoing maintenance task rather than a one-time check. Consistent monitoring protects your visibility in evolving search ecosystems.

AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.