How to check robots.txt for search and AI

Start with the exact address

Open robots.txt on the host serving the relevant page. The main domain and a subdomain can have different files and settings. Record the full content URL, User-agent group and applicable rule. A sitewide Disallow: / may have been carried over from a test environment, while a block on an individual internal path may be intentional.

Google’s guidance explains that robots.txt is not a reliable way to hide sensitive information. It can prevent content retrieval, but the address itself may still appear in search under certain conditions. Protect private documents with authentication and manage indexing using appropriate methods.

Do not fix one block with another

If you want to remove a page from search using noindex, the crawler must be able to read that signal. Blocking retrieval at the same time may prevent the directive from being processed. Do not remove every restriction for an audit: first identify the purpose of the particular page.

Test the active configuration after publication. A locally saved file does not change the CDN’s response. If a browser shows XML, HTML or an error page instead of a plain rules file, check routing. If behavior varies by request, server logs and security rules will be needed.

An example of a safe review

Imagine a shop with public product pages and an order history that requires sign-in. The aim is to let a search crawler read product pages without exposing orders. Check the list of required public URLs separately from the account area. After the edit, compare the previous and new robots.txt and repeat a product request. This illustrates the sequence of actions; it is not a universal ready-made file to copy.

How to use AURA GEO

In AURA, you can retrieve an audit and the underlying rule information, then give the agent a specific change. If the infrastructure restricts access, changing robots.txt alone will not be enough. Auri helps preserve context, and the agent applies a fix only where it has been given permission.

The completion criterion is simple: the public target page is accessible under the agreed policy, and private areas have not become public. The resulting permission to crawl is a technical outcome; indexing and the AI answer are checked next.