Changed robots.txt? Compare the Version Google Retrieved

First, compare the two files

1. Record the complete URL of the affected page.
2. Keep a copy of the robots.txt response for its protocol, host, and port.
3. Open the robots.txt report in Google Search Console.
4. Select the file for that same address.
5. Record its Checked on time.
6. Compare the displayed file with your response copy.

The file available to you now and the file Google previously retrieved can differ. A successful response from a reader does not establish which version Google used.

The Search Console report guide explains the displayed file and its retrieval history. These records give you a specific version to compare before another rule change.

Three records with different purposes

Record	Question it helps answer	Limit
File in your editor	What change do you intend?	The website can serve other text
HTTP response from the website	What did this request receive?	Another client can receive another response
File displayed in Search Console	What version does Google's report show?	It does not prove that Google retrieved the affected page again

The address also matters. For example, https://example.com/robots.txt and https://www.example.com/robots.txt belong to different hosts. Google's robots.txt specification defines the scope by protocol, host, and port.

The Enfect robots.txt reader retrieves a root robots.txt file through Enfect's server. Its response is useful evidence for that request. It does not show Google's internal cache or calculate permission for each page URL.

Why a change can appear late

A cache is a stored copy that a system can use again. Google generally keeps robots.txt contents for up to 24 hours. It can keep them longer when it cannot retrieve a new version. Its documentation also describes cache duration changes based on the Cache-Control header.

Thus, 24 hours is not a deadline for recovery. A timeout or server error needs investigation. A repeated rule change does not establish that Google received any of the new versions. See Google's cache guidance.

RFC 9309, section 2.4 permits a stored copy. Its recommendation against reuse beyond 24 hours has an exception when the file is unreachable. The standard does not promise a page crawl at that time.

One Reddit discussion describes a validator that allowed access while Search Console reported a block. The author also reported an earlier file that blocked all paths.

A WordPress discussion describes repeated edits and uncertainty about the stored version. These reports show the need for a version record. They do not prove that a cache caused either problem, or that either proposed fix is correct.

Four local responses: what a hash can show

On November 2, 2026, we used Python 3.12.6 on Windows to retrieve four responses from one local HTTP server. All four requests used the same URL and returned HTTP 200.

Response	Body change	SHA-256 result
A	Initial Disallow: /news/ rule	First hash
B	Rule changed to Disallow: /private/	Different from A
B repeated	No change	Same as B
C	One comment added before B's rules	Different from B

The script kept response A in memory. Later requests did not change that stored copy. A current response and an earlier copy therefore remained available at the same time.

The comment in C changed the bytes and the hash. The directive lines remained identical to B. A different hash establishes different bytes, not necessarily different crawl rules.

This experiment used four synthetic responses. It did not reproduce Google's cache, a CDN, or Search Console. It measured no waiting period, search effect, or recovery rate. The results demonstrate why a version record needs both the file text and the time of retrieval.

Procedure: record the difference

1. Record the time and timezone of your website response.
2. Record the response status and the final URL after any redirects.
3. Keep the response body without edits.
4. Record the file address shown in Search Console.
5. Open Versions for earlier file records, if necessary.
6. Compare the relevant directive lines in the two copies.
7. Record whether the difference changes the intended rule.

Search Console retains version history for 30 days. It adds a history entry when the retrieved file or retrieval result changes. Therefore, the history is not a list of every request. See the report documentation.

If the report shows the intended version, an old file no longer explains that specific difference. The Googlebot group guide explains the next rule check. Server records can also help establish whether a request reached the website.

When another retrieval is appropriate

Google provides Request a recrawl for a corrected error or a critical robots.txt change. The file's menu in the report contains this control. It does not guarantee an immediate crawl of the pages that the new rules allow. Google's recrawl instructions.

For an unresolved retrieval error, the official response-handling table gives the applicable behavior. A server error and a missing file are different conditions.

A useful completion record contains the intended file, the response you received, and the version Google reports. Evidence of a later page crawl is a separate record. This distinction prevents a successful file check from becoming an unsupported claim of search recovery.