Googlebot and Wildcard Rules: Find the Correct Group

Googlebot can use a different group from other crawlers. The wildcard group does not add its rules to a separate Googlebot group. A Disallow rule can be in the file but outside the group that Googlebot uses.

Procedure: find the applicable group

1. Open the robots.txt file that your website serves.
2. Find each group with a Googlebot user-agent value.
3. Identify the group that applies to the crawler.
4. Do a check of the Disallow rules in that group.
5. Compare those rules with the rules in the wildcard group.

Google's documentation gives the group selection rules. Google uses the most specific applicable user-agent group. Group order does not change that selection.

The Enfect robots.txt reader retrieves the root robots.txt file. It does not calculate crawl permission for each URL. The robots.txt introduction gives the basic steps to retrieve the file.

Example: a rule in the wrong group

This example has separate groups:

User-agent: *
Disallow: /private/
 
User-agent: Googlebot
Allow: /public/

Googlebot uses its own group. That group has no Disallow rule for /private/report. The Allow rule for /public/ does not block all other paths.

ExampleBot uses the wildcard group. Its Disallow rule blocks /private/report. Thus, the same file gives different crawl permissions to the two crawlers.

RFC 9309, sections 2.2.1 and 2.2.2 give the applicable group and path rules. A crawler uses the wildcard group when no specific group applies. Without an applicable path restriction, the crawler has permission to crawl the path.

Procedure: make the required rule explicit

First, do a review of all crawler groups and path exceptions. Keep the original file for rollback.

If Googlebot needs separate rules, copy the required Disallow rule into its group:

User-agent: *
Disallow: /private/
 
User-agent: Googlebot
Disallow: /private/
Allow: /public/

If the groups need the same rules, put their user-agent lines in one group:

User-agent: *
User-agent: Googlebot
Disallow: /private/
Allow: /public/

These examples are alternatives. Do not add either example to a production file without a review of the complete file.

Google combines applicable groups for the same specific user agent. The first applicable group does not cancel the other applicable groups. Google's group examples describe this behavior.

Results from 12 offline tests

On October 30, 2026, we did 12 tests with Python 3.12.6 and its standard-library urllib.robotparser. The experiment used only text from the examples. It did not request a public website.

All 12 results agreed with the expected results. The saved output includes the other group and path tests.

These results apply only to the specified examples. They do not show how Google's production system works. They do not show a change in the Google Search index.

The experiment does not include duplicate-group merging, wildcard path patterns, longest-match conflicts, or Unicode. Do not use it as a complete Google-compatible robots.txt validator.

Procedure: compare the complete path

A path rule applies from the start of the path. Thus, /search does not match /en-gb/search. A final slash also changes the path: /private/ and /private are different inputs. RFC 9309, section 2.2.2 gives the path rules.

1. Record the complete URL and the crawler name.
2. Keep a copy of the served robots.txt file.
3. Record the file address and the date of the check.
4. Write the expected crawl permission for each URL.
5. Include a restricted URL, an unrestricted URL, and the applicable language and slash variants.
6. After an authorized change, retrieve the file again.
7. Repeat the same tests and compare the results.

For a multilingual website, identify its actual search routes before you change the rules. A narrow path error does not justify a broad restriction.

Community reports show possible failure conditions

A TechSEO discussion describes language-prefixed search paths and separate Googlebot groups. A WordPress discussion describes a reported change to empty Googlebot restrictions.

These reports give possible failure conditions for further checks. They do not prove the cause of a problem on your website. A reported improvement does not prove which change caused it.

A rule calculation and a response to a real crawler give different information. When you compare them, record the URL, file version, and time of each check.

Crawl permission and index removal are different tasks

Robots.txt controls which paths a crawler can request. It does not give a complete instruction to remove a URL from Google's search index. The separate crawl and indexing guide explains that task.

A change to the applicable group does not prove that Google removed a search result.