Soft 404 on a Page That Opens Fine: What Google Sees That You Don't

Somebody asked this in a thread that pulled 54 replies, and it is the exact question:

"I know a 404 means the page isn't there anymore, but what exactly is a Soft 404? Why does Google sometimes show a Soft 404 even when the page opens?"

The page opens. Google says it is missing. Both of those are true at the same time, and figuring out how is the whole job.

The short version

A real 404 returns the HTTP status code 404. That tells a crawler the resource is gone, and the crawler believes it, because there is nothing else to believe.

A soft 404 is a page that returns 200 OK while having nothing on it, or while being an error page wearing a normal page's status code. Google looks at what came back, decides the page is effectively missing, and files it as a soft 404 regardless of what the header claimed.

One person in that thread put it as a site "showing a common reaction page in response to calling an URL it cannot find that doesn't clearly indicate 'this content isn't here'. Typically a redirect to the homepage or similar."

That is the textbook case: every dead URL redirects to the homepage instead of returning 404. A human clicks the link, lands somewhere real, and shrugs. A crawler asks for a specific resource and gets a 200 back for something that is not it. The reader gets a page. The crawler gets a lie.

But the textbook case is not why most people end up searching this. Most people are looking at a page that has real content, that loads for them, and that Search Console will not index.

Your browser is not the crawler

Here is the assumption that keeps this bug alive: you opened the URL, it worked, so the page is fine.

Your browser is signed in. It has cookies from previous visits. It is coming from your IP, in your country, over a warm connection, with a user agent your CDN has seen a thousand times. Googlebot has none of that. It arrives cold, from a Google IP range, announcing itself as Googlebot, with no session at all.

If anything in front of your app treats those two visitors differently, you and the crawler are looking at different websites. That is not exotic. A CDN, a bot filter, a geo redirect, and a JavaScript bundle that needs one API call are all normal things to have, and all four can produce a page that is fine for you and empty for Google.

What actually causes this

These are cases people posted about their own sites, and got debugged in the replies. Real sites, diagnosed in public by whoever happened to be reading.

The crawler gets an error page instead of your site. One site owner had a homepage stuck at soft 404 while every other page indexed fine. He had tested it with other tools and got no errors. A Google Search Advocate replied in the thread: "FWIW I can't load your pages. I get a CF timeout page instead." A Cloudflare timeout screen returns a 200 with no real content on it. The owner saw his homepage. The crawler saw a timeout page and a 200 status. Google classified what it got, which was the timeout page.

The data never arrives on a client-side rendered page. Same site, second finding. Another person answering in the thread loaded the page with a Googlebot user agent and found that calls to two of the site's own subdomains came back as CORS errors. Those calls never completed, so the parts of the page that depended on them never rendered. Nothing about that is visible in a normal browser session, because in a normal browser session the calls succeed.

robots.txt blocks the API the page needs. This is the best one, and the one worth reading twice.

Someone posted a page showing soft 404 in Search Console with an HTTP 499 in the live test, and the note that the page works for users. The diagnosis in the replies: the site is client-side rendered, and it has this in robots.txt.

Disallow: /api/

The HTML shell returns 200. Then the JavaScript tries to fetch the content, and the endpoints it needs are the ones robots.txt just told Googlebot to stay away from. So the data never loads for Google. What Google renders is a page with a shell and no content, and it calls that a soft 404. The suggested fix was to delete that one line.

Blocking /api/ feels tidy. It is the kind of thing you add because API routes are not pages and there is no reason to crawl them. But if your pages are assembled in the browser, those endpoints are not extra, they are the content. Blocking them is blocking the page. This is the same misunderstanding behind a page that stays indexed after you block it in robots.txt: robots.txt governs fetching, and fetching is how a page becomes real to Google.

The page is genuinely empty to a crawler. From the same thread, all the content available to Googlebot on one page was this.

<body>
  <div id="root"></div>
</body>

That is what a crawler gets before your framework runs. If nothing fills that div in a way Google can see, there is no page.

The page is thin. Sometimes it really is just that. The bluntest reply anyone got on a soft 404 post: "The page is super-thin. Add some content." Worth keeping on the list, but keep it last, because it is the answer people reach for first and it is usually not the answer.

The order to check in

The threads that got solved followed roughly this sequence. It is worth copying, because each step rules out a whole class of cause.

1. Fetch the URL as Googlebot, not as yourself. In Chrome, open DevTools, go to Network conditions, and set the user agent to Googlebot Smartphone. That is the advice the Google Search Advocate gave on a different soft 404 post, and it is the fastest way to see the version of your site that matters. Watch the network tab while it loads. Anything red is a candidate.

2. Check the status code, not the page. A 200 on an error page is the entire bug, and you cannot see a status code by looking at a rendered page. Run the URL through the redirect detector to see every hop and the HTTP status each one returned. If a URL you think is dead comes back 301 to the homepage and then 200, that is your soft 404 in two lines. One caveat, since it matters here: the tool requests the URL with a normal desktop browser user agent, so it tells you the status your server returns, not what it returns to Googlebot specifically. For that part, step 1 is still the test.

3. If the page renders in the browser, check what it fetches to do so. Open your robots.txt and read the Disallow lines against the requests the page makes. The robots.txt fetcher pulls the file from any domain's root and shows you the raw text, which is enough to spot a Disallow: /api/ sitting on top of the endpoints your pages depend on. Cross-check it against the requests you watched in step 1.

4. Then consider thin content. By this point you have ruled out the causes that make a full page look empty. If the page really does have three lines of text on it, that is a content problem and no amount of header debugging will fix it.

A soft 404 is a report on what came back over the wire when Googlebot asked. Find the difference between that and what came back to you, and the fix is usually one line.