SEO on a Large Site: Stop Watching Where One Keyword Ranks

These examples come from my work on a UK classifieds site. I've left out the site's name and the absolute numbers, and kept only ratios and orders of magnitude.

When I started doing SEO for a classifieds site with tens of millions of URLs, my first instinct was the same as most people's: pull the rank tracker, see which keywords had dropped, and check them one by one.

Two weeks in, I could see it wasn't going to work. Thousands of keywords had dropped, and millions of pages sat behind them, but those millions of pages came from about a dozen templates: search results, listing detail, category × city landing pages, and seller profiles. Every keyword I was chasing led back to the same thing. One type of page was broken as a whole.

This post is about that shift, and what it means for the development team.

1. On a large site, the unit of SEO is the page type

On a small site, the unit is the page. You write a post, tweak the title, add a few internal links, and the ranking moves.

A large site doesn't work like that. When a developer changes the search results template once, it touches hundreds of thousands of URLs. Going the other way, if SEO wants a keyword to recover, the template is the only thing it can change. Nobody hand-edits the H1 on a single listing.

So I now work in the opposite order:

1. Split traffic and indexing data by page type first
2. Find which type of page is falling, bloating, or not getting indexed
3. Then go back to specific keywords to confirm

The rank tracker is still useful, but it's an alarm, not a diagnostic tool.

2. What "page architecture" actually means

"Architecture" is too vague a word. A developer hears it and still doesn't know what to change. I split it into four parts, and each one maps to real code.

URL space: which URLs can be generated and indexed

This is the most expensive problem I've seen.

The site's internal search had this URL pattern: /<category path>/uk/<city>/srpsearch+<query>. Testing showed the category path in the middle did nothing at all. Any category that really existed on the site, from any line of business, returned the same search results page, and every one of those URLs set its canonical to itself.

So this set of URLs had 100% identical content, and every one of them could be indexed on its own:

/uk/hull/srpsearch+double+bed
/for-sale/uk/hull/srpsearch+double+bed
/for-sale/home-garden/uk/hull/srpsearch+double+bed
/business-services/.../massage-services/uk/hull/srpsearch+double+bed

The last one is the path for the massage services category. It returns double beds.

The URL space was "every category × every city × any query", with no upper bound. In Search Console, pages marked "Crawled – currently not indexed" outnumbered indexed pages by half again. Google crawled them and didn't want them.

The business impact was more direct. Clicks on local services halved year over year, while the number of URLs getting impressions went up. The more variants there were, the worse each one ranked, and head terms slid from the top 5 into the teens and twenties.

No single keyword would have exposed this.

Index control: canonicals, status codes, sitemaps

URL space decides what can be generated. Index control decides what you tell Google you want.

On the same srpsearch pages, the canonical didn't consolidate anything. It also rewrote the query on its own, turning massage into massage+london, but never merged across path prefixes. Two URLs with different prefixes each pointed to a different target, so the duplication stayed exactly as it was.

Expired content is the other common trap. Listings on a classifieds site come down every day, and more than half of Googlebot's requests to listing pages hit listings that had already been removed. That crawl budget could have gone to new listings. Whether a removed listing returns a 404, a 410, or redirects to its category looks like a small decision, but it decides where the crawler spends its time.

Rendered output: what's in the server-rendered HTML

We once audited heading tags on listing pages and shop pages. We sampled a little over a hundred pages and compared the server-rendered HTML with the DOM after the browser had rendered it.

The result was useful for the developers. The two matched, which meant the problem was in the server components themselves. No debate about client-side rendering, no prerendering; just fix the server components. The scope of the work became clear straight away.

An SEO requirement handed to developers should be this precise: which component to change, and what should appear in the server HTML afterwards.

Crawl budget: where the crawler spends its time

On a large site you have to look at crawler logs, and you have to clean them first.

The first time we pulled Googlebot logs, close to 40% of all requests with "Google" in the user agent came from AdsBot, the crawler that checks ad quality. It has nothing to do with organic crawling. Leave it in, and every conclusion about crawl budget is skewed.

Once the logs are clean, they answer concrete questions: how often each type of page is crawled, whether the crawler is hitting new pages or old ones, and how many requests are wasted on 404s and duplicates. Those numbers turn directly into development priorities.

3. The biggest favour SEO can do developers is less work

Up to this point it sounds like SEO is handing developers more work again. I think the most valuable thing SEO can do for developers is tell them which pages shouldn't exist.

Cut the URL space in half and it affects more than rankings:

• Fewer crawler requests, so less load on the servers
• A higher cache hit rate
• Fewer page types to maintain, monitor, and test

Developers want these things anyway. SEO doesn't have to be the team that only files requests. It can also help the development team cut complexity they don't need.

4. Write requirements as acceptance criteria, not suggestions

My early requirement docs were full of "consider optimising" and "we suggest looking at". Developers read them and had no idea when the work counted as done.

Now I try to write SEO requirements as invariants you can test:

• URLs of this type must have a canonical pointing to the address generated by rule X
• Listings removed more than N days ago return 410
• Each page's server HTML has exactly one H1, and its content is field Y
• The sitemap contains only URLs that return 200 and have a self-referencing canonical

Each of these can become an automated check in the pre-release regression tests. What SEO fears most isn't developers refusing to do the work. It's the work getting done and then quietly undone in some later redesign. Turn it into a test and that problem is half solved.

5. Is SEO an advisor or a product owner?

People ask me this a lot. My answer: on a large site, SEO should be a product owner, but only for some things.

As a pure advisor, you can't get things done on a large site. An advisor produces recommendations, and someone else makes the decision. Product managers are measured on conversion and retention, so SEO recommendations are forever "important but not urgent". Worse, nobody owns the outcome. When organic traffic drops, product says they did what SEO recommended, and SEO says it only gave advice. The srpsearch problem above had been there for a long time. It wasn't that nobody could see it. Nobody owned it.

But SEO shouldn't own everything either. Listing pages exist first for buyers and sellers. If SEO decides their features and interactions, it's overstepping, and it doesn't have the judgement for it.

This is how I split it now:

• What SEO decides: the list of page types, URL rules, canonicals, status codes, sitemaps, robots, and pages that exist purely for search, such as category × city landing pages. Here SEO is the product manager, with its own roadmap, and the organic traffic counts against SEO.
• Where SEO sets constraints: core pages such as listings, seller pages, and the homepage. SEO specifies what the server must output; product decides how the features work.
• Where SEO guards the gate: turning all of those constraints into automated checks.

To work out where you stand, ask three questions:

1. To change a URL rule, do you have to convince someone, or is it your call?
2. Do you have developers whose sprints are planned for SEO, or do you have to squeeze into product's queue?
3. If a release breaks the canonicals, does anything automatically stop it?

In my case, URL rules are my call, and I have my own developers. On the third, nothing stops it yet.

Once I had the authority and the developers, the hard part changed. It used to be getting things done. Now it's keeping them done. My team fixes the canonicals, and another team's next release may break them. Often I only find out weeks later, when the indexing numbers in Search Console look wrong, and by then traffic has already dropped. On a large site, one regression can cost more than a new feature earns.

So the third one is what I'm working on now, starting with the smallest version: pick a few dozen representative URLs for each page type, check status code, canonical, robots, H1 and structured data every day, compare with the day before, and alert on any change. First get to the point where a problem shows up the next day. Then wire those checks into the release process, so a release that fails them doesn't ship.

Authority lets you get things done. Guarding the gate stops what you've done from being broken.

6. AI citations: a hypothesis I haven't finished testing

Finally, the thing everyone is asking about now: citations in AI search.

My hypothesis is that page types may matter more for AI citations than for traditional rankings. An AI summary has to pull facts out of a page. A page type with a stable structure, complete fields and structured data should be easier to extract from than a well-written page with a loose format. Seller profiles are a good example: service area, location, reviews and opening hours are fixed fields, which makes them natural material to cite.

But this is a hypothesis. Most of the published research on "AI citation factors" comes from tool vendors, the samples and methods aren't transparent, and cause and effect haven't been untangled. I don't have enough data of my own to draw a conclusion yet. When I have observations I trust, I'll write them up separately.

7. A side lesson: double-check third-party tool data

While writing up these cases, I made a mistake of my own.

For a while, Ahrefs showed a /Services/Health path ranking #2 for "massage birmingham", and that URL returned a 404 when I tested it. My conclusion at the time: "Google is ranking a 404 page, and every click is lost."

Later I checked Search Console, and Google had never heard of the URL. What actually happened is that Google shows breadcrumbs instead of the URL in search results, and Ahrefs parsed the breadcrumb "Services › Health & Beauty" into an address that doesn't exist. The page ranking #2 was an ordinary category page.

So any conclusion about a specific URL, I now verify in Search Console or the live search results. A large site has a lot of data, and one tool's parsing error can easily be blown up into a "major finding".

Closing

On a large site, keyword rankings are the result. Page types are where you can actually do something.

If I could leave the development team only one thing, it would be a list of page types: how many URLs each type has, whether it should be indexed, what its canonical rule is, and which fields the server must output. If SEO and development both work from that table, a lot of arguments never happen.