Technical SEO for a site with 10,000 products is fundamentally different from technical SEO for a 50-page site. The scale introduces problems that do not exist at small scale: crawl budget becomes a genuine constraint, faceted navigation generates millions of URL variants, canonical tag consistency requires systematic automation, and a single template issue can affect thousands of pages simultaneously.

Crawl Budget at Enterprise Scale

As we covered in our guide to crawl budget, Google allocates a limited number of crawl slots to your domain. For large e-commerce sites, the gap between your total URL count and Google's crawl allocation is often enormous — Google may crawl only 10–20% of your total URLs in any given period.

This makes crawl prioritisation critical. Every URL type on a large e-commerce site needs an explicit crawl strategy:

Priority 1 (Maximum crawl allocation): New product pages, recently updated product pages, high-selling product pages, and category pages with recent price or inventory changes.

Priority 2 (Regular crawl): Static category pages, informational content, brand pages, and stable product pages with no recent changes.

Priority 3 (Minimal crawl): Archived products, out-of-stock products (if kept on site), and historical content with no update intent.

Block entirely: URL parameter variants, filtered navigation URLs, internal search results, and checkout flow pages.

Implement crawl prioritisation through XML sitemap segmentation (separate sitemaps for different content types with different update frequencies), robots.txt blocking for low-priority URL patterns, and last-modified timestamps in your sitemap to signal which pages have changed.

Faceted Navigation: The Largest Source of URL Bloat

A product catalogue with 10,000 products and 50 filter attributes (colour, size, brand, price range, material, etc.) can theoretically generate billions of unique filter combination URLs. Each URL is a potential crawl budget drain and duplicate content source.

The enterprise-scale faceted navigation strategy in 2026:

Canonical all filtered URLs to the base category URL. A URL like /shoes?colour=red&size=10&brand=nike should have a canonical tag pointing to /shoes/. This consolidates all filter variants' equity onto the main category page while keeping filter functionality for users.

Allow crawling of high-value filtered pages only. Some filter combinations generate significant search volume: "Nike trainers in red" has real demand. Identify these high-value combinations using Search Console query data and allow Googlebot to crawl and index these specific combinations while blocking generic filter combinations.

Block generic parameter patterns in robots.txt. As covered in our guide to advanced robots.txt: Disallow: /*?sort=, Disallow: /*?page= prevents Googlebot from crawling pure sorting and pagination variants that provide no unique indexable content.

Out-of-Stock Product Handling at Scale

Large e-commerce sites constantly have products going in and out of stock. The wrong handling of out-of-stock pages is one of the most common sources of crawl budget waste and missed link equity.

Temporarily out of stock (returning within 4–8 weeks): Keep the page live with clear out-of-stock messaging and a back-in-stock notification option. Do not noindex or redirect — the page maintains its backlinks and rankings and is ready to convert when stock returns.

Permanently discontinued (but with close replacement): 301 redirect to the most relevant replacement product. This passes link equity to the most relevant live page.

Permanently discontinued (no replacement): 410 Gone status code if the product category is discontinued, or redirect to the parent category. A 410 tells Google the resource is permanently gone — it de-indexes faster than a 404.

Template-Level Optimisation

For sites with 10,000+ products, page-level optimisation is impossible — you need template-level optimisation that improves all pages sharing a template simultaneously. Identify the five to ten core templates (product page, category page, brand page, search results, etc.) and ensure each template is technically excellent.

Use our broken link checker on template samples — scan 100 product pages, identify any systematic issues, and fix at the template level rather than page-by-page. A single template fix that improves 8,000 product pages provides dramatically better ROI than fixing 8,000 individual pages.

Summary

Enterprise e-commerce technical SEO requires explicit crawl budget prioritisation through sitemap segmentation, faceted navigation control through canonical tags and parameter blocking in robots.txt, systematic out-of-stock page handling protocols, and template-level optimisation that improves thousands of pages simultaneously. Monitor with our site scanner on regular samples and review crawl data in Search Console monthly to verify your crawl budget allocation matches your priorities.

Continue reading: SEO for Comparison and Review Sites: Surviving Google's Product Review Updates