Your website could have the best content in your industry and a strong backlink profile — but if Google cannot crawl and index your pages, nobody will ever find them through search. Understanding how Google actually discovers and processes your content is the foundation of all technical SEO.

Quick Answer: Google crawls websites using a bot called Googlebot that follows links to discover pages. It then renders those pages (executing JavaScript to see the full content), extracts the information, and stores it in the search index. Only indexed pages can appear in search results. The process can take hours for established sites and weeks for new ones.

What is Googlebot and how does it find your pages?

Googlebot is Google's automated web crawler — a computer program that systematically browses the internet to discover and read web pages. It runs continuously, visiting billions of URLs every day across the entire web.

Googlebot finds your pages through three main routes:

  • Links from already-crawled pages — this is how most pages are discovered. If a crawled page links to yours, Googlebot follows that link.
  • XML sitemaps you submit — submitting a sitemap in Google Search Console tells Google directly which pages you want crawled.
  • Manual URL submission — you can request indexing for individual URLs using the URL Inspection tool in Search Console.

Is crawling the same as indexing?

No — these are two completely separate steps. Crawling means Googlebot visited and read your page. Indexing means Google decided to store that page in its search index so it can appear in results. A page can be crawled but not indexed if Google considers the content too thin, too similar to another page, or low quality. In Search Console under Coverage, you can see exactly which pages are indexed and why others are excluded.

What happens after Googlebot crawls a page?

After fetching a page, Google does not immediately index it. It goes through a rendering process first — and this is where many sites lose traffic without knowing why.

The two-wave rendering process works like this:

  • Wave 1 — HTML fetch: Googlebot downloads the raw HTML of your page immediately. Content in that initial HTML response is processed quickly.
  • Wave 2 — JavaScript rendering: Google queues the page for JavaScript execution to see the fully rendered version. This queue is shared across all websites Google processes, so lower-authority sites may wait days or even weeks before their pages are rendered.

This means any content that only appears after JavaScript runs — menus, product descriptions, article bodies loaded via React or Vue — may not be indexed promptly. The safest approach is ensuring your critical content is present in the initial HTML response.

How do you check if Google is rendering your pages correctly?

Use the URL Inspection tool in Google Search Console. Enter your page URL, click "Test Live URL", then go to "More Info" and check the rendered HTML tab. Compare what Google sees against what you see in your browser. Any missing sections indicate Wave 2 JavaScript dependencies that may cause delayed or incomplete indexing.

How does Google decide which pages to crawl first?

Google uses crawl budget — a limit on how many pages it will crawl on your site in a given period — and it allocates this budget based on signals from your site. Pages it crawls more frequently tend to be ones that:

  • Are linked from your homepage or other high-authority pages on your site
  • Have strong external backlinks pointing to them
  • Load quickly and return consistent 200 status responses
  • Change frequently (news articles, product inventory, prices)
  • Are listed in your XML sitemap

Pages that waste crawl budget — and reduce the frequency with which Google crawls your important pages — include broken links, redirect chains, low-quality thin pages, and URL parameter variants. Run our broken link checker monthly to keep your crawl budget clean.

What stops a page from getting indexed after it has been crawled?

Several things can prevent indexing even after Googlebot successfully visits a page. These are some of the most common silent ranking killers:

  • Noindex meta tag — a <meta name="robots" content="noindex"> tag explicitly tells Google not to index the page. One wrong template-level noindex can de-index hundreds of pages at once.
  • Thin or duplicate content — if Google judges the content too short, too generic, or already available on another URL on your site, it may crawl but choose not to index the page.
  • Soft 404s — pages that return a 200 status code but display "no results found" or similar messages confuse Googlebot and often result in exclusion from the index.
  • Canonicalisation issues — if your canonical tag points to a different URL, Google indexes that other URL rather than the page itself.
  • Server instability — if Googlebot encounters repeated timeouts or 5xx errors when visiting a page, it eventually stops trying to index it.

How do you get new pages indexed faster?

The fastest route to indexing is using Google Search Console's URL Inspection tool. Enter your new page URL and click "Request Indexing". For established, trusted sites this typically triggers indexing within 24–48 hours.

To speed up indexing systematically:

  1. Submit an updated XML sitemap to Search Console every time you publish new content
  2. Add internal links to the new page from two or three of your most frequently crawled existing pages — Googlebot follows those links
  3. Ensure your site loads quickly on every new page — test with our speed tool
  4. Fix any existing server errors or crawl issues in Search Console before launching new content
  5. Share new pages on social platforms — Google crawls social media and these mentions can accelerate discovery

Why is my page indexed but still not ranking?

Indexing simply means Google knows your page exists — it does not guarantee any ranking position. To rank, your page also needs to match the search intent of the query it targets, demonstrate sufficient E-E-A-T, have relevant backlinks, and outperform competing pages on quality signals. Indexing is the prerequisite. Ranking requires everything on top of it.

How do you check which of your pages are indexed?

You have three ways to check indexing status:

  • Search Console Coverage report — shows all indexed pages and the specific reason for every excluded page across your whole site. This is the most reliable source.
  • URL Inspection tool — checks the current indexed status of any specific URL instantly.
  • Site: search operator — search site:yourdomain.com in Google to see a sample of indexed pages. Note this only shows a sample, not the full count.

Summary

Google's process has three stages your content must pass through: discovery via links and sitemaps, rendering via its two-wave JavaScript processing system, and indexing when the content meets quality thresholds. To keep your site moving efficiently through all three:

  • Submit sitemaps and use URL Inspection to request indexing for new pages immediately
  • Keep your site technically clean — no broken links, fast loading, no server errors
  • Ensure critical content is in the initial HTML response, not dependent on JavaScript rendering
  • Avoid thin or duplicate content that passes the crawl but fails the index quality threshold

Continue reading: What is the difference between on-page and off-page SEO?