Your robots.txt file is a short text document that lives at the root of your website. It is one of the first files Googlebot reads when it visits your domain. Get it right and it helps you direct Google's attention to your most important content. Get it catastrophically wrong — as happens with a single accidental line — and your entire website can disappear from Google Search within days.

Quick Answer: A robots.txt file tells search engine crawlers which parts of your website they are allowed to access. It uses simple Disallow and Allow directives to control access by specific bots. Robots.txt does not prevent pages from being indexed if they are linked to from other sites — it only prevents crawling. To prevent indexing, you need a noindex meta tag instead.

How does robots.txt actually work?

When Googlebot (or any other crawler) visits your domain, it checks yourdomain.com/robots.txt before crawling anything else. The file contains instructions in a simple format specifying which crawlers the rules apply to and which URL paths they can or cannot access.

A basic robots.txt file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

This tells all crawlers (* means all bots): do not access /admin/ or /checkout/, but everything else is accessible. The sitemap line tells crawlers where to find your sitemap.

Is robots.txt the same as noindex?

No — these are completely different signals. Blocking a URL in robots.txt prevents Googlebot from visiting and crawling that page. A noindex meta tag tells Google not to include the page in its search index, but Google can still visit and crawl it. The critical difference: if a page blocked by robots.txt has external links pointing to it, Google may still index it — as an empty listing with no content — because it knows the URL exists even without being able to crawl it. For true indexing prevention, noindex is more reliable. For blocking crawling entirely (e.g., to save crawl budget), robots.txt is the right tool.

What should you block in your robots.txt file?

Block pages that should never appear in search results and would waste crawl budget if Googlebot visited them repeatedly:

  • Admin and login pages — /admin/, /wp-admin/, /login/, /dashboard/ — these have no SEO value and should never be indexed
  • Checkout and cart pages — /checkout/, /cart/, /order/ — transactional flow pages serve no organic search purpose
  • Internal search result pages — /search?q= — search results pages typically create thin, near-duplicate content at scale
  • Thank-you and confirmation pages — /thank-you/, /order-confirmed/ — post-conversion pages should not appear in Google results
  • Tag archive pages (if they generate thin content) — /tag/ paths on blogs with shallow tag archives
  • Staging environments — if your staging site is publicly accessible, block all crawlers from it to prevent duplicate indexing of your live site

What should you never block in robots.txt?

These are the mistakes that can cause serious SEO damage:

  • CSS and JavaScript files — blocking these prevents Google from rendering your pages correctly, which can cause your entire site to appear broken in search results. This is one of the most damaging robots.txt mistakes.
  • Your homepage — accidentally blocking / with a Disallow: / directive removes your entire site from Google. This happens more often than you would think, often during site migrations.
  • Your sitemap locations — blocking /sitemap.xml defeats the purpose of having a sitemap
  • Pages with valuable backlinks — blocked pages cannot have their link equity properly passed on, even if they 301 redirect elsewhere

What is the most dangerous robots.txt mistake?

The most dangerous mistake is Disallow: / under User-agent: * — this blocks every crawler from every page on your entire domain. It is the robots.txt equivalent of taking your website offline for search engines. This error sometimes gets introduced during site rebuilds when developers copy a staging site's robots.txt (which correctly blocks crawlers) to production. Always verify your robots.txt immediately after any site migration or CMS change.

How do you test your robots.txt file?

Google Search Console has a free robots.txt tester built into it. Go to Search Console → Settings → robots.txt and enter any URL to see whether Googlebot would be allowed or blocked from accessing it. This is the definitive test — it uses Google's actual parsing of your file rather than a third-party approximation.

You can also view your robots.txt by navigating directly to yourdomain.com/robots.txt in your browser. If you see a server error rather than the file content, robots.txt is missing from your root directory.

Can you have different rules for different crawlers?

Yes — the User-agent line specifies which crawler the following rules apply to. You can have separate rule sets for different bots:

User-agent: Googlebot
Disallow: /internal/

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /admin/

This is particularly relevant in 2026 for controlling AI training crawlers. OpenAI's GPTBot, Google's Google-Extended, and others can be specifically blocked from training on your content while still allowing regular search crawlers access. As covered in our guide on meta tags for AI search, robots.txt is one layer of AI content control available to publishers.

Summary

  • Robots.txt tells crawlers which parts of your site to access — it is read before any crawling begins
  • Blocking a URL in robots.txt prevents crawling, not indexing — use noindex for indexing prevention
  • Block admin pages, checkout flows, internal search results, and staging environments
  • Never block CSS/JS files, your homepage, your sitemap, or pages with valuable backlinks
  • The most dangerous mistake is Disallow: / — check your robots.txt immediately after any migration
  • Test with Google Search Console's built-in robots.txt tester, not third-party tools

Continue reading: What is PageRank and does Google still use it in 2026?