Crawl control is not the same as hiding content

robots.txt tells well-behaved crawlers which paths they may fetch – it does not reliably remove URLs from the index or protect secrets.

If a URL is already indexed or linked elsewhere, blocking it in robots.txt can even prevent Google from seeing a noindex tag. To keep a page out of search results, use authentication, deletion, or noindex (and allow crawling of that URL when you need the directive read). Snippets: Add meta noindex and Disable archive indexing.

Verified: July 2026 – semantics: Google robots.txt intro; crawler tokens: Google common crawlers.

How WordPress exposes robots.txt

WordPress serves a virtual robots.txt and SEO plugins can override it – pick one owner.

Core generates rules (and a Sitemap: line pointing at the native sitemap when enabled).
Yoast, Rank Math, and similar can replace the file via their UI.
You can also place a physical robots.txt in the web root – that usually wins over the virtual one. Know which layer is active.

Always pair crawl rules with a correct sitemap – see XML sitemaps in WordPress.

AI crawlers: Googlebot vs Google-Extended vs GPTBot

Search crawling, AI training/grounding, and chatbots use different tokens – and not every bot honors robots.txt.

Googlebot – classic Search crawling/indexing.
Google-Extended – controls use of content for Gemini and grounding features. It does not change inclusion or ranking in Google Search.
GPTBot / other AI fetchers – separate vendors; publish explicit Allow/Disallow groups if your policy requires it.

Honest crawlers tend to respect robots.txt. Scrapers that ignore it will not. Treat robots.txt as a policy signal, not a firewall. For broader AI visibility strategy, read WordPress GEO.

Syntax that usually works

Group by user-agent, prefer specific Disallow paths, and declare your sitemap explicitly.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap_index.xml

Notes:

One group applies until the next User-agent line.
Order and longest-match rules matter; test instead of guessing.
Keep Sitemap: on its own line with an absolute HTTPS URL.

WordPress and WooCommerce examples

Block admin and noisy parameters; do not blanket-block /wp-content/ if you want CSS/JS/images fetched.

Typical safe Disallows (adjust to your stack):

/wp-admin/ with Allow for admin-ajax.php
Internal search: /?s= or your search path
WooCommerce: cart, checkout, my-account, and add-to-cart query params
Faceted navigation parameters that create infinite URL variants

Avoid blocking whole /wp-content/uploads/ unless you have a deliberate reason – Google may need product and article images.

Testing and common mistakes

Use Search Console’s robots.txt tester (or equivalent) after every change.

Do not use robots.txt to “secure” private PDFs – use auth.
Do not Disallow a URL you still want indexed.
Do not ship Disallow: / on production after copying staging.
Do not assume AI or Search ranking changes from Google-Extended alone.

Related on WP-Dude

Power of robots.txt in WordPress: Best Practices and Mistakes to Avoid