Most WordPress sites get crawled like a supermarket with no doors. Googlebot walks in, grabs whatever it wants, and wastes budget on your /wp-admin/ login page instead of your money pages. A robots.txt file for WordPress is the cheapest fix you’ll ever apply, and almost nobody configures it properly.
Let’s cut the fluff: a robots.txt is just a plain text file sitting in your root domain that tells search engine bots which paths to skip when crawling and indexing. It’s not a security tool, it’s not a magic shield, and it won’t hide your content from anyone determined to see it. It’s a traffic cop. Nothing more.
Where the hell does robots.txt live on a WordPress site
The file always lives at yourdomain.com/robots.txt. Period. No subfolders, no exceptions. If you put it anywhere else, bots won’t read it and you’ve done nothing.
By default, WordPress creates a virtual robots.txt automatically. You can see it by visiting yourdomain.com/robots.txt right now. Problem is, that virtual file is bare bones and usually only contains a sitemap line plus a generic user-agent directive. That’s not enough for a real site with plugins, feeds, taxonomies and admin URLs.
You’ve got two ways to replace it: create a physical robots.txt file via FTP or your hosting file manager, or use the do_robots filter to override the virtual one from functions.php. I’ll defend the physical file approach — it’s easier to debug, survives theme changes, and any SEO tool can verify it instantly.
A robots.txt example that actually works in 2026
Here’s a functional robots.txt for a WordPress site. Copy it, swap the domain, and stop overthinking.
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpDisallow: /wp-includes/Disallow: /wp-content/plugins/Disallow: /wp-content/cache/Disallow: /wp-content/themes/Disallow: /trackback/Disallow: /comments/feed/Disallow: /?s=Disallow: /search/Disallow: */trackback/Disallow: */feed/Disallow: */comments/feed/Sitemap: https://www.yourdomain.com/sitemap_index.xml
Notice the Allow: /wp-admin/admin-ajax.php line. Every «guru» blog forgets it, and then their site breaks because admin-ajax handles frontend requests on tons of themes and plugins. Block all of /wp-admin/ without that exception and you’ll tank your own site.
Also notice the search parameter block: Disallow: /?s= and Disallow: /search/. Internal search results create infinite crawlable URLs. Google will index thousands of them if you let it. That’s the single biggest crawl budget killer on WordPress.
What to block and what to leave alone
The right robots.txt for WordPress blocks folders that have zero SEO value and leaks crawl budget. Here’s the shortlist:
- /wp-admin/ — except admin-ajax.php, as explained.
- /wp-includes/ — core files, no reason for bots to touch them.
- /wp-content/plugins/ and /wp-content/themes/ — asset folders, not content.
- /wp-content/cache/ — generated files, pure noise.
- /trackback/ and */trackback/ — spam magnets.
- /comments/feed/ and */comments/feed/ — duplicate feed versions of your posts.
What people get wrong: blocking /wp-content/uploads/. Don’t. Your images live there, and if you block it, Google Images can’t index them. Same with /wp-content/uploads/ PDFs or media assets you actually want ranked.
Another common mistake is disallowing /category/ and /tag/ across the board. On small sites with clean taxonomy, those archive pages can rank and bring traffic. On a 500-post blog with 200 tag pages, they’re pure cannibalization. Choose based on your actual setup, not a copy-paste rule.
The sitemap line is not optional
A robots.txt file for WordPress without a Sitemap: directive is a wasted opportunity. This line tells bots exactly where to find your XML sitemap so they can crawl your real content faster.
If you use Yoast SEO, Rank Math or Google XML Sitemaps, your sitemap usually sits at /sitemap_index.xml or /sitemap.xml. Verify it in a browser first — a broken sitemap line is worse than no line at all, because it wastes a crawl request.
You can list multiple sitemaps, one per line. Useful if you run a WooCommerce store with a separate product sitemap.
Robots.txt won’t hide your pages — stop trusting it for that
This is where I’ll be blunt: robots.txt does NOT prevent indexing. It prevents crawling. If a page is linked from somewhere else on the web, Google can still index it without ever crawling it — it just shows up without a snippet.
If you want a page out of the index, use a <meta name=»robots» content=»noindex»> tag in the page header. That’s the actual tool. Combine noindex with robots.txt disallow and you get the worst result possible: the page is blocked from crawling, so Google never sees the noindex tag, and it can stay indexed forever.
Rule of thumb: robots.txt to save crawl budget, noindex to remove pages from search. Never mix them on the same URL unless you know exactly what you’re doing.
How to test and validate your robots.txt
Don’t deploy blind. Google Search Console has a robots.txt tester under
