The mistake that costs a site its traffic
A staging site carries Disallow: / so it stays out of search. Somebody copies the site to production, the file comes with it, and for two weeks nothing looks wrong. Nobody visits robots.txt. The pages still load. Then rankings collapse, and the cause is one line in a file nobody thought about.
This is the most expensive routine error in technical SEO, and it is entirely preventable by testing a robots file before it reaches a server and checking it again afterwards. A blocked path is invisible from the browser: the page you are looking at works perfectly.
How matching actually works
Four rules govern whether a crawler considers a path blocked, and getting a manual check right means holding all four:
- One group applies. A crawler finds the most specific
User-agent group naming it, obeys that, and ignores every other group in the file — including the * group. Rules do not accumulate across groups.
- Prefixes, not paths.
Disallow: /print blocks /printers/ink too. Anchor with a trailing slash when you mean a directory.
- The most specific rule wins. Where an
Allow and a Disallow both match, the longer pattern takes precedence, whichever kind it is. This is how /wp-admin/ stays blocked while admin-ajax.php inside it stays reachable.
- Two wildcards exist.
* matches any run of characters, $ anchors the end of a URL. Disallow: /*? blocks every URL with a query string; Disallow: /*.pdf$ blocks PDFs and nothing else.
What a simplified check tells you
The [msm_tool slug="robots-txt-tester"] compares your path against the Disallow lines in whatever you paste, as a prefix match. That is deliberately less than the specification, and the gaps are worth stating rather than discovering.
It ignores Allow lines, so a path caught by a broad block and released by a narrower exception is reported as blocked when a real crawler would fetch it. It ignores user-agent grouping and tests every Disallow in the file regardless of which crawler it was written for. And it does not expand * or $, so any rule depending on a wildcard is evaluated wrongly.
That makes it useful for one job: sanity-checking rules you are drafting, before they go anywhere near a server. Paste a single user-agent group, test the paths you care about, and confirm the blocks you intended are the blocks you wrote. It is not an implementation of the standard and should not be the last word on a live file.
Testing rules you are still writing
Work one group at a time. Copy only the rules for the crawler you are thinking about, so that directives meant for one bot are not applied to your path as though they were universal.
Then test the paths in three categories, because the third is the one people skip:
- Paths you intend to block. Confirm they match.
- Paths you intend to leave open. Confirm they do not.
- Paths near the boundary. If you blocked
/search, test /searchable-guide. If you blocked /tmp, test /tmpl/. Prefix matching catches neighbours, and this is where accidental blocks come from.
Always test your CSS and JavaScript paths, your images, and your sitemap URL. Blocking any of them is a self-inflicted wound that a two-second check prevents.
Then verify against the live file
Google Search Console has a robots.txt report that tests the file it actually fetched from your domain, using the real matching rules, for a property you have verified. That is the authoritative answer for Google, and it is the one to trust for anything consequential.
Two things it gives you that no offline check can. First, it shows the file Google has, which may not be the file you uploaded — a CDN or a page cache can serve an old copy for hours. Second, it reports when Google could not fetch the file at all, which matters more than most people realise: a robots.txt returning a server error is treated by Google as a signal to pause crawling, so a broken file can be worse than an aggressive one.
A deployment routine
Draft the rules and test the three path categories offline. Upload. Load /robots.txt in a browser and read what came back, because a text file uploaded to the wrong directory returns your 404 page and everything appears fine until you look. Run the Search Console report. Then check back in a few days, in the coverage report, for pages newly excluded by robots.txt.
One last thing worth holding on to: even a correct block does not remove a page from search results. Blocking prevents crawling, and a blocked URL can still be listed if other pages link to it. If the goal is that a page never appears, the answer is a noindex tag on a page the crawler is permitted to read, which is the opposite of blocking it.