Crawling is not indexing
Almost every serious mistake made with robots.txt comes from one confusion. The file controls whether a crawler fetches a URL. It says nothing about whether that URL appears in search results.
Those come apart in a way that surprises people. Block a page in robots.txt and Google may still list it, showing the URL with no description and a note that it could not read the page. It learned the address from a link somewhere and had no instruction to leave it out — only an instruction not to look.
Worse, blocking is how a page gets stuck in the index permanently. The noindex tag that would remove it lives on the page, and the crawler was told not to fetch the page, so the instruction is never read. To remove something from search results you have to let the crawler in.
The syntax, which is small
Four directives cover essentially everything:
User-agent opens a group of rules aimed at one crawler, or at all of them with *. A crawler obeys the single most specific group that names it and ignores the rest — it does not combine them.Disallow names a path prefix not to fetch. An empty value means nothing is blocked.Allow carves an exception out of a broader Disallow. Where both match, the longer and more specific rule wins.Sitemap gives an absolute URL to a sitemap. It sits outside the user-agent groups and is the one line in the file that helps discovery rather than restricting it.
Paths are matched as prefixes, so Disallow: /print blocks /printers/ as well. Two wildcards are supported by the major engines: * for any run of characters and $ to anchor the end of a URL. Matching is case-sensitive.
The four rules worth following
- Never block CSS or JavaScript. Search engines render pages before judging them. Blocking the assets makes a working page look broken to the crawler, and the old advice to block
/wp-includes/ now does exactly this.
- Use noindex for anything that must stay out of results. Thank-you pages, internal search results, filtered listings. Leave them crawlable so the tag can be read.
- Remember the file is public. Anyone can read yours at
/robots.txt, and listing a private directory there advertises it. It has been a reconnaissance step for as long as the file has existed.
- Keep it short. A file with three lines that are correct beats a file with forty inherited from a template nobody has read since.
It is advisory
Well-behaved crawlers honour robots.txt because they choose to. Scrapers, content thieves and anything hostile ignore it completely, and some read it first precisely because it lists the paths you consider interesting.
It is not a security measure and it never was. Anything that genuinely must not be reached needs authentication or a server rule. A directive in a text file is a request, and the visitors you would most like to exclude are exactly the ones who will not honour it.
Where it goes
The web root, as plain text, named exactly robots.txt in lower case, reachable at the domain root and nowhere else. A file in a subdirectory is never fetched. Each subdomain needs its own, and as far as the specification is concerned the http and https versions of a host are separate.
WordPress complicates this. It serves a virtual robots.txt that does not exist on disk, and SEO plugins add their lines to it. Upload a real file and it overrides the whole thing silently, including whatever your plugin was contributing. So load /robots.txt and read what is currently being served before you replace it, and fold anything worth keeping into the new file.
If you build one with the [msm_tool slug="robots-txt-builder"], the WordPress preset is worth understanding rather than accepting: it blocks /wp-admin/ while re-allowing admin-ajax.php, because themes and plugins call that file from the front end and blocking it breaks features on public pages.
Crawl budget is not your problem
A great deal of robots.txt advice is written for sites with hundreds of thousands of URLs, where crawl budget genuinely matters and blocking faceted-navigation parameters saves real crawler time.
Below roughly ten thousand pages this is not a constraint you have. Google will happily crawl a small site in full. Optimising a file for crawl efficiency on a two-hundred-page site is effort spent on a bottleneck that does not exist, and every extra rule is another chance to block something by accident.
Before uploading any change, test the paths that matter against the rules, and check the live file in Search Console afterwards. A robots.txt error is unusual among SEO mistakes in that it can remove a site from search within days, and nothing about the site looks wrong when you visit it yourself.