Robots.txt Builder

How to use 3 steps

Pick a preset under Select CMS Preset Strategy.
Add your Sitemap URL if you have one.
Press Generate Robots.txt, then Download robots.txt and upload it to your site root.

Select CMS Preset Strategy:

Sitemap URL (Optional):

Generated robots.txt Configuration

How this tool works

A robots.txt file controls crawling, not indexing. That distinction causes most of the damage the file does: a page blocked here can still appear in search results, listed without a description, because the crawler was told not to read it rather than told to leave it out.

Four presets cover the common cases — allow everything, a WordPress layout, an ecommerce layout, and block everything. The WordPress preset blocks /wp-admin/ while re-allowing admin-ajax.php, which themes and plugins call from the front end. Adding a sitemap URL appends the one directive in the file that helps discovery rather than restricting it.

Before you upload it

Use noindex to keep a page out of results: robots.txt is the wrong instrument. Worse, blocking a page prevents the crawler reading the noindex tag on it, so the page stays in the index indefinitely.
It belongs at the domain root: reachable at /robots.txt and nowhere else. A file in a subdirectory is never read, and a subdomain needs its own.
Never block CSS or JavaScript: search engines render pages before judging them. Blocking assets makes a working page look broken to the crawler.
Remember it is public: anyone can read yours. Listing a private directory there advertises it rather than protecting it.

Crawling is not indexing

Almost every serious mistake made with robots.txt comes from one confusion. The file controls whether a crawler fetches a URL. It says nothing about whether that URL appears in search results.

Those come apart in a way that surprises people. Block a page in robots.txt and Google may still list it, showing the URL with no description and a note that it could not read the page. It learned the address from a link somewhere and had no instruction to leave it out — only an instruction not to look.

Worse, blocking is how a page gets stuck in the index permanently. The noindex tag that would remove it lives on the page, and the crawler was told not to fetch the page, so the instruction is never read. To remove something from search results you have to let the crawler in.

The syntax, which is small

Four directives cover essentially everything:

User-agent opens a group of rules aimed at one crawler, or at all of them with *. A crawler obeys the single most specific group that names it and ignores the rest — it does not combine them.
Disallow names a path prefix not to fetch. An empty value means nothing is blocked.
Allow carves an exception out of a broader Disallow. Where both match, the longer and more specific rule wins.
Sitemap gives an absolute URL to a sitemap. It sits outside the user-agent groups and is the one line in the file that helps discovery rather than restricting it.

Paths are matched as prefixes, so Disallow: /print blocks /printers/ as well. Two wildcards are supported by the major engines: * for any run of characters and $ to anchor the end of a URL. Matching is case-sensitive.

The four rules worth following

Never block CSS or JavaScript. Search engines render pages before judging them. Blocking the assets makes a working page look broken to the crawler, and the old advice to block /wp-includes/ now does exactly this.
Use noindex for anything that must stay out of results. Thank-you pages, internal search results, filtered listings. Leave them crawlable so the tag can be read.
Remember the file is public. Anyone can read yours at /robots.txt, and listing a private directory there advertises it. It has been a reconnaissance step for as long as the file has existed.
Keep it short. A file with three lines that are correct beats a file with forty inherited from a template nobody has read since.

It is advisory

Well-behaved crawlers honour robots.txt because they choose to. Scrapers, content thieves and anything hostile ignore it completely, and some read it first precisely because it lists the paths you consider interesting.

It is not a security measure and it never was. Anything that genuinely must not be reached needs authentication or a server rule. A directive in a text file is a request, and the visitors you would most like to exclude are exactly the ones who will not honour it.

Where it goes

The web root, as plain text, named exactly robots.txt in lower case, reachable at the domain root and nowhere else. A file in a subdirectory is never fetched. Each subdomain needs its own, and as far as the specification is concerned the http and https versions of a host are separate.

WordPress complicates this. It serves a virtual robots.txt that does not exist on disk, and SEO plugins add their lines to it. Upload a real file and it overrides the whole thing silently, including whatever your plugin was contributing. So load /robots.txt and read what is currently being served before you replace it, and fold anything worth keeping into the new file.

If you build one with the [msm_tool slug="robots-txt-builder"], the WordPress preset is worth understanding rather than accepting: it blocks /wp-admin/ while re-allowing admin-ajax.php, because themes and plugins call that file from the front end and blocking it breaks features on public pages.

Crawl budget is not your problem

A great deal of robots.txt advice is written for sites with hundreds of thousands of URLs, where crawl budget genuinely matters and blocking faceted-navigation parameters saves real crawler time.

Below roughly ten thousand pages this is not a constraint you have. Google will happily crawl a small site in full. Optimising a file for crawl efficiency on a two-hundred-page site is effort spent on a bottleneck that does not exist, and every extra rule is another chance to block something by accident.

Before uploading any change, test the paths that matter against the rules, and check the live file in Search Console afterwards. A robots.txt error is unusual among SEO mistakes in that it can remove a site from search within days, and nothing about the site looks wrong when you visit it yourself.

Frequently Asked Questions

Specific to the Robots.txt Builder. Broader questions about this category are answered on the category page.

Will blocking a page here hide it from Google?

No, and this is the most common mistake made with the file. Blocked pages can still be listed, usually with no description, if anything links to them. To remove a page from results use a noindex tag or an authenticated response, and leave it crawlable so the instruction can be read.

Does robots.txt keep anyone out?

It is advisory. Well-behaved crawlers honour it; scrapers and bad actors ignore it entirely and some read it specifically to find interesting paths. It is not a security measure.

Where exactly does the file go?

In the web root, as plain text, named exactly robots.txt in lower case. It applies to the protocol, host and port it is served from, so https and http versions are separate as far as the specification is concerned.

WordPress already has a robots.txt — should I replace it?

WordPress serves a virtual one that does not exist on disk. Uploading a real file overrides it completely, including any lines your SEO plugin was adding, so check what /robots.txt currently returns before replacing it.

Should I block /wp-includes/?

It was standard advice for years and is now unnecessary. It can also do harm, because scripts and styles served from there are needed for the page to render correctly for the crawler.