Free SEO tools / Crawl & indexability

Robots.txt Generator

Create robots.txt rules, test a path and download the file.

Leave empty to allow all paths for this crawler.

How to use the Robots.txt Generator

  • Choose the crawler and paths

    Use * for the general crawler group. Add one Disallow path per line and any Allow exceptions for paths that should remain crawlable.

  • Add your sitemap locations

    Optionally enter full sitemap URLs, then generate the file. Review the output alongside any rules your website already uses.

  • Test before publishing

    Try important paths against the generated rules, download robots.txt and publish it at the root of the intended host.

Review your robots.txt before it goes live

The generator prepares a file for you to review. Your website changes only when you publish the file yourself.

  • Disallow: / blocks every path unless a more specific Allow rule applies. Leave blocked paths empty if this crawler should access everything.
  • Path rules belong to the selected user-agent group. Merge the output carefully if your existing file has several crawler groups.
  • Use robots.txt to manage crawling. It does not protect confidential files or reliably remove URLs that are already indexed.

How it works

Create and test your robots.txt file

Choose a crawler, enter one path pattern per line and optionally add sitemap URLs. Test a path against the generated rules before downloading robots.txt.

Check your complete website with URLMatcher

Where should I publish the file?

Publish it at /robots.txt on the exact protocol and host it should control. Each subdomain has its own robots.txt.

How do I allow crawling everywhere?

Leave Disallow paths empty. The generator writes an empty Disallow rule. A more specific Allow rule can create exceptions to blocked paths.

Will this remove pages from Google?

No. robots.txt controls crawling. It is not an access control mechanism or a reliable way to remove indexed URLs.

Try URLMatcher

Take your next SEO step with URLMatcher

Put your technical SEO changes in context. Create an SEO Project to inspect page status, indexability, canonicals and internal links across your website.