Free SEO tools / Crawl & indexability

Robots.txt Tester

Test a page URL against robots.txt and see the matching crawler rule.

How to use the Robots.txt Tester

  • Choose live or pasted rules

    Fetch the published robots.txt file, or paste proposed rules to test a change before putting it on your website.

  • Enter a full page URL and crawler

    Test the specific path you care about and select Googlebot, Googlebot-Image, Bingbot or the wildcard group.

  • Inspect the winning rule

    Review the selected user-agent group, the matched Allow or Disallow rule and its line number. Retest important exceptions after editing.

Understand allowed and blocked URLs

Use the tester when a crawler cannot reach a page, or before changing crawl rules for filters, search pages or private paths.

  • Allowed means the tested robots.txt rules permit this crawler to request this path. It does not confirm indexing or access through a firewall.
  • Blocked means a matching Disallow rule wins. Review the rule and any more specific Allow exceptions before changing it.
  • The crawler choice matters: a specific user-agent group can behave differently from the wildcard rules.

How it works

See which robots.txt rule applies

Enter the page URL and a crawler token. The tester fetches robots.txt from that origin, selects the most specific matching user-agent groups and explains the winning Allow or Disallow rule.

Check your complete website with URLMatcher

How are conflicting rules resolved?

The most specific matching path rule wins. Allow wins a tie. Matching groups for the same crawler are combined; the wildcard group is used only when no more specific group matches.

Can I test unpublished changes?

Yes. Choose Paste robots.txt to test proposed rules in your browser without changing your website.

Does allowed mean indexed?

No. This result only describes crawl permission under these rules. It does not test server firewalls or confirm search engine indexing.

Try URLMatcher

Take your next SEO step with URLMatcher

Turn a quick check into a website audit. Create an SEO Project to inspect redirects, indexability, canonicals and internal links across your website.