How to use the Robots.txt Tester
Choose live or pasted rules
Fetch the published robots.txt file, or paste proposed rules to test a change before putting it on your website.
Enter a full page URL and crawler
Test the specific path you care about and select Googlebot, Googlebot-Image, Bingbot or the wildcard group.
Inspect the winning rule
Review the selected user-agent group, the matched Allow or Disallow rule and its line number. Retest important exceptions after editing.
