How to use the Robots.txt Generator
Choose the crawler and paths
Use * for the general crawler group. Add one Disallow path per line and any Allow exceptions for paths that should remain crawlable.
Add your sitemap locations
Optionally enter full sitemap URLs, then generate the file. Review the output alongside any rules your website already uses.
Test before publishing
Try important paths against the generated rules, download robots.txt and publish it at the root of the intended host.
