Do I need a separate crawler before using URLMatcher?
No. You can crawl the old site, the new site or a project website directly in URLMatcher. You can also upload crawl data when you already have it.
SEO Crawler
The SEO crawler for technical audits and migrations.
Discover URLs through sitemaps, robots.txt and internal links, inspect technical SEO data at URL level and compare crawl snapshots across releases.
7-day Pro access. No credit card required.
Crawl setup
Control how your SEO crawler discovers URLs
Start with a domain, project or migration workflow and define the crawl scope before URL collection begins.
Use every discovery source
Find pages through XML sitemaps, robots.txt references and internal links.
Set clear crawl boundaries
Control the page limit, crawl depth, subdomains and robots.txt handling.
Follow crawl progress
See URL discovery and processing while the crawl runs.
URL-level analysis
Inspect the technical SEO signals behind every page
Keep the raw crawl fields and the resulting issues connected, so a finding can be traced back to the affected URLs.
Check crawlability and indexability
Review statuses, redirects, robots directives, canonicals and sitemap coverage.
Audit metadata and content
Find missing, duplicate or unusual titles, descriptions, headings and page content.
Compare structure over time
Track changes in URL sets, crawl depth, inlinks and internal PageRank between snapshots.
Inside the SEO crawler
Move from crawl progress to the URLs that need attention
Open a crawl result, filter the affected URLs and inspect the fields behind each technical SEO issue without leaving the project.
Start my free trial
Where it helps
Use the SEO crawler throughout the site lifecycle
Technical SEO audits
Build a complete URL inventory and trace crawlability, metadata, content and linking issues to individual pages.
Website migrations
Crawl old, staging and live environments to compare what changed before and after launch.
Release checks
Compare snapshots after a redesign, template update or platform release to catch unexpected changes.
FAQ
Frequently asked questions
What does the crawler collect per URL?
The crawler records status and redirect data, canonical and indexability signals, titles, descriptions, headings, body text, images, internal and external links, crawl depth, structured data, hreflang and sitemap coverage.
Does the SEO crawler respect robots.txt?
Yes. It respects robots.txt and uses an identifiable user-agent. Sitemap locations are discovered automatically when available.
Can I control which URLs are crawled?
Yes. Set a page limit and crawl depth, decide how subdomains are handled and choose whether robots.txt directives are respected.
Can I compare two crawls?
Yes. Compare crawl snapshots to identify URL, metadata, status and structural changes between releases or audit dates.
Start your next technical SEO crawl
Add the site, define the scope and inspect the resulting URLs and issues in one workflow.