1
Build the list
Merge a sitemap, a Search Console indexed export, and high-value paths from analytics or logs. Deduplicate.
Technical SEO · URL inventory
Crawl the URLs you already know. Leave the junk behind.
List crawling is an SEO audit method: you hand a crawler a finite list from a sitemap, Search Console export, or server log. No discovery. Just the pages that matter.
This site is about search-engine crawling for websites. It is not ListCrawler and not a classifieds directory.
Definition
What list crawling means here
In SEO tools, a list crawl is the opposite of a spider crawl. You supply the URLs. The crawler fetches only that set.
2
Fetch only those URLs
This site’s crawler, Screaming Frog List mode, or a script. Crawl depth stays at zero.
3
Audit the signals
Status codes, redirects, canonicals, noindex, titles, and whether a search engine can still use the page.
When to use it
List crawl vs full-site spider
| Factor | List crawl | Recursive spider |
|---|---|---|
| How URLs are chosen | You provide the set | The bot follows links |
| Scope | Narrow and known | Broad and unpredictable |
| Best for | Migrations, index checks, QA | Finding orphan and unknown pages |
| Risk on large sites | Low — no facet explosion | High — parameters and filters |
Start here
Guides
- → Migration URL audit Old URLs in. Status, hop count, and final destination out. No surprises on launch day.
- → Build a list from GSC, sitemap, and logs Three inventories, one deduplicated set. The gaps between them are the audit.
- → Screaming Frog List mode Upload the list, keep depth at zero, and follow redirects only when a migration demands it.
- → What list crawling is in SEO A finite URL set, fetched on purpose — not a spider wandering your site, and not Googlebot’s discovery loop.