Audit your whole site

What does the crawler actually do?

When you hit Start crawl, SEO Pulse does four things in order:

1. Fetches your sitemap.xml from the root of the domain and parses out the first N URLs (5–25, your choice). It handles both urlset and sitemapindex formats. If no sitemap is found, it falls back to scanning just the URL you entered.
2. Visits each page live as a real browser would — no caching, no estimates. It parses the HTML with a real parser, follows redirects, and respects robots.txt.
3. Runs the 9-category audit on every page: meta tags, headings, indexability, structured data (JSON-LD), Open Graph, Twitter cards, internal links, image alts, and performance hints. Each page gets its own 0–100 score.
4. Rolls everything into a site-level report: average score, duplicate titles, lowest-scoring pages, and aggregate critical/warning counts. The report gets a permalink you can share.

Total time is usually 30s–2min depending on page count and how fast your server responds. Each page takes 2–8 seconds to fetch + audit.

When should you run a site crawl?

• Before a redesign or migration — establish a baseline you can compare against post-launch.
• After a deploy — catch broken canonicals, accidental noindex, or duplicate metas fast.
• Quarterly content reviews — find your weakest pages and prioritize updates.
• New client onboarding — get a defensible audit in minutes, not hours.
• After a Google algorithm update — see which pages regressed and why.

Crawler limits and fair use

The free crawler is capped at 25 pages per run and respects polite rate limits. We don't store page bodies — only the audit results and metadata required to render your report. Pages behind authentication, paywalls, or aggressive bot protection (Cloudflare challenges, hCaptcha) cannot be crawled because we fetch them as a normal search engine would.