14 min read · Technical SEO · Last updated July 2026
Quick answer: Screaming Frog SEO Spider crawls your website and returns every URL with its status code, title, meta, H1, response time, and 50+ other data points. Install the desktop app, enter your domain, press Start. Then use the tab filters to find broken links, duplicate content, redirect chains, missing tags, and page speed issues. The paid licence ($259/year) unlocks crawling over 500 URLs and all advanced features.
Introduction
Every technical SEO audit starts with a crawl. Screaming Frog is the industry standard tool for that crawl — used by agencies, in-house SEO teams, and developers to baseline a site’s technical health before any other work begins.
The learning curve is real. Screaming Frog surfaces hundreds of data points across dozens of tabs, and without a framework for what to look at first, the data becomes noise rather than signal.
This guide gives you that framework. You’ll learn what each report shows, which settings matter for large sites, and how to compare crawls over time to prove your technical work is moving the needle.
What you’ll learn:
– How to configure Screaming Frog for accurate, complete crawl data
– What each tab and filter shows and which ones matter most
– How to handle large sites (100k+ URLs) without hitting memory limits
– How to compare two crawls to identify regressions and improvements
Table of Contents
- Installation and Initial Configuration
- Running Your First Crawl
- Response Codes: The Most Important Tab
- URL Report: Finding Broken and Redirected Pages
- Page Titles and Meta Descriptions
- H1 and Heading Structure
- Images, Canonicals, and Hreflang
- Configuration for Large Sites
- Comparing Crawls Over Time
- Frequently Asked Questions
Installation and Initial Configuration
Screaming Frog runs on Windows, Mac, and Linux as a desktop application. Download the free version from screamingfrog.co.uk — it crawls up to 500 URLs without a licence, which is enough to audit small sites or test configuration before a paid crawl.
Before you start crawling, configure these settings:
Configuration → Spider → Crawl: Ensure “Follow Internal Nofollow Links” is checked if you want to find all pages including those with nofollow internal links. For a complete inventory crawl, you want to see everything Google might have found.
Configuration → User-Agent: By default, Screaming Frog identifies itself as Screaming Frog. Change this to Googlebot to see how your server responds specifically to Google — some servers or Cloudflare rules behave differently for different user agents.
Configuration → Rendering: Choose “JavaScript” rendering mode if your site uses a JS framework. This takes significantly longer but returns accurate data for React, Vue, Angular, and Next.js sites. For pure HTML sites, keep the default (None).
Configuration → Speed: For most sites, the default 5 threads and 0ms throttle works fine. For shared hosting or sites with aggressive rate limiting, reduce threads to 2-3 and add a 100-500ms throttle between requests.
Running Your First Crawl
Enter your domain in the search bar (include https://) and click Start. Screaming Frog immediately begins requesting URLs and populating the report tabs in real time.
What to watch during the crawl:
– URL count: is it growing at a reasonable pace, or stalling? Stalling suggests rate limiting or redirect loops.
– Response codes: watch for a spike in 4xx or 5xx responses early in the crawl — this often reveals structural problems.
– Crawl depth: check the Crawl Depth tab filter. Pages beyond depth 5 are rarely crawled by Google; anything important that is buried that deep needs architecture changes.
When the crawl completes, save the crawl file (File → Save Crawl) before you start analysing. This preserves the data for comparison later.
Screaming Frog Report Navigator
Click any tab to see what it shows and what to look for.
Response Codes: The Most Important Tab
The Response Codes tab is where every audit starts. It groups every crawled URL by its HTTP status and lets you filter by 2xx, 3xx, 4xx, or 5xx.
4xx analysis: Export all 4xx URLs and sort by “Inlinks” (the number of internal pages linking to that broken URL). A 404 with 200 internal links pointing at it is a significant link equity leak. A 404 with zero internal links is probably a residual URL from old content — lower priority.
For each broken URL with internal links, decide: should this URL return content again (restore it), or should it redirect to the nearest relevant live page? Redirect to the most topically relevant page, not always the homepage.
3xx chain analysis: Screaming Frog’s Redirect Chains report (under Reports → Redirect Chains) shows every multi-hop redirect on your site. Any chain longer than one hop should be collapsed to a direct 301. A chain of A→B→C→D should become A→D, B→D, C→D.
Key takeaway: Sort your 4xx list by inlink count. The top 10 broken pages with the most internal links pointing at them are your highest-priority fixes.
URL Report: Finding Broken and Redirected Pages
The URL tab shows every URL Screaming Frog discovered during the crawl with its full metadata. The columns you want to sort and filter by most often:
- Status Code: filter 404 to find broken pages; filter 301 to find redirect destinations
- Indexability: filter “Non-Indexable” to see every URL excluded from Google’s index with the reason
- Crawl Depth: filter depth > 4 to find pages buried too deep for consistent Google crawling
- Inlinks: sort descending to find your most internally-linked pages (these should be your best content)
- Unique Inlinks: the number of unique pages linking to each URL (vs. total link count including duplicate links from same pages)
The most revealing combination: filter for Indexable pages, sort by Inlinks ascending. These are your indexable pages with the fewest internal links — your promotion gaps. The pages Google is supposed to rank but that receive the least internal support.
Page Titles and Meta Descriptions
The Page Titles tab and Meta Descriptions tab work identically: Screaming Frog pulls the tag content and flags it by length, duplication, and presence.
Common patterns in title audit data:
A new e-commerce client we audited had 847 product pages. Of those, 312 had duplicate title tags because the template used {Brand} | {Product Category} without including the product name. Google was being asked to differentiate 312 pages with identical signals.
A WordPress blog had 204 posts with meta descriptions over 250 characters — all auto-generated from the first paragraph of each post. Every SERP snippet was being written by Google instead.
Both are fixable at the template level. That is the key insight from title and meta audits: most problems are systematic, not page-by-page. Find the pattern, fix the template, resolve hundreds of issues at once.
H1 and Heading Structure
Missing H1s on indexable pages are always worth fixing. The H1 is the strongest on-page signal for what a page is about. A page without one forces Google to infer the topic from body content alone.
Multiple H1s are almost always a template issue. In WordPress, common culprits include:
– Genesis themes that wrap the site name in an H1 on every page
– Page builders that add an H1 wrapper around content blocks
– Headers that use H1 for the logo or brand name in addition to the page title H1
Use “View Source” on a page with multiple H1s to find the exact HTML. Then fix it in the theme template, not by editing individual pages.
The Screaming Frog heading order report (Reports → Headings) shows every page where headings appear out of sequence (H3 before H2, H2 before H1). This matters for both crawlability and accessibility.
Images, Canonicals, and Hreflang
Images: the filter you care about most is “Missing Alt Text” on indexable product and landing pages. Empty alt attributes are fine for decorative images — blank alt (alt="") tells assistive technology to skip the image. But blank alt on product images is a missed keyword opportunity and an accessibility failure.
For images over 100KB, Screaming Frog will show file size in the Images tab. Cross-reference with your PageSpeed Insights data — large images are usually the biggest contributor to poor LCP scores on image-heavy pages.
Canonicals: the most common canonical mistake Screaming Frog catches is a canonical pointing to a URL that itself redirects. After a site migration, canonical tags often still point to old URLs that now 301 elsewhere. Google does not follow canonical chains — it either ignores the canonical or uses the redirected URL, neither of which is predictable.
After any migration, always run Screaming Frog and filter the Canonical tab for “Canonical to Redirect.”
Configuration for Large Sites
Sites with 100,000+ URLs require specific Screaming Frog configuration to complete without running out of memory or time.
Crawl Configuration Checklist for Large Sites
Check off each setting before running a large-site crawl.
0/10 checked
Memory management for very large sites: Screaming Frog stores crawl data in memory by default. For sites over 500,000 URLs, switch to “Storage Mode: Database” under Configuration → System. This uses a local SQLite database instead of RAM, allowing crawls of millions of URLs at the cost of crawl speed.
Targeted crawls: For large sites, spider-crawling from the homepage is inefficient. Instead, use Configuration → Spider → Crawl and switch to “Crawl a List” mode. Feed it your XML sitemap. This audits all URLs you know about without wasting time following deep crawl paths to low-value pages.
Comparing Crawls Over Time
Save every crawl file (.seospider format). When you want to compare, open your new crawl and use File → Compare Crawl. Screaming Frog shows you:
- New URLs (pages that appeared since the last crawl)
- Missing URLs (pages that disappeared — possible accidental noindex or deletion)
- Status code changes (a page that was 200 now returns 301 or 404)
- Title, meta, H1 changes (content modifications that might affect rankings)
When to compare crawls:
– Before and after any CMS or theme update
– Before and after a migration
– Monthly as a routine health check
– After implementing a batch of technical fixes (to verify they worked)
The status code change report is the most valuable after migrations. It immediately shows pages that were 200 before the migration and are now 4xx — exactly the redirect map failures you need to find.
Frequently Asked Questions
Is Screaming Frog better than Ahrefs Site Audit?
They serve different purposes. Screaming Frog is a desktop tool that crawls from your machine — it is faster for one-off audits, configurable, and shows JavaScript-rendered content well. Ahrefs Site Audit runs in the cloud, enables scheduled crawls, and integrates backlink and traffic data. Most agencies use both: Screaming Frog for deep audits, Ahrefs for continuous monitoring.
How long does a Screaming Frog crawl take?
At default settings (5 threads, no JavaScript), expect roughly 1,000-3,000 URLs per minute. A 50,000-URL site takes 20-50 minutes. With JavaScript rendering enabled, speeds drop to 100-300 URLs per minute. A 50,000-URL JS site can take 3-8 hours.
Can Screaming Frog crawl sites that require login?
Yes. Use Configuration → Forms-Based Authentication for login forms, or Configuration → Cookies to paste session cookies directly. This is how to audit pages behind a paywall, inside a membership portal, or within a staging environment that requires authentication.
How do I find orphan pages in Screaming Frog?
Crawl via XML sitemap (to get all known pages), then cross-reference with a standard spider crawl from the homepage. Pages in the sitemap that are not found by the spider crawl are orphans — they exist but have no internal links leading to them.
What is the maximum site size Screaming Frog can handle?
With sufficient RAM (16GB+) and database storage mode, Screaming Frog has crawled sites with tens of millions of URLs. For most agency and in-house teams, sites up to 1 million URLs are manageable with a well-configured workstation.
Conclusion
Screaming Frog is the most configurable and fastest crawling tool in the SEO industry. The investment in learning it pays back immediately — a two-hour audit that surfaces redirect chains, broken internal links, and duplicate title tags gives you a prioritised fix list that can meaningfully lift organic performance.
The key habit: save every crawl. The comparison data is often more valuable than the crawl itself, especially when you need to explain to a client or developer exactly what changed and when.
Let Ignited Nepal Handle This
Screaming Frog delivers raw data. Turning that data into a prioritised fix plan, coordinating implementation, and measuring results is a different skill. Our team runs site crawls, triages issues, and delivers implementation-ready reports for clients across Nepal, Australia, UAE, and beyond.
→ Request a Free Technical SEO Audit
Written by the Ignited Nepal SEO team. We build organic search systems for businesses across Nepal, Australia, UAE, USA, UK, and beyond. ignitednepal.com