Free and open source · MIT License · v1.0.0

Scrawly: the free SEO crawler built for Google and AI search

Scrawly is a free, open-source desktop SEO crawler and technical site audit tool for macOS, Windows and Linux. It crawls your website, runs 250 technical SEO, GEO and agentic-web checks on every page, and explains in plain language what to fix. There is no account, no page cap and no subscription.

Download free on GitHub Install guide

$ curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash

Paste into Terminal. Scrawly installs itself and appears in your Applications.

Version 1.0.0, released October 6, 2026. Needs Python 3.11+ and git, which the installer can add for you.

MIT licensed
Open source
No account
No page cap
macOS, Windows, Linux
250 checks in 23 areas

The short version

What is Scrawly?

Scrawly is a free desktop app that audits any website for the technical issues that affect Google rankings and AI search visibility in tools like ChatGPT, Perplexity and Google AI Overviews. It runs on your own computer, with no account and no page limit.

Think of it as a site crawler and a patient SEO teacher in one window. Scrawly visits your pages, renders JavaScript in a real Chromium browser, and checks every page against 250 rules. Each finding tells you why it matters, how to fix it and exactly which pages are affected.

Ilias Sami built Scrawly and released it for the SEO community under the MIT License. The engine is written in Python and opens in a native window on macOS, Windows and Linux. Your audits are saved in a local database on your machine, not in someone else's cloud.

What is in version 1.0.0

Version 1.0.0, released on October 6, 2026, is the first public release of Scrawly Free. It includes:

250 checks across 23 areas, with plain-language fixes
HTML and PDF reports, saved wherever you choose
Safe WordPress fixes with a rollback snapshot before every change
One-line installers that add Scrawly to your apps
Automatic updates that only install versions whose tests passed
Send feedback from inside the app, with an email or GitHub fallback

Scrawly · example.comIllustration

DashboardSite ExplorerIssues & AuditsAgenticVisualizationsReports

Finding and crawling pages

Crawled

Discovered

Queued

Pages/s

Now crawling

example.com/

example.com/blog/

example.com/pricing

example.com/docs/install

example.com/about

example.com/blog/first-post

example.com/contact

example.com/

example.com/blog/

example.com/pricing

example.com/docs/install

example.com/about

example.com/blog/first-post

example.com/contact

Top priority issuesSorted by impact

CriticalRedirect loop
The URL never resolves, so crawlers and visitors give up.
Copy promptPages
HighAI search crawlers blocked
OAI-SearchBot, Claude-SearchBot or PerplexityBot cannot fetch these pages.
Copy promptPages
Infollms.txt missing at the root
AI systems find no curated guide to your most useful pages.
Copy promptPages

Illustration of the Scrawly interface using the placeholder site example.com. It shows the layout and real check names, not audit results.

Check catalogue

What does Scrawly check? 250 checks in 23 areas

Scrawly runs 250 checks grouped into 23 areas, from crawlability and redirects to Core Web Vitals, structured data and AI search readiness. Each finding explains why it matters, how to fix it and which pages are affected.

The 23 check areas

Crawlability and indexability
15 checks
robots.txt blocks, noindex in meta tags and headers, conflicting index signals, orphan URLs and pages more than 4 clicks deep.
Response codes and redirects
14 checks
4xx and 5xx pages, soft 404s, broken internal links with their sources, redirect chains and loops, and 302s that should be 301s.
Canonicalization
11 checks
Missing, conflicting or relative canonicals, canonicals that point at redirects or noindex pages, and canonicals changed by JavaScript.
Titles, meta and headings
21 checks
Missing and duplicate titles, pixel-width truncation, meta descriptions too thin for AI, H1 problems, heading order and Open Graph tags.
Content quality
12 checks
Exact and near-duplicate pages, thin content, keyword cannibalization, placeholder text, low text-to-HTML ratio and missing dates.
Internal linking and architecture
12 checks
Orphan pages, pages with one inlink, generic anchors like "click here", links to redirects and links that only appear after JavaScript runs.
External and outbound links
6 checks
Broken and redirecting outbound links, plain HTTP links, missing rel sponsored or ugc, and target=_blank without noopener.
Images and media
11 checks
Missing or overlong alt text, images over 100 KB, missing width and height, broken sources, no WebP or AVIF, and lazy-loading issues.
Hreflang and international
8 checks
Missing return tags, invalid language codes, x-default, canonical conflicts and mismatches between HTML, headers and sitemaps.
Structured data and schema
15 checks
Invalid JSON-LD, missing required or recommended properties, @id conflicts, and Organization, Breadcrumb, Article, Product and FAQ markup.
XML sitemaps
10 checks
Missing or broken sitemaps, noindexed or redirected URLs inside them, missing indexable pages, size limits, invalid XML and stale lastmod.
Robots directives (file level)
7 checks
"Disallow: /" on the whole site, blocked CSS and JS paths, wildcard mistakes, the wrong content type, a missing Sitemap line and a high crawl-delay.
Security
10 checks
HTTPS, mixed content, HSTS, expiring TLS certificates, security headers, server version leaks and exposed WordPress versions.
Performance and Core Web Vitals
14 checks
LCP over 2.5 seconds, INP over 200 ms from field data, CLS over 0.1, slow TTFB, render-blocking files, page weight and DOM size.
Mobile
7 checks
Viewport, small tap targets, sideways scrolling, small fonts, mobile and desktop content parity, and intrusive pop-ups.
JavaScript rendering
6 checks
Content that only exists after rendering, and titles, canonicals or robots tags that differ between the raw response and the render.
Accessibility
9 checks
A WCAG subset that helps both SEO and AI agents: accessible names, ARIA, form labels, color contrast, page language and landmarks.
AI search readiness (GEO)
13 checks
AI crawlers blocked in robots.txt or silently by a CDN or firewall, client-side-only content, weak answer structure and missing entity schema.
Agentic web and llms.txt
13 checks
llms.txt presence and quality, the accessibility tree AI agents read, layout shifts after load, and whether agents can finish key flows.
URL structure and hygiene
9 checks
Uppercase letters, unsafe characters, double slashes, URLs over 115 characters, session parameters, underscores and trailing-slash mix-ups.
Pagination
5 checks
Noindexed paginated pages, page 2 and later canonicalized to page 1, and infinite scroll with no paginated URLs.
WordPress and HTML validation
17 checks
Indexed attachment, tag and search pages, Yoast and Rank Math both active, indexable staging sites, plus DOCTYPE, lang and charset checks.
Analytics, tracking and config
5 checks
GA4 and Google Tag Manager tags, Search Console verification, the favicon, the 404 template and multiple homepage versions.
250checks in 23 areas, each with why it matters, how to fix it and the pages affected.

Some checks need extra data to run, such as JavaScript rendering, a PageSpeed Insights key, Search Console access or a WordPress site. Counts are unique check IDs in the Scrawly source code.

Findings you can act on, ranked by impact

Critical
High
Medium
Low
Info

Every finding has a severity: Critical, High, Medium, Low or Info. Scrawly sorts by impact, so the problems that hurt the most pages come first.

Severity is coverage-aware. If at least 20% of your live pages share a problem, it moves up one level, and at 40% it moves up two. It needs at least 5 affected URLs, it stops at High, and it never turns a problem into a Critical on its own.

Precedence rules stop double counting. A page that returns an error is not also flagged for a missing title, and an exact duplicate is not also reported as a near duplicate.

Site health score and internal authority

Each audit gets a site health score from 0 to 100. It weighs the share of pages free of Critical or High issues, the share that can be indexed and the share that is not broken, minus a few points for serious site-wide problems. The dashboard and the report use the same model.

Scrawly also computes Authority, a PageRank-style score from 0 to 100 built from your own internal links. It shows which pages your site structure really favors, not just raw inlink counts.

GEO and AEO

How does Scrawly audit for AI search?

Scrawly checks whether AI crawlers can reach your pages, whether your llms.txt and meta descriptions give AI systems enough context, and how ready your site is for AI agents. It runs these GEO checks in the same crawl as the classic technical SEO audit.

Scrawly does not track rankings or citations in AI answers. It audits the technical signals that decide whether AI systems can reach, read and understand your pages.

AI crawler access matrix

Scrawly tests 10 user agents. It reads your robots.txt rules for each one, then fetches your start page as that bot. If robots.txt says yes but the live fetch fails, Scrawly flags a silent block by your CDN or firewall, which a robots.txt checker alone would miss.

Training bots and search bots

Blocking training crawlers such as GPTBot, CCBot, ClaudeBot or Google-Extended is a policy choice, so Scrawly reports it as Info. Blocking AI search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot is rated High, because it can keep your pages out of AI answers.

llms.txt checks

Scrawly checks whether /llms.txt exists, whether it has a title and real links, and whether it has turned into a link dump. It flags files with more than 800 links and suggests 20 to 50 high-value links instead. It also notes when the optional llms-full.txt is missing.

Meta descriptions as AI context

A meta description is often the first thing an AI system reads about a page. Scrawly flags descriptions under 120 characters as too thin. With your own AI key, it can draft an entity-rich description from the page's own content.

Answers, entities and freshness

GEO checks also look for a direct answer before the context, a clear heading hierarchy, author and freshness signals, Organization and WebSite schema, and main content that only appears after JavaScript runs.

AI crawler matrix

10 user agents, each tested two ways

Every agent is checked against your robots.txt rules and with a live fetch of your start page.

Classic search

Googlebotrobots.txtlive fetch
bingbotrobots.txtlive fetch

AI search

OAI-SearchBotrobots.txtlive fetch
Claude-SearchBotrobots.txtlive fetch
PerplexityBotrobots.txtlive fetch

AI user fetch

ChatGPT-Userrobots.txtlive fetch

AI training

GPTBotrobots.txtlive fetch
ClaudeBotrobots.txtlive fetch
CCBotrobots.txtlive fetch
Google-Extendedrobots.txtlive fetch

Agentic web

Agentic web readiness audit

A separate audit asks a simple question: can an AI agent discover, read, sign in to and buy from this site? It runs 21 probes in 6 categories against your site root, at a fixed cost of about 25 requests, whether the site has 10 pages or 100,000.

Probes cover robots.txt AI bot rules, sitemaps, Link headers, Markdown negotiation, llms.txt, Content Signals, OAuth discovery, MCP Server Cards, A2A Agent Cards, Agent Skills and agentic commerce protocols such as x402 and ACP. Commerce probes only count when the site really sells something, so a blog is never marked down for lacking a payment rail.

Every probe records the exact requests it made and what it concluded, so each verdict can be checked. The score maps to six readiness levels:

Level 0Not agent-ready
Level 1Basic web presence
Level 2Agent-discoverable
Level 3Agent-readable
Level 4Agent-operable
Level 5Agent-native

Features

Everything you expect from a desktop SEO crawler

Scrawly covers the core jobs of a desktop crawler: JavaScript rendering, a filterable Site Explorer, custom extraction, site graphs, crawl comparison and white-label reports. On top of that it adds Google data, fix prompts for coding agents and an MCP server.

JavaScript rendering
Pages render in Playwright Chromium, first on desktop and then as a smartphone. Scrawly compares the raw response with the rendered page, so you see what JavaScript adds, removes or changes.
Site Explorer
A spreadsheet-style view with tabs for Internal, Response Codes, Page Titles, Meta Description, H1, Directives, Canonicals, Content, Analytics, PDF and Custom. Every tab has filters with live counts.
Seven site graph layouts
Includes a 3D architecture map, a crawl-depth tree, content clusters and redirect chains. Size nodes by authority, inlinks or words, and color them by status, issues or depth.
Custom extraction
Pull any value from every page with CSS selectors, XPath or regex, or count regex matches across the raw HTML.
Platform presets and profiles
Scrawly detects WordPress, Shopify, Wix, Squarespace, Webflow, Framer, Next.js, React, Laravel, Drupal, Joomla, Magento or Ghost and skips admin pages, carts and other crawl traps. Four built-in profiles cover quick checks to full technical audits.
Compare crawls
See resolved issues, new issues and regressions between two audits, plus which URLs appeared, vanished or changed their title, canonical, status or indexability.
Scheduled audits
A small script runs a new audit, compares it with the last one and exits with code 2 if new Critical or High issues appear, so cron, Task Scheduler or CI can alert you. The app must be open.
Search Console and GA4
One read-only Google connection adds clicks, impressions, CTR and position from Google Search Console, plus sessions, engaged sessions, views and conversions from GA4.
PageSpeed Insights and CrUX
Add a free PageSpeed Insights API key to get Lighthouse lab scores and real-user Core Web Vitals from CrUX, when Google has enough field data for the URL.
White-label reports
A polished, self-contained HTML report with your agency name, color, logo and contact line. Save it as a PDF wherever you like, or export selected rows to CSV.
AI writer and audit assistant
Bring your own key for Anthropic (Claude), OpenAI, DeepSeek, OpenRouter or any OpenAI-compatible service. Draft titles and meta descriptions, or ask an assistant that only reasons over the current audit.
Fix prompts for coding agents
Every finding comes with a copy-paste prompt that names the right file on your stack (WordPress, Next.js, Shopify, Astro, Hugo, Django, Rails, Cloudflare and more), with acceptance criteria and a check to verify the fix.

Also included: list mode for an exact set of URLs, crawling behind a login form, PDF crawling, on-demand desktop and smartphone screenshots, and checks on the CSS, JS and media files each page loads. Crawling is polite by default: robots.txt is respected, and 429 or 5xx responses are retried with backoff and then recorded as findings instead of being dropped.

Ready to see your site the way crawlers do?

Scrawly is free for everyone, with every feature included. Install it in one line and start your first audit.

Download free on GitHub Read the install steps

Releases Changelog Desktop guide

Screaming Frog alternative

Scrawly vs Screaming Frog and Sitebulb

Screaming Frog SEO Spider and Sitebulb are excellent tools. Scrawly is a free, open-source option for people who want a deep technical audit without a license, plus a dedicated set of checks for AI search.

How Scrawly compares with Screaming Frog SEO Spider and Sitebulb
Feature	Screaming Frog / Sitebulb	Scrawly
Price	Paid license or subscription (limited free tier or trial)	Free and open source (MIT)
Account or license key	Needed for full use	None
Page limit	The free tier is capped	No cap: you set the limit
AI search (GEO) and agentic-web checks	Some, depending on version	A dedicated set: AI crawler access, llms.txt, citation and agent readiness
Fixes	Report only	Safe, reversible fixes on WordPress
Source code	Closed	Open: read, audit, contribute

Scrawly is independent and not affiliated with Screaming Frog Ltd or Sitebulb. This comparison follows the Scrawly README. Check each vendor's website for current pricing and features.

Already happy with a paid crawler? Scrawly can still run next to it as a free second opinion, especially for AI search and agent readiness.

Install

How to install Scrawly on macOS, Windows and Linux

Install Scrawly with one command: paste the curl line into Terminal on macOS or Linux, or the irm line into PowerShell on Windows. The installer sets everything up and adds Scrawly to your apps, with nothing to sign up for.

What you need

Python 3.11 or newer (3.11, 3.12 and 3.13 are supported)
git, which powers automatic updates
About 150 MB for a one-time download of the crawler browser
Node.js is not needed, because releases ship the interface prebuilt

Missing git or Python? The installer adds them for you where it safely can, with Homebrew on macOS or winget on Windows. Otherwise it prints the exact command to run.

What the installer does

Checks for git and Python 3.11 or newer, and installs them where it safely can.
Downloads Scrawly into a folder called Scrawly in your home folder. Set SCRAWLY_HOME to choose another folder.
Creates a private Python environment and installs the crawler browser (about 150 MB, once).
Adds Scrawly to Applications on macOS, the Start Menu and Desktop on Windows, or the app menu on Linux.

Running the same command again updates an existing install.

Install on macOS

Terminal

Open Terminal
Find Terminal in Applications, then Utilities, or search for it with Spotlight.

Run the install command

Paste this line and press Return.

Terminal

$ curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash

Accept the Command Line Tools if asked
If git is missing, macOS offers to install the Command Line Tools. Accept, then run the command again.
Open Scrawly
Launch Scrawly from Launchpad, Spotlight or the Applications folder in your home folder. A window opens and you are ready.

Install on Windows

PowerShell

Open PowerShell
Open PowerShell from the Start Menu. Windows PowerShell 5.1 works.

Run the install command

Paste this line and press Enter.

PowerShell

PS> irm https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.ps1 | iex

Let winget add anything missing
If git or Python is missing, the installer uses winget to add them. If you install Python yourself from python.org, tick "Add python.exe to PATH".
Open Scrawly
Launch Scrawly from the Start Menu or the Desktop shortcut.

Install on Linux

Terminal

Open a terminal
Use the terminal app that comes with your distribution.

Run the install command

Paste this line and press Enter.

Terminal

$ curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash

Add git or Python if needed
If the installer reports that git or Python is missing, install them with your package manager. On Ubuntu or Debian:
Terminal
```
$ sudo apt install git python3.12 python3.12-venv
```
Open Scrawly
Launch Scrawly from your app menu, or run ./scrawly.sh in the Scrawly folder. If the window does not open on Ubuntu or Debian, install these libraries:
Terminal
```
$ sudo apt install libxkbcommon-x11-0 libxcb-cursor0 libnss3 libgbm1
```

Manual install

Terminal

Install the prerequisites
Make sure Python 3.11 or newer and git are installed.
Clone the repo and run the installer
Run these three commands in a terminal.
Terminal
```
$ git clone https://github.com/IliasSami/scrawly-seo-crawler.git Scrawly
$ cd Scrawly
$ python3 scripts/install.py
```
On Windows, run py scripts\install.py as the last line instead.
Skip shortcuts if you prefer
Set SCRAWLY_NO_SHORTCUTS=1 before running the installer if you do not want app shortcuts.
Open Scrawly
Launch it from your apps, or use the launcher in the Scrawly folder: Scrawly.command on macOS, Scrawly.bat on Windows or scrawly.sh on Linux.

Full details live in the desktop guide on GitHub.

How to use

How to use Scrawly, from first crawl to report

Open Scrawly, click the logo, paste a web address and start the audit. When it finishes, the dashboard opens, Issues & Audits shows what to fix first, and the Reports tab saves an HTML or PDF report.

Your first audit, step by step

Open Scrawly
A window opens and you are ready. There is no sign-in screen.
Start a quick audit
Click the Scrawly logo at the top of the left bar, paste any web address and click Analyze.
Let Scrawly pick the settings
It detects the platform, shows how confident it is and applies a matching preset. You can still change the pages to check, pages at a time, JavaScript loading, robots.txt rules and sitemap discovery.
Start the audit
Scrawly only reads your pages. Nothing on your site is changed.
Watch the live console
Progress, pages per second and a live feed of crawled URLs stay visible on every screen. You can keep using the app while the audit runs in the background.
Read the dashboard
See the site health score, pages crawled, the indexable share, broken pages and total issues, with charts for severity, response codes, crawl depth and more.
Fix what matters first
Issues & Audits sorts findings by impact. Open one to see why it matters, the recommended fix and the evidence, then click Pages to see every affected URL.
Hand off the fix
Copy prompt gives you a ready-made fix prompt for your site's platform. On a connected WordPress site, you can apply supported fixes from the same row.
Dig deeper
Use Site Explorer for raw data, Titles for pixel widths, Agentic for AI agent readiness and Visualizations for your site's structure.
Share the report
In Reports, open the HTML report in your browser or click Print / Save PDF. Add your agency name, color and logo first if the report is for a client.

Keep audit history for your sites

A quick audit keeps only the latest run. To keep history, compare crawls and run scheduled audits, add the site under Clients with + New Client, then run the crawl wizard: Target, Scope (pick a crawl profile) and Review & Launch.

Optional connections

AI provider: Add your own key under Settings for the AI writer and the audit assistant. The key is stored on your computer.
Google Search Console and GA4: Connect both under Settings, then Integrations. The panel walks you through a one-time setup with your own Google Cloud OAuth client.
PageSpeed Insights: Add a free API key as PAGESPEED_API_KEY in the .env file in your Scrawly folder to get Core Web Vitals.

Scrawly works fine with none of these. They only add extra data.

Tutorial

Scrawly tutorial: your first audit, screen by screen

This walkthrough follows one audit from start to report: start a quick audit, let Scrawly pick the settings, watch the crawl, read the dashboard, fix the top issues, tune titles for Google and AI, check AI search readiness and share the report.

Illustrations of Scrawly screens using the placeholder site example.com. Labels are real; there are no real audit results.

Step 1
Start a quick audit
Click the Scrawly logo at the top of the left bar. Paste any web address and click Analyze. You do not need to connect the site or prove you own it.
Tip: run your first audit on a site you know well, so you can judge the findings.
Step 2
Let Scrawly pick the settings
Scrawly fingerprints the platform first, shows how confident it is and applies a matching preset. A WordPress or WooCommerce preset, for example, skips cart, checkout and admin URLs that waste crawl budget.
You can still change the page limit, JavaScript rendering, robots.txt rules and sitemap discovery before you start.
Step 3
Watch the crawl live
The live console shows progress, pages per second and the URLs being crawled right now. It stays visible on every screen, so you can keep working while the audit runs.
Scrawly only reads your pages. Nothing on your site changes.
Step 4
Read the dashboard
The dashboard opens with a site health score from 0 to 100, pages crawled, the indexable share, broken pages and total issues, plus charts for severity, response codes and crawl depth.
Use the score to track progress between crawls, not as a grade on its own.
Step 5
Fix what matters first
Issues & Audits sorts findings by impact. Open one to see why it matters, the recommended fix and the evidence. Click Pages for every affected URL, or Copy prompt for a fix prompt written for your platform.
On a connected WordPress site, supported fixes apply from the same row, with an Undo button.
Step 6
Tune titles for Google and AI
The Titles view measures titles and descriptions in pixels, because Google cuts snippets by width, not characters. It also flags descriptions under 120 characters as too thin for AI systems.
With your own AI key, Write with AI drafts a stronger title and an entity-rich description from the live page.
Step 7
Check AI search readiness
The AI crawler matrix tests 10 bots against your robots.txt and with a live fetch, so you can see a silent CDN or firewall block. The Agentic tab scores how ready the site is for AI agents, from Level 0 to Level 5.
Blocking AI search crawlers is rated High, because it can keep you out of AI answers.
Scrawly · example.comIllustration
DashboardIssues & AuditsTitlesAgenticReports
AI crawler access
OAI-SearchBotrobots.txtlive fetch
PerplexityBotrobots.txtlive fetch
ClaudeBotrobots.txtlive fetch
GPTBotrobots.txtlive fetch
PerplexityBot: allowed by robots.txt, blocked live
Agent readinessL0L1L2L3L4L5
Step 8
Share the report
In Reports, open the HTML report in your browser or click Print / Save PDF. Add your agency name, colour, logo and contact line first if the report is for a client.
Run the next crawl after your fixes ship, then compare the two crawls to show what improved.

WordPress

Safe, reversible WordPress fixes

On WordPress, Scrawly can fix titles, meta descriptions, canonicals and redirects for you through the small Scrawly Connector plugin. It saves a rollback snapshot before every change, asks before touching the live site, and never auto-applies higher-risk fixes.

What it can fix

Page titles and meta descriptions, written with your AI provider
Canonical URLs, computed with no AI needed
301 redirects
Suggestions only for image alt text, H1s and anchor text, which you apply in WordPress

How it stays safe

A snapshot of the old value is saved first, then the change is written. Every fix can be undone from the issue's Fixed button.
Live changes always ask first: fix all pages, just the first page, or cancel.
Report-only issues are never applied automatically.
The plugin only writes title, meta description, canonical and robots fields, mapped to Yoast or Rank Math. It never edits files, post content or other settings.
If Yoast and Rank Math are both active, it refuses to write.
A per-site "Allow fixes" switch keeps the connection read-only whenever you want.

How to connect a WordPress site

Add the site
In Scrawly, go to Clients, click + New Client and enter the site name and URL.
Download the plugin
Choose WordPress Connector as the connection method and download the plugin .zip.
Install it in WordPress
In wp-admin, go to Plugins, Add New, Upload Plugin. Choose the .zip, click Install Now, then Activate.
Copy the Connection Key
Open the new Scrawly menu in WordPress and copy the Connection Key.
Test the connection
Paste the key into Scrawly and click Test connection.

The Connector is a single file of about 6 KB with no dependencies. It needs WordPress 5.6 or newer and PHP 7.4 or newer, and it does not need Application Passwords. Like WordPress itself, the plugin is licensed GPL-2.0-or-later.

Not on WordPress? Shopify, Wix, Webflow, Next.js and custom sites get the full read-only audit, plus fix prompts for your developer or coding agent.

Read the WordPress Connector guide

Model Context Protocol

MCP server: use your audits from AI assistants

Scrawly includes an MCP (Model Context Protocol) server, so assistants like Claude Desktop, Claude Code or Cursor can read your audits, explain findings and compare crawls. With your confirmation, they can also apply WordPress fixes.

11 tools your assistant can call

list_crawls: Your recent audits with their IDs
get_issues: Findings for an audit, optionally by severity or fix tier
get_page_detail: Everything Scrawly recorded about one page
propose_fix: The recommended fix and affected pages (writes nothing)
diff_crawls: What was resolved, what is new and what persists
check_ai_access: Which AI crawlers a site allows or blocks
run_lighthouse: PageSpeed Insights data for a URL (needs a PageSpeed key)
generate_report: The audit report as HTML or PDF
apply_fix: Apply a fix on WordPress (requires confirm=True)
create_redirect: Add a 301 redirect on WordPress (requires confirm=True)
revert: Undo a fix using its stored rollback snapshot

Set it up

Add Scrawly to your assistant's MCP configuration. Replace /path/to/Scrawly with your install folder (by default, Scrawly in your home folder) and point SCRAWLY_DB_PATH at your audit database. On Windows, the command is C:\Users\you\Scrawly\.venv\Scripts\fastmcp.exe.

MCP config (JSON)

{
  "mcpServers": {
    "scrawly": {
      "command": "/path/to/Scrawly/.venv/bin/fastmcp",
      "args": ["run", "/path/to/Scrawly/src/sentinelseo/mcp/server.py"],
      "env": {
        "SCRAWLY_DB_PATH": "/Users/you/Library/Application Support/Scrawly/scrawly.db"
      }
    }
  }
}

macOS~/Library/Application Support/Scrawly/scrawly.db
Windows%APPDATA%\Scrawly\scrawly.db
Linux~/.local/share/scrawly/scrawly.db

Audits run in the app, and the MCP server works with the audits stored on your computer. Nothing is written without an explicit confirm=True, a rollback snapshot is stored before every write, and manual-only findings are never auto-fixed. Write tools need WordPress credentials in the env block, so leave them out for read-only use.

Read the MCP guide

Private by design

Privacy: your audits stay on your computer

Scrawly has no accounts and no analytics. Your crawls, findings and reports are stored on your own computer, and the app only talks to the sites you audit and the services you choose.

Scrawly only connects to

The websites you choose to audit
GitHub, to check for updates (turn this off with SCRAWLY_AUTO_UPDATE=0)
Services you set up yourself, such as PageSpeed Insights, Search Console or your AI provider
The feedback service, only when you click Send on a feedback note

How the app protects you

The local engine listens only on 127.0.0.1 and only accepts requests from the Scrawly window, so websites you visit cannot control it.
Crawled content such as page titles and URLs is always shown as text and never run as code, in the app and in reports.
API keys you add are stored on your computer and sent only to the service they belong to.
The Google connection asks for read-only access, and its token is encrypted on your machine.

Where your data lives

macOS~/Library/Application Support/Scrawly
Windows%APPDATA%\Scrawly
Linux~/.local/share/scrawly

Set SCRAWLY_DATA_DIR to keep your data somewhere else, such as a portable drive.

One opt-in exception: the spelling and grammar check is off by default. If you turn it on with the free public LanguageTool service, page text is sent to languagetool.org. The local LanguageTool option keeps text on your machine.

Who it is for

Who is Scrawly for?

Scrawly is for anyone who needs a deep technical SEO audit without a license fee: agencies, in-house SEO teams, freelancers and developers. It is also a practical GEO audit tool for anyone getting a site ready for AI search.

Agencies
White-label HTML and PDF reports, a Clients list for every site, crawl comparisons that show progress, and scheduled audits that catch regressions.
In-house SEO teams
Search Console and GA4 data next to crawl data, release checks with crawl comparison, and a CI-friendly script that fails on new Critical or High issues.
Freelancers
No license to buy and no page cap, so you can audit prospect and client sites as often as you need, then hand over a branded report.
Developers
Fix prompts for your stack, a response-vs-render diff for JavaScript sites, an MCP server, and open Python source you can read and extend.

Updates and support

Updates, troubleshooting and uninstalling

Scrawly updates itself. Each time it opens, it checks GitHub for a new version and installs it once the project's automated tests have passed. To uninstall, delete the Scrawly folder and its app shortcut.

Automatic updates

Turn off automatic updates with SCRAWLY_AUTO_UPDATE=0 in the .env file in your Scrawly folder.
Offline? Nothing updates and your installed version keeps working. A failed update never blocks launch.
If GitHub cannot confirm the test results, for example because of rate limits, the update goes ahead and is logged so your install never gets stuck.
If you edited files in the Scrawly folder, automatic updates pause. Run git pull yourself.
After a system Python upgrade, Scrawly rebuilds its private environment on the next launch.

Troubleshooting

"backend not installed": Run python3 scripts/install.py in the Scrawly folder.
A PDF will not save: The first PDF may need to download a browser component. Check your connection and try again.
"Scrawly's engine isn't responding": Close and reopen Scrawly. On macOS and Linux, check scrawly.log in the data folder.
An update did not apply: You may have local changes. Run git status, then git pull.
The window does not open on Linux: Install the Linux window libraries listed in the install steps above.

How to uninstall Scrawly

Delete the Scrawly folder
By default it is the Scrawly folder in your home folder.
Remove the app entry
Delete ~/Applications/Scrawly.app on macOS, the Start Menu and Desktop shortcuts on Windows, or ~/.local/share/applications/scrawly.desktop on Linux.
Remove your audits (optional)
To delete your audits too, remove the data folder listed in the privacy section.

Community

Open source and contributing

Scrawly is open source under the MIT License, so you can read, audit, change and share the code. The fastest way to help is to send feedback from inside the app or open an issue on GitHub.

Send feedback in the app
Click the speech-bubble icon at the bottom of the left bar. No GitHub account is needed, and notes go straight to the maintainer.
Report a bug
Open a GitHub issue with your operating system, your Scrawly version (under About, the "i" icon) and scrawly.log. Never include passwords or API keys.
Contribute code
Fork, branch and pass the CI checks (ruff, mypy, pytest and the UI build) before you open a pull request. Development needs Python 3.11+, Node.js 18+ and git.
Report security issues privately
Follow the steps in SECURITY.md and do not open a public issue until a fix ships.

Ground rules: only crawl and audit websites you own or are allowed to test, keep user-facing text plain and friendly, and propose new checks in an issue first. Contributions are MIT licensed.

Star Scrawly on GitHub Contributing guide Open an issue Security policy

If Scrawly helps you, a star on GitHub helps others find it.

The maker

Made by Ilias Sami

Scrawly was created by Ilias Sami, an SEO consultant and the silent AEO & GEO infrastructure partner for digital agencies, as a free contribution to the SEO community.

The tool stays free, with every feature open to everyone. If your agency would rather have the audit done for you, Ilias also delivers white-label technical SEO, AEO and GEO audits that your team can present as its own.

About Ilias Sami Want the audit done for you? Book a call

FAQ

Scrawly FAQ

Quick answers to the questions people ask most about Scrawly: cost, limits, privacy, AI search and how it compares with Screaming Frog.

Is Scrawly really free?

Direct answer: Yes. Scrawly is free and open source under the MIT License, with no account, no subscription and no usage limit. Audits run on your own computer, so there is no per-crawl cost. Optional services you connect, like an AI provider or a paid backlink tool, bill you directly if they charge.

Is there a page or URL limit?

Direct answer: There is no fixed cap. You choose the page limit for each audit, and how far you can go depends on your computer and the website. The quick audit window has a built-in ceiling of 100,000 pages per audit, which is a sanity limit, not a quota.

How does Scrawly compare to Screaming Frog?

Direct answer: Screaming Frog SEO Spider is an excellent paid crawler with a capped free tier. Scrawly is free and open source with no page cap and no license key, and it adds a dedicated set of AI search and agentic-web checks plus safe, reversible fixes on WordPress. Scrawly is independent and not affiliated with Screaming Frog Ltd.

Do I need an account to use Scrawly?

Direct answer: No. There is no sign-up, login or approval. Install Scrawly, open it and start an audit.

What are the system requirements?

Direct answer: Scrawly runs on macOS, Windows and Linux. It needs Python 3.11 or newer and git, plus a one-time download of about 150 MB for the crawler browser. Node.js is not needed. The installer adds git and Python for you where it safely can.

Does my data leave my computer?

Direct answer: Your audits do not. Crawls, findings and reports are stored on your computer only, and Scrawly has no accounts and no analytics. It connects only to the sites you audit, GitHub for update checks, services you set up yourself such as PageSpeed Insights, Search Console or your AI provider, and the feedback service when you click Send.

Can Scrawly render JavaScript?

Direct answer: Yes. Scrawly renders pages in a real Chromium browser through Playwright, with a desktop pass and a mobile pass. It compares the raw HTML response with the rendered page, so it can flag content, links, titles or canonicals that only exist after JavaScript runs.

Can Scrawly audit my site for ChatGPT, Perplexity and Google AI Overviews?

Direct answer: It audits the technical side. Scrawly checks whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your pages, whether a CDN or firewall silently blocks them, your llms.txt, meta descriptions as AI context, structured data and agent readiness. It does not track rankings or citations in AI answers.

How do I update or uninstall Scrawly?

Direct answer: Updates are automatic: Scrawly checks GitHub each time it opens and installs new versions once their tests pass. You can also run the install command again. To uninstall, delete the Scrawly folder in your home folder, remove the app shortcut, and delete the data folder if you want your audits gone too.

How do I report a bug or contribute?

Direct answer: Click the speech-bubble icon at the bottom of the left bar in Scrawly to send feedback, with no GitHub account needed. You can also open an issue or a pull request on GitHub. Include your operating system, Scrawly version and scrawly.log, but never passwords or API keys.

Does Scrawly work on sites that are not built with WordPress?

Direct answer: Yes. Every site gets the full read-only audit, including Shopify, Wix, Webflow, Next.js and custom builds. Only the one-click fixes are WordPress-specific. For other platforms, Scrawly gives you a fix prompt that names the right file on your stack.

Can Scrawly fix SEO issues for me?

Direct answer: On WordPress, yes. With the Scrawly Connector plugin it can fix titles, meta descriptions, canonicals and redirects, saving a rollback snapshot first and asking before any live change. Higher-risk issues are never applied automatically.

Can I white-label Scrawly reports for clients?

Direct answer: Yes. Set your agency name, accent color, logo and contact line, and every HTML and PDF report uses your branding.

Does Scrawly connect to Google Search Console, GA4 and PageSpeed Insights?

Direct answer: Yes, if you want it to. One read-only Google connection adds Search Console and GA4 data for each page, using your own Google Cloud OAuth client. A free PageSpeed Insights API key adds Lighthouse lab data and CrUX field data for Core Web Vitals.

Who made Scrawly?

Direct answer: Ilias Sami, an SEO consultant and the silent AEO & GEO infrastructure partner for digital agencies, created Scrawly as a free contribution to the SEO community.

Still have a question? Ask on GitHub or use Send feedback inside the app.

Run your first free audit today

Install Scrawly with one command and audit any website for Google and AI search. No account, no page cap, MIT licensed.

Download free on GitHub Read the install steps

Releases Changelog Desktop guide

$ curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash

Paste into Terminal. Scrawly installs itself and appears in your Applications.

github.com/IliasSami/scrawly-seo-crawlerMIT License

Scrawly: the free SEO crawler built for Google and AI search

What is in version 1.0.0

The 23 check areas

Crawlability and indexability

Response codes and redirects

Canonicalization

Titles, meta and headings

Content quality

Internal linking and architecture

External and outbound links

Images and media

Hreflang and international

Structured data and schema

XML sitemaps

Robots directives (file level)

Security

Performance and Core Web Vitals

Mobile

JavaScript rendering

Accessibility

AI search readiness (GEO)

Agentic web and llms.txt

URL structure and hygiene

Pagination

WordPress and HTML validation

Analytics, tracking and config

Findings you can act on, ranked by impact

Site health score and internal authority

AI crawler access matrix

Training bots and search bots

llms.txt checks

Meta descriptions as AI context

Answers, entities and freshness

Agentic web readiness audit

JavaScript rendering

Site Explorer

Seven site graph layouts

Custom extraction

Platform presets and profiles

Compare crawls

Scheduled audits

Search Console and GA4

PageSpeed Insights and CrUX

White-label reports

AI writer and audit assistant

Fix prompts for coding agents

What you need

What the installer does

Install on macOS

Install on Windows

Install on Linux

Manual install

Your first audit, step by step

Keep audit history for your sites

Optional connections

Start a quick audit

Let Scrawly pick the settings

Watch the crawl live

Read the dashboard

Fix what matters first

Tune titles for Google and AI

Check AI search readiness

Share the report

What it can fix

How it stays safe

How to connect a WordPress site

11 tools your assistant can call

Set it up

Scrawly only connects to

How the app protects you

Where your data lives

Agencies

In-house SEO teams

Freelancers

Developers

Automatic updates

Troubleshooting

How to uninstall Scrawly

Send feedback in the app

Report a bug