# Scrawly: the free SEO crawler built for Google and AI search

> Scrawly is a free, open-source SEO crawler for macOS, Windows and Linux. Run 250 technical SEO and GEO checks with no account and no page cap.

Scrawly is a free, open-source desktop SEO crawler and technical site audit tool for macOS, Windows and Linux. It crawls your website, runs 250 technical SEO, GEO and agentic-web checks on every page, and explains in plain language what to fix. There is no account, no page cap and no subscription.

MIT licensed · Open source · No account · No page cap · macOS, Windows, Linux · 250 checks in 23 areas

- Download (free, MIT): https://github.com/IliasSami/scrawly-seo-crawler
- Product Hunt: https://www.producthunt.com/products/scrawly
- Releases: https://github.com/IliasSami/scrawly-seo-crawler/releases
- Changelog: https://github.com/IliasSami/scrawly-seo-crawler/blob/main/CHANGELOG.md

Install on macOS or Linux:

```bash
curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash
```

Install on Windows (PowerShell):

```powershell
irm https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.ps1 | iex
```

## What is Scrawly?

Scrawly is a free desktop app that audits any website for the technical issues that affect Google rankings and AI search visibility in tools like ChatGPT, Perplexity and Google AI Overviews. It runs on your own computer, with no account and no page limit.

Think of it as a site crawler and a patient SEO teacher in one window. Scrawly visits your pages, renders JavaScript in a real Chromium browser, and checks every page against 250 rules. Each finding tells you why it matters, how to fix it and exactly which pages are affected.

Ilias Sami built Scrawly and released it for the SEO community under the MIT License. The engine is written in Python and opens in a native window on macOS, Windows and Linux. Your audits are saved in a local database on your machine, not in someone else's cloud.

### What is in version 1.0.0

Version 1.0.0, released on October 6, 2026, is the first public release of Scrawly Free. It includes:

- 250 checks across 23 areas, with plain-language fixes
- HTML and PDF reports, saved wherever you choose
- Safe WordPress fixes with a rollback snapshot before every change
- One-line installers that add Scrawly to your apps
- Automatic updates that only install versions whose tests passed
- Send feedback from inside the app, with an email or GitHub fallback

## What does Scrawly check? 250 checks in 23 areas

Scrawly runs 250 checks grouped into 23 areas, from crawlability and redirects to Core Web Vitals, structured data and AI search readiness. Each finding explains why it matters, how to fix it and which pages are affected.

| Area | Checks | Examples |
|---|---|---|
| Crawlability and indexability | 15 | robots.txt blocks, noindex in meta tags and headers, conflicting index signals, orphan URLs and pages more than 4 clicks deep. |
| Response codes and redirects | 14 | 4xx and 5xx pages, soft 404s, broken internal links with their sources, redirect chains and loops, and 302s that should be 301s. |
| Canonicalization | 11 | Missing, conflicting or relative canonicals, canonicals that point at redirects or noindex pages, and canonicals changed by JavaScript. |
| Titles, meta and headings | 21 | Missing and duplicate titles, pixel-width truncation, meta descriptions too thin for AI, H1 problems, heading order and Open Graph tags. |
| Content quality | 12 | Exact and near-duplicate pages, thin content, keyword cannibalization, placeholder text, low text-to-HTML ratio and missing dates. |
| Internal linking and architecture | 12 | Orphan pages, pages with one inlink, generic anchors like "click here", links to redirects and links that only appear after JavaScript runs. |
| External and outbound links | 6 | Broken and redirecting outbound links, plain HTTP links, missing rel sponsored or ugc, and target=_blank without noopener. |
| Images and media | 11 | Missing or overlong alt text, images over 100 KB, missing width and height, broken sources, no WebP or AVIF, and lazy-loading issues. |
| Hreflang and international | 8 | Missing return tags, invalid language codes, x-default, canonical conflicts and mismatches between HTML, headers and sitemaps. |
| Structured data and schema | 15 | Invalid JSON-LD, missing required or recommended properties, @id conflicts, and Organization, Breadcrumb, Article, Product and FAQ markup. |
| XML sitemaps | 10 | Missing or broken sitemaps, noindexed or redirected URLs inside them, missing indexable pages, size limits, invalid XML and stale lastmod. |
| Robots directives (file level) | 7 | "Disallow: /" on the whole site, blocked CSS and JS paths, wildcard mistakes, the wrong content type, a missing Sitemap line and a high crawl-delay. |
| Security | 10 | HTTPS, mixed content, HSTS, expiring TLS certificates, security headers, server version leaks and exposed WordPress versions. |
| Performance and Core Web Vitals | 14 | LCP over 2.5 seconds, INP over 200 ms from field data, CLS over 0.1, slow TTFB, render-blocking files, page weight and DOM size. |
| Mobile | 7 | Viewport, small tap targets, sideways scrolling, small fonts, mobile and desktop content parity, and intrusive pop-ups. |
| JavaScript rendering | 6 | Content that only exists after rendering, and titles, canonicals or robots tags that differ between the raw response and the render. |
| Accessibility | 9 | A WCAG subset that helps both SEO and AI agents: accessible names, ARIA, form labels, color contrast, page language and landmarks. |
| AI search readiness (GEO) | 13 | AI crawlers blocked in robots.txt or silently by a CDN or firewall, client-side-only content, weak answer structure and missing entity schema. |
| Agentic web and llms.txt | 13 | llms.txt presence and quality, the accessibility tree AI agents read, layout shifts after load, and whether agents can finish key flows. |
| URL structure and hygiene | 9 | Uppercase letters, unsafe characters, double slashes, URLs over 115 characters, session parameters, underscores and trailing-slash mix-ups. |
| Pagination | 5 | Noindexed paginated pages, page 2 and later canonicalized to page 1, and infinite scroll with no paginated URLs. |
| WordPress and HTML validation | 17 | Indexed attachment, tag and search pages, Yoast and Rank Math both active, indexable staging sites, plus DOCTYPE, lang and charset checks. |
| Analytics, tracking and config | 5 | GA4 and Google Tag Manager tags, Search Console verification, the favicon, the 404 template and multiple homepage versions. |

Some checks need extra data to run, such as JavaScript rendering, a PageSpeed Insights key, Search Console access or a WordPress site. Counts are unique check IDs in the Scrawly source code.

### Findings you can act on, ranked by impact

Every finding has a severity: Critical, High, Medium, Low or Info. Scrawly sorts by impact, so the problems that hurt the most pages come first.

Severity is coverage-aware. If at least 20% of your live pages share a problem, it moves up one level, and at 40% it moves up two. It needs at least 5 affected URLs, it stops at High, and it never turns a problem into a Critical on its own.

Precedence rules stop double counting. A page that returns an error is not also flagged for a missing title, and an exact duplicate is not also reported as a near duplicate.

### Site health score and internal authority

Each audit gets a site health score from 0 to 100. It weighs the share of pages free of Critical or High issues, the share that can be indexed and the share that is not broken, minus a few points for serious site-wide problems. The dashboard and the report use the same model.

Scrawly also computes Authority, a PageRank-style score from 0 to 100 built from your own internal links. It shows which pages your site structure really favors, not just raw inlink counts.

## How does Scrawly audit for AI search?

Scrawly checks whether AI crawlers can reach your pages, whether your llms.txt and meta descriptions give AI systems enough context, and how ready your site is for AI agents. It runs these GEO checks in the same crawl as the classic technical SEO audit.

### AI crawler access matrix

Scrawly tests 10 user agents. It reads your robots.txt rules for each one, then fetches your start page as that bot. If robots.txt says yes but the live fetch fails, Scrawly flags a silent block by your CDN or firewall, which a robots.txt checker alone would miss.

User agents tested: Googlebot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, CCBot, Google-Extended, bingbot.

### Training bots and search bots

Blocking training crawlers such as GPTBot, CCBot, ClaudeBot or Google-Extended is a policy choice, so Scrawly reports it as Info. Blocking AI search crawlers such as OAI-SearchBot, Claude-SearchBot or PerplexityBot is rated High, because it can keep your pages out of AI answers.

### llms.txt checks

Scrawly checks whether /llms.txt exists, whether it has a title and real links, and whether it has turned into a link dump. It flags files with more than 800 links and suggests 20 to 50 high-value links instead. It also notes when the optional llms-full.txt is missing.

### Meta descriptions as AI context

A meta description is often the first thing an AI system reads about a page. Scrawly flags descriptions under 120 characters as too thin. With your own AI key, it can draft an entity-rich description from the page's own content.

### Answers, entities and freshness

GEO checks also look for a direct answer before the context, a clear heading hierarchy, author and freshness signals, Organization and WebSite schema, and main content that only appears after JavaScript runs.

### Agentic web readiness audit

A separate audit asks a simple question: can an AI agent discover, read, sign in to and buy from this site? It runs 21 probes in 6 categories against your site root, at a fixed cost of about 25 requests, whether the site has 10 pages or 100,000.

Probes cover robots.txt AI bot rules, sitemaps, Link headers, Markdown negotiation, llms.txt, Content Signals, OAuth discovery, MCP Server Cards, A2A Agent Cards, Agent Skills and agentic commerce protocols such as x402 and ACP. Commerce probes only count when the site really sells something, so a blog is never marked down for lacking a payment rail.

Every probe records the exact requests it made and what it concluded, so each verdict can be checked. The score maps to six readiness levels:

- Level 0: Not agent-ready
- Level 1: Basic web presence
- Level 2: Agent-discoverable
- Level 3: Agent-readable
- Level 4: Agent-operable
- Level 5: Agent-native

Scrawly does not track rankings or citations in AI answers. It audits the technical signals that decide whether AI systems can reach, read and understand your pages.

## Everything you expect from a desktop SEO crawler

Scrawly covers the core jobs of a desktop crawler: JavaScript rendering, a filterable Site Explorer, custom extraction, site graphs, crawl comparison and white-label reports. On top of that it adds Google data, fix prompts for coding agents and an MCP server.

- **JavaScript rendering:** Pages render in Playwright Chromium, first on desktop and then as a smartphone. Scrawly compares the raw response with the rendered page, so you see what JavaScript adds, removes or changes.
- **Site Explorer:** A spreadsheet-style view with tabs for Internal, Response Codes, Page Titles, Meta Description, H1, Directives, Canonicals, Content, Analytics, PDF and Custom. Every tab has filters with live counts.
- **Seven site graph layouts:** Includes a 3D architecture map, a crawl-depth tree, content clusters and redirect chains. Size nodes by authority, inlinks or words, and color them by status, issues or depth.
- **Custom extraction:** Pull any value from every page with CSS selectors, XPath or regex, or count regex matches across the raw HTML.
- **Platform presets and profiles:** Scrawly detects WordPress, Shopify, Wix, Squarespace, Webflow, Framer, Next.js, React, Laravel, Drupal, Joomla, Magento or Ghost and skips admin pages, carts and other crawl traps. Four built-in profiles cover quick checks to full technical audits.
- **Compare crawls:** See resolved issues, new issues and regressions between two audits, plus which URLs appeared, vanished or changed their title, canonical, status or indexability.
- **Scheduled audits:** A small script runs a new audit, compares it with the last one and exits with code 2 if new Critical or High issues appear, so cron, Task Scheduler or CI can alert you. The app must be open.
- **Search Console and GA4:** One read-only Google connection adds clicks, impressions, CTR and position from Google Search Console, plus sessions, engaged sessions, views and conversions from GA4.
- **PageSpeed Insights and CrUX:** Add a free PageSpeed Insights API key to get Lighthouse lab scores and real-user Core Web Vitals from CrUX, when Google has enough field data for the URL.
- **White-label reports:** A polished, self-contained HTML report with your agency name, color, logo and contact line. Save it as a PDF wherever you like, or export selected rows to CSV.
- **AI writer and audit assistant:** Bring your own key for Anthropic (Claude), OpenAI, DeepSeek, OpenRouter or any OpenAI-compatible service. Draft titles and meta descriptions, or ask an assistant that only reasons over the current audit.
- **Fix prompts for coding agents:** Every finding comes with a copy-paste prompt that names the right file on your stack (WordPress, Next.js, Shopify, Astro, Hugo, Django, Rails, Cloudflare and more), with acceptance criteria and a check to verify the fix.

Also included: list mode for an exact set of URLs, crawling behind a login form, PDF crawling, on-demand desktop and smartphone screenshots, and checks on the CSS, JS and media files each page loads. Crawling is polite by default: robots.txt is respected, and 429 or 5xx responses are retried with backoff and then recorded as findings instead of being dropped.

## Scrawly vs Screaming Frog and Sitebulb

Screaming Frog SEO Spider and Sitebulb are excellent tools. Scrawly is a free, open-source option for people who want a deep technical audit without a license, plus a dedicated set of checks for AI search.

| | Screaming Frog / Sitebulb | Scrawly |
|---|---|---|
| Price | Paid license or subscription (limited free tier or trial) | **Free and open source (MIT)** |
| Account or license key | Needed for full use | **None** |
| Page limit | The free tier is capped | **No cap: you set the limit** |
| AI search (GEO) and agentic-web checks | Some, depending on version | **A dedicated set: AI crawler access, llms.txt, citation and agent readiness** |
| Fixes | Report only | **Safe, reversible fixes on WordPress** |
| Source code | Closed | **Open: read, audit, contribute** |

Scrawly is independent and not affiliated with Screaming Frog Ltd or Sitebulb. This comparison follows the Scrawly README. Check each vendor's website for current pricing and features.

Already happy with a paid crawler? Scrawly can still run next to it as a free second opinion, especially for AI search and agent readiness.

## How to install Scrawly on macOS, Windows and Linux

Install Scrawly with one command: paste the curl line into Terminal on macOS or Linux, or the irm line into PowerShell on Windows. The installer sets everything up and adds Scrawly to your apps, with nothing to sign up for.

### What you need

- Python 3.11 or newer (3.11, 3.12 and 3.13 are supported)
- git, which powers automatic updates
- About 150 MB for a one-time download of the crawler browser
- Node.js is not needed, because releases ship the interface prebuilt

Missing git or Python? The installer adds them for you where it safely can, with Homebrew on macOS or winget on Windows. Otherwise it prints the exact command to run.

### Install on macOS

1. **Open Terminal.** Find Terminal in Applications, then Utilities, or search for it with Spotlight.
2. **Run the install command.** Paste this line and press Return.

```bash
curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash
```

3. **Accept the Command Line Tools if asked.** If git is missing, macOS offers to install the Command Line Tools. Accept, then run the command again.
4. **Open Scrawly.** Launch Scrawly from Launchpad, Spotlight or the Applications folder in your home folder. A window opens and you are ready.

### Install on Windows

1. **Open PowerShell.** Open PowerShell from the Start Menu. Windows PowerShell 5.1 works.
2. **Run the install command.** Paste this line and press Enter.

```powershell
irm https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.ps1 | iex
```

3. **Let winget add anything missing.** If git or Python is missing, the installer uses winget to add them. If you install Python yourself from python.org, tick "Add python.exe to PATH".
4. **Open Scrawly.** Launch Scrawly from the Start Menu or the Desktop shortcut.

### Install on Linux

1. **Open a terminal.** Use the terminal app that comes with your distribution.
2. **Run the install command.** Paste this line and press Enter.

```bash
curl -fsSL https://raw.githubusercontent.com/IliasSami/scrawly-seo-crawler/main/install.sh | bash
```

3. **Add git or Python if needed.** If the installer reports that git or Python is missing, install them with your package manager. On Ubuntu or Debian:

```bash
sudo apt install git python3.12 python3.12-venv
```

4. **Open Scrawly.** Launch Scrawly from your app menu, or run ./scrawly.sh in the Scrawly folder. If the window does not open on Ubuntu or Debian, install these libraries:

```bash
sudo apt install libxkbcommon-x11-0 libxcb-cursor0 libnss3 libgbm1
```


### Manual install

1. **Install the prerequisites.** Make sure Python 3.11 or newer and git are installed.
2. **Clone the repo and run the installer.** Run these three commands in a terminal.

```bash
git clone https://github.com/IliasSami/scrawly-seo-crawler.git Scrawly
cd Scrawly
python3 scripts/install.py
```

   On Windows, run py scripts\install.py as the last line instead.
3. **Skip shortcuts if you prefer.** Set SCRAWLY_NO_SHORTCUTS=1 before running the installer if you do not want app shortcuts.
4. **Open Scrawly.** Launch it from your apps, or use the launcher in the Scrawly folder: Scrawly.command on macOS, Scrawly.bat on Windows or scrawly.sh on Linux.

### What the installer does

1. Checks for git and Python 3.11 or newer, and installs them where it safely can.
2. Downloads Scrawly into a folder called Scrawly in your home folder. Set SCRAWLY_HOME to choose another folder.
3. Creates a private Python environment and installs the crawler browser (about 150 MB, once).
4. Adds Scrawly to Applications on macOS, the Start Menu and Desktop on Windows, or the app menu on Linux.

Running the same command again updates an existing install.

## How to use Scrawly, from first crawl to report

Open Scrawly, click the logo, paste a web address and start the audit. When it finishes, the dashboard opens, Issues & Audits shows what to fix first, and the Reports tab saves an HTML or PDF report.

1. **Open Scrawly.** A window opens and you are ready. There is no sign-in screen.
2. **Start a quick audit.** Click the Scrawly logo at the top of the left bar, paste any web address and click Analyze.
3. **Let Scrawly pick the settings.** It detects the platform, shows how confident it is and applies a matching preset. You can still change the pages to check, pages at a time, JavaScript loading, robots.txt rules and sitemap discovery.
4. **Start the audit.** Scrawly only reads your pages. Nothing on your site is changed.
5. **Watch the live console.** Progress, pages per second and a live feed of crawled URLs stay visible on every screen. You can keep using the app while the audit runs in the background.
6. **Read the dashboard.** See the site health score, pages crawled, the indexable share, broken pages and total issues, with charts for severity, response codes, crawl depth and more.
7. **Fix what matters first.** Issues & Audits sorts findings by impact. Open one to see why it matters, the recommended fix and the evidence, then click Pages to see every affected URL.
8. **Hand off the fix.** Copy prompt gives you a ready-made fix prompt for your site's platform. On a connected WordPress site, you can apply supported fixes from the same row.
9. **Dig deeper.** Use Site Explorer for raw data, Titles for pixel widths, Agentic for AI agent readiness and Visualizations for your site's structure.
10. **Share the report.** In Reports, open the HTML report in your browser or click Print / Save PDF. Add your agency name, color and logo first if the report is for a client.

### Keep audit history for your sites

A quick audit keeps only the latest run. To keep history, compare crawls and run scheduled audits, add the site under Clients with + New Client, then run the crawl wizard: Target, Scope (pick a crawl profile) and Review & Launch.

### Optional connections

- **AI provider:** Add your own key under Settings for the AI writer and the audit assistant. The key is stored on your computer.
- **Google Search Console and GA4:** Connect both under Settings, then Integrations. The panel walks you through a one-time setup with your own Google Cloud OAuth client.
- **PageSpeed Insights:** Add a free API key as PAGESPEED_API_KEY in the .env file in your Scrawly folder to get Core Web Vitals.

Scrawly works fine with none of these. They only add extra data.

## Scrawly tutorial: your first audit, screen by screen

This walkthrough follows one audit from start to report: start a quick audit, let Scrawly pick the settings, watch the crawl, read the dashboard, fix the top issues, tune titles for Google and AI, check AI search readiness and share the report.

1. **Start a quick audit.** Click the Scrawly logo at the top of the left bar. Paste any web address and click Analyze. You do not need to connect the site or prove you own it. Tip: run your first audit on a site you know well, so you can judge the findings.
2. **Let Scrawly pick the settings.** Scrawly fingerprints the platform first, shows how confident it is and applies a matching preset. A WordPress or WooCommerce preset, for example, skips cart, checkout and admin URLs that waste crawl budget. You can still change the page limit, JavaScript rendering, robots.txt rules and sitemap discovery before you start.
3. **Watch the crawl live.** The live console shows progress, pages per second and the URLs being crawled right now. It stays visible on every screen, so you can keep working while the audit runs. Scrawly only reads your pages. Nothing on your site changes.
4. **Read the dashboard.** The dashboard opens with a site health score from 0 to 100, pages crawled, the indexable share, broken pages and total issues, plus charts for severity, response codes and crawl depth. Use the score to track progress between crawls, not as a grade on its own.
5. **Fix what matters first.** Issues & Audits sorts findings by impact. Open one to see why it matters, the recommended fix and the evidence. Click Pages for every affected URL, or Copy prompt for a fix prompt written for your platform. On a connected WordPress site, supported fixes apply from the same row, with an Undo button.
6. **Tune titles for Google and AI.** The Titles view measures titles and descriptions in pixels, because Google cuts snippets by width, not characters. It also flags descriptions under 120 characters as too thin for AI systems. With your own AI key, Write with AI drafts a stronger title and an entity-rich description from the live page.
7. **Check AI search readiness.** The AI crawler matrix tests 10 bots against your robots.txt and with a live fetch, so you can see a silent CDN or firewall block. The Agentic tab scores how ready the site is for AI agents, from Level 0 to Level 5. Blocking AI search crawlers is rated High, because it can keep you out of AI answers.
8. **Share the report.** In Reports, open the HTML report in your browser or click Print / Save PDF. Add your agency name, colour, logo and contact line first if the report is for a client. Run the next crawl after your fixes ship, then compare the two crawls to show what improved.

## Learn more and go further

Scrawly finds the issues. These guides, free tools and services on iliassami.com help you understand them, check a single URL fast, or hand the whole fix to an expert.

### Guides

- [AI crawlers explained](https://iliassami.com/blog/ai-crawlers-explained)
- [The complete AI crawler list](https://iliassami.com/blog/complete-ai-crawler-list)
- [robots.txt mistakes that block AI bots](https://iliassami.com/blog/robots-txt-mistakes-block-ai-bots)
- [JavaScript rendering and AI crawlers](https://iliassami.com/blog/javascript-rendering-ai-crawlers)
- [Manual vs automated technical audits](https://iliassami.com/blog/manual-vs-automated-technical-audit)
- [The GEO guide](https://iliassami.com/geo-guide)

### Free tools for one URL

- [SEO audit checker](https://iliassami.com/tools/seo-audit)
- [robots.txt tester](https://iliassami.com/tools/robots-txt-tester)
- [llms.txt generator](https://iliassami.com/tools/llms-txt-generator)
- [JSON-LD validator](https://iliassami.com/tools/json-ld-validator)
- [Redirect chain checker](https://iliassami.com/tools/redirect-chain-checker)
- [All 89 free SEO tools](https://iliassami.com/tools)

### Done for you

- [Manual technical SEO audit](https://iliassami.com/services/manual-technical-seo-audit)
- [Agentic AI readiness audit](https://iliassami.com/services/agentic-ai-readiness-audit)
- [AI visibility audit](https://iliassami.com/services/ai-visibility-audit)
- [Answer engine optimization](https://iliassami.com/services/answer-engine-optimization)
- [White label SEO for agencies](https://iliassami.com/agencies)
- [SEO glossary](https://iliassami.com/seo-glossary)

## Safe, reversible WordPress fixes

On WordPress, Scrawly can fix titles, meta descriptions, canonicals and redirects for you through the small Scrawly Connector plugin. It saves a rollback snapshot before every change, asks before touching the live site, and never auto-applies higher-risk fixes.

### What it can fix

- Page titles and meta descriptions, written with your AI provider
- Canonical URLs, computed with no AI needed
- 301 redirects
- Suggestions only for image alt text, H1s and anchor text, which you apply in WordPress

### How it stays safe

- A snapshot of the old value is saved first, then the change is written. Every fix can be undone from the issue's Fixed button.
- Live changes always ask first: fix all pages, just the first page, or cancel.
- Report-only issues are never applied automatically.
- The plugin only writes title, meta description, canonical and robots fields, mapped to Yoast or Rank Math. It never edits files, post content or other settings.
- If Yoast and Rank Math are both active, it refuses to write.
- A per-site "Allow fixes" switch keeps the connection read-only whenever you want.

### How to connect a WordPress site

1. **Add the site.** In Scrawly, go to Clients, click + New Client and enter the site name and URL.
2. **Download the plugin.** Choose WordPress Connector as the connection method and download the plugin .zip.
3. **Install it in WordPress.** In wp-admin, go to Plugins, Add New, Upload Plugin. Choose the .zip, click Install Now, then Activate.
4. **Copy the Connection Key.** Open the new Scrawly menu in WordPress and copy the Connection Key.
5. **Test the connection.** Paste the key into Scrawly and click Test connection.

The Connector is a single file of about 6 KB with no dependencies. It needs WordPress 5.6 or newer and PHP 7.4 or newer, and it does not need Application Passwords. Like WordPress itself, the plugin is licensed GPL-2.0-or-later.

Not on WordPress? Shopify, Wix, Webflow, Next.js and custom sites get the full read-only audit, plus fix prompts for your developer or coding agent.

## MCP server: use your audits from AI assistants

Scrawly includes an MCP (Model Context Protocol) server, so assistants like Claude Desktop, Claude Code or Cursor can read your audits, explain findings and compare crawls. With your confirmation, they can also apply WordPress fixes.

Add Scrawly to your assistant's MCP configuration. Replace /path/to/Scrawly with your install folder (by default, Scrawly in your home folder) and point SCRAWLY_DB_PATH at your audit database. On Windows, the command is C:\Users\you\Scrawly\.venv\Scripts\fastmcp.exe.

```json
{
  "mcpServers": {
    "scrawly": {
      "command": "/path/to/Scrawly/.venv/bin/fastmcp",
      "args": ["run", "/path/to/Scrawly/src/sentinelseo/mcp/server.py"],
      "env": {
        "SCRAWLY_DB_PATH": "/Users/you/Library/Application Support/Scrawly/scrawly.db"
      }
    }
  }
}
```

- macOS: `~/Library/Application Support/Scrawly/scrawly.db`
- Windows: `%APPDATA%\Scrawly\scrawly.db`
- Linux: `~/.local/share/scrawly/scrawly.db`

### 11 tools your assistant can call

- `list_crawls`: Your recent audits with their IDs
- `get_issues`: Findings for an audit, optionally by severity or fix tier
- `get_page_detail`: Everything Scrawly recorded about one page
- `propose_fix`: The recommended fix and affected pages (writes nothing)
- `diff_crawls`: What was resolved, what is new and what persists
- `check_ai_access`: Which AI crawlers a site allows or blocks
- `run_lighthouse`: PageSpeed Insights data for a URL (needs a PageSpeed key)
- `generate_report`: The audit report as HTML or PDF
- `apply_fix`: Apply a fix on WordPress (requires confirm=True)
- `create_redirect`: Add a 301 redirect on WordPress (requires confirm=True)
- `revert`: Undo a fix using its stored rollback snapshot

Audits run in the app, and the MCP server works with the audits stored on your computer. Nothing is written without an explicit confirm=True, a rollback snapshot is stored before every write, and manual-only findings are never auto-fixed. Write tools need WordPress credentials in the env block, so leave them out for read-only use.

## Privacy: your audits stay on your computer

Scrawly has no accounts and no analytics. Your crawls, findings and reports are stored on your own computer, and the app only talks to the sites you audit and the services you choose.

### Scrawly only connects to

- The websites you choose to audit
- GitHub, to check for updates (turn this off with SCRAWLY_AUTO_UPDATE=0)
- Services you set up yourself, such as PageSpeed Insights, Search Console or your AI provider
- The feedback service, only when you click Send on a feedback note

### How the app protects you

- The local engine listens only on 127.0.0.1 and only accepts requests from the Scrawly window, so websites you visit cannot control it.
- Crawled content such as page titles and URLs is always shown as text and never run as code, in the app and in reports.
- API keys you add are stored on your computer and sent only to the service they belong to.
- The Google connection asks for read-only access, and its token is encrypted on your machine.

### Where your data lives

- macOS: `~/Library/Application Support/Scrawly`
- Windows: `%APPDATA%\Scrawly`
- Linux: `~/.local/share/scrawly`

Set SCRAWLY_DATA_DIR to keep your data somewhere else, such as a portable drive.

One opt-in exception: the spelling and grammar check is off by default. If you turn it on with the free public LanguageTool service, page text is sent to languagetool.org. The local LanguageTool option keeps text on your machine.

## Who is Scrawly for?

Scrawly is for anyone who needs a deep technical SEO audit without a license fee: agencies, in-house SEO teams, freelancers and developers. It is also a practical GEO audit tool for anyone getting a site ready for AI search.

- **Agencies:** White-label HTML and PDF reports, a Clients list for every site, crawl comparisons that show progress, and scheduled audits that catch regressions.
- **In-house SEO teams:** Search Console and GA4 data next to crawl data, release checks with crawl comparison, and a CI-friendly script that fails on new Critical or High issues.
- **Freelancers:** No license to buy and no page cap, so you can audit prospect and client sites as often as you need, then hand over a branded report.
- **Developers:** Fix prompts for your stack, a response-vs-render diff for JavaScript sites, an MCP server, and open Python source you can read and extend.

## Updates, troubleshooting and uninstalling

Scrawly updates itself. Each time it opens, it checks GitHub for a new version and installs it once the project's automated tests have passed. To uninstall, delete the Scrawly folder and its app shortcut.

### Automatic updates

- Turn off automatic updates with SCRAWLY_AUTO_UPDATE=0 in the .env file in your Scrawly folder.
- Offline? Nothing updates and your installed version keeps working. A failed update never blocks launch.
- If GitHub cannot confirm the test results, for example because of rate limits, the update goes ahead and is logged so your install never gets stuck.
- If you edited files in the Scrawly folder, automatic updates pause. Run git pull yourself.
- After a system Python upgrade, Scrawly rebuilds its private environment on the next launch.

### Troubleshooting

- **"backend not installed":** Run python3 scripts/install.py in the Scrawly folder.
- **A PDF will not save:** The first PDF may need to download a browser component. Check your connection and try again.
- **"Scrawly's engine isn't responding":** Close and reopen Scrawly. On macOS and Linux, check scrawly.log in the data folder.
- **An update did not apply:** You may have local changes. Run git status, then git pull.
- **The window does not open on Linux:** Install the Linux window libraries listed in the install steps above.

### How to uninstall Scrawly

1. **Delete the Scrawly folder.** By default it is the Scrawly folder in your home folder.
2. **Remove the app entry.** Delete ~/Applications/Scrawly.app on macOS, the Start Menu and Desktop shortcuts on Windows, or ~/.local/share/applications/scrawly.desktop on Linux.
3. **Remove your audits (optional).** To delete your audits too, remove the data folder listed in the privacy section.

## Open source and contributing

Scrawly is open source under the MIT License, so you can read, audit, change and share the code. The fastest way to help is to send feedback from inside the app or open an issue on GitHub.

- **Send feedback in the app:** Click the speech-bubble icon at the bottom of the left bar. No GitHub account is needed, and notes go straight to the maintainer.
- **Report a bug:** Open a GitHub issue with your operating system, your Scrawly version (under About, the "i" icon) and scrawly.log. Never include passwords or API keys.
- **Contribute code:** Fork, branch and pass the CI checks (ruff, mypy, pytest and the UI build) before you open a pull request. Development needs Python 3.11+, Node.js 18+ and git.
- **Report security issues privately:** Follow the steps in SECURITY.md and do not open a public issue until a fix ships.

Ground rules: only crawl and audit websites you own or are allowed to test, keep user-facing text plain and friendly, and propose new checks in an issue first. Contributions are MIT licensed.

If Scrawly helps you, a star on GitHub helps others find it. Source: https://github.com/IliasSami/scrawly-seo-crawler · Contributing: https://github.com/IliasSami/scrawly-seo-crawler/blob/main/CONTRIBUTING.md · Security: https://github.com/IliasSami/scrawly-seo-crawler/blob/main/SECURITY.md

## Made by Ilias Sami

Scrawly was created by Ilias Sami, an SEO consultant and the silent AEO & GEO infrastructure partner for digital agencies, as a free contribution to the SEO community.

The tool stays free, with every feature open to everyone. If your agency would rather have the audit done for you, Ilias also delivers white-label technical SEO, AEO and GEO audits that your team can present as its own. More about Ilias: https://iliassami.com/about · Book a call: https://iliassami.com/#booking

## Scrawly FAQ

**Is Scrawly really free?**

**Direct answer:** Yes. Scrawly is free and open source under the MIT License, with no account, no subscription and no usage limit. Audits run on your own computer, so there is no per-crawl cost. Optional services you connect, like an AI provider or a paid backlink tool, bill you directly if they charge.

**Is there a page or URL limit?**

**Direct answer:** There is no fixed cap. You choose the page limit for each audit, and how far you can go depends on your computer and the website. The quick audit window has a built-in ceiling of 100,000 pages per audit, which is a sanity limit, not a quota.

**How does Scrawly compare to Screaming Frog?**

**Direct answer:** Screaming Frog SEO Spider is an excellent paid crawler with a capped free tier. Scrawly is free and open source with no page cap and no license key, and it adds a dedicated set of AI search and agentic-web checks plus safe, reversible fixes on WordPress. Scrawly is independent and not affiliated with Screaming Frog Ltd.

**Do I need an account to use Scrawly?**

**Direct answer:** No. There is no sign-up, login or approval. Install Scrawly, open it and start an audit.

**What are the system requirements?**

**Direct answer:** Scrawly runs on macOS, Windows and Linux. It needs Python 3.11 or newer and git, plus a one-time download of about 150 MB for the crawler browser. Node.js is not needed. The installer adds git and Python for you where it safely can.

**Does my data leave my computer?**

**Direct answer:** Your audits do not. Crawls, findings and reports are stored on your computer only, and Scrawly has no accounts and no analytics. It connects only to the sites you audit, GitHub for update checks, services you set up yourself such as PageSpeed Insights, Search Console or your AI provider, and the feedback service when you click Send.

**Can Scrawly render JavaScript?**

**Direct answer:** Yes. Scrawly renders pages in a real Chromium browser through Playwright, with a desktop pass and a mobile pass. It compares the raw HTML response with the rendered page, so it can flag content, links, titles or canonicals that only exist after JavaScript runs.

**Can Scrawly audit my site for ChatGPT, Perplexity and Google AI Overviews?**

**Direct answer:** It audits the technical side. Scrawly checks whether AI crawlers such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your pages, whether a CDN or firewall silently blocks them, your llms.txt, meta descriptions as AI context, structured data and agent readiness. It does not track rankings or citations in AI answers.

**How do I update or uninstall Scrawly?**

**Direct answer:** Updates are automatic: Scrawly checks GitHub each time it opens and installs new versions once their tests pass. You can also run the install command again. To uninstall, delete the Scrawly folder in your home folder, remove the app shortcut, and delete the data folder if you want your audits gone too.

**How do I report a bug or contribute?**

**Direct answer:** Click the speech-bubble icon at the bottom of the left bar in Scrawly to send feedback, with no GitHub account needed. You can also open an issue or a pull request on GitHub. Include your operating system, Scrawly version and scrawly.log, but never passwords or API keys.

**Does Scrawly work on sites that are not built with WordPress?**

**Direct answer:** Yes. Every site gets the full read-only audit, including Shopify, Wix, Webflow, Next.js and custom builds. Only the one-click fixes are WordPress-specific. For other platforms, Scrawly gives you a fix prompt that names the right file on your stack.

**Can Scrawly fix SEO issues for me?**

**Direct answer:** On WordPress, yes. With the Scrawly Connector plugin it can fix titles, meta descriptions, canonicals and redirects, saving a rollback snapshot first and asking before any live change. Higher-risk issues are never applied automatically.

**Can I white-label Scrawly reports for clients?**

**Direct answer:** Yes. Set your agency name, accent color, logo and contact line, and every HTML and PDF report uses your branding.

**Does Scrawly connect to Google Search Console, GA4 and PageSpeed Insights?**

**Direct answer:** Yes, if you want it to. One read-only Google connection adds Search Console and GA4 data for each page, using your own Google Cloud OAuth client. A free PageSpeed Insights API key adds Lighthouse lab data and CrUX field data for Core Web Vitals.

**Who made Scrawly?**

**Direct answer:** Ilias Sami, an SEO consultant and the silent AEO & GEO infrastructure partner for digital agencies, created Scrawly as a free contribution to the SEO community.

## Run your first free audit today

Install Scrawly with one command and audit any website for Google and AI search. No account, no page cap, MIT licensed. Download Scrawly free on GitHub: https://github.com/IliasSami/scrawly-seo-crawler

---
_Canonical page: [https://iliassami.com/scrawly](https://iliassami.com/scrawly) · Markdown generated on request from the live site content._