Technical SEO

robots.txt Generator

Build a valid robots.txt with AI crawler directives. Fetch your existing file to inspect and improve it.

Inspect Existing robots.txt

Rule 1

robots.txt Output

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

# AI Search Crawlers

User-agent: GPTBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Googlebot
Allow: /

Upload as robots.txt to your domain root. Verify at yoursite.com/robots.txt