Free Tool
robots.txt Generator for AI Crawlers
Build a robots.txt that either welcomes AI crawlers (GPTBot, ClaudeBot, PerplexityBot…) so you can be cited in AI answers — or blocks them. Copy the result to your site root.
robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /cart # Welcome AI crawlers User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: PerplexityBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: CCBot User-agent: Meta-ExternalAgent Allow: / Disallow: /admin Disallow: /cart
Upload to yoursite.com/robots.txt. Do AI crawlers respect robots.txt? →
Want the full picture?
These tools fix one thing at a time. A full SiteScoreNow audit scores your whole site across SEO, GEO, AEO & LLMO and hands you a prioritized action plan.