Do AI Crawlers Respect robots.txt? GPTBot, ClaudeBot & PerplexityBot Explained

Your robots.txt file controls which automated crawlers can access your site. In the AI era, that includes a growing list of bots from OpenAI, Anthropic, Perplexity, Google, and others. If you want to be recommended and cited in AI answers — the goal of GEO — you need to make sure the right ones are allowed in.

Do AI crawlers actually obey robots.txt?

The major, reputable AI crawlers publish their bot names and honor robots.txt directives. If you disallow them, they'll generally stay out — which also means you won't appear in the answers they generate. Less reputable scrapers may ignore the rules, but you can't control those; focus on configuring access for the ones that matter.

The AI crawlers worth knowing

• GPTBot — OpenAI's crawler for training and browsing.
• OAI-SearchBot — OpenAI's search/citation crawler.
• ClaudeBot — Anthropic's crawler (powers Claude).
• PerplexityBot — Perplexity's answer-engine crawler.
• Google-Extended — controls whether Google uses your content for Gemini/AI.
• Applebot-Extended — Apple's AI-training control.

How to welcome AI crawlers

To be eligible for AI citations, allow these bots in your robots.txt with an Allow: / rule for each user-agent. You can still disallow private or utility paths (like an admin area) for every bot while allowing your public content.

Should you ever block them?

Some publishers block AI crawlers to protect content or negotiate licensing. For most small businesses, the opposite is true: being absent from AI answers is a bigger risk than being cited. If your goal is visibility, let them in.

Don't forget the rest of the stack

Allowing crawlers is step one. To actually get cited, pair it with an llms.txt file, structured data, and clear factual content — the full LLMO and AEO playbook.

See how your site scores

SiteScoreNow audits any website across SEO, GEO, AEO, and LLMO and gives you a prioritized, plain-English action plan. Get a free instant score, or order the full report to see exactly what to fix first.