Powered by KwikChexAI crawlers and robots.txt: let the search bots in
By About This Business editorial team · Published · Updated
Short answer
To appear in AI assistants' answers, allow their search crawlers in robots.txt: OAI-SearchBot for ChatGPT, Claude-SearchBot for Claude, PerplexityBot for Perplexity, and Googlebot for Google's AI features. Training crawlers such as GPTBot and ClaudeBot are separate, so you can block training while allowing search. Also check that your firewall or CDN isn't blocking them.
Key takeaways
- Search crawlers and training crawlers are separate, and each has its own robots.txt name.
- Blocking a search crawler can remove you from that assistant's answers.
- A broad "block all bots" rule, firewall or CDN setting can block AI search crawlers without you noticing.
- Robots.txt is a request. Check your server logs to see who actually visits.
Why do AI crawlers matter for your business?
Because an assistant can only use your pages if its crawler is allowed to read them. If its search crawler is blocked, the assistant falls back on whatever other sites say about you, which may be older or wrong. This is part of How AI assistants decide what to say about your business.
Which AI crawlers should you know?
Each assistant separates search, training and user-requested fetches into different named crawlers.
| Crawler | Operator | Job | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | You won't be shown in ChatGPT search answers |
| GPTBot | OpenAI | Training data | Excluded from training; search unaffected |
| ChatGPT-User | OpenAI | Fetches pages for user requests | ChatGPT can't open your pages on request |
| Claude-SearchBot | Anthropic | Indexes for Claude's search | May reduce visibility in Claude's search results |
| ClaudeBot | Anthropic | Training data | Future content excluded from training |
| Claude-User | Anthropic | Fetches pages for user questions | Claude can't retrieve your pages for users |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity | Reduced visibility in Perplexity |
| Perplexity-User | Perplexity | Fetches pages users ask about | Generally ignores robots.txt |
| Googlebot | Google Search, including AI Overviews and AI Mode | Removed from Search and its AI features | |
| Google-Extended | Controls use in other Google AI systems | Doesn't affect Google Search |
How do you allow search but block training?
Name each crawler in robots.txt. This example allows the search crawlers and user fetches, and blocks the training crawlers:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Whether to block training is your choice. Blocking it doesn't affect search visibility. Keep your existing rules for private areas such as account and admin pages, and include your sitemap: Sitemap: https://yourdomain/sitemap.xml.
What else can block AI crawlers?
Firewalls, CDNs, security plugins and "block AI bots" presets. These can refuse a crawler even when robots.txt allows it. Check your hosting or CDN bot settings, and look in your server logs for visits from the crawler names above.
How do you check it's working?
Read your robots.txt, check your bot settings, then look for real visits.
- Open
yourdomain/robots.txtand check every rule that names, or catches, the crawlers above. - Check your CDN or firewall bot settings.
- Search your server logs for
OAI-SearchBot,Claude-SearchBotandPerplexityBot. - Make sure key pages render their main content as HTML text. See Structured data for AI search.
Once crawlers can reach you, the next question is what they find. See How ChatGPT, Claude, Perplexity and Google AI find their sources.
Part of How AI assistants decide what to say about your business.
FAQ
If I block GPTBot, will I disappear from ChatGPT search?
No. GPTBot is OpenAI's training crawler. ChatGPT search visibility is controlled by OAI-SearchBot, which you can allow separately.
Does blocking ClaudeBot block Claude's search?
No. ClaudeBot is for training. Claude-SearchBot and Claude-User handle search and user requests, and each needs its own rule.
Does robots.txt stop every AI fetch?
Not always. Perplexity documents that Perplexity-User, which fetches pages a user asked for, generally ignores robots.txt because the request is user-initiated.
My robots.txt has a catch-all "User-agent: *" rule. Is that enough?
If it allows crawling, the AI search crawlers are allowed too. If it blocks crawling, it blocks them as well. Named rules make your intent unambiguous.