Learn · AI crawlers
What is ClaudeBot? And what are Claude-User and Claude-SearchBot?
By Cloute · Published October 1, 2026
The short answer
ClaudeBot is Anthropic's crawler for collecting web content that could contribute to training Claude. Anthropic runs two more: Claude-SearchBot, which indexes content to improve Claude's search results, and Claude-User, which visits pages when a Claude user asks a question. Each has its own robots.txt token, and Anthropic says all three honor robots.txt.
ClaudeBot visits in Cloute's server logs of local business pages in 2026
busiest AI crawler in that dataset
What does ClaudeBot do?
Anthropic says ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training." If a site blocks it, Anthropic says the site's future materials should be excluded from its AI model training datasets.
ClaudeBot was the most active AI crawler in Cloute's server logs of local business pages in 2026, with 17,471 visits.
What are Claude-User and Claude-SearchBot?
| Bot | What Anthropic says it does | What blocking it does |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to training | Excludes the site's future content from training |
| Claude-SearchBot | Analyzes content to improve the relevance and accuracy of search responses | Stops indexing for search, which may reduce visibility in Claude's search results |
| Claude-User | Visits websites when people ask Claude questions | Stops Claude retrieving the content for a user's question, which may reduce visibility |
Do Anthropic's crawlers follow robots.txt?
Yes. Anthropic says its bots honor industry standard robots.txt directives and also support the non-standard Crawl-delay extension. Unlike OpenAI and Perplexity, Anthropic does not carve out its user-initiated fetcher: blocking Claude-User in robots.txt is respected.
Each bot is controlled separately. Blocking ClaudeBot does not block Claude-SearchBot or Claude-User.
How do I block ClaudeBot but stay visible in Claude?
- User-agent: ClaudeBot
- Disallow: /
- User-agent: Claude-SearchBot
- Allow: /
- User-agent: Claude-User
- Allow: /
Anthropic publishes the IP addresses its crawlers use at claude.com/crawling/bots.json, which lets a site check that a request claiming to be from Anthropic is genuine. Older tokens such as anthropic-ai and Claude-Web are not the ones Anthropic documents today, so rules written only for them may not apply to current crawlers.
Why does this matter for a local business?
Claude depends heavily on what it reads. Across AI answers about local businesses, Claude named a business 86.6% of the time when it had read one of the business's own pages and 11.9% when it had not, the largest gap of any engine measured. Blocking Claude-SearchBot or Claude-User cuts off that reading.
Like the other AI crawlers, it does not run JavaScript. A page whose text only appears after scripts load looks empty to it, so the content should be in the HTML the server sends. Cloute found its own website was invisible to AI crawlers for exactly this reason in 2026, and fixed it.
Read next
Research: 80,000+ AI crawler visits to local business websites What is GPTBot? What is PerplexityBot? (and Perplexity-User) AI crawlers: the complete list for 2026 Should a local business block AI crawlers?Questions people ask
Does blocking ClaudeBot remove my site from Claude's answers?
Not by itself. ClaudeBot is for training. Claude's search indexing and live page visits use Claude-SearchBot and Claude-User, which are controlled separately.
Does Claude-User respect robots.txt?
Anthropic says all of its bots honor robots.txt, including Claude-User. OpenAI and Perplexity say robots.txt may not or generally does not apply to their user-initiated fetchers.
Is ClaudeBot crawling local business websites?
Yes. It made 17,471 visits in Cloute's server logs of local business pages in 2026, more than any other AI crawler.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic says its bots support the Crawl-delay directive in robots.txt.
About Cloute. Cloute is an AI visibility company for local businesses. It measures how often ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews recommend a business against its local competitors, and works every month to move it to the top.