Definition and user agents
Anthropic documents three user agents with separate purposes:
- ClaudeBot: collects training data for future Claude models — the counterpart of GPTBot.
- Claude-SearchBot: crawls for Claude's search functionality — determines which sources can appear in answers with web search.
- Claude-User: fetches pages live when users trigger a URL or research task in Claude.
Older labels such as anthropic-ai and Claude-Web still circulate in robots.txt templates; Anthropic's current documentation is authoritative. According to Anthropic, all crawlers respect robots.txt and are verifiable via published IP ranges.
The visibility trade-off
The training question (ClaudeBot) and the visibility question (Claude-SearchBot, Claude-User) must be answered separately. Training presence works long-term: content that flows into training corpora shapes what future model generations “know” about a brand — the slowest but most durable channel of LLM SEO. Search accessibility acts immediately on citations in Claude answers. A blanket Anthropic block ends both at once.
# example: allow everything (visibility position) User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: /
Practice checks
- Update robots.txt: list the current Anthropic user agents explicitly instead of outdated aliases.
- Check WAF rules: bot-protection systems often block AI crawlers wholesale — analyse logs for 200 status codes for the Claude agents.
- Monitoring: test your representation in Claude regularly — entity resolution and citations, documented via prompt sets (see the entity resolution study, which measures exactly this layer).
Related terms
ClaudeBot belongs to the crawler policy of the retrieval layer beside GPTBot, OAI-SearchBot, PerplexityBot and Google-Extended; curated hints for all systems come from llms.txt. Model-specific optimisation: Claude citation optimisation.
Training and search are two decisions
ClaudeBot fills training corpora, Claude-SearchBot the search layer. Blocking Anthropic wholesale gives up both — training presence in future models and citations in today's answers.
FAQ on ClaudeBot
What is the difference between ClaudeBot, Claude-SearchBot and Claude-User? ▾
ClaudeBot collects training data for future Claude models. Claude-SearchBot crawls for the search function and determines the citable sources. Claude-User fetches pages live when users trigger a URL or research task. All three can be controlled separately via robots.txt.
Should you block ClaudeBot? ▾
That is a business decision between content protection and model presence: blocking keeps content out of future training corpora — and thereby gives up the most long-term channel of brand anchoring in LLMs. For visibility-oriented brands, opening usually prevails.
Do the Anthropic crawlers respect robots.txt? ▾
According to Anthropic's documentation, yes, for all three user agents. Anthropic publishes IP ranges for verifying real hits; in practice, blocks happen more often unintentionally through WAF and bot-protection rules than deliberately via robots.txt.
How do I check my representation in Claude? ▾
With documented prompt sets: entity prompts (“who is …?”) and subject prompts across several runs, evaluated against verified facts. The methodology is documented in the entity resolution study — it measures exactly this training-data layer.