Research · AI crawlers
Which AI crawlers read local business websites: 80,000+ visits from 19 crawlers
By Cloute · Published October 1, 2026
The short answer
In server logs of local business pages Cloute publishes, ClaudeBot made the most AI crawler visits in 2026 (17,471), followed by GPTBot (13,991), Amazonbot (10,389), OAI-SearchBot (8,861), ChatGPT-User (7,582), Meta's AI crawler (5,120) and PerplexityBot (4,160), out of 80,420 visits from 19 identified AI crawlers. Counted by company, OpenAI's three crawlers made the most visits, 30,434.
AI crawler visits logged, 2026
different AI crawlers identified
visits from OpenAI's three crawlers combined
Three kinds of AI crawler
Not every AI crawler does the same job, and the difference matters when you decide what to allow.
| Crawler | Company | What it does | Visits |
|---|---|---|---|
| ClaudeBot | Anthropic | Collects pages used to train Claude | 17,471 |
| GPTBot | OpenAI | Collects pages used to train OpenAI's models | 13,991 |
| Amazonbot | Amazon | Crawls for Amazon's services, including Alexa answers | 10,389 |
| OAI-SearchBot | OpenAI | Builds the index ChatGPT search draws on | 8,861 |
| ChatGPT-User | OpenAI | Opens a page live, while ChatGPT is answering someone | 7,582 |
| Meta AI crawler | Meta | Collects pages for Meta's AI models | 5,120 |
| PerplexityBot | Perplexity | Builds Perplexity's search index | 4,160 |
- Training crawlers (GPTBot, ClaudeBot, Meta's crawler) gather pages that may shape what a model knows in its next version.
- Search crawlers (OAI-SearchBot, PerplexityBot) build the indexes an engine searches when it answers with live results.
- Live fetchers (ChatGPT-User, Perplexity-User, Claude-User) open a page at the moment a person's question sends the engine there. These are the closest thing to a customer reading your page through an AI.
What a local business should take from this
- AI engines do read local business websites, at scale. 7,582 of these visits were ChatGPT opening a page live while answering a real person's question.
- Blocking OAI-SearchBot or PerplexityBot in robots.txt keeps a site out of those engines' search results. Blocking only the training crawlers does not.
- These crawlers do not run JavaScript. A site that only shows its content after scripts run looks empty to them. We found this on our own website in 2026 and fixed it.
- Google's AI Overviews and AI Mode use Google's regular search crawler, so they do not appear as a separate AI crawler in server logs.
How the data was collected
Every request to the local business pages Cloute publishes for its clients is logged on the server. A visit is counted when its user agent matches a published AI crawler name. Image, script and style files are not counted. Crawlers that hide their identity are not counted, so these figures are a floor.
Questions people ask
Does ChatGPT crawl websites?
Yes. OpenAI runs three crawlers: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for pages opened live during an answer. Together they made 30,434 visits to the local business pages in this dataset in 2026.
Which AI crawler visits websites the most?
In this dataset, ClaudeBot (17,471 visits), then GPTBot (13,991). By company, OpenAI's crawlers combined made the most visits.
Should a local business block AI crawlers?
Usually not the search crawlers. Blocking OAI-SearchBot, PerplexityBot or the live fetchers removes the site from those engines' answers. Training crawlers can be blocked separately if a business prefers.
Do AI crawlers run JavaScript?
Generally no. Pages that only render their text with JavaScript appear empty to AI crawlers, so the content should be in the HTML the server sends.
How to cite this research
Cloute (2026). AI crawlers on local business websites: 80,420 visits from 19 crawlers. https://cloute.ai/research/ai-crawlers-local-business-websites-2026
The underlying data is open under CC BY 4.0: github.com/luked9/cloute-ai-visibility-research.
About Cloute. Cloute is an AI visibility company for local businesses. It measures how often ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews recommend a business against its local competitors, and works every month to move it to the top.