Overview: All AI crawlers at a glance

Every major AI platform operates its own web crawlers to collect content for training, knowledge updates or real-time search. These crawlers identify themselves via their user agent string and respect robots.txt – but beyond that, they differ significantly in crawl frequency, purpose, transparency and what they actually do with your content.

Core principle for all crawlers: They all respect robots.txt, read only the HTML source without JavaScript rendering (with partial exceptions) and prefer technically clean, fast websites. The differences lie in crawl frequency, purpose, whether they link back to your site and the degree of transparency each provider offers.

As of 2026, eight major AI crawlers are actively indexing the web. Understanding each one lets you make informed decisions about who gets access to your content – and who doesn't.

GPTBot – ChatGPT (OpenAI)

OpenAI

GPTBot

User agent

GPTBot

Platform

ChatGPT

Purpose

Training + updates

IP ranges

Published

Transparency

High

Source links

No

GPTBot is the best-known and most thoroughly documented AI crawler. OpenAI officially introduced it in August 2023 and publishes both a documentation page and the current IP address ranges – making it one of the most verifiable AI crawlers.

GPTBot crawls for two purposes: training future GPT models and updating the knowledge of already deployed models. Crawl frequency is moderate compared to Googlebot – important pages are typically visited on a cycle of weeks to months rather than days.

One key consideration: allowing GPTBot means your content may appear in ChatGPT answers, but ChatGPT rarely links directly to sources. You gain AI visibility but not referral traffic. If your goal is traffic over brand presence in AI answers, this trade-off is worth thinking about.

robots.txt control: User-agent: GPTBot – fully supported and reliably respected by OpenAI.

ClaudeBot – Claude (Anthropic)

Anthropic

ClaudeBot

User agent

ClaudeBot

Platform

Claude

Purpose

Context + answers

IP ranges

Not published

Transparency

Medium

Source links

Partial

ClaudeBot is Anthropic's crawler for the Claude language model. It operates on the same basic principles as GPTBot: reading HTML source, respecting robots.txt, no JavaScript rendering. Anthropic does not publish IP ranges, making it harder to verify crawl activity independently.

A distinctive feature of ClaudeBot is its focus on contextual understanding – Anthropic places great emphasis on Claude understanding relationships, nuance and long-form reasoning rather than just retrieving isolated facts. This means well-structured, content-rich pages with clear hierarchy are particularly valued.

Claude increasingly includes source citations in its answers, especially in its web search mode. This makes ClaudeBot more valuable for traffic than GPTBot in certain contexts.

robots.txt control: User-agent: ClaudeBot – fully supported.

PerplexityBot – Perplexity AI

User agent

PerplexityBot

Platform

Perplexity AI

Purpose

Real-time search

IP ranges

Published

Transparency

Medium

Source links

Yes

PerplexityBot differs from GPTBot and ClaudeBot in one critical respect: Perplexity is primarily a search engine with AI answers – not a pure chatbot. This means PerplexityBot crawls more actively and more frequently than training crawlers, often on a days-to-weeks cycle.

Perplexity cites sources directly and visibly in every answer, with clickable links back to the original pages. A citation by Perplexity is therefore the most traffic-valuable AI citation available right now. Any website wanting to appear as a Perplexity source needs technically clean, citable pages and an allowed PerplexityBot.

For most websites with public informational content, PerplexityBot is the single most important AI crawler to prioritise. Even if you block training crawlers, keeping PerplexityBot allowed is usually worthwhile.

robots.txt control: User-agent: PerplexityBot – respected. Perplexity also operates a secondary crawler called Perplexity-User for real-time lookups triggered by user queries.

Google-Extended – Gemini & AI Overviews

Google

Google-Extended

User agent

Google-Extended

Platform

Gemini, AI Overviews

Purpose

Gemini training

IP ranges

Published

Transparency

High

Source links

Yes (AI Overviews)

Google-Extended is a separate crawler from Google for AI-specific purposes, introduced in September 2023. It is completely distinct from Googlebot, which crawls for classic Google Search. Google-Extended collects data for training Gemini and for generating Google AI Overviews – the AI answers appearing at the top of search results.

The key advantage of Google-Extended: it can be controlled entirely independently from Googlebot. Blocking Google-Extended has no direct impact on your Google Search ranking – Googlebot continues crawling regardless. This gives website owners a genuine choice about AI training participation without SEO risk.

If you appear in AI Overviews, Google does include source citations. This makes Google-Extended particularly valuable for high-traffic informational queries where AI Overviews appear prominently.

robots.txt control: User-agent: Google-Extended – fully supported and independently controllable from Googlebot.

xAI-Bot – Grok (xAI)

User agent

xAI-Bot

Platform

Grok (X / Twitter)

Purpose

Training + search

IP ranges

Not published

Transparency

Low

Source links

Partial

Grok is the AI model from xAI, Elon Musk's AI company, deeply integrated into the X (formerly Twitter) platform. The web crawler identifies itself as xAI-Bot and crawls for both training and knowledge updates.

xAI is the least transparent of the major AI providers: no published IP ranges, limited official documentation and less clear communication about crawling scope or data usage. This makes it harder to verify whether robots.txt rules are reliably respected.

A key differentiator: Grok has real-time access to X posts and can incorporate social media signals directly into answers – independently of web crawling. This means your X presence and web presence work together for Grok visibility in a way that doesn't apply to other AI platforms.

robots.txt control: User-agent: xAI-Bot – respected according to current knowledge, but less verifiable than other crawlers.

---

Amazonbot – Amazon / Alexa AI

Amazon

Amazonbot

User agent

Amazonbot

Platform

Alexa, Amazon AI

Purpose

Alexa answers + training

IP ranges

Published

Transparency

Medium

Source links

No

Amazonbot is Amazon's web crawler, operating primarily for Alexa voice assistant answers and Amazon's broader AI initiatives. Amazon publishes its IP address ranges, which allows server-side verification of crawler authenticity – a notable plus for transparency.

While Alexa's web-based smart speaker market share has declined, Amazon is actively integrating AI into its shopping, AWS and Alexa ecosystems. Amazonbot's crawl scope reflects this: it focuses on factual, structured content that can be used to answer voice queries and power AI features across Amazon's product line.

For most websites, Amazonbot has lower immediate traffic impact than PerplexityBot or Google-Extended. However, e-commerce sites, local businesses and informational content producers benefit from maintaining Alexa visibility, especially as Amazon expands its AI answer features.

robots.txt control: User-agent: Amazonbot – fully supported. Amazon publishes clear documentation and IP ranges for verification.

Applebot-Extended – Apple Intelligence / Siri

Apple

Applebot-Extended

User agent

Applebot-Extended

Platform

Apple Intelligence, Siri

Purpose

AI training (Apple)

IP ranges

Published

Transparency

High

Source links

No

Applebot-Extended is Apple's dedicated AI training crawler, introduced in 2024 alongside the rollout of Apple Intelligence. It is completely separate from the regular Applebot, which crawls for Spotlight search and Safari Suggestions. The two can be controlled independently.

Apple introduced this separation deliberately to give website owners control over AI training participation without affecting their presence in Apple's regular search features. Blocking Applebot-Extended does not affect Spotlight indexing or Siri's ability to surface your website as a regular result.

Apple Intelligence is tightly integrated into iOS, iPadOS and macOS – a user base of over a billion active devices. As Apple Intelligence expands its capabilities, Applebot-Extended's strategic importance will increase significantly. Websites targeting iOS users in particular should consider their Applebot-Extended policy carefully.

Apple publishes IP ranges for verification, making Applebot-Extended one of the more transparent AI crawlers despite being relatively new.

robots.txt control: User-agent: Applebot-Extended – fully supported and independently controllable from regular Applebot.

Bytespider – ByteDance / TikTok AI

ByteDance

Bytespider

User agent

Bytespider

Platform

ByteDance / TikTok AI

Purpose

Training

IP ranges

Not published

Transparency

Low

Source links

No

Bytespider is ByteDance's web crawler – the company behind TikTok and the large language model family used in their AI products. It has been observed crawling at very high volumes, in some cases more aggressively than other AI crawlers, which has led to concerns among webmasters and hosting providers.

ByteDance does not publish IP ranges or comprehensive documentation for Bytespider, making it the least transparent of all major AI crawlers. There is no clear public-facing AI product that surfaces web content with source citations – Bytespider appears to be primarily a training data crawler.

Given the lack of transparency, the absence of source links and reported aggressive crawl behaviour, many website owners choose to block Bytespider by default unless they have a specific reason to allow it. This is a reasonable precaution that has no known negative impact on any user-facing AI search product.

robots.txt control: User-agent: Bytespider – listed as respected, but less verifiable due to limited documentation.

Direct comparison: all crawlers in one table

Crawler	User agent	JS rendering	Crawl freq.	Source links	IP ranges	Transparency	robots.txt	Recommendation
GPTBot	GPTBot	No	Weeks–months	No	Yes	High	Yes	Allow for AI presence
ClaudeBot	ClaudeBot	No	Weeks–months	Partial	No	Medium	Yes	Allow for AI presence
PerplexityBot	PerplexityBot	No	Days–weeks	Yes	Yes	Medium	Yes	Highest priority
Google-Extended	Google-Extended	Partial	Regular	Yes (AI Overviews)	Yes	High	Yes	Allow for AI Overviews
xAI-Bot	xAI-Bot	No	Unknown	Partial	No	Low	Yes	Optional
Amazonbot	Amazonbot	No	Weeks	No	Yes	Medium	Yes	Allow if targeting Alexa
Applebot-Extended	Applebot-Extended	No	Weeks–months	No	Yes	High	Yes	Allow for iOS audience
Bytespider	Bytespider	No	High / aggressive	No	No	Low	Unclear	Block by default

💡 Tip: Use the robots.txt Validator to check whether your current configuration correctly controls each of these crawlers. Capitalisation in user agent names matters – GPTBot and gptbot are treated differently.

Which strategy is right for you?

Maximum AI visibility – allow all except Bytespider

For public websites with informational content, service providers, blogs and tools: allow all crawlers except Bytespider. This is the most sensible default for anyone wanting to be cited in as many AI answers as possible. Bytespider's lack of transparency and aggressive crawl behaviour makes it the one exception worth blocking in most cases.

Selective – prioritise search-based crawlers

If your primary goal is clickable traffic from AI sources, prioritise PerplexityBot and Google-Extended. These are the two crawlers where a citation directly translates into a link back to your website. GPTBot, ClaudeBot and Amazonbot build AI presence without driving direct traffic.

Block training, allow search

Anyone who does not want their content used for AI model training – but wants to appear in real-time AI search results – can block GPTBot, ClaudeBot, Applebot-Extended and Amazonbot while keeping PerplexityBot and Google-Extended active. This separates the training use case from the search visibility use case.

Block all AI crawlers

For websites with copyrighted, paid or sensitive content, blocking all AI crawlers is a reasonable choice. This is a conscious decision against AI visibility and carries the trade-off of not appearing in any AI-generated answers. For publishers concerned about content use without compensation, this may be the right call.

Ready-made robots.txt templates

Allow all AI crawlers (maximum visibility)

User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml

Recommended default: allow all, block Bytespider

User-agent: Bytespider Disallow: / User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml

Block training crawlers only (allow search)

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Amazonbot Disallow: / User-agent: xAI-Bot Disallow: / User-agent: Bytespider Disallow: / User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml

Block all AI crawlers

User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Perplexity-User Disallow: / User-agent: Google-Extended Disallow: / User-agent: xAI-Bot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml

• Make a conscious decision: which AI crawlers should have access to your content?
• Check robots.txt for correct user agent names – capitalisation matters (GPTBot not gptbot)
• Add Perplexity-User alongside PerplexityBot if you want to block all Perplexity access
• Blocking Google-Extended does not affect your Google Search ranking
• Blocking Applebot-Extended does not affect Spotlight or regular Siri results
• Test robots.txt with a validator after every change
• TTFB under 800ms – so all crawlers can fully read your pages