We’ve updated our Akamai Bot Directory to split the single AI Bots category into three distinct types: AI training crawlers, AI search crawlers, and AI fetchers and agents.
Businesses can now move beyond blanket “block or allow” rules and apply precise security policies based on what a bot actually does (e.g., restrict training scrapers but allow search indexers).
Differentiating real-time user agents from background crawlers enables sites to safely facilitate AI-driven customer interactions, research, and transactions.
The update pairs traffic visibility and control with optimization strategies, helping organizations safely protect their IP while maintaining brand presence in AI search results.
AI bots have quickly become a regular part of internet traffic, but not every AI bot interacts with a website for the same reason. Some crawl content to train AI models. Some index content for AI-powered search. Others access websites in real time to answer a question or complete a task for a user.
We have been detecting and individually identifying these bots since they emerged. Now, we're making it easier for customers to see, manage, and optimize for them, too.
We've updated the Akamai Bot Directory by splitting our existing AI Bots category into three more granular categories:
AI training crawlers: Scan and collect content to train LLMs
AI search crawlers: Index content for AI search and citations
AI fetchers and agents: Access content in real time for user requests and autonomous tasks
This simple change has an important benefit: Businesses can better understand the AI traffic that is reaching their sites and make more precise decisions about what to block, what to allow, and what to optimize for.
These new categories make it easier to apply different policies based on what the AI bot is actually there to do.
For example, imagine that you run a website that includes valuable content. You do not want an AI company crawling that content to train its models. But you do want an AI search engine to index it so your brand will appear when someone searches for information via an AI platform. You may also want to allow an AI assistant to access a page when one of your customers specifically asks it to.
Let's dive deeper into each of these new categories to see how they operate and why treating them differently matters for your business.
AI training crawlers (like GPTBot from OpenAI, ClaudeBot from Anthropic, and Meta-ExternalAgent from Meta) collect large volumes of web content to train AI models, potentially consuming resources and valuable content without driving traffic or value back to your site.
Our customers can now manage AI training crawlers as a distinct category, giving them greater control over how and where their content is used for AI training.
AI search crawlers (including include OAI-SearchBot from OpenAI, Claude-SearchBot from Anthropic, and PerplexityBot from Perplexity) index web content so AI-powered search engines can discover, understand, and surface information in responses, often with citations or links back to the original website. Although this can increase brand visibility across AI search, it doesn’t always translate into site traffic, as users may get the information they need directly from the AI response.
Separating search crawlers from training crawlers gives businesses more flexibility to protect their content while maintaining visibility across AI search.
AI fetchers and agents (such as ChatGPT-User, Claude-User, Perplexity-User, ChatGPT Agent, and GoogleAgent-Mariner) access websites on an ad hoc basis to fulfill immediate user requests or complete tasks on their behalf.
A fetcher might access a specific page because someone asked an AI assistant to summarize it. An agent may interact with a website to research something, compare options, or complete a task. Both represent a fundamentally different interaction from large-scale crawling, which is why they are now categorized separately from AI training and AI search crawlers.
More granular classification helps organizations move beyond simply identifying AI traffic to deciding how they want to engage with it. With Akamai Bot & Agent Control, businesses can define and enforce policies based on the type of AI interaction, whether that means limiting training crawlers, allowing search crawlers, or enabling trusted agents.
That distinction becomes even more important as agentic commerce takes shape. AI agents are beginning to do more than search for information. They can research products, compare options, navigate websites, and take actions on behalf of consumers.
As these interactions grow, businesses will need to distinguish between the AI traffic they want to welcome and the traffic they want to restrict. Granular controls make that choice possible without treating every AI bot the same.
Control is only part of the opportunity. As AI becomes a new way people discover brands and interact with businesses, brands also need to think about how they show up in these experiences. Akamai AI Brand Presence extends that strategy to optimization, helping businesses understand and improve their visibility across AI platforms.
Together, Bot Directory and AI Brand Presence create a more intentional approach to AI traffic: You can now understand who is accessing your site, control how they interact with it, and optimize for the AI experiences that create value.
The new categories are also part of a broader effort to continually strengthen Akamai's AI bot detection as the ecosystem evolves.
New AI bots, agents, and user agents are emerging quickly. Akamai continues to identify and add these entities to our detection capabilities, helping customers recognize more of the AI traffic interacting with their applications.
To learn more about how to gain granular control over your AI bot traffic, contact an expert.
© 2026 Akamai Technologies
