AI and LLM Bot Management Has Become a Business-Critical Issue: Do It Right
AI bots and LLM scrapers all want your content. Akamai stops the scrapers that hinder your business and manages the bots that help it.
TL;DR
- AI bot traffic has grown 300% in the past year.
- The number of AI/LLM bots is increasing substantially, but their use patterns help determine intent.
- With Akamai, you can know who’s accessing your content, stop scrapers, and take back control of your traffic by deciding who can access which content, under what conditions.
AI is driving increased traffic on the internet
Artificial intelligence (AI)-driven bot traffic went from next to nothing late last year to 0.27% of traffic across the Akamai platform and 0.9% of known bots, tracked in our library of self-identified, Akamai-categorized bots (Figure 1). This traffic is now already in the billions of requests per day.
Fig. 1: Verified AI bot daily triggers have increased massively and steadily since we began tracking them in late 2024
This dramatic growth shows no signs of abating, and we see more AI bots (scraping, user-driven, agent-driven) entering the fray weekly. Some are better behaved than others, and the intent of the AI bots varies widely.
We actively track both verified and inferred AI bots:
- Verified AI bots — via user/agent plus additional factors like source ASN
- Inferred AI bots — via user/agent (but no additional factor) or threat intel signals
Akamai protects the most valued — and targeted — brands in the world
We protect many of the biggest names worldwide and across industries, and we also protect their most valuable assets.
Because Akamai operates the world’s most distributed cloud platform, we have unmatched visibility into good intent — and can therefore more easily detect malicious intent.
We know what the AI bots and large language models (LLMs) are after, giving us unparalleled insights into activity and use cases, and affording us the ability to provide the best management solutions on the planet.
This is particularly interesting in user- or agent-driven AI/LLM requests (in which specific information is sought out in real time), as opposed to model building (in which exhaustive crawls are conducted in less frequent intervals).
A front-row seat for the LLM explosion
Akamai is uniquely positioned for visibility into the accelerated use of LLMs. Here are five things that we’re seeing:
- The numbers of AI/LLM bots are increasing substantially as a proportion of bots.
- Inbound AI bot requests are coming from a wide variety of vendors.
- We know which LLMs are targeting specific industries and where they’re located.
- Some industries are targeted more heavily by AI bots.
- Usage patterns help determine intent.
AI/LLM bots increasing as a proportion of bots
While the number of verified bots has increased significantly since late 2024, most of that growth occurred in the beginning of the time frame. We started tracking verified AI bots around the same time, and that growth has been steady and sharp (Figure 2).
Fig. 2: Verified AI bots are growing at a much faster rate than verified bots in general
Inbound AI bot requests from a wide variety of vendors
We are actively tracking dozens of AI bots from AI vendor data centers, those running in-house AI models, and AI vendor mobile apps (Figure 3).
Fig. 3: We see a wide variety of AI bot–sourced traffic, with the top six vendors accounting for two-thirds of traffic
LLMs targeting specific industries
We know which LLMs are targeting specific industries and locations. Many top AI bots (such as those from Google, Claude, and Meta) focus on our commerce customers (Figure 4).
- GPTBot (OpenAI) has a penchant for high tech customers.
- Operator (OpenAI agent) almost exclusively targets high tech customers.
- PetalBot (Huawei) heavily targets social media customers.
Fig. 4: AI bot traffic from various vendors varies by day
Industries targeted more heavily by AI bots
As shown in Figure 5, different AI bots target different industries.
Fig. 5: Different AI bots target different industries
In Figure 6, we can see that publishing customers are more than 7x more likely to see AI-driven bot traffic than our broader customer base. AI bots comprise some of the top user agents seen across the vertical (red bars to the left).
Fig. 6: AI bots are not among the top user agents across all customers, but they are among the top when looking at publishers
Usage patterns help determine intent
In Figure 7, the red bars (number of customers) indicate scraping for model building/updating. Blue bars indicate user/agent-driven requests.
Fig. 7: While many AI bots are active across a large number of accounts, only a handful are generating sizable volumes of requests
How Akamai manages AI/LLM bots
For more than two decades, Akamai has partnered closely with publishers to protect their digital properties and stay ahead of constantly shifting threats.
- Know who’s accessing your content: Understand the landscape of known bots and how they adapt to evade detection
- Stop scraping: Instantly allow or block bots from crawling your content
- Take back control: Decide who can access which content, under what conditions
We have specific detections to recognize malicious bots that are trying to impersonate known bots. Attackers, knowing that organizations will allow some AI bots — e.g., ChatGPT, will exploit this.
Take control of your relationship with AI bots
The key word in our bot management solution is … management. Blocking AI bots outright is an aggressive strategy, as user- and agent-driven AI traffic grows rapidly. But allowing all AI bots may also be the wrong move.
Our bot management suite lets customers take control of their relationship with AI bots — allowing those that they find beneficial, blocking the adversarial ones, and taking granular context-specific actions on others. As a result, they can then make the right decisions for their businesses.
Consider tomorrow’s real-world use cases, today
For instance, let’s say that a retailer would like to share product details but not pricing information with a particular LLM. They could serve a product page without pricing details to that specific LLM, while tarpitting requests from LLMs that it isn’t interested in interacting with (or impersonators of LLM bots).
Perhaps they wouldn’t allow an AI agent to check out, but instead present a user challenge to ensure that the buyer is a human.
Let’s consider another example. In this scenario, a publisher would like to accept queries from AI vendors that it has business monetization arrangements with (such as Google AI’s Offerwall), while inserting a challenge (e.g., JavaScript or CAPTCHA) for those it doesn’t.
How you manage LLM bots is a strategic business need
The industry tends to think of bots as a security and abuse issue. However, we increasingly see that how companies choose to handle bot traffic has a direct impact on their business objectives.
Customers need to think about AI bots in a business context, and we can help. This is an emerging space, and our ability to see the whole picture is invaluable. Consider the following questions:
- Which LLMs are you interested in serving?
- Which use cases (model building vs. users/agents) matter to you?
- Which information do you want to share, and which information do you not want to share?
- Which mechanisms do you want to employ to manage inbound AI?
In the future, how you manage your relationships with AI/LLMs will be a large part of how you operate your business — and whether or not you achieve your objectives.
