Manage, Don’t Mitigate Bot Traffic
Traditionally, organizations apply a one-size-fits-all approach of mitigating (blocking) web crawlers: block all bots and site scrapers in the same manner. This approach can lead to lower search engine visibility or otherwise negatively affect online business objectives. In addition, blocking bots is ineffective over the long term, as blocked bots simply mutate, evolve, and return to sites in an even more difficult-to-detect form.
Traditional bot mitigation tactics are ineffective for three reasons:
- Blocking treats all bot traffic the same, regardless of whether some bots are beneficial to business.
- Blocking sends the bot back to the operator, which can prompt the operator to alter the bot so it can return under a hidden identity.
- Blocking beneficial (good) bots may result in lower search-engine visibility and hinder other business objectives.
Managing bot traffic using predefined signature analysis and real-time detection is a more effective approach by enabling organizations to allow and prioritize beneficial bots while denying or delaying access to bad bots, or even serving them an alternate origin, alternate content, or content out of cache.
Identify Bots Based on Signatures
With visibility into 15% to 30% of global web traffic, the Akamai Intelligent Platform™ includes 15 categories of self-identified bots, which include more than 1,200 bots. A granular view of site traffic shows that as much as 40% of traffic volume received may be comprised of bots and scrapers.
A typical traffic profile may reveal:
- 60% User traffic
- 20% Search engines
- 10% Unknown
- 5% Site development & monitoring
- 2% Web archiving
- 1% Advertising
- 1% Other
Continuously updating bot directories based on signatures, along with creating custom signatures to identify known bots, enables immediate classification of bot signatures and assignment of real-time response actions, lowering business risks and reducing IT load.
Categorize Bots For Better Response Actions
Bot signatures contain factors that enable a robust solution like Bot Manager to classify different bots into three categories:
- Akamai-known bots — A continuously updated directory of known bots based on recent interactions with other Akamai customers, using its Cloud Security Intelligence (CSI) data analysis engine.
- Customer-known bots — Custom signatures identify known bots that regularly (and legitimately) interact with a site, using factors such as IP address, subnet, ASN, and header values to assign a specific action to be taken.
- Unknown bots —Unknown bots are detected and identified using a variety of characteristics such as request rate, request characteristics, bot behavior, and workflow validation.
Reduce IT Burden with Specific Bot Management Actions
The burden on IT infrastructure can be minimized by assigning appropriate actions based upon a bot category. Response can include a range of advanced actions:
- Silent Deny, which blocks a request from the site but doesn’t return an error code to the client, cloaking the response.
- Delay allows a request to the site after inserting a three-second delay. It can be useful for managing bots that are beneficial but are imposing a significant performance load on the origin.
- Slow allows a request to the site after inserting an 8-10 second delay.
- Serve Alternate Content redirects a request to an alternate configured page. This enables a response to bots with information that differs from a customer-facing site – for example, an alternate e-commerce site that has the same products with different prices.
- Serve Alternate Origin redirects a request to a preconfigured alternate origin infrastructure. This action is similar to Serve Alternate Content but also helps minimize the performance load on customer-facing origin infrastructure.
- Serve Cached responds to requests with content out of cache, which can also help minimize the performance load on the origin infrastructure.