How AI Bots Are Rewriting the Rules of Publishing

The internet itself is transforming. Gone are the days of sifting through Google search results, clicking link after link to find what you need. Now, people are heading straight to ChatGPT for answers or relying on Gemini’s AI-powered summaries to get information instantly. The fundamentals of organic search are shifting.

AI bots are crawling the internet to fuel training data and improve AI model performance. What does that mean in practice? First and foremost, it’s hitting content creators and publishers the hardest:

• Traffic loss: As AI models consume and repackage your content into direct answers, fewer users click through to your site. This undermines the page views you rely on for advertising revenue.
• Revenue erosion: Publishers are seeing a decline in programmatic ad impressions and paid subscriptions as audiences stay within AI interfaces rather than visiting original sources.
• Brand dilution: When your content is stripped of context and attribution, your brand loses visibility. Users often don’t know where the information originated.
• Data misuse: Your proprietary content, sometimes behind paywalls, is harvested to train models that can then compete with your offerings.
• Increased infrastructure costs: High-volume scraping drives up bandwidth and compute resources without delivering any return.
  

The clicks that publishers relied on for monetization are declining. And the reality is that most of this extraction happens without compensation or an awareness of how content may be reused.

For more than two decades, Akamai has partnered closely with publishers to protect their digital properties and stay ahead of constantly shifting threats. We’ve evolved alongside our customers as business models have changed, from the rise of paywalls to new approaches for monetization and audience engagement. 

Today, the most forward-thinking publishers understand that bot management isn’t only about blocking unwanted traffic; it’s also a strategic pillar that supports revenue, protects user experience, and safeguards long-term growth.

This isn’t about being anti-AI. It’s about evolving responsibly in an era where consumers are embracing AI. And it’s about recognizing that journalism is expensive to produce and easy to exploit if it’s left undefended. AI bots don’t just borrow headlines — they extract the value that publishers have worked hard to create.

We see AI scraping not only as a threat but also as a chance to rethink access. Our tools help you move beyond blocking, so you can identify, control, and even monetize AI bots.

While some may recommend blocking AI bots outright, we don’t believe that’s always the best approach. Our strategy is more nuanced:

• Consider how adaptable bot operators can be
• Make informed decisions about blocking bots
• Measure short- and long-term effects of bot traffic

We’ve been in the bot business for a long time. We know how operators think. They expect to be blocked and are prepared to adapt. Sooner or later, they’ll find ways around blanket blocks, and it becomes an endless back-and-forth.

By blocking AI bots completely, what are you aiming to achieve? If your goal is to push GenAI platforms to negotiate licensing, be realistic about whether that move will work or whether it will just send them to scrape from sources that aren’t protecting their content. 

Start by building visibility into who’s accessing your content and how often, so you can make informed decisions about when to block, when to allow, and when to engage.

And before you dismiss AI bots as a small percentage of traffic today, consider these questions:

• If a GenAI crawler ingests your content once, how many times will that same content be surfaced in AI-generated responses, replacing future visits you would have monetized? 
• Are you measuring only the initial scrape, or the long-term impact on engagement as habits shift?

Our approach is simple: put you in control. You decide how to handle AI bots. If you want to block them entirely, we give you the tools to do that. If you’d rather start by measuring and understanding their activity, we make that possible too.

Importantly, we can also help you monetize this new reality. Through edge compute and our partnerships with third-party solutions, Akamai makes it possible to enforce policies at the edge, requiring AI bots to authenticate, agree to licensing terms, or pay to access your content. Instead of simply blocking, you can transform scraping from a liability into a revenue stream.

We believe that the decision about whether and how AI bots access your content belongs to you, not the bots. Blanket blocking often just escalates the problem, prompting a cat-and-mouse game of defense and evasion. Instead, start with full visibility so you can make informed choices.

Akamai Content Protector paired with Akamai App & API Protector delivers granular controls and enforcement to help you decide what’s fair, what’s allowed, and what should be stopped outright.