How Akamai Is Solving Distributed AI Inference

[00:00 - 00:50] Introduction to AI Bottlenecks: Swapnil Bhartiya opens the discussion with Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai, highlighting how latency is replacing GPU availability as the primary challenge in production AI.

[00:50 - 02:46] Evolution of AI Workloads: Jon Alexander traces the progression from early model training and simple single-turn chatbots to complex, real-time agentic systems executing tasks at machine speed.

[02:46 - 05:08] Architectural Challenges of Multi-Agent Systems: Jon Alexander compares multi-agent workflows to microservice architectures, detailing how multi-step loops and distributed third-party dependencies compound latency in centralized clouds.

[05:08 - 06:58] GPU vs. CPU Bottlenecks: Jon Alexander explains that up to 90% of execution time in agentic workflows occurs off the GPU during CPU-bound tasks like tool calls, file writes, and code execution.

[06:58 - 09:09] Real-World Latency Demands: Swapnil and Jon discuss the stark difference in latency tolerance between human-facing chatbots and real-time physical AI, autonomous vehicles, and robotic systems.

[09:09 - 11:48] Parallels to Web History & CDN Innovation: Jon Alexander draws comparisons between today's AI inflection point and the 1996 web era, explaining how Akamai's early CDN innovations address modern distributed AI challenges.

[11:48 - 14:16] Orchestration & Workload Placement: Jon Alexander outlines Akamai's strategy for orchestrating inference across CPUs and GPUs, optimizing placement near data and end-users to reduce idle times.

[14:16 - 15:58] Crawl, Walk, Run Strategy for Enterprises: Jon Alexander presents a 3-step enterprise roadmap: Crawl (solve real business problems with guardrails), Walk (scale for security and load), and Run (optimize for SLOs and performance monitoring).

Well, it's true that the biggest challenge in AI is securing GPUs, which are hard to find. But when it comes to production, the real bottleneck is latency. But as AI moves from simple chatbots that answer simple questions into complex agentic systems that take actions on your behalf, autonomously centralized cloud architectures are starting to choke. When AI agents coordinate and make real-time decisions, latency becomes an architectural constraint. And the future of production AI requires a totally different approach. It requires a distributed approach. And to explain this, today we have with us once again, John Alexander, SVP of Product for the Cloud Technology Group at Akamai. John, it's great to have you on the show.

Great, Swap. Yeah, good to be back on the show. Great to chat again. Let's talk about latency. Why is it becoming the defining infrastructure problem for production AI, especially as enterprises transition from simple chatbots to complex agentic systems that work autonomously? That's right.

Yeah, no, and I think that's the key point. Like, we're on the cusp of a significant change in the types of AI workloads that our customers are looking to deploy. And so, again, if we think back a couple of years, a lot of the focus was on developing the model. So big training clusters were being developed, aggregating huge amounts of data into centralized infrastructure to create these amazingly powerful foundational models. Then we started to shift into application of AI, relatively simple workloads, simple in terms of the application path, in terms of the capabilities that are being deployed, but primarily things like chatbots, customer support, relatively simple, sort of one-dimensional interactions that end users were having. Today, one of the most prominent uses for AI that we see is for coding. So we're starting to get into much more sophisticated use cases, but again, deployed in a relatively constrained environment.

What we're seeing our customers talking about now is moving into more real-time AI where it becomes pervasive throughout their business processes. And they're looking at deploying agents to handle many of the tasks that previously were handled by other algorithms and applications that they had or even humans. And so they're looking at a much broader range of use cases. And this is where... inferences evolving and becoming more real-time and needing to make decisions at machine speed, not at human speed, is the key transition that we're seeing.

When multi-agent workflows are forced through traditional centralized cloud architecture, what actually starts to break? As we move from relatively simple question-answer type interactions with AI and we're moving into agentic workflows, you're moving into a multi-step looping process where the agent is making multiple requests to the model. And that is a multi-step process. I personally think my background is in internet architecture, so I think about this as a lot like a microservices type deployment where An application is built with decomposed into all these different components, and all of these pieces have a separate purpose that needs to be coordinated to achieve a specific business outcome. In an agent, you have kind of a similar concept where you have multiple components of the system, different tools, different models that you're talking to, and different interactions at different steps of the process, which are all required to achieve the outcome. So it's not a single shot where I just fire off one response, get one answer, and deliver that back to the user. I go around maybe 100 times, maybe 200 times, I call these different systems, I pull in data from APIs, I generate code that I might execute. All of these are multiple steps. And depending upon where these are running, it can compound the latency. Now, a centralized cloud can work well if all of the data, if all of the users, if the models, if everything is located close. Often what we're finding is that isn't the case. Users are distributed, data is distributed, and often the systems that you're talking to, the APIs that you're calling, the tools that you're calling, these third-party systems, they're distributed as well. And this all pulls us back to A centralized system is not going to be the optimal location to run all of those components of the solution. And so in these multi-agent workflows, we're often finding that they don't work well for real world applications where distribution of data of users and the tools exist.

No matter who you talk to, the conversation always comes back to GPU capacity and token generation. Is that focus right or is it incomplete? What are organizations actually missing and what should they be focused on as we shift from generating answers to taking actions? This is one of the big items that we've identified as we're seeing customers deploying real-world agentic applications. GPUs are incredibly important. They're one of the most expensive and constrained parts of the architecture, so capacity there is limited. You want to make sure you're using it effectively. But the way that we're seeing these agentic applications built, a large portion of the execution duration is isn't waiting for the GPU to generate tokens to generate the response. It's actually in the tool calling. So the agent is calling out to third party systems, is writing files to the local file system, it's executing code. So it's doing all of this other work. And that can be up to 90%of the overall execution time isn't on the GPU. It's actually running on CPUs where it's doing this other work. And so You definitely see a lot of benefit from the GPUs being fast. That can be a bottleneck. But increasingly, we're seeing that the end-to-end latency is being controlled by all these other parts of the agentic system. And so orchestrating that and coordinating between the CPUs and the GPUs strategically making sure that you're leveraging that infrastructure in the right locations, making sure you've got connectivity to external systems. All of that is as important, if not more important, for improving the performance in these real-world scenarios.

What are the biggest friction points organizations face in today's infrastructure as they move from simple question and answer to multi-step, multi-agent system, pulling from APIs, data sources, and knowledge bases? I think you hit on the key point. We're moving from relatively simple, like almost one-dimensional problems into a multi-dimensional space. Like all of the use cases that you talked about, this is what we see customers starting to build. And again, we're at the very beginning of the real-world adoption of agents, like these complex systems. But there's a disconnect. What those real-world agents need So they need the ability to call tools. They need access to a file system. They need to be able to execute code. They need to be able to coordinate across different models running on different cloud providers or on different AI labs. All of that needs a coordination of infrastructure that needs to be distributed. And so... Putting or trying to force all of that into one central location is not the optimal architecture. It means that you're compounding the latency on all of these iterations, all of the different sequences in that flow. And your end-to-end duration goes up exponentially. So if one iteration on the loop takes 100 milliseconds, you've got a loop that takes 100 milliseconds. iterations, that's 10 seconds. So you've got 10 seconds end-to-end. Now, if this is a human talking to a chatbot, maybe we're willing to wait 10 seconds. If this is a physical AI, if this is a robot trying to make a decision in real time, if this is an autonomous vehicle, 10 seconds is a lifetime. It's way too long and that task is going to fail.

Today's AI landscape reminds me of the early days of the web. Big promise, real performance problems and that's exactly the problem Akamai solved back then by inventing CDN and bringing content closer to users. Are we at the same inflection point when it comes to AI? And if so, how should we rethink where workloads should run? closer to users and data, or they should be centralized? Yeah, no, I think there's a lot of parallels to the evolution of the web. So the web was the killer application for the internet and technology. If we rewind 30 years now, I think we're at that same point. So this is where we are today in 2026 is equivalent to the web in 1996, right at the very early stage. I think the early adopters are seeing the potential. They're building the experiences, but we haven't achieved that sort of pervasive deployment. I think that's what's coming. And this is very much kind of the problem that Akamai set out to solve 30 years ago, solving what at that time we called the worldwide weight. Getting capacity from these or getting content from these central servers to users was a challenge. Like there just wasn't the physical capacity in the Internet to achieve that. And therefore... Akamai solved this through creating kind of the content delivery network, and then we were able to help those sites deliver the experiences that today are common and kind of expected.

I think there's very similar transformation that's coming today with AI. We're moving from this phase of experimentation early adoption, and now we're starting to move into real commercialization and the technology starting to become pervasive and really solving business problems. So chatbots were They were great proofs of concept for the technology, but they didn't transform the business economics or change kind of the fundamental capabilities of companies. I think the opportunity that we're seeing going forward as people are deploying agents, they can fundamentally change how the business operates, their ability to innovate, the scale they can operate. There's lots of potential benefits. And so that's what we're seeing is many orders of magnitude more adoption. as we look forward. Now, how do we achieve that? This is the challenge. I think if you follow the tech press, I read far too many stories every single day about new data sensors being deployed, kind of supply chain constraints, scarcity of memory, limitations in the overall supply to meet the demand that's coming. And that's where I think Akamai sees an opportunity for us to help scale out the adoption of inference. And so part of this is making sure that we can achieve the kind of ubiquitous availability of inference, make sure that we're able to deliver the performance expectations that the customers expect. And so like the real world latency budgets that people have, like that's a fundamental requirement that we need to be able to meet. But we also need to make sure that it's cost effective, that it's reliable, resilient, it can meet the kind of compliance and security requirements that customers have. That's multiple dimensions of optimization. It needs... A slightly different set of technologies than Akamai originally identified with CDN, but some of them are very similar. One of the core problems that we see is this idea of orchestration across the inference platform. And so we believe that the solution is to leverage infrastructure that is in the right location with the right capabilities for the right workload. That doesn't mean that everything needs to be distributed. Some pieces can be centralized, but some pieces do need to be close to either where the data sits, where the GPUs are, or even where the end user is. And so optimizing for placement of workloads is scheduling between CPUs and GPUs to handle those handoffs of the tool calling. Make sure you're not leaving GPUs idle as you're waiting to pull data from third parties or you're executing code locally to generate a response. That's a big optimization problem that is very, very similar to a lot of the early work that Akamai did around kind of routing and load balancing, placement of workloads, core technology that really helped us scale out the early stages of the internet that we think is going to be very relevant here. So similar problems that are very close to the DNA of Akamai.

Infrared providers like Akamai continue to evolve. How should enterprises rethink their own architecture to prepare for the shift towards distributed AI execution? I think it's a crawl, walk, run. I mean, typically what we're seeing is people are experimenting. They're building the application. And I think one of the key things is focus on Are you solving a real problem? There's a lot of agents that don't end up delivering value. And therefore, confirm that you're really solving a business problem. Make sure you've got the right guardrails. Make sure you've got the right controls in place. That's the first thing. So that's the crawl phase. Then the walk phase is like, hey, how are you going to scale this out? Make sure that this is able to handle the... The load that's coming, make sure it's ready for real-world deployment, make sure it's secure. That's key. So that's the walk phase. The run phase, though, is where you get into the optimization around performance and make sure it's meeting the expectations. Again, back to some of the examples that we talked about, so for physical AI, for for an autonomous vehicle, even for commerce. There are expectations that end users have or that systems have. And if you're not meeting those expectations, the system's going to fail. It's not going to meet the business objectives. And so that's the final level of like, hey, make sure that you're actually achieving the SLOs. Make sure you've got really clear expectations performance monitoring benchmarks that you're trying to target and then you've activated to achieve those. And that's really where Akamai can help you scale out and scale up. the performance to meet those objectives.

John, thank you so much for joining us today and sharing these insights on latency and what it really means for the future of AI infrastructure in production. For everyone else watching, please make sure to check out Akamai and their blog to see how this market is evolving and how you should be preparing. John, thank you so much for your time and I look forward to chatting with you again. Great. Thank you very much, Swap. Cheers.