[00:00–01:10] Einführung in KI‑Latenzengpässe: Swapnil Bhartiya stellt Jon Alexander vor, um zu erörtern, wie Produktions‑KI bei zentralen Cloud-Architekturen an ihre Grenzen stößt, da sich Workloads von einfachen Chatbots hin zu autonomen, komplexen Multi-Agenten-Systemen verlagern, die Echtzeitentscheidungen erfordern.
[01:10–02:18] Weiterentwicklung von KI‑Inferenz-Workloads: Alexander skizziert die Entwicklung der KI: von frühen zentralisierten Trainingsclustern und einfachen, eindimensionalen Chatbots hin zu allgegenwärtigen, agentischen Echtzeitsystemen, die darauf ausgelegt sind, mit Maschinengeschwindigkeit zu arbeiten statt mit menschlicher Latenztoleranz.
[02:18–03:22] Microservices-Parallele und Latenz aus Schleifen: Alexander vergleicht moderne agentische Workflows mit dezentralen Internet-Microservices und erklärt, wie mehrstufige Agentenschleifen die Latenz vervielfachen, wenn sie zwingend über einen einzigen zentralen Standort ausgeführt werden, während sie mit verteilten Daten, Nutzern und APIs kommunizieren.
[03:22–04:22] Verarbeitung abseits der GPU: Die Referenten hinterfragen den engen Fokus der Branche auf reine GPU‑Kapazität. Alexander erklärt, dass bis zu 90 % der gesamten Aufgabenausführung eines Agenten auf CPUs erfolgen – für Toolaufrufe, die Bearbeitung des lokalen Dateisystems und die Codeausführung.
[04:22–05:14] Probleme für autonome Systeme: Alexander erläutert, wie die Tatsache, dass mehrdimensionale Agenten gezwungen sind, eine zentralisierte Schleife hunderte Male zu durchlaufen, geringfügige Verzögerungen im Millisekundenbereich in eine massive Verarbeitungslatenz von 10 Sekunden verwandelt – was bei zeitkritischen Aufgaben wie Robotik oder autonomen Fahrzeugen fatal ist.
[05:14–06:17] Workload-Transformation und die Analogie zum Internet 1996: Alexander zieht historische Parallelen zwischen der KI‑Landschaft 2026 und dem frühen Internet im Jahr 1996. Er beschreibt, wie Akamai damals CDNs erfunden hat, um das Problem des „World Wide Wait“ zu lösen, und merkt an, dass KI einen ähnlichen strukturellen Wandel durchlaufen muss.
[06:17–07:14] Orchestrierung, Planung und Edge-Platzierung: Alexander definiert die Strategie, die Akamai zur Skalierung der Inferenz verfolgt, und betont dabei ein Orchestrierungsmodell, das zentrale Komponenten mit Edge-platzierten Knoten in Einklang bringt. Ziel ist es hierbei, die Auslastung der Infrastruktur zu maximieren, die Übergabe von CPU zu GPU zu optimieren und zu verhindern, dass GPUs ungenutzt bleiben.
[07:14–08:08] Architekturentwicklung über eine „Crawl, Walk, Run“-Strategie: Alexander stellt ein Implementierungs-Framework für die Unternehmensarchitektur vor: „Crawl“ (Krabbeln): die Überprüfung des geschäftlichen Nutzens und die Einrichtung von Leitplanken; „Walk“ (Gehen): die Gewährleistung von Sicherheit und Skalierbarkeit; „Run“ (Laufen): die Architekturgestaltung für strenge Performanceziele und SLOs.
[08:08–08:34] Schlusswort: Bhartiya beendet den Beitrag, indem er sich bei Alexander bedankt und die Zuschauer dazu anregt, den Akamai-Blog zu besuchen, um dynamischen Marktveränderungen immer einen Schritt voraus zu sein.
Wichtige Erkenntnisse
-
Hohe Latenz in Schleifen:
-
Workflows mit mehreren Agenten arbeiten mit mehrstufigen Schleifen, nicht mit einzelnen Prompts. Wenn diese verteilten Komponenten (APIs, Tools und Nutzer) über zentrale Rechenzentren laufen, addieren sich die Latenzen über hunderte Iterationen auf. Und das kann zu einer Verarbeitungszeit von bis zu 10 Sekunden führen, was für physische KI untragbar ist.
-
CPU‑Ausführung als Engpass:
-
Auch wenn sich die Branche stark auf die Geschwindigkeit der GPU‑Tokengenerierung konzentriert, erfolgen bis zu 90 % der Aufgabenausführung eines KI‑Agenten nicht über GPUs, sondern über normale CPUs, darunter die Verarbeitung von Toolaufrufen, die Änderung lokaler Dateisysteme oder die Ausführung externer Code-Sandboxes.
-
Parallelen zum Internet im Jahr 1996:
-
Die aktuelle Phase der KI‑Kommerzialisierung im Jahr 2026 ähnelt der Internetlandschaft aus dem Jahr 1996: So wie Netzwerke zur Inhaltsbereitstellung (Content Delivery Networks, CDNs) entwickelt wurden, um die Engpässe des ursprünglichen zentralen Webservers zu beseitigen, erfordert moderne KI eine verteilte Architektur, um die Anforderungen der breiten Agentenbereitstellung zu bewältigen.
-
Strategische Workload-Platzierung:
-
Die Skalierung der KI‑Inferenz erfordert ein hybrides Orchestrierungsmodell. Die Infrastruktur muss segmentiert werden, damit rechenintensive analytische Modelle zentralisiert bleiben können, während Mikroaufgaben rund um Kommunikation, Toolaufrufe und Datenerfassung näher am Nutzer oder am Speicherort der physischen Daten ausgeführt werden.
-
Performanceorientierte Architektur:
-
Unternehmen, die ihre Agenteninfrastruktur skalieren, müssen von einfachen Experimenten zu strikter Performance-Budgetierung übergehen. Um die erforderlichen Servicelevel-Ziele (Service-Level Objectives, SLOs) zu erreichen, muss die Übergabe von CPU zu GPU präzise geplant werden, damit teure GPU‑Cluster niemals inaktiv sind, während sie auf externe API‑Daten warten.
Zusammenfassung mit Zeitstempeln
Videotranskript
Videotranskript
**Swapnil Bhartiya:** While it's true that the biggest challenge in AI is securing GPUs, which are hard to find, but when it comes to production, the real bottleneck is latency. But as AI moves from simple chatbots that answer simple questions into complex agentic systems that take actions on your behalf autonomously, centralized cloud architectures are starting to choke. When AI agents coordinate and make real time decisions, latency becomes an architectural constraint. And the future of production AI requires a totally different approach. It requires a distributed approach. And to explain this, today, we have with us once again Jon Alexander, SVP of Product for the Cloud Technology Group at Akamai. Jon, it's great to have you on the show.
**Jon Alexander:** Great swap. Yeah, good to be back on the show. Great to chat again.
**Swapnil Bhartiya:** Let's talk about latency. Why is it becoming the defining infrastructure problem for production AI, especially as enterprises transition from simple chatbots to complex agentic systems that work autonomously?
**Jon Alexander:** That's right, yeah. No, and I think that's the key point. Like we're on the cusp of a significant change in the types of AI workloads that our customers are looking to deploy. And so again, if we think back a couple of years, a lot of the focus was on developing the model. So big training clusters were being developed, aggregating huge amounts of data into centralized infrastructure to create these amazingly powerful foundational models. Then we started to shift into application of AI into relatively simple workloads. Simple in terms of the application, powerful in terms of the capabilities that are being deployed, but primarily things like chatbots, customer support, relatively simple sort of one dimensional interactions that end users were having with AI. Today, one of the most prominent uses for AI that we see is for coding. So we're starting to get into much more sophisticated use cases. But again, deployed in a relatively constrained environment. What we're seeing our customers talking about now is moving into more real time AI where it becomes pervasive throughout their business process. And they're looking at deploying agents to handle many of the tasks that previously were handled by other algorithms and applications that they had, or even humans. And so they're looking at a much broader range of use cases. And this is where inference is evolving and becoming more real time. And needing to make decisions at machine speed, not at human speed is the key transition that we're seeing when multi
**Swapnil Bhartiya:** agent workflows are forced through traditional centralized cloud architecture. What actually starts to break as we
**Jon Alexander:** move from relatively simple question, answer type interactions with AI and we're moving into agentic workflows, you're Moving into a multi step looping process where the agent is making multiple requests to the model, and that is a multi step process. I personally think my background is in Internet architecture, so I think about this as a lot like a microservices type deployment where an application is built with decomposed into all these different components and all of these pieces have a separate purpose that needs to be coordinated to achieve a specific business outcome. In an agent you have kind of a similar concept where you have multiple components of the system, different tools, different models that you're talking to, and different interactions at different steps of the process which are all required to achieve the outcome. So it's not a single shot where I just fire off one response, get one answer and deliver that back to the user. I go around maybe 100 times, maybe 200 times, I call these different systems, I pull in data from APIs, I generate code that I might execute. All of these are multiple steps and depending upon where these are running, it can compound the latency. Now, a centralized cloud can work well if all of the data, if all of the users, if the models, if everything is located close. Often what we're finding is that isn't the case. Users are distributed, data is distributed, and often the systems that you're talking to, the APIs that you're calling, the tools that you're calling, these third party systems, they're distributed as well. And this all pulls us back to a centralized system is not going to be the optimal location to run all of those components of the solution. And so in these multi agent workflows, we're often finding that they don't work well. For real world applications where distribution of data, of users and the tools exist,
**Swapnil Bhartiya:** no matter who you talk to, the conversation always comes back to GPU capacity and token generation. Is that focus right or is it incomplete? What are organizations actually missing and what should they be focused on as we shift from generating answers to taking actions?
**Jon Alexander:** This is one of the kind of big items that we've identified as we're seeing customers deploying real world agentic applications. GPUs are incredibly important. Like they're one of the most expensive and constrained parts of the architecture. So capacity there is limited. You want to make sure you're using it effectively. But the way that we're seeing these agentic applications built a large portion of the execution duration, waiting for the GPU to generate tokens, to generate the response. It's actually in the tool calling. So the agent is calling out to third party systems. It's Writing files to the local file system, it's executing code, so it's doing all of this other work. And that can be up to 90% of the overall execution time isn't on the GPU. It's actually running on CPUs where it's doing this other work. And so you definitely see a lot of benefit from the GPUs being fast. That can be a bolt on it. But increasingly we're seeing that the end to end latency is being controlled by all these other parts of the Agentix system. And so orchestrating that and coordinating between the CPUs and the GPUs, strategically making sure that you're leveraging that infrastructure in the right locations, making sure you've got connectivity to external systems, all of that is as important, if not more important, for improving the performance in these real world scenarios.
**Swapnil Bhartiya:** What are the biggest friction points organizations face in today's infrastructure as they move from simple question answer to multi step, multi agent system pulling from APIs, data sources and knowledge bases?
**Jon Alexander:** I think you hit on the key point. We're moving from relatively simple, almost one dimensional problems into a multi dimensional space. All of the use cases that you talked about, this is what we see customers starting to build. And again, we're at the very beginning of the real world adoption of agents like these complex systems. But there's a disconnect what those real world agents need. So they need the ability to call tools, they need access to a file system, they need to be able to execute code, they need to be able to coordinate across different models, running on different cloud providers or on different AI labs. All of that needs a coordination of infrastructure that needs to be distributed. And so putting or trying to force all of that into one central location is not the optimal architecture. It means that you're compounding the latency on all of these iterations, all of the different sequences in that flow, and your end to end duration goes up exponentially. So if one iteration on the loop takes 100 milliseconds, you've got a loop that takes 100 iterations, that's 10 seconds. So you've got 10 seconds end to end. Now, if this is a human talking to a chatbot, maybe we're willing to wait 10 seconds. If this is a physical AI, if this is a robot trying to make a decision in real time, if this is an autonomous vehicle, 10 seconds is a lifetime, it's way too long. And like that, that task is going to fail. And so this is the fundamental disconnect that we're seeing with real world agents, their expectations are exponentially different than traditionally. What we've seen with chatbots, where it's been a more human centric, human latency tolerance.
**Swapnil Bhartiya:** Today's AI landscape reminds me of the early days of the web. Big promise, real performance problems. And that's exactly the problem Akamai solved back then. By inventing CDN and bringing content closer to users. Are we at the same inflection point when it comes to AI? And if so, how should we rethink where workloads should run closer to users and data, or they should be centralized?
**Jon Alexander:** Yeah, I think there's a lot of parallels to the evolution of the web. So the web was the killer application for the Internet. And if we rewind 30 years now I think we're at that same point. This is where we are today in 2026 is equivalent to the web in 1996. Right. At the very early stage, I think the early adopters are seeing the potential, they're building the experiences, but we haven't achieved that sort of pervasive deployment. I think that's what's coming. And this is very much kind of a problem that Akamai set out to solve 30 years ago. Solving what at that time we called the worldwide wait. Getting capacity from these or getting content from these central servers to users was a challenge. There just wasn't the physical capacity in the Internet to achieve that. And therefore Akamai solved this through creating the content delivery network. And then we were able to help those sites deliver the experiences that today are common and kind of expected. I think there's very similar transformation that's coming today with AI, we're moving from this phase of experimentation, early, early adoption, and now we're starting to move into real commercialization and the technology starting to become pervasive and really solving business problems. So chatbots were powerful. They were great proofs of concept for the technology, but they didn't transform the business economics or change kind of the fundamental capabilities of companies. I think the opportunity that we're seeing going forward as people are deploying agents, they can fundamentally change how the business operates, their ability to innovate, the scale they can operate. There's lots of potential benefits. And so that's what we're seeing is many orders of magnitude more adoption
**Swapnil Bhartiya:** as
**Jon Alexander:** we look forward now. How do we achieve that? This is the challenge, I think, if you follow the tech press, I read far too many stories every single day about new data centers being deployed, kind of supply chain constraints, scarcity of memory limitations, in the overall supply to meet the demand that's coming. And that's where I think Akamai sees an opportunity for us to help scale out the adoption of inference. And so part of this is making sure that we can achieve the kind of ubiquitous availability of inference, make sure that we're able to deliver the performance expectations that the customers expect. And so the real world latency budget that people have, like, that's a fundamental requirement that we need to be able to meet. But we also need to make sure that it's cost effective, that it's reliable, resilient, it can meet the kind of compliance and security requirements that customers have. That's multiple dimensions of optimization. It needs a slightly different set of technologies than Akamai originally kind of identified with cdn, but some of them are very similar. And like one of the pieces, maybe just to kind of finish with is one of the core problems that we see is this idea of orchestration across the. Across the inference platform. And so we believe that the solution is to leverage infrastructure that is in the right location with the right capabilities for the right workload. That doesn't mean that everything needs to be distributed. Some pieces can be centralized, but, but some pieces do need to be close to either where the data its, where the GPUs are, or even where the end user is. And so optimizing for placement of workloads, scheduling between CPUs and GPUs to handle those handoffs of like the tool calling, make sure you're not leaving GPUs idle as you're waiting to pull data from third parties, or you're executing code locally to generate a response. That's a big optimization problem that is very, very similar to a lot of the early work that Akamai did around routing and load balancing, placement of workloads, Core technology that really helped us scale out the early stages of the Internet that we think is going to be very relevant here. So similar problems that are very close to the DNA of Akamai, infrastructure providers
**Swapnil Bhartiya:** like Akamai continue to evolve. How should enterprises rethink their own architecture to prepare for the shift towards distributed AI execution?
**Jon Alexander:** I think it's a cruel walk, Ron. I mean, typically what we're seeing is people are experimenting, they're building the application. And I think one of the key things is focus on are you solving a real problem? Like there's a lot of agents that don't end up delivering value and therefore confirm that you're really solving a business problem. Make sure you've got the right guardrails, make sure you've got the right controls in place. That's the first thing. So that's the crawl phase. Then the walk phase is like, hey, how are you going to scale this out? Make sure that this is able to handle the load that's coming, make sure that's ready for real world deployment, make sure it's secure. Like that's key. So that's the walk phase. The run phase though, is where you get into the optimization around performance and make sure it's meeting the expectations. Again, back to some of the examples that we talked about. So for physical AI, for, for an autonomous vehicle, even for commerce, there are expectations that end users have or that systems have, and if you're not meeting those expectations, the system's going to fail, it's not going to meet the business objectives. And so that's the final level of like, hey, make sure that you're actually achieving the SLOs. Make sure you've got really clear performance monitoring benchmarks that you're trying to target and then you've architected to achieve those. And that's really where Akamai can help you scale out and scale up the performance to meet those objectives.
**Swapnil Bhartiya:** John, thank you so much for joining us today and sharing these insights on latency and what it really means for the future of AI infrastructure in production. For everyone else watching, please make sure to check out Akamai and their blog to see how this market is evolving and how you should be preparing. John, thank you so much for your time and I look forward to chat with you again.
**Jon Alexander:** Great, thank you very much. Cheers.