중앙 집중식 AI 추론의 숨겨진 비용

**Swapnil Bhartiya:** For the last couple of decades we have been talking about one thing and one thing only, cloud. Centralize everything, consolidate workflows, move to the cloud, and centralized cloud did become single answer to all your problems. But as AI moves from training into real time agentic inferencing, that model is hitting a wall. Forcing complex workflows into a central core creates bottlenecks that hurt performance and drive up cost. Architecture now has to follow the user and Akamai is helping enterprises build a true compute continuum from core to edge to solve exactly this problem. And today we have with us once again Jon Alexander SVP of Product for the Cloud Technology Group at Akamai. John, it's good to have you back on the show.

**Jon Alexander:** Great swap. Yeah, good to be here.

**Swapnil Bhartiya:** We have spent decades moving everything to the cloud and it solved a lot of problems. But when it comes to AI inferencing specifically it demands a fundamentally different architectural approach. Why is the centralized cloud model not working for Agent Ki?

**Jon Alexander:** It's a good point. And I mean the cloud has been massively transformational for many enterprises. So the ability to rent infrastructure versus own infrastructure, the ability to consume infrastructure as a service versus as infrastructure like these have been transformed. But when you think about AI, we're going through a big transformation in terms of the types of applications that we're seeing running on top of the infrastructure. So a couple of years ago a lot of focus was on training models and so a lot of the discussion and architecture focus was on how do we support very large clusters of coherent GPUs so we can connect these GPUs together with very high speed interconnects to create this large coherent cluster with high speed interconnects between each of the GPUs, they can share the memory that's available to them. And so memory is really the constraint that we're optimizing for. And so we create these very closely connected clusters to achieve that high performance computing infrastructure that allows us to create these amazing foundational models that we're all familiar with today that started to shift as we put more of our focus into inference. So running workloads on top of those models, early inference workloads were chatbots. They weren't particularly complex. And so those could be deployed into centralized infrastructure. And again, having those run on centralized infrastructure has got some values of you got massive compute all in one location. You can often drive economies of scale there. You can drive efficiency in terms of placing workloads and optimizing usage across many users. We're at the CUSP now of the next wave of adoption where we're moving into the rise of agents. So OpenClore, I'm trying to remember exactly how long ago it was. Six months ago. I think it kind of exploded onto the scene really kind of started the main wave of adoption around kind of agents. And what we're seeing from our customers now is as they think about agents, these aren't sort of one dimensional chat type applications. They're multi stage, multi turn applications that are calling tools. They're looping multiple times around these kind of inference loops. They're pulling in huge amounts of context to, to give the best answers possible. And physical geography is becoming a really important constraint in these architectures.

**Swapnil Bhartiya:** When enterprises try to force these new agenting workflow through that same legacy centralized infrastructure, what exactly breaks? Because often they don't realize it. All they see is performance rate, higher bill or runaway token usage. But what is actually breaking underneath?

**Jon Alexander:** I think that the kind of simple answer is like, hey, the real world isn't centralized data users, the devices, the APIs and all the decisions you're making. They're distributed. Like when we're delivering a service in production, it doesn't all sit in one data center. And not everyone is physically close to that data center. If you think about what an agent really looks like, you've got a combination of the actual agent which runs on top of CPU infrastructure. It calls out to models that are running on GPUs. Those can often be multiple models. You can have different types of models that it's calling. Maybe it's calling OpenAI, maybe it's calling Anthropic, maybe it's calling open source models that are hosted on different GPU infrastructure elsewhere. So even the models aren't all co located. But increasingly what we're seeing is the agents are dominated by the tools that they're calling. The power of agents is around the memory system that it's got, the access to, the file system, the ability to execute code in some kind of a sandbox, generate output from the code that it's running to call out to third parties. So to call APIs, to call MCPS, to call out to these external systems so it can interact with the real world. So this is the real power of an agent. And as it's interacting with the real world, that's not all in one location. And so this is the problem that we're seeing is agents have to interact with the real world. That real world is distributed. Forcing all of that to run in one location means that you're compounding latency. And these agentic systems, they run in a loop, they run multiple interaction or multiple, multiple times around the loop and you're compounding that latency on every loop. And so even adding 100 milliseconds of latency, if you're looping 100 times, that's 10 seconds of latency that you've added. Which might be okay for a chatbot. Maybe a human's willing to wait 10 seconds for an answer to come back. But for physical AI, for any type of real time system, 10 seconds is way too long. It's a lifetime.

**Swapnil Bhartiya:** In traditional cloud we talk about CPUs, storage, networking, memory, but when it comes to AI, we mostly talk about GPU capacity and token generation speed. What metrics are missing from that conversation? The ones that actually matter for real world AI agent performance?

**Jon Alexander:** Yeah, no, this is kind of a key optimization that we've been looking at again is as we've been working with customers, is that they're building these agents. The GPU is really important like the, and we've been deploying a lot of the Blackwell GPUs and the speed of those GPUs. So time to first token, the throughput, the number of tokens they can generate per second, those are incredibly important metrics that we're seeing that customers are optimizing for. But for these applications, in a gentic application, often what we're seeing is up to 90% of the overall task execution isn't on the GPU, so isn't kind of generating tokens. The majority of that time is actually in the tool calling portion of the agent. So calling out to these third party system, calling external APIs, pulling data into, pulling in context, pulling in real time information that's needed to generate the answers. This is what's actually generating the majority of the end to end latency for the task to be successful and so optimizing for the end to end system is what's important. So GPUs are a really important part, but often what we're seeing is there's a really big portion of the end to end time that's not on the GPU at all. So you can have the most powerful fastest model running on the latest gpu. But if your compute is in the wrong location, if you don't have good connectivity out to the tools, if you're not running with good connectivity to storage and other data systems that you need, your system is still going to be slow and isn't going to meet the expectations of the Users.

**Swapnil Bhartiya:** Akamai has been talking about a compute continuum for a while now. What does that actually look like in practice? Spanning from centralized data centers all the way to the network edge. When we talk about AI, this is

**Jon Alexander:** a big optimization problem. So something that we've been working on is what we call AI grid Orchestrator. So Nvidia's talked a lot about their kind of AI grid reference architecture. So as we move from kind of centralized token factories to kind of these real world systems that can deploy applications for kind of real time token generation, having the ability to have that in a grid distributed infrastructure is important, but that needs orchestration. So you've got all of these different locations where you can run workloads. And what we find is there are certainly very large models that need to be deployed on powerful GPUs, often clusters of TPUs. Makes sense to have that in more centralized locations, so you don't have those deployed in hundreds of locations. It's very hard to maintain utilization of highly distributed clusters of GPUs like that. But then what we see is for the cpu, where the agent is running, where we're deploying sandboxes, where we're spinning up containers to kind of run kind of code execution, that's something that is isolating sandbox to an individual user, an individual session. And it makes sense as we look to optimize end to end performance, manage cost for that to be close to the end user. That's where what we're seeing is the separation between CPU and GPU is often beneficial in these types of distributed systems. We can place those different components of the application in the right physical location to drive end to end latency down and then also make sure we're driving effective utilization of that infrastructure. And then key requirement for the Orchestrator is to coordinate between the CPU and the GPU and schedule the execution effectively across those different components of the of the infrastructure.

**Swapnil Bhartiya:** Before enterprises commit to large scale long term inference deployments, what should they be expecting and demanding from their AI infrastructure?

**Jon Alexander:** It's an important question. And so like generally I think some of the kind of cloud buying criteria are as important here. And so think carefully about sort of portability. Think about like how you can actually have flexibility around where you deploy applications. You don't want to tie into one vendor's technology stack running on top of cloud. Native technologies like Kubernetes, obviously important that gives you flexibility around where you deploy. I think making sure that you've got elasticity, think about how it's going to scale up often. A lot of applications work great for that demo. And then as soon as you've got 100 users, 1,000 users, 10,000 users, it starts to look very, very different. And eventually, when it's wildly successful and you've got tens of billions of users, that's when things can get really challenging. And so think about how can you have that architecture that will scale. So you don't want to be provisioning a huge amount of capacity upfront, but you want to have flexibility to grow. You want to have the ability to expand into new markets, and you want to be able to do that with low friction. You don't want to be tied in into one infrastructure that's going to give you penalties for moving the data. So again, a lot of the way I think about this is AI is really a data problem. It's a context management problem. It's making sure you've got the right context at the right time, loaded into memory. But having that available to wherever you want to run the application is super important. So you don't want to tie into a database or a storage infrastructure that can only be accessed through, through one provider. And then I think last piece is think about the performance. We talked a lot here about latency. Cost is an important dimension. A lot of people think about managing costs. A lot of horror stories going around at the moment about people blowing their entire token budget in a few days with kind of uncontrolled costs. Obviously that's, that's an important factor. But thinking about the performance budget that you have, what are the latency targets that you need to hit as the scales out and how are you going to achieve that? Because a lot of people are starting again, just simple deployments. One location may be running on a local machine or just on a single server. Everything works great when it's one user. How's it going to work when it's much more, more widely adopted? That's kind of a key area to think about. How do you scale out to achieve that performance?

**Swapnil Bhartiya:** How hard is it for organizations to untangle their current setup and move towards a distributed model? And how can Akamai make that easier so enterprises can focus on business outcome and not all the plumbing involved?

**Jon Alexander:** Again, I think this is one of the kind of key decisions that everyone has as part of their cloud journey as well. So I'm not sure this is different than people have seen before. If you make the right architectural decisions upfront, if you think about where you want to end up, you can architect in the cloud to be able to deploy and run in many locations. But if you make a certain set of decisions early on, if you tie your application to a single availability zone or a single region or a single database location, then you really end up with challenges. And I think that the same is true for AI. We've heavily invested in serverless as one of the kind of fundamental presentations of compute and infrastructure that we think is going to be incredibly important for AI adoption. We're working a lot right now in our roadmap on building a serverless inference platform that customers can just consume as a service and they don't need to think about where something's running, how many locations, they don't need to think about that placement. So I think that's the type of advice I would give is if you architect and tie to just one location, yeah, you're going to have problems. You're going to have to unwind a lot of those decisions you made early on. If you're making decisions where even if you are deploying into one region on day one, but if you've got a plan of how you could scale this out to 2 locations, 5 locations, 10 locations, 20 hundred locations, you'll be fine. So again, it's planning ahead. You don't need to build all of that infrastructure on day one, but at least having a pathway is going to give you the optionality later on.

**Swapnil Bhartiya:** Is this distributed approach specific to certain industries, highly regulated, real time or latency sensitive ones, or should it become a standard model for AI inferencing across the board?

**Jon Alexander:** Yeah, I think this will have broad applicability. And so not every AI inference use case needs to be real time or low latency. It doesn't need that performance. So there are certainly applications you can think of that will have a high tolerance to latency. Certainly things like batch processing, those aren't time sensitive, so those can be run in centralized infrastructure. And so not everything needs to be distributed. But a large number of use cases are performance sensitive. Certainly as you're thinking about anything that a user is in the loop, like a user's waiting for a response, human tolerance is finite. Machine tolerance is going to be even more sensitive. Again, if we're thinking about machines needing to take action based on the output of an agent, that's where I think we're going to see very low tolerance for latency. So that's where I think we're going to find that it's going to require very, very sensitive latency targets to be achieved.

**Swapnil Bhartiya:** John, thank you so much for joining me. Today and shedding light on this crucial shift in AI architecture. The compute continuum is not just a concept, it is quickly becoming a necessity. So those who are watching, please go and check out Akamai and the blog post to understand how they are helping. Enter Build for what's coming next. John, once again, thanks for your time and I look forward to chat with you again.

**Jon Alexander:** Thank you. Always a pleasure to be here.