推論をどこで実行するかが重要な理由

**Swapnil Bhartiya:** Today, most companies, or most of us, think that the hard part of AI is building the model. It is not. The hard part is running it in production without cost spiraling out of control. You have to look at egress, surprises, latency wall, GPU capacity that you cannot predict or plan for. And here is what makes this even worse. Most teams cannot even track whether their AI workloads are scaling efficiently or are just getting more expensive. And today we have with us Ari Weil, VP of Product Marketing at Akamai, to break down the real economics of AI inference and why where you run it changes everything.

**Ari Weil:** Absolutely. So, first of all, it's great to be back on the program with you again, and I'm glad to have a chance to talk about the cost side of the equation when we think about AI applications and workloads. And I think to start off with, I would suggest that training is really the headline that everybody loves. It's driving these massive data center deals, big investments, and some of the circular investment that we've seen coming out of anything from chip manufacturers and card manufacturers to the major hyperscale clouds that are also building out their own infrastructure and procuring it from others. And even if we talk about the landscape of companies that are looking to do their first ipo, a lot of the focus has been on how large of a data center they're building out to build the next multi billion, multi trillion weight model. I think that training and that headline sort of factor is real. But if we think about it, inference is the bill that you get after you get all excited about training and it arrives every single time your model takes a breath, every time that you're not training, either pre training or post training or fine tuning that model. Inference is what happens when somebody needs your intelligence to be deployed. And so I think the highlight from our perspective, where we would start a conversation, is to say you need to stop budgeting your AI applications as line items to do with just the GPUs that you're procuring or just the applications that you're building. You need to budget, like building a distributed system, which is something that many large organizations have familiarity with, or at least there are representative architectures for in the marketplace, because you're going to hear people quote token price, how much does it cost per token for me to generate a workload? But the architecture is what ultimately, ultimately is going to decide your bill. And that's the part that people need to get more conversant on and take some of their cues from building cloud Native systems and to some extent edge native systems for the last several years.

**Swapnil Bhartiya:** When it comes to cost, most of the times when we look at and you will see, oh, output token, $5 sorry, input token, $5 output token. Depending on what you're using. If you're using Opus or Gemini Pro, you're looking at $130 per million token output tokens. But the thing is not as simple as that. There's a context window. The more you use it, those context, those pile up. So you even if you're sending five messages, it will also look at the last 20 messages. So that cost is not the message you send, it is your message plus five other message. Sometimes we use. Sometimes running it locally on a local machine is the cheapest and the most perfect best way. No latency. But not everybody can run it on their local machine. Can you talk about what role does physical location of AI inference matter or play a role when it comes to performance latency and most importantly, cost. Because cost is going to become a very big factor. Because as there are reports coming out, it's actually cheaper to hire an employee than to use AI.

**Ari Weil:** That is true. I mean, the first thing that I would start with, and you're absolutely right about the way that some people are modeling applications, distance does come at a cost. And a lot of times you're going to price that cost at zero. Because when you're modeling your application, you're thinking about things like what is the workflow that I'm enabling? Or what is the output of a given function or a given prompt supposed to be? And really focusing on that. But every time a request crosses a zone or a region or a cloud boundary, you're going to pay twice. Effectively, the first time you pay, you're going to pay for latency. How much is my time to first token? What is the overall overall time that I'm going to expend consuming all of the tokens from my request and response? The other one is the transfer fees that basically are not part of your token price. It's what you're charging anytime that you need to move data either between microservices, overall services or other cloud boundaries. And if you think about that latter piece, cross availability zone traffic can run a penny per gigabyte in each direction. So if I'm going to be sending data from a model to a set of GPUs that are deployed in a different part of my architecture, and then as part of that workload, I'm going to be sending either some of my EMBEDDINGS back to store, or if I have to go and call other services along the way, I am going to pay again, penny per gigabyte each time that I send that data across. Egress fees then if I send them across the Internet are 9 to 12 cents each. So there's a sort of tension between how do I build my application to take full advantage of the cloud capabilities that I know how to build on, or the different providers are making available. Does it make sense for me to keep my workload on a given cloud or where I have to cross cloud boundaries, Am I ready to incur that 9 to 12 cents per gigabyte? Understanding how my app needs to scale so things like latency and egress, that's not physics, that's not the speed of light. That is an architectural choice that you're billed for as though it were physics. The physics is what you have to think about. And again, where people are building AI applications and afraid that they don't have a frame of reference for what they're building, they can look at the last 10 to 15 years of building distributed cloud apps and understand what does data movement cost me and then really start to think to your point, when I move from a prototype to production, how many zones do I have to move from and to where does my application need to be? And one of the levers that we treat late in that cycle, late in that consideration phase, is cost. And I would argue that you have to really bring that into a lot of your user and scalability testings before you launch your app.

**Swapnil Bhartiya:** Now, I always argue that we should not look at AI any differently than we look at software. The closer it is to a user, the better performance you get. Of course, just because piece of code or software is massive or monolith, doesn't mean it is more powerful and more efficient in most cases is smaller. That's why we break it down. A smaller piece of code is better. Same way you don't need large language model with massive billions and billions of parameters. Sometimes smaller models work better and they are most cost efficient as well. Another factor is culture, as we have seen a lot of cultural shift. DevOps, DevSecOps, sorry, all the label, all the Personas are there in terms of AI. How much role do you see culture, process practices as we have adopted in the software world also play in AI world when it comes to taming cost?

**Ari Weil:** I mean to me we have to look at the different cost factors that we use to model our business. Many times people will look at what is my return on investment, what is my total cost of ownership, how quickly am I scaling and what is my cost of customer acquisition? Those are three numbers, sort of when you think about ROI, TCO and CAC that many people in SaaS environments are very used to looking at all the time. But if we think about where AI is a little bit different in this equation, it could be in the past, using in its recent past, but using a non AI centric workload that I would build in the cloud that I could scale more readily by deploying additional compute and deploying my microservices to additional environments. That would help me with architectural availability and resilience, but it could also help me with latency and scale up if I needed to put my microservices in more places to accommodate more users or more concurrent users. The challenge when it comes to AI is that if you think about what you would need to put in multiple locations, a lot of companies don't have the experience a lot of developers haven't yet figured out. As an example, can I afford to or does it make sense to deploy my model in multiple locations? Because there's a fair amount of data and you mentioned some of these very very large models that people are used to licensing or used to referencing versus a small fine tuned model that I might use for another purpose. Now the first thing that I would say is make sure that you understand what model you're using for which purpose. If I am an industry or a domain specific application developer and and I need to make sure that my app is going to give me quick responses, it's not going to hallucinate and I've got all of the specific information that I want coded into the model that I need. I might be looking at a fine tuned or a domain or an industry specific model that typically will be smaller than these frontier large language models. Because it doesn't need to be trained on all of the same things for me to address the use case for my domain or for this use case that then also has to do with the size of the model. So if I have a fine tuned model it might encourage me or I might be tempted to deploy that in multiple places. But here's where we can run into a challenge. You can fix a distance problem with more compute if it's just pure and sheer compute power. If I need to keep my model up to date, continuously learning, if I need to think about the size of that model and how I'm moving data into and out of it, how that is going to relate to where I place the workload and where my GPU or CPU infrastructure is going to be for it. Sometimes dealing with proximity by having copies of my model or my data in multiple places is just going to incur me a certain amount of additional cost because I'm going to pay twice for the same answer. Arguably, if I could come to one model from multiple locations where the user is making a request, and I know that ultimately I'm going to be serving the same answer. But I'm copying that model to overcome a latency barrier. Now I've incurred cost just to overcome latency by copying my model. And again, that could be okay. If you think about my model isn't really changing that much and I just need that proximity to maintain a latency budget. That's a consideration that you would make at design time or at architecture time, and then you can think about the cost that you would incur. But if I think about how workloads can become interactive and chatty, a lot of the interfaces for the use cases that modern AI inference are addressing are voice based, are sort of video to text based, or speech to text based or real time decisions like fraud interception for example, or a voice agent. Those types of things need low latency, they need real time responsiveness. And so if you think about concentrating that sort of a workload into centralized regions, it'll work great when people are close to that region. But your performance and your cost will suffer the further away you get because you will start to implement additional capabilities to overcome that latency, some of which could be as we were saying before copying that model. So I think the centralized cloud thesis makes sense if you have centralized functions that are sparsely or sparingly used, that are not directly in the request response timing for an end user request, or if that request doesn't require real time. But if you need something like anomaly detection or fraud detection, or to provide somebody with a real time query based on something where they've already been conditioned to anticipate a real time response, then you need to be thinking about how am I going to have the orchestration capability to decompose a request and go to where the right information or the right code is being managed for my use case. And that is an extra layer of architectural consideration that goes beyond just how much do my tokens and how much do my GPUs cost.

**Swapnil Bhartiya:** One thing more is that a lot of organizations, they are kind of used to thinking about AI infrastructure in terms of centralized big cloud and huge massive GPU clusters where does that model start to break down when AI gets embedded into real time applications?

**Ari Weil:** So if I think about things like real time personalization, if I'm in a retail or an E commerce sort of a workflow, my anticipation is that as I click, there's going to be a response from the application, whether that's a web application or a native mobile application, or even if I'm doing it through some sort of an agentic platform. So real time is real time. If I think about things like fraud detection I've mentioned a few times now, those things need to happen incredibly quickly because of the pace and the frequency of financial transactions when themselves. If I think about a voice agent or something that is doing real time logistics, for example, to send first responders to where they need to go and manage traffic events while they're going to their first response or responder locations, these are all things where something is waiting on that answer. And that answer can't be a continent or a country away, it needs to be nearby. And so if I think about the common use case of you fan out to try to go out and find the resources that you need, get all of the tools, get the data, get the context that's required, and then you fan in. The heavy part of that equation is that last mile of reaching the device that made the request in the first place. And those are sort of the things that the centralized model handles the worst. Because fanning out, fanning in and doing so quickly is something that works well in a centralized location for batch process processing or infrequent processing. But the reason that we built Akamai inference cloud using the Nvidia hardware stack, using the Nvidia enterprise software stack, was that we needed to run inference across an edge footprint that supported low single to double digit millisecond response times. And the way that Akamai has found that we are able to repeatedly hit those types of SLAs is by federating things out across our high performance network and making sure that there are points of presence to respond to those requests nearer to where the requests are being made and also nearer to where we have to apply some of the security considerations, leaving that heavier weight, slower centralized compute to be things like persistence and logging and the sorts of things that are not going to get in the way of that real time answer that either your robotics, your physical AI or your humans are looking for responses to

**Swapnil Bhartiya:** are there specific workloads or use cases that make the strongest case for running inference closer to users, devices or data sources, or you Feel that every use case, every workload is ideal if it is running closer?

**Ari Weil:** Oh no, definitely not. I mean if I were going to be training a model, pre training or post training of my model to do anything from adding weights to the model or embeddings that I might need later, if I'm training, paring down a model that I might use in a future workload, and I want to make it a smaller overall model for storage reasons and what I can hold resident in memory, we don't need those things to be federated. Those don't have to be real time and responsive in this way. If I've already got a model and the use case is, hey, I have an LLM backed mortgage application or mortgage approval application, then I can think about when the user logs in, they're going to provide me a certain amount of data, I might want to access my data about that user and maybe I'll perform some rag activity to do a credit check or to do a background check on that person. But all of it needs to come together in under, you know, five seconds, typically certainly under 30 seconds and provide me with an answer that says you are or not approved for this loan. You are approved for this much and here are the next steps to actually capitalize. And if I think about that in a workload, like an end to end workload sort of perspective, imagine that I am a real estate company and I might have anything from my online listings. I need to make sure that all of my listings render completely correctly, consistently. But certainly that cwicly piece needs to be top of mind for me. I need to ensure that in LLMs any zero click searches are actually having my page be cited more frequently than my competitors page so that people find me in the first place. Just finding that real estate listing already has a significant amount of right now architectural rework and application rework that companies are going through because of this phenomenon of zero click searches and LLM scrapers. Now that's just again finding my listing. Next thing is maybe I want to understand can I set up a open house time? Can I schedule a walkthrough with the realtor? Maybe I want to find out what it might take for me to actually get approved for a loan on this property. Those are different work streams that I'm building into my application and each of those as well needs to be real time. You would argue that scheduling something on the realtor's calendar is a pretty lightweight use case and you would be right. There's not much of a appeal for AI in that sort of a use case. But confirming somebody's eligibility and what level or what, what size of loan they could get is very much something that AI can help with. And it's very common in the financial services industry, specifically around insurance and warranties and mortgages that we need to run quickly, securely and get an answer back to that user before they lose interest and walk away, or before they worry that maybe their information was compromised. So now I've got a listing that I've received, I've applied for a loan, maybe I've set up time for an open open house. The next thing that I might consider is the actual set of transactions where I want to procure the home or I want to finalize that mortgage, I want to purchase the house, I want to go through other signatories. Those types of things typically as well will require some form of multi step approval process that is currently being automated by the people that are managing it. And so those types of things where you're sitting there interacting in real time, where you're expecting a response, response for something that is very high value and typically has a pretty good amount of risk associated with it, that's a great example and a great use case for distributed AI because you are running inference and you want those answers to be fast, reliable and secure. If I think about some other ones that are related, you know, anytime that I might do anomaly detection, that could be in an autonomous vehicle, it could be in a robot or other physical AI device, it could be for all augmented reality. If I'm doing, you know, work on a, on a conveyor belt or an assembly line, I might be in the medical field, I may be in the public sector or even in a military sort of a facility. But there's real time inference happening through augmented reality and virtual reality goggles today that lend themselves to this. And then we can think about, you know, even simpler things like real time multiplayer online gaming. You're creating worlds, you're, you're having avatars and other characters in the game that are reacting and responding to people in real time. These types of things you keep hearing real time or where you have a physical engagement with the application, those are typically the things that would warrant distribution to maintain that low latency and that real time performance. But I think we're just starting to understand this as an industry and even things like how we'll create online sports broadcasting, news broadcasting, broadcasting, you know, gambling, slash E gaming, things that really do reward latency and instantaneous responses are where we are seeing companies invest in this area and start to experiment with new ways of shuttling data across systems and starting to think really deeply about where does the hardware go, where does the data go, including the models, and then where is that application experience going to be interacted with with. And that's what's driving the considerations you

**Swapnil Bhartiya:** touched upon egress cost earlier. When teams move AI into production, egress cost often come as a surprise to them. How much of that hidden cost of centralized inference is actually just data movement getting results back to users and applications? How big of a factor is that and how can teams plan for that as well?

**Ari Weil:** So I think, you know, moving into the next 12 to 36 months, the companies that are going to grow the fastest and the most successfully will have modeled the whole bill and not just the line item that the hyperscalers are touting or that all of the different AI platforms are touting. They need to really understand all of the use cases that will incur cost for them as they scale. And that can be anything from what is my user acquisition cost, what is my concurrency cost with multiple use cases users, how many use cases have I enabled on this platform? Where is some of that data that I own completely, my intellectual property, my systems of record, and where am I learning actively and developing from the world or the industry at large and my customer base? Because the first thing that I can do is I can understand what would that cost be if I said I don't want to use multiple cloud providers or even my principal cloud provider, and I want to manage the data in my own environment? My own environment could be my own data center, it could be leased space, it could mean a lot of different things. But that's the first question of what if you just owned all of your data in your application code that will have you start to understand availability, zones and data boundaries, and where distribution can become more expensive if you use your own versus using a hyperscalers infrastructure. The second thing is once, once you've decided to partner with somebody, what would that cost to leave? Because now, if you look at all of the last 12 to 18 months of news around the cloud space, there have been all sorts of news stories about how much does it cost to leave a cloud provider? What does it cost for companies to take over their own data to export or download it themselves? And so the reality is, 9 to 12 cents a gigabyte is something that is designed to keep your data where it is on most hyperscaler platforms, because the cost of moving it is prohibitive. Some vendors have Said if you're leaving their cloud altogether, then they will give you a one time exemption where you can download your data. But you should be making sure that you indeed intend to exit that cloud environment. If you do so, it's also given rise to something that people are considering to be more of a neutral architecture. So a neutral architecture is one that will allow you to store data in one place that might then federate or synchronize it in a bunch of different areas, allow you to retrieve it somewhere else and then serve it where it makes the most sense. So a converged architecture like Akamai, where we have core to edge, can in some cases look like you might be thinking about an expensive distributed architecture on the face of it. But when you think about all of the costs that you don't have to pay from moving data between microservices or other services, moving it across geographic boundaries, and what it would take to manage all of your data and application code and then serve it as needed on demand with that low latency could actually turn into a cost benefit for you if you've modeled out all of your use cases across the platform. And I think that's the ultimate question here is where does inference run for your application? How often and where do you need to do things like pre and post training and fine tuning of your model? And then what? Where is the rest of the activity in your application actually occurring? Because if you know those things and you know how you're modeling user acquisition and user growth and longtime value, then you can put all of those factors together and think about what sort of an architecture am I really focused on and which parts of my application can be centralized versus where I'll benefit from distribution.

**Swapnil Bhartiya:** Now let's talk about tracking and measuring. If most organizations can't even track unit label economics for inference, how do they even know whether their AI is scaling efficiently or just getting more expensive?

**Ari Weil:** Yeah, so I think in some cases I'll use an example from Akamai. We have a product called CloudTest. That product is designed to look at the real user monitoring of all of the users of Your application, your SaaS application in the case of a proxied app on Akamai, and to show you how many people are using it, where are they coming from, what type of devices and network connections are they, how many requests and responses are part of an average user session. And then you can start to replay those things through your cloud and your Edge network so that you understand exactly the dynamics. Whether you use a wireshark to look at the network connections and the network dynamics. Or you use another active or passive testing tool to look at those real user activities and replay them and start to decompress them into repeatable steps that you might use in a testing harness. But there are multiple ways to get whether you've architected the whole user interaction or you're just recording the reality of a user interaction coming to you. All of those individual steps, the systems that you hit, the amount of data that you typically transfer and then it becomes a modeling exercise. You can do it virtually or you can do it maybe conceptually in spreadsheets and databases and actually use math. Just to extrapolate out this is what what my growth curves will look like. You can use real user examples like that CloudTest product that I mentioned so that we can look at the volume of users coming to you. We can look at their recorded sessions and actually show you what that user interaction looks like. And there's a myriad of other observability tools that can help you do something similar. There are also finops tools that you can bring to bear to help you make sense of the cloud builds that you're seeing, including if you have, or maybe especially if you have a multi cloud environment where the way that things are being metered and built are different. So I would take my FinOps tool, I would take my observability tool, I would take my active and passive testing tools and if you're an aquamai customer, we have a number of these for you. You can also use services if you want to to help you with sort of this modeling and observing your actual user workloads. But then it basically becomes models and math. It's how much does it cost for me to enable this use case? Is that use case as optimized as I might want it to be, how much does that cost me? And then by channel you can start to understand with your revenue operations or product or business operations teams. Am I incurring a cost that's commensurate with what I'm making? And does the margin profile for this channel or this workload or this application makes sense? And I think every business has the tools to do this. It's just when we look at AI, a lot of times we're thinking that this special thing that is a frontier LLM that maybe we've licensed, or maybe it's a fine tuned or a post trained model that we've created ourselves somehow completely changes the game. It might change the game and the capability that we're enabling it doesn't necessarily fundamentally change the way that I'm building and distributing an application. It's just adding a new capability to it. But that capability has a size and it has a cost, and it has a set of activities that workloads spawn from it or to it. And you can measure and make meter, all of those things, to come up with the right architecture and the right scalability model for your business.

**Swapnil Bhartiya:** For those businesses and technology leaders who are trying to manage AI costs, what should they be evaluating beyond raw compute pricing?

**Ari Weil:** Yeah, I think. I mean, the raw compute is certainly important. Understanding if I expect my model to grow and how much data that it might be sending out from various queries is also important. But I would argue you could start simpler than that and just say, what are the core use cases for my app? Where do I anticipate my users to be? Most companies, most product teams, are used to looking at user stories that define an ideal customer profile. You've got demographic and firmographic information about who those users are that you're looking to attain. If you've done a tech preview, an alpha or a beta release, you've probably got an idea of how you might need to fine tune some of those assumptions around ICP and target account lists and things of that nature. But once you know where your users are and those use cases that you're modeling, I do think that it's fairly straightforward for you to understand what are the costs of compute in those regions that I'm trying to serve? What are the nuances that I need to understand about data privacy laws, data egress and ingress fees that I might be accumulating when I want to do business in that region? And then you can start to ask yourself, how often do I have to think about things like being audited and what might be the scope for that audit, what sort of privacy regimes or sovereignty regimes I may need to sign up for, which might specify how my architecture is set up, where I need to persist, data where I'm allowed to actually make decisions based on the data that I've been collecting, how long I can save that data, because all of that will have an impact for storage modeling and database modeling. And then you need to think about how am I thinking of growing the capabilities of this application over time, and what does that need to grow? Do I just need additional hardware so that I can process more off of the existing LLMs or SLMs that I'm managing? Do I need to continuously deploy more models for these capabilities, and are they going to be consumed by all of the same users the same way. And so to me it becomes really a microservices architecture scaling exercise. And between the finops teams that have been helping cloud practitioners understand what their architecture architecting, what sort of costs they're incurring, giving way to some new versions of FinOps that now are taking into account AI considerations, build by partner, and also how you would architect the AI request response calls. As people are getting more examples and more comfortable with AI applications. We've got a lot of the raw materials for this, but the decisions ultimately will be what do I have? Where are my users? And what does it cost for me to make those two things connect?

**Swapnil Bhartiya:** If I ask you what does good cost discipline look like for teams that are scaling inference globally right now?

**Ari Weil:** I mean, that's a great question. I think if I were to think about my existing architectures and how our teams are used to looking at them, typically the question is going to be if I said that I was prioritizing one thing, it would be growth, right? I want my user count to grow, I want my active users to grow, and I want my revenue to grow. The number one factor will ultimately depend on what kind of a business you are and what you prioritize. Some businesses need to grow a large community to have any hope of scaling and becoming successful. I think about some developer platforms out there. Unless you have the support of a large group of developers, there's simply no way for you to be successful because ultimately developer sentiment and developer mindshare drive you. And so in those cases, maybe my cost for customer acquisition acquisition can be a bit higher. And off of some of the centralized indexes that have become standard, maybe the ultimate revenue that I get generated by user will be a little bit smaller. But if I look at customer acquisition and daily active use, those are the optimal metrics that I'm trying to prioritize. If I look at an enterprise applications, it probably comes back down to margin. And there I'm thinking about how do I procure more users, how much are they paying me by user, perhaps by seat or by month or year. So I'm looking at mrr, ARR, or I'm looking at cost per user, cost per seat, and then I'm thinking about how much does it cost for me to serve them an experience, depending on what that experience is. And does that math balance like what is my CAC and what is my ltv? And from that can I get the ROI from all of the sales and marketing that I need to invest to get and keep people on that platform. So I think the answer varies. I always think about my favorite architect in Australia when I was a young product manager and he used to always ask me how long is a piece of string? When I'd ask a non deterministic question. And so I think that is sort of the answer to your question. Swapnil it depends what you're optimizing for. If it's mind share and growth and consistent usage, then you really need to prioritize those numbers and the way that you're updating your application to drive that engagement. If it's margin based because you're an enterprise app or otherwise somebody that needs to scale up to maintain a consistent cost of service with a consistent benefit from the service that you're providing, then I think you need to think about unit economics and scale and it's a different architecture altogether. But that's such a use case and sort of company specific question that it really does depend on what you're building for.

**Swapnil Bhartiya:** Ari, this is exactly the conversation Enterprise Team needs to be having right now. Now. Thanks for breaking it down and sharing with our audience and folks. Please head over to Akamai and check out their latest blog post where they are going to talk about cost and other challenges related to AI. Thanks for watching and we'll see you in the next episode. Thank you, thank you.