Is Your AI Budget Built for Inference?

[00:00 - 00:11] Introduction: Swapnil Bhartiya introduces Ari Weil, VP of Product Marketing at Akamai, to discuss the real economics of AI inference and location impact.

[00:11 - 00:57] AI Model Training Overview: Ari Weil outlines the major investments, data center builds, and hype surrounding AI model training.

[00:57 - 01:21] Shift to AI Inference: Ari Weil contrasts AI training with inference, highlighting inference as the ongoing bill and active deployment phase.

[01:21 - 02:04] Architectural & Budgeting Insights: Ari Weil recommends budgeting AI applications as distributed systems, emphasizing architectural impact over raw token pricing.

And today we have with us Ari Weil, VP of Product Marketing at Akamai, to break down the real economics of AI inference and why where you run it changes everything.

Absolutely. So first of all, it's great to be back on the program with you again, and I'm glad to have a chance to talk about the cost side of the equation when we think about AI applications and workloads. And I think to start off with, I would suggest that training is really the headline that everybody loves. It's driving these massive data center deals, big investments, and some of the circular investment that we've seen coming out of anything from chip manufacturers and card manufacturers to the major hyperscale clouds that are also building out their own infrastructure and procuring it from others. And even if we talk about the, the landscape of companies that are looking to do their first IPO, a lot of the focus has been on how large of a data center they're building out to build the next multi-billion, multi-trillion weight model.

I think that training and that headline sort of factor is real. But if we think about it, inference is the bill that you get after you get all excited about training, and it arrives every single time your model takes a breath. Every time that you're not training, either pre-training or post-training or fine-tuning that model, inference is what happens when somebody needs your intelligence to be deployed.

And so I think the highlight from our perspective, where we would start a conversation, is to say, "You need to stop budgeting your AI applications as line items to do with just the GPUs that you're procuring or just the applications that you're building. You need to budget like building a distributed system, which is something that many large organizations have familiarity with or at least there are representative architectures for in the marketplace, because you're gonna hear people quote token price. How much does it cost per token for me to generate a workload? But the architecture is what ultimately is gonna decide your bill, and that's the part that people need to get more conversant on and take some of their cues from building cloud-native systems and to some extent, edge-native systems for the last several years."