AI Governance in Regulated Industries

[00:00 - 00:11] Introducing AI Governance: Swapnil Bhartiya opens the discussion asking Ari Weil how enterprises should manage governance when inference runs in distributed environments.

[00:11 - 01:23] Governance Guardrails & Compliance: Ari Weil explains how AI governance evolves iteratively and discusses compliance obligations, audit trails, and managing non-deterministic AI workflows.

[01:23 - 02:03] Risk Management & Code Review Practices: Ari Weil outlines risk management strategies regarding human oversight, AI-generated code, automated test harnesses, and the pressure to ship quickly.

[02:03 - 03:20] Evaluating Risk Frameworks: Ari Weil covers quantifying business risks, referencing NIST and MITRE ATT&CK frameworks, and explaining how Akamai helps clients scale safely.

[03:20 - 03:31] Early Adopters Question: Swapnil Bhartiya asks about specific industries leading the shift toward decentralized inference.

[03:31 - 04:31] Media & Gaming Industries: Ari Weil details how media companies utilize decentralized compute for high-fidelity video streams, transcoding, speech-to-text, and real-time processing.

[04:31 - 05:08] Robotics & Spatial Computing Use Cases: Ari Weil highlights the expansion of physical AI, robotics, AR/VR, and spatial computing into healthcare, assembly lines, and modern warfare.

[05:08 - 05:58] Retail & Web Personalization: Ari Weil explains how retail and e-commerce leverage edge compute and agentic AI to deliver personalized recommendations and enhance consumer experiences.

Let's not also talk about governance, how should enterprises think about governance when inference is running across more distributed environments?

So the interesting thing about governance is that it's coming together iteratively like so much of the architecture, so much of the workloads are as well. It's easy to say that you should have a human in a loop, it's easy to say that you should buy things like API gateways and AI gateways and start thinking about testing harness evolutions and how you should be evaluating cloud code and having people review that before it goes into production environments. But I think the reality is that across the industry, the governance question is really going to be very situational and specific to the business, to their maturity, and to the environment that they exist in or selling into.

So for example, if I'm part of a traditionally heavily regulated industry like the education industry, government, financial services like banking, insurance, or trading, there are a given set of rules that I have to abide by. There's a certain burden that I have of what I can log in how I provide audit trails and recoverability in things of that nature to all of my data. And the challenge that I'm facing right now in the AI era is when something is non-deterministic, how much scope do I allow up to have inside of that very scoped and audited workflow that I am responsible for a testing compliance to?

And I think that's one set of guardrails, compliance guardrails. Another one is just purely from a risk management basis. How much am I going to allow people to write code and run code from conception into production with whatever level of human guardrails I have or without? I mean, some companies are famously going on podcasts right now talking about how everything is cloud-coded and they're less and less reviewing it with human reviews and they're allowing AI to build its own test harnesses to evaluate the code that it's written. There's many businesses that think about that and it's a horrifying scenario and other businesses that look at that and say that's just the cost of doing business today because of the need to ship and ship so frequently.

I think when it comes to governance, the same sort of rules apply to how I think about want to find risk for my business, how I measure that risk for my business, and how I start to think about introduction of new technologies, especially when they're autonomous. I don't believe that we have well-established frameworks for this, but there have been frameworks like using NIST, like mapping things to MITRE attacks, to really understand where you feel like you need a given set of checks and balances. Often, I work with a lot of our customers, for example, through our global services team on this architecture, on testing scenarios, on how they can think about scaling and penetration testing and other things like that that they have to evaluate as they move to production and evolve their applications.

But when it comes to governance, I don't think that there's a broad brush approach to what we need to do other than most people saying we are not ready for a fully autonomous and self-contained AI sort of concept to production deployment right now without human interaction. The amount of human interaction, what the humans are actually doing, what you have people auditing, reviewing and reporting on, I think is very specific to the business. And regulators, as we've seen, are just now starting to kind of get their ideas together, and they're leaving a lot of this to the individual companies to define.

Are there any specific industries where you see this shift towards decentralized interests is getting moment or makes more sense or they are the early adopters of this idea?

We definitely see some of our typical early adopters from a media and technology perspective adopting things quicker. And I think that that's part of the overall industry and their approach to technology throughout the years. Media has famously, as they kept pace with all sorts of digital disruptions, had to continuously reinvent themselves. And we're seeing this now from the way that video needs to be captured in very high fidelity, in a 4K, in AK streams, processed very quickly, turned into a derivative work and then stream back out to people at a global scale, without incurring too much latency, being a huge driver of how you can use different types of compute infrastructure to actually do the transporting, to do the anomaly detection, to put together the social clips, the streaming clips, the downloads, the people all expect, to be able to take speech to text and text to speech, and really have that working across a number of different languages and mediums, very, very quickly driving a fair amount of innovation.

We're seeing the same sort of thing around gaming. And we're even seeing a quick evolution of things that were originally conceived of through neural networks and using physical AI and robotics to support gaming use cases now evolved into other sorts of robotics and physical AI use cases. So people that have cut their teeth on creating early versions of the metaverse and immersive gaming are now starting to think about how they can optimize assembly lines and healthcare and thinking about even modern warfare in the way that AR and VR are currently being brought into the real world as a different technology or a different purpose than what they were originally conceived of.

And then we're seeing from just a wrote sort of recommendations engine and being able to mine data and come out with tailored recommendations, new sorts of form factors of the work or the work product being produced. A lot of effort is being put in from the retail and commerce segment where the web is being redefined to be more agentic to now really needing to take that technology that we've done through edge computing and serverless functions and start to apply that more to inference use cases. And we're seeing those meet production very, very quickly. And then scale up as people are realizing that the end consumer really does, they've always sort of looked for this human like personalization recommendation and the early versions of it are really starting to show people that there's a big there and a big competitive opportunity to be seized.