[00:00–02:30] The disappearing AI boundary: Swapnil Bhartiya introduces the shift from AI living in isolated lanes to permeating every application and workflow. Danielle Cook explains that the clear distinction between AI and non-AI applications no longer exists. Apps now contain AI, connect to it via APIs, feed it data, or support it, making every app a participant in the AI ecosystem.
[02:31–03:25] Visibility and shadow AI: Cook discusses the varying levels of visibility organizations have into AI usage. While regulated industries may maintain strict control, most organizations must assume widespread, unapproved shadow AI usage across their workflows, presenting a significant monitoring challenge for platform teams and developers.
[03:26–04:56] Experimentation vs. production demands: Moving AI to production shifts the focus from model performance to handling real users, latency requirements, continuous inference, and actual costs. Cook highlights that GPUs are only part of the architecture. Production also requires compute, storage, and networking, as well as recognizing that many supplementary tasks (like API calls and databases) run effectively on CPUs.
[04:57–06:09] Rethinking infrastructure strategy: Architects must view the entire infrastructure through an AI lens due to the real-time, interactive nature of AI and continuous data movement. This requires implementing strategies like model routing and semantic caching to deliver cost-effective, high-quality customer experiences at scale.
[06:09–08:01] Data gravity and edge computing: The paradigm is shifting to move compute to the data rather than moving data to compute. Because data is often the least movable component due to size, compliance, or sensitivity, workload placement and anticipating egress costs are becoming critical architectural decisions.
[08:02–09:56] The complexity of a single AI request: A single user request to an AI application can trigger dozens of distributed background requests involving multiple models, vector databases, and APIs. Cook notes that because these systems aren’t always colocated, repeated minor network latency can compound, making the network or storage the actual bottleneck rather than the GPU.
[09:57–11:35] Managing distributed data and egress costs: With enterprise data scattered across public clouds, private infrastructure, and edge locations, moving data becomes costly and complex due to regulatory and geographic factors. Cook emphasizes the necessity of architecting for data transfer fees up front to avoid unexpected egress costs that can reach billions annually.
[11:36–13:08] Designing AI architecture from scratch: For teams building a platform today, Cook advises designing for “data in motion” from day one. This involves assuming distributed applications, flexible workload placement to service users effectively, and fundamental portability to accommodate rapid changes in models and infrastructure.
[13:09–14:23] Starting points for rearchitecting: Teams rearchitecting their infrastructure should begin by evaluating available compute resources, determining which models to run, assessing data ownership, deciding between building and buying infrastructure, and analyzing tokenomics.
[14:24–15:16] The value of smaller models: While the industry obsesses over the largest models, Cook points out that smaller models can sometimes deliver better, more cost-effective results. Using a smaller model on a CPU for specific tasks can save money and improve efficiency through proper workload routing.
[15:17–16:15] The role of edge computing: Edge computing places compute resources near the end user. This can involve distributed GPU locations or containerized edge environments, depending on what the customer needs to deliver their required experience while managing financial impacts.
[16:16–20:23] Navigating AI strategies beyond the hype: Cook notes that there is no universal maturity model for AI. Businesses must align their AI infrastructure choices with their specific business objectives, whether that means simply using an external AI service, running internal GPUs for fine-tuning, training models from scratch, or using serverless inference.