Running Millions of (Millisecond) AI Sandboxes without Breaking the Piggy Bank

Agents are great, but they place difficult requirements on the underlying infrastructure they run on: (1) they need to be strongly isolated (eg, within a VM); (2) they need to start up as quickly as possible (ideally in milliseconds) and put to sleep when not used; and (3) they require massive scale (eg, millions of them for even a single provider/product). Using standard infra to run sandboxes at this level of scale can result in eye-watering cloud-infra bills. And attempting to start sandboxes in milliseconds is an unsolved challenge.

In this talk we’ll cover our years-long journey aimed at severely optimizing and increasing the efficiency of how workloads are deployed on the cloud, beginning with research and OSS work. Along the way, we’ll  cover the basics of virtualization and isolation primitives (e.g., virtual machines, microVMs, containers, isolates) and their performance and security trade-offs. With that in place, we will describe how we leveraged the research and OSS work  to build a virtualization system that can start any workload in a few milliseconds, and cram up to 1M+ such lightweight VMs into a single, off-the-shelf server, allowing for millions of strongly-isolated agents to be hosted in a rack, rather than an entire data center. Finally, we will show a brief live demo of this in action.