The AI Conference for Humans Who Ship

This talk goes beyond architecture diagrams to share what actually happens when you operate an agentic search engine on trillions of documents.

We'll dig into how an object storage-native design allows a small team of engineers to manage an AI search engine that scales to:

• Peak load of 1M+ writes per second and 30k+ searches per second
• 1+ trillion documents
• 5+ PB of logical data
• 400+ tenants
• p90 query latency <100 ms

• How using a modern storage architecture decreases COGS by 10x or more
• Optimizing traditional vector and FTS indexes for the high latency of object storage
• Building search algorithms that are fine-tuned for LLM-initiated searches
• A simple rate-limiting technique that provides strong performance isolation in multi-tenant environments
• Observability, reliability, and performance lessons learned from production incidents.

As models become more capable and reliable, it’s more important than ever to design tools and context thoughtfully. When models had short context windows and low agency, basic document retrieval into the model’s context window made a dramatic difference; but as frontier models boast millions of tokens of context and run for minutes and hours through tool calls and cycles of compaction, there is an ever growing list of concerns all vying for our models’ scarce attention. Extending context windows is an expensive and incomplete workaround.

In this talk, we will share some of the principles and techniques we found useful in navigating these problems ourselves as we worked on the goal of improving our assistant, Puck, from a simple retrieval-based chatbot to a deeply knowledgeable general assistant. In particular, we will touch on two prevailing challenges. First, we’ll share how we’ve sought to raise the signal-to-noise ratio in Puck’s context by thinking of context engineering itself as a search problem, involving every step of the pipeline from indexing to subagents. Second, we’ll share how Puck approaches blending faithful, up-to-date structured data queries with the richness, breadth, incompleteness, and frequent conflicts latent in unstructured data in production. We’ll walk through a few concrete tactics in detail within both of these pillars, demonstrating that creative and useful approaches often come when we stop thinking of databases, search indexes, tools, and subagents as separate components, but as different solutions to the same underlying, age-old problem: searching for signal in a confusing and noisy world.

As AI features become standard across products, teams face a critical measurement challenge: understanding how much AI actually contributed to user outcomes.

This talk introduces a practical framework for AI attribution — measuring not just whether users engaged with an AI feature, but the degree to which AI shaped the final result. We'll cover the full spectrum: from full acceptance, to partial edits, to abandoned attempts where users generated output but ultimately reverted to manual input. We'll explore AI attribution measurement approaches and a clearer understanding of how to assess AI feature impact.

Most AI/ML systems in production rely on predictive models. Businesses score users for churn risk, conversion likelihood, or click probability, and then make decisions based on those scores. But accurate predictions do not imply better decisions. A model that answers the question “Who will convert anyway?” very well is often a terrible guide for deciding who to target with an offer, notification, or intervention. More often than not, what we really care about is answering the question “Whose behavior will actually change because of our intervention?” rather than “Who will do X?”.

This gap between prediction and decision-making shows up in more and more places, such as marketing campaigns, product experiments, and even healthcare and risk interventions. Uplift modeling is a way to bridge this gap by estimating the causal effect of an intervention at the individual level—who is persuadable, who is a sure thing, who is a lost cause, and who might even be harmed or annoyed if we intervene. At Intuit, we use uplift modeling to enhance targeting and experimentation, helping us allocate marketing and product interventions more effectively across millions of customers.

In this talk, I will introduce uplift modeling as a practical form of causal AI that fits into existing experimentation and ML workflows. I will start by contrasting predictive and causal thinking, followed by core concepts like treatment effects and counterfactuals, using concrete examples. After that, I will dive into uplift modeling techniques, specifically using meta-learners in Python. Finally, we will go over evaluation techniques that are unique to uplift modeling and causal inference, and how they differ from standard ML metrics like F1 score or AUC.

This talk is aimed at data scientists and ML engineers at any level, including those with no background in causal inference. Basic familiarity with Python and machine learning concepts is helpful but not required. Attendees will leave with a clear mental model of when predictive models are not enough, an intuitive understanding of uplift modeling and its causal foundations, and a sense of how to start applying uplift models in their own AI/ML systems to target the right people.

Most organizations are experimenting with AI at work. Proof of concepts are thriving in small-scale silos, but how can you scale those wins across an enterprise? This talk discusses the symptoms that trap teams in the cycle of perpetual prototyping and provides a practical framework for breaking through common technical, organizational, and cultural barriers to scale. Learn how to plan for production from day one and turn your AI experiments into solutions that stick.

Datadog has grown from a startup focused on infrastructure monitoring into a platform processing over a hundred trillion events daily.

Over the years, we expanded beyond metrics to include traces, logs, profiling, real user monitoring, and security. Our user base has broadened from operations to developers, analysts, and business users. More recently, automated agents have also become key consumers of our platform.

Our bottom-up culture encourages teams to take initiative. Consequently, we developed multiple specialized ingestion pipelines and query engines. These were built to satisfy strict real-time requirements and interactive experience, providing users with the insights necessary for success.

This focus on efficiency led to custom-built, proprietary solutions designed for our unique constraints. Today, however, the evolving landscape allows us to reconcile these specialized engines with open standards, blending the versatility of the ecosystem with our purpose-built designs.

In recent years, we have refactored the interfaces of these query engines to create a composable data system. This allows us to better leverage shared capabilities, enabling cross-dataset querying, advanced analytics, and more versatile access patterns.
Our goal is to scale our bottom up culture. By defining clear contracts and high-level components, we enable decentralized decision-making. This improves performance, efficiency, and flexibility across the platform, while reducing silos.

By adopting a deconstructed stack, we combine the efficiency of the open-source ecosystem with our internal capabilities to build a truly composable system. This architecture provides the flexibility to adapt to immediate and future demands, specifically addressing requirements for scale, velocity, and operational resilience, while ensuring readiness for growing challenges such as data intensive operations like AI.

In this talk, we will discuss how we rely on and contribute to key projects in the data ecosystem: Arrow for data interchange, Substrait for plans, Calcite as an optimizer, DataFusion as an execution core, and Parquet for columnar storage.

• Why does AI struggle with quantitative data, and where are there divergences from human experts?
• What data representations are most meaningful for today's frontier AI models?
• How can better data representation let AI become a partner in data science?
• What are the best practices for deploying AI to teams to maximize acceleration without compromising quality or security?

The biggest barrier to AI ROI is developers' inability to define what "good" means for their use case.

After a decade building ML infrastructure at Meta , I co-founded Fireworks AI. We now serve 13 trillion tokens daily for thousands of companies like Cursor, Perplexity, Vercel, Uber, and Notion, helping them customize open models for production. Across all these customers, one pattern has become clear: the teams that can build rigorous evaluations are the only ones shipping successful AI systems. Everyone else is just throwing spaghetti at the wall.

Too many teams are shipping AI products entirely based on intuition: manually testing a few examples and hoping for the best. This works early on, but when you venture into specialized domains like contract analysis or insurance formulary navigation, untested edge cases can make applications completely unusable. Models might hallucinate their way to the right answer half the time, but in scrutinized industries, that batting average can end careers.

What separates good evals from bad: I’ll walk through anti-patterns we see repeatedly, like evals that are too easy or that test capabilities instead of requirements. I’ll also cover how to build safety tests that automatically fail when an agent leaks system prompts or attempts destructive queries, alongside functional tests that validate outputs against your requirements.. 

The Evaluation-Driven Development (EDD) to building AI systems: Code is deterministic, but AI is probabilistic. My EDD approach adapts test-driven development's red-green-refactor cycle for probabilistic systems. I’ll break down how this approach works: developers write a failing evaluation first, make the minimal change to pass it, and refactor while maintaining coverage.

How to translate vague business requirements into measurable assertions: We’ll walk through how to use Eval Protocol, an open-source testing framework that makes AI evaluations work like unit tests, integrating with pytest and CI/CD pipelines. I’ll share examples of how to handle scoring probabilistic outputs, and the tradeoffs between single-turn and multi-turn agent evaluations. 

The audience will leave with a practical framework for building evaluation infrastructure that catches problems before they ship and the discipline to never change an AI system without a failing evaluation that demands it.

Most AI agents work in demos. Few survive in production. LLMs are stateless. Infrastructure fails. Context windows reset. Real-world objectives span hours or days. Building long-running autonomous agents requires durability engineered across the entire system.

This talk compares and contrasts dominant approaches to durability for agents and presents three pillars of durable agentic systems.

1. Durable Execution

   Agents must survive crashes, retries, and partial task completion. Durable execution engines like Temporal persist workflow state and enable deterministic replay. Graph-based orchestrators such as LangGraph model control flow as explicit state machines. These approaches reflect different assumptions about recovery, replayability, and operational resilience, and directly shape how agents behave under failure.

2. Durable Autonomy

   Autonomous systems inevitably encounter ambiguity and incomplete information. Durable autonomy means designing agents that recognize uncertainty, escalate intelligently to humans when necessary, and resume coherently without losing progress. We’ll examine architectural patterns for human-in-the-loop integration that preserve control while maintaining forward momentum.

3. Durable Statefulness

   Long-running agents cannot rely on ever-growing prompts. Some systems serialize state into resumable bursts using patterns like Anthropic’s Git-Commit approach. Others externalize cognition into layered memory architectures - separating working, episodic, semantic, or procedural memory through memory virtualization. Different workloads and time horizons demand different state strategies.

Attendees will leave with a deeper understanding of agent durability and a practical architectural framework for building resilient agents, systems designed not just to respond, but to endure.

Reinforcement learning systems often fail not because rewards are wrong, but because optimization pressure is unbounded. Policies exploit edge cases, drift over time, and converge to brittle strategies that look fine in training but break in deployment, especially under bounded actions, safety requirements, resource budgets, and long-term user impact.

This talk focuses on controlling optimization directly: practical techniques for training RL agents that remain stable and predictable under hard constraints. Rather than modifying rewards, we explore structural and system-level approaches that shape behavior by construction.

Topics include:

• Why reward penalties alone fail to enforce hard constraints under scale and distribution shift

• Structural constraint mechanisms such as action masking, feasibility filters, and sandboxed execution

• How training inside hard boundaries changes policy behavior and improves long-horizon stability, including across retraining cycles

• Detecting constraint violations and failure modes that do not appear in aggregate return metrics

• Lessons from applying constrained RL in production-like systems, including failures only discovered after deployment and what ultimately stopped them

• The goal is to share concrete algorithmic and system design strategies for deploying reinforcement learning in settings where violations are suboptimal.

The same way computing for agents is evolving into sandboxes, data infrastructure for agents will necessitate the provisioning and maintenance of trillions of databases. From agent memory, session data, to the mind-boggling number of databases vibe-coded applications will need, it is clear we need databases that come online instantly and are individually cheap. SQLite is widely acknowledged to have the right shape for this, but it also at the same time lacks the extended feature set that modern applications need. In this talk we will present Turso, an open- source rewrite of SQLite in Rust, that keeps full compatibility with its file-based nature while expanding what it can do.

Vibe coding is fast, but it often skips the safety rails: features look fine in a demo and then break in real user flows. Especially when you iterate on them. This talk shows how to make vibe-coded web apps reliable by adding end-to-end tests with Playwright that are quick to write, stable in CI, and focused on what actually matters.

A big shift is that modern coding assistants like Cursor and Claude Code can run commands and iterate on real failures. Whats missing is the glue, between e.g. Claude Code to run a test and gets its state. I will show practical workflows for writing tests faster using MCP Skills and Playwright MCP, both in an editor and inside Claude Code environments.

Based on lessons from building multiple websites over the last months, I will share a repeatable approach for growing a small, high-signal test suite that keeps up with rapid development and gives you the confidence to ship more changes without fear.

Datadog has grown from a startup focused on infrastructure monitoring into a platform processing over a hundred trillion events daily.

Over the years, we expanded beyond metrics to include traces, logs, profiling, real user monitoring, and security. Our user base has broadened from operations to developers, analysts, and business users. More recently, automated agents have also become key consumers of our platform.

Our bottom-up culture encourages teams to take initiative. Consequently, we developed multiple specialized ingestion pipelines and query engines. These were built to satisfy strict real-time requirements and interactive experience, providing users with the insights necessary for success.

This focus on efficiency led to custom-built, proprietary solutions designed for our unique constraints. Today, however, the evolving landscape allows us to reconcile these specialized engines with open standards, blending the versatility of the ecosystem with our purpose-built designs.

In recent years, we have refactored the interfaces of these query engines to create a composable data system. This allows us to better leverage shared capabilities, enabling cross-dataset querying, advanced analytics, and more versatile access patterns.
Our goal is to scale our bottom up culture. By defining clear contracts and high-level components, we enable decentralized decision-making. This improves performance, efficiency, and flexibility across the platform, while reducing silos.

By adopting a deconstructed stack, we combine the efficiency of the open-source ecosystem with our internal capabilities to build a truly composable system. This architecture provides the flexibility to adapt to immediate and future demands, specifically addressing requirements for scale, velocity, and operational resilience, while ensuring readiness for growing challenges such as data intensive operations like AI.

In this talk, we will discuss how we rely on and contribute to key projects in the data ecosystem: Arrow for data interchange, Substrait for plans, Calcite as an optimizer, DataFusion as an execution core, and Parquet for columnar storage.

Education researchers have spent a century figuring out how to build environments where complex, unpredictable systems self-correct through structured feedback. Software engineers building AI agents are solving the exact same problem from scratch and ignoring all of it. This talk bridges that gap. I'm a former classroom teacher turned AI engineer, and I'll walk you through five pedagogical frameworks that map directly to eval design patterns for AI agents: backward design (define success criteria before you build), formative assessment (eval continuously, not just at the end), rubric design (multi-dimensional scoring instead of pass/fail), error analysis (categorize failure modes because same symptom doesn't mean same cause), and differentiated feedback (the agent, the user, and the knowledge base each need their own signal channel). What ties them together is back pressure: each framework is a way to capture signal from problems and route it to where it drives change. That's what makes a system self-correcting instead of just self-reporting. Most AI observability is still about watching systems after the fact. This talk is about designing systems where the eval layer captures back pressure from failures and feeds it back upstream, so the system iterates on itself. I've built production agents and won hackathons with this approach, and the core insight is simple: the best eval systems aren't tests, they're environments. And nobody knows more about designing those environments than teachers.

Most AI problems are really data problems. AI workloads bring with them ever larger amounts of data from multiple modalities (e.g., text, images, audio, video, sensor data). If you were indexing say, the internet, you need to solve a number of new data infra challenges:

1. Storing large blobs and avoiding copying them over and over during processing

2. Dealing with much larger table sizes: trillion rows with a capital T

3. Supporting workloads like Search, Curation, and Training directly from your dataset instead of having to move data to/from point-solution systems

4. Dealing with *really* distributed pipelines: what happens when your storage, CPUs, and GPUs are with different clouds / vendors?

In this talk we will dive into detail on why it's challenging to manage trillion scale wide tables with multimodal data. We'll see why existing data infra doesn't support new these data types, workloads, or scale. And we'll do a quick under the hood peek at how Lance format and LanceDB solves these problems at a foundational level. Zooming out, we'll cover how LanceDB fits into the existing data stack alongside Iceberg. Finally, we'll talk through our roadmap and show you the big improvements we're working on in 2026.

Whether you're looking to do large scale search or building the next frontier model, this will help you scale easier, get to production faster, and save on infra cost.