The AI Conference for Humans Who Ship

There are many benchmarks that attempt to measure how well LLMs and AI agents can write SQL queries or do complicated statistical analysis. But as most practitioners know, this is only a small part of our job. Before we can write a query, we have to figure out the business context behind the question. We have figure out which tables to use in a messy database. We have to make subjective decisions about vaguely defined problems. All of this makes benchmarking analytical agents difficult.

We built a new benchmark—ADE-bench—that aspires to do exactly that. It gives agents complex analytical environments to work in and ambiguous tasks to solve, and measures how well they perform.

In this talk, we'll share how we built the benchmark, the results of our tests, a bunch of things we learned along the way, and what we think is coming next. The benchmark harness is open source, and can be found here: https://github.com/dbt-labs/ade-bench

I lead large-scale AI implementations and design agents for some of the most security-sensitive enterprises in the world. The pattern is always the same: prototypes and MVPs are easy, running agents in production is not. The moment an agent touches real data and real systems, security, policy, identity, orchestration, and auditability dominate the design. You also discover how many teams must align to make it safe. Without a shared and evolving playbook, the deployment stalls. In this talk I outline the approach I use to build enterprise agents that actually run in production. I cover the core architecture, the supervisory runtime, and the secure execution model that keeps agents predictable: identity boundaries, retrieval controls, RBAC and ABAC alignment, sandboxed tools, and evaluation loops. Through this talk I'd like to show what it really takes to operate agents inside large organizations and provide a practical blueprint you can adapt to your own environment.

The idea of incremental reads from data lakes has been cooking for years, but few are serving it up. As a user, you must wrangle change feeds, snapshots, time travel, that one corrupted manifest file. Do you need to be a "Big Data Engineer" to get it right? In this lightning talk, we’ll explore what’s broken, what's just hard, and why making data lake CDC accessible is a problem worth solving.

Cloud compute is expensive, and wasting runs on the guise of a "just scale will fix any problems" leaves you with less time to fix errors, and less compute to train the model you want. In this talk, I will discuss what are the easy optimizatiosn you might miss (minimizing communications, using the most effective algorithms, ensuring you're getting the most FLOPs possible) at the small scale, before ensuring that when you do scale up nothing is going to waste. In this particular talk, I'll be focusing on what worked at home, that then let me scale it further onto the cloud.

After more than a decade of limited innovation, the modern data stack is still slow, fragmented, and painfully repetitive. Every new dataframe library promises better ergonomics or performance, yet converges on the same API with the same limitations. The failure isn’t execution or scale, it is ignoring that appearance doesn't matter anymore: Agents and AI-first developers couldn't care less about how an API looks today. In this talk we’ll examine why the dataframe paradigm keeps reproducing itself, why it consistently underdelivers, and why the data stack’s biggest problems were never going to be solved at the API layer first. Princess Leia once put her trust into Obi-Wan Kenobi - what if we did the same with simple Python functions? Could we defeat the evil empire of DataFrame APIs?

Reinforcement Learning from Verifiable Rewards (RLVR) is increasingly common in post-training pipelines, but the practical details are often glossed over. How do you design reward functions that programmatically verify model outputs? What makes synthetic training data effective? How do you build a custom RL environment that doesn't silently break your training?

Training a large language model is an exercise in tradeoffs you didn't expect. Should you spend a week optimizing infrastructure and architecture, or just start training? This talk covers how to think about pre-training decisions: why architecture changes are rarely about accuracy and almost always about performance, how frontier open labs actually design models, and how to make principled calls when everything is a tradeoff. We'll walk through real examples of decisions that looked obvious in retrospect, and the ones that still don't have clean answers.

Most organizations are experimenting with AI at work. Proof of concepts are thriving in small-scale silos, but how can you scale those wins across an enterprise. This talk discusses the symptoms that trap teams in the cycle of perpetual prototyping and provides a practical framework for breaking through common technical, organizational, and cultural barriers to scale. Learn how to plan for production from day one and turn your AI experiments into scalable, business-impacting solutions.

Training sparse Mixture-of-Experts models at scale is notoriously unstable. Experts collapse, routers drift, and loss spikes appear out of nowhere. This talk covers how we built Trinity Large, a 400B parameter MoE (13B active), trained on 17 trillion tokens with zero loss spikes.

We'll walk through the decisions that actually mattered: why we replaced standard aux-loss-free balancing with a momentum-based approach (SMEBU), how interleaved local/global attention made context extension surprisingly smooth, and what broke when we first tried running Muon at scale.

I'll also cover the less glamorous stuff: our Random Sequential Document Buffer to reduce batch heterogeneity, recovering from B300 GPU faults on brand-new hardware, and the six changes we shipped at once when routing started collapsing mid-run.

Practical lessons for teams training their own MoEs or scaling up sparse architectures

Moondream is a vision language model that runs in real time on video streams. This talk covers the model-side work behind it.

I'll start with architecture: upcycling from dense to MoE, and the tradeoffs when you're optimizing for latency rather than just parameter count. Then tokenization: why we built a custom SuperBPE tokenizer and what it bought us. The goal throughout was to avoid modeling decisions that would hurt us at inference time.

I'll also cover training infrastructure. We wrote custom training engines and RL systems because existing open source projects were pushing us toward design decisions that didn't fit. I'll talk about where we diverged and what we got out of it.

Finally, inference. Real-time VLM isn't just a serving problem or a modeling problem. We built a custom inference engine alongside the model, and I'll cover how the two informed each other.

This talk goes beyond architecture diagrams to share what actually happens when you operate an agentic search engine on trillions of documents.

We'll dig into how an object storage-native design allows a small team of engineers to manage an AI search engine that scales to:

• Peak load of 1M+ writes per second and 30k+ searches per second
• 1+ trillion documents
• 5+ PB of logical data
• 400+ tenants
• p90 query latency <100 ms

Topics include:

• How using a modern storage architecture decreases COGS by 10x or more
• Optimizing traditional vector and FTS indexes for the high latency of object storage
• Building search algorithms that are fine-tuned for LLM-initiated searches
• A simple rate-limiting technique that provides strong performance isolation in multi-tenant environments
• Observability, reliability, and performance lessons learned from production incidents.

Attendees will leave with a concrete understanding of how separating storage from compute-and treating object storage as the primary database changes not only the cost structure, but the entire operational model of large-scale AI search.

There are many benchmarks that attempt to measure how well LLMs and AI agents can write SQL queries or do complicated statistical analysis. But as most practitioners know, this is only a small part of our job. Before we can write a query, we have to figure out the business context behind the question. We have figure out which tables to use in a messy database. We have to make subjective decisions about vaguely defined problems. All of this makes benchmarking analytical agents difficult.

We built a new benchmark—ADE-bench—that aspires to do exactly that. It gives agents complex analytical environments to work in and ambiguous tasks to solve, and measures how well they perform.

In this talk, we'll share how we built the benchmark, the results of our tests, a bunch of things we learned along the way, and what we think is coming next. The benchmark harness is open source, and can be found here: https://github.com/dbt-labs/ade-bench

Reinforcement learning systems often fail not because rewards are wrong, but because optimization pressure is unbounded. Policies exploit edge cases, drift over time, and converge to brittle strategies that look fine in training but break in deployment, especially under bounded actions, safety requirements, resource budgets, and long-term user impact.

This talk focuses on controlling optimization directly: practical techniques for training RL agents that remain stable and predictable under hard constraints. Rather than modifying rewards, we explore structural and system-level approaches that shape behavior by construction.

Topics include:

• Why reward penalties alone fail to enforce hard constraints under scale and distribution shift

• Structural constraint mechanisms such as action masking, feasibility filters, and sandboxed execution

• How training inside hard boundaries changes policy behavior and improves long-horizon stability, including across retraining cycles

• Detecting constraint violations and failure modes that do not appear in aggregate return metrics

• Lessons from applying constrained RL in production-like systems, including failures only discovered after deployment and what ultimately stopped them

• The goal is to share concrete algorithmic and system design strategies for deploying reinforcement learning in settings where violations are suboptimal.

Vibe coding is fast, but it often skips the safety rails: features look fine in a demo and then break in real user flows. Especially when you iterate on them. This talk shows how to make vibe-coded web apps reliable by adding end-to-end tests with Playwright that are quick to write, stable in CI, and focused on what actually matters.

A big shift is that modern coding assistants like Cursor and Claude Code can run commands and iterate on real failures. Whats missing is the glue, between e.g. Claude Code to run a test and gets its state. I will show practical workflows for writing tests faster using MCP Skills and Playwright MCP, both in an editor and inside Claude Code environments.

Based on lessons from building multiple websites over the last months, I will share a repeatable approach for growing a small, high-signal test suite that keeps up with rapid development and gives you the confidence to ship more changes without fear.

Agentic AI has certainly secured mindspace of industry. As more & more people continue to play with LLMs, they begin developing an intuition aka vibes around whether LLM is performing the task at hand well or not. While usecases like math problem, classification are easier to verify; generative content creation etc are not. Adding agents to the mix makes things lot more complex. This talk will highlight the particular challenges one may face as they continue their journey towards productionizing their Agentic application: outputs are non deterministic, ground truth is hard to find, and what used to work before no longer does. Yet ""LGTM"" isn't a deployment strategy.

While typical LLM chatbot can be seen as a more sophisticated Google search, we're beginning to expect more from Agents: Don't just give an answer but actually perform the task. Security, bias, privacy etc suddenly become non-negotiable. Only after we handle these complexities, we can unblock real usecases in healthcare, finance, legal etc.

This talk tackles why agent evaluation is fundamentally harder than traditional ML testing: multi-step reasoning chains, tool use side effects & more. How to build evaluation datasets that actually reflect production scenarios, not just cherry-picked examples. We'll cover automated evaluation pipelines using LLM-as-judge patterns, and when you can not avoid human in the loop. The session addresses detecting regressions before users do: setting up continuous evaluation that catches model degradation. Tricky cases when agent aces public evals but fails in production, and how to build evaluations that predict real-world performance.

Software engineering has seen a massive lift from AI augmentation over the last 3 years. However, data teams have been left behind. The fundamental loop of software engineering differs significantly from that of data science, making distinct agents necessary.

• Why does AI struggle with quantitative data, and where are there divergences from human experts?
• What data representations are most meaningful for today's frontier AI models?
• How can better data representation let AI become a partner in data science?
• What are the best practices for deploying AI to teams to maximize acceleration without compromising quality or security?