The AI Conference for Humans Who Ship

There are many benchmarks that attempt to measure how well LLMs and AI agents can write SQL queries or do complicated statistical analysis. But as most practitioners know, this is only a small part of our job. Before we can write a query, we have to figure out the business context behind the question. We have figure out which tables to use in a messy database. We have to make subjective decisions about vaguely defined problems. All of this makes benchmarking analytical agents difficult.

We built a new benchmark—ADE-bench—that aspires to do exactly that. It gives agents complex analytical environments to work in and ambiguous tasks to solve, and measures how well they perform.

In this talk, we'll share how we built the benchmark, the results of our tests, a bunch of things we learned along the way, and what we think is coming next. The benchmark harness is open source, and can be found here: https://github.com/dbt-labs/ade-bench

I lead large-scale AI implementations and design agents for some of the most security-sensitive enterprises in the world. The pattern is always the same: prototypes and MVPs are easy, running agents in production is not. The moment an agent touches real data and real systems, security, policy, identity, orchestration, and auditability dominate the design. You also discover how many teams must align to make it safe. Without a shared and evolving playbook, the deployment stalls. In this talk I outline the approach I use to build enterprise agents that actually run in production. I cover the core architecture, the supervisory runtime, and the secure execution model that keeps agents predictable: identity boundaries, retrieval controls, RBAC and ABAC alignment, sandboxed tools, and evaluation loops. Through this talk I'd like to show what it really takes to operate agents inside large organizations and provide a practical blueprint you can adapt to your own environment.

The idea of incremental reads from data lakes has been cooking for years, but few are serving it up. As a user, you must wrangle change feeds, snapshots, time travel, that one corrupted manifest file. Do you need to be a "Big Data Engineer" to get it right? In this lightning talk, we’ll explore what’s broken, what's just hard, and why making data lake CDC accessible is a problem worth solving.

Cloud compute is expensive, and wasting runs on the guise of a "just scale will fix any problems" leaves you with less time to fix errors, and less compute to train the model you want. In this talk, I will discuss what are the easy optimizatiosn you might miss (minimizing communications, using the most effective algorithms, ensuring you're getting the most FLOPs possible) at the small scale, before ensuring that when you do scale up nothing is going to waste. In this particular talk, I'll be focusing on what worked at home, that then let me scale it further onto the cloud.

After more than a decade of limited innovation, the modern data stack is still slow, fragmented, and painfully repetitive. Every new dataframe library promises better ergonomics or performance, yet converges on the same API with the same limitations. The failure isn’t execution or scale, it is ignoring that appearance doesn't matter anymore: Agents and AI-first developers couldn't care less about how an API looks today. In this talk we’ll examine why the dataframe paradigm keeps reproducing itself, why it consistently underdelivers, and why the data stack’s biggest problems were never going to be solved at the API layer first. Princess Leia once put her trust into Obi-Wan Kenobi - what if we did the same with simple Python functions? Could we defeat the evil empire of DataFrame APIs?

Reinforcement Learning from Verifiable Rewards (RLVR) is increasingly common in post-training pipelines, but the practical details are often glossed over. How do you design reward functions that programmatically verify model outputs? What makes synthetic training data effective? How do you build a custom RL environment that doesn't silently break your training?

Training a large language model is an exercise in tradeoffs you didn't expect. Should you spend a week optimizing infrastructure and architecture, or just start training? This talk covers how to think about pre-training decisions: why architecture changes are rarely about accuracy and almost always about performance, how frontier open labs actually design models, and how to make principled calls when everything is a tradeoff. We'll walk through real examples of decisions that looked obvious in retrospect, and the ones that still don't have clean answers.

Most organizations are experimenting with AI at work. Proof of concepts are thriving in small-scale silos, but how can you scale those wins across an enterprise. This talk discusses the symptoms that trap teams in the cycle of perpetual prototyping and provides a practical framework for breaking through common technical, organizational, and cultural barriers to scale. Learn how to plan for production from day one and turn your AI experiments into scalable, business-impacting solutions.

Training sparse Mixture-of-Experts models at scale is notoriously unstable. Experts collapse, routers drift, and loss spikes appear out of nowhere. This talk covers how we built Trinity Large, a 400B parameter MoE (13B active), trained on 17 trillion tokens with zero loss spikes.

We'll walk through the decisions that actually mattered: why we replaced standard aux-loss-free balancing with a momentum-based approach (SMEBU), how interleaved local/global attention made context extension surprisingly smooth, and what broke when we first tried running Muon at scale.

I'll also cover the less glamorous stuff: our Random Sequential Document Buffer to reduce batch heterogeneity, recovering from B300 GPU faults on brand-new hardware, and the six changes we shipped at once when routing started collapsing mid-run.

Practical lessons for teams training their own MoEs or scaling up sparse architectures

Moondream is a vision language model that runs in real time on video streams. This talk covers the model-side work behind it.

I'll start with architecture: upcycling from dense to MoE, and the tradeoffs when you're optimizing for latency rather than just parameter count. Then tokenization: why we built a custom SuperBPE tokenizer and what it bought us. The goal throughout was to avoid modeling decisions that would hurt us at inference time.

I'll also cover training infrastructure. We wrote custom training engines and RL systems because existing open source projects were pushing us toward design decisions that didn't fit. I'll talk about where we diverged and what we got out of it.

Finally, inference. Real-time VLM isn't just a serving problem or a modeling problem. We built a custom inference engine alongside the model, and I'll cover how the two informed each other.

This talk goes beyond architecture diagrams to share what actually happens when you operate an agentic search engine on trillions of documents.

We'll dig into how an object storage-native design allows a small team of engineers to manage an AI search engine that scales to:

• Peak load of 1M+ writes per second and 30k+ searches per second
• 1+ trillion documents
• 5+ PB of logical data
• 400+ tenants
• p90 query latency <100 ms

Topics include:

• How using a modern storage architecture decreases COGS by 10x or more
• Optimizing traditional vector and FTS indexes for the high latency of object storage
• Building search algorithms that are fine-tuned for LLM-initiated searches
• A simple rate-limiting technique that provides strong performance isolation in multi-tenant environments
• Observability, reliability, and performance lessons learned from production incidents.

Attendees will leave with a concrete understanding of how separating storage from compute-and treating object storage as the primary database changes not only the cost structure, but the entire operational model of large-scale AI search.

There are many benchmarks that attempt to measure how well LLMs and AI agents can write SQL queries or do complicated statistical analysis. But as most practitioners know, this is only a small part of our job. Before we can write a query, we have to figure out the business context behind the question. We have figure out which tables to use in a messy database. We have to make subjective decisions about vaguely defined problems. All of this makes benchmarking analytical agents difficult.

We built a new benchmark—ADE-bench—that aspires to do exactly that. It gives agents complex analytical environments to work in and ambiguous tasks to solve, and measures how well they perform.

In this talk, we'll share how we built the benchmark, the results of our tests, a bunch of things we learned along the way, and what we think is coming next. The benchmark harness is open source, and can be found here: https://github.com/dbt-labs/ade-bench

Reinforcement learning systems often fail not because rewards are wrong, but because optimization pressure is unbounded. Policies exploit edge cases, drift over time, and converge to brittle strategies that look fine in training but break in deployment, especially under bounded actions, safety requirements, resource budgets, and long-term user impact.

This talk focuses on controlling optimization directly: practical techniques for training RL agents that remain stable and predictable under hard constraints. Rather than modifying rewards, we explore structural and system-level approaches that shape behavior by construction.

Topics include:

• Why reward penalties alone fail to enforce hard constraints under scale and distribution shift
• Structural constraint mechanisms such as action masking, feasibility filters, and sandboxed execution
• How training inside hard boundaries changes policy behavior and improves long-horizon stability, including across retraining cycles
• Detecting constraint violations and failure modes that do not appear in aggregate return metrics
• Lessons from applying constrained RL in production-like systems, including failures only discovered after deployment and what ultimately stopped them
• The goal is to share concrete algorithmic and system design strategies for deploying reinforcement learning in settings where violations are suboptimal.

Software engineering has seen a massive lift from AI augmentation over the last 3 years. However, data teams have been left behind. The fundamental loop of software engineering differs significantly from that of data science, making distinct agents necessary.

• Why does AI struggle with quantitative data, and where are there divergences from human experts?
• What data representations are most meaningful for today's frontier AI models?
• How can better data representation let AI become a partner in data science?
• What are the best practices for deploying AI to teams to maximize acceleration without compromising quality or security?

At Snowflake I have been straddling between product engineering and model training. I have been involved with training snowflake's own embedding model series arctic embed v1 and arctic embed v2, and snowflake's text2sql model. This talk is about how at Snowflake we decided when to train a model vs when to take advantage of open source models / partner with frontier model providers.  Next I describe briefly the models we have trained and the technical learnings we discovered along the way.