Lore

AI evals

Braintrust CEO: Evals are the new PRD for AI products

Ankur Goyal (CEO of Braintrust) argues that evals are the modern equivalent of a PRD: by quantifying "what" success looks like instead of specifying "how" to achieve it, evals let coding agents run rigorous, exhaustive experimentation on both AI products and hard technical/infrastructure problems (like database query performance) at a scale and consistency no human engineer manually matches.

How I AI · 2026-06-15 · English

Key ideas

  1. Goyal opens by rejecting the claim that AI can't handle complex engineering work, asserting no staff engineer manually runs as many rigorous benchmarks or tries as many algorithms as someone using an agent.

  2. He introduces the "agent line" — the rising threshold of work that can be handed to agents instead of humans — and argues people should re-evaluate how much of their own work already falls below it.

  3. Because models now write code well, he claims the highest-leverage move is building hard evals for any engineering problem (not just AI evals), since good success criteria let a model work creatively and autonomously in the background.

  4. He supports this with Braintrust's own query-performance work: rather than manually optimizing, they mine slow-query patterns and have agents exhaustively test literature-derived database techniques (column store formats, execution engines) at a scale no human team would attempt manually.

  5. A concrete case (a Bloom-filter index found after a week of continuous agent experimentation) illustrates how ad hoc human "probability" reasoning about complexity was wrong about half the time, removing the excuse to skip benchmarking now that agents can run it.

  6. He distinguishes theoretical human quality (could run every benchmark) from practical human quality (attention decays over hard, tedious work), arguing AI integration raises practical quality specifically because agents sustain rigor longer and more consistently than people.

  7. This reframes previously unaffordable engineering investigations (e.g., a year testing six indexing solutions) as tractable, low-risk background work a business would approve, since it runs alongside normal shipping rather than blocking it.

  8. The core underlying thesis is a "what not how" framing: machine learning already shifted programming from specifying computation steps to specifying desired outcomes (regression, transformers), and applying the same framing to any AI-assisted work — traditional engineering or AI products — increases productivity.

  9. Building on this, he states evals are literally "the modern version of a PRD": where a PRD states success in prose plus examples, an eval makes that quantifiable so a model can own the "how," demonstrated live by building a doc-answering agent, a real-user-question dataset, and a model-generated scoring function inside a sandboxed "unhinged mode" playground where full agent autonomy carries low risk.

  10. He addresses the objection that systematizing one's own taste is "building your own replacement" by arguing the opposite: encoding a taste-maker's (David's) feedback back into the scorers lets that person's quality bar apply to far more surface area than manual review ever could, making them more valuable, not less.

  11. On managing feature velocity and quality without a backlog, he claims modern product building is "carving, not constructing": about 90% of customer complaints are solved by removing the confusing feature rather than patching it, and engineering throughput is protected by investing in CI rather than shipping on a weak one.

  12. He closes with a structural claim: every engineer is now building a platform on top of which agents do the manual work engineers used to do, so the number-one job for an AI-product engineering team is building the feedback loop that turns real usage data into evals — the same principle he says applies to CI in traditional software engineering.

  13. A final anecdote qualifies the whole thesis: even he abandoned a vibe-coded eval script (judged "3,000 lines of complete trash") and hand-wrote the eval with no AI assistance to solve it and build his own understanding — treated as a deliberate, bounded exception rather than a reversal of the evals-first philosophy.

  14. The agent line — Goyal's heuristic for the rising threshold of work that can be handed to an agent instead of a human. Apply: Audit your own decisions/interactions and ask which ones would produce the same outcome if given to an agent instead of discussed in person, then delegate those.

  15. Hard evals as a general engineering technique — Building rigorous, quantifiable success criteria for any technical problem, not just AI-output quality (e.g., "why is this query slow"). Apply: Before asking an agent to fix a performance or engineering problem, define a measurable test/benchmark it must pass so the agent can work autonomously against it.

  16. Slow-query pattern mining and reproduction — Braintrust's approach of finding patterns in slow queries and reproducing them as test cases before optimizing. Apply: Instead of manually tuning a slow system, first isolate reproducible failure patterns, then hand those reproductions to an agent to iterate against.

  17. Exhaustive matrix search over storage formats/engines — Testing every combination of open-source column store format and execution engine to find the best-performing pairing. Apply: When choosing among many technical options, let an agent enumerate and benchmark the full combinatorial matrix rather than picking one based on intuition.

  18. Tantivy — An open-source search/indexing library Braintrust uses under the hood, whose built-in column store is described as weak. Apply: Cited as the motivating case for why Braintrust searched for alternative column-store solutions rather than accepting the default.

  19. Bloom filters — A probabilistic indexing data structure, adopted by Braintrust despite a "bad reputation," after a week of agent-driven experimentation showed it worked well. Apply: Don't dismiss data structures with a poor reputation without empirically testing them against your actual workload via agent-run experiments.

  20. Codex — A coding agent both speakers use for hard technical problems, noted as the only model that regularly disagrees with the user. Apply: Use Codex specifically when you need pushback/disagreement on hard problems rather than an agreeable model.

  21. Agent-driven production-data testing — Running test experiments directly against real production data with proper engineering safeguards, using agents rather than humans. Apply: When safe engineering controls exist, let an agent test against production data directly instead of relying only on staging/synthetic data.

  22. Theoretical vs. practical quality framework — Claire Vo's distinction between what humans could theoretically achieve (running every benchmark) and what they practically achieve given attention decay. Apply: Use this framing to justify AI integration on hard problems: the gain isn't theoretical capability, it's sustained practical rigor over time.

  23. Paul Graham's maker vs. manager schedule — A referenced essay/framework distinguishing maker time (uninterrupted focus) from manager time (meetings). Apply: Goyal cites it to justify blocking all meetings after noon so afternoons remain protected for deep coding work.

  24. Foreground / background / remote agent execution model — A taxonomy distinguishing local interactive agent sessions, autonomous background agents, and remote high-compute agent runs. Apply: Match the execution mode to the task: run interactive foreground agents for guided work, and offload compute-heavy or long-running experiments to remote/background agents.

  25. Named per-project terminal-multiplexer sessions — Goyal's personal practice of running each concurrent agent in its own tmux-style session (e.g., "Brain Trust One" through "Four"). Apply: Isolate concurrent agent workstreams into separate named sessions to track which agent is doing what and reduce port/resource collisions.

  26. Remote execution for compute-heavy experiments — Running experiments (e.g., EC2-to-S3 latency under thousands of concurrent reads) remotely at near-production scale instead of locally. Apply: When a local machine can't sustain a realistic-scale experiment, move it to a remote environment that mirrors production conditions.

  27. "No backlog" / backlog zero philosophy — A philosophy (also attributed to Intercom's Brian in an earlier episode) of fixing any complaint immediately rather than queuing it. Apply: Treat even minor issues (a UI paper cut) as immediately fixable given agent capacity, rather than deferring them to a backlog.

  28. In-house custom background agents on third-party infra — An emerging pattern where teams build their own background-agent tooling on top of infrastructure from major model providers. Apply: Consider building a custom background-agent layer if off-the-shelf options don't fit your workflow, since the underlying primitives (from model providers) now support it.

  29. Portless — A tool by Chris Tate (Vercel) for managing multiple local development ports more easily. Apply: Use it to avoid manual port-conflict management when running several concurrent local agent/coding sessions.

  30. Work-trees-everywhere / dedicated port ranges — An informal convention of dedicating a fixed port range (3000–3009) to each parallel task/work-tree. Apply: Assign each concurrent agent session its own reserved port range to avoid collisions when running multiple projects at once.

  31. "What not how" framing — The core mental model that ML (regression, transformers) and AI-assisted work generally shift the task from specifying implementation ("how") to specifying desired outcomes ("what"). Apply: When directing an agent or defining a project, focus effort on precisely specifying the desired outcome/success criteria and let the model handle implementation.

  32. Evals as the modern PRD — The claim that evals serve the same function a PRD does (stating success via prose/examples) but make it quantifiable so a model can act on it. Apply: Replace or supplement written PRDs with a dataset of real inputs plus a scoring function that quantifies what "good" looks like for the feature.

  33. Braintrust live eval-building workflow — The demoed end-to-end process: collect real user questions into a dataset, build an answering prompt, attach an MCP server, and generate a scoring function to evaluate outputs. Apply: Mirror this pipeline for any AI feature: gather real usage questions, build a baseline prompt/agent, then have a model draft a scorer against your stated criteria and run it across outputs.

  34. MCP (Model Context Protocol) server — A protocol/server (Braintrust's own) attached to a model in the demo to give it access to documentation context. Apply: Attach an MCP server to give an agent grounded access to your product's own documentation or data during eval-building or normal use.

  35. Context7 — A documentation-indexing tool tried as an alternative/complement to MCP in the demo. Apply: Use it to index external docs for a model to consult, comparable to or alongside an MCP server.

  36. Model-generated scoring functions — Having the AI itself draft the scoring/evaluation function from natural-language criteria instead of hand-writing it. Apply: State your scoring criteria in plain language (e.g., "concise code, one language, no em dashes") and ask a model to turn that into an executable scorer.

  37. "Unhinged mode" sandboxed agent autonomy — Letting an agent operate with high autonomy inside a sandboxed playground (data/prompts only, no bash/system access), where the blast radius of mistakes is low. Apply: Grant an agent maximal freedom to experiment only within environments where a wrong move can't damage real systems, such as an eval playground.

  38. Two-tier quality process (evals + "quantify David" vibe check) — Running automated evals first, then a periodic human taste-maker review, whose specific feedback is then encoded back into the scorers. Apply: Pair automated evals with recurring human spot-checks from your best taste-holder, and translate each correction they give into a new scoring rule.

  39. CI investment as a velocity lever — The claim that engineering velocity under AI-heavy workflows is bottlenecked by CI quality, not by agent capability. Apply: When asked how to speed up engineering with AI, invest in strengthening CI first and pause feature shipping until CI "earns" the ability to move faster.

  40. Feedback-loop pipeline (real-world data → evals) — The claim that the primary job of an AI-product engineering team is building the pipeline that converts real usage data into evals. Apply: Prioritize instrumenting and piping real user interactions into your eval dataset over prompt engineering or agent-framework selection.

  41. Running evals on internal tool usage — Using evals to analyze internal usage of an AI tool itself (Intercom's example of evaluating their team's use of Claude Code) to find pain points and permission-escalation friction. Apply: Apply the same eval methodology internally to your team's own AI-tool usage to surface where engineers struggle or agents request excessive permissions.

  42. "Close the session and start from scratch" recovery strategy — The default response to a failing/uncooperative agent session: abandon it, improve the evals, and restart rather than manually steering it harder. Apply: When an agent session goes off track, close it and improve your evals/success criteria before retrying, instead of trying to argue or coax the existing session.

  43. Deliberate hand-coding without AI assistance — A bounded fallback of writing code with no co-pilot or autocomplete when agent-generated code becomes unmanageably complex, partly to rebuild personal understanding. Apply: If AI-generated code balloons into unmanageable complexity, deliberately turn off AI assistance for that specific task and hand-write the solution to regain understanding and control.

Insights

Teams historically skipped rigorous benchmarking by "probability-ing" it based on complexity intuition (e.g., assuming something is log n, not n²) and were wrong about half the time — agents remove the excuse to keep doing this.

Testing directly against production data is framed as arguably safer with agents than with humans, because human production testing is typically unsupervised ("no one's looking at it") whereas agent-driven testing can be engineered with guardrails.

In large data-migration work, the real bottleneck was never scripting capability but the throughput of human judgment on ambiguous edge cases (yes/no/left/right decisions); pairing precise success criteria with an agent that can grind through the long tail removes that bottleneck.

"Chunking" AI usage into deliberate blocks — rather than running agents continuously and feeling guilty when not actively "kicking off agents" — is presented as both more enjoyable and more productive, directly countering the "cloud anxiety" burnout some AI-heavy coders report.

Sandboxed environments (data/prompts only, no bash or system access) enable a qualitatively different "unhinged mode" of agent autonomy that would be unacceptably risky on a real machine but is safe in a playground — autonomy risk is a property of the environment, not just the agent.

The vibe check isn't eliminated by quantitative evals — it's preserved and compounded: a human taste-maker's specific corrections get fed back into the scorers ("quantify David"), turning subjective taste into reusable infrastructure that improves each round.

CI quality, not agent tooling itself, is positioned as the actual lever for engineering velocity under AI-heavy workflows: weak CI should trigger pausing to strengthen it rather than shipping faster on top of it.

The company most identified with "no excuse not to eval" still reaches a hand-coded, no-AI exception when agent-generated code becomes unmanageably complex — showing the philosophy has a practical, self-imposed limit rather than being applied dogmatically.

«I so viscerally disagree with there's no staff engineer who is running as many rigorous benchmarks and trying out different algorithms and analyzing ideas manually than someone who's using an agent.»

— 00:09

«Everyone should take a hard look in the mirror and re-evaluate how they spend their time.»

— 00:21

«I think like many of these things to me fit below the agent line. I think the agent line keeps going up.»

— 00:30

«Now that models are so good at actually writing code, one of the best things that we can do is create really hard evals.»

— 00:41

«I have a lot of people saying, "Wow, if I go as so far as to turn my own taste or my own skills or my own expertise into a system, I'm functionally just building my own replacement.»

— 00:57

«We're able to have David's pallet applied to more things. I think the quality bar that we're able to hit is higher because we're able to get more things to that bar.»

— 01:08

«Half the time you're wrong. Now there is no excuse to not do those benchmarks.»

— 10:19

«The practical quality of integrating AI into your engineering process on very hard technical problems goes up simply because of how hard you can run at the problem and how long and consistently you can run against the problem.»

— 11:19

«There's just no excuse to not have rigor... there's no excuse to to just not improve these things.»

— 15:29

«I think it's currently the only model that will disagree with you regularly. And I think if you're working on hard problems, it's very important.»

— 17:35

«I like the idea of chunking your time with AI a little bit more.»

— 21:11

«So, yes, close your laptop, people. Close your laptop.»

— 22:08

«Machine learning specifically shifts the task of programming from being about the how to being about the what.»

— 23:47

«In my opinion, EVELs are actually the modern version of a PRD.»

— 25:18

«This agent is running inside of this playground and it's using like data and and some prompts and stuff. So the risk of letting it just go and try stuff out is actually very low.»

— 27:56

«Product building and code writing is now looks like carving rather than constructing.»

— 34:28

«Every engineer is now building a platform, and upon the platform agents are doing the work that the engineers were doing manually.»

— 36:18

«If you're an engineering team and you're building an AI product, the number one job for you is to build a feedback loop.»

— 36:35

«Close the session. Um, and then I improve the evals and then I try from scratch again.»

— 37:56

«and it's like 3,000 lines of complete trash and it had like all these scoring functions and all this crap and it was getting confused»

— 38:38

Reception

Comments are overwhelmingly positive and engaged, praising the episode's insights on evals with substantive discussion and practical advice, alongside a few clarifying questions and minor critiques of AI coding tools.

A dense, practitioner-level walkthrough that pairs a concrete live demo of eval-building in Braintrust with reusable heuristics (the agent line, what-not-how, vibe-check-then-quantify, carving not constructing), making it a strong reference for eval methodology and agent-driven engineering practice despite spending most of its technical depth on a single company's internal database-performance anecdotes.

40:12

↳ How I AI · YouTube

Watch original