AI evals
Hamel Husain and Shreya Shankar argue that building evals — a systematic, human-led error-analysis process (open coding, then axial coding, then narrowly-scoped automated evaluators) — is the highest-ROI, most necessary skill for AI product builders, and that most 'evals don't work' backlash comes from people who skipped this grounding process and built ungrounded, unvalidated LLM judges.
Hamel and Shreya argue that building evals is the single highest-ROI skill for AI product builders, and that the underlying error-analysis process, once done well, is typically a one-time setup that then gets reused rather than repeated from scratch.
They frame the goal of evals as actionable product improvement, not procedural perfection — imperfect but frequent iteration beats an exhaustive, correct-on-paper eval suite.
They attribute widespread 'anti-eval' sentiment on platforms like X to people being burned by badly built evals (e.g., naive Likert-scale judges) rather than to evals being inherently useless, and argue much of the debate is really a definitional disagreement about what 'eval' even means.
A core claim is that AI cannot substitute for humans at the initial data-review stage, because an LLM asked to spot errors in a trace lacks the product/domain context to catch subtle problems (e.g., a hallucinated feature) and will typically just say the output 'looks good.'
They argue that most teams fail by writing formal tests/evals before grounding themselves in real observed errors, and insist error analysis (reading real traces) must come first.
They argue eval-building should be selective and prioritized, not exhaustive: most identified issues are fixable by editing the prompt directly, so a product typically only needs a handful of formal LLM-judge evaluators (roughly four to seven) reserved for recurring or business-critical failure modes.
They argue that skipping validation of an LLM judge against human judgment is how teams end up with evals that don't reflect reality, and that this failure erodes not just trust in the eval but in the person who built it.
They reframe a well-built LLM-judge prompt as functioning like a living, continuously running product requirements document, grounded in real data and updated as reviewers encounter failure modes ('criteria drift') they couldn't have anticipated upfront — while still recommending a traditional PRD as a starting point.
They argue evals and A/B testing are not rival or separate practices but both instances of applying data-science thinking to AI products, since a meaningful A/B test requires an eval metric to compare against; the broader claim is that eval work doesn't require inventing new tooling, just applying established analytics discipline to a stochastic new kind of system.
They argue that coding agents like Claude Code only appear to skip formal evals ('we just vibe') because they inherit rigor from their foundation model's own extensive benchmarking and because their expert developer-users constantly dogfood the product — a shortcut that doesn't generalize to domains lacking natural expert dogfooding, like medicine.
They present the entire methodology as a learnable, teachable skill rather than an innate talent or something requiring a purchased tool, framing their own course's popularity and their consulting outcomes (e.g., issues found in every client trace review) as evidence that structured error analysis reliably surfaces real, actionable problems.
Error analysis — The foundational first step of building evals — a long-standing ML/data-science practice of systematically reviewing an AI application's real output data to find what's going wrong, which the hosts say they didn't invent but adapted (citing an ~8-year-old Andrew Ng video on the technique for stochastic systems). Apply: Before writing any tests, pull a sample of real traces/logs and read through them to ground what actually needs fixing.
Trace — A detailed log of a sequence of events in an AI application's run (tool calls, retrieval, responses), viewed via an observability tool such as Brain Trust, Phoenix Arize, or Langsmith. Apply: Inspect individual traces in an observability tool as the raw material for error analysis rather than relying on aggregate metrics alone.
Open coding — Writing free-form, informal notes on individual traces to flag what's wrong, a term borrowed from qualitative research methodology and described as unsuitable for full automation because it requires product/domain context an LLM lacks. Apply: For each sampled trace, write one specific note capturing the first/most upstream thing that went wrong, in plain language a later reader can understand (not vague words like 'janky').
Benevolent dictator — A term Hamel coined for appointing one trusted domain-expert (often the PM) to make subjective judgment calls during open coding, instead of running the process as a committee. Apply: Pick a single person with domain expertise to own the open-coding judgment calls so the process stays cheap and fast rather than bogged down in consensus-building.
Most-upstream-error rule — A rule for open coding stating you should log only the first/most upstream error observed in a trace and move on, rather than cataloguing every issue in it. Apply: When a trace has multiple problems, note only the earliest one and skip to the next trace to keep review throughput high.
Theoretical saturation — A stopping heuristic borrowed from qualitative data analysis: stop reviewing traces once you stop discovering new note types/concepts, rather than hitting a fixed count. Apply: Keep sampling and open-coding traces (roughly 100 as a non-intimidating starting target, sometimes as few as 15-60 in practice) until new problems stop appearing.
Axial coding — Clustering the raw open-code notes into named failure-mode categories, typically with LLM assistance (e.g., feeding a CSV of notes into a Claude project or ChatGPT and asking it to synthesize categories). Apply: Export open-code notes to a spreadsheet/CSV, prompt an LLM to group them into actionable, specific failure-mode categories, then edit those categories yourself rather than accepting them as-is.
'None of the above' diagnostic category — Adding a catch-all category when using an LLM to auto-label notes into axial codes, used as a signal that the taxonomy is incomplete. Apply: If the AI frequently classifies notes as 'none of the above,' return to open coding to add or reword categories rather than forcing bad fits.
Basic counting / pivot-table frequency analysis — Tallying how often each axial code occurs across labeled traces using a pivot table, described as the most powerful and undervalued analytical technique in data science. Apply: Build a pivot table of axial-code counts to prioritize which failure mode to tackle first, and drill into underlying traces per category by double-clicking cells.
Code-based evals / automated evaluators — Unit-test-like, deterministic checks (e.g., valid JSON, correct format) used for failure modes that are binary/deterministic and cheap to check in code. Apply: For failure modes with a clear yes/no correctness check, write a plain code assertion instead of building an LLM judge.
LLM-as-judge — An LLM prompted to evaluate one narrow, specific failure mode with a binary pass/fail verdict, used for subjective/complex issues that can't be captured in code (e.g., whether a human handoff was needed). Apply: Write a scoped judge prompt covering explicit trigger conditions for one failure mode, run it on traces, and use it in both CI/unit tests and production monitoring.
Binary (true/false) scoring principle — A design rule that LLM judges should output binary pass/fail rather than 1-5 or 1-7 rating scales, because scales let evaluators avoid decisions and produce uninterpretable averages. Apply: When designing a judge prompt, force a true/false verdict rather than a numeric scale.
Confusion-matrix judge validation — Comparing an LLM judge's verdicts against human/axial-code labels in a human-label-by-judge-label pivot table, rather than relying on a single bare agreement percentage which can be misleadingly high for rare failure modes. Apply: Before trusting a judge, build a confusion matrix of human vs. judge labels and iterate on the judge prompt until the off-diagonal (disagreement) cells shrink.
Online/production monitoring sampling — Running validated LLM judges continuously on a sample of live production traces (e.g., ~1000/day) to track failure rates over time, distinct from offline unit testing. Apply: Deploy the same judge used in CI as a daily/hourly monitor on sampled production traffic to get a real, current quality measure.
Evals-as-PRD framing — Framing an LLM-judge prompt as functioning like a living product requirements document — encoding conditional 'if this, then that' behavior expectations that run automatically and continuously and get updated as data reveals unanticipated expectations ('criteria drift'). Apply: Still write a traditional PRD before building the product, then treat the judge prompt as a data-driven refinement of that PRD that evolves as new failure modes surface.
Vibe checks — Informal, unsystematic spot-checking of an AI product's outputs, described as fine for an early/first pass but unmanageable as the product scales. Apply: Use vibe checks only as an initial sanity check, then replace them with structured error analysis once the product has real usage.
Dogfooding as informal/human eval — Heavy internal usage of a product by domain-expert developers (e.g., coding-agent engineers using their own tool all day), argued to function as a manual, human substitute for formal evals when the practitioner is also the domain expert. Apply: In domains where your team are genuine domain experts and heavy users of the product, rigorous dogfooding plus attentive review of results can substitute for or supplement formal LLM judges — though this doesn't generalize to domains lacking natural expert-user overlap, like medicine.
AB testing as eval-dependent — The claim that A/B testing is itself a form of eval — comparing two conditions requires a quality metric, which comes from eval work — so eval and A/B testing are the same underlying data-science discipline rather than competing approaches. Apply: Ground A/B test hypotheses in prior error analysis of real failure modes rather than testing pre-conceived feature ideas, and use eval metrics as the comparison criterion.
Custom annotation tool building — Building lightweight, often 'vibe-coded' internal web apps/interfaces tailored to a specific product to remove friction from viewing traces and running error analysis (e.g., Nurture Boss's internal tool built in a few hours). Apply: If existing observability tools create friction, quickly build (or vibe-code) a custom interface showing channels, threads, and automated axial-code counts to make trace review as easy as possible.
Cost optimization via model substitution — Replacing an expensive state-of-the-art model (e.g., GPT-5) with a cheaper variant (e.g., a 'nano' or 'mini' model) in parts of the pipeline while maintaining quality, taught as a dedicated course topic. Apply: Once a task's quality bar is validated, test whether a smaller/cheaper model can meet it and swap it in to cut cost.
Criteria drift ('Who Validates the Validator') — A concept from Shreya Shankar's research paper 'Who Validates the Validator,' stating that people's opinions of what counts as good/bad output shift as they review more outputs, and they typically discover new failure modes only after seeing roughly ten outputs they hadn't anticipated — even as experienced practitioners. Apply: Expect and plan for your evaluation rubric to change as you review more data; don't treat an initial rubric as final.
Adding a deliberate 'none of the above' bucket when an LLM auto-labels notes into categories works as a diagnostic: if the AI reaches for it often, the failure-mode taxonomy is incomplete, not the data.
Vague open-code notes like 'janky' break the whole downstream pipeline — neither a human nor an AI can later categorize or act on them — so the detail discipline of the first note-taking pass is the lynchpin of the entire method.
A judge that simply always says 'pass' can still score 90% agreement with human labels when the true error rate is only about 10%, which is why the speakers call bare agreement percentages a 'dangerous metric' on their own.
Manual trace review catches issues an LLM reviewer misses entirely — e.g., an AI dropping a lead by saying 'we don't have that, have a nice day' instead of escalating to a human — because judging it 'wrong' requires product context the LLM doesn't have.
Once the error-analysis pipeline is built, ongoing maintenance is reportedly cheap (about 30 minutes a week), which cuts against the common perception that evals are a heavy, continuous burden.
Mature products end up needing surprisingly few formal LLM judges (four to seven, per Shreya), because most discovered failures turn out to be missing prompt instructions rather than genuinely ambiguous judgment calls.
Even experienced practitioners who've built many LLM pipelines reportedly only discover new failure modes after reviewing roughly ten outputs they hadn't anticipated — evidence, per Shreya's 'Who Validates the Validator' research, that evaluation rubrics can't be fully specified before you start looking at data.
«To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in.»
— 00:01
«The goal is not to do evals perfectly. It's to actionably improve your product.»
— 00:18
«The first step in conquering data like this is just to write notes.»
— 17:21
«I like that you're using the word janky. Shows you just how informal this can be at this stage.»
— 21:53
«basic counting is the most powerful analytical technique in data science.»
— 32:20
«We didn't invent error analysis. We don't actually want to invent things. That's bad signal.»
— 37:26
«This is like the purest sense of what a product requirements document should be is this eval judge that's telling you exactly what it should be and it's automatic and running constantly.»
— 61:37
«People's opinions of good and bad change as they review more outputs. They think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place.»
— 64:24
«you can easily have 90% agreement by just having a judge say... it passes all the time.»
— 58:59
«coding agents are fundamentally very different than other AI products because the developer is the domain expert.»
— 73:53
«It's not magic. Like anyone can do this. You're going to have to practice the skill like any new skill you have to practice but you can do it.»
— 69:38
«you can't really do an AB test without the eval to compare.»
— 76:34
Reception
Comments are overwhelmingly positive and appreciative of the podcast's depth on AI evals, with a handful of substantive critiques about scope limitations (multi-agent systems, A/B testing coverage) and one dismissive troll comment.
A methodologically dense, example-driven walkthrough that grounds its claims in a real (anonymized) production system and named techniques from qualitative research rather than abstract advice, though it is explicitly framed by its own creators as 'scratching the surface' of a multi-day course.

106:33