Lore

Error Analysis (Manual Trace Review)

Overview

Reading a modest sample of raw AI conversation traces by hand and writing a short note on what went wrong in each — without yet trying to root-cause or fix it — is, per Hamel Husain, the single highest-value and most commonly skipped step in building AI evals. It outperforms throwing generic automated scores at the data because it surfaces the actual, application-specific failures rather than proxies for them.

Practice

Read roughly 100 traces per product/feature and capture a one-line note of the failure per trace (see Open Coding). The pace of noticing new failure types speeds up markedly by around the 10th trace, suggesting the value is front-loaded.

Case study: Nurture Boss

A client initially viewed this manual reading as a waste of an "expensive consulting engagement." The exercise surfaced specific, real failures — such as the model not knowing today's date — that off-the-shelf hallucination, toxicity, and coherence scores would have missed entirely, because those generic metrics aren't matched to the application's actual problems.

Outcome

Once real failures are catalogued this way, teams can route each one through Eval Investment Triage: fix it outright, build a cheap code-based check, or build a validated LLM-as-judge. This is meant as a continuous, ongoing discipline — repeated on a cadence with lightweight custom tooling — not a one-off audit.

Custom Annotation Tooling

Manual trace review is sustained in practice by building a lightweight, purpose-built internal tool rather than relying indefinitely on spreadsheets or ad hoc scripts. In the Nurture Boss case study, Hamel Husain describes building such a tool — a trace viewer with channel selectors, a hidden system-prompt toggle, a free-text note field, and built-in support for Axial Coding (tagging failure categories and counting their frequency) — in under four hours.

Once a team has this minimal UI, the pace of labeling reportedly gets fast enough that reviewers get 'addicted' to it, reversing the usual reluctance to do the tedious parts of eval work. The lesson: build the bespoke tool only after manual review has already proven valuable by hand (e.g. in a spreadsheet, see Spreadsheet-Based AI Judging); don't front-load tooling investment before knowing the review is worth doing regularly.

Stopping Criterion: Theoretical Saturation

How many traces to read by hand is often anchored to a round number like 100, but this is not presented as a rigorous threshold — it's just a motivating starting point. The actual stopping rule, borrowed from qualitative research, is theoretical saturation: keep reading traces until you experience diminishing returns and stop discovering new failure patterns, whether that happens at 40 traces or 400.

This pairs with Axial Coding: saturation is reached when new traces stop producing new categories, not when an arbitrary count is hit.

Applying to Internal Tool Usage

The same review methodology can be pointed at your own team's use of an AI tool, not just at the product's outputs. Intercom, for example, runs evals over their engineers' internal usage of Claude Code to find pain points and cases where the agent hits permission-escalation friction. Treat your team's own tool-usage traces as another data source worth systematic review, feeding into the same Feedback-Loop Pipeline: Real-World Data to Evals.

ROI and Sequencing

Hamel Husain and Shreya Shankar (in Why AI Evals Are the Hottest New Skill for Product Builders) frame error analysis as the highest-ROI activity an AI product builder can do, and note that once done well it is typically a one-time grounding exercise whose findings get reused rather than repeated from scratch for every change.

They argue most teams fail by skipping straight to writing formal tests/evals before grounding themselves in real observed errors — evaluators built on assumptions about failure modes, rather than on read traces, tend to miss what's actually going wrong. Error analysis (reading real traces) must come first, and only after patterns emerge (via Open Coding then Axial Coding) should teams build narrowly-scoped LLM judges.

A key claim is that AI cannot substitute for humans at this initial review stage: an LLM asked to spot errors in a trace lacks the product/domain context to catch subtle problems (e.g., a hallucinated feature) and will typically just say the output 'looks good.' Traces themselves — logs of an AI run's tool calls, retrieval steps, and responses — are inspected via observability tools (Brain Trust, Phoenix Arize, Langsmith) as the raw material for this review, rather than relying on aggregate metrics alone.

"To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in."

Tooling and What Manual Review Catches

Custom annotation tools. When existing observability tools create friction, teams build lightweight, often 'vibe-coded' internal web apps to make trace review painless — Nurture Boss's internal tool, built in a few hours, surfaces channels, threads, and automated Axial Coding counts in one view. The tool doesn't replace the human review pass; it removes the friction that makes people skip it.

What manual review catches that an LLM reviewer won't. Example from the source: an AI dropped a lead by saying 'we don't have that, have a nice day' instead of escalating to a human. An LLM-as-judge reviewer missed this entirely, because judging it as wrong requires product context (that this class of message should always escalate) the judge was never given. This is the practical case for keeping a human trace-review pass even after LLM-as-Judge Validation is in place — some failures are only visible to someone who knows the product.