Lore

Eval Investment Triage

Overview

Not every problem found in Error Analysis (Manual Trace Review) deserves a formal eval, and not every product needs a formal eval program at all — effort should scale to the problem and to the product's stage.

Triage by problem type

Triage by product stage

Pre-launch or low-usage products can rely on the builder's own hands-on use of the product as informal error analysis — no formal eval program needed yet. Formal evals become necessary once usage outgrows what one person can personally track by hand. Gradual rollouts are used to accumulate real usage data safely during this transition.

Framing

This triage is what turns the generic advice "look at your data" into a concrete workflow: read traces, annotate failures, then route each failure to the cheapest sufficient fix — direct fix, code check, or validated judge — rather than reflexively building an eval for everything.

Selective Scope and Pragmatic Goal

Husain and Shankar frame the goal of eval work as actionable product improvement, not procedural perfection: imperfect but frequent iteration beats an exhaustive, correct-on-paper eval suite.

"The goal is not to do evals perfectly. It's to actionably improve your product."

They argue this means eval-building should be selective rather than exhaustive: most issues surfaced by Error Analysis (Manual Trace Review) are fixable by directly editing the prompt, so a product typically only needs a handful of formal LLM-judge evaluators — roughly four to seven — reserved for failure modes that are recurring or business-critical, not every issue found during review.

Maintenance Cost and Right-Sizing Judge Count

Two data points from Hamel Husain and Shreya Shankar's course, both cutting against the perception that evals are a heavy, continuous burden. First, once the Error Analysis (Manual Trace Review) pipeline (open coding → Axial Coding → automated evaluators) is built, ongoing maintenance is reportedly cheap — about 30 minutes a week. Second, mature production systems end up needing surprisingly few formal LLM judges — four to seven, per Shreya Shankar — because most failures discovered through error analysis turn out to be missing prompt instructions rather than genuinely ambiguous judgment calls that need a dedicated judge.

Linear: Don't Evaluate Where Variance Is Legitimate

Linear's team explicitly pushes back on maximal eval coverage for agent behavior: evaluating every agent output as if there's one correct answer produces false signal when variance between good outputs is legitimate and expected. Their rule is to target evals only where consistency genuinely matters — e.g., the agent's hero paths — and to leave looser, exploratory behavior unmeasured rather than force it into a pass/fail harness it doesn't fit.