Hamel Husain and Shreya Shankar argue that evals and A/B testing are not rival or separate practices but both instances of applying established data-science thinking to AI products: a meaningful A/B test requires an eval metric to compare treatment against control, so you cannot run one without the other. Their broader claim is that eval work doesn't require inventing new tooling or a new discipline — it's the same analytics rigor teams already use elsewhere, applied to a stochastic new kind of system (LLM-based products). This connects to Error Analysis (Manual Trace Review) as the grounding step that produces the metric an A/B test would need, and to Eval Investment Triage for deciding which metrics are worth formalizing.
Source: Why AI Evals Are the Hottest New Skill for Product Builders — Hamel Husain & Shreya Shankar.
Direct quote from the source: 'you can't really do an AB test without the eval to compare.' Comparing two conditions in an A/B test requires a metric of quality, and that metric is produced by eval work — so the two aren't competing practices, they're the same underlying data-science discipline applied at different points. Practical implication: ground A/B test hypotheses in prior Error Analysis (Manual Trace Review) of real, observed failure modes rather than in pre-conceived feature ideas, and reuse the eval's scoring criteria as the A/B comparison metric.