Quantitative evals (see Eval-Building Workflow (Dataset → Prompt → MCP → Scorer)) don't replace human taste — they compound it. The process runs automated evals first, then periodically has a human taste-maker do a qualitative "vibe check" pass. Braintrust calls the internal version of this "quantify David": whatever specific correction the taste-maker gives gets encoded back into the scoring functions, so the next round of evals already reflects that judgment.
Each iteration: (1) automated evals catch the bulk of regressions cheaply, (2) a trusted human reviewer periodically samples output for taste/quality issues evals miss, (3) their specific feedback is translated into a new or adjusted scorer, so the qualitative signal becomes reusable, quantitative infrastructure rather than being re-litigated every time.
Pair automated evals with recurring spot-checks from your best taste-holder, and treat every correction they give as a spec for a new scoring rule, not just a one-off fix.
Related: Taste as the Human Differentiator, LLM-as-Judge Validation
Hamel Husain and Shreya Shankar frame the informal 'vibe check' half of this process as a fine first pass — an early, unsystematic spot-check of outputs — but explicitly unmanageable once a product has real usage and volume. Vibe checks have no mechanism for surfacing failure modes systematically; that's the job Error Analysis (Manual Trace Review) (open coding → Axial Coding → narrow automated evaluators) takes over as soon as ad hoc spot-checking stops being tractable.