Lore

Eval-Building Workflow (Dataset → Prompt → MCP → Scorer)

Overview

A concrete, repeatable pipeline for turning a product surface into an eval, demonstrated live by Ankur Goyal (Braintrust):

  1. Collect real user questions into a dataset (not synthetic examples).
  2. Build a baseline prompt/agent that answers those questions.
  3. Attach an model-context-protocol (MCP) server so the model has grounded access to the product's own documentation.
  4. Generate a scoring function from the stated success criteria and run it across the outputs.

Application

This is the practical instantiation of Evals as the Modern PRD ("What Not How"): instead of writing a prose PRD describing what a good answer looks like, the team encodes "good" as a dataset of real inputs plus a scorer, then lets a model draft and refine the scorer against stated criteria. Mirror this pipeline for any AI feature — gather real usage, build/attach context, then have a model generate the scorer rather than hand-writing it.

Related

Evals as the Modern PRD ("What Not How"), model-context-protocol

Model-Generated Scoring Functions

Rather than hand-writing the scorer, the model itself can draft the scoring function from natural-language criteria: state what you want in plain language (e.g., "concise code, one language, no em dashes") and ask a model to turn that into an executable scorer. This turns the Scorer step of the workflow into a prompting task instead of a coding task.