Lore

Feedback-Loop Pipeline: Real-World Data to Evals

Overview

For a team building an AI product, the primary engineering job isn't prompt engineering or picking an agent framework — it's building the pipeline that converts real usage data into evals. Braintrust CEO Ankur Goyal: "If you're an engineering team and you're building an AI product, the number one job for you is to build a feedback loop."

Mechanism

Production interactions — successes, failures, edge cases users actually hit — get piped into the eval dataset, so the eval suite stays grounded in what real usage demands rather than what engineers guess it demands.

Apply

Prioritize instrumenting and piping real user interactions into your eval dataset over prompt-tuning or framework selection — the pipeline itself is the deliverable.

Related: Eval-Building Workflow (Dataset → Prompt → MCP → Scorer), Error Analysis (Manual Trace Review)

Continuous Production Monitoring

Once an LLM judge is validated, the same prompt used in CI is run continuously on a sample of live production traffic (roughly ~1000 traces/day in the example given) to track failure rates over time. This online monitoring is distinct from offline unit testing — it is what closes the loop back to real-world data, surfacing new failure modes that then feed back into another round of Open Coding and Axial Coding.