Criteria drift is Hamel Husain and Shreya Shankar's term for the way an eval's judging criteria change over time as reviewers encounter real failure modes they could not have anticipated when the criteria were first written. Because a product's actual failure surface is only discovered through Error Analysis (Manual Trace Review) on live traces, the rubric used by an LLM judge is expected to evolve — it is not a spec fixed at design time. This is the mechanism underlying why they treat the judge prompt as a living document (see Evals as the Modern PRD ("What Not How")) rather than a one-time deliverable.
Source: Why AI Evals Are the Hottest New Skill for Product Builders — Hamel Husain & Shreya Shankar.
Shreya Shankar's research paper Who Validates the Validator? is the named empirical source for this phenomenon: reviewers' opinions of what counts as good or bad output shift as they see more outputs, and even experienced practitioners — people who have built many LLM pipelines before — typically only surface a failure mode they hadn't anticipated after reviewing roughly ten outputs. See theoretical-saturation for the related stopping-rule this finding underpins, and Open Coding for the note-taking pass where drift first shows up in practice.