Open coding is the raw, unstructured note-taking format used during Error Analysis (Manual Trace Review): for each trace read, the reviewer jots down whatever they notice going wrong as free text, without pre-imposed categories. The term is borrowed from qualitative social-science research and has long been used in ML/data-annotation work.
Capture an open code per trace as you go rather than trying to fit it into a taxonomy immediately. Organizing these free-text notes into named failure categories is a separate, later step.
From the same source: the hosts describe the entry point to Error Analysis (Manual Trace Review) as simply writing free-form notes while reading traces, before any structure is imposed — the axial-coding step comes later. They deliberately embrace the informality of this stage.
"The first step in conquering data like this is just to write notes."
"I like that you're using the word janky. Shows you just how informal this can be at this stage."
When a trace exhibits multiple problems, log only the first/most upstream error and move to the next trace rather than cataloguing everything wrong with it — this keeps review throughput high enough to reach theoretical-saturation. Notes should be specific enough for someone else to act on ("the retrieval returned no results for a valid query," not "janky"), since open coding requires product/domain context an LLM lacks and can't be fully automated. Judgment calls about what counts as an error are best made by one person — see benevolent-dictator — rather than by committee.
Vague notes taken during the open-coding pass — e.g. just 'janky' — break everything downstream: neither a human nor an AI can later group or act on a note that doesn't say what was actually wrong. The detail discipline of this first note-taking pass is the lynchpin of the whole Error Analysis (Manual Trace Review) method; sloppiness here isn't recoverable in Axial Coding or in the automated evaluators built afterward.