failures
Ten defect categories. An item can fall in several. Categories that depend on the same-family judge are marked, because that judge is not independent of the generators.
| Category | Rule | Evidence |
|---|---|---|
| wrong answer | the key is contradicted (PoT, or both answering probes) | deterministic / answer proxy |
| duplicate or multiple correct | duplicate options, numerically equal options, or PoT matches the key and another option | deterministic / PoT |
| ambiguous wording | judge: unanswerable, or clarity at most 2 | same-family LLM judge |
| unrealistic distractor | a distractor is an impossible value, or judge distractor quality 1 | deterministic + judge |
| construct drift | judge: answering does not require the stated subskill | same-family LLM judge |
| difficulty mismatch | judge difficulty differs from the requested difficulty (generated items) | same-family LLM judge |
| surface leakage | any cue flag, or the option-only probe finds the key on every draw | deterministic + probe |
| explanation error | a rationale computes a value that is not its option, or wrong options labelled as the key | deterministic |
| overlong stem | stem longer than 110 words | deterministic |
| format artifact | parse failure, mixed numeric formats, unit mismatch, all/none of the above | deterministic |