shortcuts
If the key can be found without the statistics, the item measures test-wiseness. Every probe here runs after the options are re-shuffled with a seeded generator, so position bias cannot pass for leakage.
| Probe | The answerer sees | Chance |
|---|---|---|
| Option-only (LLM) | the four options; the stem is withheld | 0.25 |
| Context-removed (LLM) | the final question sentence with digits masked, and the options | 0.25 |
| Option-only lexical model | option text and within-item shallow features; cross-validated by construct | 0.25 |
| Heuristic solvers | options (one also reads the stem) | 0.25 |
| Stem-only (LLM) | the stem only; free response mapped to an option | — |
can a model find the key without the question?
test-taking heuristics
Expected accuracy with ties split evenly, so a heuristic never wins a tie it did not earn.