Generation

Five methods, one model (Yuu v1, Qwen3-8B according to its gateway's documentation), the same 60 blueprint cells, two samples each. Arm B also runs with the reasoning trace enabled, giving six arms. Prompts never say where to put the key; generated key positions are recorded and then re-shuffled with a seeded generator before any test.

Generation arms
Arm	What the model is given
A · zero-shot	One prompt naming the topic and a difficulty word. What a busy instructor would type.
B · blueprint	The full blueprint cell: construct definition, objective, operational demand rule, difficulty spec, target misconceptions, item-writing rules. Run with the reasoning trace off and on.
C · critic + revise	B, then a reasoning-enabled critic that solves the item and reports defects, then a revision pass (skipped when the critic reports none).
D · multi-agent	A stem writer, a distractor writer who executes each misconception, a validity reviewer, and an editor.
E · retrieval-grounded	B plus two vetted exemplar items for the same subskill, freshly drawn from the reference templates.

calls and cost

the exact prompts

Rendered for blueprint BP008 (addition rule with overlap, a development-split cell). Angle brackets mark the parts filled at run time.

A · zero-shot · 1 step

writer · reasoning false

[user]
Write one medium multiple-choice question for an introductory statistics course about addition rule with overlap. It must have four options and exactly one correct answer.

Return ONLY JSON: {"stem": "...", "options": ["...", "...", "...", "..."], "correct_index": <0-3>, "explanation": "..."}

B · blueprint · 1 step

writer · reasoning nothink or think (arm B/think)

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
Write one four-option multiple-choice item for the following blueprint cell.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Item-writing rules:
1. The stem must contain every piece of information needed; a knowledgeable student could answer it before seeing the options.
2. Exactly one option is correct. No other option may be defensible.
3. Options must be homogeneous in content and grammatical form, and similar in length.
4. Do not use "all of the above", "none of the above", or "both A and B".
5. Do not phrase the stem negatively (NOT / EXCEPT).
6. Numeric options must use the same format and precision, and every option must be a legal value for the quantity asked (e.g. a probability between 0 and 1).
7. Each incorrect option must be the answer produced by a specific misconception applied to this stem.
8. Do not put absolute words (always, never, proves) only in the incorrect options.

Return ONLY a JSON object with exactly these fields:
{
  "stem": "the question text",
  "options": ["option text", "option text", "option text", "option text"],
  "correct_index": <integer 0-3, the position of the correct option in "options">,
  "explanation": "a worked solution showing why the correct option is correct",
  "distractor_misconceptions": ["<code or KEY>", "<code or KEY>", "<code or KEY>", "<code or KEY>"],
  "distractor_rationales": ["why this option is right or what error produces it", "...", "...", "..."]
}
Use "KEY" as the misconception entry for the correct option. Do not prefix options with letters.

C · critic + revise · 3 steps

writer · reasoning false

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
Write one four-option multiple-choice item for the following blueprint cell.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Item-writing rules:
1. The stem must contain every piece of information needed; a knowledgeable student could answer it before seeing the options.
2. Exactly one option is correct. No other option may be defensible.
3. Options must be homogeneous in content and grammatical form, and similar in length.
4. Do not use "all of the above", "none of the above", or "both A and B".
5. Do not phrase the stem negatively (NOT / EXCEPT).
6. Numeric options must use the same format and precision, and every option must be a legal value for the quantity asked (e.g. a probability between 0 and 1).
7. Each incorrect option must be the answer produced by a specific misconception applied to this stem.
8. Do not put absolute words (always, never, proves) only in the incorrect options.

Return ONLY a JSON object with exactly these fields:
{
  "stem": "the question text",
  "options": ["option text", "option text", "option text", "option text"],
  "correct_index": <integer 0-3, the position of the correct option in "options">,
  "explanation": "a worked solution showing why the correct option is correct",
  "distractor_misconceptions": ["<code or KEY>", "<code or KEY>", "<code or KEY>", "<code or KEY>"],
  "distractor_rationales": ["why this option is right or what error produces it", "...", "...", "..."]
}
Use "KEY" as the misconception entry for the correct option. Do not prefix options with letters.

critic · reasoning true

[system]
You are a strict psychometric reviewer of statistics assessment items. You solve every item yourself before judging it, and you report problems precisely. You return valid JSON and nothing else.

[user]
Review this draft item against its blueprint.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Draft item (JSON):
{
 "stem": "<draft stem>",
 "options": [
  "<o1>",
  "<o2>",
  "<o3>",
  "<o4>"
 ],
 "correct_index": 0,
 "explanation": "<draft explanation>",
 "distractor_misconceptions": [
  "KEY",
  "<code>",
  "<code>",
  "<code>"
 ],
 "distractor_rationales": [
  "<r>",
  "<r>",
  "<r>",
  "<r>"
 ]
}

Procedure: first solve the item yourself without looking at "correct_index". Then check:
(1) Is the keyed option actually correct? (2) Is any other option also correct or defensible?
(3) Is information missing, or is any term, reference, or unit ambiguous?
(4) Does the item measure the stated subskill at the stated cognitive demand and difficulty?
(5) Is each incorrect option the product of a plausible, distinct misconception?
(6) Surface cues: is the key longer, more qualified, or grammatically different from the others; is any option an impossible value; is there "all/none of the above"?

Return ONLY JSON:
{"your_answer_index": <0-3 or null>, "key_is_correct": true/false, "other_correct_option": <index or null>,
  "issues": ["short, specific problem statements"], "severity": "none" | "minor" | "major"}

reviser (skipped if critic reports no issues) · reasoning false

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
Revise the draft item so that it fixes every issue raised by the reviewer, while still
matching the blueprint.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Draft item:
{
 "stem": "<draft stem>",
 "options": [
  "<o1>",
  "<o2>",
  "<o3>",
  "<o4>"
 ],
 "correct_index": 0,
 "explanation": "<draft explanation>",
 "distractor_misconceptions": [
  "KEY",
  "<code>",
  "<code>",
  "<code>"
 ],
 "distractor_rationales": [
  "<r>",
  "<r>",
  "<r>",
  "<r>"
 ]
}

Reviewer report:
{
 "issues": [
  "<critic issues>"
 ],
 "severity": "<none|minor|major>"
}

Item-writing rules:
1. The stem must contain every piece of information needed; a knowledgeable student could answer it before seeing the options.
2. Exactly one option is correct. No other option may be defensible.
3. Options must be homogeneous in content and grammatical form, and similar in length.
4. Do not use "all of the above", "none of the above", or "both A and B".
5. Do not phrase the stem negatively (NOT / EXCEPT).
6. Numeric options must use the same format and precision, and every option must be a legal value for the quantity asked (e.g. a probability between 0 and 1).
7. Each incorrect option must be the answer produced by a specific misconception applied to this stem.
8. Do not put absolute words (always, never, proves) only in the incorrect options.

Return ONLY a JSON object with exactly these fields:
{
  "stem": "the question text",
  "options": ["option text", "option text", "option text", "option text"],
  "correct_index": <integer 0-3, the position of the correct option in "options">,
  "explanation": "a worked solution showing why the correct option is correct",
  "distractor_misconceptions": ["<code or KEY>", "<code or KEY>", "<code or KEY>", "<code or KEY>"],
  "distractor_rationales": ["why this option is right or what error produces it", "...", "...", "..."]
}
Use "KEY" as the misconception entry for the correct option. Do not prefix options with letters.

D · multi-agent · 4 steps

stem writer · reasoning true

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
You are the STEM WRITER on an item-development team. Write only the question stem and its
correct answer for the blueprint below; another team member will write the incorrect options.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
- M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
- M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

The stem must contain every piece of information needed and have exactly one correct answer that can be
stated briefly (a number, or one short sentence).

Return ONLY JSON: {"stem": "...", "correct_answer": "...", "solution": "step-by-step worked solution"}

distractor writer · reasoning false

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
You are the DISTRACTOR WRITER on an item-development team. Write three incorrect options
for this stem. Produce each one by actually carrying out a specific misconception on this stem,
so that a student holding that misconception would choose it.

Stem: <stem>
Correct answer: <answer>
Worked solution: <solution>

Misconceptions to use (codes); use MISC-OTHER only if none fits:
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Every incorrect option must match the correct answer in format, precision and approximate length, must be a
legal value for the quantity asked, and must be different from the correct answer and from each other.

Return ONLY JSON: {"distractors": [{"text": "...", "misconception": "<code>", "rationale": "..."}, ...3 items]}

validity reviewer · reasoning true

[system]
You are a strict psychometric reviewer of statistics assessment items. You solve every item yourself before judging it, and you report problems precisely. You return valid JSON and nothing else.

[user]
Review this draft item against its blueprint.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Draft item (JSON):
{
 "stem": "<draft stem>",
 "options": [
  "<o1>",
  "<o2>",
  "<o3>",
  "<o4>"
 ],
 "correct_index": 0,
 "explanation": "<draft explanation>",
 "distractor_misconceptions": [
  "KEY",
  "<code>",
  "<code>",
  "<code>"
 ],
 "distractor_rationales": [
  "<r>",
  "<r>",
  "<r>",
  "<r>"
 ]
}

Procedure: first solve the item yourself without looking at "correct_index". Then check:
(1) Is the keyed option actually correct? (2) Is any other option also correct or defensible?
(3) Is information missing, or is any term, reference, or unit ambiguous?
(4) Does the item measure the stated subskill at the stated cognitive demand and difficulty?
(5) Is each incorrect option the product of a plausible, distinct misconception?
(6) Surface cues: is the key longer, more qualified, or grammatically different from the others; is any option an impossible value; is there "all/none of the above"?

Return ONLY JSON:
{"your_answer_index": <0-3 or null>, "key_is_correct": true/false, "other_correct_option": <index or null>,
  "issues": ["short, specific problem statements"], "severity": "none" | "minor" | "major"}

editor (skipped if reviewer reports no issues) · reasoning false

[system]
You are an experienced assessment item writer for a first course in probability and statistics. You write precise, unambiguous multiple-choice items. You always return valid JSON and nothing else.

[user]
Revise the draft item so that it fixes every issue raised by the reviewer, while still
matching the blueprint.

Assessment blueprint cell BP008
Construct: probability rules -- Complement, addition and multiplication rules over events in a common sample space.
Subskill: addition rule with overlap
Learning objective: Apply P(A or B) = P(A) + P(B) - P(A and B).
Prerequisites: set membership
Cognitive demand: CD2 procedural application -- Execute a known procedure on given values to produce a numeric or categorical result.
  Decision rule the item must satisfy: A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required.
Target difficulty: medium (2 steps). Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. At least one distractor is the result of a single named misconception.
Target misconceptions for the distractors (use these codes; use MISC-OTHER only if none fits):
  - M-DOUBLE-COUNT: Applies the addition rule without subtracting the overlap.
  - M-ADD-NOT-MULT: Adds probabilities of independent events where multiplication is required (or vice versa).

Draft item:
{
 "stem": "<draft stem>",
 "options": [
  "<o1>",
  "<o2>",
  "<o3>",
  "<o4>"
 ],
 "correct_index": 0,
 "explanation": "<draft explanation>",
 "distractor_misconceptions": [
  "KEY",
  "<code>",
  "<code>",
  "<code>"
 ],
 "distractor_rationales": [
  "<r>",
  "<r>",
  "<r>",
  "<r>"
 ]
}

Reviewer report:
{
 "issues": [
  "<critic issues>"
 ],
 "severity": "<none|minor|major>"
}

Item-writing rules:
1. The stem must contain every piece of information needed; a knowledgeable student could answer it before seeing the options.
2. Exactly one option is correct. No other option may be defensible.
3. Options must be homogeneous in content and grammatical form, and similar in length.
4. Do not use "all of the above", "none of the above", or "both A and B".
5. Do not phrase the stem negatively (NOT / EXCEPT).
6. Numeric options must use the same format and precision, and every option must be a legal value for the quantity asked (e.g. a probability between 0 and 1).
7. Each incorrect option must be the answer produced by a specific misconception applied to this stem.
8. Do not put absolute words (always, never, proves) only in the incorrect options.

Return ONLY a JSON object with exactly these fields:
{
  "stem": "the question text",
  "options": ["option text", "option text", "option text", "option text"],
  "correct_index": <integer 0-3, the position of the correct option in "options">,
  "explanation": "a worked solution showing why the correct option is correct",
  "distractor_misconceptions": ["<code or KEY>", "<code or KEY>", "<code or KEY>", "<code or KEY>"],
  "distractor_rationales": ["why this option is right or what error produces it", "...", "...", "..."]
}
Use "KEY" as the misconception entry for the correct option. Do not prefix options with letters.

E · retrieval-grounded · 1 step

writer · reasoning false

[user]
Write one NEW four-option multiple-choice item for the blueprint cell below. Two vetted
exemplar items for the same subskill are provided as reference for level and style. Do not copy their
numbers, scenario or wording: write a different situation that assesses the same subskill.

Exemplar 1:
Stem: In a logistics review, the probability that a randomly chosen order is express is 0.55, the probability that it arrives late is 0.27, and the probability that both are true is 0.11. What is the probability that a randomly chosen order is express or arrives late (or both)?
Options: ["0.149", "0.29", "0.82", "0.71"]
Correct option: 0.71
Why the distractors exist: Multiplies as if the question asked for the intersection of independent events.; Computes the complement of the correct answer.; Adds the two probabilities without removing the double-counted overlap.

Exemplar 2:
Stem: In a factory audit, the probability that a randomly chosen employee is a night-shift worker is 0.4, the probability that it uses the shuttle is 0.52, and the probability that both are true is 0.2. What is the probability that a randomly chosen employee is a night-shift worker or uses the shuttle (or both)?
Options: ["0.208", "0.92", "0.72", "0.28"]
Correct option: 0.72
Why the distractors exist: Multiplies as if the question asked for the intersection of independent events.; Adds the two probabilities without removing the double-counted overlap.; Computes the complement of the correct answer.

Item-writing rules:
1. The stem must contain every piece of information needed; a knowledgeable student could answer it before seeing the options.
2. Exactly one option is correct. No other option may be defensible.
3. Options must be homogeneous in content and grammatical form, and similar in length.
4. Do not use "all of the above", "none of the above", or "both A and B".
5. Do not phrase the stem negatively (NOT / EXCEPT).
6. Numeric options must use the same format and precision, and every option must be a legal value for the quantity asked (e.g. a probability between 0 and 1).
7. Each incorrect option must be the answer produced by a specific misconception applied to this stem.
8. Do not put absolute words (always, never, proves) only in the incorrect options.

Return ONLY a JSON object with exactly these fields:
{
"stem": "the question text",
"options": ["option text", "option text", "option text", "option text"],
"correct_index": <integer 0-3, the position of the correct option in "options">,
"explanation": "a worked solution showing why the correct option is correct",
"distractor_misconceptions": ["<code or KEY>", "<code or KEY>", "<code or KEY>", "<code or KEY>"],
"distractor_rationales": ["why this option is right or what error produces it", "...", "...", "..."]
}
Use "KEY" as the misconception entry for the correct option. Do not prefix options with letters.

generation lab

Audit an item in the browser with the same structural checker the benchmark uses (a JavaScript port, kept identical to the Python checker by a parity test).

Start from

Stem

Option A

Option B

Option C · key

Option D

Keyed option

A B C D

Audit · no integrity, cue or style flag

duplicate options (integrity) not firedTwo options are identical after normalisation.
numerically equal options (integrity) not firedTwo options denote the same number (e.g. 0.5 and 1/2 and 50%).
missing options (integrity) not firedFewer than four non-empty options.
key out of range (integrity) not firedThe keyed answer is a probability outside [0, 1].
all none of above (cue) not firedAn option is 'all/none of the above' or 'both A and B'.
key uniquely longest (cue) not firedThe key is the longest option by at least 20% (length cue).
key uniquely shortest (cue) not firedThe key is the shortest option by at least 20%.
distractor out of range (cue) not firedA distractor is a probability outside [0, 1]; it can be eliminated without the construct.
absolute terms in distractors (cue) not firedDistractors (only) use absolute terms such as always/never/must/proves.
stem option overlap cue (cue) not firedThe key alone repeats a content word from the stem.
grammatical mismatch (cue) not firedThe stem ends in 'a'/'an' and some options do not agree with it.
key is hedged only (cue) not firedOnly the key contains hedging language (may/might/plausibly/likely).
numeric middle key (info) not firedNumeric options with the key among the two middle values (set-level only; chance 0.5).
negative stem (style) not firedStem uses NOT/EXCEPT/LEAST phrasing.
overlong stem (style) not firedStem exceeds 110 words.
inconsistent numeric format (style) not firedNumeric options mix fractions, decimals and percentages, or differ in precision.
unit mismatch (style) not firedOptions carry different units.
option length imbalance (style) not firedLongest option is more than 2.5x the shortest.

Longest-option heuristic would answer A.