methods
One domain, defined narrowly enough that construct alignment means something: introductory probability and statistics, at the level of a first university course.
cognitive demand
Five levels, each with an observable decision rule instead of a verb list.
| Code | Level | Decision rule | Example |
|---|---|---|---|
| CD1 | recall | The item can be answered correctly by someone who has memorised a definition and has performed zero arithmetic and zero interpretation of a context. No numbers need be combined. | Which symbol denotes the population standard deviation? |
| CD2 | procedural application | A single named procedure applies, the values needed are given explicitly, and the solution is 1-2 arithmetic steps. No decision about *which* procedure applies is required. | A binomial has n=10, p=0.3. Compute P(X=2). |
| CD3 | conceptual interpretation | The item asks what a quantity *means*, or which statement about it is true. The correct answer cannot be obtained by computation alone; distractors encode plausible misreadings. | A 95% confidence interval for mu is (4.1, 6.3). Which interpretation is correct? |
| CD4 | multistep reasoning | A correct solution requires at least two distinct steps that could each be performed wrongly, and at least one step involves choosing the applicable rule rather than being told it. | Given sensitivity, specificity and prevalence, find the positive predictive value. |
| CD5 | transfer | The item inverts the usual direction (solve for an input given an output), uses a representation not used in instruction, or requires judging whether the construct applies at all. | What sample size makes the standard error of the mean at most 0.5, given sigma=4? |
| Target | Solution steps | Distractor policy | Description |
|---|---|---|---|
| easy | 1 step | Distractors are clearly wrong to a student who knows the definition. | A student who has learned the target subskill and nothing else should answer correctly. All required values are stated; no distractor is one arithmetic slip away from the key. |
| medium | 2 steps | At least one distractor is the result of a single named misconception. | Requires combining two given quantities or one procedure plus one interpretation. A student with a partial understanding should be attracted to a specific distractor. |
| hard | 3+ steps, or inverted direction | Every distractor corresponds to a distinct, documented misconception. | Requires three or more dependent steps, or solving backwards from a desired result, or recognising that a commonly applied rule does not apply here. |
constructs
C01 conditional probability
The probability of an event evaluated within a restricted sample space defined by a second event. Prerequisites: proportions, set membership.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C01.1 direction of conditioning | Distinguish P(A|B) from P(B|A) in a described situation. | M-INV-COND, M-JOINT-AS-COND |
| C01.2 conditional from two way table | Compute a conditional probability from counts in a two-way table. | M-WRONG-DENOM, M-JOINT-AS-COND |
| C01.3 conditional vs joint | Distinguish P(A and B) from P(A|B) given a verbal description. | M-JOINT-AS-COND, M-WRONG-DENOM |
C02 bayes rule
Updating a prior probability with evidence of known reliability to obtain a posterior probability. Prerequisites: conditional_probability, law of total probability.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C02.1 posterior from screening | Compute a positive predictive value from prevalence, sensitivity and specificity. | M-BASERATE, M-INV-COND |
| C02.2 base rate reasoning | Explain why a highly accurate test yields a low posterior when prevalence is low. | M-BASERATE, M-INV-COND |
C03 independence
Two events are independent when conditioning on one leaves the probability of the other unchanged. Prerequisites: conditional_probability.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C03.1 test independence from table | Decide whether two events are independent by comparing P(A|B) with P(A). | M-INDEP-VS-EXCL, M-ADD-NOT-MULT |
| C03.2 independent vs mutually exclusive | Distinguish independence from mutual exclusivity. | M-INDEP-VS-EXCL |
C04 probability rules
Complement, addition and multiplication rules over events in a common sample space. Prerequisites: set membership.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C04.1 addition rule with overlap | Apply P(A or B) = P(A) + P(B) - P(A and B). | M-DOUBLE-COUNT, M-ADD-NOT-MULT |
| C04.2 complement rule | Use the complement rule, including 'at least one' phrasing. | M-COMPLEMENT, M-ADD-NOT-MULT |
C05 expected value
The probability-weighted mean of a random variable, and its behaviour under linear transformation. Prerequisites: probability_rules.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C05.1 expectation from pmf | Compute E[X] from a probability mass function. | M-SD-VS-VAR, MISC-OTHER |
| C05.2 linearity of expectation | Apply E[aX+b] = aE[X] + b. | M-VAR-LINEAR, MISC-OTHER |
C06 variance and sd
Spread of a random variable, and its behaviour under linear transformation. Prerequisites: expected_value.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C06.1 variance from pmf | Compute Var(X) from a pmf and distinguish it from SD(X). | M-SD-VS-VAR |
| C06.2 variance under linear transform | Apply Var(aX+b) = a^2 Var(X). | M-VAR-LINEAR, M-SD-VS-VAR |
C07 binomial distribution
The distribution of the number of successes in a fixed number of independent Bernoulli trials. Prerequisites: probability_rules, independence.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C07.1 binomial point probability | Compute P(X = k) for a binomial random variable. | M-BINOM-NO-COEF, M-BINOM-AT-MOST |
| C07.2 binomial cumulative | Distinguish and compute P(X <= k) versus P(X = k). | M-BINOM-AT-MOST, M-COMPLEMENT |
| C07.3 binomial conditions | Judge whether the binomial conditions hold in a described situation. | M-INDEP-VS-EXCL, MISC-OTHER |
C08 normal distribution
Standardisation and area under the normal curve. Prerequisites: variance_and_sd.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C08.1 zscore computation | Standardise a value and interpret the resulting z-score. | M-Z-SIGN, M-SD-VS-VAR |
| C08.2 normal area | Find the probability that a normal variable falls in an interval. | M-Z-SIGN, M-COMPLEMENT |
| C08.3 inverse normal | Find the value cutting off a stated proportion of a normal distribution. | M-Z-SIGN, M-COMPLEMENT |
C09 sampling distribution
The distribution of a statistic over repeated samples, and how its spread depends on n. Prerequisites: normal_distribution, variance_and_sd.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C09.1 standard error of mean | Compute and interpret sigma/sqrt(n). | M-SIGMA-VS-SE, M-SE-NO-SQRT |
| C09.2 effect of sample size | Predict how the sampling distribution changes as n increases. | M-SE-NO-SQRT, M-SAMPLE-SIZE-POP |
C10 confidence interval
An interval estimate whose coverage refers to the procedure across repeated samples. Prerequisites: sampling_distribution.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C10.1 ci computation | Compute a confidence interval for a mean given the standard error and critical value. | M-SIGMA-VS-SE, M-SE-NO-SQRT |
| C10.2 ci interpretation | Select the correct long-run-coverage interpretation of an interval. | M-CI-PROB-PARAM, M-CI-INDIVIDUALS |
C11 hypothesis test logic
The logic of assuming a null model and measuring the extremeness of observed data under it. Prerequisites: sampling_distribution.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C11.1 pvalue interpretation | State what a p-value is the probability of. | M-P-AS-H0-PROB, M-P-AS-EFFECT |
| C11.2 conclusion from test | Draw the correct conclusion, including the meaning of failing to reject. | M-ACCEPT-NULL, M-P-AS-EFFECT |
C12 error types and power
Type I and Type II error, their probabilities, and power. Prerequisites: hypothesis_test_logic.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C12.1 identify error type | Classify a described decision error as Type I or Type II. | M-ALPHA-BETA |
| C12.2 power relationships | Predict how power changes with sample size, effect size and alpha. | M-ALPHA-BETA, M-SAMPLE-SIZE-POP |
C13 correlation
The strength and direction of a linear association between two quantitative variables. Prerequisites: descriptive_measures.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C13.1 interpret r | Interpret the value and sign of the correlation coefficient. | M-R-NONLINEAR, M-CORR-SLOPE |
| C13.2 correlation not causation | Identify why an observational association does not establish causation. | M-CORR-CAUSE |
C14 regression
The least-squares line, the meaning of its coefficients, and its use for prediction. Prerequisites: correlation.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C14.1 slope interpretation | State the meaning of the slope in the units of the problem. | M-SLOPE-UNITS, M-CORR-SLOPE |
| C14.2 prediction and residual | Compute a predicted value and a residual. | M-EXTRAPOLATE, MISC-OTHER |
C15 descriptive measures
Centre, spread and shape of a distribution, and the resistance of each measure. Prerequisites: none.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C15.1 resistance to outliers | Predict the effect of an extreme value on mean, median, SD and IQR. | M-MEAN-OUTLIER |
| C15.2 shape and center | Relate skew to the relative position of mean and median. | M-SKEW-DIRECTION, M-MEAN-OUTLIER |
C16 study design
How data were produced, and what that permits by way of causal and generalising claims. Prerequisites: none.
| Subskill | Objective | Target misconceptions |
|---|---|---|
| C16.1 scope of inference | Determine whether a study supports a causal claim, a generalisation, both, or neither. | M-RANDOM-CONFUSE, M-CORR-CAUSE |
| C16.2 confounding | Identify a confounding variable and its effect on a conclusion. | M-CORR-CAUSE |
| Code | Description |
|---|---|
| M-INV-COND | Inverts the conditional: reports P(B|A) when P(A|B) was asked (confusion of the inverse). |
| M-BASERATE | Neglects the base rate and equates the posterior with the test's sensitivity. |
| M-JOINT-AS-COND | Uses the joint probability P(A and B) where the conditional P(A|B) was required. |
| M-WRONG-DENOM | Divides by the full sample size rather than by the size of the conditioning group. |
| M-COMPLEMENT | Forgets to take a complement, or takes 1 - p when p already answered the question. |
| M-ADD-NOT-MULT | Adds probabilities of independent events where multiplication is required (or vice versa). |
| M-INDEP-VS-EXCL | Treats mutually exclusive events as independent, or independence as exclusivity. |
| M-DOUBLE-COUNT | Applies the addition rule without subtracting the overlap. |
| M-VAR-LINEAR | Treats variance as linear: Var(aX) = a*Var(X) instead of a^2*Var(X). |
| M-SD-VS-VAR | Reports the variance where the standard deviation was asked, or vice versa. |
| M-BINOM-NO-COEF | Omits the binomial coefficient, reporting only p^k(1-p)^(n-k). |
| M-BINOM-AT-MOST | Confuses P(X = k) with P(X <= k) or P(X >= k). |
| M-Z-SIGN | Drops or flips the sign of a z-score, or reads the wrong tail of the normal table. |
| M-SIGMA-VS-SE | Uses the population SD where the standard error sigma/sqrt(n) is required. |
| M-SE-NO-SQRT | Divides by n rather than sqrt(n) when computing a standard error. |
| M-CI-PROB-PARAM | Interprets a confidence interval as a probability statement about the parameter. |
| M-CI-INDIVIDUALS | Interprets a confidence interval as containing a stated fraction of individual observations. |
| M-P-AS-H0-PROB | Reads the p-value as the probability that the null hypothesis is true. |
| M-P-AS-EFFECT | Treats a smaller p-value as evidence of a larger effect. |
| M-ACCEPT-NULL | Concludes the null hypothesis is true from a failure to reject it. |
| M-ALPHA-BETA | Swaps Type I and Type II error, or confuses beta with power. |
| M-CORR-CAUSE | Infers causation from an observational association. |
| M-CORR-SLOPE | Confuses the correlation coefficient with the regression slope. |
| M-R-NONLINEAR | Reads r near zero as 'no relationship' rather than 'no linear relationship'. |
| M-SLOPE-UNITS | States the slope without units, or as a change in x per unit y. |
| M-EXTRAPOLATE | Predicts outside the observed range of x without qualification. |
| M-MEAN-OUTLIER | Treats the mean as resistant to outliers, or the median as sensitive. |
| M-SKEW-DIRECTION | Reverses the direction of skew relative to the mean-median comparison. |
| M-RANDOM-CONFUSE | Confuses random assignment (causal scope) with random sampling (generalisation scope). |
| M-SAMPLE-SIZE-POP | Believes the required sample size depends on the population size. |
| MISC-OTHER | Distractor is not attributable to any catalogued misconception (itself a quality signal). |
blueprint
60 cells, assigned to splits per cell so no measurement target appears in two splits.
| Cell | Split | Subskill | Demand | Difficulty |
|---|---|---|---|---|
| BP001 | test | C01.1 direction of conditioning | CD3 | easy |
| BP002 | val | C01.2 conditional from two way table | CD2 | medium |
| BP003 | dev | C01.3 conditional vs joint | CD3 | hard |
| BP004 | test | C02.1 posterior from screening | CD4 | easy |
| BP005 | dev | C02.2 base rate reasoning | CD3 | medium |
| BP006 | test | C03.1 test independence from table | CD4 | hard |
| BP007 | val | C03.2 independent vs mutually exclusive | CD3 | easy |
| BP008 | dev | C04.1 addition rule with overlap | CD2 | medium |
| BP009 | test | C04.2 complement rule | CD2 | hard |
| BP010 | dev | C05.1 expectation from pmf | CD2 | easy |
| BP011 | test | C05.2 linearity of expectation | CD2 | medium |
| BP012 | val | C06.1 variance from pmf | CD2 | hard |
| BP013 | dev | C06.2 variance under linear transform | CD2 | easy |
| BP014 | test | C07.1 binomial point probability | CD2 | medium |
| BP015 | dev | C07.2 binomial cumulative | CD4 | hard |
| BP016 | test | C07.3 binomial conditions | CD3 | easy |
| BP017 | val | C08.1 zscore computation | CD2 | medium |
| BP018 | dev | C08.2 normal area | CD2 | hard |
| BP019 | test | C08.3 inverse normal | CD5 | easy |
| BP020 | dev | C09.1 standard error of mean | CD2 | medium |
| BP021 | test | C09.2 effect of sample size | CD3 | hard |
| BP022 | val | C10.1 ci computation | CD2 | easy |
| BP023 | dev | C10.2 ci interpretation | CD3 | medium |
| BP024 | test | C11.1 pvalue interpretation | CD3 | hard |
| BP025 | dev | C11.2 conclusion from test | CD3 | easy |
| BP026 | test | C12.1 identify error type | CD3 | medium |
| BP027 | val | C12.2 power relationships | CD4 | hard |
| BP028 | dev | C13.1 interpret r | CD3 | easy |
| BP029 | test | C13.2 correlation not causation | CD3 | medium |
| BP030 | dev | C14.1 slope interpretation | CD3 | hard |
| BP031 | test | C14.2 prediction and residual | CD2 | easy |
| BP032 | val | C15.1 resistance to outliers | CD3 | medium |
| BP033 | dev | C15.2 shape and center | CD3 | hard |
| BP034 | test | C16.1 scope of inference | CD4 | easy |
| BP035 | dev | C16.2 confounding | CD3 | medium |
| BP036 | test | C01.1 direction of conditioning | CD4 | hard |
| BP037 | val | C01.2 conditional from two way table | CD4 | easy |
| BP038 | dev | C01.3 conditional vs joint | CD2 | medium |
| BP039 | test | C02.1 posterior from screening | CD5 | hard |
| BP040 | dev | C02.2 base rate reasoning | CD4 | easy |
| BP041 | test | C03.1 test independence from table | CD2 | medium |
| BP042 | val | C03.2 independent vs mutually exclusive | CD1 | hard |
| BP043 | dev | C04.1 addition rule with overlap | CD4 | easy |
| BP044 | test | C04.2 complement rule | CD5 | medium |
| BP045 | dev | C05.1 expectation from pmf | CD4 | hard |
| BP046 | test | C05.2 linearity of expectation | CD5 | easy |
| BP047 | val | C06.1 variance from pmf | CD3 | medium |
| BP048 | dev | C06.2 variance under linear transform | CD5 | hard |
| BP049 | test | C07.1 binomial point probability | CD4 | easy |
| BP050 | dev | C07.2 binomial cumulative | CD3 | medium |
| BP051 | test | C07.3 binomial conditions | CD5 | hard |
| BP052 | val | C08.1 zscore computation | CD3 | easy |
| BP053 | dev | C08.2 normal area | CD4 | medium |
| BP054 | test | C08.3 inverse normal | CD4 | hard |
| BP055 | dev | C09.1 standard error of mean | CD3 | easy |
| BP056 | test | C09.2 effect of sample size | CD5 | medium |
| BP057 | val | C10.1 ci computation | CD4 | hard |
| BP058 | dev | C10.2 ci interpretation | CD1 | easy |
| BP059 | test | C11.1 pvalue interpretation | CD1 | medium |
| BP060 | dev | C11.2 conclusion from test | CD4 | hard |
protocol and pre-registration
The evaluation protocol and hypotheses were frozen on 2026-09-10, before full-scale generation. At that point the only generated items anyone had seen were the six items of a development-split slice (BP008).
Changes made after the freeze
- No-reasoning probes were truncated before stating a letter in 81 of 90 slice probes (40-token cap). The cap was raised to 300. No full-scale data existed.
- The middle-value flag was reclassified from cue to set-level information: half of all numeric items have a middle-value key by chance.
- Generation diversity was added as a descriptive metric after four arms independently produced the same textbook instance.
- The probe budget was reduced to fit the gateway's 60 requests/minute limit. No additional keys were used to raise throughput.
Reference items and contamination
Reference items come from 35 parameterised templates written by the build agent (an LLM, not one of the models under test) and not yet reviewed by a human educator; they are not a scraped item bank. Fresh numbers are drawn at build time, so no model can have memorised a specific instance; each numeric key is admitted only if an independent Monte-Carlo, SymPy or coverage check reproduces it.
Reproduction
Every model call is cached. make reproduce re-derives every number on this site offline, from the committed cache, without an API key.