Calibrating generated items with predicted priors

Automatic item generation produces items faster than pretesting can calibrate them. coldstart predicts each new item’s difficulty from its features, uses the prediction as a robust prior, and finds template families where the prediction cannot be trusted.

A bank with features

Legacy items have calibrated difficulties; new generated items have only features (any numeric representation works, such as embeddings from any text model or cognitive-attribute codes) and a template family. In the simulation, one family’s template drifted, making its new items harder than its history suggests.

library(coldstart)
sim <- cs_simulate(n_train = 400, n_new = 150, seed = 3)
it <- sim$items
tr <- it$set == "train"
table(it$set)
#> 
#>   new train 
#>   150   400

Predict difficulty, with honest uncertainty

pr <- cs_predictor(it$b_legacy[tr], sim$features[tr, ], it$family[tr], seed = 1)
pr
#> <cs_predictor> ridge, lambda = 1 | 400 legacy items | 10 families
#> predictive SD: 0.548 (seen family), 0.788 (unseen family)
#>   shared within family: 0.000 (seen), 0.559 (unseen)
pred <- predict(pr, sim$features[!tr, ], it$family[!tr])
head(pred)
#>        item family       mean        sd sd_shared family_seen
#> G0401 G0401    F07  1.3051977 0.5480338 0.0000000        TRUE
#> G0402 G0402    F08 -1.8811880 0.5480338 0.0000000        TRUE
#> G0403 G0403    F11 -0.4888554 0.7876555 0.5589692       FALSE
#> G0404 G0404    F09 -0.5278875 0.5480338 0.0000000        TRUE
#> G0405 G0405    F06 -0.1683571 0.5480338 0.0000000        TRUE
#> G0406 G0406    F02 -0.4489074 0.5480338 0.0000000        TRUE

The predictive SD is estimated out of sample. It is larger for a family never seen in training, because the family’s own effect is then unknown.

How many pretest responses do we need?

plan <- cs_plan(pred, target_sd = 0.3)
summary(plan[c("n_with_prior", "n_without_prior")])
#>   n_with_prior   n_without_prior 
#>  Min.   :38.00   Min.   : 54.00  
#>  1st Qu.:39.00   1st Qu.: 55.00  
#>  Median :41.00   Median : 58.00  
#>  Mean   :46.24   Mean   : 64.15  
#>  3rd Qu.:48.75   3rd Qu.: 64.75  
#>  Max.   :94.00   Max.   :134.00

Calibrate from a small pretest

resp <- cs_responses(sim, n_per_item = 25, seed = 4)
cal <- cs_calibrate(resp, pred)
truth <- it$b_true[match(cal$item, it$item)]
c(baseline = sqrt(mean((cal$base_mean - truth)^2)),
  with_prior = sqrt(mean((cal$post_mean - truth)^2)))
#>   baseline with_prior 
#>  0.5752105  0.3839569

Which families can we trust?

A prediction is only as good as the family template it comes from. If a template has drifted (the items are now harder or easier than the features say), every item of that family is mispredicted in the same direction. cs_check() compares the pretest estimates with the predictions, family by family, and flags families whose items conflict with their priors:

fam <- setNames(it$family[!tr], it$item[!tr])
chk <- cs_check(cs_calibrate(cs_responses(sim, 100, seed = 5), pred), fam)
chk
#>    family n_items      mean_z     rms_z  coverage      p_value trustworthy
#> 1     F01      13  1.91170195 2.2174671 0.3076923 1.033704e-08       FALSE
#> 9     F09      18  0.26358231 1.1677767 0.7777778 1.379219e-01        TRUE
#> 7     F07      10 -0.08620829 1.2031628 0.9000000 1.523653e-01        TRUE
#> 6     F06      19 -0.03607136 1.0696458 0.8947368 2.974532e-01        TRUE
#> 4     F04       7  0.23966538 1.0288285 0.8571429 3.875306e-01        TRUE
#> 3     F03      12  0.19518559 1.0150236 0.8333333 4.169631e-01        TRUE
#> 8     F08      13  0.23874499 1.0076796 0.8461538 4.324513e-01        TRUE
#> 11    F11      19 -0.31884214 0.7773376 1.0000000 5.212788e-01        TRUE
#> 2     F02       9 -0.10144038 0.8624153 0.8888889 6.689603e-01        TRUE
#> 5     F05      15  0.32898050 0.8516693 0.9333333 7.610460e-01        TRUE
#> 10    F10      15 -0.08314448 0.7429080 1.0000000 9.121243e-01        TRUE
unique(it$family[it$rogue])
#> [1] "F01"

The drifted family is flagged. The family that never appeared in training is not, even though its predictions are the least certain.

Priors for families that fail the check are withdrawn before the final calibration:

resp100 <- cs_responses(sim, 100, seed = 5)
final <- cs_calibrate(resp100, cs_distrust(pred, chk))
head(final[c("item", "n", "post_mean", "post_sd")])
#>    item   n  post_mean   post_sd
#> 1 G0401 100  0.7082320 0.2175301
#> 2 G0402 100 -1.3150924 0.2362350
#> 3 G0403 100 -1.1280780 0.2314078
#> 4 G0404 100  0.6417285 0.2345004
#> 5 G0405 100 -0.7933419 0.2231148
#> 6 G0406 100 -0.6901677 0.2011189

Items of a family share prediction error

Items of the same family do not err independently: they share the error in the family’s estimated effect. For a family unseen in training, the whole family effect is unknown, so all of its items are off by the same unknown amount. predict() reports this shared part as sd_shared, alongside each item’s total predictive SD:

unique(pred[c("family_seen", "sd", "sd_shared")])
#>       family_seen        sd sd_shared
#> G0401        TRUE 0.5480338 0.0000000
#> G0403       FALSE 0.7876555 0.5589692

Since version 0.2.0, cs_check() tests each family’s conflicts under this correlation. Version 0.1.0 treated the items as independent, so a new family whose effect happened to sit away from the average could fail the check merely for being new, even when its predictions were as honest as they claimed. The difference is easy to see on a bank where that happens. Setting prior_sd_shared to zero reproduces the old, independent test:

sim2 <- cs_simulate(n_train = 400, n_new = 150, seed = 110)
it2 <- sim2$items; tr2 <- it2$set == "train"
pr2 <- cs_predictor(it2$b_legacy[tr2], sim2$features[tr2, ], it2$family[tr2], seed = 10)
pred2 <- predict(pr2, sim2$features[!tr2, ], it2$family[!tr2])
cal2 <- cs_calibrate(cs_responses(sim2, 100, seed = 10), pred2)
fam2 <- setNames(it2$family[!tr2], it2$item[!tr2])
c(unseen = unique(it2$family[it2$unseen_family]), drifted = unique(it2$family[it2$rogue]))
#>  unseen drifted 
#>   "F11"   "F01"

correlated <- cs_check(cal2, fam2)
independent <- cs_check(transform(cal2, prior_sd_shared = 0), fam2)
correlated[!correlated$trustworthy, c("family", "n_items", "mean_z", "p_value")]
#>   family n_items   mean_z      p_value
#> 1    F01      18 1.550548 2.661835e-07
independent[!independent$trustworthy, c("family", "n_items", "mean_z", "p_value")]
#>    family n_items    mean_z      p_value
#> 1     F01      18  1.550548 2.661835e-07
#> 11    F11      18 -1.309219 3.501922e-03

Both tests flag the drifted family. Only the independent test also flags the new family, whose items sit on average 1.3 SD below their predictions: the kind of common shift that sd_shared says to expect for a new family. In the package’s simulations, this correction lowered the rate at which the unseen family was falsely flagged from 3–7% to about 1% (at alpha = 0.01), while the drifted family was still detected. See ?cs_check for the test.