---
title: "Calibrating generated items with predicted priors"
output: markdown::html_format
vignette: >
  %\VignetteIndexEntry{Calibrating generated items with predicted priors}
  %\VignetteEngine{knitr::knitr}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

Automatic item generation produces items faster than pretesting can calibrate
them. coldstart predicts each new item's difficulty from its features, uses
the prediction as a robust prior, and finds template families where the
prediction cannot be trusted.

## A bank with features

Legacy items have calibrated difficulties; new generated items have only
features (any numeric representation works, such as embeddings from any text
model or cognitive-attribute codes) and a template family. In the simulation,
one family's template drifted, making its new items harder than its history
suggests.

```{r}
library(coldstart)
sim <- cs_simulate(n_train = 400, n_new = 150, seed = 3)
it <- sim$items
tr <- it$set == "train"
table(it$set)
```

## Predict difficulty, with honest uncertainty

```{r}
pr <- cs_predictor(it$b_legacy[tr], sim$features[tr, ], it$family[tr], seed = 1)
pr
pred <- predict(pr, sim$features[!tr, ], it$family[!tr])
head(pred)
```

The predictive SD is estimated out of sample. It is larger for a family never
seen in training, because the family's own effect is then unknown.

## How many pretest responses do we need?

```{r}
plan <- cs_plan(pred, target_sd = 0.3)
summary(plan[c("n_with_prior", "n_without_prior")])
```

## Calibrate from a small pretest

```{r}
resp <- cs_responses(sim, n_per_item = 25, seed = 4)
cal <- cs_calibrate(resp, pred)
truth <- it$b_true[match(cal$item, it$item)]
c(baseline = sqrt(mean((cal$base_mean - truth)^2)),
  with_prior = sqrt(mean((cal$post_mean - truth)^2)))
```

## Which families can we trust?

A prediction is only as good as the family template it comes from. If a
template has drifted (the items are now harder or easier than the features
say), every item of that family is mispredicted in the same direction.
`cs_check()` compares the pretest estimates with the predictions, family by
family, and flags families whose items conflict with their priors:

```{r}
fam <- setNames(it$family[!tr], it$item[!tr])
chk <- cs_check(cs_calibrate(cs_responses(sim, 100, seed = 5), pred), fam)
chk
unique(it$family[it$rogue])
```

The drifted family is flagged. The family that never appeared in training
is not, even though its predictions are the least certain.

Priors for families that fail the check are withdrawn before the final
calibration:

```{r}
resp100 <- cs_responses(sim, 100, seed = 5)
final <- cs_calibrate(resp100, cs_distrust(pred, chk))
head(final[c("item", "n", "post_mean", "post_sd")])
```

### Items of a family share prediction error

Items of the same family do not err independently: they share the error in
the family's estimated effect. For a family unseen in training, the whole
family effect is unknown, so all of its items are off by the same unknown
amount. `predict()` reports this shared part as `sd_shared`, alongside each
item's total predictive SD:

```{r}
unique(pred[c("family_seen", "sd", "sd_shared")])
```

Since version 0.2.0, `cs_check()` tests each family's conflicts under this
correlation. Version 0.1.0 treated the items as independent, so a new family
whose effect happened to sit away from the average could fail the check
merely for being new, even when its predictions were as honest as they
claimed. The difference is easy to see on a bank where that happens. Setting
`prior_sd_shared` to zero reproduces the old, independent test:

```{r}
sim2 <- cs_simulate(n_train = 400, n_new = 150, seed = 110)
it2 <- sim2$items; tr2 <- it2$set == "train"
pr2 <- cs_predictor(it2$b_legacy[tr2], sim2$features[tr2, ], it2$family[tr2], seed = 10)
pred2 <- predict(pr2, sim2$features[!tr2, ], it2$family[!tr2])
cal2 <- cs_calibrate(cs_responses(sim2, 100, seed = 10), pred2)
fam2 <- setNames(it2$family[!tr2], it2$item[!tr2])
c(unseen = unique(it2$family[it2$unseen_family]), drifted = unique(it2$family[it2$rogue]))

correlated <- cs_check(cal2, fam2)
independent <- cs_check(transform(cal2, prior_sd_shared = 0), fam2)
correlated[!correlated$trustworthy, c("family", "n_items", "mean_z", "p_value")]
independent[!independent$trustworthy, c("family", "n_items", "mean_z", "p_value")]
```

Both tests flag the drifted family. Only the independent test also flags
the new family, whose items sit on average 1.3 SD below their predictions:
the kind of common shift that `sd_shared` says to expect for a new family.
In the package's simulations, this correction lowered the rate at which the unseen family was
falsely flagged from 3–7% to about 1% (at `alpha = 0.01`), while the
drifted family was still detected. See `?cs_check` for the test.