---
title: "Would this candidate have passed with different raters?"
output: markdown::html_format
vignette: >
  %\VignetteIndexEntry{Would this candidate have passed with different raters?}
  %\VignetteEngine{knitr::knitr}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

Boards that run oral exams, OSCEs or essay-scored certifications must defend
individual pass/fail decisions. Rater severity is well studied; what it *did*
to decisions usually is not. decisionfacets answers that question directly.

## A simulated administration

Every candidate is scored on four tasks by two raters drawn from a pool of
twelve whose severities differ. Because the data are simulated, the true
abilities and severities are known.

```{r}
library(decisionfacets)
sim <- df_simulate(n_persons = 400, n_items = 4, n_raters = 12,
                   raters_per_person = 2, severity_sd = 0.6, seed = 2026)
head(sim$data)
round(sim$par$lambda, 2)   # true rater severities (positive = harsher)
```

## Fit the many-facet Rasch model

`df_fit()` uses 'TAM' when it is installed and a built-in joint maximum
likelihood estimator otherwise. We use the built-in one here so the vignette
has no dependencies.

```{r}
fit <- df_fit(sim$data, engine = "jmle")
df_rater_effects(fit)
```

## The decision rule is the point

The same substantive standard (an average rating of 2 per cell) can be applied
to raw totals or to severity-adjusted measures. Rater severity reaches the
decision only in the first case.

```{r}
raw <- df_cut(2 * 4 * 2, "raw_total")
fa  <- df_cut(2, "fair_average")
```

## Counterfactual pass probabilities

For each candidate, `df_counterfactual()` gives the probability of passing a
re-rating by the observed panel, by an average-severity panel and by a random
panel from the pool. `advantage` is how far the assigned panel pushed the
candidate toward the outcome they received.

```{r}
cf_raw <- df_counterfactual(fit, raw)
cf_raw
summary(cf_raw)
summary(df_counterfactual(fit, fa))
```

Under raw totals, a sizable share of candidates are rater-dependent; under the
fair average, essentially none are.

## Where do wrong decisions come from?

```{r}
df_attribute(cf_raw)
```

The decomposition separates error that no rater could remove (measurement)
from error added by the particular raters assigned, and from the expected cost
of random assignment.

## Checking against the truth

With the true parameters the same analysis gives the known answer, so the
estimates can be compared with it:

```{r}
truth <- df_counterfactual(sim, raw)
cor(truth$advantage, cf_raw$advantage)
table(truth = truth$rater_dependent, estimated = cf_raw$rater_dependent)
```

## Oral examinations: one examiner per case, scores 0-100

Many oral certification exams do not use panels. Each candidate works
through a series of cases, and each case is scored by a different examiner on
a 0-100 scale. decisionfacets handles this design and these scores directly.

`oral_exam` is a synthetic example: 300 candidates each take 12 cases, and
each case is scored by one of 8 examiners qualified for it (48 examiners in
all), never the same examiner twice for one candidate.

```{r}
head(oral_exam)
d <- df_data(oral_exam, person = "candidate", item = "case",
             rater = "examiner", scale = "continuous")
```

Continuous scores are fitted with a linear many-facet model,
`score = ability - case difficulty - examiner severity + error`. Measures,
difficulties and severities are all in score points, so a severity of 3 means
that examiner scores about 3 points lower than an average examiner would.

```{r}
ofit <- df_fit(d)
round(ofit$par$sigma, 1)                # residual SD (score points)
head(df_rater_effects(ofit))
range(round(ofit$par$lambda, 1))        # most lenient and harshest examiners
```

The natural standard is a mean score, for example 70. The counterfactual
reassigns examiners case by case: each draw gives every case an examiner
qualified for it.

```{r}
cut70 <- df_cut(70, "raw_mean")
cf_oral <- df_counterfactual(ofit, cut70, max_panels = 500, seed = 1)
summary(cf_oral)
df_attribute(cf_oral)
```

With twelve different examiners per candidate, severity largely averages
out, so examiner assignment usually accounts for a smaller share of wrong
decisions than with two-rater panels. Scoring with severity-adjusted
measures removes it altogether:

```{r}
summary(df_counterfactual(ofit, df_cut(70, "measure"), max_panels = 500, seed = 1))
```

`df_fit()` also checks that the rating network is connected. If some
examiners never share candidates or cases with the rest, their severities
cannot be compared, and `df_fit()` stops with an explanation instead of
returning misleading estimates.