---
title: "Guide: Rubric Calibration"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{Guide: Rubric Calibration}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

Rubric calibration turns writing-quality scores from pairwise comparisons into
ordered levels such as “developing”, “proficient” and “advanced”. Comparing samples
two at a time is called **comparative judgment (CJ)**. The ordinal methods use human
rubric scores to learn where the levels belong on the CJ scale. Percentile scoring
instead assigns levels according to proportions you choose.

**You need:** a completed Bayesian CJ analysis and, for ordinal calibration,
human rubric labels for some of its items. **You get:** a rubric label for each
eligible item, full category probabilities for ordinal methods, and diagnostics.
The frequentist BT/Elo fits in Getting Started are not accepted calibration inputs.

For example, you might have compared essays for organization and want to report
“developing”, “proficient”, or “advanced”. Human scores teach the ordinal model
how that trait's CJ scale relates to those labels. If you have no human scores,
percentile levels describe relative standing under your chosen distribution;
they do not establish mastery of rubric criteria.

## Choose the workflow

| Available evidence | Method and design | Interpretation |
|---|---|---|
| Completed CJ, no human rubric labels | `percentile`, `same_set` | Levels describe relative standing in this set |
| Human labels for some samples in one completed CJ analysis | `ordinal_linear`, `same_set` | The default calibration learns from those human scores |
| A human-scored reference and new samples | `ordinal_linear`, `linked_anchors` | Reuse the saved calibration after linking the sets |
| A curved relationship between CJ scores and rubric levels | `ordinal_monotone`, either ordinal design | A more flexible curve that must be checked on separate human scores |

`same_set` means that samples with and without human scores participated in
**one CJ analysis**. Calibration uses its completed scores; it does not change the
pairwise analysis. Both ordinal methods assume the effect of CJ quality is shared
across rubric boundaries. Allowing a curve does not remove that assumption.
You choose the method; the package does not choose one automatically.

## What the research shows

[Mercer (2026)](https://doi.org/10.2139/ssrn.7549240) compared direct LLM rubric
scoring with percentile scoring and human-score calibration using pairwiseLLM.
The study assessed 1,714 essays from students in Grades 8–12 on six writing traits.
Without human-scored references, percentile scoring placed roughly one fifth of
essays in each of five levels. Compared with direct scoring without examples,
it better preserved relative differences among essays, but individual scores were
farther from human scores. Calibrated CJ scores were also closer to human scores
than percentile scores were.

With human-score calibration, CJ agreement changed little between 20 and 80
reference essays. Direct scoring improved with more examples until overall
agreement was similar, although the approaches retained different strengths and
results varied by writing trait. Calibrated CJ may be more useful when relative
differences matter—for example, when examining correlations with external
measures. Direct scoring may be more useful when the aim is to place each essay
accurately on the rubric, especially when its levels describe specific skills or
standards that students should meet (criterion-referenced scores).

These findings concern the study's particular tasks and model. Reference and
target essays participated in one comparison analysis (`same_set`); the study
does not establish performance for separately linked sets or make 20 references
a general recommendation. Equal-sized percentile groups describe relative
standing; they do not establish that essays meet rubric criteria.

Mercer, S. H. (2026). *Comparative judgment versus direct rubric scoring with large
language models: Effects of human-scored reference samples* [Preprint]. SSRN.
https://doi.org/10.2139/ssrn.7549240

## Start from completed CJ results

Use a completed `fit_bayes_btl_mcmc()` result, a completed within-set
`adaptive_rank()` result (or its state), or an import-ready Phase A artifact.
Supply the completed result, not just a table of scores or an unfinished adaptive
run. If a fixed-pair analysis was fitted several times, calibration uses its last
fit. All four Bayesian variants, `btl`, `btl_e`, `btl_b`, and `btl_e_b`, use the
same calibration functions. Whether fixed comparisons were collected live or in
batches does not change calibration.

Before calibration, check that the samples are connected through comparisons and
that the CJ estimates and diagnostics are adequate for your purpose. Rubric
calibration cannot repair a weak ranking. A saved Phase A result being ready to
import does not, by itself, show that its original run finished successfully.

For constructing these inputs, see [Standalone Bayesian BTL](bayesian-btl.html),
[Guide: Adaptive Pairing](adaptive-pairing.html), and
[Guide: Adaptive Linking](adaptive-linking.html). Rubric functions do not generate
pairs, collect judgments, or control adaptive stopping.

The code below starts from your completed objects and label tables. Those objects
are not bundled example data, so dependent chunks are not evaluated during normal
documentation builds. No provider calls or CmdStan sampling occur in the rubric
examples. The downstream snippets are tested with deterministic completed-result
fixtures in the package's test suite.

pairwiseLLM requires **R >= 4.4**. Percentile scoring needs no optional modeling
backend. Linear fitting requires `ordinal`; saved linear prediction does not.
Monotone fitting requires `mgcv >= 1.9-4` and `withr`; saved monotone prediction
requires `mgcv`. Install the optional packages only when you use those methods:

```{r install, eval=FALSE}
install.packages("ordinal")
install.packages(c("mgcv", "withr"))
```

## Define the ordered rubric and labels

```{r rubric-levels}
rubric_levels <- c("developing", "proficient", "advanced")
```

`completed_cj` below is the completed organization-specific CJ result.
`training_labels` and `evaluation_labels` are disjoint data frames with columns
`item_id` and `rubric_score`, optionally `trait`. Each ID must occur in that CJ
result. Use global IDs when the adaptive result or artifact provides them;
otherwise use the original fixed-fit IDs. Alignment is by ID, never row position.
Omitted items or `NA` rubric scores are unlabeled.

A label table has this shape (illustrative IDs only; use IDs from your completed
analysis and enough labels to support fitting and validation):

| item_id | rubric_score |
| --- | --- |
| essay_01 | developing |
| essay_07 | proficient |
| essay_12 | advanced |

```{r read-labels, eval=FALSE}
training_labels <- read.csv("training_labels.csv", colClasses = "character")
evaluation_labels <- read.csv("evaluation_labels.csv", colClasses = "character")
```

List levels from lowest to highest quality. For text labels, supply `levels`
explicitly. Numeric scores are ordered from low to high; ordered factors keep
their declared order. Use at least three levels, commonly three to six. Every
intended level must appear in the training labels. The package stops if a level
is missing rather than silently combining levels. Higher CJ and rubric scores
must both mean better performance.

Choose calibration responses that cover the CJ range, including its extremes,
and check category counts. There is no universal sufficient number of labels.
Keep evaluation labels separate before fitting: evaluation on training labels
measures apparent performance, not held-out accuracy.

## Same-set linear calibration

```{r same-set, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  calibration <- fit_rubric_calibration(
    completed_cj, training_labels,
    trait = "organization", levels = rubric_levels
  )
  scores <- predict(calibration)
  scores[, c("item_id", "rubric_score", "extrapolated")]
  calibration$category_counts
  calibration$calibration_range
  calibration$transformation
}
```

The model learns from samples with training labels and saves their score scale
for future predictions. Higher CJ scores should favor higher rubric levels.
A warning about a zero or negative slope means this relationship needs review;
the package does not reverse your scoring direction automatically. Also inspect
warnings about sparse categories or an unstable fit. The formal model and
scaling rules are documented in `?fit_rubric_calibration`.

`predict(calibration)` scores every original sample, including those without human
labels. If you supply `newdata` for a same-set calibration, it must be the original
completed result; changing row order is fine, but changing the samples or their
fit is not. Use the linked-reference workflow below for a separate set.

## Read probabilities and hard scores

```{r probabilities, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  probability_matrix <- do.call(rbind, scores$probabilities)
  head(probability_matrix)
  scores[, c("median_category", "modal_category", "expected_level")]
  modal_scores <- predict(calibration, hard_score = "mode")
}
```

Each `probabilities` entry gives the probability of every rubric level, named by
its label. The default single score is the **median category**: starting from the
lowest level, add probabilities until the total reaches 0.5. If it equals 0.5
exactly, use that lower category. With `hard_score = "mode"`, use the most probable
category instead; ties go to the lower category. Both summaries and all
probabilities remain available whichever score you report.

`expected_level` is an average of category positions (1, 2, 3, and so on), not an
average of your numeric labels. It is a summary rather than an assigned rubric
level. `extrapolated` flags scores outside the range covered by training labels,
even for samples in the original analysis.

For an illustrative probability vector `(developing = 0.20, proficient = 0.55,
advanced = 0.25)`, cumulative probability first reaches 0.5 at “proficient”, so
that is the default median-category score. The probabilities also show substantial
uncertainty; reporting the category alone discards it. These numbers illustrate
the decision rule and are not fitted results.

For a report, retain `item_id`, `rubric_score`, `probabilities`, and `extrapolated`
for ordinal predictions. Explain an extrapolation flag as a score outside the
labeled calibration range. Percentile predictions instead require requested and
achieved category proportions. Keep the method, trait, and validation results
with either kind of output.

## Evaluate held-out labels and inspect diagnostics

```{r evaluation, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  assessment <- evaluate_rubric_predictions(
    calibration, evaluation_labels, diagnostics = TRUE
  )
  assessment$metrics
  assessment$calibration
  assessment$metadata$n_training_label_overlap
  assessment$diagnostics$common_effect
}
```

Start with ranked probability score (`rps`): lower values indicate better
predictions, accounting for how far a prediction falls from the observed level.
Log loss is another probability-based check. For the assigned categories, inspect
exact agreement, agreement within one level, average category error and weighted
kappa (agreement adjusted for chance). Distances count category steps. If a metric
cannot be calculated, the output explains why.

Evaluation matches samples by ID, counts missing labels and reports any overlap
with training labels. Unlike training, an evaluation set need not contain every
category. `assessment$calibration` groups similar predictions and compares them
with observed human scores, helping you see where the model is too optimistic or
too pessimistic.

Diagnostics are off by default. For the linear method, they check the assumption
that CJ quality has a shared effect across rubric boundaries. For the curved
method, they provide descriptive boundary checks. These checks use training data;
the separate evaluation labels do not change your fitted calibration. If a
check fails to run, it is marked unavailable and the original fit is preserved.

For repeated validation, keep the completed CJ analysis fixed. Repeatedly set
aside some human labels, fit a fresh calibration using only the remaining labels,
and evaluate the labels you set aside. Every training group must still include
all rubric levels. If you use those results to choose a model or its settings,
reserve another independent evaluation set for your final report.

## Explicit monotone alternative

```{r monotone, eval=FALSE}
if (requireNamespace("mgcv", quietly = TRUE) &&
    packageVersion("mgcv") >= "1.9.4" && requireNamespace("withr", quietly = TRUE)) {
  monotone_calibration <- fit_rubric_calibration(
    completed_cj, training_labels, method = "ordinal_monotone",
    trait = "organization", levels = rubric_levels, k = 6
  )
  monotone_scores <- predict(monotone_calibration)
  monotone_assessment <- evaluate_rubric_predictions(
    monotone_calibration, evaluation_labels
  )
  monotone_assessment$metrics
}
```

This fits a smooth curve that can bend while never decreasing as CJ quality
increases. `k` controls how much flexibility is available; its default is 6 and
it must be at least 5. The package may reduce it with a warning when too few
distinct training scores are available. At least five distinct scores are needed
to fit this curve, but that minimum does not guarantee enough data for useful
calibration. See the function help for smoothing controls.

Check fit diagnostics and performance on separate human labels before preferring
the more flexible method. A flat part of the curve may make a category cutpoint
unavailable; predictions still use category probabilities. Scores outside the
training range are flagged because the curve is being extended beyond the scores
used to learn it.

## Percentile levels without human labels

```{r percentile, eval=FALSE}
percentile_calibration <- fit_rubric_calibration(
  completed_cj, method = "percentile", trait = "organization",
  levels = rubric_levels, target_distribution = c(0.2, 0.5, 0.3)
)
percentile_scores <- predict(percentile_calibration)
percentile_calibration$backend$requested_proportions
percentile_calibration$backend$achieved_proportions
percentile_calibration$backend$cutpoints
percentile_calibration$backend$cutpoint_tie_counts
```

The example asks for 20% of samples in the lowest level, 50% in the middle and
30% in the highest. Omit `target_distribution` for equal proportions. You can
supply the number of levels with `K` instead of naming them.

Tied scores stay together. This means the achieved proportions may differ from
what you requested, and some categories may be empty. A score exactly on a
boundary enters the higher category; if all scores are identical, this rule puts
every sample in the highest category. Always inspect the achieved proportions.

These levels describe **relative standing within this set**. Matching a historical
rubric distribution does not establish agreement with human raters for individual
samples. Percentile scoring reuses only the original analysis; it cannot transport
levels to a separate cohort through `linked_anchors`. It returns categories without
category probabilities, so evaluation uses only category-based metrics.

See [What the research shows](#what-the-research-shows) for a study comparing
percentile scoring, human-score calibration, and direct LLM rubric scoring.

## Reuse a historical rubric reference for a new set

You can use a human-scored reference set to score new writing samples after linking
both sets onto a common scale. The reference is the **hub** and the new set is the
**spoke**. **Phase A** means ranking each set separately; **Phase B** links them
using comparisons between sets. A hub sample is a reference for comparison, not a
sample whose writing quality is known without uncertainty.

1. Rank the reference samples on the chosen trait and keep the original Phase A result.
2. Fit and save the rubric calibration using that result and its human scores.
3. Rank the new samples using compatible settings and keep their Phase A result.
4. Choose E1, E2 or E3 and link the sets using between-set comparisons.
5. Check the link, then apply the saved calibration to the linked target scores.

The following example assumes `reference_phase_a` contains the original reference
artifact (the saved Phase A result) and `reference_labels` contains its human
scores. Read a saved artifact with `readRDS()` first; supply the artifact itself,
not a filename or the entire collection of saved-run metadata.

```{r linked-calibration, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  linked_calibration <- fit_rubric_calibration(
    reference_phase_a, reference_labels,
    calibration_design = "linked_anchors",
    trait = "organization", levels = rubric_levels
  )
  reference_scores <- predict(linked_calibration)
}
```

### Prepare a reference from a standalone ranking

An import-ready adaptive Phase A result is one supported reference source. You
can also use a completed standalone Bayesian ranking, including one fitted from
pooled historical comparisons of reference samples. Freeze this reference before
evaluating the new samples.

Keep the exact comparison table supplied to `fit_bayes_btl_mcmc()`. If you fitted
several subsets, keep the rows used for the final fit. The constructor checks
these rows against the record saved when the fit was made. Older standalone
results without that record must be refitted from their original comparisons;
they remain usable for ordinary same-set calibration.

```{r standalone-reference, eval=FALSE}
reference <- prepare_linked_rubric_reference(
  completed_reference, reference_comparisons,
  set_id = "reference", items = reference_items, trait = "organization"
)
standalone_calibration <- fit_rubric_calibration(
  reference, reference_labels, calibration_design = "linked_anchors",
  levels = rubric_levels
)
saveRDS(reference, "rubric-reference.rds")
saveRDS(standalone_calibration, "rubric-calibration.rds")
```

`reference_items` maps the fit's `item_id` values to unique `global_item_id`
values. Use those global IDs in `reference_labels`. Omit `items` when the
original IDs already uniquely identify your samples across sets.

The [linking sessions guide](linking-sessions.html#link-a-prepared-standalone-reference)
shows which saved reference inputs to use with E1, E2, and E3. All three retain
the same reference identifier. The saved reference and calibration can be read
with `readRDS()` later; the original sampler does not need to stay open.

### Link the reference and new samples

Use the same BTL model variant for the reference and target rankings. The
[linking sessions guide](linking-sessions.html) explains the required input tables
and how to save and resume work. Here we explicitly choose E2, which needs saved
joint posterior draws in both artifacts. `active_cross` holds the comparisons to
use for fitting; `frozen_judge` holds the shared bias and lapse settings from Phase A.
Keep comparisons reserved for checking accuracy in a separate table.

```{r phase-b-workflow, eval=FALSE}
link_input <- prepare_link_input(
  estimator = "gaussian_posterior_bridge",
  hub = list(set_id = "1",
    items = reference_phase_a$items[c("item_id", "global_item_id")]),
  spoke = list(set_id = "2",
    items = target_phase_a$items[c("item_id", "global_item_id")]),
  phase_a = list(hub = list(artifact = reference_phase_a),
    spoke = list(artifact = target_phase_a)),
  cross = active_cross, judge = frozen_judge
)
completed_phase_b <- start_link_session(link_input)
```

For several target sets, prepare one linking input per set and pass the list to
`start_link_session()`. All use the same hub, judge settings and chosen method.
E2 and E3 can update hub scores during linking, but the rubric calibration stays
attached to the original reference result. You do not need target rubric labels
to link the sets or apply the saved calibration.

```{r linked-prediction, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  target_scores <- predict(linked_calibration, newdata = completed_phase_b)
  target_scores[, c("global_item_id", "set_id", "rubric_score", "theta_sd", "extrapolated")]
  attr(target_scores, "linking")$diagnostics
}
```

The prediction contains only the target samples. Use `newdata = NULL` to predict
scores for the original reference samples. Every target set needs a valid link
supported by between-set comparisons; a fit based only on starting assumptions
is insufficient.

Linking scores are named `theta_link_mean` (also available as `theta_link_eap`).
E1 and E3-MCMC report posterior averages; E2 and the usual E3 fit report the most
probable joint scores with approximate uncertainty. E1 treats the earlier rankings
as fixed. E2 and E3 include uncertainty in those rankings as well as the shift
between sets. The [design guide](adaptive-linking-design.html) explains these
differences. Rubric prediction uses the linked point scores; it retains score
uncertainty for inspection but does not propagate it through the rubric model.

Prediction checks that you are using the original reference samples, trait,
scoring direction and compatible model settings. It restores the original
reference's score origin before applying the saved calibration. **Do not
standardize the target set separately**: doing so would change the meaning of its
scores relative to the reference. The output includes information about the
linking method, comparisons used and fit diagnostics.

Old anchored-joint Phase B sessions cannot be used with this workflow. Restart
linking from compatible original Phase A results using E1, E2 or E3.

Reference holdout performance assesses calibration on the reference scale. To
assess transport, use external human labels for linked targets:

```{r linked-evaluation, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  target_assessment <- evaluate_rubric_predictions(
    linked_calibration, target_labels, newdata = completed_phase_b
  )
  target_assessment$metrics
  target_assessment$metadata$linking
}
```

Phase B diagnostics alone do not demonstrate rubric accuracy, and reference
validation alone does not measure the additional error from linking.

## Separate analytic traits

Organization and evidence require separate trait-specific comparisons, CJ fits,
labels, and calibrations even when item IDs overlap. Each may have different
levels or a different Bayesian variant. Compatibility must hold within each
trait's reference/target linking workflow. One holistic CJ scale cannot supply
multiple analytic trait calibrations.

```{r analytic-traits, eval=FALSE}
if (requireNamespace("ordinal", quietly = TRUE)) {
  evidence_calibration <- fit_rubric_calibration(
    evidence_cj, evidence_training_labels, trait = "evidence",
    levels = c("limited", "adequate", "strong")
  )
  evidence_scores <- predict(evidence_calibration)
}
```

## Uncertainty and reporting

All three methods condition on accepted **CJ point locations**. Retained CJ SDs,
posterior draws, or linking uncertainty are not propagated into rubric category
probabilities. Ordinal parameter uncertainty is conditional on those estimated
predictors; monotone threshold uncertainty is not supplied by the wrapper.
There is no posterior/bootstrap propagation engine or joint likelihood here.

Report the trait, BTL variant, method/design, ordered levels, labeled category
counts and range, held-out metrics, extrapolation, and CJ/calibration/linking
diagnostics. Distinguish numerical feasibility from adequate measurement and
external rubric validity. Small deterministic examples and package recovery
tests do not establish a universal calibration sample size or validity for a new
population.

## Citation

> Mercer, S. H. (2026). *Guide: Rubric Calibration* [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM