Guide: Rubric Calibration

Choose the workflow

same_set means that samples with and without human scores participated in one CJ analysis. Calibration uses its completed scores; it does not change the pairwise analysis. Both ordinal methods assume the effect of CJ quality is shared across rubric boundaries. Allowing a curve does not remove that assumption. You choose the method; the package does not choose one automatically.

What the research shows

Mercer (2026) compared direct LLM rubric scoring with percentile scoring and human-score calibration using pairwiseLLM. The study assessed 1,714 essays from students in Grades 8–12 on six writing traits. Without human-scored references, percentile scoring placed roughly one fifth of essays in each of five levels. Compared with direct scoring without examples, it better preserved relative differences among essays, but individual scores were farther from human scores. Calibrated CJ scores were also closer to human scores than percentile scores were.

With human-score calibration, CJ agreement changed little between 20 and 80 reference essays. Direct scoring improved with more examples until overall agreement was similar, although the approaches retained different strengths and results varied by writing trait. Calibrated CJ may be more useful when relative differences matter—for example, when examining correlations with external measures. Direct scoring may be more useful when the aim is to place each essay accurately on the rubric, especially when its levels describe specific skills or standards that students should meet (criterion-referenced scores).

These findings concern the study’s particular tasks and model. Reference and target essays participated in one comparison analysis (same_set); the study does not establish performance for separately linked sets or make 20 references a general recommendation. Equal-sized percentile groups describe relative standing; they do not establish that essays meet rubric criteria.

Mercer, S. H. (2026). Comparative judgment versus direct rubric scoring with large language models: Effects of human-scored reference samples [Preprint]. SSRN. https://doi.org/10.2139/ssrn.7549240

Start from completed CJ results

Use a completed fit_bayes_btl_mcmc() result, a completed within-set adaptive_rank() result (or its state), or an import-ready Phase A artifact. Supply the completed result, not just a table of scores or an unfinished adaptive run. If a fixed-pair analysis was fitted several times, calibration uses its last fit. All four Bayesian variants, btl, btl_e, btl_b, and btl_e_b, use the same calibration functions. Whether fixed comparisons were collected live or in batches does not change calibration.

Before calibration, check that the samples are connected through comparisons and that the CJ estimates and diagnostics are adequate for your purpose. Rubric calibration cannot repair a weak ranking. A saved Phase A result being ready to import does not, by itself, show that its original run finished successfully.

For constructing these inputs, see Standalone Bayesian BTL, Guide: Adaptive Pairing, and Guide: Adaptive Linking. Rubric functions do not generate pairs, collect judgments, or control adaptive stopping.

The code below starts from your completed objects and label tables. Those objects are not bundled example data, so dependent chunks are not evaluated during normal documentation builds. No provider calls or CmdStan sampling occur in the rubric examples. The downstream snippets are tested with deterministic completed-result fixtures in the package’s test suite.

pairwiseLLM requires R >= 4.4. Percentile scoring needs no optional modeling backend. Linear fitting requires ordinal; saved linear prediction does not. Monotone fitting requires mgcv >= 1.9-4 and withr; saved monotone prediction requires mgcv. Install the optional packages only when you use those methods:

install.packages("ordinal")
install.packages(c("mgcv", "withr"))

Define the ordered rubric and labels

rubric_levels <- c("developing", "proficient", "advanced")

completed_cj below is the completed organization-specific CJ result. training_labels and evaluation_labels are disjoint data frames with columns item_id and rubric_score, optionally trait. Each ID must occur in that CJ result. Use global IDs when the adaptive result or artifact provides them; otherwise use the original fixed-fit IDs. Alignment is by ID, never row position. Omitted items or NA rubric scores are unlabeled.

A label table has this shape (illustrative IDs only; use IDs from your completed analysis and enough labels to support fitting and validation):

training_labels <- read.csv("training_labels.csv", colClasses = "character")
evaluation_labels <- read.csv("evaluation_labels.csv", colClasses = "character")

List levels from lowest to highest quality. For text labels, supply levels explicitly. Numeric scores are ordered from low to high; ordered factors keep their declared order. Use at least three levels, commonly three to six. Every intended level must appear in the training labels. The package stops if a level is missing rather than silently combining levels. Higher CJ and rubric scores must both mean better performance.

Choose calibration responses that cover the CJ range, including its extremes, and check category counts. There is no universal sufficient number of labels. Keep evaluation labels separate before fitting: evaluation on training labels measures apparent performance, not held-out accuracy.

Same-set linear calibration

if (requireNamespace("ordinal", quietly = TRUE)) {
  calibration <- fit_rubric_calibration(
    completed_cj, training_labels,
    trait = "organization", levels = rubric_levels
  )
  scores <- predict(calibration)
  scores[, c("item_id", "rubric_score", "extrapolated")]
  calibration$category_counts
  calibration$calibration_range
  calibration$transformation
}

The model learns from samples with training labels and saves their score scale for future predictions. Higher CJ scores should favor higher rubric levels. A warning about a zero or negative slope means this relationship needs review; the package does not reverse your scoring direction automatically. Also inspect warnings about sparse categories or an unstable fit. The formal model and scaling rules are documented in ?fit_rubric_calibration.

predict(calibration) scores every original sample, including those without human labels. If you supply newdata for a same-set calibration, it must be the original completed result; changing row order is fine, but changing the samples or their fit is not. Use the linked-reference workflow below for a separate set.

Read probabilities and hard scores

if (requireNamespace("ordinal", quietly = TRUE)) {
  probability_matrix <- do.call(rbind, scores$probabilities)
  head(probability_matrix)
  scores[, c("median_category", "modal_category", "expected_level")]
  modal_scores <- predict(calibration, hard_score = "mode")
}

Each probabilities entry gives the probability of every rubric level, named by its label. The default single score is the median category: starting from the lowest level, add probabilities until the total reaches 0.5. If it equals 0.5 exactly, use that lower category. With hard_score = "mode", use the most probable category instead; ties go to the lower category. Both summaries and all probabilities remain available whichever score you report.

expected_level is an average of category positions (1, 2, 3, and so on), not an average of your numeric labels. It is a summary rather than an assigned rubric level. extrapolated flags scores outside the range covered by training labels, even for samples in the original analysis.

For an illustrative probability vector (developing = 0.20, proficient = 0.55, advanced = 0.25), cumulative probability first reaches 0.5 at “proficient”, so that is the default median-category score. The probabilities also show substantial uncertainty; reporting the category alone discards it. These numbers illustrate the decision rule and are not fitted results.

For a report, retain item_id, rubric_score, probabilities, and extrapolated for ordinal predictions. Explain an extrapolation flag as a score outside the labeled calibration range. Percentile predictions instead require requested and achieved category proportions. Keep the method, trait, and validation results with either kind of output.

Evaluate held-out labels and inspect diagnostics

if (requireNamespace("ordinal", quietly = TRUE)) {
  assessment <- evaluate_rubric_predictions(
    calibration, evaluation_labels, diagnostics = TRUE
  )
  assessment$metrics
  assessment$calibration
  assessment$metadata$n_training_label_overlap
  assessment$diagnostics$common_effect
}

Start with ranked probability score (rps): lower values indicate better predictions, accounting for how far a prediction falls from the observed level. Log loss is another probability-based check. For the assigned categories, inspect exact agreement, agreement within one level, average category error and weighted kappa (agreement adjusted for chance). Distances count category steps. If a metric cannot be calculated, the output explains why.

Evaluation matches samples by ID, counts missing labels and reports any overlap with training labels. Unlike training, an evaluation set need not contain every category. assessment$calibration groups similar predictions and compares them with observed human scores, helping you see where the model is too optimistic or too pessimistic.

Diagnostics are off by default. For the linear method, they check the assumption that CJ quality has a shared effect across rubric boundaries. For the curved method, they provide descriptive boundary checks. These checks use training data; the separate evaluation labels do not change your fitted calibration. If a check fails to run, it is marked unavailable and the original fit is preserved.

For repeated validation, keep the completed CJ analysis fixed. Repeatedly set aside some human labels, fit a fresh calibration using only the remaining labels, and evaluate the labels you set aside. Every training group must still include all rubric levels. If you use those results to choose a model or its settings, reserve another independent evaluation set for your final report.

Explicit monotone alternative

if (requireNamespace("mgcv", quietly = TRUE) &&
    packageVersion("mgcv") >= "1.9.4" && requireNamespace("withr", quietly = TRUE)) {
  monotone_calibration <- fit_rubric_calibration(
    completed_cj, training_labels, method = "ordinal_monotone",
    trait = "organization", levels = rubric_levels, k = 6
  )
  monotone_scores <- predict(monotone_calibration)
  monotone_assessment <- evaluate_rubric_predictions(
    monotone_calibration, evaluation_labels
  )
  monotone_assessment$metrics
}

This fits a smooth curve that can bend while never decreasing as CJ quality increases. k controls how much flexibility is available; its default is 6 and it must be at least 5. The package may reduce it with a warning when too few distinct training scores are available. At least five distinct scores are needed to fit this curve, but that minimum does not guarantee enough data for useful calibration. See the function help for smoothing controls.

Check fit diagnostics and performance on separate human labels before preferring the more flexible method. A flat part of the curve may make a category cutpoint unavailable; predictions still use category probabilities. Scores outside the training range are flagged because the curve is being extended beyond the scores used to learn it.

Percentile levels without human labels

percentile_calibration <- fit_rubric_calibration(
  completed_cj, method = "percentile", trait = "organization",
  levels = rubric_levels, target_distribution = c(0.2, 0.5, 0.3)
)
percentile_scores <- predict(percentile_calibration)
percentile_calibration$backend$requested_proportions
percentile_calibration$backend$achieved_proportions
percentile_calibration$backend$cutpoints
percentile_calibration$backend$cutpoint_tie_counts

The example asks for 20% of samples in the lowest level, 50% in the middle and 30% in the highest. Omit target_distribution for equal proportions. You can supply the number of levels with K instead of naming them.

Tied scores stay together. This means the achieved proportions may differ from what you requested, and some categories may be empty. A score exactly on a boundary enters the higher category; if all scores are identical, this rule puts every sample in the highest category. Always inspect the achieved proportions.

These levels describe relative standing within this set. Matching a historical rubric distribution does not establish agreement with human raters for individual samples. Percentile scoring reuses only the original analysis; it cannot transport levels to a separate cohort through linked_anchors. It returns categories without category probabilities, so evaluation uses only category-based metrics.

See What the research shows for a study comparing percentile scoring, human-score calibration, and direct LLM rubric scoring.

Reuse a historical rubric reference for a new set

You can use a human-scored reference set to score new writing samples after linking both sets onto a common scale. The reference is the hub and the new set is the spoke. Phase A means ranking each set separately; Phase B links them using comparisons between sets. A hub sample is a reference for comparison, not a sample whose writing quality is known without uncertainty.

1. Rank the reference samples on the chosen trait and keep the original Phase A result.
2. Fit and save the rubric calibration using that result and its human scores.
3. Rank the new samples using compatible settings and keep their Phase A result.
4. Choose E1, E2 or E3 and link the sets using between-set comparisons.
5. Check the link, then apply the saved calibration to the linked target scores.

The following example assumes reference_phase_a contains the original reference artifact (the saved Phase A result) and reference_labels contains its human scores. Read a saved artifact with readRDS() first; supply the artifact itself, not a filename or the entire collection of saved-run metadata.

if (requireNamespace("ordinal", quietly = TRUE)) {
  linked_calibration <- fit_rubric_calibration(
    reference_phase_a, reference_labels,
    calibration_design = "linked_anchors",
    trait = "organization", levels = rubric_levels
  )
  reference_scores <- predict(linked_calibration)
}

reference <- prepare_linked_rubric_reference(
  completed_reference, reference_comparisons,
  set_id = "reference", items = reference_items, trait = "organization"
)
standalone_calibration <- fit_rubric_calibration(
  reference, reference_labels, calibration_design = "linked_anchors",
  levels = rubric_levels
)
saveRDS(reference, "rubric-reference.rds")
saveRDS(standalone_calibration, "rubric-calibration.rds")

link_input <- prepare_link_input(
  estimator = "gaussian_posterior_bridge",
  hub = list(set_id = "1",
    items = reference_phase_a$items[c("item_id", "global_item_id")]),
  spoke = list(set_id = "2",
    items = target_phase_a$items[c("item_id", "global_item_id")]),
  phase_a = list(hub = list(artifact = reference_phase_a),
    spoke = list(artifact = target_phase_a)),
  cross = active_cross, judge = frozen_judge
)
completed_phase_b <- start_link_session(link_input)

if (requireNamespace("ordinal", quietly = TRUE)) {
  target_scores <- predict(linked_calibration, newdata = completed_phase_b)
  target_scores[, c("global_item_id", "set_id", "rubric_score", "theta_sd", "extrapolated")]
  attr(target_scores, "linking")$diagnostics
}

if (requireNamespace("ordinal", quietly = TRUE)) {
  target_assessment <- evaluate_rubric_predictions(
    linked_calibration, target_labels, newdata = completed_phase_b
  )
  target_assessment$metrics
  target_assessment$metadata$linking
}

Separate analytic traits

Organization and evidence require separate trait-specific comparisons, CJ fits, labels, and calibrations even when item IDs overlap. Each may have different levels or a different Bayesian variant. Compatibility must hold within each trait’s reference/target linking workflow. One holistic CJ scale cannot supply multiple analytic trait calibrations.

if (requireNamespace("ordinal", quietly = TRUE)) {
  evidence_calibration <- fit_rubric_calibration(
    evidence_cj, evidence_training_labels, trait = "evidence",
    levels = c("limited", "adequate", "strong")
  )
  evidence_scores <- predict(evidence_calibration)
}

Uncertainty and reporting

All three methods condition on accepted CJ point locations. Retained CJ SDs, posterior draws, or linking uncertainty are not propagated into rubric category probabilities. Ordinal parameter uncertainty is conditional on those estimated predictors; monotone threshold uncertainty is not supplied by the wrapper. There is no posterior/bootstrap propagation engine or joint likelihood here.

Report the trait, BTL variant, method/design, ordered levels, labeled category counts and range, held-out metrics, extrapolation, and CJ/calibration/linking diagnostics. Distinguish numerical feasibility from adequate measurement and external rubric validity. Small deterministic examples and package recovery tests do not establish a universal calibration sample size or validity for a new population.

Citation

Mercer, S. H. (2026). Guide: Rubric Calibration [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM