Guide: Adaptive Pairing

What you will do

The integrated CJ workflow is the conceptual entry point for selection, estimation, reliability, recovery, and held-out prediction. Use this guide for the practical collection and persistence steps.

This article shows how to rank one set of items with [adaptive_rank()]. The wrapper reads and validates the items, starts or resumes a run, requests comparisons from a judge, performs Bayesian refits, and returns the ranking and audit logs. Most users should start with this wrapper rather than assembling the lower-level state and runner functions themselves. Executable CmdStan chunks are disabled during ordinary package builds. Set PAIRWISELLM_RUN_CMDSTAN_VIGNETTES=true to opt in when rendering this source locally.

You need: one set of texts with stable IDs and a working CmdStan setup. The first example uses a simulated local judge; the later live example also needs a provider account. You get: item scores and uncertainty, a stopping status, and logs of the comparisons and refits. A run returning successfully is not by itself evidence that its statistical stopping criteria passed.

Read the offline run and its inspection steps first. Strategy details, custom judges, replay, and predictive initialization are optional follow-up sections.

Choose a comparison workflow

Use exhaustive pairing when the set is small enough to evaluate every one of the choose(N, 2) unordered pairs and complete coverage is important. Use a random sample when you want a simple fixed-budget design or a baseline for comparison. Use adaptive selection when each judgment is costly and you want the next comparison to respond to evidence already collected.

The runtime accepts N >= 2 unique items. A range such as 30–2,000 items is operating guidance, not an enforced limit. Very small sets may exhaust all eligible candidates before a Bayesian refit, while large studies should be planned around provider cost, refit time, and storage.

Install and check CmdStan

Bayesian refits require the suggested cmdstanr and parallelly packages, a working C++ toolchain, and CmdStan. This is a one-time machine setup; it does not require an LLM credential.

install.packages("parallelly")
install.packages(
  "cmdstanr",
  repos = c("https://stan-dev.r-universe.dev", getOption("repos"))
)
cmdstanr::check_cmdstan_toolchain(fix = TRUE)
cmdstanr::install_cmdstan()

Before starting a study, confirm that CmdStan can be found:

cmdstan_available <- requireNamespace("cmdstanr", quietly = TRUE) &&
  tryCatch(!is.null(cmdstanr::cmdstan_version()), error = function(e) FALSE)
cmdstan_available
#> [1] TRUE

if (cmdstan_available) {
  cmdstanr::cmdstan_version()
}
#> [1] "2.39.0"

A TRUE check means CmdStan was found; it does not test compilation. The ranking chunks also require the opt-in flag when rendering this document. You can run the displayed R code interactively after completing setup. The deterministic judge is local, so the example never makes a network request or incurs provider charges.

MCMC refits use the same CPU allocation rules as standalone Bayesian fits. Automatic scheduling uses at most two CPU slots with core_fraction = 0.8; explicit parallel-chain requests must fit the available chain/thread budget. These settings control execution, not total chains or adaptive stopping. See Chains and CPU allocation for the controls and resource fields recorded in the refit log.

A complete offline run with adaptive_rank()

Start from a clean R session and load the package data. The ID column already contains unique, non-missing identifiers. We retain quality_score only to simulate a judge.

library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")
samples <- example_writing_samples[, c("ID", "text", "quality_score")]

trait <- trait_description("overall_quality")
prompt_template <- set_prompt_template()

deterministic_judge <- function(A, B, state, ...) {
  a_wins <- A$quality_score[[1]] >= B$quality_score[[1]]
  list(
    is_valid = TRUE,
    Y = as.integer(a_wins),
    invalid_reason = NA_character_
  )
}

A custom judge receives one-row item tables A and B plus the current state. A valid response must contain is_valid = TRUE and Y equal to 1 when A wins or 0 when B wins. The simulated judge uses a fixture score so its choices are deterministic. Because it bypasses an LLM, the trait and prompt are not consulted until the live example later in this article.

The minimum useful wrapper call supplies data, its ID and text columns, a judge, and a step budget. We also choose a session directory so progress is persisted. With 20 items, the implemented default refit_pairs_target is 20 committed comparisons: ceiling(N / 2) clamped to [20, 5000]. A budget of 22 attempted steps covers the connected bootstrap and produces one refit under this deterministic fixture.

session_dir <- tempfile("pairwisellm-adaptive-")

out <- adaptive_rank(
  data = samples,
  id_col = "ID",
  text_col = "text",
  judge = deterministic_judge,
  n_steps = 22L,
  session_dir = session_dir,
  persist_item_log = TRUE,
  resume = FALSE,
  seed = 42L,
  progress = "none"
)

n_steps is a maximum number of attempted steps for that call, not a promise that every attempt will commit and not a global stopping criterion. A run can return earlier after Bayesian stopping or candidate starvation. A requested comparison budget does not guarantee that enough permitted pairs remain to reach it. Hybrid usually allows two observations of a pair; a third is allowed only for selected close comparisons. These repeat rules can stop a run even when the requested budget is below the maximum obtained by allowing three observations of every pair.

out$summary

out$items[, c(
  "item_id", "theta_raw_eap", "theta_raw_sd", "rank_raw", "degree"
)] |>
  head()

out$refits[, c(
  "refit_id", "total_pairs_done", "diagnostics_pass",
  "reliability_EAP", "stop_decision", "stop_reason"
)]

names(out$logs)

step_log <- adaptive_step_log(out$state)
step_log[, c(
  "step_id", "pair_id", "A_id", "B_id", "Y", "status",
  "round_stage", "fallback_used", "starvation_reason"
)] |>
  head()

summarize_adaptive(out$state)
summarize_refits(out$state, last_n = 1L, include_optional = FALSE)
summarize_items(out$state, top_n = 5L)

result_history <- adaptive_results_history(out$state)
head(result_history)

What happens during a run?

Within-set ranking has three phases.

1. Connected shuffled bootstrap. Every predictive mode uses the same seeded chain of N - 1 valid comparisons to connect the observed graph. Predictive initialization occurs before these observations and does not change their schedule.
2. Post-bootstrap pairing. TrueSkill updates after every valid comparison. The selected pairing strategy chooses subsequent pairs; the default hybrid balances anchor, long, mid, and local comparisons, exposure, repeated-pair limits, and fallbacks.
3. Phase 3: near stop. After a Bayesian refit passes diagnostics and its EAP reliability is within 0.05 of the stopping threshold, later refits use the stricter near-stop ESS requirement. Pair-selection behavior does not otherwise change merely because Phase 3 was entered.

TrueSkill is the fast, step-by-step selection model. Bayesian Bradley–Terry–Luce (BTL) refits are slower and intermittent. They provide posterior item estimates and uncertainty, convergence diagnostics, EAP reliability, stability checks, and stopping decisions. Hybrid live ranks, strata, rolling anchors, pair probabilities, base utility, and the long-link gate use TrueSkill throughout. BTL still contributes to global_identified, which can change later hybrid tapering and routing; selection is not wholly independent of BTL.

items -> predictive initialization -> connected shuffled bootstrap
  -> select one pair -> judge -> commit valid outcome and update TrueSkill
           ^                          |
           |                          v
           +-- continue <- periodic BTL refit -> stop check

Adaptive selection is intended to direct effort toward useful comparisons, but it does not guarantee a particular ranking, reliability, cost reduction, or improvement over random pairing. Those outcomes depend on the items, judge, budget, and model assumptions.

For sparse reservoir replay, the bootstrap is instead a seeded spanning tree of N - 1 allowed edges. Its schedule is shared by all predictive modes and pairing strategies; see Replay a sparse reservoir.

Choose how later pairs are selected

Every strategy starts with the same initial comparisons that connect the samples. After that, choose a strategy with, for example, adaptive_config = list(pairing_strategy = "trueskill_p50").

The last three are called direct strategies. They first choose a sample from those with the fewest completed comparisons, then choose its partner using the rule above. A fixed seed makes those choices reproducible. The last method is Pollitt-inspired: it uses TrueSkill probabilities, while the earlier published approach used BTL probabilities. The design guide gives the exact selection rules and tie handling.

Direct strategies allow at most two judgments of a pair. They balance presentation order and reverse it for a repeat. An invalid result does not advance the selection. They stop if the chosen sample has no allowed partner, even if some other pairs remain. Direct strategies support ordinary within-set runs only; linking Phase A runs require hybrid. Automatic Phase B selection is unavailable pending separate validation. To link saved comparisons, see the linking guide.

In the step log, direct_pairing identifies these comparisons, pairing_strategy records the chosen rule and i_id records the sample chosen first. p_ij is the predicted chance that the sample presented as A beats B. target_distance records how close a pair came to the desired probability; it is missing for random pairing. BTL fits and stopping checks continue at their usual intervals for every strategy.

Hybrid recovery attempts remain part of hybrid, including base, expand_locality, uncertainty_pool, dup_relax, and global_safe; they do not substitute another strategy. See the integrated explanation and the exact recovery rules.

Lower-level lifecycle and offline judges

Most users should use adaptive_rank(). The lower-level lifecycle is useful when an application needs to hold the state itself: create it with adaptive_rank_start(), advance it with adaptive_rank_run_live(), persist it, and reconstruct it with adaptive_rank_resume() or load_adaptive_session(). Do not modify state lists or canonical log columns by hand.

low_level_items <- samples[1:5, ]
low_level_items$item_id <- as.character(low_level_items$ID)

low_level_state <- adaptive_rank_start(low_level_items, seed = 17L)
low_level_state <- adaptive_rank_run_live(
  state = low_level_state,
  judge = deterministic_judge,
  n_steps = 3L,
  btl_config = list(refit_pairs_target = 5000L),
  progress = "none"
)

summarize_adaptive(low_level_state)

A custom judge has the contract judge(A, B, state, ...). A and B are one-row item tables. Return list(is_valid = TRUE, Y = 1L) when A wins or Y = 0L when B wins. To simulate refusal, timeout, or parsing failure, return is_valid = FALSE, Y = NA_integer_, and a stable invalid_reason. Use a local score column, a fixed lookup table, or a separately seeded random generator to make simulations reproducible. The package’s internal scenario harness is test infrastructure, not a public workflow function.

make_adaptive_judge_llm() builds the same judge interface around llm_compare_pair(). It is a lower-level alternative to supplying backend/model arguments to adaptive_rank(), and its calls are live, billable, and subject to the selected provider’s failure and privacy behavior.

sparse_ids <- c("a", "b", "c", "d")
sparse_outcomes <- data.frame(
  A_id = c("b", "a", "d", "c"),
  B_id = c("a", "c", "a", "d"),
  Y = c(1L, 0L, 1L, 1L)
)
reservoir <- make_adaptive_replay_reservoir(sparse_outcomes, sparse_ids)
sparse_state <- adaptive_rank_start(sparse_ids, seed = 17L,
  replay_reservoir = reservoir,
  adaptive_config = list(pairing_strategy = "random"))
sparse_judge <- make_adaptive_judge_replay(reservoir)
sparse_state <- adaptive_rank_run_live(sparse_state, sparse_judge,
  n_steps = 4L, progress = "none")
adaptive_step_log(sparse_state)[, c("A_id", "B_id", "Y", "pairing_strategy")]
#> # A tibble: 4 × 4
#>   A_id  B_id      Y pairing_strategy
#>   <chr> <chr> <int> <chr>           
#> 1 a     c         0 random          
#> 2 b     a         1 random          
#> 3 c     d         1 random          
#> 4 d     a         1 random

sparse_dir <- tempfile("sparse-replay-")
save_adaptive_session(sparse_state, sparse_dir)
sparse_restored <- adaptive_rank_resume(sparse_dir)
sparse_judge <- make_adaptive_judge_replay(reservoir)
# Continue with adaptive_rank_run_live(sparse_restored, sparse_judge, ...).
unlink(sparse_dir, recursive = TRUE)

replay_ids <- c("a", "b", "c", "d")
frozen <- expand.grid(A_id = replay_ids, B_id = replay_ids, stringsAsFactors = FALSE)
frozen <- frozen[frozen$A_id != frozen$B_id, ]
# Fabricated ordered outcomes, used only to demonstrate the interface.
frozen$Y <- as.integer(frozen$A_id < frozen$B_id)
frozen <- validate_adaptive_replay(frozen, item_ids = replay_ids)
replay_judge <- make_adaptive_judge_replay(frozen, item_ids = replay_ids)
replay_state <- adaptive_rank_start(replay_ids, seed = 17L,
  adaptive_config = list(pairing_strategy = "trueskill_p50", dup_max_obs_relaxed = 2L))
replay_state <- adaptive_rank_run_live(replay_state, replay_judge,
  n_steps = 5L, progress = "none")
adaptive_step_log(replay_state)[, c("A_id", "B_id", "Y", "pairing_strategy")]
#> # A tibble: 5 × 4
#>   A_id  B_id      Y pairing_strategy
#>   <chr> <chr> <int> <chr>           
#> 1 a     b         1 trueskill_p50   
#> 2 d     a         0 trueskill_p50   
#> 3 c     d         1 trueskill_p50   
#> 4 b     d         1 trueskill_p50   
#> 5 a     c         1 trueskill_p50

Persistence, interruption, and resume

Supplying session_dir makes the wrapper persist the initial state, every completed refit, terminal stops, the end of an ordinary call, and intermediate checkpoints. The default checkpoint cadence is every 100 attempted steps; change it with checkpoint_every_steps when losing that much work would be costly. Frequent checkpoints increase disk I/O.

An abrupt process termination can lose attempts since the most recent completed save. After an interrupt, validate what is on disk rather than assuming the in-memory state was written.

session_metadata <- validate_session_dir(session_dir)
session_metadata[c("schema_version", "package_version", "n_items",
  "warm_start_mode", "pairing_strategy")]

loaded_state <- load_adaptive_session(session_dir)
summarize_adaptive(loaded_state)

resumed <- adaptive_rank(
  data = samples,
  id_col = "ID",
  text_col = "text",
  judge = deterministic_judge,
  n_steps = 2L,
  session_dir = session_dir,
  persist_item_log = TRUE,
  resume = TRUE,
  progress = "none"
)

c(
  before = nrow(out$logs$step_log),
  after = nrow(resumed$logs$step_log)
)

Resume is intentionally strict. If saved artifacts are present but invalid, adaptive_rank() aborts instead of silently creating a new run. The input IDs and their order must exactly match the saved session. Set resume = FALSE only when you deliberately want a new state, and choose a new or empty session directory to avoid mixing studies.

Resume preserves mode, strategy, predictive prior/provenance, current TrueSkill state, committed history, bootstrap progress, and round state. Omit all predictive warm-start arguments; omit strategy or supply the saved strategy. A legacy session without mode becomes cold if no prior is saved and btl_only if one is saved; missing strategy becomes hybrid. Migration never warms TrueSkill retroactively. Within-set continuation also reuses saved btl_config when it is omitted.

The wrapper handles ordinary saving. For an explicit snapshot outside a wrapper call, use the lower-level persistence helpers:

snapshot_dir <- tempfile("pairwisellm-snapshot-")
save_adaptive_session(resumed$state, snapshot_dir, overwrite = TRUE)
validate_session_dir(snapshot_dir)
snapshot <- load_adaptive_session(snapshot_dir)

Setting persist_item_log = TRUE writes a separate item-log file for each refit. The canonical state, step log, round log, and metadata are persisted regardless; the option is useful when per-refit item histories must be inspected independently.

Invalid judgments and failed calls

Invalid judge output is transactional: it creates an attempted step with no committed pair_id, does not update TrueSkill or comparison history, and does not count toward the BTL refit cadence. It still consumes one unit of n_steps. The LLM judge created internally by adaptive_rank() maps provider errors to invalid results and preserves available status, error, token, and raw response fields in the step log.

This small example deliberately rejects its first attempt, then accepts the next one:

invalid_once_judge <- function(A, B, state, ...) {
  if (nrow(state$step_log) == 0L) {
    return(list(
      is_valid = FALSE,
      Y = NA_integer_,
      invalid_reason = "simulated_parse_failure"
    ))
  }
  deterministic_judge(A, B, state)
}

invalid_demo <- adaptive_rank(
  data = samples[1:4, ],
  id_col = "ID",
  text_col = "text",
  judge = invalid_once_judge,
  n_steps = 2L,
  resume = FALSE,
  seed = 7L,
  progress = "none"
)

adaptive_step_log(invalid_demo$state)[, c(
  "step_id", "pair_id", "status", "judge_valid", "judge_invalid_reason"
)]
#> # A tibble: 2 × 5
#>   step_id pair_id status  judge_valid judge_invalid_reason   
#>     <int>   <int> <chr>   <lgl>       <chr>                  
#> 1       1      NA invalid FALSE       simulated_parse_failure
#> 2       2       1 ok      TRUE        <NA>

Investigate repeated invalid rows before increasing the budget. Common causes include a thrown judge error, a response that does not name either presented ID, or a custom judge that returns a missing or non-binary Y.

A detailed live-LLM workflow

The following is a live-API example and is not evaluated. It requires an OpenAI key, network access, and incurs provider charges. The request shape shown here was recorded as tested with pairwiseLLM 1.3.1 on 2026-09-05. That is dated evidence about this configuration, not a promise that the provider still offers the model. Check Backends and Tested Model Configurations and the provider catalog before a long run.

Set OPENAI_API_KEY outside the script. Replace real_samples with a data frame or file containing one row per item, a unique ID, and the text to compare. Preserve that input in its exact row order for resume.

library(pairwiseLLM)

stopifnot(nzchar(Sys.getenv("OPENAI_API_KEY")))

real_samples <- utils::read.csv(
  "writing-samples.csv",
  stringsAsFactors = FALSE
)
stopifnot(nrow(real_samples) >= 2L)
stopifnot(all(c("ID", "text") %in% names(real_samples)))
stopifnot(!anyNA(real_samples$ID), !anyDuplicated(real_samples$ID))
stopifnot(!anyNA(real_samples$text), all(nzchar(real_samples$text)))

live_trait <- trait_description("overall_quality")
live_prompt <- set_prompt_template()
live_session <- "adaptive-live-session"

live <- adaptive_rank(
  data = real_samples,
  id_col = "ID",
  text_col = "text",
  backend = "openai",
  model = "gpt-5.6-luna",
  endpoint = "responses",
  trait_name = live_trait$name,
  trait_description = live_trait$description,
  prompt_template = live_prompt,
  judge_args = list(reasoning = "none"),
  n_steps = 200L,
  session_dir = live_session,
  checkpoint_every_steps = 10L,
  persist_item_log = TRUE,
  resume = FALSE,
  seed = 20260904L,
  progress = "refits",
  save_outputs = TRUE
)

This one call validates and normalizes the data, constructs the LLM judge, initializes the within-set controller, attempts at most 200 judgments, performs Bayesian refits when due, persists the session, and returns reporting views. n_steps is a per-call ceiling, not a target number of valid comparisons or a guarantee that the stopping criteria will pass. Use a new or empty session_dir with resume = FALSE.

The call intentionally retains the package’s inference and stopping defaults. Do not copy reduced diagnostic thresholds from a test fixture into a real study merely to make it finish sooner.

live$summary

live$items[, c(
  "item_id", "theta_raw_eap", "theta_raw_sd", "rank_raw", "degree"
)] |>
  head(10L)

live$refits[, c(
  "refit_id", "total_pairs_done", "diagnostics_pass",
  "reliability_EAP", "stop_decision", "stop_reason"
)] |>
  tail()

live_steps <- live$logs$step_log[, c(
  "step_id", "pair_id", "A_id", "B_id", "Y", "status",
  "judge_backend", "judge_model", "judge_endpoint", "judge_invalid_reason",
  "llm_status_code", "llm_error_message",
  "prompt_tokens", "completion_tokens", "total_tokens"
)]

tail(live_steps)

colSums(live_steps[, c(
  "prompt_tokens", "completion_tokens", "total_tokens"
)], na.rm = TRUE)

live_steps[live_steps$status == "invalid", ]

live <- adaptive_rank(
  data = real_samples,
  id_col = "ID",
  text_col = "text",
  backend = "openai",
  model = "gpt-5.6-luna",
  endpoint = "responses",
  trait_name = live_trait$name,
  trait_description = live_trait$description,
  prompt_template = live_prompt,
  judge_args = list(reasoning = "none"),
  n_steps = 100L,
  session_dir = live_session,
  checkpoint_every_steps = 10L,
  persist_item_log = TRUE,
  resume = TRUE,
  progress = "refits",
  save_outputs = TRUE
)

Troubleshooting

CmdStan is missing or compilation fails. Run cmdstanr::check_cmdstan_toolchain() and cmdstanr::cmdstan_version(). Install or repair the compiler toolchain before retrying. An LLM credential cannot substitute for CmdStan.

IDs are missing or duplicated. id_col must identify a non-missing, unique column. For the shipped data it is id_col = "ID". Resume also requires the same IDs in the same order as the saved state.

The judge output is malformed. A custom judge must return is_valid, Y, and preferably invalid_reason. For a valid result, Y must be exactly 0 or 1. Review judge_invalid_reason, llm_status_code, and llm_error_message in adaptive_step_log().

The run reports candidate starvation. Hybrid tries its fallbacks before stopping when none of the examined pairs can be selected. Repeat rules or limits on how often an item is used can leave no permitted comparison before the requested budget is reached. Passing the Bayesian model checks does not establish that another pair is available, and this stop is not Bayesian convergence. Increasing the budget alone does not remove the pairing restrictions.

For a saved or returned state, inspect the comparisons completed and the recorded explanation:

availability <- summarize_adaptive(out$state, include_starvation = TRUE)
availability[, c("committed_pairs", "last_stop_reason")]
availability$starvation_diagnostic[[1L]]

The optional report is NULL if there is no current terminal report, including older saved sessions. Otherwise it shows the remaining arithmetic upper bound and what happened in each attempt. A positive upper bound means that some repeat slots have not been used; those repeats may still be prohibited by the pairing rules. Candidate counts describe the pairs examined in each attempt, and the same pair may appear in several attempts. For large sets, some attempts examine a sample, so an empty examined pool does not prove that every possible pair was checked. See [summarize_adaptive()] for the report’s fields.

Direct strategies stop when the chosen minimum-degree focal item has no legal partner; they do not search other focal items or invoke hybrid fallbacks. They do not produce this hybrid report.

There is no item summary yet. out$items is empty until the first successful BTL refit. Under the default cadence, at least 20 new committed comparisons are needed. Invalid and starved attempts do not advance that count.

Related documentation

See [adaptive_rank()], [adaptive_step_log()], [adaptive_round_log()], [adaptive_item_log()], [save_adaptive_session()], and [validate_session_dir()] for complete argument and schema details. For the within-set algorithm and statistical rationale, see Design: Adaptive Pairing.

Predictive warm-start priors

Warm starts are optional. They give BTL, TrueSkill, or both an initial estimate of each item’s relative quality. BTL estimates scores and uncertainty; TrueSkill provides the fast estimates used to help choose comparisons.

Supply either a fitted warm_start_model or a warm_start_prior containing resolved scores and your chosen prior standard deviations. Models may be single predictors, cross-task ensembles from separate assessments, or same-task ensembles of several algorithms. The warm-start guide walks through features, training, validation, prediction, and initialization offline.

Choose warm_start_mode = "btl_only", "trueskill_only", or "both" to select where predictions are used. With predictive input the default is btl_only; without it the default is cold. TrueSkill’s starting uncertainty is unchanged. Disagreement between prediction models does not determine either model’s prior uncertainty. All modes begin with the same seeded connected comparisons; later TrueSkill-based choices can differ.

With model input, precomputed warm_start_features avoids Python. Otherwise, warm_start_python selects an already configured extraction environment. Prediction runs once and the session stores the numeric prior. On resume, omit all warm-start arguments: the saved values remain authoritative even if the original model file is removed.

Citation

Mercer, S. H. (2026). Guide: Adaptive pairing [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM