Package 'pairwiseLLM'

Description

Retrieve canonical adaptive logs.

Usage

adaptive_get_logs(state)

Arguments

Details

Returns the canonical adaptive logs as currently held in memory: step_log, round_log, item_log, and link_stage_log. These correspond to step attempts, posterior refit rounds, item-level refit summaries, and per-refit linking summaries.

Value

A named list with four elements:

step_log

A tibble with one row per attempted step.

round_log

A tibble with one row per BTL refit round.

item_log

A list of per-refit item tibbles.

link_stage_log

A tibble with one row per (refit_id, spoke_id) linking summary when linking mode is active.

See Also

adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive logs: adaptive_item_log(), adaptive_results_history(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
logs <- adaptive_get_logs(state)
names(logs)

Description

Adaptive item log accessor.

Usage

adaptive_item_log(state, refit_id = NULL, stack = FALSE)

Arguments

Details

item_log stores per-item posterior summaries by refit. The underlying state stores a list of refit tables; this accessor can return one refit table (default: most recent) or stack all refits into a single tibble.

Item-level summaries are domain-explicit:

• theta_raw_*: raw/within-set posterior summaries (EAP, fixed quantiles, SD, rank) at the current refit.

• theta_link_*: linked/global posterior summaries (EAP, fixed quantiles, SD, rank) after transform application.

• During linking Phase A (phase_scope = "phase_a_set"), theta_link_* is typed NA by design.

• phase_scope, phase_scope_set_id, and in_phase_scope indicate which item domain is currently optimized.

Value

A tibble of item-level summaries. When stack = FALSE, one row per item for the selected refit. When stack = TRUE, one row per item per refit with refit_id identifying source refit.

See Also

adaptive_get_logs(), summarize_items(), adaptive_round_log()

Other adaptive logs: adaptive_get_logs(), adaptive_results_history(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_item_log(state)

Description

High-level workflow wrapper that reads sample data, constructs an LLM judge, starts or resumes adaptive state, runs adaptive_rank_run_live(), and returns state plus summary outputs.

Usage

adaptive_rank(
  data,
  id_col = 1,
  text_col = 2,
  backend = c("openai", "anthropic", "gemini", "vertex", "together", "ollama"),
  model = NULL,
  trait = "overall_quality",
  trait_name = NULL,
  trait_description = NULL,
  prompt_template = set_prompt_template(),
  endpoint = "chat.completions",
  api_key = NULL,
  include_raw = FALSE,
  judge_args = list(),
  judge_call_args = list(),
  n_steps = 1L,
  fit_fn = NULL,
  adaptive_config = NULL,
  btl_config = NULL,
  session_dir = NULL,
  persist_item_log = FALSE,
  checkpoint_every_steps = NULL,
  resume = TRUE,
  seed = 1L,
  progress = c("all", "refits", "steps", "none"),
  progress_redraw_every = 10L,
  progress_show_events = TRUE,
  progress_errors = TRUE,
  save_outputs = FALSE,
  output_file = NULL,
  judge = NULL,
  warm_start_model = NULL,
  warm_start_prior = NULL,
  warm_start_features = NULL,
  warm_start_python = NULL,
  warm_start_prior_sd = NULL,
  warm_start_mode = NULL,
  replay_reservoir = NULL,
  warm_start_trueskill = NULL,
  bootstrap_policy = "shuffled_connected"
)

Arguments

Details

This helper is designed for end users who want one entry point for adaptive runs. It supports:

• data input from a data frame, file (.csv, .tsv, .txt, .rds), or a directory of .txt files;

• model/backend configuration through make_adaptive_judge_llm();

• all adaptive runtime controls exposed by adaptive_rank_run_live();

• wrapper-visible phase_a reuse surfaces (manifest, artifact_dir, and per-set status) for separate-run then later-link workflows;

• resumability via session_dir and resume;

• optional saving of run outputs to an .rds artifact.

Model options: use judge_args (fixed) and judge_call_args (per-run overrides) to pass any additional llm_compare_pair() arguments, including provider-specific controls such as reasoning, service_tier, temperature, top_p, logprobs, include_thoughts, or host.

Adaptive options: all key controls from adaptive_rank_run_live() are available directly: n_steps, fit_fn, adaptive_config, btl_config, progress, progress_redraw_every, progress_show_events, progress_errors, session_dir, and persist_item_log. Use adaptive_config for identifiability-gated controller behavior and btl_config for inference/diagnostics cadence only.

Linking run modes: run_mode = "within_set" is the single-set workflow. run_mode = "link_one_spoke" and run_mode = "link_multi_spoke" require multi-set input (set_id/global_item_id), enforce hub<->spoke routing defaults, and preserve Phase A artifact gating before Phase B cross-set comparisons begin. An explicit link_estimation_mode is required. Adaptive Phase B is unavailable pending selector validation; use start_link_session() with explicit evidence for E1–E3 estimation. Every wrapper call returns canonical phase_a outputs that can be fed back into a later linking run through adaptive_config$phase_a_artifacts.

Selection semantics: selection uses one-pair transactional steps after the configured connected bootstrap. In the default hybrid strategy, TrueSkill supplies live ranks, strata, pair probabilities, base utility, and rolling anchors throughout within-set and Phase-A work. Rolling anchors use current TrueSkill ranks, and anchor-link routing compares exactly one anchor endpoint with one non-anchor endpoint. Long/mid-link routing excludes anchor-anchor and anchor-non-anchor pairs, while local-link routing admits same-stratum pairs and anchor-involving pairs according to stage bounds.

Wrapper-visible defaults include top-band refinement (top_band_pct = 0.10, top_band_bins = 5) with top-band size computed as ceiling(top_band_pct * N).

Hybrid exposure and repeat routing: under-represented routing is degree-based (deg <= D_min + 1), while repeat-pressure gating is based on recent exposure (bottom-quantile recent_deg with quantile default 0.25) and per-endpoint repeat slot accounting.

Inference separation: BTL refits supply item estimates, posterior uncertainty, EAP reliability, diagnostics, stopping, and the existing global_identified signal. That signal can change later hybrid tapering and routing, so selection is not wholly independent of BTL. The within-set/Phase-A long-link gate uses TrueSkill probabilities throughout. Automatic linking Phase B selection is unavailable pending validation; use prepare_link_input() and start_link_session().

Resume behavior: when resume = TRUE and session_dir already contains adaptive artifacts, failed session loads abort with an actionable error instead of starting a fresh run silently.

Predictive destinations and initial connectivity are separate choices. By default, every warm mode retains the same seeded shuffled bootstrap of N - 1 valid comparisons. The explicit predictive graph policy uses a frozen allowed spanning tree with Pollitt probability targets and degree-cap relaxation. Both policies preserve invalid-result retries and recorded reservoir orientation. Later pairing retains the configured strategy. For budgets expressed as mean comparison exposures per item, each committed pair counts twice. The synthetic five-arm example uses floor(B * N / 2) total committed pairs, including bootstrap, and reports realized exposure 2 * committed_pairs / N. B = 0 is prior-only; B = 0.5 and B = 1 precede connectivity. The tree completes at N - 1 successful commits; comparison N uses the post-bootstrap strategy. n_steps counts attempts, so invalid or retried judgments must not be counted as additional committed evidence. See vignette("adaptive-warm-start") for count and continuation examples. Without the distribution opt-in, BTL prior SD does not determine TrueSkill sigma. Ensemble disagreement never supplies SD automatically. No historical training-score units are restored.

Distribution initialization assumes the supplied BTL prior SD and TrueSkill uncertainty describe comparable latent scales under the documented affine convention. BTL priors apply to theta_raw; centering induces dependence, so this is not the marginal SD of centered BTL effects and does not equate the models' posteriors. Existing BTL identifiability and prior rules are unchanged. Upstream predictive workflows must establish uncertainty calibration using training data only; supplying SD explicitly does not establish calibration. Scalar SD supports sensitivity analyses, not essay-specific calibration. Saved metadata records the versioned mapping, SD source and scalar/per-item rule, and integrity digests; calibration is recorded as upstream, unverified.

Predictive BTL priors apply only in btl_only and both, including run-required linking Phase A. TrueSkill initialization applies in trueskill_only and both. Imported Phase-A artifacts retain their own generation identity and are not rerun because predictive input exists. Linking and pooled judge refits keep their existing prior rules; predictive evidence is not injected into Phase B priors, D-optimal selection, or probes. Custom BTL fit functions should consume state$predictive_prior only when state$meta$warm_start_mode is btl_only or both; its presence alone does not imply BTL warming. Resume preserves saved predictions, current TrueSkill state, mode, strategy, and bootstrap progress; omit all warm-start arguments.

Value

A list with:

state

Final adaptive_state.

summary

Run-level summary from summarize_adaptive().

refits

Per-refit summary from summarize_refits().

items

Item summary from summarize_items(), sorted by a usable canonical rank column (rank_link for linking runs when available, otherwise rank_raw).

logs

Canonical logs from adaptive_get_logs().

phase_a

Canonical wrapper-visible Phase A discovery surface with per-set status, artifact_dir, artifact_paths, and a reusable manifest that can be fed back into a later linking run via adaptive_config$phase_a_artifacts.

output_file

Saved output path when save_outputs = TRUE, otherwise NULL.

See Also

make_adaptive_judge_llm(), adaptive_rank_run_live(), adaptive_rank_start(), adaptive_rank_resume(), llm_compare_pair()

make_warm_start_prior(), fit_warm_start_model() and the integrated CJ workflow

Other adaptive ranking: adaptive_rank_resume(), adaptive_rank_run_live(), adaptive_rank_start(), make_adaptive_judge_llm(), make_adaptive_judge_replay(), make_adaptive_replay_reservoir(), summarize_adaptive(), validate_adaptive_replay()

Examples

data("example_writing_samples", package = "pairwiseLLM")

out <- adaptive_rank(
  data = example_writing_samples[1:8, c("ID", "text", "quality_score")],
  id_col = "ID",
  text_col = "text",
  model = "gpt-5.1",
  judge = function(A, B, state, ...) {
    y <- as.integer(A$quality_score[[1]] >= B$quality_score[[1]])
    list(is_valid = TRUE, Y = y, invalid_reason = NA_character_)
  },
  n_steps = 4,
  progress = "none"
)

out$summary
head(out$logs$step_log)

## Not run: 
# Live run with OpenAI gpt-5.1 + lower-cost Flex processing.
live <- adaptive_rank(
  data = example_writing_samples[1:12, c("ID", "text")],
  backend = "openai",
  model = "gpt-5.1",
  endpoint = "responses",
  judge_args = list(
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE
  ),
  btl_config = list(
    refit_pairs_target = 20L,
    ess_bulk_min = 500,
    eap_reliability_min = 0.92
  ),
  adaptive_config = list(
    explore_taper_mult = 0.40,
    star_override_budget_per_round = 2L
  ),
  n_steps = 120,
  session_dir = file.path(tempdir(), "adaptive-live"),
  persist_item_log = TRUE,
  resume = TRUE,
  progress = "all",
  save_outputs = TRUE
)

print(live$state)
live$summary

# Wrapper-driven linking workflow (hub + one spoke).
linking_samples <- example_writing_samples[1:12, c("ID", "text")]
linking_samples$set_id <- rep(c(1L, 2L), each = 6L)
linking_samples$global_item_id <- paste0("g_", linking_samples$ID)

link_out <- adaptive_rank(
  data = linking_samples,
  id_col = "ID",
  text_col = "text",
  backend = "openai",
  model = "gpt-5.1",
  adaptive_config = list(
    run_mode = "link_one_spoke",
    hub_id = 1L,
    phase_a_mode = "run",
    probe_panel_edges = 48L,
    hub_anchor_required_phase_b = TRUE,
    max_pairs_after_stop = 0L
  ),
  n_steps = 200,
  session_dir = file.path(tempdir(), "adaptive-link"),
  resume = TRUE,
  progress = "refits"
)

# Later linking from prior wrapper outputs:
# hub_run <- adaptive_rank(
#   data = linking_samples[linking_samples$set_id == 1L, c("ID", "text")],
#   backend = "openai",
#   model = "gpt-5.1",
#   n_steps = 120,
#   progress = "none"
# )
# spoke_run <- adaptive_rank(
#   data = linking_samples[linking_samples$set_id == 2L, c("ID", "text")],
#   backend = "openai",
#   model = "gpt-5.1",
#   n_steps = 120,
#   progress = "none"
# )
#
# link_out <- adaptive_rank(
#   data = linking_samples,
#   id_col = "ID",
#   text_col = "text",
#   backend = "openai",
#   model = "gpt-5.1",
#   adaptive_config = list(
#     run_mode = "link_one_spoke",
#     hub_id = 1L,
#     phase_a_mode = "import",
#     phase_a_artifacts = list(
#       `1` = hub_run$phase_a$manifest,
#       `2` = spoke_run$phase_a$artifact_dir
#     )
#   ),
#   n_steps = 200,
#   progress = "refits"
# )

names(link_out$logs)

## End(Not run)

Description

Resume a previously persisted adaptive pairing session.

Usage

adaptive_rank_resume(session_dir, ...)

Arguments

Details

This is a thin wrapper around load_adaptive_session() and performs schema and log-shape checks during load. Returned state preserves canonical step_log, round_log, and item_log contents used for adaptive auditability. The saved predictive mode, prior, pairing strategy, current TrueSkill state, bootstrap policy, and frozen queue are authoritative. Resume does not reload a predictive model, regenerate predictions, initialize ratings, or rebuild a predictive tree. SHA-256 validates the frozen predictive inputs and queue against saved session metadata. Legacy shuffled sessions retain their original behavior.

Value

An adaptive_state object restored from disk.

See Also

make_warm_start_prior(), adaptive_rank_start(), adaptive_rank_run_live(), save_adaptive_session(), load_adaptive_session()

Other adaptive ranking: adaptive_rank(), adaptive_rank_run_live(), adaptive_rank_start(), make_adaptive_judge_llm(), make_adaptive_judge_replay(), make_adaptive_replay_reservoir(), summarize_adaptive(), validate_adaptive_replay()

Examples

dir <- tempfile("pwllm-session-")
state <- adaptive_rank_start(c("a", "b", "c"), seed = 3)
save_adaptive_session(state, dir, overwrite = TRUE)
restored <- adaptive_rank_resume(dir)
summarize_adaptive(restored)

Description

Execute stepwise adaptive ranking with a user-supplied judge.

Usage

adaptive_rank_run_live(
  state,
  judge,
  n_steps = 1L,
  fit_fn = NULL,
  adaptive_config = NULL,
  btl_config = NULL,
  session_dir = NULL,
  persist_item_log = NULL,
  checkpoint_every_steps = NULL,
  progress = c("all", "refits", "steps", "none"),
  progress_redraw_every = 10L,
  progress_show_events = TRUE,
  progress_errors = TRUE,
  ...
)

Arguments

Details

Each iteration attempts at most one pair evaluation ("one-pair step"), then applies transactional updates if and only if the judge response is valid. Invalid responses produce a logged step with pair_id = NA and must not update committed-comparison state.

Within-set/Phase-A hybrid routing uses TrueSkill ranks, strata, rolling anchors, pair probabilities, and utility

$U_0 = p_{ij}(1 - p_{ij})$

. The long-link probability gate uses TrueSkill throughout within-set/Phase-A hybrid selection. Direct strategies apply their partner targets after the same connected bootstrap and currently require ordinary within-set mode. Adaptive Phase B selection is unavailable pending separate validation. Use prepare_link_input(), fit_link(), and start_link_session() with explicit cross-set evidence and an explicit E1–E3 estimator. Legacy Phase B posteriors cannot be resumed or migrated into a new estimator. Exploration/exploitation routing and fallback handling are recorded in step_log.

Hybrid round scheduling uses stage-specific admissibility:

• rolling-anchor links compare one anchor and one non-anchor endpoint;

• long/mid links exclude anchor endpoints and enforce stratum-distance bounds;

• local-link routing admits same-stratum pairs and anchor-involving pairs within local stage bounds.

Hybrid exposure and repeat handling are soft, stage-local constraints: under-represented exploration uses degree set deg <= D_min + 1, while repeat-pressure gating uses bottom-quantile recent_deg (default quantile 0.25) and per-endpoint repeat-slot accounting against repeat_in_round_budget.

Top-band defaults for stratum construction are top_band_pct = 0.10 and top_band_bins = 5, with top-band size ceiling(top_band_pct * N).

Bayesian BTL refits are triggered by committed comparisons and evaluated with diagnostics gates (including ESS thresholds), reliability, and lagged stability criteria. Refit-level outcomes are appended to round_log; per-item posterior summaries are appended to item_log. Controller behavior can change after refits via identifiability-gated settings in adaptive_config; those controls affect hybrid pair routing and quotas through the existing global_identified signal. BTL supplies item estimates, posterior uncertainty, EAP reliability, diagnostics, and stopping. Automatic Phase B selection is unavailable pending separate validation. If adaptive_config$max_pairs_after_stop > 0, the run records a stop boundary at the first refit with stop_decision = TRUE and allows at most that many additional committed comparisons before deterministic termination. Round logs record max_pairs_after_stop and pairs_committed_after_stop.

Value

An updated adaptive_state. The returned state includes appended step_log rows for attempted steps and, when refits occur, appended round_log and item_log entries.

Phase B linking restriction

Adaptive Phase B D-optimal selection is unavailable pending a separate selector validation study. This includes all E1–E3 engines and legacy D-optimal aliases; execution fails before selecting or judging a Phase B pair. Phase A and ordinary within-set ranking remain available. For linking, use prepare_link_input(), fit_link(), and start_link_session() with explicit cross-set evidence and an explicit estimator choice.

See Also

make_warm_start_prior(), adaptive_rank_start(), adaptive_rank_resume(), adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_start(), make_adaptive_judge_llm(), make_adaptive_judge_replay(), make_adaptive_replay_reservoir(), summarize_adaptive(), validate_adaptive_replay()

Examples

# ------------------------------------------------------------------
# Offline end-to-end workflow (fast, deterministic, CRAN-safe)
# ------------------------------------------------------------------
data("example_writing_samples", package = "pairwiseLLM")

items <- dplyr::rename(
  example_writing_samples[1:8, c("ID", "text", "quality_score")],
  item_id = ID
)

# Use the package defaults for trait and prompt template.
trait <- trait_description("overall_quality")
prompt_template <- set_prompt_template()

# Deterministic local judge based on fixture quality scores.
sim_judge <- function(A, B, state, ...) {
  y <- as.integer(A$quality_score[[1]] >= B$quality_score[[1]])
  list(is_valid = TRUE, Y = y, invalid_reason = NA_character_)
}

session_dir <- tempfile("pwllm-adaptive-session-")

state <- adaptive_rank_start(
  items = items,
  seed = 42,
  adaptive_config = list(
    global_identified_reliability_min = 0.85,
    star_override_budget_per_round = 2L
  ),
  session_dir = session_dir,
  persist_item_log = TRUE
)

state <- adaptive_rank_run_live(
  state = state,
  judge = sim_judge,
  n_steps = 6,
  btl_config = list(
    # Keep examples lightweight while showing custom stop config inputs.
    refit_pairs_target = 50L,
    ess_bulk_min = 400,
    eap_reliability_min = 0.90
  ),
  adaptive_config = list(
    explore_taper_mult = 0.40,
    boundary_frac = 0.20
  ),
  progress = "steps",
  progress_redraw_every = 1L,
  progress_show_events = TRUE,
  progress_errors = TRUE
)

# Print and inspect run outputs.
print(state)
run_summary <- summarize_adaptive(state)
step_view <- adaptive_step_log(state)
logs <- adaptive_get_logs(state)

run_summary
head(step_view)
names(logs)

# Resume from disk and continue.
resumed <- adaptive_rank_resume(session_dir)
resumed <- adaptive_rank_run_live(
  state = resumed,
  judge = sim_judge,
  n_steps = 4,
  progress = "none"
)
summarize_adaptive(resumed)

# ------------------------------------------------------------------
# Live OpenAI workflow via backend-agnostic llm_compare_pair()
# ------------------------------------------------------------------
## Not run: 
# Requires network + OPENAI_API_KEY. This incurs API cost.
# check_llm_api_keys() is a quick preflight.
check_llm_api_keys()

data("example_writing_samples", package = "pairwiseLLM")
live_items <- dplyr::rename(
  example_writing_samples[1:12, c("ID", "text")],
  item_id = ID
)

# Default trait/template setup used by the backend-agnostic runner.
trait <- trait_description("overall_quality")
prompt_template <- set_prompt_template()

live_session_dir <- file.path(tempdir(), "pwllm-adaptive-openai")

judge_openai <- function(A, B, state, ...) {
  res <- llm_compare_pair(
    ID1 = A$item_id[[1]],
    text1 = A$text[[1]],
    ID2 = B$item_id[[1]],
    text2 = B$text[[1]],
    model = "gpt-5.1",
    trait_name = trait$name,
    trait_description = trait$description,
    prompt_template = prompt_template,
    backend = "openai",
    endpoint = "responses",
    reasoning = "low",
    service_tier = "flex",
    include_thoughts = FALSE,
    temperature = NULL,
    top_p = NULL,
    logprobs = NULL
  )

  better_id <- res$better_id[[1]]
  ok_ids <- c(A$item_id[[1]], B$item_id[[1]])
  if (is.na(better_id) || !(better_id %in% ok_ids)) {
    return(list(
      is_valid = FALSE,
      Y = NA_integer_,
      invalid_reason = "model_response_invalid"
    ))
  }

  list(
    is_valid = TRUE,
    Y = as.integer(identical(better_id, A$item_id[[1]])),
    invalid_reason = NA_character_
  )
}

state_live <- adaptive_rank_start(
  items = live_items,
  seed = 2026,
  session_dir = live_session_dir,
  persist_item_log = TRUE
)

state_live <- adaptive_rank_run_live(
  state = state_live,
  judge = judge_openai,
  n_steps = 120L,
  btl_config = list(
    refit_pairs_target = 20L,
    ess_bulk_min = 500,
    ess_bulk_min_near_stop = 1200,
    max_rhat = 1.01,
    divergences_max = 0L,
    eap_reliability_min = 0.92,
    stability_lag = 2L,
    theta_corr_min = 0.97,
    theta_sd_rel_change_max = 0.08,
    rank_spearman_min = 0.97
  ),
  progress = "all",
  progress_redraw_every = 1L,
  progress_show_events = TRUE,
  progress_errors = TRUE
)

# Reporting outputs for end users.
print(state_live)
run_summary <- summarize_adaptive(state_live)
refit_summary <- summarize_refits(state_live)
item_summary <- summarize_items(state_live)
logs <- adaptive_get_logs(state_live)

# Store outputs for audit/reproducibility.
saveRDS(
  list(
    run_summary = run_summary,
    refit_summary = refit_summary,
    item_summary = item_summary,
    logs = logs
  ),
  file.path(live_session_dir, "adaptive_outputs.rds")
)

# Resume from stored state and continue sampling.
state_live <- adaptive_rank_resume(live_session_dir)
state_live <- adaptive_rank_run_live(
  state = state_live,
  judge = judge_openai,
  n_steps = 40L,
  progress = "refits"
)
print(summarize_adaptive(state_live))

## End(Not run)

Description

Initialize an adaptive ranking session and canonical state object.

Usage

adaptive_rank_start(
  items,
  seed = 1L,
  session_dir = NULL,
  persist_item_log = FALSE,
  ...,
  adaptive_config = NULL,
  checkpoint_every_steps = NULL,
  warm_start_model = NULL,
  warm_start_prior = NULL,
  warm_start_features = NULL,
  warm_start_python = NULL,
  warm_start_prior_sd = NULL,
  warm_start_mode = NULL,
  replay_reservoir = NULL,
  warm_start_trueskill = NULL,
  bootstrap_policy = "shuffled_connected"
)

Arguments

Details

This function creates the stepwise controller state and seeds all canonical logs used in the adaptive pairing workflow. The default connected bootstrap uses a seeded shuffled chain, or a shuffled allowed tree for replay reservoirs. Either graph policy connects all items after $N - 1$ committed comparisons.

Pair selection in this framework is stepwise and uncertainty-aware. Within-set/Phase-A hybrid routing uses TrueSkill ranks, strata, rolling anchors, pair probabilities, and base utility

$U_0 = p_{ij}(1 - p_{ij})$

where $p_{ij}$ is the current TrueSkill win probability for pair $\{i, j\}$. Linking Phase A preparation requires an explicit adaptive_config$link_estimation_mode. Adaptive Phase B selection remains unavailable pending separate validation. Use prepare_link_input(), fit_link(), and start_link_session() with explicit cross-set evidence and frozen shared judge parameters for E1–E3 linking. The within-set/Phase-A hybrid long-link gate uses TrueSkill throughout. Bayesian BTL supplies item estimates, posterior uncertainty, EAP reliability, diagnostics, stopping, and the existing global_identified signal. This signal can affect later hybrid tapering and routing; selection is not wholly independent of BTL. Direct within-set strategies use their documented partner targets after the common bootstrap. Automatic Phase B selection is unavailable pending separate validation. Linking Phase B refits use Bayesian posterior estimation and posterior summaries/diagnostics are logged per spoke at each linking refit.

The returned state contains canonical logs:

• step_log: one row per attempted step,

• round_log: one row per posterior refit,

• item_log: per-item posterior summaries by refit.

If session_dir is supplied, the initialized state is persisted immediately using save_adaptive_session().

Predictive destinations and initial connectivity are separate choices. By default, every warm mode retains the same seeded shuffled bootstrap of N - 1 valid comparisons. The explicit predictive graph policy uses a frozen allowed spanning tree with Pollitt probability targets and degree-cap relaxation. Both policies preserve invalid-result retries and recorded reservoir orientation. Later pairing retains the configured strategy. When expressing budgets as mean comparison exposures per item, each committed pair contributes two exposures. A prespecified rounding-down convention gives floor(B * N / 2) total pairs, including bootstrap, with realized exposure 2 * committed_pairs / N. B = 0 is prior-only; B = 0.5 and B = 1 precede connectivity and are not fully connected comparative-judgment estimates. The tree completes at N - 1 successful commits; comparison N uses the post-bootstrap strategy. Invalid attempts and retries do not add evidence. See vignette("adaptive-warm-start") for the five-arm synthetic count example. Without the distribution opt-in, BTL prior SD does not determine TrueSkill sigma. Ensemble disagreement never supplies SD automatically. No historical training-score units are restored.

Distribution initialization assumes the supplied BTL prior SD and TrueSkill uncertainty describe comparable latent scales under the documented affine convention. BTL priors apply to theta_raw; centering induces dependence, so this is not the marginal SD of centered BTL effects and does not equate the models' posteriors. Existing BTL identifiability and prior rules are unchanged. Upstream predictive workflows must establish uncertainty calibration using training data only; supplying SD explicitly does not establish calibration. Scalar SD supports sensitivity analyses, not essay-specific calibration. Saved metadata records the versioned mapping, SD source and scalar/per-item rule, and integrity digests; calibration is recorded as upstream, unverified.

Predictive BTL priors apply only in btl_only and both, including run-required linking Phase A. TrueSkill initialization applies in trueskill_only and both. Imported Phase-A artifacts retain their own generation identity and are not rerun because predictive input exists. Linking and pooled judge refits keep their existing prior rules; predictive evidence is not injected into Phase B priors, D-optimal selection, or probes. Custom BTL fit functions should consume state$predictive_prior only when state$meta$warm_start_mode is btl_only or both; its presence alone does not imply BTL warming. Resume preserves saved predictions, current TrueSkill state, mode, strategy, and bootstrap progress; omit all warm-start arguments.

Value

An adaptive state object containing step_log, round_log, and item_log. The object includes class "adaptive_state", item ID mappings, TrueSkill state, frozen bootstrap policy/queue, refit metadata, and runtime configuration.

Phase B linking restriction

Adaptive Phase B D-optimal selection is unavailable pending a separate selector validation study. This includes all E1–E3 engines and legacy D-optimal aliases; execution fails before selecting or judging a Phase B pair. Phase A and ordinary within-set ranking remain available. For linking, use prepare_link_input(), fit_link(), and start_link_session() with explicit cross-set evidence and an explicit estimator choice.

See Also

make_warm_start_prior(), adaptive_rank_run_live(), adaptive_rank_resume(), adaptive_step_log(), adaptive_round_log(), adaptive_item_log()

Other adaptive ranking: adaptive_rank(), adaptive_rank_resume(), adaptive_rank_run_live(), make_adaptive_judge_llm(), make_adaptive_judge_replay(), make_adaptive_replay_reservoir(), summarize_adaptive(), validate_adaptive_replay()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 11)
summarize_adaptive(state)
# Prior-only exposure checkpoint: B=0 commits no comparisons.
B <- c(0, 0.5, 1, 2)
data.frame(B = B, target_pairs = floor(B * length(state$item_ids) / 2))

Description

Adaptive results history ready for BT fitting

Usage

adaptive_results_history(state, committed_only = TRUE)

Arguments

Details

Converts adaptive step outcomes into the three-column format accepted by fit_bt_model() (object1, object2, result). Pass the result directly to fit_bt_model(); it is already in the output format of build_bt_data() and needs no further conversion. With committed_only = TRUE, only committed steps (pair_id not missing) are retained. This preserves the transactional invariant that invalid steps do not contribute to inferred comparisons.

Value

A tibble with columns:

object1

Character item id shown in position A.

object2

Character item id shown in position B.

result

Numeric outcome in {0, 1} where 1 means object1 wins.

See Also

fit_bt_model(), build_bt_data(), adaptive_step_log(), integrated CJ workflow

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_round_log(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_results_history(state)

Description

Adaptive round log accessor.

Usage

adaptive_round_log(state, reconstruct_deferred = FALSE)

Arguments

Details

round_log is the canonical per-refit audit log for the adaptive pairing workflow. Each row summarizes one Bayesian BTL refit and includes diagnostics, reliability, and stopping-gate fields used to justify stop decisions.

Core columns:

• Refit identity/state: refit_id, round_id_at_refit, step_id_at_refit, timestamp, model_variant, n_items, total_pairs_done, new_pairs_since_last_refit, n_unique_pairs_seen.

• Candidate health: proposed_pairs_mode, starve_rate_since_last_refit, fallback_rate_since_last_refit, fallback_used_mode, starvation_reason_mode.

• Identifiability/quota adaptation: global_identified, global_identified_reliability_min, global_identified_rank_corr_min, long_quota_raw, long_quota_effective, long_quota_removed, realloc_to_mid, realloc_to_local, phase_scope, phase_scope_set_id, phase_scope_n_items.

• Coverage/imbalance: mean_degree, min_degree, mean_degree_scope, min_degree_scope, pos_balance_sd, star_cap_rejects_since_last_refit, star_cap_reject_rate_since_last_refit, recent_deg_median_since_last_refit, recent_deg_max_since_last_refit.

• Posterior parameter summaries: epsilon_mean/percentiles and b_mean/percentiles.

• Audit diagnostics: ts_sigma_mean, ts_sigma_max, ts_degree_sigma_corr, ts_btl_theta_corr, ts_btl_rank_spearman, ci95_theta_width_*, near_tie_adj_frac, near_tie_adj_count, p_adj_median, cov_trace_theta, cov_logdet_diag_theta, post_sd_theta_p10, post_sd_theta_p50, post_sd_theta_p90, top20_boundary_entropy_*, nn_diff_sd_*.

• Stopping diagnostics: diagnostics_pass, diagnostics_divergences_pass, diagnostics_rhat_pass, diagnostics_ess_pass, divergences, divergences_max_allowed, max_rhat, max_rhat_allowed, min_ess_bulk, ess_bulk_required, near_stop_active, reliability_EAP, reliability_EAP_scope, eap_reliability_min, eap_pass, eap_pass_scope, theta_sd_eap, theta_sd_eap_scope, rho_theta, rho_theta_scope, lag_eligible, lag_eligible_scope, theta_corr_min, theta_corr_pass, theta_corr_pass_scope, delta_sd_theta, delta_sd_theta_scope, theta_sd_rel_change_max, delta_sd_theta_pass, delta_sd_theta_pass_scope, rho_rank, rho_rank_scope, rank_spearman_min, rho_rank_pass, rho_rank_pass_scope.

• Refit execution metadata: mcmc_chains, mcmc_parallel_chains, mcmc_core_fraction, mcmc_cores_detected_physical, mcmc_cores_detected_logical, mcmc_threads_per_chain, mcmc_cores_available, mcmc_parallel_chains_requested, mcmc_concurrency_budget, mcmc_concurrency_used, mcmc_cmdstanr_version. Requested parallel chains are NA for automatic scheduling; the budget and usage count CPU slots across chains and threads. Historical fits without allocation metadata report NA for those fields.

• Stop output: stop_decision, stop_reason, max_pairs_after_stop, pairs_committed_after_stop.

Value

A tibble with one row per completed posterior refit round.

See Also

adaptive_get_logs(), summarize_refits(), adaptive_rank_run_live()

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_results_history(), adaptive_step_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_round_log(state)

Description

Adaptive step log accessor.

Usage

adaptive_step_log(state)

Arguments

Details

step_log is the canonical per-step audit log for the adaptive workflow. It records candidate pipeline outcomes, selected pair/order, and commit status. A step with invalid judge response keeps committed fields as NA and must not update model state. The selected endpoints i/j are the pre-orientation item indices, while A/B are the displayed / judged item indices after order assignment. Y is defined relative to displayed order: Y = 1 means A wins and Y = 0 means B wins. For cross-run audit and reuse, prefer the stable *_id columns plus unordered_key/ordered_key rather than transient integer item positions from the live state. Judge provenance, token counts, and raw_response_json are canonical step-log outputs, with raw_response_json stored as serialized character data rather than a list-column.

Core columns:

• Identity/outcome: step_id, timestamp, pair_id, i, j, i_id, j_id, A, B, A_id, B_id, unordered_key, ordered_key, Y, status.

• Judge audit: judge_backend, judge_model, judge_endpoint, judge_valid, judge_invalid_reason, llm_status_code, llm_error_message, llm_custom_id, prompt_tokens, completion_tokens, total_tokens, raw_response_json.

• Routing/scheduling: round_id, round_stage, pair_type, pairing_strategy, stage_committed_so_far, stage_quota. Direct strategies use direct_pairing rather than hybrid stages; hybrid-only diagnostics are missing where inapplicable.

• Exposure/strata: used_in_round_i, used_in_round_j, is_anchor_i, is_anchor_j, stratum_i, stratum_j, dist_stratum.

• Candidate health: is_explore_step, explore_mode, explore_reason, explore_rate_used, local_priority_mode, long_gate_pass, long_gate_reason, star_override_used, star_override_reason, candidate_starved, fallback_used, fallback_path, starvation_reason.

• Candidate counts: n_candidates_generated, n_candidates_after_hard_filters, n_candidates_after_duplicates, n_candidates_after_star_caps, n_candidates_scored.

• Endpoint diagnostics: deg_i, deg_j, recent_deg_i, recent_deg_j, mu_i, mu_j, sigma_i, sigma_j, p_ij, U0_ij, target_distance. For direct strategies, i_id is the focal item and p_ij is the pre-judgment TrueSkill probability for presented A over B. target_distance is missing for random selection.

• Star-cap diagnostics: star_cap_rejects, star_cap_reject_items.

Value

A tibble with one row per attempted step, in execution order.

See Also

adaptive_get_logs(), adaptive_round_log(), adaptive_rank_run_live()

Other adaptive logs: adaptive_get_logs(), adaptive_item_log(), adaptive_results_history(), adaptive_round_log(), summarize_items(), summarize_refits()

Examples

state <- adaptive_rank_start(c("a", "b", "c"), seed = 1)
adaptive_step_log(state)

Description

This helper takes a table of paired writing samples (with columns ID1, text1, ID2, and text2) and reverses the sample order for every second row (rows 2, 4, 6, ...). This provides a perfectly balanced reversal pattern without the randomness of randomize_pair_order().

Usage

alternate_pair_order(pairs)

Arguments

Details

This is useful when you want a fixed 50/50 mix of original and reversed pairs for bias control, benchmarking, or debugging, without relying on the random number generator or seeds.

Value

A tibble identical to pairs except that rows 2, 4, 6, ... have ID1/text1 and ID2/text2 swapped.

See Also

read_samples_df(), read_samples_dir()

Other pairing and data: make_pairs(), randomize_pair_order(), read_samples_df(), read_samples_dir(), sample_pairs(), sample_reverse_pairs()

Examples

data("example_writing_samples")
pairs <- make_pairs(example_writing_samples)

pairs_alt <- alternate_pair_order(pairs)

head(pairs[, c("ID1", "ID2")])
head(pairs_alt[, c("ID1", "ID2")])

Description

This function sends a single pairwise comparison prompt to the Anthropic Messages API (Claude models) and parses the result into a small tibble.

Usage

anthropic_compare_pair_live(
  ID1,
  text1,
  ID2,
  text2,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  tag_prefix = "<BETTER_SAMPLE>",
  tag_suffix = "</BETTER_SAMPLE>",
  api_key = NULL,
  anthropic_version = "2023-06-01",
  reasoning = c("none", "enabled"),
  include_raw = FALSE,
  include_thoughts = NULL,
  ...
)

Arguments

Details

It mirrors the behaviour and output schema of openai_compare_pair_live, but targets Anthropic's /v1/messages endpoint. The prompt template, ⁠<BETTER_SAMPLE>⁠ tag convention, and downstream parsing / BT modelling can remain unchanged.

The function is designed to work with Claude models such as Sonnet, Haiku, and Opus in the "4.5" family. You can pass any valid Anthropic model string, for example:

• "claude-sonnet-4-5"

• "claude-haiku-4-5"

• "claude-opus-4-5"

The API typically responds with a dated model string such as "claude-sonnet-4-5-20250929" in the model field.

Recommended defaults for pairwise writing comparisons

For stable, reproducible comparisons we recommend:

• reasoning = "none" with model-default sampling and max_tokens = 768 for standard pairwise scoring.

• reasoning = "enabled" when you explicitly want extended thinking; in this mode Anthropic requires temperature = 1. The default in this function is max_tokens = 2048 and thinking_budget_tokens = 1024, which satisfies the documented constraints thinking_budget_tokens >= 1024 and thinking_budget_tokens < max_tokens.

When reasoning = "enabled", this function also sends a thinking block to the Anthropic API:

"thinking": {
  "type": "enabled",
  "budget_tokens": <thinking_budget_tokens>
}

Setting include_thoughts = TRUE when reasoning = "none" is a convenient way to opt into Anthropic's extended thinking mode without changing the reasoning argument explicitly. In that case, reasoning is upgraded to "enabled", the default temperature becomes 1, and a thinking block is included in the request. When reasoning = "none" and include_thoughts is FALSE or NULL, omitted sampling parameters use the model/provider defaults.

Value

A tibble with one row and columns:

custom_id

Stable ID for the pair (pair_uid if supplied via ...; otherwise "LIVE_<ID1>_vs_<ID2>").

ID1, ID2

The sample IDs you supplied.

model

Model name reported by the API.

object_type

Anthropic object type (for example "message").

status_code

HTTP-style status code (200 if successful).

error_message

Error message if something goes wrong; otherwise NA.

thoughts

Summarised thinking / reasoning text when reasoning = "enabled" and the API returns thinking blocks; otherwise NA.

content

Concatenated text from the assistant output (excluding thinking blocks).

better_sample

"SAMPLE_1", "SAMPLE_2", or NA.

better_id

ID1 if SAMPLE_1 is chosen, ID2 if SAMPLE_2 is chosen, otherwise NA.

prompt_tokens

Prompt / input token count (if reported).

completion_tokens

Completion / output token count (if reported).

total_tokens

Total token count (reported by the API or computed as input + output tokens when not provided).

raw_response

(Optional) list-column containing the parsed JSON body.

See Also

check_llm_api_keys(), llm_compare_pair()

Other live backends: check_llm_api_keys(), gemini_compare_pair_live(), llm_compare_pair(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_typesafe_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), typesafe_compare_pair_live(), vertex_compare_pair_live()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")
samples <- example_writing_samples[1:2, ]

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Standard comparison with model-default sampling and no thinking block
res_claude <- anthropic_compare_pair_live(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "none",
  include_raw       = FALSE
)

res_claude$better_id

# Allow more internal thinking and a longer explanation
res_claude_reason <- anthropic_compare_pair_live(
  ID1               = samples$ID[1],
  text1             = samples$text[1],
  ID2               = samples$ID[2],
  text2             = samples$text[2],
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "enabled",
  include_raw       = TRUE,
  include_thoughts  = TRUE
)

res_claude_reason$total_tokens
substr(res_claude_reason$content, 1, 200)

## End(Not run)

Description

This is a thin wrapper around Anthropic's /v1/messages/batches endpoint. It accepts a list of request objects (each with custom_id and params) and returns the resulting Message Batch object.

Usage

anthropic_create_batch(
  requests,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Details

Typically you will not call this directly; instead, use run_anthropic_batch_pipeline which builds requests from a tibble of pairs, creates the batch, polls for completion, and downloads the results.

Value

A list representing the Message Batch object returned by Anthropic. Important fields include id, processing_status, request_counts, and (after completion) results_url.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
library(pairwiseLLM)

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 2, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

req_tbl <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl
)

requests <- lapply(seq_len(nrow(req_tbl)), function(i) {
  list(
    custom_id = req_tbl$custom_id[i],
    params    = req_tbl$params[[i]]
  )
})

batch <- anthropic_create_batch(requests = requests)
batch$id
batch$processing_status

## End(Not run)

Description

Once a Message Batch has finished processing (status "ended"), Anthropic exposes a results_url field pointing to a .jsonl file containing one JSON object per request result.

Usage

anthropic_download_batch_results(
  batch_id,
  output_path,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Details

This helper downloads that file and writes it to disk. It is the Anthropic counterpart to openai_download_batch_output().

Value

Invisibly, the output_path.

Retrieval retries

Batch metadata and result-file GET requests retry HTTP 408, 429, all 5xx responses, and transport failures, with at most three total HTTP attempts per GET. Valid Retry-After seconds or HTTP dates take precedence; otherwise retries use exponential backoff starting at 0.5 seconds plus up to 0.25 seconds of jitter, capped at 30 seconds. Other HTTP errors fail immediately. Exhaustion raises the original error with the additional class pairwiseLLM_batch_retry_exhausted. These retries retrieve the same batch; they do not resubmit comparisons or create scientific failed-attempt rows.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
final <- anthropic_poll_batch_until_complete(batch$id)
jsonl_path <- tempfile(fileext = ".jsonl")
anthropic_download_batch_results(final$id, jsonl_path)

## End(Not run)

Description

This retrieves the latest state of a Message Batch using its id. It corresponds to a GET request on /v1/messages/batches/<MESSAGE_BATCH_ID>.

Usage

anthropic_get_batch(
  batch_id,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01"
)

Arguments

Value

A list representing the Message Batch object, including fields such as id, processing_status, request_counts, and (after completion) results_url.

Retrieval retries

Batch metadata and result-file GET requests retry HTTP 408, 429, all 5xx responses, and transport failures, with at most three total HTTP attempts per GET. Valid Retry-After seconds or HTTP dates take precedence; otherwise retries use exponential backoff starting at 0.5 seconds plus up to 0.25 seconds of jitter, capped at 30 seconds. Other HTTP errors fail immediately. Exhaustion raises the original error with the additional class pairwiseLLM_batch_retry_exhausted. These retries retrieve the same batch; they do not resubmit comparisons or create scientific failed-attempt rows.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
# After creating a batch:
batch <- anthropic_create_batch(requests = my_requests)
batch_id <- batch$id

latest <- anthropic_get_batch(batch_id)
latest$processing_status

## End(Not run)

Description

This helper repeatedly calls anthropic_get_batch until the batch's processing_status becomes "ended" or a time limit is reached. It is analogous to openai_poll_batch_until_complete() but for Anthropic's Message Batches API.

Usage

anthropic_poll_batch_until_complete(
  batch_id,
  interval_seconds = 60,
  timeout_seconds = 86400,
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  anthropic_version = "2023-06-01",
  verbose = TRUE
)

Arguments

Value

The final Message Batch object as returned by anthropic_get_batch once processing_status == "ended" or the last object retrieved before timing out.

Polling limits

HTTP retries occur within a logical status poll. Elapsed time includes retry waits, but timeout_seconds is checked between status requests; an in-flight GET and its retries can finish after that limit. Existing terminal-status and timeout return behavior is preserved.

Retrieval retries

Batch metadata and result-file GET requests retry HTTP 408, 429, all 5xx responses, and transport failures, with at most three total HTTP attempts per GET. Valid Retry-After seconds or HTTP dates take precedence; otherwise retries use exponential backoff starting at 0.5 seconds plus up to 0.25 seconds of jitter, capped at 30 seconds. Other HTTP errors fail immediately. Exhaustion raises the original error with the additional class pairwiseLLM_batch_retry_exhausted. These retries retrieve the same batch; they do not resubmit comparisons or create scientific failed-attempt rows.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), build_anthropic_batch_requests(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

## Not run: 
# Requires ANTHROPIC_API_KEY and network access.
batch <- anthropic_create_batch(requests = my_requests)
final <- anthropic_poll_batch_until_complete(batch$id, interval_seconds = 30)
final$processing_status

## End(Not run)

Description

Simulate binary comparisons from fitted BT scores, refit the same estimator, and estimate itemwise bias. Adaptive runs repeat the actual package scheduling algorithm, including changes driven by simulated earlier outcomes.

Usage

bootstrap_bt_model(
  object,
  mode,
  n_rep,
  seed,
  schedule = NULL,
  initial_state = NULL,
  budget = NULL,
  estimator = NULL,
  estimator_args = list(),
  btl_config = NULL,
  schedule_fit_fn = NULL,
  min_success = n_rep,
  workers = 1L,
  keep = c("summary", "theta", "full"),
  executor = NULL
)

Arguments

Details

Each outcome has probability plogis(theta[A] - theta[B]). Initial scores and refits are aligned by ID and centered to sum zero. With successful refits only, bias = mean(theta_boot) - theta_initial and theta_corrected = theta_initial - bias. Corrections can change rankings and need not improve every item or every application.

Adaptive p50, Pollitt-inspired, hybrid and random strategies use the canonical live runner with a synthetic judge. Initialization stays frozen, while later selector randomness changes by replicate. Scheduled refits, TrueSkill updates, hybrid identifiability/tapering and constraints remain active. The existing max_pairs_after_stop continuation control is set to budget so statistical stopping cannot truncate the requested fixed budget. Terminal exhaustion is a failed replicate; it never triggers replacement with another strategy.

RNG algorithm is Mersenne-Twister, with Inversion normals and Rejection sampling. For replicate b and stage s (selector, outcomes, scheduling refits, final estimator, numbered 1–4), the integer seed is max(1, floor((seed * 1000003 + b * 10007 + s * 101 + 304) %% 2147483647)). Scheduling-refit seeds use the canonical stage derivation again, with the scheduling seed, committed comparison count, stage 1, and offset zero. Serial and parallel results use the same seeds and aggregation order. Exact reproducibility requires compatible R/package/engine versions and deterministic callbacks. There is no checkpoint/resume API; restarting repeats the run. Built-in final estimators reuse immutable item contrasts within this call. Fixed schedules also reuse pair indexes and designs; alpha retains its all-pairs pseudo-count design, including unobserved pairs. Outcome counts, estimation and numerical validity checks are recomputed for every replicate. Worker chunks hold at most eight replicates for summary/theta retention and two for full retention. Designs are constructed inside each chunk, never transferred from the parent or retained in a global cache. An executor is an execution resource and is not stored in scientific provenance. Its transport errors propagate without replacement replicates or a serial fallback.

bootstrap_sd describes replicate-estimate dispersion, whereas mcse_bias is bootstrap_sd / sqrt(n_success): simulation error in the estimated bias. Neither is automatically a standard error for the bias-corrected estimator. This function does not construct confidence intervals or corrected SSR.

An unmet min_success raises pairwiseLLM_bt_bootstrap_error with a result field containing diagnostics and missing bias/corrected scores. Tolerated failures raise a warning and remain in the diagnostic table. Numerical warnings from successful fits are recorded per replicate and signaled once at completion.

Value

A pairwiseLLM_bt_bootstrap list containing theta (ID, initial theta, bootstrap mean, bias, corrected theta, bootstrap SD and Monte Carlo error), n_success, n_failed, status, replicates (seeds, failures and schedule diagnostics), failures (available error diagnostics), optional draws and artifacts, and provenance. Provenance includes normalized generating parameters, schedule/initial state, estimator/settings, master seed, RNG convention, scheduling configuration, callbacks and software versions.

See Also

fit_bt_model(), adaptive_rank_start(), adaptive_rank_run_live() and the integrated CJ workflow

Other frequentist models: build_bt_data(), build_elo_data(), fit_bt_model(), fit_elo_model(), predict.pairwiseLLM_bt_alpha(), predict.pairwiseLLM_bt_firth(), predict.pairwiseLLM_bt_lapse(), scale_separation_reliability(), summarize_bt_fit()

Examples

dat <- data.frame(object1 = rep("a", 8), object2 = rep("b", 8),
                  result = c(rep(1, 6), rep(0, 2)))
fit <- fit_bt_model(dat, engine = "alpha", alpha = 1, verbose = FALSE)
boot <- bootstrap_bt_model(fit, mode = "fixed", n_rep = 20, seed = 304)
boot$theta

Description

This helper converts a tibble of writing pairs into a list of Anthropic Message Batch requests. Each request has a unique custom_id of the form "ANTH_<ID1>_vs_<ID2>" and a params object compatible with the /v1/messages API.

Usage

build_anthropic_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  reasoning = c("none", "enabled"),
  custom_id_prefix = "ANTH",
  ...
)

Arguments

Details

The function mirrors the behaviour of build_openai_batch_requests but targets Anthropic's /v1/messages/batches endpoint. It applies the same recommended defaults and reasoning constraints as anthropic_compare_pair_live:

• reasoning = "none":

  • Omitted temperature and top_p values are not sent, so the model/provider defaults apply.

  • Default max_tokens = 768, unless overridden via max_tokens in ....

• reasoning = "enabled" (extended thinking):

  • temperature must be 1. If you supply a different value in ..., this function throws an error.

  • Defaults to max_tokens = 2048 and thinking_budget_tokens = 1024, with the constraint 1024 <= thinking_budget_tokens < max_tokens. Violations of this constraint produce an error.

As a result, batches without extended thinking (reasoning = "none") use model-default sampling. When you opt into extended thinking (reasoning = "enabled"), Anthropic's requirement of temperature = 1 is enforced for all batch requests.

Value

A tibble with one row per pair and two main columns:

custom_id

Character ID of the form "<PREFIX>_<ID1>_vs_<ID2>".

params

List-column containing the Anthropic Messages API params object for each request, ready to be used in the requests array of /v1/messages/batches.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_gemini_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Standard batch requests without extended thinking
reqs_none <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "none"
)

reqs_none

# Batch requests with extended thinking
reqs_reason <- build_anthropic_batch_requests(
  pairs             = pairs,
  model             = "claude-sonnet-4-5",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  reasoning         = "enabled"
)

reqs_reason

Description

This function converts pairwise comparison results into the three-column format commonly used for Bradley-Terry models: the first two columns contain object labels and the third column contains the comparison result (1 for a win of the first object, 0 for a win of the second).

Usage

build_bt_data(results)

Arguments

Details

It accepts either:

• legacy columns ID1, ID2, better_id, or

• canonical columns A_id, B_id, better_id.

Rows where better_id does not match either side of the pair (including NA) are excluded.

Value

A tibble with three columns:

• object1: ID from ID1

• object2: ID from ID2

• result: numeric value, 1 if better_id == ID1, 0 if better_id == ID2

Rows with invalid or missing better_id are dropped.

See Also

fit_bt_model(), summarize_bt_fit()

Other frequentist models: bootstrap_bt_model(), build_elo_data(), fit_bt_model(), fit_elo_model(), predict.pairwiseLLM_bt_alpha(), predict.pairwiseLLM_bt_firth(), predict.pairwiseLLM_bt_lapse(), scale_separation_reliability(), summarize_bt_fit()

Examples

results <- tibble::tibble(
  ID1       = c("S1", "S1", "S2"),
  ID2       = c("S2", "S3", "S3"),
  better_id = c("S1", "S3", "S2")
)

bt_data <- build_bt_data(results)
bt_data

# Using the example writing pairs
data("example_writing_pairs")
bt_ex <- build_bt_data(example_writing_pairs)
head(bt_ex)

Description

Converts non-adaptive pairwise outcomes (for example, rows like example_writing_pairs with ID1, ID2, better_id) into the canonical results_tbl schema required by fit_bayes_btl_mcmc().

Usage

build_btl_results_data(
  results,
  phase = "phase2",
  backend = "non_adaptive_import",
  model = "unknown",
  iter_start = 1L,
  received_at_start = as.POSIXct("1970-01-01 00:00:00", tz = "UTC")
)

Arguments

Details

The output is deterministic and schema-valid:

• stable unordered_key / ordered_key values,

• deterministic pair_uid as "<unordered_key>#<occurrence>",

• deterministic iter and received_at sequences.

Value

A tibble in canonical results_tbl format with columns: pair_uid, unordered_key, ordered_key, A_id, B_id, better_id, winner_pos, phase, iter, received_at, backend, model.

See Also

fit_bayes_btl_mcmc()

Other Bayesian models: fit_bayes_btl_mcmc()

Examples

data("example_writing_pairs", package = "pairwiseLLM")

results_tbl <- build_btl_results_data(example_writing_pairs)
head(results_tbl)

ids <- sort(unique(c(results_tbl$A_id, results_tbl$B_id)))
ids

Description

This function converts pairwise comparison results into the two-column format used by the EloChoice package: one column for the winner and one for the loser of each trial.

Usage

build_elo_data(results)

Arguments

Details

It accepts either:

• legacy columns ID1, ID2, better_id, or

• canonical columns A_id, B_id, better_id.

Rows where better_id does not match either side of the pair (including NA) are excluded.

Value

A tibble with two columns:

• winner: ID of the winning sample

• loser: ID of the losing sample

Rows with invalid or missing better_id are dropped.

See Also

build_bt_data(), fit_bt_model()

Other frequentist models: bootstrap_bt_model(), build_bt_data(), fit_bt_model(), fit_elo_model(), predict.pairwiseLLM_bt_alpha(), predict.pairwiseLLM_bt_firth(), predict.pairwiseLLM_bt_lapse(), scale_separation_reliability(), summarize_bt_fit()

Examples

results <- tibble::tibble(
  ID1       = c("S1", "S1", "S2", "S3"),
  ID2       = c("S2", "S3", "S3", "S4"),
  better_id = c("S1", "S3", "S2", "S4")
)

elo_data <- build_elo_data(results)
elo_data

Description

This helper converts a tibble of writing pairs into a set of Gemini GenerateContent requests suitable for use with the Batch API (models/*:batchGenerateContent).

Usage

build_gemini_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  thinking_level = "low",
  custom_id_prefix = "GEM",
  temperature = NULL,
  top_p = NULL,
  top_k = NULL,
  max_output_tokens = NULL,
  service_tier = "standard",
  include_thoughts = FALSE,
  ...
)

Arguments

Details

Each pair receives a unique custom_id of the form "GEM_<ID1>_vs_<ID2>" and a corresponding request object containing the prompt and generation configuration.

Value

A tibble with one row per pair and two main columns:

custom_id

Character ID of the form "<PREFIX>_<ID1>_vs_<ID2>".

request

List-column containing the Gemini GenerateContent request object for each pair.

See Also

llm_submit_pairs_batch(), llm_download_batch_results()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_openai_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# Dated tested Gemini Developer API configuration
reqs <- build_gemini_batch_requests(
  pairs             = pairs,
  model             = "gemini-3.5-flash-lite",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "low",
  include_thoughts  = TRUE
)

reqs

# Gemini 3 Flash example (minimal thinking)
reqs_flash <- build_gemini_batch_requests(
  pairs             = pairs,
  model             = "gemini-3-flash-preview",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  thinking_level    = "minimal",
  include_thoughts  = FALSE
)

reqs_flash

Description

This helper constructs one JSON object per pair of writing samples, suitable for use with the OpenAI batch API. It supports both /v1/chat/completions and /v1/responses endpoints.

Usage

build_openai_batch_requests(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  endpoint = c("chat.completions", "responses"),
  temperature = NULL,
  top_p = NULL,
  logprobs = NULL,
  reasoning = NULL,
  include_thoughts = FALSE,
  request_id_prefix = "EXP",
  store = NULL,
  max_output_tokens = NULL,
  prompt_caching = NULL,
  ...
)

Arguments

Details

Invalid controls fail even for empty pairs. Responses are stored for later API retrieval by default; set store = FALSE to disable response storage. For Chat Completions, store controls storage for distillation or evaluations. Omitting it preserves that endpoint's provider default. This parameter does not control Batch input/output/error file retention and does not imply zero data retention. See https://developers.openai.com/api/docs/guides/your-data.

Value

A tibble with one row per pair and columns:

• custom_id: ID string used by the batch API.

• method: HTTP method ("POST").

• url: Endpoint path ("/v1/chat/completions" or "/v1/responses").

• body: List column containing the request body.

Batch prompt caching

The default disables prompt caching for these verified model IDs: gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol, gpt-6-luna, gpt-6-sol, gpt-6-astra, and gpt-6.1-sol. Each request body gets prompt_cache_options = list(mode = "explicit"), with no breakpoints, on both endpoints. Set prompt_caching = "implicit" to omit that field and retain provider caching when prefix reuse is expected to be worthwhile.

Recognized earlier OpenAI models retain their historical request bodies when the policy is omitted or NULL; explicitly disabling caching on those models errors. Unknown IDs (including unlisted snapshots, future models and custom aliases) require an explicit "implicit" opt-in or a verified model ID. Support is an offline allowlist; version numbers and snapshot suffixes never establish support for disabling caching. Older GPT-3.5/4, GPT-5 through GPT-5.5 and o-series naming forms, including recognized legacy date-suffixed names, retain provider defaults.

Invalid policies and manual cache controls fail before submission, even with zero pairs. Raw prompt_cache_options, prompt_cache_breakpoint, prompt_cache_key, retention and TTL controls are not accepted by these builders. Keys affect cache routing/accounting; retention and TTL affect lifetime; store = FALSE affects response storage. None substitutes for disabling implicit cache breakpoints. Prompt text and keys are not rewritten. This policy affects future package-built Batch requests only, not live requests or already submitted jobs. See the prompt-caching guide and vignette("advanced-batch-workflows") for usage-based cost comparisons.

See Also

llm_submit_pairs_batch(), llm_resume_multi_batches()

Other batch backends: anthropic_create_batch(), anthropic_download_batch_results(), anthropic_get_batch(), anthropic_poll_batch_until_complete(), build_anthropic_batch_requests(), build_gemini_batch_requests(), gemini_create_batch(), gemini_download_batch_results(), gemini_get_batch(), gemini_poll_batch_until_complete(), llm_download_batch_results(), llm_resume_multi_batches(), llm_submit_pairs_batch(), llm_submit_pairs_multi_batch(), openai_create_batch(), openai_download_batch_errors(), openai_download_batch_output(), openai_get_batch(), openai_poll_batch_until_complete(), openai_upload_batch_file(), run_anthropic_batch_pipeline(), run_gemini_batch_pipeline(), run_openai_batch_pipeline(), write_openai_batch_file()

Examples

data("example_writing_samples", package = "pairwiseLLM")

pairs <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 3, seed = 123) |>
  randomize_pair_order(seed = 456)

td <- trait_description("overall_quality")
tmpl <- set_prompt_template()

# 1. Basic chat.completions batch with no thoughts
batch_tbl_chat <- build_openai_batch_requests(
  pairs             = pairs,
  model             = "gpt-4.1",
  trait_name        = td$name,
  trait_description = td$description,
  prompt_template   = tmpl,
  endpoint          = "chat.completions"
)

# 2. GPT-5.6 Sol Responses Batch with Reasoning
batch_tbl_resp <- build_openai_batch_requests(
  pairs = pairs,
  model = "gpt-5.6-sol",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  endpoint = "responses",
  include_thoughts = TRUE, # implies reasoning="low" if not set
  reasoning = "medium",
  store = FALSE,
  max_output_tokens = 1024
)

batch_tbl_chat
batch_tbl_resp

Description

This function takes a prompt template (typically from set_prompt_template), a trait name and description, and two writing samples, and fills in the required placeholders.

Usage

build_prompt(template, trait_name, trait_desc, text1, text2)

Arguments

Details

The template must contain the placeholders: {TRAIT_NAME}, {TRAIT_DESCRIPTION}, {SAMPLE_1}, and {SAMPLE_2}.

Value

A single character string containing the completed prompt.

See Also

trait_description(), set_prompt_template()

Other prompts and traits: get_prompt_template(), list_prompt_templates(), register_prompt_template(), remove_prompt_template(), set_prompt_template(), trait_description()

Examples

tmpl <- set_prompt_template()
td <- trait_description("overall_quality")
prompt <- build_prompt(
  template   = tmpl,
  trait_name = td$name,
  trait_desc = td$description,
  text1      = "This is sample 1.",
  text2      = "This is sample 2."
)
cat(substr(prompt, 1, 200), "...\n")

Description

This function inspects the current R session for configured API keys used by pairwiseLLM. It checks for known environment variables such as OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, VERTEX_API_KEY, TOGETHER_API_KEY, and TYPESAFE_API_KEY, and returns a small tibble summarising which keys are available.

Usage

check_llm_api_keys(verbose = TRUE)

Arguments

Details

It does not print or return the key values themselves - only whether each key is present. This makes it safe to run in logs, scripts, and shared environments.

Value

A tibble (data frame) with one row per backend and columns:

backend

Short backend identifier, e.g. "openai", "anthropic", "gemini", "vertex", "together", "typesafe".

service

Human-readable service name, e.g. "OpenAI", "Anthropic", "Google Gemini", "Vertex AI Gemini API", "Together.ai", "TypeSafe".

env_var

Name of the environment variable that is checked.

has_key

Logical flag indicating whether the key is set and non-empty.

See Also

llm_compare_pair(), submit_llm_pairs(), typesafe_compare_pair_live()

Other live backends: anthropic_compare_pair_live(), gemini_compare_pair_live(), llm_compare_pair(), ollama_compare_pair_live(), openai_compare_pair_live(), submit_anthropic_pairs_live(), submit_gemini_pairs_live(), submit_llm_pairs(), submit_ollama_pairs_live(), submit_openai_pairs_live(), submit_together_pairs_live(), submit_typesafe_pairs_live(), submit_vertex_pairs_live(), together_compare_pair_live(), typesafe_compare_pair_live(), vertex_compare_pair_live()

Examples

# In an interactive session, quickly check which keys are configured:
check_llm_api_keys()

# In non-interactive scripts, you can disable messages and just use the
# result:
status <- check_llm_api_keys(verbose = FALSE)
status

Description

This function diagnoses positional preference in LLM-based paired comparison data and provides a bootstrapped confidence interval for the overall agreement of forward vs. reverse comparisons.

Usage

check_positional_bias(
  consistency,
  n_boot = 1000,
  conf_level = 0.95,
  seed = NULL
)

Arguments

Details

It is designed to work with the output of compute_reverse_consistency, but will also accept a tibble that looks like its $details component.

Each row of details is one unordered pair after any duplicate judgments have been reduced to a per-direction majority by compute_reverse_consistency(). The agreement estimate and its percentile bootstrap interval therefore use unordered pairs as the unit of analysis and treat those rows as independently resampled units. This assumption may be inappropriate when pairs share items.

The direction-specific binomial tests likewise treat unordered-pair outcomes within a direction as independent. The overall test is paired: among inconsistent pairs, it compares the number for which position 1 won both presentations with the number for which position 2 won both. This is the exact conditional form of McNemar's test. It returns NA when there are no informative inconsistent pairs. A large p-value is not evidence that positional preference is absent.

Value

A list with two elements:

summary

A tibble with:

• n_pairs: number of unordered pairs

• prop_consistent: observed proportion of consistent pairs

• boot_mean: mean of bootstrap consistency proportions

• boot_lwr, boot_upr: bootstrap confidence interval

• p_sample1_main: p-value from a binomial test for the null hypothesis that SAMPLE_1 wins 50\ main (forward) comparisons

• p_sample1_rev: analogous p-value for the reverse comparisons

• p_sample1_overall: p-value from the paired exact test that position-1 and position-2 inconsistencies are equally likely

• total_pos1_wins: total number of wins by position 1 across forward + reverse comparisons

• total_comparisons: total number of valid forward + reverse comparisons included in the overall test

• n_inconsistent: number of pairs with inconsistent forward vs. reverse outcomes

• n_inconsistent_pos1_bias: among inconsistent pairs, how many times the winner is in position 1 in both directions

• n_inconsistent_pos2_bias: analogous for position 2

details

The input details tibble augmented with:

• winner_pos_main: "pos1" or "pos2" (or NA) indicating which position won in the main direction

• winner_pos_rev: analogous for the reversed direction

• is_pos1_bias: logical; TRUE if the pair is inconsistent and position 1 wins in both directions

• is_pos2_bias: analogous for position 2

See Also

compute_reverse_consistency()

Other bias and consistency: compute_reverse_consistency()

Examples

# Simple synthetic example
main <- tibble::tibble(
  ID1       = c("S1", "S1", "S2"),
  ID2       = c("S2", "S3", "S3"),
  better_id = c("S1", "S3", "S2")
)

rev <- tibble::tibble(
  ID1       = c("S2", "S3", "S3"),
  ID2       = c("S1", "S1", "S2"),
  better_id = c("S1", "S3", "S2")
)

rc <- compute_reverse_consistency(main, rev)
rc$summary

bias <- check_positional_bias(rc)
bias$summary

Description

Given two data frames of pairwise comparison results (one for the "forward" ordering of pairs, one for the "reverse" ordering), this function identifies unordered pairs that were evaluated in both directions and computes the proportion of consistent judgments.

Usage

compute_reverse_consistency(main_results, reverse_results)

Arguments

Details

Consistency is defined at the level of IDs: a pair is consistent if the same ID is selected as better in both directions. This function assumes each input contains columns ID1, ID2, and better_id, where better_id is the ID of the better sample (not "SAMPLE_1"/"SAMPLE_2").

Per-key majority agreement (duplicates supported). If a pair appears multiple times in main_results and/or reverse_results (e.g., submitted twice), this function aggregates each unordered pair key separately in each direction and takes the majority better_id. If there is a tie for the majority winner within a direction, that direction's majority winner is set to NA and the key is excluded from the consistency calculation.

The output details contains exactly one row per unordered pair key, which keeps it compatible with check_positional_bias. Columns other than ID1, ID2, and better_id are not grouping variables. Split results by model, template, trait, reasoning or thinking condition, and any intended replicate unit before calling this function; otherwise their judgments are pooled in the per-key majority.

Value

A list with two elements:

• summary: a tibble with one row and columns n_pairs, n_consistent, and prop_consistent. Here, n_pairs counts unordered pair keys with a non-missing majority winner in both directions.

• details: a tibble with one row per unordered pair key, including columns key, ID1_main, ID2_main, ID1_rev, ID2_rev, better_id_main, better_id_rev, and is_consistent. Additional columns provide vote counts and tie flags.

See Also

check_positional_bias()

Other bias and consistency: check_positional_bias()

Examples

main <- tibble::tibble(
  ID1       = c("A", "A", "X"),
  ID2       = c("B", "B", "Y"),
  better_id = c("A", "B", "X")  # duplicate A-B with disagreement
)
rev <- tibble::tibble(
  ID1       = c("B"),
  ID2       = c("A"),
  better_id = c("A")
)
compute_reverse_consistency(main, rev)$summary

Description

Collect and resume OpenAI Decisions and TypeSafe Jev comparisons through submit_llm_pairs(), submit_openai_pairs_live() or submit_typesafe_pairs_live(). Each pair occurrence is one live request; these endpoints do not use provider-managed Batch jobs.

Execution controls

The bulk wrappers accept execution_control, a uniquely named list. Unknown names, invalid scalars and unsupported provider options fail before HTTP. parallel = FALSE or workers = 1 selects sequential execution. ⁠parallel = TRUE, workers > 1⁠ uses optional promises and later through httr2's experimental public asynchronous API. Missing packages cause an informative error before requests; no packages are installed automatically. Workers bound simultaneous in-flight requests, not R processes. One private event loop and curl pool belong to each session and are cleaned up on exit. The coordinator explicitly implements retries and admission; asynchronous httr2 retry/throttle policies are not assumed to apply.

Supported controls and defaults are:

requests_per_minute = NULL

Optional positive integer maximum transmission reservations in a rolling 60-second window.

tokens_per_minute = NULL

Optional positive integer maximum estimated token reservations in that window. Requires token_estimates.

token_estimates = NULL

Positive integer scalar or vector in input row order, including an output allowance. A reservation must fit the token limit. No tokenizer or missing provider usage is guessed. This bounds reservations, not actual usage when estimates are too small.

max_attempts = 3L

Persistent total send-reservation budget per occurrence, including initial, transient and ambiguous-recovery sends. It is fixed in the saved manifest; restarting does not replenish it.

timeout_seconds = 120

Positive finite timeout per transmission.

retry_ambiguous = TRUE

Allow at most one automatic reissue of an interrupted occurrence across all restarts, also within max_attempts. FALSE leaves ambiguous work unresolved. Toggling it never replenishes the consumed allowance.

lock_timeout_ms = 10000

Nonnegative finite milliseconds to wait for an exclusive saved-run lock before failing without provider calls.

Rate limits are caller configuration, not assumed provider quotas. Recent reservations, retry deadlines and shared provider cooldowns survive restart. Limits apply to this saved run; other sessions sharing an account can consume the same provider quota. Limits and timeout may change on resume without changing scientific identity; prior reservations remain counted.

Retries and ambiguous requests

Bulk OpenAI retries HTTP 408, 409, 429 and all 5xx statuses; TypeSafe retries 408, 429 and all 5xx, including 529. Recognized OpenAI insufficient-quota or billing-limit errors are terminal. Timeouts receive bounded transient retries; other transport failures and permanent HTTP errors remain unresolved. Exponential delays start at 0.5 seconds, double up to 30 seconds and add uniform jitter of up to 0.25 seconds. Valid Retry-After seconds or HTTP dates are a minimum delay, with shared cooldowns for 429, 503 and 529. Invalid headers use backoff. Single-pair retry defaults are unchanged.

A send reservation is committed before transmission. An interruption before transmission may therefore conservatively consume an attempt. Reservations without a committed response have an unknown provider outcome. Recovery first checks local committed evidence. The currently supported endpoint contracts provide no documented remote result lookup/idempotency mechanism: reconciliation is recorded as unsupported, without speculative retrieval requests. One automatic ambiguous reissue is permitted by default, across all restarts. Exhaustion leaves the occurrence in failed_pairs for explicit operator action; an operator can deliberately start a separate run containing only unresolved source pairs. Preserve the old journal rather than editing it or resetting budgets. Timeout retries can also incur duplicate charges. Neither locking nor checkpoints establish exactly-once provider execution.

Malformed/refused decisions, incomplete probabilities, invalid confidence and choice/probability inconsistencies are recorded without automatic retry and never become completed evidence. Only normalized valid results enter modeling.

Request identity

All four input ID/text columns must be nonmissing character columns. Supplied pair_uid values must be unique and nonempty. Otherwise deterministic scientific-fingerprint IDs plus occurrence suffixes distinguish repeats and reversed presentations. Supply explicit IDs when scheduled occurrences need independent labels. IDs are never matched by row position alone.

The saved workset must have the same occurrence membership, scientific payload and provider wire controls. Matching occurrences can be reordered. Adding or removing occurrences, or changing texts, orientation, trait, instructions, choices, model/provider/endpoint or OpenAI cache controls requires a separate save path. Omitted decision prompts select the exact decision_cj_v1 wording; custom instructions follow decision_request. Generative templates fail.

Every current request is rebuilt and validated even when its result is already saved. The shared scientific validator checks body, ordered evidence hashes, approved question core, provenance and fingerprint. Serialization and attached raw UTF-8 bytes are checked before every transmission, including retries. scientific_payload_sha256 hashes the serialized scientific body; wire_body_sha256 hashes the exact transmitted JSON body, including supported cache controls. These digests exclude authentication headers and timeout. Digests detect identity/integrity mismatches, not malicious modification with recomputed checksums. Requested and provider-reported resolved models remain separate. Moving aliases may change weights; saved results retain the originally reported resolution, and no availability or alias-resolution calls are made.

Persistence and output

A decision save_path names a CSV export and requires optional filelock. The dedicated ⁠<save_path>.lock⁠ is locked before checkpoints are read and held through all asynchronous calls, retries and writes. The OS lock releases on exit/process termination; its file is retained. Only one writer is admitted.

⁠<save_path>.checkpoint/⁠ is the authoritative lossless RDS journal. Its immutable checksummed manifest and sequenced records are written through same-directory temporary files, flushed/closed, then renamed to new final names. Each response is validated and committed immediately, independently of other in-flight work. Completed committed occurrences are never automatically submitted again. Uncommitted temporary files are ignored; corrupt committed files, missing interior records and unsupported schemas stop execution before HTTP. CSV alone cannot establish completion and is not imported as a resume journal.

CSV contains valid results with stable scalar columns, including all probability and provenance columns. Numeric values use 17 significant digits to round-trip R doubles with a correctly rounded decimal reader. Some readers (including base numeric conversion on macOS ARM in qualification) can change the last bit. For exact numeric CSV recovery, read columns as character and decode probability cells with jsonlite::parse_json(); use RDS for exact native recovery. Read IDs as character to preserve leading zeros. A missing or torn CSV export is rebuilt from RDS on resume. Journal envelopes are RDS lists with a SHA-256 and serialized raw data; unserialize(readRDS(record)$data) exposes the versioned event and its state/attempt records. Treat these as trusted local files, not a public cross-version interchange format.

The returned list retains results, failed_pairs and failed_attempts. Results preserve decision_results fields, exact parsed R double probabilities, full probabilities_json, independent confidence, usage and model provenance. Missing usage/resolution stays missing; no price or cost estimate is fabricated. Additional diagnostics include payload digests, controls, local attempt_id, attempt_count, ambiguous_reissues, possible_duplicate_charge, elapsed_seconds since admission, and recovery_status. Failed attempts retain unique local IDs, reason (initial, transient_retry, ambiguous_recovery), timestamps, available original/new provider request IDs, provenance, and reconciliation status. Provider IDs unknown at interruption remain missing. Successful outcomes and failed attempts remain separate; history is not re-added on every restart. Probabilities never become BT/adaptive weights, fractional wins or extra comparisons.

No durability is promised without save_path. Saved-run guarantees concern process interruption on a filesystem supporting OS locks and atomic rename; they are not power-loss/fsync guarantees. Network/cloud-synchronized storage and cross-machine locking require filesystem guarantees beyond this package.

Privacy

Samples go to the selected provider. Normal messages and scalar checkpoints omit API keys and sample text. Provenance retains complete custom instructions and trait definitions, which can themselves be sensitive. CSV excludes input text columns and arbitrary caller metadata. include_raw = TRUE explicitly retains parsed provider responses in RDS and may contain sensitive content; FALSE omits them from new persisted records, including failures. Existing opted-in records are not erased. Increasing raw retention on resume cannot recreate responses previously omitted and never triggers resubmission. In-memory failure objects may contain raw responses even with include_raw = FALSE. Explicitly saving or serializing the complete returned object (for example with saveRDS()) can persist sensitive text. Successful normalized results can also retain source text columns. The default journal/export privacy policy does not sanitize caller-created stores. Keep journals and returned diagnostics private.

See Also

submit_llm_pairs(), decision_request, decision_results

Examples

## Not run: 
# Requires provider credentials; incurs usage charges.
pairs <- data.frame(ID1 = c("A", "B"), ID2 = c("B", "A"),
                    text1 = c("Synthetic A.", "Synthetic B."),
                    text2 = c("Synthetic B.", "Synthetic A."),
                    pair_uid = c("forward", "reverse"))
result <- submit_llm_pairs(
  pairs, model = "jev-1.13.0", backend = "typesafe",
  trait_name = "Organization", trait_description = "Logical sequencing.",
  parallel = TRUE, workers = 4, save_path = "decisions.csv",
  execution_control = list(requests_per_minute = 60,
                           retry_ambiguous = FALSE)
)
# Repeat the same call to resume; preserve its .checkpoint directory.
result$failed_pairs
result$failed_attempts

## End(Not run)

Description

Unlike ensemble_warm_start_models(), this constructor requires auditable shared training outcomes and evaluation partitions. Validation averages the untouched outer-held-out calibrated predictions, not full-fit predictions or component validation metrics.

Usage

ensemble_warm_start_algorithms(...)

Arguments

Details

Components must share the exact task, ordered IDs, original outcomes, schema, CV plan and outer observed values/folds. Every component requires learned oof_linear calibration. Legacy and reduced models cannot establish the necessary audit evidence and must not be used to construct a new ensemble. Deployment averages each component's calibrated prediction with equal weight. There are no learned weights, automatic component selection or ensemble-level recalibration. Validation calibration intercept/slope are diagnostics only. Component names identify members; distinct engine names are not required.

prepare_warm_start_model() can reduce an existing ensemble to summary-only audit status. Its portable component payloads and identity survive, but the stored validation summaries cannot then be recomputed. Between-component SD is diagnostic disagreement, never a Bayesian prior SD.

Value

A standalone pairwiseLLM_warm_algorithm_ensemble, format 1, with artifact_type = "algorithm_ensemble", ordered components, common schema, outcome and CV identity, and honest nested-CV ensemble validation.

See Also

make_warm_start_cv_plan(), make_warm_start_prior(), predict.pairwiseLLM_warm_algorithm_ensemble()

Other adaptive warm start: ensemble_warm_start_models(), extract_warm_start_features(), fit_warm_start_model(), make_warm_start_cv_plan(), make_warm_start_prior(), pairwiseLLM_warm_model, predict.pairwiseLLM_warm_algorithm_ensemble(), predict.pairwiseLLM_warm_ensemble(), predict.pairwiseLLM_warm_model(), prepare_warm_start_model(), register_warm_start_model(), save_warm_start_model(), summary.pairwiseLLM_warm_algorithm_ensemble(), summary.pairwiseLLM_warm_ensemble(), summary.pairwiseLLM_warm_predictions(), warm_start_coefficients(), warm_start_feature_schema(), warm_start_python_status()

Examples

if (requireNamespace("glmnet", quietly = TRUE) &&
    requireNamespace("pls", quietly = TRUE) && requireNamespace("withr", quietly = TRUE)) {
  local({
    withr::local_seed(259L)
    fields <- warm_start_feature_schema()$feature
    x <- as.data.frame(matrix(runif(20 * length(fields)), nrow = 20))
    names(x) <- fields
    x$n_tokens <- 11:30
    x <- data.frame(item_id = as.character(1:20), x)
    attr(x, "warm_start_schema") <- "writing_features_v1"
    theta <- x$n_tokens - 3 * x$token_length_mean
    plan <- make_warm_start_cv_plan(x$item_id, theta, "synthetic", seed = 259L)
    en <- fit_warm_start_model(x$item_id, theta, "synthetic", features = x,
      cv_plan = plan, alpha_grid = c(0, 1))
    pls <- fit_warm_start_model(x$item_id, theta, "synthetic", features = x,
      cv_plan = plan, engine = "pls", engine_control = list(ncomp = 1L))
    ensemble <- ensemble_warm_start_algorithms(elastic_net = en, pls = pls)
    print(ensemble)
    summary(ensemble)$validation$metrics
    predictions <- predict(ensemble, x)
    make_warm_start_prior(predictions, prior_sd = 0.5)
  })
}

Description

An ensemble averages calibrated standardized predictions from separate task models. Each task standardizes its own outcome, so original BT/BTL scales need not be linked. Callers are responsible for independently sourced training data; task labels cannot establish independence.

Usage

ensemble_warm_start_models(...)

Arguments

Details

Components must have learned OOF calibration and compatible frozen schemas. Legacy model formats 1/2 and full or summary-only format-3 models can be mixed. Each component keeps its own preprocessing, calibration, and training metadata. References are loaded once; prediction uses stored components. Character references always mean paths, never implicit registry names. Registry names retain existing normalization and source ambiguity rules; component names do not use registry normalization. Custom or learned weights are not supported.

Repeated models and task labels remain accepted; neither establishes independent training data. Validation reports component metrics only. Use ensemble_warm_start_algorithms() for aligned validation of algorithms trained on one assessment.

prepare_warm_start_model() can add ensemble metadata or explicitly reduce all component audits. Ordinary save/load is lossless. Ensemble format 1 is independent of component formats and remains 1 after reduction.

Value

A portable pairwiseLLM_warm_ensemble list with ensemble format version 1, named components, common schema, original features, standardized outcome definition/sample-SD convention, and weighting = "equal".

See Also

fit_warm_start_model(), predict.pairwiseLLM_warm_ensemble(), ensemble_warm_start_algorithms(), warm_start_coefficients(), summary.pairwiseLLM_warm_ensemble()

Other adaptive warm start: ensemble_warm_start_algorithms(), extract_warm_start_features(), fit_warm_start_model(), make_warm_start_cv_plan(), make_warm_start_prior(), pairwiseLLM_warm_model, predict.pairwiseLLM_warm_algorithm_ensemble(), predict.pairwiseLLM_warm_ensemble(), predict.pairwiseLLM_warm_model(), prepare_warm_start_model(), register_warm_start_model(), save_warm_start_model(), summary.pairwiseLLM_warm_algorithm_ensemble(), summary.pairwiseLLM_warm_ensemble(), summary.pairwiseLLM_warm_predictions(), warm_start_coefficients(), warm_start_feature_schema(), warm_start_python_status()

Examples

if (requireNamespace("glmnet", quietly = TRUE) &&
    requireNamespace("withr", quietly = TRUE)) {
  local({
    # Synthetic features illustrate the interface, not predictive validity.
    example_features <- function(seed) {
      withr::local_seed(seed)
      fields <- warm_start_feature_schema()$feature
      x <- as.data.frame(matrix(runif(15 * length(fields)), nrow = 15))
      names(x) <- fields
      x$n_tokens <- 11:25
      x$token_length_mean <- 2 + 10 * x$token_length_mean
      x$token_length_std <- 0.2 + x$token_length_std
      x$dale_chall_readability_score <- 5 + 20 * x$dale_chall_readability_score
      x <- data.frame(item_id = as.character(1:15), x)
      attr(x, "warm_start_schema") <- "writing_features_v1"
      x
    }
    features <- example_features(3103)
    theta <- 10 + 0.4 * features$n_tokens - 2 * features$token_length_mean
    # A small alpha grid keeps this example fast; the default has 41 values.
    model <- fit_warm_start_model(features$item_id, theta, "synthetic-a",
      features = features, alpha_grid = c(0, 1))
    features_b <- example_features(3104)
    theta_b <- 30 + features_b$n_tokens - 3 * features_b$token_length_mean
    model_b <- fit_warm_start_model(features_b$item_id, theta_b, "synthetic-b",
      features = features_b, alpha_grid = c(0, 1))
    ensemble <- ensemble_warm_start_models(assessment_a = model, assessment_b = model_b)
    print(ensemble)
  })
}

Description

ensure_only_ollama_model_loaded() is a small convenience helper for managing memory when working with large local models via Ollama. It inspects the current set of active models using the ⁠ollama ps⁠ command and attempts to unload any models that are not the one you specify.

Usage

ensure_only_ollama_model_loaded(model, verbose = TRUE)

Arguments

Details

This can be useful when running multiple large models (for example "mistral-small3.2:24b", "qwen3:32b", "gemma3:27b") on a single machine, where keeping all of them loaded simultaneously may exhaust GPU or system memory.

The function is intentionally conservative:

• If the ollama command is not available on the system or ⁠ollama ps⁠ returns an error or empty output, no action is taken and a message is printed when verbose = TRUE.

• If no active models are reported, no action is taken.

• Only models with names different from model are passed to ollama stop <name>.

This helper is not called automatically by the package; it is intended to be used programmatically in development scripts and ad hoc workflows before running comparisons with ollama_compare_pair_live() or submit_ollama_pairs_live().

This function relies on the ollama command-line interface being available on the system PATH. If the command cannot be executed or returns a non-zero status code, the function will issue a message (when verbose = TRUE) and return without making any changes.

The exact output format of ollama ps is treated as an implementation detail: this helper assumes that the first non-empty line is a header and that subsequent non-empty lines begin with the model name as the first whitespace-separated field. If the format changes in a future version of Ollama, parsing may fail and the function will simply fall back to doing nothing.

Because ⁠ollama stop⁠ affects the global Ollama server state for the current machine, you should only use this helper in environments where you are comfortable unloading models that might be in use by other processes.

Value

Invisibly returns a character vector containing the names of models that were requested to be unloaded (i.e., those passed to ollama stop). If no models were unloaded, an empty character vector is returned.

See Also

• ollama_compare_pair_live() for single-pair Ollama comparisons.

• submit_ollama_pairs_live() for row-wise Ollama comparisons across many pairs.

Examples

## Not run: 
# Keep only mistral-small3.2:24b loaded in Ollama, unloading any
# other active models
ensure_only_ollama_model_loaded("mistral-small3.2:24b")

## End(Not run)

Description

Estimate total token usage and cost for running a large set of pairwise comparisons by:

• running a small pilot on n_test pairs (live calls) to observe prompt_tokens and completion_tokens, and

• using the pilot to calibrate a prompt-bytes-to-input-token model for the remaining pairs, and

• prorating output tokens for the remaining pairs from the pilot distribution.

Usage

estimate_llm_pairs_cost(
  pairs,
  model,
  trait_name,
  trait_description,
  prompt_template = set_prompt_template(),
  backend = c("openai", "anthropic", "gemini", "vertex", "together"),
  endpoint = c("chat.completions", "responses"),
  mode = c("live", "batch"),
  n_test = 25,
  test_strategy = c("stratified_prompt_bytes", "random", "first"),
  seed = NULL,
  cost_per_million_input,
  cost_per_million_output,
  batch_discount = 1,
  budget_quantile = 0.9,
  return_test_results = TRUE,
  return_remaining_pairs = TRUE,
  ...
)

Arguments