Package: pairwiseLLM Title: Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation Version: 1.6.0 Authors@R: person("Sterett H.", "Mercer", , "sterett.mercer@ubc.ca", role = c("aut", "cre", "cph"), comment = c(ORCID = "0000-0002-7940-4221")) Description: Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', 'TypeSafe', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. The package additionally supports adaptive pairing workflows that iteratively select comparisons based on model uncertainty to improve ranking efficiency. Results can be modeled using frequentist or Bayesian Bradley–Terry–Luce models (Bradley & Terry, 1952 ; see also Caron & Doucet, 2012 ) or Elo rating methods (see Clark et al., 2018 ) to derive writing quality scores. The Bayesian writing quality scores can be converted to distribution-matched performance levels or calibrated to ordered rubric scores using linear or monotone ordinal models. For information on pairwise comparisons and comparative judgement, see Thurstone (1927) and Heldsinger & Humphry (2010) . License: MIT + file LICENSE Encoding: UTF-8 Roxygen: list(markdown = TRUE) Imports: cli, curl, dplyr, httr2 (>= 1.1.0), jsonlite, openssl, rlang (>= 1.3.0), stats, tibble, tidyselect, tools, utils Suggests: BradleyTerry2, brglm2, cmdstanr, e1071, EloChoice, filelock, future, future.apply, glmnet, knitr, later, mgcv (>= 1.9-4), mockery, ordinal, parallelly, pkgload, pls, posterior, promises, purrr, readr, rmarkdown, reticulate (>= 1.41), sirt, stringr, testthat (>= 3.0.0), tidyr, vctrs, withr (>= 3.0.3) Additional_repositories: https://stan-dev.r-universe.dev/ SystemRequirements: CmdStan and a C++17 toolchain (optional; needed only for Bayesian and adaptive modeling); Python 3.12.3 with spaCy, TextDescriptives, textstat and English resources (optional; needed only for warm-start text feature extraction) Config/testthat/edition: 3 URL: https://github.com/shmercer/pairwiseLLM, https://shmercer.github.io/pairwiseLLM/ BugReports: https://github.com/shmercer/pairwiseLLM/issues Depends: R (>= 4.4) VignetteBuilder: knitr Collate: 'adaptive_benchmark_metrics.R' 'adaptive_bootstrap.R' 'adaptive_btl_refit.R' 'adaptive_candidates.R' 'adaptive_constraints.R' 'adaptive_linking_calibration.R' 'adaptive_linking_phase_a.R' 'adaptive_logs.R' 'adaptive_pairing_strategy.R' 'adaptive_persist.R' 'adaptive_predictive_tree.R' 'adaptive_print.R' 'adaptive_rank.R' 'adaptive_replay.R' 'adaptive_replay_reservoir.R' 'adaptive_replay_selection.R' 'adaptive_round_candidates.R' 'adaptive_run.R' 'adaptive_schemas.R' 'adaptive_select.R' 'adaptive_simulation_harness.R' 'adaptive_starvation.R' 'adaptive_state.R' 'adaptive_step.R' 'adaptive_trueskill.R' 'adaptive_utility.R' 'anthropic_batch_api.R' 'anthropic_live.R' 'api_keys.R' 'batch_retrieval.R' 'bayes_btl_mcmc.R' 'bayes_btl_mcmc_adaptive.R' 'bayes_btl_summarize.R' 'bt_alpha.R' 'bt_bootstrap.R' 'bt_bootstrap_contracts.R' 'bt_bootstrap_schedule.R' 'bt_common.R' 'bt_firth.R' 'bt_helpers.R' 'bt_lapse.R' 'bt_lapse_likelihood.R' 'bt_lapse_optimizer.R' 'bt_model.R' 'bt_provenance.R' 'btl_evidence_identity.R' 'btl_mcmc_constraints.R' 'btl_mcmc_contracts.R' 'btl_mcmc_fit_contracts.R' 'btl_mcmc_ingest.R' 'btl_mcmc_model_variant.R' 'btl_mcmc_resources.R' 'btl_mcmc_state.R' 'btl_mcmc_stopping.R' 'btl_mcmc_summaries.R' 'btl_mcmc_theta_summary_stubs.R' 'core_budget.R' 'cost_estimator.R' 'custom_id.R' 'data-example_writing.R' 'data_import.R' 'decision_bulk.R' 'decision_checkpoint.R' 'decision_live.R' 'decision_prompts.R' 'decision_results.R' 'draws_sanitize.R' 'elo_model.R' 'gemini_batch_api.R' 'gemini_live.R' 'gemini_params.R' 'httr2_retry.R' 'link_contract.R' 'link_coordinates.R' 'link_fixed_shape.R' 'link_gaussian_bridge.R' 'link_joint_mcmc.R' 'link_joint_offset.R' 'link_legacy.R' 'link_likelihood.R' 'link_orchestration.R' 'link_phase_a_draws.R' 'link_phase_a_evidence.R' 'link_phase_a_points.R' 'link_probe_quality.R' 'link_reporting.R' 'link_result.R' 'link_session.R' 'llm_backends.R' 'llm_batch.R' 'llm_multi_batch.R' 'normalize_results.R' 'ollama_live.R' 'openai_batch_api.R' 'openai_batch_cache.R' 'openai_decisions.R' 'openai_live.R' 'openai_params.R' 'pairing.R' 'pairwiseLLM-package.R' 'prompt_template.R' 'reverse_consistency.R' 'rubric_calibration.R' 'rubric_cj_contract.R' 'rubric_cj_linked.R' 'rubric_diagnostics.R' 'rubric_linked_anchors.R' 'rubric_metrics.R' 'rubric_ordinal_linear.R' 'rubric_ordinal_monotone.R' 'rubric_percentile.R' 'rubric_reference.R' 'rubric_validation.R' 'seed_helpers.R' 'together_live.R' 'traits.R' 'typesafe_live.R' 'utils-null-coalesce.R' 'vertex_live.R' 'warm_start_adaptive.R' 'warm_start_algorithm_ensemble.R' 'warm_start_bundle.R' 'warm_start_calibration.R' 'warm_start_candidate_validity.R' 'warm_start_coefficients.R' 'warm_start_cv.R' 'warm_start_cv_plan.R' 'warm_start_engine.R' 'warm_start_ensemble.R' 'warm_start_feature_schema.R' 'warm_start_features.R' 'warm_start_format3.R' 'warm_start_model.R' 'warm_start_model_artifact.R' 'warm_start_model_io.R' 'warm_start_model_registry.R' 'warm_start_pls.R' 'warm_start_pls_validation.R' 'warm_start_predict.R' 'warm_start_predictions.R' 'warm_start_preprocess.R' 'warm_start_prior.R' 'warm_start_python.R' 'warm_start_svr.R' 'warm_start_svr_validation.R' 'warm_start_tune.R' 'warm_start_validation.R' Config/roxygen2/version: 8.1.0 Config/pak/sysreqs: libssl-dev python3 Repository: https://shmercer.r-universe.dev Date/Publication: 2026-10-19 19:15:35 UTC RemoteUrl: https://github.com/shmercer/pairwisellm RemoteRef: HEAD RemoteSha: 3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5 NeedsCompilation: no Packaged: 2026-10-19 20:25:47 UTC; root Author: Sterett H. Mercer [aut, cre, cph] (ORCID: ) Maintainer: Sterett H. Mercer