{ "_id": "6ad685230d12c08e591b7952", "Package": "pairwiseLLM", "Title": "Pairwise Comparison Tools for Large Language Model-Based Writing\nEvaluation", "Version": "1.6.0", "Authors@R": "person(\"Sterett H.\", \"Mercer\", , \"sterett.mercer@ubc.ca\", role = c(\"aut\", \"cre\", \"cph\"),\ncomment = c(ORCID = \"0000-0002-7940-4221\"))", "Description": "Provides a unified framework for generating, submitting,\nand analyzing pairwise comparisons of writing quality using\nlarge language models (LLMs). The package supports live and/or\nbatch evaluation workflows across multiple providers ('OpenAI',\n'Anthropic', 'Google Gemini', 'Together AI', 'TypeSafe', and\nlocally-hosted 'Ollama' models), includes bias-tested prompt\ntemplates and a flexible template registry, and offers tools\nfor constructing forward and reversed comparison sets to\nanalyze consistency and positional bias. The package\nadditionally supports adaptive pairing workflows that\niteratively select comparisons based on model uncertainty to\nimprove ranking efficiency. Results can be modeled using\nfrequentist or Bayesian Bradley–Terry–Luce models (Bradley &\nTerry, 1952 ; see also Caron & Doucet,\n2012 ) or Elo rating methods\n(see Clark et al., 2018 ) to\nderive writing quality scores. The Bayesian writing quality\nscores can be converted to distribution-matched performance\nlevels or calibrated to ordered rubric scores using linear or\nmonotone ordinal models. For information on pairwise\ncomparisons and comparative judgement, see Thurstone (1927)\n and Heldsinger & Humphry (2010)\n.", "License": "MIT + file LICENSE", "Encoding": "UTF-8", "Roxygen": "list(markdown = TRUE)", "Additional_repositories": "https://stan-dev.r-universe.dev/", "SystemRequirements": "CmdStan and a C++17 toolchain (optional; needed\nonly for Bayesian and adaptive modeling); Python 3.12.3 with\nspaCy, TextDescriptives, textstat and English resources\n(optional; needed only for warm-start text feature extraction)", "Config/testthat/edition": "3", "URL": "https://github.com/shmercer/pairwiseLLM,\nhttps://shmercer.github.io/pairwiseLLM/", "BugReports": "https://github.com/shmercer/pairwiseLLM/issues", "VignetteBuilder": "knitr", "Collate": "'adaptive_benchmark_metrics.R' 'adaptive_bootstrap.R'\n'adaptive_btl_refit.R' 'adaptive_candidates.R'\n'adaptive_constraints.R' 'adaptive_linking_calibration.R'\n'adaptive_linking_phase_a.R' 'adaptive_logs.R'\n'adaptive_pairing_strategy.R' 'adaptive_persist.R'\n'adaptive_predictive_tree.R' 'adaptive_print.R'\n'adaptive_rank.R' 'adaptive_replay.R'\n'adaptive_replay_reservoir.R' 'adaptive_replay_selection.R'\n'adaptive_round_candidates.R' 'adaptive_run.R'\n'adaptive_schemas.R' 'adaptive_select.R'\n'adaptive_simulation_harness.R' 'adaptive_starvation.R'\n'adaptive_state.R' 'adaptive_step.R' 'adaptive_trueskill.R'\n'adaptive_utility.R' 'anthropic_batch_api.R' 'anthropic_live.R'\n'api_keys.R' 'batch_retrieval.R' 'bayes_btl_mcmc.R'\n'bayes_btl_mcmc_adaptive.R' 'bayes_btl_summarize.R'\n'bt_alpha.R' 'bt_bootstrap.R' 'bt_bootstrap_contracts.R'\n'bt_bootstrap_schedule.R' 'bt_common.R' 'bt_firth.R'\n'bt_helpers.R' 'bt_lapse.R' 'bt_lapse_likelihood.R'\n'bt_lapse_optimizer.R' 'bt_model.R' 'bt_provenance.R'\n'btl_evidence_identity.R' 'btl_mcmc_constraints.R'\n'btl_mcmc_contracts.R' 'btl_mcmc_fit_contracts.R'\n'btl_mcmc_ingest.R' 'btl_mcmc_model_variant.R'\n'btl_mcmc_resources.R' 'btl_mcmc_state.R' 'btl_mcmc_stopping.R'\n'btl_mcmc_summaries.R' 'btl_mcmc_theta_summary_stubs.R'\n'core_budget.R' 'cost_estimator.R' 'custom_id.R'\n'data-example_writing.R' 'data_import.R' 'decision_bulk.R'\n'decision_checkpoint.R' 'decision_live.R' 'decision_prompts.R'\n'decision_results.R' 'draws_sanitize.R' 'elo_model.R'\n'gemini_batch_api.R' 'gemini_live.R' 'gemini_params.R'\n'httr2_retry.R' 'link_contract.R' 'link_coordinates.R'\n'link_fixed_shape.R' 'link_gaussian_bridge.R'\n'link_joint_mcmc.R' 'link_joint_offset.R' 'link_legacy.R'\n'link_likelihood.R' 'link_orchestration.R'\n'link_phase_a_draws.R' 'link_phase_a_evidence.R'\n'link_phase_a_points.R' 'link_probe_quality.R'\n'link_reporting.R' 'link_result.R' 'link_session.R'\n'llm_backends.R' 'llm_batch.R' 'llm_multi_batch.R'\n'normalize_results.R' 'ollama_live.R' 'openai_batch_api.R'\n'openai_batch_cache.R' 'openai_decisions.R' 'openai_live.R'\n'openai_params.R' 'pairing.R' 'pairwiseLLM-package.R'\n'prompt_template.R' 'reverse_consistency.R'\n'rubric_calibration.R' 'rubric_cj_contract.R'\n'rubric_cj_linked.R' 'rubric_diagnostics.R'\n'rubric_linked_anchors.R' 'rubric_metrics.R'\n'rubric_ordinal_linear.R' 'rubric_ordinal_monotone.R'\n'rubric_percentile.R' 'rubric_reference.R'\n'rubric_validation.R' 'seed_helpers.R' 'together_live.R'\n'traits.R' 'typesafe_live.R' 'utils-null-coalesce.R'\n'vertex_live.R' 'warm_start_adaptive.R'\n'warm_start_algorithm_ensemble.R' 'warm_start_bundle.R'\n'warm_start_calibration.R' 'warm_start_candidate_validity.R'\n'warm_start_coefficients.R' 'warm_start_cv.R'\n'warm_start_cv_plan.R' 'warm_start_engine.R'\n'warm_start_ensemble.R' 'warm_start_feature_schema.R'\n'warm_start_features.R' 'warm_start_format3.R'\n'warm_start_model.R' 'warm_start_model_artifact.R'\n'warm_start_model_io.R' 'warm_start_model_registry.R'\n'warm_start_pls.R' 'warm_start_pls_validation.R'\n'warm_start_predict.R' 'warm_start_predictions.R'\n'warm_start_preprocess.R' 'warm_start_prior.R'\n'warm_start_python.R' 'warm_start_svr.R'\n'warm_start_svr_validation.R' 'warm_start_tune.R'\n'warm_start_validation.R'", "Config/roxygen2/version": "8.1.0", "Config/pak/sysreqs": "libssl-dev python3", "Repository": "https://shmercer.r-universe.dev", "Date/Publication": "2026-10-19 19:15:35 UTC", "RemoteUrl": "https://github.com/shmercer/pairwisellm", "RemoteRef": "HEAD", "RemoteSha": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "NeedsCompilation": "no", "Packaged": { "Date": "2026-10-19 20:25:47 UTC", "User": "root" }, "Author": "Sterett H. Mercer [aut, cre, cph] (ORCID:\n)", "Maintainer": "Sterett H. Mercer ", "_user": "shmercer", "_type": "src", "_file": "pairwiseLLM_1.6.0.tar.gz", "_fileid": "https://r2.ropensci.org/2f5dc4c59b2451944f7e5076865453809791d67952e29da1afc64a6051260838", "_filesize": 5061337, "_sha256": "2f5dc4c59b2451944f7e5076865453809791d67952e29da1afc64a6051260838", "_expires": "2027-01-27T21:01:21.000Z", "_created": "2026-10-19T20:25:47.000Z", "_published": "2026-10-19T21:01:23.775Z", "_upstream": "https://github.com/shmercer/pairwisellm", "_status": "success", "_maintainer": { "name": "Sterett H. Mercer", "email": "sterett.mercer@ubc.ca", "login": "shmercer", "description": "", "uuid": 64615368, "orcid": "0000-0002-7940-4221" }, "_jobs": [ { "job": 118289152331, "time": 1247, "config": "linux-devel-x86_64", "r": "4.7.0", "check": "OK", "artifact": "12219754569" }, { "job": 118289152532, "time": 938, "config": "linux-release-x86_64", "r": "4.6.1", "check": "OK", "artifact": "12219783419" }, { "job": 118289152125, "time": 1097, "config": "macos-oldrel-arm64", "r": "4.5.3", "check": "OK", "artifact": "12221716817" }, { "job": 118289152049, "time": 1096, "config": "macos-release-arm64", "r": "4.6.1", "check": "OK", "artifact": "12221836389" }, { "job": 118283717731, "time": 678, "config": "source", "r": "4.6.1", "check": "OK", "artifact": "12219651463" }, { "job": 118289152126, "time": 192, "config": "wasm-release", "r": "4.6.0", "check": "OK", "artifact": "12219666967" }, { "job": 118289152478, "time": 1464, "config": "windows-devel-x86_64", "r": "4.7.0", "check": "OK", "artifact": "12220039047" }, { "job": 118289152357, "time": 1847, "config": "windows-oldrel-x86_64", "r": "4.5.3", "check": "OK", "artifact": "12221950455" }, { "job": 118289152150, "time": 1921, "config": "windows-release-x86_64", "r": "4.6.1", "check": "OK", "artifact": "12221766238" } ], "_buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115", "_commit": { "id": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "author": "Sterett Mercer ", "committer": "GitHub ", "message": "docs: add Matta et al. writing screening preprint (#345)", "time": 1792437335 }, "_distro": "resolute", "_host": "GitHub-Actions", "_registered": true, "_dependencies": [ { "package": "R", "version": ">= 4.4", "role": "Depends" }, { "package": "cli", "role": "Imports" }, { "package": "curl", "role": "Imports" }, { "package": "dplyr", "role": "Imports" }, { "package": "httr2", "version": ">= 1.1.0", "role": "Imports" }, { "package": "jsonlite", "role": "Imports" }, { "package": "openssl", "role": "Imports" }, { "package": "rlang", "version": ">= 1.3.0", "role": "Imports" }, { "package": "stats", "role": "Imports" }, { "package": "tibble", "role": "Imports" }, { "package": "tidyselect", "role": "Imports" }, { "package": "tools", "role": "Imports" }, { "package": "utils", "role": "Imports" }, { "package": "BradleyTerry2", "role": "Suggests" }, { "package": "brglm2", "role": "Suggests" }, { "package": "cmdstanr", "role": "Suggests" }, { "package": "e1071", "role": "Suggests" }, { "package": "EloChoice", "role": "Suggests" }, { "package": "filelock", "role": "Suggests" }, { "package": "future", "role": "Suggests" }, { "package": "future.apply", "role": "Suggests" }, { "package": "glmnet", "role": "Suggests" }, { "package": "knitr", "role": "Suggests" }, { "package": "later", "role": "Suggests" }, { "package": "mgcv", "version": ">= 1.9-4", "role": "Suggests" }, { "package": "mockery", "role": "Suggests" }, { "package": "ordinal", "role": "Suggests" }, { "package": "parallelly", "role": "Suggests" }, { "package": "pkgload", "role": "Suggests" }, { "package": "pls", "role": "Suggests" }, { "package": "posterior", "role": "Suggests" }, { "package": "promises", "role": "Suggests" }, { "package": "purrr", "role": "Suggests" }, { "package": "readr", "role": "Suggests" }, { "package": "rmarkdown", "role": "Suggests" }, { "package": "reticulate", "version": ">= 1.41", "role": "Suggests" }, { "package": "sirt", "role": "Suggests" }, { "package": "stringr", "role": "Suggests" }, { "package": "testthat", "version": ">= 3.0.0", "role": "Suggests" }, { "package": "tidyr", "role": "Suggests" }, { "package": "vctrs", "role": "Suggests" }, { "package": "withr", "version": ">= 3.0.3", "role": "Suggests" } ], "_owner": "shmercer", "_selfowned": true, "_usedby": 0, "_updates": [ { "week": "2025-48", "n": 59 }, { "week": "2025-49", "n": 53 }, { "week": "2025-50", "n": 13 }, { "week": "2025-51", "n": 59 }, { "week": "2025-52", "n": 16 }, { "week": "2026-01", "n": 2 }, { "week": "2026-03", "n": 6 }, { "week": "2026-07", "n": 8 }, { "week": "2026-36", "n": 4 }, { "week": "2026-37", "n": 5 }, { "week": "2026-38", "n": 17 }, { "week": "2026-39", "n": 13 }, { "week": "2026-40", "n": 8 }, { "week": "2026-41", "n": 8 }, { "week": "2026-42", "n": 8 }, { "week": "2026-43", "n": 1 } ], "_tags": [ { "name": "v1.0.0", "date": "2025-12-10" }, { "name": "v1.1.0", "date": "2025-12-18" }, { "name": "v1.3.1", "date": "2026-09-06" } ], "_topics": [ "anthropic", "assessment-tool", "automated-scoring", "bradley-terry", "elo-rating", "gemini", "large-language-models", "llm", "llm-as-a-judge", "ollama", "openai", "pairwise-comparison", "ranking", "together-ai", "writing-assessment", "writing-evaluation" ], "_stars": 3, "_contributors": [ { "user": "shmercer", "count": 388, "uuid": 64615368 } ], "_userbio": { "uuid": 64615368, "type": "user", "name": "Sterett Mercer", "followers": 1 }, "_downloads": { "count": 390, "source": "https://cranlogs.r-pkg.org/downloads/total/last-month/pairwiseLLM" }, "_devurl": "https://github.com/shmercer/pairwisellm", "_pkgdown": "https://shmercer.github.io/pairwiseLLM/", "_searchresults": 39, "_rbuild": "4.6.1", "_assets": [ "extra/citation.cff", "extra/citation.html", "extra/citation.json", "extra/citation.txt", "extra/contents.json", "extra/manual.pdf", "extra/NEWS.html", "extra/NEWS.txt", "extra/pairwiseLLM.html", "extra/readme.html", "extra/readme.md", "LICENSE" ], "_homeurl": "https://github.com/shmercer/pairwisellm", "_realowner": "shmercer", "_cranurl": true, "_releases": [ { "version": "1.1.0", "date": "2025-12-22" }, { "version": "1.3.1", "date": "2026-09-08" } ], "_exports": [ "adaptive_get_logs", "adaptive_item_log", "adaptive_rank", "adaptive_rank_resume", "adaptive_rank_run_live", "adaptive_rank_start", "adaptive_results_history", "adaptive_round_log", "adaptive_step_log", "alternate_pair_order", "anthropic_compare_pair_live", "anthropic_create_batch", "anthropic_download_batch_results", "anthropic_get_batch", "anthropic_poll_batch_until_complete", "bootstrap_bt_model", "build_anthropic_batch_requests", "build_bt_data", "build_btl_results_data", "build_elo_data", "build_gemini_batch_requests", "build_openai_batch_requests", "build_prompt", "check_llm_api_keys", "check_positional_bias", "compute_reverse_consistency", "ensemble_warm_start_algorithms", "ensemble_warm_start_models", "ensure_only_ollama_model_loaded", "estimate_llm_pairs_cost", "evaluate_rubric_predictions", "extract_warm_start_features", "fit_bayes_btl_mcmc", "fit_bt_model", "fit_elo_model", "fit_link", "fit_rubric_calibration", "fit_warm_start_model", "gemini_compare_pair_live", "gemini_create_batch", "gemini_download_batch_results", "gemini_get_batch", "gemini_poll_batch_until_complete", "get_prompt_template", "list_prompt_templates", "list_warm_start_models", "llm_compare_pair", "llm_download_batch_results", "llm_resume_multi_batches", "llm_submit_pairs_batch", "llm_submit_pairs_multi_batch", "load_adaptive_session", "load_link_session", "load_warm_start_model", "make_adaptive_judge_llm", "make_adaptive_judge_replay", "make_adaptive_replay_reservoir", "make_pairs", "make_warm_start_cv_plan", "make_warm_start_prior", "ollama_compare_pair_live", "openai_compare_pair_live", "openai_create_batch", "openai_download_batch_errors", "openai_download_batch_output", "openai_get_batch", "openai_poll_batch_until_complete", "openai_upload_batch_file", "parse_anthropic_batch_output", "parse_gemini_batch_output", "parse_openai_batch_output", "predict_link", "prepare_link_input", "prepare_linked_rubric_reference", "prepare_warm_start_model", "randomize_pair_order", "read_samples_df", "read_samples_dir", "register_prompt_template", "register_warm_start_model", "remove_prompt_template", "remove_warm_start_model", "resume_link_session", "run_anthropic_batch_pipeline", "run_gemini_batch_pipeline", "run_openai_batch_pipeline", "sample_pairs", "sample_reverse_pairs", "save_adaptive_session", "save_link_session", "save_warm_start_model", "scale_separation_reliability", "set_prompt_template", "start_link_session", "submit_anthropic_pairs_live", "submit_gemini_pairs_live", "submit_llm_pairs", "submit_ollama_pairs_live", "submit_openai_pairs_live", "submit_together_pairs_live", "submit_typesafe_pairs_live", "submit_vertex_pairs_live", "summarize_adaptive", "summarize_bt_fit", "summarize_items", "summarize_refits", "together_compare_pair_live", "trait_description", "typesafe_compare_pair_live", "validate_adaptive_replay", "validate_session_dir", "vertex_compare_pair_live", "warm_start_coefficients", "warm_start_feature_schema", "warm_start_python_status", "write_openai_batch_file" ], "_datasets": [ { "name": "example_openai_batch_output", "title": "Example OpenAI Batch output (JSONL lines)", "object": "example_openai_batch_output", "file": "example_openai_batch_output.rda", "class": [ "character" ], "fields": [], "table": false, "tojson": true }, { "name": "example_writing_pairs", "title": "Example dataset of paired comparisons for writing samples", "object": "example_writing_pairs", "file": "example_writing_pairs.rda", "class": [ "tbl_df", "tbl", "data.frame" ], "fields": [ "ID1", "ID2", "better_id" ], "rows": 190, "table": true, "tojson": true }, { "name": "example_writing_results", "title": "Example canonical results table for writing comparisons", "object": "example_writing_results", "file": "example_writing_results.rda", "class": [ "tbl_df", "tbl", "data.frame" ], "fields": [ "pair_uid", "unordered_key", "ordered_key", "A_id", "B_id", "better_id", "winner_pos", "phase", "iter", "received_at", "backend", "model" ], "rows": 190, "table": true, "tojson": true }, { "name": "example_writing_samples", "title": "Example dataset of writing samples", "object": "example_writing_samples", "file": "example_writing_samples.rda", "class": [ "tbl_df", "tbl", "data.frame" ], "fields": [ "ID", "text", "quality_score" ], "rows": 20, "table": true, "tojson": true }, { "name": "example_writing_samples1000", "title": "Synthetic Writing Samples with Controlled Quality Levels (N = 1000)", "object": "example_writing_samples1000", "file": "example_writing_samples1000.rda", "class": [ "tbl_df", "tbl", "data.frame" ], "fields": [ "ID", "text", "quality_level", "theta_true", "prompt_id", "model", "created_at" ], "rows": 1000, "table": true, "tojson": true } ], "_help": [ { "page": "adaptive_get_logs", "title": "Retrieve canonical adaptive logs.", "concept": [ "adaptive logs" ], "topics": [ "adaptive_get_logs" ] }, { "page": "adaptive_item_log", "title": "Adaptive item log accessor.", "concept": [ "adaptive logs" ], "topics": [ "adaptive_item_log" ] }, { "page": "adaptive_rank", "title": "Run adaptive ranking end-to-end from data and model settings", "concept": [ "adaptive ranking" ], "topics": [ "adaptive_rank" ] }, { "page": "adaptive_rank_resume", "title": "Adaptive ranking resume", "concept": [ "adaptive ranking" ], "topics": [ "adaptive_rank_resume" ] }, { "page": "adaptive_rank_run_live", "title": "Adaptive ranking live runner", "concept": [ "adaptive ranking" ], "topics": [ "adaptive_rank_run_live" ] }, { "page": "adaptive_rank_start", "title": "Adaptive ranking", "concept": [ "adaptive ranking" ], "topics": [ "adaptive_rank_start" ] }, { "page": "adaptive_results_history", "title": "Adaptive results history ready for BT fitting", "concept": [ "adaptive logs" ], "topics": [ "adaptive_results_history" ] }, { "page": "adaptive_round_log", "title": "Adaptive round log accessor.", "concept": [ "adaptive logs" ], "topics": [ "adaptive_round_log" ] }, { "page": "adaptive_step_log", "title": "Adaptive step log accessor.", "concept": [ "adaptive logs" ], "topics": [ "adaptive_step_log" ] }, { "page": "alternate_pair_order", "title": "Deterministically alternate sample order in pairs", "concept": [ "pairing and data" ], "topics": [ "alternate_pair_order" ] }, { "page": "anthropic_compare_pair_live", "title": "Live Anthropic (Claude) comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "anthropic_compare_pair_live" ] }, { "page": "anthropic_create_batch", "title": "Create an Anthropic Message Batch", "concept": [ "batch backends" ], "topics": [ "anthropic_create_batch" ] }, { "page": "anthropic_download_batch_results", "title": "Download Anthropic Message Batch results (.jsonl)", "concept": [ "batch backends" ], "topics": [ "anthropic_download_batch_results" ] }, { "page": "anthropic_get_batch", "title": "Retrieve an Anthropic Message Batch by ID", "concept": [ "batch backends" ], "topics": [ "anthropic_get_batch" ] }, { "page": "anthropic_poll_batch_until_complete", "title": "Poll an Anthropic Message Batch until completion", "concept": [ "batch backends" ], "topics": [ "anthropic_poll_batch_until_complete" ] }, { "page": "bootstrap_bt_model", "title": "Schedule-aware parametric bootstrap bias correction for Bradley-Terry scores", "concept": [ "frequentist models" ], "topics": [ "bootstrap_bt_model" ] }, { "page": "build_anthropic_batch_requests", "title": "Build Anthropic Message Batch requests from a tibble of pairs", "concept": [ "batch backends" ], "topics": [ "build_anthropic_batch_requests" ] }, { "page": "build_bt_data", "title": "Build Bradley-Terry comparison data from pairwise results", "concept": [ "frequentist models" ], "topics": [ "build_bt_data" ] }, { "page": "build_btl_results_data", "title": "Build canonical 'results_tbl' data for Bayesian BTL MCMC", "concept": [ "Bayesian models" ], "topics": [ "build_btl_results_data" ] }, { "page": "build_elo_data", "title": "Build EloChoice comparison data from pairwise results", "concept": [ "frequentist models" ], "topics": [ "build_elo_data" ] }, { "page": "build_gemini_batch_requests", "title": "Build Gemini batch requests from a tibble of pairs", "concept": [ "batch backends" ], "topics": [ "build_gemini_batch_requests" ] }, { "page": "build_openai_batch_requests", "title": "Build OpenAI batch JSONL lines for paired comparisons", "concept": [ "batch backends" ], "topics": [ "build_openai_batch_requests" ] }, { "page": "build_prompt", "title": "Build a concrete LLM prompt from a template", "concept": [ "prompts and traits" ], "topics": [ "build_prompt" ] }, { "page": "check_llm_api_keys", "title": "Check configured API keys for LLM backends", "concept": [ "live backends" ], "topics": [ "check_llm_api_keys" ] }, { "page": "check_positional_bias", "title": "Check positional preference and bootstrap reversal agreement", "concept": [ "bias and consistency" ], "topics": [ "check_positional_bias" ] }, { "page": "compute_reverse_consistency", "title": "Compute consistency between forward and reverse pair comparisons", "concept": [ "bias and consistency" ], "topics": [ "compute_reverse_consistency" ] }, { "page": "decision_bulk", "title": "Live decision execution, persistence and recovery contracts", "topics": [ "decision_bulk" ] }, { "page": "ensemble_warm_start_algorithms", "title": "Average algorithms trained on the same task", "concept": [ "adaptive warm start" ], "topics": [ "ensemble_warm_start_algorithms" ] }, { "page": "ensemble_warm_start_models", "title": "Combine independently trained warm-start models", "concept": [ "adaptive warm start" ], "topics": [ "ensemble_warm_start_models" ] }, { "page": "ensure_only_ollama_model_loaded", "title": "Ensure only one Ollama model is loaded in memory", "concept": [ "Ollama resource management" ], "topics": [ "ensure_only_ollama_model_loaded" ] }, { "page": "estimate_llm_pairs_cost", "title": "Estimate LLM token usage and cost for a set of pairwise comparisons", "concept": [ "cost estimation" ], "topics": [ "estimate_llm_pairs_cost" ] }, { "page": "evaluate_rubric_predictions", "title": "Evaluate rubric predictions on observed ordered labels", "concept": [ "rubric calibration" ], "topics": [ "evaluate_rubric_predictions" ] }, { "page": "example_openai_batch_output", "title": "Example OpenAI Batch output (JSONL lines)", "topics": [ "example_openai_batch_output" ] }, { "page": "example_writing_pairs", "title": "Example dataset of paired comparisons for writing samples", "topics": [ "example_writing_pairs" ] }, { "page": "example_writing_results", "title": "Example canonical results table for writing comparisons", "topics": [ "example_writing_results" ] }, { "page": "example_writing_samples", "title": "Example dataset of writing samples", "topics": [ "example_writing_samples" ] }, { "page": "example_writing_samples1000", "title": "Synthetic Writing Samples with Controlled Quality Levels (N = 1000)", "topics": [ "example_writing_samples1000" ] }, { "page": "extract_warm_start_features", "title": "Extract frozen writing features for warm-start prediction", "concept": [ "adaptive warm start" ], "topics": [ "extract_warm_start_features" ] }, { "page": "fit_bayes_btl_mcmc", "title": "Full Bayesian BTL inference via CmdStanR (adaptive-compatible)", "concept": [ "Bayesian models" ], "topics": [ "fit_bayes_btl_mcmc" ] }, { "page": "fit_bt_model", "title": "Fit a Bradley-Terry model with optional frequentist engines", "concept": [ "frequentist models" ], "topics": [ "fit_bt_model" ] }, { "page": "fit_elo_model", "title": "Fit an EloChoice model to pairwise comparison data", "concept": [ "frequentist models" ], "topics": [ "fit_elo_model" ] }, { "page": "fit_link", "title": "Fit a linker using prepared explicit evidence", "concept": [ "linking" ], "topics": [ "fit_link" ] }, { "page": "fit_rubric_calibration", "title": "Fit a rubric calibration to completed comparative judgments", "concept": [ "rubric calibration" ], "topics": [ "fit_rubric_calibration" ] }, { "page": "fit_warm_start_model", "title": "Train a task-specific warm-start model with nested validation", "concept": [ "adaptive warm start" ], "topics": [ "fit_warm_start_model" ] }, { "page": "gemini_compare_pair_live", "title": "Live Google Gemini comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "gemini_compare_pair_live" ] }, { "page": "gemini_create_batch", "title": "Create a Gemini Batch job from request objects", "concept": [ "batch backends" ], "topics": [ "gemini_create_batch" ] }, { "page": "gemini_download_batch_results", "title": "Download Gemini Batch results to a JSONL file", "concept": [ "batch backends" ], "topics": [ "gemini_download_batch_results" ] }, { "page": "gemini_get_batch", "title": "Retrieve a Gemini Batch job by name", "concept": [ "batch backends" ], "topics": [ "gemini_get_batch" ] }, { "page": "gemini_poll_batch_until_complete", "title": "Poll a Gemini Batch job until completion", "concept": [ "batch backends" ], "topics": [ "gemini_poll_batch_until_complete" ] }, { "page": "get_prompt_template", "title": "Retrieve a named prompt template", "concept": [ "prompts and traits" ], "topics": [ "get_prompt_template" ] }, { "page": "list_prompt_templates", "title": "List available prompt templates", "concept": [ "prompts and traits" ], "topics": [ "list_prompt_templates" ] }, { "page": "llm_compare_pair", "title": "Backend-agnostic live comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "llm_compare_pair" ] }, { "page": "llm_download_batch_results", "title": "Extract results from a pairwiseLLM batch object", "concept": [ "batch backends" ], "topics": [ "llm_download_batch_results" ] }, { "page": "llm_resume_multi_batches", "title": "Resume polling and download results for multiple batch jobs", "concept": [ "batch backends" ], "topics": [ "llm_resume_multi_batches" ] }, { "page": "llm_submit_pairs_batch", "title": "Submit pairs to an LLM backend via batch API", "concept": [ "batch backends" ], "topics": [ "llm_submit_pairs_batch" ] }, { "page": "llm_submit_pairs_multi_batch", "title": "Multi‑batch submission and polling wrappers", "concept": [ "batch backends" ], "topics": [ "llm_submit_pairs_multi_batch" ] }, { "page": "load_adaptive_session", "title": "Load an adaptive session from disk.", "concept": [ "adaptive persistence" ], "topics": [ "load_adaptive_session" ] }, { "page": "make_adaptive_judge_llm", "title": "Build an LLM judge function for adaptive ranking", "concept": [ "adaptive ranking" ], "topics": [ "make_adaptive_judge_llm" ] }, { "page": "make_adaptive_judge_replay", "title": "Create an offline judge from frozen directed outcomes", "concept": [ "adaptive ranking" ], "topics": [ "make_adaptive_judge_replay" ] }, { "page": "make_adaptive_replay_reservoir", "title": "Create a sparse frozen adaptive replay reservoir", "concept": [ "adaptive ranking" ], "topics": [ "make_adaptive_replay_reservoir" ] }, { "page": "make_pairs", "title": "Create all unordered pairs of writing samples", "concept": [ "pairing and data" ], "topics": [ "make_pairs" ] }, { "page": "make_warm_start_cv_plan", "title": "Construct reusable warm-start cross-validation partitions", "concept": [ "adaptive warm start" ], "topics": [ "make_warm_start_cv_plan" ] }, { "page": "make_warm_start_prior", "title": "Convert warm-start predictions to Bayesian BTL priors", "concept": [ "adaptive warm start" ], "topics": [ "make_warm_start_prior" ] }, { "page": "ollama_compare_pair_live", "title": "Live Ollama comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "ollama_compare_pair_live" ] }, { "page": "openai_compare_pair_live", "title": "Live OpenAI comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "openai_compare_pair_live" ] }, { "page": "openai_create_batch", "title": "Create an OpenAI batch from an uploaded file", "concept": [ "batch backends" ], "topics": [ "openai_create_batch" ] }, { "page": "openai_download_batch_errors", "title": "Download the error file for an OpenAI batch", "concept": [ "batch backends" ], "topics": [ "openai_download_batch_errors" ] }, { "page": "openai_download_batch_output", "title": "Download the output file for a completed batch", "concept": [ "batch backends" ], "topics": [ "openai_download_batch_output" ] }, { "page": "openai_get_batch", "title": "Retrieve an OpenAI batch", "concept": [ "batch backends" ], "topics": [ "openai_get_batch" ] }, { "page": "openai_poll_batch_until_complete", "title": "Poll an OpenAI batch until it completes or fails", "concept": [ "batch backends" ], "topics": [ "openai_poll_batch_until_complete" ] }, { "page": "openai_upload_batch_file", "title": "Upload a JSONL batch file to OpenAI", "concept": [ "batch backends" ], "topics": [ "openai_upload_batch_file" ] }, { "page": "pairwiseLLM_warm_model", "title": "Portable task-specific warm-start models", "concept": [ "adaptive warm start" ], "topics": [ "pairwiseLLM_warm_model", "print.pairwiseLLM_warm_model", "summary.pairwiseLLM_warm_model" ] }, { "page": "parse_anthropic_batch_output", "title": "Parse Anthropic Message Batch output into a tibble", "concept": [ "result normalization" ], "topics": [ "parse_anthropic_batch_output" ] }, { "page": "parse_gemini_batch_output", "title": "Parse Gemini batch JSONL output into a tibble of pairwise results", "concept": [ "result normalization" ], "topics": [ "parse_gemini_batch_output" ] }, { "page": "parse_openai_batch_output", "title": "Parse an OpenAI Batch output JSONL file", "concept": [ "result normalization" ], "topics": [ "parse_openai_batch_output" ] }, { "page": "predict_link", "title": "Predict oriented hub-spoke comparison probabilities", "concept": [ "linking" ], "topics": [ "predict_link" ] }, { "page": "predict.pairwiseLLM_bt_alpha", "title": "Predict pairwise win probabilities from an alpha-adjusted Bradley-Terry fit", "concept": [ "frequentist models" ], "topics": [ "predict.pairwiseLLM_bt_alpha" ] }, { "page": "predict.pairwiseLLM_bt_firth", "title": "Predict pairwise win probabilities from a Firth Bradley-Terry fit", "concept": [ "frequentist models" ], "topics": [ "predict.pairwiseLLM_bt_firth" ] }, { "page": "predict.pairwiseLLM_bt_lapse", "title": "Predict ordered comparisons from an experimental lapse BTL fit", "concept": [ "frequentist models" ], "topics": [ "predict.pairwiseLLM_bt_lapse" ] }, { "page": "predict.pairwiseLLM_rubric_calibration", "title": "Predict rubric scores from a calibration", "concept": [ "rubric calibration" ], "topics": [ "predict.pairwiseLLM_rubric_calibration" ] }, { "page": "predict.pairwiseLLM_warm_algorithm_ensemble", "title": "Predict with a same-task algorithm ensemble", "concept": [ "adaptive warm start" ], "topics": [ "predict.pairwiseLLM_warm_algorithm_ensemble" ] }, { "page": "predict.pairwiseLLM_warm_ensemble", "title": "Predict with an equal-weight warm-start ensemble", "concept": [ "adaptive warm start" ], "topics": [ "predict.pairwiseLLM_warm_ensemble" ] }, { "page": "predict.pairwiseLLM_warm_model", "title": "Predict relative quality from a portable warm-start model", "concept": [ "adaptive warm start" ], "topics": [ "predict.pairwiseLLM_warm_model" ] }, { "page": "prepare_link_input", "title": "Prepare explicit evidence for a linking estimator", "concept": [ "linking" ], "topics": [ "prepare_link_input" ] }, { "page": "prepare_linked_rubric_reference", "title": "Save a standalone Bayesian ranking as a reusable rubric reference", "concept": [ "rubric calibration" ], "topics": [ "prepare_linked_rubric_reference" ] }, { "page": "prepare_warm_start_model", "title": "Prepare metadata or a summary-only warm-start artifact", "concept": [ "adaptive warm start" ], "topics": [ "prepare_warm_start_model" ] }, { "page": "print.adaptive_state", "title": "Print an adaptive state summary.", "topics": [ "print.adaptive_state" ] }, { "page": "print.pairwiseLLM_cost_estimate", "title": "Print a pairwiseLLM cost estimate", "topics": [ "print.pairwiseLLM_cost_estimate" ] }, { "page": "randomize_pair_order", "title": "Randomly assign samples to positions SAMPLE_1 and SAMPLE_2", "concept": [ "pairing and data" ], "topics": [ "randomize_pair_order" ] }, { "page": "read_samples_df", "title": "Read writing samples from a data frame", "concept": [ "pairing and data" ], "topics": [ "read_samples_df" ] }, { "page": "read_samples_dir", "title": "Read writing samples from a directory of .txt files", "concept": [ "pairing and data" ], "topics": [ "read_samples_dir" ] }, { "page": "register_prompt_template", "title": "Register a named prompt template", "concept": [ "prompts and traits" ], "topics": [ "register_prompt_template" ] }, { "page": "register_warm_start_model", "title": "Register, inspect, or remove user warm-start models", "concept": [ "adaptive warm start" ], "topics": [ "list_warm_start_models", "register_warm_start_model", "remove_warm_start_model" ] }, { "page": "remove_prompt_template", "title": "Remove a registered prompt template", "concept": [ "prompts and traits" ], "topics": [ "remove_prompt_template" ] }, { "page": "run_anthropic_batch_pipeline", "title": "Run an Anthropic batch pipeline for pairwise comparisons", "concept": [ "batch backends" ], "topics": [ "run_anthropic_batch_pipeline" ] }, { "page": "run_gemini_batch_pipeline", "title": "Run a Gemini batch pipeline for pairwise comparisons", "concept": [ "batch backends" ], "topics": [ "run_gemini_batch_pipeline" ] }, { "page": "run_openai_batch_pipeline", "title": "Run a full OpenAI batch pipeline for pairwise comparisons", "concept": [ "batch backends" ], "topics": [ "run_openai_batch_pipeline" ] }, { "page": "sample_pairs", "title": "Randomly sample pairs of writing samples", "concept": [ "pairing and data" ], "topics": [ "sample_pairs" ] }, { "page": "sample_reverse_pairs", "title": "Sample reversed versions of a subset of pairs", "concept": [ "pairing and data" ], "topics": [ "sample_reverse_pairs" ] }, { "page": "save_adaptive_session", "title": "Save an adaptive session to disk.", "concept": [ "adaptive persistence" ], "topics": [ "save_adaptive_session" ] }, { "page": "save_link_session", "title": "Save and load exact linking sessions", "concept": [ "linking" ], "topics": [ "load_link_session", "save_link_session" ] }, { "page": "save_warm_start_model", "title": "Save or load a portable warm-start model", "concept": [ "adaptive warm start" ], "topics": [ "load_warm_start_model", "save_warm_start_model" ] }, { "page": "scale_separation_reliability", "title": "Calculate scale-separation reliability", "concept": [ "frequentist models" ], "topics": [ "scale_separation_reliability" ] }, { "page": "set_prompt_template", "title": "Read or validate a prompt template for pairwise comparisons", "concept": [ "prompts and traits" ], "topics": [ "set_prompt_template" ] }, { "page": "start_link_session", "title": "Run and resume explicit-evidence linking sessions", "concept": [ "linking" ], "topics": [ "resume_link_session", "start_link_session" ] }, { "page": "submit_anthropic_pairs_live", "title": "Live Anthropic (Claude) comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_anthropic_pairs_live" ] }, { "page": "submit_gemini_pairs_live", "title": "Live Google Gemini comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_gemini_pairs_live" ] }, { "page": "submit_llm_pairs", "title": "Backend-agnostic live comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_llm_pairs" ] }, { "page": "submit_ollama_pairs_live", "title": "Live Ollama comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_ollama_pairs_live" ] }, { "page": "submit_openai_pairs_live", "title": "Live OpenAI comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_openai_pairs_live" ] }, { "page": "submit_together_pairs_live", "title": "Live Together.ai comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_together_pairs_live" ] }, { "page": "submit_typesafe_pairs_live", "title": "Collect and resume live TypeSafe Jev decisions", "concept": [ "live backends" ], "topics": [ "submit_typesafe_pairs_live" ] }, { "page": "submit_vertex_pairs_live", "title": "Live Vertex AI Gemini comparisons for a tibble of pairs", "concept": [ "live backends" ], "topics": [ "submit_vertex_pairs_live" ] }, { "page": "summarize_adaptive", "title": "Summarize an adaptive state.", "concept": [ "adaptive ranking" ], "topics": [ "summarize_adaptive" ] }, { "page": "summarize_bt_fit", "title": "Summarize a Bradley–Terry model fit", "concept": [ "frequentist models" ], "topics": [ "summarize_bt_fit" ] }, { "page": "summarize_items", "title": "Summarize adaptive items", "concept": [ "adaptive logs" ], "topics": [ "summarize_items" ] }, { "page": "summarize_refits", "title": "Summarize adaptive refits", "concept": [ "adaptive logs" ], "topics": [ "summarize_refits" ] }, { "page": "summary.pairwiseLLM_link_result", "title": "Inspect explicit-evidence linking results and sessions", "concept": [ "linking" ], "topics": [ "print.pairwiseLLM_link_result", "print.pairwiseLLM_link_session", "summary.pairwiseLLM_link_result", "summary.pairwiseLLM_link_session" ] }, { "page": "summary.pairwiseLLM_warm_algorithm_ensemble", "title": "Inspect a same-task algorithm ensemble", "concept": [ "adaptive warm start" ], "topics": [ "print.pairwiseLLM_warm_algorithm_ensemble", "summary.pairwiseLLM_warm_algorithm_ensemble" ] }, { "page": "summary.pairwiseLLM_warm_ensemble", "title": "Inspect a warm-start ensemble", "concept": [ "adaptive warm start" ], "topics": [ "print.pairwiseLLM_warm_ensemble", "summary.pairwiseLLM_warm_ensemble" ] }, { "page": "summary.pairwiseLLM_warm_predictions", "title": "Inspect ensemble predictions", "concept": [ "adaptive warm start" ], "topics": [ "print.pairwiseLLM_warm_predictions", "summary.pairwiseLLM_warm_predictions" ] }, { "page": "together_compare_pair_live", "title": "Live Together.ai comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "together_compare_pair_live" ] }, { "page": "trait_description", "title": "Get a trait name and description for prompts", "concept": [ "prompts and traits" ], "topics": [ "trait_description" ] }, { "page": "typesafe_compare_pair_live", "title": "Live TypeSafe Jev comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "typesafe_compare_pair_live" ] }, { "page": "validate_adaptive_replay", "title": "Validate frozen directed judgments for an adaptive panel", "concept": [ "adaptive ranking" ], "topics": [ "validate_adaptive_replay" ] }, { "page": "validate_session_dir", "title": "Validate an adaptive session directory.", "concept": [ "adaptive persistence" ], "topics": [ "validate_session_dir" ] }, { "page": "vertex_compare_pair_live", "title": "Live Vertex AI Gemini comparison for a single pair of samples", "concept": [ "live backends" ], "topics": [ "vertex_compare_pair_live" ] }, { "page": "warm_start_coefficients", "title": "Inspect calibrated standardized warm-start coefficients", "concept": [ "adaptive warm start" ], "topics": [ "warm_start_coefficients", "warm_start_coefficients.pairwiseLLM_warm_algorithm_ensemble", "warm_start_coefficients.pairwiseLLM_warm_ensemble", "warm_start_coefficients.pairwiseLLM_warm_model" ] }, { "page": "warm_start_feature_schema", "title": "Inspect the frozen warm-start writing feature schema", "concept": [ "adaptive warm start" ], "topics": [ "warm_start_feature_schema" ] }, { "page": "warm_start_python_status", "title": "Check the optional warm-start feature environment", "concept": [ "adaptive warm start" ], "topics": [ "warm_start_python_status" ] }, { "page": "write_openai_batch_file", "title": "Write an OpenAI batch table to a JSONL file", "concept": [ "batch backends" ], "topics": [ "write_openai_batch_file" ] } ], "_readme": "https://github.com/shmercer/pairwisellm/raw/HEAD/README.md", "_rundeps": [ "askpass", "cli", "curl", "dplyr", "generics", "glue", "httr2", "jsonlite", "lifecycle", "magrittr", "openssl", "pillar", "pkgconfig", "R6", "rlang", "sys", "tibble", "tidyselect", "utf8", "vctrs", "withr" ], "_vignettes": [ { "source": "adaptive-cj-workflow.Rmd", "filename": "adaptive-cj-workflow.html", "title": "Adaptive comparative judgment: estimation, reliability, and evaluation", "engine": "knitr::rmarkdown", "headings": [ "Five related questions", "Start with comparisons chosen in advance", "Conventional fitting and estimator provenance", "What the adjustments change", "Internal reliability: SSR and Bayesian EAP", "Scale Separation Reliability (SSR)", "Bayesian BTL and EAP reliability", "Collect an adaptive schedule", "Hybrid recovery stays within hybrid", "A small provider-free p50 run", "Empirical recovery and scale dispersion", "Held-out predictive evaluation", "Schedule-aware parametric bootstrap", "Fixed versus adaptive scheduling", "What the Monte Carlo summaries mean", "Experimental lapse and positional-bias model matching", "Reporting the complete workflow", "Citation" ], "created": "2026-10-03 01:15:44", "modified": "2026-10-03 01:15:44", "commits": 1 }, { "source": "advanced-batch-workflows.Rmd", "filename": "advanced-batch-workflows.html", "title": "Advanced: Submitting and Polling Multiple Batches", "engine": "knitr::rmarkdown", "headings": [ "Overview", "Setup and API Keys", "Example Data and Prompt Template", "First batch: submit once, then resume", "OpenAI storage, output limits, and error files", "OpenAI Batch prompt caching and costs", "Designing the Batch Grid", "Submitting Many Batches with the Multi‑Batch Helpers", "Create a run plan and output directory", "Submit all runs (no polling)", "Polling, Downloading, and Parsing (Resumable)", "Building a single merged results table (all runs)", "Resuming After Interruption", "Next Steps", "Citation" ], "created": "2025-12-05 21:39:39", "modified": "2026-10-16 20:12:09", "commits": 22 }, { "source": "model-compatibility.Rmd", "filename": "model-compatibility.html", "title": "Backends and Tested Model Configurations", "engine": "knitr::rmarkdown", "headings": [ "Choose a backend and model", "What the package supports", "Dated compatibility record", "Optional future checks", "Decision-model live qualification", "Reproduce the design without sharing private data", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-10-16 20:12:09", "commits": 3 }, { "source": "bt-bootstrap.Rmd", "filename": "bt-bootstrap.html", "title": "Bootstrap bias correction for BT scores", "engine": "knitr::rmarkdown", "headings": [ "What the correction estimates", "A fixed random schedule", "An adaptive p50 schedule", "Reading the results and managing cost", "Reusing an external worker pool", "Citation" ], "created": "2026-10-02 15:35:55", "modified": "2026-10-10 23:26:23", "commits": 3 }, { "source": "data-and-prompts.Rmd", "filename": "data-and-prompts.html", "title": "Data Schemas and Prompt Management", "engine": "knitr::rmarkdown", "headings": [ "Bring a spreadsheet or data frame", "Samples and pairs", "Traits and prompt templates", "Decision instructions and probabilities", "Provider and result schemas", "Modeling and adaptive branches", "Failure checks at schema boundaries", "Related documentation", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-10-16 20:12:09", "commits": 4 }, { "source": "adaptive-linking-design.Rmd", "filename": "adaptive-linking-design.html", "title": "Design: Adaptive Linking", "engine": "knitr::rmarkdown", "headings": [ "Coordinates, identification, and the judge model", "E1: fixed-shape offset", "E2: full-covariance Gaussian posterior bridge", "E3: single-use joint-offset model", "MAP/Laplace calculation", "E3-MCMC audit engine", "Evidence, prediction, and uncertainty contracts", "Selection, probes, and stopping", "Persistence, multiple spokes, and rubric transport", "Frozen standalone rubric-reference identity", "Why old Phase B sessions cannot resume", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-09-23 18:18:46", "commits": 8 }, { "source": "within-set-adaptive-design.Rmd", "filename": "within-set-adaptive-design.html", "title": "Design: Adaptive Pairing", "engine": "knitr::rmarkdown", "headings": [ "Reading guide", "Foundational concepts", "Glossary and units", "Algorithm overview", "Predictive initialization and connected bootstrap", "Online TrueSkill update and selection utility", "Post-bootstrap pairing strategies", "Hybrid rounds, anchors, and rank strata", "Hybrid candidate filters, coverage, and fallback", "Conceptual example of one adaptive step", "Coverage and exploration", "Duplicate and degree controls", "Fallback ladder", "Global identification and hybrid long-link tapering", "Bayesian BTL refits", "Code-derived defaults", "Diagnostics, reliability, and stopping", "Audit trail and reproducibility", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-10-09 17:49:30", "commits": 10 }, { "source": "getting-started.Rmd", "filename": "getting-started.html", "title": "Getting Started with pairwiseLLM", "engine": "knitr::rmarkdown", "headings": [ "What you will do", "Install the tools for this example", "Your first result: scores from bundled comparisons", "Read the scores", "Read the reliability summary", "An option for comparisons chosen in advance", "An option for comparisons chosen adaptively", "An experimental comparison with the Bayesian likelihood", "Save an analysis", "Prepare your own writing samples", "Prepare a judging prompt", "Collect a small set of new judgments", "Configure one provider", "Submit the pairs", "Check what succeeded", "Choose the next workflow", "Common first-run problems", "Citation" ], "created": "2025-12-05 21:39:39", "modified": "2026-10-16 20:12:09", "commits": 28 }, { "source": "adaptive-linking.Rmd", "filename": "adaptive-linking.html", "title": "Guide: Adaptive Linking", "engine": "knitr::rmarkdown", "headings": [ "Choose what to retain from Phase A", "Prepare your comparisons", "Fit and inspect the link", "Save, add comparisons, and reuse a reference", "Restart a session from an older version", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-09-23 17:06:24", "commits": 8 }, { "source": "adaptive-pairing.Rmd", "filename": "adaptive-pairing.html", "title": "Guide: Adaptive Pairing", "engine": "knitr::rmarkdown", "headings": [ "What you will do", "Choose a comparison workflow", "Install and check CmdStan", "A complete offline run with adaptive_rank()", "Inspect the ranking and stopping state", "Inspect comparisons and audit logs", "What happens during a run?", "Choose how later pairs are selected", "Lower-level lifecycle and offline judges", "Replay a sparse reservoir", "Replay frozen directed judgments", "Persistence, interruption, and resume", "Invalid judgments and failed calls", "A detailed live-LLM workflow", "Inspect the live run", "Continue the same study", "Troubleshooting", "Related documentation", "Predictive warm-start priors", "Citation" ], "created": "2026-02-10 21:06:53", "modified": "2026-10-03 01:15:44", "commits": 13 }, { "source": "adaptive-warm-start.Rmd", "filename": "adaptive-warm-start.html", "title": "Guide: Adaptive Warm Start", "engine": "knitr::rmarkdown", "headings": [ "Before you start", "Choose where starting scores are used", "Opt in to predictive uncertainty", "Choose the initial graph independently", "Count evidence consistently across arms", "A complete example with three algorithms", "Load example texts and inspect the features", "Extract features when you have Python configured", "Share the same training and validation splits", "Fit the three algorithms", "Inspect validation before combining models", "Combine algorithms trained on this assessment", "Predict new items and initialize an assessment", "Save a model and resume a session", "Combining models from separate assessments", "Inspect calibrated standardized coefficients", "Understand saved models", "Citation" ], "created": "2026-09-09 19:56:45", "modified": "2026-10-09 21:33:46", "commits": 13 }, { "source": "linking-sessions.Rmd", "filename": "linking-sessions.html", "title": "Guide: Linking Saved Comparisons", "engine": "knitr::rmarkdown", "headings": [ "Start with the samples and their comparisons", "Choose a method and fit the first comparison", "Save your work and add the next comparison", "Use saved uncertainty with E2", "Return to the original comparisons with E3", "Link a prepared standalone reference", "Predict a comparison and check the result", "Use saved Phase A results or several spokes", "Dependency upgrades and saved hashes", "Citation" ], "created": "2026-09-23 01:55:48", "modified": "2026-10-14 18:33:17", "commits": 6 }, { "source": "rubric-calibration.Rmd", "filename": "rubric-calibration.html", "title": "Guide: Rubric Calibration", "engine": "knitr::rmarkdown", "headings": [ "Choose the workflow", "What the research shows", "Start from completed CJ results", "Define the ordered rubric and labels", "Same-set linear calibration", "Read probabilities and hard scores", "Evaluate held-out labels and inspect diagnostics", "Explicit monotone alternative", "Percentile levels without human labels", "Reuse a historical rubric reference for a new set", "Prepare a reference from a standalone ranking", "Link the reference and new samples", "Separate analytic traits", "Uncertainty and reporting", "Citation" ], "created": "2026-09-13 05:10:12", "modified": "2026-10-01 21:16:50", "commits": 6 }, { "source": "prompt-template-bias.Rmd", "filename": "prompt-template-bias.html", "title": "Prompt Template Positional Bias Testing", "engine": "knitr::rmarkdown", "headings": [ "How to read this guide", "Motivation", "Testing Process Summary", "Trait descriptions and custom traits", "Built-in traits", "Setting a different built-in trait", "Creating a custom trait", "Why traits matter for positional bias testing", "Example data used in tests", "Built-in prompt templates", "Forward and reverse pairs", "Thinking / Reasoning Configurations Used in Testing", "Anthropic (Claude 4.5 models)", "thinking = \"no_thinking\"", "thinking = \"with_thinking\"", "Gemini 3 Pro Preview", "Only thinking = \"with_thinking\" was used", "OpenAI (gpt-4.1, gpt-4o, gpt-5.1)", "thinking = \"with_thinking\" (gpt-5.1 only)", "TogetherAI (Deepseek-R1, Deepseek-V3, Kimi-K2, Qwen3)", "Summary of historical request configurations", "Loading summary results", "Column definitions", "Interpreting the statistics", "Summary results by prompt", "Template test1", "Template text", "Summary table", "Template test2", "Template test3", "Template test4", "Template test5", "Per-backend summary", "Anthropic models", "Gemini models", "OpenAI models", "TogetherAI-hosted models", "Conclusion", "Citation" ], "created": "2025-12-11 19:40:28", "modified": "2026-09-16 22:24:56", "commits": 9 }, { "source": "provider-controls-and-recovery.Rmd", "filename": "provider-controls-and-recovery.html", "title": "Provider Controls and Recovery", "engine": "knitr::rmarkdown", "headings": [ "Start with the symptom", "Storage, sampling, and service tiers", "Controls by backend", "Live calls: retain successes and failures", "Decision models: save, resume, and inspect", "Batch recovery", "Adaptive recovery", "Ollama setup and resources", "Privacy and retention", "Estimate costs before a larger run", "Deterministic calculation example", "Reuse pilot results", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-10-16 20:12:09", "commits": 5 }, { "source": "bayesian-btl.Rmd", "filename": "bayesian-btl.html", "title": "Standalone Bayesian BTL with CmdStan", "engine": "knitr::rmarkdown", "headings": [ "When to use this guide", "Prerequisites", "Prepare modeling input", "Fit and summarize", "What the output means", "Chains and CPU allocation", "Cumulative refits", "Files, failures, and reproducibility", "Predictive warm-start priors", "Citation" ], "created": "2026-09-06 21:09:31", "modified": "2026-10-03 01:15:44", "commits": 7 } ], "_score": 7.175395831394031, "_indexed": true, "_nocasepkg": "pairwisellm", "_universes": [ "shmercer" ], "_binaries": [ { "r": "4.7.0", "os": "linux", "version": "1.6.0", "date": "2026-10-19T20:29:42.000Z", "distro": "resolute", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/1362e0b4331d5e958dd54bdc0f0f9b95e12b98defd96148f243720c39cd36edc", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.6.1", "os": "linux", "version": "1.6.0", "date": "2026-10-19T20:29:23.000Z", "distro": "resolute", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/871d47a3080fc375fd1a9a157282b5786a7eb3ebcf0c8840392f13dd0b457f56", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.5.3", "os": "mac", "version": "1.6.0", "date": "2026-10-19T20:44:23.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/def3154d668fbede3823d58c93247a8c7d27e9424ae4510c2009a23c737ed71a", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.6.1", "os": "mac", "version": "1.6.0", "date": "2026-10-19T20:43:17.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/60ee74997d126bc212a495d03615ce527ee3d14d584bad3413cc9cb0fbe68b7f", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.6.0", "os": "wasm", "version": "1.6.0", "date": "2026-10-19T20:30:01.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/39299e36f5df4a777aeb347e6a619d4170b271fc8388a3d488167e93295ee3a5", "status": "success", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.7.0", "os": "win", "version": "1.6.0", "date": "2026-10-19T20:28:19.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/aed2a9e62acc248eac923f0437ee10ed61929e44d88431c5c146f32ac52d3109", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.5.3", "os": "win", "version": "1.6.0", "date": "2026-10-19T20:27:46.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/8ccd65bc9cdc25629b1233cc44dd1e688aa02387ec052c740b325ac287e98880", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" }, { "r": "4.6.1", "os": "win", "version": "1.6.0", "date": "2026-10-19T20:27:53.000Z", "commit": "3cf49a9a2559f341e30f2e77e8df2ad4883ab7f5", "fileid": "https://r2.ropensci.org/00896138512a15720f2a9afba20000e78970f9b3fcd032566a9e79406938c8a1", "status": "success", "check": "OK", "buildurl": "https://github.com/r-universe/shmercer/actions/runs/39317760115" } ] }