Getting Started with pairwiseLLM

Read the scores

The displayed rows are ordered by score. The full scores table has 20 rows. A difference between two estimates is not automatically a meaningful difference: consider their uncertainty and the quality of the judgments.

Zero is a location on a relative scale, not a pass mark. These values are not percentages or rubric grades, and independently fitted cohorts do not share a common scale. For rubric categories, use the rubric calibration guide, which requires completed Bayesian CJ results; the frequentist fit above is not an accepted input.

Read the reliability summary

The fit also reports scale-separation reliability, or SSR. This compares how spread out the estimated scores are with how uncertain those estimates are. You can read it from the fit you already created:

fit$reliability
#> [1] 0.6223254

Values closer to 1 mean that estimated score differences are large compared with their uncertainty. A value below zero means that uncertainty is large compared with the spread of scores, so treat the ranking cautiously.

A high SSR does not prove that the ranking is accurate. Changing the fitting method can spread the scores farther apart and raise SSR without improving the ranking. Consider SSR alongside the score uncertainties and the quality of the judgments.

For the sirt fit above, if the package cannot calculate a valid reliability value, it stops with an error and explains the problem. The comparisons must also connect all samples, directly or through other samples, to place them on one shared scale. If you use BradleyTerry2, reliability is shown as NA, meaning it is not available from that fitting method in this package.

For the formula and a worked example, open the help page with ?scale_separation_reliability. The ?fit_bt_model help page explains the fitting settings and the additional details saved with each fit.

An option for comparisons chosen in advance

For comparisons chosen randomly or fixed before seeing the results, you can choose Firth estimation. This method reduces small-sample bias and can still estimate scores when a sample always wins or always loses. The comparisons must connect all samples, and the optional brglm2 package must be installed.

firth_fit <- fit_bt_model(build_bt_data(example_writing_pairs),
                          engine = "brglm2", verbose = FALSE)
head(summarize_bt_fit(firth_fit), 5)
#> # A tibble: 5 × 6
#>   ID    theta    se  rank engine reliability
#>   <chr> <dbl> <dbl> <int> <chr>        <dbl>
#> 1 S01   -4.68 1.17     19 brglm2       0.908
#> 2 S02   -3.41 0.976    17 brglm2       0.908
#> 3 S03   -4.68 1.17     20 brglm2       0.908
#> 4 S04   -4.02 1.05     18 brglm2       0.908
#> 5 S05   -2.80 0.917    16 brglm2       0.908
head(predict(firth_fit)) # probability that the first sample wins each comparison
#> [1] 0.21830274 0.50000000 0.34047109 0.13188116 0.07605926 0.04301231

The same cautions about uncertainty and reliability apply. If every estimated score is equal, this method keeps the fit but reports reliability as NA, because there is no score spread to compare with uncertainty. This option is intended for random or fixed schedules; it is not the package’s correction for adaptive pairing, where later comparisons depend on earlier results. See ?fit_bt_model for the method and assumptions.

An option for comparisons chosen adaptively

When later comparisons depend on earlier results, alpha adjustment is an option motivated by the estimation bias that adaptive scheduling can introduce. It gently pulls scores together by adding a small, equal amount of evidence in both directions for every possible pair. This differs from the conventional sirt epsilon adjustment, which depends on observed win proportions. Both methods remain available; Firth is the intended modern comparator for random schedules.

Choose alpha before examining your substantive results. The package accepts nonnegative values, including 0.30 and 0.50, and never chooses or tunes it for you. The bundled example below demonstrates the API with 0.50; it does not establish which setting is best for a particular study. This option needs no additional engine package.

alpha_fit <- fit_bt_model(build_bt_data(example_writing_pairs),
                          engine = "alpha", alpha = 0.50, verbose = FALSE)
head(summarize_bt_fit(alpha_fit), 5) # includes model-based standard errors
#> # A tibble: 5 × 6
#>   ID    theta    se  rank engine reliability
#>   <chr> <dbl> <dbl> <int> <chr>        <dbl>
#> 1 S01   -4.20 1.07     19 alpha        0.900
#> 2 S02   -2.92 0.870    17 alpha        0.900
#> 3 S03   -4.20 1.07     20 alpha        0.900
#> 4 S04   -3.51 0.947    18 alpha        0.900
#> 5 S05   -2.38 0.815    16 alpha        0.900
head(predict(alpha_fit))           # first-sample win probabilities
#> [1] 0.21722290 0.50000000 0.33427851 0.13887040 0.08836331 0.05650570
alpha_fit$ssr                     # reliability and its variance components
#> $observed_variance
#> [1] 6.666692
#> 
#> $mean_squared_se
#> [1] 0.6653468
#> 
#> $true_score_variance
#> [1] 6.001345
#> 
#> $ssr
#> [1] 0.9001984
#> 
#> $n_items
#> [1] 20
#> 
#> $n_finite
#> [1] 20
#> 
#> $valid
#> [1] TRUE
#> 
#> $status
#> [1] "ok"
alpha_fit$provenance$adjustment   # exact penalty used
#> $method
#> [1] "hamilton_alpha"
#> 
#> $alpha
#> [1] 0.5
#> 
#> $pseudo_count
#> [1] 0.02631579
#> 
#> $pairs
#> [1] "all_unordered_pairs"
#> 
#> $objective
#> [1] "log_likelihood + alpha/(N-1) * sum(log(p*(1-p)))"
#> 
#> $reference
#> [1] "doi:10.1111/jedm.70022, equation (3)"

These standard errors describe uncertainty conditional on the comparisons that actually occurred. They do not account for how an adaptive schedule might change if the assessment were repeated. Alpha adjustment does not select pairs and does not guarantee unbiased reliability for every adaptive schedule. Schedule-aware bootstrap bias correction is available in the bootstrap guide. For the mathematical definition, uncertainty assumptions and failure diagnostics, see ?fit_bt_model.

An experimental comparison with the Bayesian likelihood

fit_bt_model(..., engine = "lapse") jointly estimates sample strengths, a preference for the first presented sample, and a probability of guessing. It uses the same comparison probability as the Bayesian btl_e_b model, with unpenalized maximum likelihood. This helps investigate whether differences come from the probability model or from Bayesian versus frequentist estimation. It is a prototype, not a replacement for the Bayesian fit.

A connected set of comparisons is not always enough to separate these effects. The prototype stops when numerical or identification checks fail. The error retains available estimates and diagnostic details without standard errors or reliability.

A fitted guessing probability of exactly zero can be a valid result: the best fit needs no guessing component. When this boundary solution passes the required checks, the fit retains scores, positional bias, and predictions. Its convergence status is converged_boundary. Ordinary standard errors are unavailable because the usual uncertainty approximation does not apply at this boundary. The fit records uncertainty status nonregular_boundary, missing score standard errors, and no covariance matrices. You can still use predict() and summarize_bt_fit().

Valid interior fits provide local standard errors that include uncertainty in bias and guessing. Numerical checks alone do not prove that these errors are well calibrated across repeated samples. Conventional reliability (SSR) remains unavailable for every lapse fit, and the simple-BT bootstrap rejects these fits.

The original frozen synthetic assessment recovered 324 of 450 cases and rejected 100 boundary candidates and 26 numerical failures. With the same cases and recovery tolerances, the follow-up recovered all 450: 352 interior fits and 98 valid zero-boundary fits. There were no numerical or identification failures, and no valid fits failed parameter recovery. Read the follow-up with readLines(system.file("validation", "bt-lapse-305-boundary.txt", package = "pairwiseLLM")). The original evidence is preserved. These synthetic results do not establish production validity or uncertainty calibration. See ?fit_bt_model for the model, checks, numerical controls, and diagnostic fields.

Save an analysis

write.csv(scores, "writing_scores.csv", row.names = FALSE)
saveRDS(fit, "writing_bt_fit.rds")
# In a later session: fit <- readRDS("writing_bt_fit.rds")

The CSV is convenient for viewing scores in a spreadsheet. The RDS preserves the fitted R object, including the method and settings used to produce it. Keep this file if you need to revisit or explain your analysis later. These paths are relative to your current working directory.

Configure one provider

A backend selects the service; model selects a model offered by that service. This example uses OpenAI. Use an identifier available to your account; example identifiers are not a guarantee of current availability. See Backends and Tested Model Configurations for the provider/key table and official catalogs.

For the current R session:

Sys.setenv(OPENAI_API_KEY = "your-key-here")
check_llm_api_keys()

has_key = TRUE means R can see a nonempty value; it does not test whether the key is valid or the account can access a model. Missing keys for other providers are expected. Never put a real key in a shared script or report.

For persistent local setup, put OPENAI_API_KEY="your-key-here" in your ~/.Renviron file and restart R. If you use usethis, usethis::edit_r_environ() opens that file. This is a file you choose to store; the package does not save credentials for you. Ollama uses a local server and does not require a provider API key.

Submit the pairs

res_list <- submit_llm_pairs(
  pairs = pairs_small,
  backend = "openai",
  model = "gpt-4o",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  parallel = FALSE,
  save_path = "live_results.csv"
)

The run processes pairs sequentially and saves completed results incrementally. Use a distinct file for each study/model/trait configuration. To continue the same interrupted run, keep its inputs and saved file and repeat the call; matching saved pairs are skipped. Parallel execution is optional and requires future and future.apply; leave it off for a first run.

Check what succeeded

head(res_list$results)
res_list$failed_pairs
res_list$failed_attempts

An HTTP success alone is not a valid judgment. Inspect unresolved pairs before analysis; missing winners are not ties. The recovery guide explains retrying failures and keeping successful work.

Once you have collected an adequate comparison design and inspected failures, prepare its successful outcomes with build_bt_data(res_list$results) and use the same fit-and-inspect steps as the offline example. Do not interpret the small submission demonstration as an adequate design for all 20 samples.