--- title: "Guide: Linking Saved Comparisons" output: rmarkdown::html_vignette: toc: true vignette: > %\VignetteIndexEntry{Guide: Linking Saved Comparisons} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- This guide shows how to put two separately ranked sets of writing samples on a common scale using comparisons you already have. It uses small, invented data and makes no provider requests. You need only `pairwiseLLM` for the runnable examples. The reference set is the **hub** and the other set is the **spoke**. Their original, separate rankings are called **Phase A**. Linking them is called **Phase B**. Read the [linking overview](adaptive-linking.html) for help choosing inputs, or the [design guide](adaptive-linking-design.html) for the statistical models. ## Start with the samples and their comparisons Each set has two samples. The IDs below identify samples within their own set. Add unique `global_item_id` values if you will later reuse a rubric calibration. ```{r setup} library(pairwiseLLM) hub <- list(set_id = "hub", items = data.frame(item_id = c("h1", "h2"))) spoke <- list(set_id = "spoke", items = data.frame(item_id = c("s1", "s2"))) cross <- data.frame( observation_id = c("cross-1", "cross-2"), A_set = "hub", A_item = c("h1", "h2"), B_set = "spoke", B_item = c("s1", "s2"), y_A = c(0L, 1L) ) ``` Each row describes one judgment: `y_A = 1` means sample A won and `y_A = 0` means sample B won. Retain the order in which samples were presented. Give a new judgment a new ID even when the same pair is compared again. Keep any comparisons reserved for checking accuracy outside this table. For this example, assume there is no presentation-order bias or random lapse. For an analysis, use the shared judge settings estimated in Phase A; keep them fixed when linking. ```{r judge} judge <- list(beta = 0, epsilon = 0, model_variant = "btl", link = "logit", source = "invented example: no bias or lapses") ``` ## Choose a method and fit the first comparison There is no default method. We start with E1 (`fixed_shape_offset`) to illustrate the workflow. It keeps the original within-set score differences fixed and estimates how much to shift the spoke scores. ```{r first-fit} phase_a <- list( hub = list(points = c(h1 = -1, h2 = 1)), spoke = list(points = c(s1 = -.5, s2 = .5)) ) input <- prepare_link_input("fixed_shape_offset", hub, spoke, phase_a, cross[1, ], judge) state <- start_link_session(input) print(state) summarize_items(state) ``` Higher `theta_link_mean` values mean stronger estimated writing on the chosen trait. `theta_link_sd` describes uncertainty; it is not a rubric grade. For E1, this uncertainty concerns only the shift between sets. The earlier scores are treated as fixed, so the hub's zero uncertainty here does **not** mean its original scores were known perfectly. `theta_link_eap` is another name for `theta_link_mean`, retained for compatibility. E1 reports posterior averages. E2 and the usual E3 fit report the most probable joint scores, with approximate uncertainty. See the design guide before comparing uncertainty across methods. ## Save your work and add the next comparison ```{r save-resume} path <- tempfile(fileext = ".rds") save_link_session(state, path) restored <- load_link_session(path, input = input) stopifnot(identical(state, restored)) next_input <- prepare_link_input("fixed_shape_offset", hub, spoke, phase_a, cross, judge) continued <- resume_link_session(restored, next_input) summary(continued)[c("estimator_id", "delta_spoke_mean", "delta_spoke_sd", "phase_b_active_edges_used", "uncertainty_scope", "fit_valid", "status")] unlink(path) ``` `delta_spoke_mean` is the estimated shift from hub to spoke; positive values put the spoke higher. `delta_spoke_sd` describes uncertainty in that shift. `phase_b_active_edges_used` counts judgments used for fitting, including legitimate repeated judgments. Check `fit_valid` and the uncertainty description before interpreting the scores. When adding data, supply all previous rows in their original order followed by the new rows. Keep the same Phase A inputs, method and settings. Reusing unchanged input returns the saved fit without fitting again. Start a new session if you need to change those inputs or settings. `adaptive_get_logs(continued)` provides the record of the fits made along the way. ## Use saved uncertainty with E2 E2 (`gaussian_posterior_bridge`) uses joint posterior draws from each Phase A fit. Each row is one plausible set of scores; columns identify samples. The tiny matrices below are invented to demonstrate the required format, not recommended sample sizes. ```{r e2} hub_draws <- cbind(h1 = c(-1, -2, -.5, -1.5), h2 = c(1, 2, .5, 1.5)) spoke_draws <- cbind(s1 = c(-.3, -.7, -.4, -.6), s2 = c(.3, .7, .4, .6)) e2_input <- prepare_link_input("gaussian_posterior_bridge", hub, spoke, phase_a = list(hub = list(draws = hub_draws), spoke = list(draws = spoke_draws)), cross = cross, judge = judge) e2_fit <- fit_link(e2_input) e2_fit$offset ``` E2 can update both sets' scores and the shift between them. It needs the joint draws to retain relationships among uncertain scores. Separate standard errors cannot replace them. It does not fit the original Phase A comparisons a second time. ## Return to the original comparisons with E3 E3 (`joint_offset`) fits the original within-set comparisons together with the between-set comparisons, using each judgment once. It requires compatibility with the original Phase A model, including its prior assumptions. Saved scores alone are insufficient. These invented within-set rows use the same format as `cross`. ```{r e3} within_hub <- data.frame(observation_id = c("hub-1", "hub-2"), A_set = "hub", A_item = "h1", B_set = "hub", B_item = "h2", y_A = c(0L, 1L)) within_spoke <- data.frame(observation_id = c("spoke-1", "spoke-2"), A_set = "spoke", A_item = "s1", B_set = "spoke", B_item = "s2", y_A = c(0L, 0L)) e3_input <- prepare_link_input("joint_offset", hub, spoke, phase_a = list(hub = list(observations = within_hub), spoke = list(observations = within_spoke)), cross = cross, judge = judge) e3_fit <- fit_link(e3_input) e3_fit$offset ``` The usual E3 fit needs no sampler. An optional MCMC engine provides a more expensive statistical audit and requires CmdStan. It must be requested explicitly; see `?fit_link` and the design guide. It is not selected automatically when a fit fails. The three examples illustrate input formats. Their invented Phase A summaries were not estimated from the same comparisons, so their numerical results should not be used to judge which method works best. ## Link a prepared standalone reference For rubric scoring, the hub can be an import-ready adaptive Phase A result or a standalone Bayesian ranking saved with `prepare_linked_rubric_reference()`. The [rubric guide](rubric-calibration.html#prepare-a-reference-from-a-standalone-ranking) shows how to prepare and save the latter, including rankings based on pooled historical comparisons. A prepared `reference` provides the matching hub inputs for each method: | Method | Hub input | | --- | --- | | E1 | `list(points = reference$points, source = reference$source)` | | E2 | `list(draws = reference$posterior_draws, source = reference$source)` | | E3, including its MCMC engine | `list(observations = reference$evidence, source = reference$source)` | Always carry `reference$source` with the hub input. It identifies the frozen reference across methods. The package also checks the actual hub inputs before applying the saved rubric calibration; copying an identifier onto different scores or comparisons will not work. Here is an E2 example using a prepared reference and an existing target Phase A artifact. `active_cross` must use the set and sample IDs in these inputs. ```{r standalone-link, eval=FALSE} standalone_input <- prepare_link_input( "gaussian_posterior_bridge", hub = reference$hub, spoke = list(set_id = target_phase_a$set_id, items = target_phase_a$items[c("item_id", "global_item_id")]), phase_a = list( hub = list(draws = reference$posterior_draws, source = reference$source), spoke = list(artifact = target_phase_a) ), cross = active_cross, judge = reference$judge ) standalone_link <- fit_link(standalone_input) target_scores <- predict(standalone_calibration, standalone_link) ``` Use compatible model settings for the target ranking. `reference$judge` keeps the shared bias and lapse settings from the reference fit. E3 also requires compatible original prior assumptions, as described in the design guide. You do not need target rubric labels for linking or prediction. E2 and E3 may update hub scores during linking, but the saved rubric calibration stays attached to the original reference. Prediction restores that reference's score origin; it does not adjust the scores to match the new cohort's distribution. ## Predict a comparison and check the result To predict a winner, provide sample identities in their presentation order, without an outcome column: ```{r prediction} pairs_to_predict <- cross[, c("observation_id", "A_set", "A_item", "B_set", "B_item")] predict_link(e3_fit, pairs_to_predict) e3_fit$diagnostics[c("fit_valid", "failure_code", "uncertainty_scope")] ``` Predictions are probabilities that A wins. They average over the method's estimated uncertainty. Keep observed outcomes for checking those predictions in a separate table. An offset fitted without between-set comparisons reflects only its starting assumption and does not establish a link. Failed fits or unavailable uncertainty need investigation before you use the results. ## Use saved Phase A results or several spokes Instead of supplying points, draws or raw observations directly, you can supply `list(artifact = artifact)` for each set's Phase A input. Read saved files with `readRDS()` first. The artifact must contain what the selected method needs and must describe the original Phase A fit. Compatible older Phase A results remain usable; linked Phase B results cannot substitute for them. For several spokes, prepare one input per spoke and pass the list to `start_link_session()`. All must share the same hub information, judge settings and estimator. Reports include `link_spoke_id`: keep the separate hub estimates associated with their spokes. The package does not combine them into one hub fit. The optional status labels `active`, `probe` and `frozen` help record your workflow. They do not decide when enough comparisons have been collected. A frozen spoke must be explicitly reactivated before adding observations. Automatic Phase B selection and stopping remain unavailable pending validation. The [rubric guide](rubric-calibration.html) explains how to apply a saved hub calibration to linked spoke scores. Old anchored-joint Phase B sessions must be restarted from compatible Phase A results; they cannot be resumed as E1, E2 or E3. ## Dependency upgrades and saved hashes The package requires rlang >= 1.3.0. That release changed the hash algorithm; hashes created with earlier versions cannot be verified with the new engine. When recorded provenance establishes this incompatibility, loading or validating raises `pairwiseLLM_link_hash_incompatibility`, identifying the saved scheme, saved rlang version, and current version. A tampered current artifact remains an integrity error. An older artifact without engine provenance cannot establish which version produced its hashes; failed verification does not prove that a dependency upgrade caused it. Keep the original artifacts and their hashes. To build a **new** input, supply the original saved arguments, judgments, and compatible Phase A points or draws: ```{r rebuild-link-input, eval=FALSE} args <- readRDS("original-link-input-arguments.rds") new_input <- do.call(prepare_link_input, args) saveRDS(new_input, "rebuilt-link-input.rds") ``` Preparing this input makes no LLM requests and performs no sampling. It does not repair or resume the old session. Reuse saved original outcomes; new judgments are unnecessary. Do not replace the old input's recorded hashes with new ones. Old normalized inputs are not a substitute for original Phase A payloads and their recorded centering/source information. A standalone reference can be prepared again from a saved completed fit and its exact evidence only if the fit-time evidence identity still verifies. That identity also uses rlang hashing, so some older completed fits cannot be reused this way. Investigate with the original producing environment before separately deciding whether to refit. A new E3 result refits the saved outcomes, and choosing E3-MCMC can require expensive sampling. Loading, validation, and input preparation never initiate that work automatically. A rebuilt reference has a new identity; dependent calibration/linking artifacts must be prepared explicitly as well. ## Citation Mercer, S. H. (2026). *Guide: Linking Saved Comparisons*. In *pairwiseLLM* [R package vignette]. Comprehensive R Archive Network.