Advanced: Submitting and Polling Multiple Batches

Overview

This vignette demonstrates how to use pairwiseLLM for Batch API workflows (server-side batching), which are distinct from the live API calls described in the Getting Started with pairwiseLLM vignette.

Batch workflows can suit larger fixed-pair jobs when turnaround is flexible. They:

• Allow submitting thousands of pairs at once
  
• Are often cheaper (e.g., discounted batch pricing on some providers)
  
• Continue remotely after submission even if your R session ends
• Can be polled and resumed even if your local R session ends

OpenAI Decisions and TypeSafe Jev support live multi-pair collection through submit_llm_pairs(), including concurrency and resumption, but cannot enter these provider-managed Batch workflows. See the decision recovery guide.

Supported Batch API providers:

• OpenAI (batch pipeline: run_openai_batch_pipeline())
• Anthropic (batch pipeline: run_anthropic_batch_pipeline())
• Gemini (batch pipeline: run_gemini_batch_pipeline())

Vertex AI Gemini API is intentionally excluded here. In this series, Vertex is supported on the live path only via backend = "vertex" with VERTEX_API_KEY, and generic batch wrappers reject Vertex explicitly.

Recommended approach: For multiple batches (e.g., templates × providers × models × forward/reverse), use:

• llm_submit_pairs_multi_batch() to split + submit jobs (no polling; writes an optional registry CSV)
• llm_resume_multi_batches() to poll + download + parse results (can resume from a registry on disk)

These helpers orchestrate the provider-specific pipelines without forcing you to write your own polling loops.

Note: Together.ai and Ollama do not currently support a native Batch API compatible with this workflow. For those providers, use the live API wrapper submit_llm_pairs() as described in Getting Started with pairwiseLLM.

In this vignette, we will cover:

• Designing a grid of provider/model/thinking/direction combinations
• Submitting many batch jobs using the multi-batch helpers
• Polling and resuming safely via on-disk registries
• Producing per-run and merged results tables

Submission, polling, and downloads can still fail or time out; saving job IDs lets you recover existing work. The local setup and pair preparation below run without credentials. Submitted examples require a provider account and may incur charges. This guide uses dplyr, tidyr, purrr, readr, and stringr; install these optional packages if needed. In particular, readr is needed to save and reload the job registry even in the first single-batch example.

Note: All heavy API calls in this vignette are set to eval = FALSE so that the vignette remains CRAN-safe. You can enable them in your own project.

For basic function usage, see the companion vignette:

• Getting Started with pairwiseLLM

For prompt evaluation and positional-bias diagnostics, see the companion vignette:

• Prompt Template Positional Bias Testing

Setup and API Keys

library(pairwiseLLM)
library(dplyr)
library(tidyr)
library(purrr)
library(readr)
library(stringr)

Configure the environment variable only for each provider included in your run plan:

This vignette only covers batch-capable providers. Vertex live requests use a separate VERTEX_API_KEY surface and are documented in Getting Started with pairwiseLLM.

Check which are set:

check_llm_api_keys()

Example Data and Prompt Template

We use the built-in writing samples and a single trait (overall_quality).

data("example_writing_samples", package = "pairwiseLLM")

td <- trait_description("overall_quality")
td
#> $name
#> [1] "Overall Quality"
#> 
#> $description
#> [1] "Overall quality of the writing, considering how well ideas are expressed,\nhow clearly the writing is organized, and how effective the language and\nconventions are."

Default prompt template:

tmpl <- set_prompt_template()
cat(substr(tmpl, 1, 400), "...
")
#> You are a debate adjudicator. Your task is to weigh the comparative strengths of two writing samples regarding a specific trait.
#> 
#> TRAIT: {TRAIT_NAME}
#> DEFINITION: {TRAIT_DESCRIPTION}
#> 
#> SAMPLES:
#> 
#> === SAMPLE_1 ===
#> {SAMPLE_1}
#> 
#> === SAMPLE_2 ===
#> {SAMPLE_2}
#> 
#> EVALUATION PROCESS (Mental Simulation):
#> 
#> 1.  **Advocate for SAMPLE_1**: Mentally list the single strongest point of evidence that makes SAMPLE_1 the  ...

Construct a modest number of pairs to keep the example light:

set.seed(123)

pairs_all <- example_writing_samples |>
  make_pairs()

n_pairs <- min(40L, nrow(pairs_all))

pairs_forward <- pairs_all |>
  sample_pairs(n_pairs = n_pairs, seed = 123) |>
  randomize_pair_order(seed = 456)

pairs_reverse <- sample_reverse_pairs(
  pairs_forward,
  reverse_pct = 1.0,
  seed        = 789
)

get_pairs_for_direction <- function(direction = c("forward", "reverse")) {
  direction <- match.arg(direction)
  if (identical(direction, "forward")) {
    pairs_forward
  } else {
    pairs_reverse
  }
}

First batch: submit once, then resume

Start with one provider and the pairs_forward, td, and tmpl objects created above. This example creates one remote batch and a local job registry. Use a new output directory for a new job; keep it to resume the same job.

first_job <- llm_submit_pairs_multi_batch(
  pairs = pairs_forward,
  backend = "openai",
  model = "gpt-4.1",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  n_segments = 1,
  output_dir = "first_batch",
  write_registry = TRUE
)

Later, including in a new R session, load the saved registry to poll and download that same job. Do not rerun submission merely because it has not finished.

library(pairwiseLLM)
first_results <- llm_resume_multi_batches(
  jobs = NULL,
  output_dir = "first_batch",
  write_combined_csv = TRUE
)
head(first_results$combined)

Inspect the registry and per-job results before treating the combined rows as a complete collection. Preserve error files and reconcile unsuccessful request IDs; a returned table alone does not establish that every requested comparison succeeded. For a simpler single-provider pipeline without multi-job registry management, see llm_submit_pairs_batch() and llm_download_batch_results().

OpenAI storage, output limits, and error files

OpenAI Batch accepts store = TRUE or FALSE on both endpoints. Omitted or NULL values preserve provider defaults: Responses are stored for later API retrieval by default. store = FALSE disables that response storage; it does not disable Batch input/output/error file retention or imply zero data retention. See OpenAI data controls.

Select endpoint = "responses" explicitly when setting max_output_tokens. The limit includes both visible output and reasoning tokens. These controls pass through run_openai_batch_pipeline() and llm_submit_pairs_batch().

job <- run_openai_batch_pipeline(
  pairs = pairs_forward,
  model = "gpt-5.6-terra",
  trait_name = "Cohesion",
  trait_description = "How clearly the ideas connect.",
  endpoint = "responses",
  reasoning = "none",
  store = FALSE,
  max_output_tokens = 64,
  poll = FALSE
)
batch <- openai_poll_batch_until_complete(job$batch$id)
if (!is.null(batch$output_file_id) && nzchar(batch$output_file_id)) {
  openai_download_batch_output(batch$id, "batch_output.jsonl")
}
if (!is.null(batch$error_file_id) && nzchar(batch$error_file_id)) {
  openai_download_batch_errors(batch$id, "batch_errors.jsonl")
  errors <- lapply(readLines("batch_errors.jsonl"), jsonlite::fromJSON)
}

Successful output and request failures are separate files. Reconcile both with the submitted requests by custom_id, never line order. The error downloader preserves the raw JSONL and works when no output file exists. It errors clearly when no error file is available. Batch-level validation errors can instead appear in batch$errors. The pipeline’s poll = TRUE path still expects an output file; the separate steps above also handle errors-only batches. See the OpenAI Batch guide.

OpenAI Batch prompt caching and costs

For verified GPT-5.6 and later model IDs, new package-built Batch requests turn off implicit caching by default. Both endpoints receive exactly prompt_cache_options: {"mode": "explicit"} without explicit breakpoints. This avoids cache writes for unique long prompts. Reuse can instead make caching cheaper: opt in with prompt_caching = "implicit", which omits the field and keeps provider caching available. The same option passes through build_openai_batch_requests(), run_openai_batch_pipeline(), llm_submit_pairs_batch(backend = "openai"), and llm_submit_pairs_multi_batch(backend = "openai").

The supported allowlist, verified on 2026-10-09, is gpt-5.6-luna, gpt-5.6-terra, gpt-5.6-sol, gpt-6-luna, gpt-6-sol, gpt-6-astra, and gpt-6.1-sol. Recognized earlier model names retain their historical bodies for omitted/NULL policy, but reject "disabled". Unknown IDs, including unlisted snapshots, fine-tuned/custom aliases and future models, require "implicit" or a verified ID. The allowlist is offline and requires a package update to add support; a newer-looking version alone is insufficient.

requests <- build_openai_batch_requests(
  pairs = pairs_forward,
  model = "gpt-5.6-luna",
  trait_name = "Cohesion",
  trait_description = "How clearly the ideas connect.",
  endpoint = "responses",
  prompt_caching = "implicit"
)

The builder accepts only this policy, not raw provider cache options or breakpoints. prompt_cache_key concerns routing/accounting, prompt_cache_retention and prompt_cache_options.ttl concern lifetime, and store = FALSE concerns response storage. None disables implicit caching. See the OpenAI prompt-caching guide.

parse_openai_batch_output() reports prompt_cached_tokens and prompt_cache_write_tokens from either endpoint. Missing counters remain NA, even when the submitted policy was disabled. Use the parsed raw output for billing audits: normalized comparison tables can exclude failed or unparseable judgments that still consumed tokens. Reconcile all provider output/error records and any retried submissions before estimating total spend.

Compute ordinary input as total input minus cached reads minus cache writes. Multiply each category by its own applicable rate. As documented on 2026-10-09, GPT-5.6+ writes cost 1.25 times ordinary input; reads cost 0.1 times on most models, but 0.05 times on GPT-6.1 Sol. Select rates for the actual model, context length and Batch tier from OpenAI pricing. When using Batch rates, do not apply its discount a second time. Output tokens and any additional charges are separate.

This executable example uses hypothetical rates per million tokens, already chosen for the desired billing tier. It demonstrates calculation, not savings measured on a provider workload. Replace the synthetic counts with parsed usage and the rates with those applicable to each request.

cache_usage <- tibble::tibble(
  scenario = c("unique prefix write", "reused prefix", "missing counters"),
  prompt_tokens = c(4000, 4000, 4000),
  prompt_cached_tokens = c(0, 3000, NA_real_),
  prompt_cache_write_tokens = c(3000, 0, NA_real_)
)
input_rates <- c(ordinary = 1, cached = 0.1, write = 1.25)
cache_costs <- cache_usage |>
  dplyr::mutate(
    ordinary_tokens = prompt_tokens - prompt_cached_tokens - prompt_cache_write_tokens,
    input_charge = (ordinary_tokens * input_rates[["ordinary"]] +
      prompt_cached_tokens * input_rates[["cached"]] +
      prompt_cache_write_tokens * input_rates[["write"]]) / 1e6,
    uncached_input_charge = prompt_tokens * input_rates[["ordinary"]] / 1e6
  )
cache_costs
#> # A tibble: 3 × 7
#>   scenario            prompt_tokens prompt_cached_tokens prompt_cache_write_to…¹
#>   <chr>                       <dbl>                <dbl>                   <dbl>
#> 1 unique prefix write          4000                    0                    3000
#> 2 reused prefix                4000                 3000                       0
#> 3 missing counters             4000                   NA                      NA
#> # ℹ abbreviated name: ¹​prompt_cache_write_tokens
#> # ℹ 3 more variables: ordinary_tokens <dbl>, input_charge <dbl>,
#> #   uncached_input_charge <dbl>
# Missing counters make the total unknown; do not hide them with na.rm = TRUE.
sum(cache_costs$input_charge)
#> [1] NA

Submission validates the policy before IO, including for zero rows. Multi-batch segments and retries reuse the resolved policy. Resume only retrieves existing jobs; it does not change caching or resubmit their requests.

This change applies only to future JSONL built by the package. Frozen W007/W010 Luna requests, running adaptive analyses and their installed libraries remain untouched. Separately authored W-series builders need their own explicit code audit, authorized package repin where applicable, and JSONL preflight; updating this package does not update those builders.

Designing the Batch Grid

Suppose we want to test several prompt templates across:

• Anthropic models (with/without “thinking”)
• OpenAI models (with/without “thinking” for specific models)
• Gemini models (with “thinking” enabled)

Here we define a small grid:

anthropic_models <- c(
  "claude-haiku-4-5-20251001"
)

gemini_models <- c(
  "gemini-3.8-flash"
)

openai_models <- c(
  "gpt-4.1",
  "gpt-5.6-luna"
)

thinking_levels <- c("no_thinking", "with_thinking")
directions <- c("forward", "reverse")

anthropic_grid <- tidyr::expand_grid(
  provider  = "anthropic",
  model     = anthropic_models,
  thinking  = "no_thinking",
  direction = directions
)

gemini_grid <- tidyr::expand_grid(
  provider  = "gemini",
  model     = gemini_models,
  thinking  = "with_thinking",
  direction = directions
)

openai_grid <- tidyr::expand_grid(
  provider  = "openai",
  model     = openai_models,
  thinking  = "no_thinking",
  direction = directions
)

batch_grid <- dplyr::bind_rows(
  anthropic_grid,
  gemini_grid,
  openai_grid
)

batch_grid
#> # A tibble: 8 × 4
#>   provider  model                     thinking      direction
#>   <chr>     <chr>                     <chr>         <chr>    
#> 1 anthropic claude-haiku-4-5-20251001 no_thinking   forward  
#> 2 anthropic claude-haiku-4-5-20251001 no_thinking   reverse  
#> 3 gemini    gemini-3.8-flash          with_thinking forward  
#> 4 gemini    gemini-3.8-flash          with_thinking reverse  
#> 5 openai    gpt-4.1                   no_thinking   forward  
#> 6 openai    gpt-4.1                   no_thinking   reverse  
#> 7 openai    gpt-5.6-luna              no_thinking   forward  
#> 8 openai    gpt-5.6-luna              no_thinking   reverse

We will also imagine multiple prompt templates have been registered. For simplicity, we use the same tmpl string, but in practice you would substitute different text:

templates_tbl <- tibble::tibble(
  template_id     = c("test1", "test2", "test3", "test4", "test5"),
  prompt_template = list(tmpl, tmpl, tmpl, tmpl, tmpl)
)

templates_tbl
#> # A tibble: 5 × 2
#>   template_id prompt_template
#>   <chr>       <list>         
#> 1 test1       <chr [1]>      
#> 2 test2       <chr [1]>      
#> 3 test3       <chr [1]>      
#> 4 test4       <chr [1]>      
#> 5 test5       <chr [1]>

Submitting Many Batches with the Multi‑Batch Helpers

The key idea is:

• Each combination of (template_id, provider, model, thinking, direction) becomes a run
• Each run writes its files into its own subdirectory (so file names never collide)
• Within each run you can still split into multiple segments using batch_size or n_segments

out_root <- "dev-output/advanced-multi-batch"
dir.create(out_root, recursive = TRUE, showWarnings = FALSE)

run_plan <- tidyr::crossing(
  templates_tbl |> tidyr::unnest(prompt_template),
  batch_grid
) |>
  mutate(
    run_id = paste(template_id, provider, model, thinking, direction, sep = "__"),
    run_id = gsub("[^A-Za-z0-9_.-]+", "-", run_id),
    run_dir = file.path(out_root, run_id)
  )

run_plan |> dplyr::select(run_id, template_id, provider, model, thinking, direction, run_dir)

submit_one_run <- function(template_id, prompt_template, provider, model, thinking, direction, run_dir) {
  pairs_use   <- get_pairs_for_direction(direction)
  is_thinking <- identical(thinking, "with_thinking")

  # Provider-specific knobs (passed through via ...)
  extra_args <- list()

  if (identical(provider, "openai")) {
    # Only request thoughts for models that support them in this workflow
    extra_args$include_thoughts <- is_thinking && grepl("^gpt-5", model)
    extra_args$include_raw      <- TRUE
  } else if (identical(provider, "anthropic")) {
    extra_args$reasoning        <- if (is_thinking) "enabled" else "none"
    extra_args$include_thoughts <- is_thinking
    extra_args$include_raw      <- TRUE
    # Optional: set deterministic temperature when not using reasoning
    if (!is_thinking) extra_args$temperature <- 0
  } else if (identical(provider, "gemini")) {
    extra_args$include_thoughts <- TRUE
    extra_args$thinking_level   <- "low"   # example
    extra_args$include_raw      <- TRUE
  }

  message(
    "Submitting: ", template_id, " | ", provider, " / ", model,
    " / ", thinking, " / ", direction
  )

  # Split strategy:
  # - For real jobs, use batch_size (e.g., 500–5000) or n_segments (e.g., 10–50)
  # - Here we keep it simple and submit a single segment per run
  do.call(
    llm_submit_pairs_multi_batch,
    c(
      list(
        pairs             = pairs_use,
        backend           = provider,
        model             = model,
        trait_name        = td$name,
        trait_description = td$description,
        prompt_template   = prompt_template,
        n_segments        = 1L,
        output_dir        = run_dir,
        write_registry    = TRUE,
        verbose           = TRUE
      ),
      extra_args
    )
  )
}

run_results <- purrr::pmap(
  run_plan |>
    dplyr::select(
      template_id, prompt_template, provider, model, thinking, direction,
      run_dir
    ),
  submit_one_run
)

# Store a lightweight manifest so you can resume later without rebuilding run_plan
manifest <- run_plan |>
  mutate(registry_path = file.path(run_dir, "jobs_registry.csv"))

manifest_path <- file.path(out_root, "run_manifest.csv")
readr::write_csv(manifest, manifest_path)

manifest_path

Polling, Downloading, and Parsing (Resumable)

To poll all runs, read the manifest and call llm_resume_multi_batches() for each run_dir. If you restart R, you can resume without keeping the jobs objects in memory by setting jobs = NULL and pointing to output_dir (the function will load jobs_registry.csv).

manifest_path <- file.path(out_root, "run_manifest.csv")
manifest <- readr::read_csv(manifest_path, show_col_types = FALSE)

poll_one_run <- function(run_dir) {
  llm_resume_multi_batches(
    jobs               = NULL,   # load from jobs_registry.csv in run_dir
    output_dir         = run_dir,
    interval_seconds   = 60,
    per_job_delay      = 2,
    write_results_csv  = TRUE,   # writes batch_XX_results.csv files
    write_registry     = TRUE,   # refreshes jobs_registry.csv with done flags
    keep_jsonl         = TRUE,
    verbose            = TRUE,
    write_combined_csv = TRUE,   # writes combined_results.csv inside run_dir
    combined_csv_path  = "combined_results.csv"
  )
}

polled <- purrr::map(manifest$run_dir, poll_one_run)

combined_all <- purrr::map2_dfr(
  polled,
  seq_len(nrow(manifest)),
  function(res, i) {
    meta <- manifest[i, ]
    if (is.null(res$combined)) return(NULL)

    res$combined |>
      mutate(
        template_id = meta$template_id,
        provider    = meta$provider,
        model       = meta$model,
        thinking    = meta$thinking,
        direction   = meta$direction,
        run_id      = meta$run_id
      )
  }
)

combined_path <- file.path(out_root, "combined_all_runs.csv")
readr::write_csv(combined_all, combined_path)

combined_path

Resuming After Interruption

Resuming jobs is possible:

• Submission writes jobs_registry.csv under each run directory
• Polling can be restarted at any time by calling llm_resume_multi_batches(jobs = NULL, output_dir = <run_dir>)
• If you keep a run_manifest.csv with run_dir paths, resuming all runs is just a loop

Example: resume only unfinished runs (based on each run’s registry):

manifest <- readr::read_csv(file.path(out_root, "run_manifest.csv"), show_col_types = FALSE)

needs_poll <- function(run_dir) {
  reg_path <- file.path(run_dir, "jobs_registry.csv")
  if (!file.exists(reg_path)) return(FALSE)
  reg <- readr::read_csv(reg_path, show_col_types = FALSE)
  any(!as.logical(reg$done))
}

unfinished_dirs <- manifest$run_dir[vapply(manifest$run_dir, needs_poll, logical(1))]

polled <- purrr::map(unfinished_dirs, poll_one_run)

Next Steps

Once you have per-run results CSVs (e.g., one per template × model × thinking × direction), you can:

• Compute reverse consistency with compute_reverse_consistency()
• Analyze positional bias with check_positional_bias()
• Aggregate results by provider/model/template using standard dplyr pipelines
• Fit Bradley–Terry models with build_bt_data() + fit_bt_model()
• Fit Elo models with fit_elo_model() (when EloChoice is installed)

Citation

Mercer, S. H. (2026). Advanced: Submitting and polling multiple batches [R package vignette]. Comprehensive R Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM