---
title: "Provider Controls and Recovery"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 2
vignette: >
  %\VignetteIndexEntry{Provider Controls and Recovery}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

This article explains the package's reasoning/thinking controls and how to recover useful work after
provider failures. Model availability changes independently of package behavior. Consult
[Backends and Tested Model Configurations](https://shmercer.github.io/pairwiseLLM/articles/model-compatibility.html)
for the dated registry rather
than treating examples here as a current model catalog. The registry preserves dated generative tests from 2026-09-05 and decision-model
tests from 2026-10-16; a past success does not guarantee present availability.

## Start with the symptom

You need the original inputs and any saved result files or job IDs. The goal is
to identify what succeeded and continue the remaining work. For a first request,
use [Getting Started](getting-started.html); add provider controls only when your
chosen model or study needs them.

| What you see | What to inspect or do next |
| --- | --- |
| Some live pairs have no winner | Inspect `failed_pairs` and `failed_attempts`; preserve `results`. |
| A batch is pending | Poll the saved job ID or resume its registry; avoid a second submission. |
| Polling or download hits a temporary network/server error | Retrieval retries transient errors; if exhausted, resume the existing job later. |
| An invalid key or job ID | Correct the credential/identifier; permanent errors are surfaced immediately. |
| Adaptive attempts produce no committed judgments | Inspect `judge_invalid_reason` and provider details in the step log. |
| A local model is unavailable or runs out of memory | Check the Ollama server/tag, then model size, context, and concurrency. |

## Storage, sampling, and service tiers

Omitted `temperature` and `top_p` leave model/provider sampling defaults in effect;
explicit values are forwarded subject to endpoint constraints. A `service_tier`
is a provider-specific processing option, not a portable speed or price promise.
OpenAI generative live requests accept `"standard"`/`"default"`, `"auto"`, `"flex"`, and
`"priority"`; explicit standard/default sends `"default"`. Gemini Developer API
and Vertex use `"standard"`, `"flex"`, or `"priority"`, through different request
surfaces. Availability is provider- and model-dependent. See `llm_compare_pair()`
and the selected provider's help for supported arguments.

For OpenAI generative endpoints, `store` controls response storage; for Gemini Developer API it controls
request logging. `NULL` omits the setting, preserving provider/project defaults.
Neither `store = FALSE` nor omission establishes a general zero-retention policy.
For OpenAI Batch storage, output limits, and separate error files, see the
[batch guide](advanced-batch-workflows.html#openai-storage-output-limits-and-error-files).

## Controls by backend

| Backend | Public controls | Important constraint |
|---|---|---|
| OpenAI generative | `reasoning`, `include_thoughts`, endpoint sampling controls | Reasoning support depends on model and endpoint; unsupported combinations abort during request normalization. |
| OpenAI Decisions | `decision_template`, `prompt_cache_options` | Live only; generative sampling, reasoning, storage and tier controls are rejected. |
| TypeSafe Jev | `decision_template` | Live only; sampling, reasoning, caching, storage and tier controls are unsupported. |
| Anthropic | `reasoning = "none"` or `"enabled"`, `include_thoughts`, `thinking_budget_tokens` | Extended thinking requires `temperature = 1` and `1024 <= thinking_budget_tokens < max_tokens`. Claude 5 adaptive thinking is not implemented or advertised. |
| Gemini Developer API | `thinking_level`, `include_thoughts`, `service_tier` | Accepted thinking levels depend on the detected model family; `service_tier` is `NULL`/`"standard"`, `"flex"`, or `"priority"`. |
| Vertex | `thinking_level` or `thinking_budget`, `include_thoughts`, `service_tier` | Use the control supported by the selected Gemini family; Vertex batch is not implemented. |
| Together | OpenAI-compatible sampling arguments | The package does not expose a separate tested reasoning mode for this backend. |
| Ollama | `think`, `include_thoughts`, `num_ctx`, sampling options | Capabilities and memory use depend on the locally installed tag and server. |

Omitting a sampling or reasoning option generally leaves the provider/model default in effect; it
does not mean that every provider uses the same value. `include_thoughts` controls whether available
reasoning text is requested or retained, but providers may return summaries, redacted blocks, or no
reasoning text. Never make a workflow depend on hidden reasoning being present.

The exact tested request profiles are in the installed registry:

```{r compatibility-registry}
library(pairwiseLLM)

registry_path <- system.file("extdata", "model_compatibility.csv", package = "pairwiseLLM")
if (!nzchar(registry_path)) {
  source_paths <- c(
    file.path("inst", "extdata", "model_compatibility.csv"),
    file.path("..", "inst", "extdata", "model_compatibility.csv")
  )
  registry_path <- source_paths[file.exists(source_paths)][1]
}
registry <- utils::read.csv(registry_path, check.names = FALSE)

unique(registry[c(
  "backend", "endpoint", "reasoning_mode", "package_version", "test_date", "status"
)])
```

## Live calls: retain successes and failures

`submit_llm_pairs()` and backend-specific row-wise submitters return three views:

- `results`: normalized rows with a valid winner belonging to the submitted pair;
- `failed_pairs`: source pairs that did not produce a valid judgment;
- `failed_attempts`: retry- or attempt-level HTTP, timeout, parse, refusal, and validation details.

```{r live-shape, eval=FALSE, purl=TRUE}
# Live API example; requires the selected provider credential and may incur cost.
library(pairwiseLLM)
data("example_writing_samples", package = "pairwiseLLM")
pairs <- make_pairs(example_writing_samples) |> sample_pairs(n_pairs = 10, seed = 123)
trait <- trait_description("overall_quality")
template <- get_prompt_template("default")
result <- submit_llm_pairs(
  pairs = pairs,
  backend = "openai",
  model = "gpt-5.6-luna",
  endpoint = "responses",
  trait_name = trait$name,
  trait_description = trait$description,
  prompt_template = template,
  reasoning = "none",
  save_path = "live-results.csv",
  parallel = FALSE
)

result$results
result$failed_pairs
result$failed_attempts
```

With `save_path`, completed rows are written incrementally and matching pairs are skipped on a later
call. Preserve the saved file, retry only `failed_pairs`, and re-bind results after confirming that
the retry did not duplicate successful `custom_id` values. A returned HTTP success is not a valid
judgment unless parsing also produced a winner belonging to the submitted pair.

### Decision models: save, resume, and inspect

Use `submit_llm_pairs()` with `backend = "openai", endpoint = "decisions",
model = "gpt-6-luna"`, or `backend = "typesafe", model = "jev-1.13.0"`.
Configure `OPENAI_API_KEY` or `TYPESAFE_API_KEY`; explicit `api_key` values take
precedence. Omit generative prompts and controls, including explicit `NULL`
settings for unsupported options. See the [prompts guide](data-and-prompts.html#decision-instructions-and-probabilities)
for custom traits and decision instructions.

```{r decision-resume, eval=FALSE}
# Requires TYPESAFE_API_KEY and incurs charges for unresolved work.
result <- submit_llm_pairs(
  pairs, backend = "typesafe", model = "jev-1.13.0",
  trait_name = trait$name, trait_description = trait$description,
  save_path = "jev-decisions.csv", parallel = FALSE,
  execution_control = list(requests_per_minute = 30, retry_ambiguous = FALSE)
)
# Repeat this same call after interruption; preserve the .checkpoint directory.
result$failed_pairs
result$failed_attempts
```

Saved decision runs require optional `filelock`. The `.checkpoint` directory
beside `save_path` is the authoritative record; CSV alone cannot resume a run.
Keep it with its export. Repeating the same call reuses committed decisions;
changing samples, presentation order, trait, instructions, model, workset, or
OpenAI cache controls requires a separate save path. Corrupt checkpoints stop
before calls; preserve them for inspection instead of editing them. Prefer a
local filesystem with reliable locking and atomic renames. Cloud-synchronized
folders do not establish cross-machine locking or power-loss durability.

Sequential calls need no asynchronous packages. Concurrent collection uses
optional `promises` and `later` with `parallel = TRUE, workers = 4`; workers
limit simultaneous requests. Set request limits for your account and workload;
token limiting additionally needs your token estimates, including an output
allowance. These controls cover one run, not all activity on the account, and
token reservations are estimates. Retry budgets and recent reservations survive
restart. Temporary errors receive bounded retries; invalid decisions stay
unresolved. In-flight requests may already have incurred charges. By default,
one ambiguous reissue is allowed across restarts; the example disables it.
Checkpoints do not guarantee exactly-once provider execution.

OpenAI Decisions uses its own `prompt_cache_options`, separate from generative
Batch caching controls. In this helper, `list(mode = "explicit")` disables
caching because no explicit breakpoints are added. Omission uses implicit
caching; caching does not guarantee zero retention. TypeSafe exposes no caching
or retention control here. Full limits and recovery details are in
[`decision_bulk`](../reference/decision_bulk.html) and the provider helpers.
Compare reversal probabilities after mapping positions back to the same sample
ID, and inspect both probabilities alongside the selected winner.

## Batch recovery

Batch workflows separate local request creation, remote submission, polling, download, and parsing.
Keep the provider job ID, input JSONL, downloaded output/error files, and parser results together.
The multi-batch helpers write `jobs_registry.csv`; after interruption, use
`llm_resume_multi_batches(jobs = NULL, output_dir = ...)` to reload that registry.

Do not resubmit merely because a remote job is still pending. Poll the existing job first. If a job
reaches a terminal failure, preserve its status/error payload, isolate the affected request IDs, and
submit only those requests after correcting the cause. See
[Advanced: Submitting and Polling Multiple Batches](https://shmercer.github.io/pairwiseLLM/articles/advanced-batch-workflows.html)
for the complete unevaluated API flow.

Transient batch status/download failures are retried without creating a new job.
Multi-batch resume defers exhausted transient retrievals to later polling rounds;
permanent authentication, invalid-ID, parsing, and local-write errors surface
immediately. Polling timeouts are checked between requests, so a request and its
retries can extend the elapsed time beyond the configured polling limit.

## Adaptive recovery

An invalid adaptive judgment consumes an attempted step but is not committed to the comparison
history and does not advance the refit cadence. Inspect `judge_invalid_reason`, `llm_status_code`,
and `llm_error_message` in `adaptive_step_log()`. Validate and load an existing session before
adding budget; never overwrite a failed directory merely to force resume.

## Ollama setup and resources

Install Ollama separately, start its server, and pull a model before calling the package. Tags are
local and environment-dependent. Large models and context windows can exhaust accelerator or system
memory; reduce `num_ctx`, concurrency, or model size when requests are evicted or time out.

```{r ollama, eval=FALSE, purl=TRUE}
# Local-system example; requires the Ollama CLI/server and a previously pulled tag.
options(pairwiseLLM.ollama_host = "http://127.0.0.1:11434")
ensure_only_ollama_model_loaded("qwen3:32b")
```

`ensure_only_ollama_model_loaded()` runs `ollama ps`, parses the first whitespace-delimited field,
and calls `ollama stop` for every active model except the requested one. It does not pull or load the
requested model. If listing fails or cannot be parsed, it returns without unloading anything; a
failed stop is reported and processing continues. Because unloading affects the machine-wide
server, do not use it when another process may need those models.

## Privacy and retention

Cloud requests transmit sample and prompt text to the selected provider. Raw responses and thoughts
may repeat submitted text and increase storage. In-memory decision failure objects
may retain raw responses even with `include_raw = FALSE`; saving an entire
returned object can persist them and retained source text columns. Default decision checkpoints and ordinary
exports omit raw responses. See `decision_bulk` for detailed retention rules. Set `include_raw = TRUE` only under an appropriate
privacy and retention policy, and avoid placing credentials or raw responses in registries,
vignettes, bug reports, or version control.

## Estimate costs before a larger run

For large jobs, it is often useful to estimate token usage and cost before launching a live run or submitting a batch.
`pairwiseLLM` includes `estimate_llm_pairs_cost()`, which runs a small **pilot** (paid live calls) and then estimates
the rest of the job by calibrating input tokens from prompt byte length.

The output includes both:

- **Expected cost** (using mean output tokens from usable pilot calls)
- **Budget cost** (using a high quantile of pilot output tokens, controlled by `budget_quantile`)

If you are running a discounted batch workflow, set `mode = "batch"` and supply a `batch_discount` multiplier.

```{r cost-pilot, eval=FALSE}
library(pairwiseLLM)
data("example_writing_samples", package = "pairwiseLLM")

# Create a moderate set of pairs
pairs_big <- example_writing_samples |>
  make_pairs() |>
  sample_pairs(n_pairs = 100, seed = 123) |>
  randomize_pair_order(seed = 456)

td   <- trait_description("overall_quality")
tmpl <- set_prompt_template()

est <- estimate_llm_pairs_cost(
  pairs = pairs_big,
  backend = "anthropic",                # "openai", "anthropic", "gemini", "together"
  model = "claude-sonnet-4-5",
  trait_name = td$name,
  trait_description = td$description,
  prompt_template = tmpl,
  mode = "batch",
  batch_discount = 0.5,                 # set to 1 for no discount
  n_test = 10,                          # paid pilot calls (live)
  budget_quantile = 0.9,                # p90 output tokens
  cost_per_million_input = 3.0,         # fill in your provider pricing
  cost_per_million_output = 15.0
)

est$summary
```

### Deterministic calculation example

Suppose a two-pair live pilot records input-token counts of 20 and 40 and
completion-token counts of 10 and 30. The prompt-byte calibration predicts 60
and 80 input tokens for the two remaining pairs. With
`budget_quantile = 0.9`, R's type-7 sample quantile of `c(10, 30)` is 28.

At input and output prices of 1 currency unit per million tokens and
`batch_discount = 0.5`, the estimator calculates:

- observed live pilot tokens: 60 input and 40 output;
- remaining expected tokens: 140 input and `mean(c(10, 30)) * 2 = 40` output;
- remaining budget output tokens: `28 * 2 = 56`;
- expected cost: `(60 + 40 + 0.5 * (140 + 40)) / 1e6 = 0.000190`; and
- budget cost: `(60 + 40 + 0.5 * (140 + 56)) / 1e6 = 0.000198`.

The pilot observations are included once at live prices. They are not averaged
into the estimated remaining count and are not discounted.

### Reuse pilot results

The estimator returns the original pilot output object and the pairs not
included in the pilot (`remaining_pairs`). Use `remaining_pairs` to submit only
the remaining work after you are satisfied with the estimate. Pilot judgments
are not automatically merged into the later submission result:

```{r, eval=FALSE}
remaining_pairs <- est$remaining_pairs

# Example: submit only the remaining pairs as a batch

batch <- llm_submit_pairs_batch(
          backend = "anthropic",
          model = "claude-sonnet-4-5",
          pairs = remaining_pairs,
          trait_name = td$name,
          trait_description = td$description,
          prompt_template = tmpl)

results <- llm_download_batch_results(batch)
```

Notes:

- The estimator does not require a provider tokenizer; it uses prompt byte length calibrated on the pilot.
- Ollama is not supported in the estimator (local models do not incur token costs).
- Reasoning/thinking tokens are treated as output tokens for pricing.
- Supply current prices for the selected provider, model, endpoint, and mode;
  the package does not maintain or validate a pricing catalog.

## Citation

> Mercer, S. H. (2026). *Provider controls and recovery* [R package vignette]. Comprehensive R
> Archive Network. https://doi.org/10.32614/CRAN.package.pairwiseLLM