# LLM Reproducibility Harness

A local Streamlit UI for testing how repeatable OpenAI Chat Completions are when you hold the prompt, context, model, seed, and generation parameters constant.

The goal is to demonstrate operational reproducibility: identical request payloads, identical outputs, and identical backend metadata where available. It is not a proof that the hosted model is mathematically deterministic.

## What it records

Each run writes a JSON bundle under `runs/` containing:

- exact request payload sent to `v1/chat/completions`
- SHA-256 hash of the canonicalized request payload
- repeated trial outputs
- SHA-256 hash of each output string
- `system_fingerprint` returned by the API, when available
- returned model id, finish reason, usage, response id, and raw response

The API key is not stored in bundles.

## Setup

```bash
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env and set OPENAI_API_KEY, or export it in your shell.
streamlit run app.py
```

## Recommended first experiment

Use the default prompt and these settings:

- model: a dated snapshot, for example `gpt-4.1-mini-2025-04-14`
- repetitions: `3` or `5`
- seed: `42`
- temperature: `0`
- top_p: `1`
- max_completion_tokens: keep modest, for example `200`
- streaming: off; tools: not used by this harness

Then change only the seed and run again. You should usually see a different output hash while the request hash changes by exactly the seed field.

## Baseline comparison

After a run is saved, select it in the sidebar as a baseline and run the same request again. The comparison checks:

- whether the canonical request hash is identical
- whether output hashes are identical in the same order
- whether the set of returned system fingerprints is identical

## Offline verification

To recompute bundle and output hashes without calling the API:

```bash
python verify_bundle.py runs/run_*.json
```

## Caveats

- Hosted LLM reproducibility is best-effort. Matching `seed`, prompt, parameters, and `system_fingerprint` improves repeatability but does not guarantee identical text forever.
- Rolling model aliases can change. Use snapshot model identifiers where your account and model family support them.
- Longer completions provide more opportunities for divergence.
- Tool calls, retrieval, web access, time-dependent prompts, and streaming add more variables. This scaffold deliberately avoids them.