LLM Reproducibility Harness

A local Streamlit UI for testing how repeatable OpenAI Chat Completions are when you hold the prompt, context, model, seed, and generation parameters constant.

The goal is to demonstrate operational reproducibility: identical request payloads, identical outputs, and identical backend metadata where available. It is not a proof that the hosted model is mathematically deterministic.

What it records

Each run writes a JSON bundle under runs/ containing:

exact request payload sent to v1/chat/completions
SHA-256 hash of the canonicalized request payload
repeated trial outputs
SHA-256 hash of each output string
system_fingerprint returned by the API, when available
returned model id, finish reason, usage, response id, and raw response

The API key is not stored in bundles.

Setup

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env and set OPENAI_API_KEY, or export it in your shell.
streamlit run app.py

Recommended first experiment

Use the default prompt and these settings:

model: a dated snapshot, for example gpt-4.1-mini-2025-04-14
repetitions: 3 or 5
seed: 42
temperature: 0
top_p: 1
max_completion_tokens: keep modest, for example 200
streaming: off; tools: not used by this harness

Then change only the seed and run again. You should usually see a different output hash while the request hash changes by exactly the seed field.

Baseline comparison

After a run is saved, select it in the sidebar as a baseline and run the same request again. The comparison checks:

whether the canonical request hash is identical
whether output hashes are identical in the same order
whether the set of returned system fingerprints is identical

Offline verification

To recompute bundle and output hashes without calling the API:

python verify_bundle.py runs/run_*.json

Caveats

Hosted LLM reproducibility is best-effort. Matching seed, prompt, parameters, and system_fingerprint improves repeatability but does not guarantee identical text forever.
Rolling model aliases can change. Use snapshot model identifiers where your account and model family support them.
Longer completions provide more opportunities for divergence.
Tool calls, retrieval, web access, time-dependent prompts, and streaming add more variables. This scaffold deliberately avoids them.

2.4 KiB Raw Permalink Blame History