Initial commit, very early alpha

README.md

@@ -1,2 +1,66 @@
-# llm-repro-ui
+# LLM Reproducibility Harness
 
+A local Streamlit UI for testing how repeatable OpenAI Chat Completions are when you hold the prompt, context, model, seed, and generation parameters constant.
+
+The goal is to demonstrate operational reproducibility: identical request payloads, identical outputs, and identical backend metadata where available. It is not a proof that the hosted model is mathematically deterministic.
+
+## What it records
+
+Each run writes a JSON bundle under `runs/` containing:
+
+- exact request payload sent to `v1/chat/completions`
+- SHA-256 hash of the canonicalized request payload
+- repeated trial outputs
+- SHA-256 hash of each output string
+- `system_fingerprint` returned by the API, when available
+- returned model id, finish reason, usage, response id, and raw response
+
+The API key is not stored in bundles.
+
+## Setup
+
+```bash
+python -m venv .venv
+source .venv/bin/activate  # Windows: .venv\Scripts\activate
+pip install -r requirements.txt
+cp .env.example .env
+# Edit .env and set OPENAI_API_KEY, or export it in your shell.
+streamlit run app.py
+```
+
+## Recommended first experiment
+
+Use the default prompt and these settings:
+
+- model: a dated snapshot, for example `gpt-4.1-mini-2025-04-14`
+- repetitions: `3` or `5`
+- seed: `42`
+- temperature: `0`
+- top_p: `1`
+- max_completion_tokens: keep modest, for example `200`
+- streaming: off; tools: not used by this harness
+
+Then change only the seed and run again. You should usually see a different output hash while the request hash changes by exactly the seed field.
+
+## Baseline comparison
+
+After a run is saved, select it in the sidebar as a baseline and run the same request again. The comparison checks:
+
+- whether the canonical request hash is identical
+- whether output hashes are identical in the same order
+- whether the set of returned system fingerprints is identical
+
+## Offline verification
+
+To recompute bundle and output hashes without calling the API:
+
+```bash
+python verify_bundle.py runs/run_*.json
+```
+
+## Caveats
+
+- Hosted LLM reproducibility is best-effort. Matching `seed`, prompt, parameters, and `system_fingerprint` improves repeatability but does not guarantee identical text forever.
+- Rolling model aliases can change. Use snapshot model identifiers where your account and model family support them.
+- Longer completions provide more opportunities for divergence.
+- Tool calls, retrieval, web access, time-dependent prompts, and streaming add more variables. This scaffold deliberately avoids them.