Document production target evidence handoff

docs/PRODUCTION_TARGET_HANDOFF.md

@@ -0,0 +1,182 @@
+# Production Target And Independent Evidence Handoff
+
+This runbook identifies the external inputs needed to finish
+[GovOPlaN #27](https://git.add-ideas.de/GovOPlaN/govoplan/issues/27) and
+[GovOPlaN #37](https://git.add-ideas.de/GovOPlaN/govoplan/issues/37). The
+repository can render, inspect and sign evidence for a target, but it cannot
+manufacture an independent failure domain or an independent approval authority.
+
+## GovOPlaN #27: real two-node target
+
+The bounded acceptance target is two independently schedulable worker nodes.
+The API and WebUI must each have ready replicas on both nodes, all Deployments
+must be available, the active module composition and software versions must be
+consistent, every configured queue must have a worker, and the database
+connection budget must pass. The validation then deletes one ready API pod and
+requires replacement without an observed readiness outage.
+
+Two virtual machines on different physical hosts or availability zones meet the
+failure-domain intent. Two containers, VMs or Kubernetes nodes on one physical
+host are useful development targets but do not close #27. A two-worker cluster
+also does not prove control-plane high availability. For a self-managed
+production cluster, use three control-plane nodes plus at least two workers; a
+managed control plane plus two workers is the shorter path.
+
+### What the target owner must provide
+
+Provide these through a secure handoff, not an issue, chat message or Git:
+
+1. A kubeconfig path with access to the target, for example
+   `~/.config/govoplan/targets/<target>.kubeconfig`, mode `0600`.
+2. A stable installation ID, public HTTPS hostname, namespace, ingress class and
+   TLS-secret or certificate-manager arrangement.
+3. Two independently schedulable workers and permission to place API and WebUI
+   replicas on both.
+4. External, logically shared PostgreSQL, Redis and S3 endpoints with trusted
+   CA material and network reachability from every worker. Do not co-locate the
+   only copies of these services on the two workers used for the failure drill.
+5. The six runtime secret values required by the generated manifest:
+   `MASTER_KEY_B64`, `DATABASE_URL`, `GOVOPLAN_DATABASE_URL_PGTOOLS`,
+   `REDIS_URL`, `FILE_STORAGE_S3_ACCESS_KEY_ID` and
+   `FILE_STORAGE_S3_SECRET_ACCESS_KEY`.
+6. A short-lived GovOPlaN API key limited to `ops:operations:read`, supplied in
+   `GOVOPLAN_OPS_API_KEY` only for evidence collection.
+7. An approved drill window and permission to delete one API pod.
+
+If no Kubernetes target exists, provide hostnames/IP addresses for the machines,
+an SSH user and key path, the internal/external DNS plan, and the permitted
+firewall ports. Those inputs are sufficient to provision a k3s target. They are
+not sufficient to claim control-plane HA unless three control-plane failure
+domains are present.
+
+### Separate deployment and evidence authorities
+
+The deployment identity may create and update the namespace, Secret,
+ConfigMap, Deployments, Services, Jobs, PodDisruptionBudgets and Ingress. The
+evidence collector only needs:
+
+- cluster scope: `get` and `list` for `nodes`;
+- target namespace: `get` and `list` for `pods` and `deployments`;
+- target namespace during the approved drill: `delete` for `pods`.
+
+Use separate kubeconfig contexts or service accounts when the same person does
+not hold both roles.
+
+### Render, apply and verify
+
+Use the signed, digest-pinned installation bundle selected for the target:
+
+```bash
+export KUBECONFIG="$HOME/.config/govoplan/targets/<target>.kubeconfig"
+
+python tools/deployment/govoplan-deploy.py render-kubernetes \
+  --directory /srv/govoplan/<installation-id> \
+  --namespace govoplan \
+  --secret-name govoplan-runtime \
+  --tls-secret-name govoplan-tls \
+  --ingress-class-name nginx \
+  --output /srv/govoplan/<installation-id>/kubernetes.json
+
+kubectl apply -f /srv/govoplan/<installation-id>/kubernetes.json
+kubectl -n govoplan wait --for=condition=available deployment --all --timeout=10m
+
+export GOVOPLAN_OPS_API_KEY="$(cat /run/secrets/govoplan-ops-evidence-key)"
+python tools/deployment/govoplan-deploy.py verify-kubernetes \
+  --directory /srv/govoplan/<installation-id> \
+  --namespace govoplan \
+  --exercise-api-pod-loss \
+  --output /srv/govoplan/<installation-id>/evidence/kubernetes-multi-host.json
+unset GOVOPLAN_OPS_API_KEY
+```
+
+The verifier emits sanitized JSON and exits nonzero if the topology, runtime,
+queue, connection-budget or pod-loss checks fail. Preserve the private cluster
+logs and manifest alongside the sanitized result in the controlled evidence
+store.
+
+## GovOPlaN #37: controlled signed target evidence
+
+Yes, collection, review and signing can run in containers. A container provides
+repeatability and process isolation; it does not create independent authority.
+The production approver must control a different private key from the target
+operator/assessor and must review the evidence before signing the
+`production_approval` scope.
+
+Use at least these three key boundaries:
+
+1. **Installer authority:** signs installed-release-origin receipts only.
+2. **Target assessment authority:** signs the permitted target, accessibility,
+   privacy, security, operations and recovery scopes.
+3. **Production approval authority:** independently signs only
+   `production_approval` after reviewing the other evidence.
+
+Do not reuse release-catalog keys for any of these roles. Keep private Ed25519
+keys outside Git, Gitea, GovOPlaN application storage and chat. Publish only the
+public keyrings. The proof issuer already rejects key reuse across release,
+installer and proof trust domains.
+
+### Generate independently held keys
+
+Each authority runs this command in its own `0700` directory. The generator
+refuses existing output paths and writes both files as `0600`:
+
+```bash
+install -d -m 0700 "$HOME/.config/govoplan/authority-keys"
+
+python tools/assessments/generate-authority-keypair.py \
+  --purpose proof \
+  --key-id authority:target-2026 \
+  --scope target_environment \
+  --scope accessibility \
+  --scope privacy \
+  --scope security \
+  --scope operations \
+  --scope recovery \
+  --private-key "$HOME/.config/govoplan/authority-keys/target-2026.pem" \
+  --keyring "$HOME/.config/govoplan/authority-keys/target-2026-public.json"
+```
+
+The independent production approver generates another key with only
+`--scope production_approval`. An installer authority uses `--purpose installer`
+and no `--scope`. Merge public key entries into the separately controlled
+keyrings only after the responsible authorities verify fingerprints out of
+band.
+
+### Container boundary
+
+Use two one-shot jobs or containers:
+
+- **Collector/assessor:** network access, read-only source and trust mounts,
+  read/write private evidence output, and the narrowly scoped kubeconfig. It
+  must not receive the production-approval private key.
+- **Production approver:** `--network none`, read-only assessment/evidence/trust
+  mounts, a read-only secret mount containing only the approval key, and a
+  separate output mount. It must not receive deployment credentials.
+
+Build or select the assessment image by digest and record that digest in the
+evidence log. A representative runtime shape is:
+
+```bash
+docker run --rm --network none --read-only --tmpfs /tmp \
+  --user "$(id -u):$(id -g)" \
+  --mount type=bind,src="$PWD/evidence",dst=/evidence,readonly \
+  --mount type=bind,src="$PWD/trust",dst=/trust,readonly \
+  --mount type=bind,src="$HOME/.config/govoplan/authority-keys",dst=/run/keys,readonly \
+  --mount type=bind,src="$PWD/approved",dst=/output \
+  <assessment-image>@sha256:<digest> \
+  <assessment command>
+```
+
+The current evidence commands and required scopes are documented in
+[`TARGET_MATURITY_EVIDENCE_RUNBOOK.md`](TARGET_MATURITY_EVIDENCE_RUNBOOK.md).
+The final proof must cover `target_environment`, `accessibility`, `privacy`,
+`security`, `operations`, `recovery` and independent `production_approval`, and
+must bind to the verified installed composition and installer receipt.
+
+## Completion boundary
+
+#27 can close after the real target produces a passing pod-loss result. #37 can
+close after an independently approved, schema-valid proof is generated for that
+same installed composition and the public authority keyrings, proof and private
+evidence custody references are recorded. Neither issue should close from a
+single-host simulation or a self-approved signature.

docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md

@@ -1,5 +1,9 @@
 # Scaling And Multi-Host Deployment
 
+For the exact external handoff, least-privilege collector permissions and live
+two-node acceptance procedure, see
+[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
+
 ## Implemented Contract
 
 GovOPlaN now supports a stateless application tier backed by logically shared

docs/TARGET_MATURITY_EVIDENCE_RUNBOOK.md

@@ -1,5 +1,9 @@
 # Target Maturity Evidence Runbook
 
+For authority-key generation, container isolation and the concrete inputs that
+must be supplied by the target owner and independent production approver, see
+[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
+
 This runbook turns retained target-environment results into a sanitized,
 signed GovOPlaN capability-fit proof. It does not make a deployment suitable,
 certified, supported, or production-approved by itself. The proof records what

tests/test_assessment_authority_keypair.py

@@ -0,0 +1,120 @@
+from __future__ import annotations
+
+import base64
+import json
+from pathlib import Path
+import stat
+import subprocess
+import sys
+import tempfile
+import unittest
+
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
+from jsonschema import Draft202012Validator, FormatChecker
+
+
+META_ROOT = Path(__file__).resolve().parents[1]
+GENERATOR = META_ROOT / "tools" / "assessments" / "generate-authority-keypair.py"
+
+
+class AssessmentAuthorityKeypairTests(unittest.TestCase):
+    def test_generates_schema_valid_scoped_proof_authority(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            output_dir = Path(temp_dir)
+            output_dir.chmod(0o700)
+            private_path = output_dir / "target.pem"
+            keyring_path = output_dir / "target.json"
+
+            result = subprocess.run(
+                (
+                    sys.executable,
+                    str(GENERATOR),
+                    "--purpose",
+                    "proof",
+                    "--key-id",
+                    "authority:target-2026",
+                    "--scope",
+                    "target_environment",
+                    "--scope",
+                    "operations",
+                    "--private-key",
+                    str(private_path),
+                    "--keyring",
+                    str(keyring_path),
+                ),
+                check=False,
+                capture_output=True,
+                text=True,
+            )
+
+            self.assertEqual(0, result.returncode, result.stderr)
+            self.assertEqual(0o600, stat.S_IMODE(private_path.stat().st_mode))
+            self.assertEqual(0o600, stat.S_IMODE(keyring_path.stat().st_mode))
+            keyring = json.loads(keyring_path.read_text(encoding="utf-8"))
+            schema = json.loads(
+                (
+                    META_ROOT
+                    / "docs"
+                    / "capability-fit-proof-authority-keyring.schema.json"
+                ).read_text(encoding="utf-8")
+            )
+            errors = tuple(
+                Draft202012Validator(
+                    schema, format_checker=FormatChecker()
+                ).iter_errors(keyring)
+            )
+            self.assertEqual((), errors)
+            self.assertEqual(
+                ["target_environment", "operations"],
+                keyring["keys"][0]["allowed_scopes"],
+            )
+            private_key = serialization.load_pem_private_key(
+                private_path.read_bytes(), password=None
+            )
+            self.assertIsInstance(private_key, Ed25519PrivateKey)
+            public_key = base64.b64encode(
+                private_key.public_key().public_bytes(
+                    encoding=serialization.Encoding.Raw,
+                    format=serialization.PublicFormat.Raw,
+                )
+            ).decode("ascii")
+            self.assertEqual(public_key, keyring["keys"][0]["public_key"])
+
+    def test_installer_authority_uses_fixed_scope_and_refuses_overwrite(self) -> None:
+        with tempfile.TemporaryDirectory() as temp_dir:
+            output_dir = Path(temp_dir)
+            output_dir.chmod(0o700)
+            private_path = output_dir / "installer.pem"
+            keyring_path = output_dir / "installer.json"
+            command = (
+                sys.executable,
+                str(GENERATOR),
+                "--purpose",
+                "installer",
+                "--key-id",
+                "authority:installer-2026",
+                "--private-key",
+                str(private_path),
+                "--keyring",
+                str(keyring_path),
+            )
+
+            first = subprocess.run(
+                command, check=False, capture_output=True, text=True
+            )
+            second = subprocess.run(
+                command, check=False, capture_output=True, text=True
+            )
+
+            self.assertEqual(0, first.returncode, first.stderr)
+            self.assertNotEqual(0, second.returncode)
+            keyring = json.loads(keyring_path.read_text(encoding="utf-8"))
+            self.assertEqual(
+                ["installed_release_origin"],
+                keyring["keys"][0]["allowed_scopes"],
+            )
+
+
+if __name__ == "__main__":
+    unittest.main()

tools/assessments/generate-authority-keypair.py

@@ -0,0 +1,194 @@
+#!/usr/bin/env python3
+"""Generate an independently held Ed25519 assessment-authority keypair."""
+
+from __future__ import annotations
+
+import argparse
+import base64
+from datetime import UTC, datetime, timedelta
+import json
+import os
+from pathlib import Path
+import re
+import stat
+
+from cryptography.hazmat.primitives import serialization
+from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
+
+
+KEY_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$")
+PROOF_SCOPES = (
+    "target_environment",
+    "external_providers",
+    "accessibility",
+    "privacy",
+    "security",
+    "operations",
+    "recovery",
+    "production_approval",
+)
+PURPOSES = {
+    "proof": (
+        "govoplan.capability-fit-proof-authorities",
+        "./capability-fit-proof-authority-keyring.schema.json",
+    ),
+    "installer": (
+        "govoplan.installer-receipt-authorities",
+        "./installer-receipt-authority-keyring.schema.json",
+    ),
+}
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--purpose", choices=tuple(PURPOSES), required=True)
+    parser.add_argument("--key-id", required=True)
+    parser.add_argument(
+        "--scope",
+        action="append",
+        choices=PROOF_SCOPES,
+        default=[],
+        help="Authorized proof scope; repeat as needed. Not used for installer keys.",
+    )
+    parser.add_argument("--private-key", type=Path, required=True)
+    parser.add_argument("--keyring", type=Path, required=True)
+    parser.add_argument(
+        "--valid-days",
+        type=int,
+        default=365,
+        help="Validity from generation time (default: 365 days).",
+    )
+    parser.add_argument(
+        "--status",
+        choices=("active", "next"),
+        default="active",
+    )
+    args = parser.parse_args(argv)
+
+    if not KEY_ID_PATTERN.fullmatch(args.key_id):
+        parser.error("--key-id must be a valid opaque identifier")
+    if args.valid_days < 1 or args.valid_days > 3660:
+        parser.error("--valid-days must be between 1 and 3660")
+    scopes = _resolve_scopes(parser, purpose=args.purpose, scopes=args.scope)
+
+    private_path = args.private_key.expanduser().resolve()
+    keyring_path = args.keyring.expanduser().resolve()
+    _require_fresh_output(parser, private_path, label="private key")
+    _require_fresh_output(parser, keyring_path, label="keyring")
+    _require_private_directory(parser, private_path.parent)
+    _require_output_directory(parser, keyring_path.parent)
+
+    private_key = Ed25519PrivateKey.generate()
+    private_bytes = private_key.private_bytes(
+        encoding=serialization.Encoding.PEM,
+        format=serialization.PrivateFormat.PKCS8,
+        encryption_algorithm=serialization.NoEncryption(),
+    )
+    public_bytes = private_key.public_key().public_bytes(
+        encoding=serialization.Encoding.Raw,
+        format=serialization.PublicFormat.Raw,
+    )
+    public_base64 = base64.b64encode(public_bytes).decode("ascii")
+    now = datetime.now(UTC).replace(microsecond=0)
+    not_after = now + timedelta(days=args.valid_days)
+    purpose, schema = PURPOSES[args.purpose]
+    keyring = {
+        "$schema": schema,
+        "schema_version": "0.1.0",
+        "purpose": purpose,
+        "keys": [
+            {
+                "key_id": args.key_id,
+                "status": args.status,
+                "public_key": public_base64,
+                "allowed_scopes": scopes,
+                "not_before": _rfc3339(now),
+                "not_after": _rfc3339(not_after),
+            }
+        ],
+    }
+
+    _write_new_private_file(private_path, private_bytes)
+    try:
+        _write_new_private_file(
+            keyring_path,
+            (json.dumps(keyring, indent=2, sort_keys=True) + "\n").encode("utf-8"),
+        )
+    except BaseException:
+        private_path.unlink(missing_ok=True)
+        keyring_path.unlink(missing_ok=True)
+        raise
+
+    print(f"private_key={private_path}")
+    print(f"keyring={keyring_path}")
+    print(f"key_id={args.key_id}")
+    print(f"allowed_scopes={','.join(scopes)}")
+    return 0
+
+
+def _resolve_scopes(
+    parser: argparse.ArgumentParser, *, purpose: str, scopes: list[str]
+) -> list[str]:
+    if purpose == "installer":
+        if scopes:
+            parser.error("installer authorities do not accept --scope")
+        return ["installed_release_origin"]
+    unique = list(dict.fromkeys(scopes))
+    if not unique:
+        parser.error("proof authorities require at least one --scope")
+    return unique
+
+
+def _require_fresh_output(
+    parser: argparse.ArgumentParser, path: Path, *, label: str
+) -> None:
+    if path.exists() or path.is_symlink():
+        parser.error(f"{label.capitalize()} output already exists: {path}")
+
+
+def _require_private_directory(
+    parser: argparse.ArgumentParser, directory: Path
+) -> None:
+    _require_output_directory(parser, directory)
+    mode = stat.S_IMODE(directory.stat().st_mode)
+    if mode & (stat.S_IRWXG | stat.S_IRWXO):
+        parser.error(
+            "Private-key parent directory must not be accessible by group or others"
+        )
+
+
+def _require_output_directory(
+    parser: argparse.ArgumentParser, directory: Path
+) -> None:
+    try:
+        metadata = directory.lstat()
+    except OSError as exc:
+        parser.error(f"Output parent directory is unavailable: {directory}")
+        raise AssertionError from exc
+    if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
+        parser.error(f"Output parent must be a real directory: {directory}")
+
+
+def _write_new_private_file(path: Path, payload: bytes) -> None:
+    flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
+    if hasattr(os, "O_NOFOLLOW"):
+        flags |= os.O_NOFOLLOW
+    descriptor = os.open(path, flags, 0o600)
+    try:
+        with os.fdopen(descriptor, "wb", closefd=False) as handle:
+            handle.write(payload)
+            handle.flush()
+            os.fsync(handle.fileno())
+        metadata = os.fstat(descriptor)
+        if not stat.S_ISREG(metadata.st_mode) or stat.S_IMODE(metadata.st_mode) != 0o600:
+            raise OSError("Authority output could not be secured")
+    finally:
+        os.close(descriptor)
+
+
+def _rfc3339(value: datetime) -> str:
+    return value.isoformat().replace("+00:00", "Z")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())