77 Commits
Author SHA1 Message Date
zemion a24c94435e Release v0.1.15
Dependency Audit / dependency-audit (push) Failing after 1m49s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m50s
Developer Meta-package Release / publish-package (push) Failing after 4s
2026-08-04 15:20:50 +02:00
zemion 774793976c Support legacy module version declarations
Deployment Installer / deployment-installer (push) Successful in 6s
Dependency Audit / dependency-audit (push) Failing after 1m42s
Security Audit / security-audit (push) Successful in 10m43s
2026-08-04 15:08:46 +02:00
zemion 492449a4e2 Align all release version declarations
Dependency Audit / dependency-audit (push) Failing after 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Failing after 10m42s
2026-08-04 15:06:36 +02:00
zemion 8e890b37ed Validate candidate migration baseline during release
Dependency Audit / dependency-audit (push) Failing after 1m42s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m38s
2026-08-04 15:01:50 +02:00
zemion 077735bc24 Bind migration heads to coordinated releases
Deployment Installer / deployment-installer (push) Successful in 6s
Dependency Audit / dependency-audit (push) Failing after 1m44s
Security Audit / security-audit (push) Successful in 10m36s
2026-08-04 14:55:58 +02:00
zemion d9522d3cc4 Publish release tags in dependency order
Dependency Audit / dependency-audit (push) Failing after 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m53s
2026-08-04 14:54:54 +02:00
zemion bad0ea37a7 Automate exact package-set publication
Dependency Audit / dependency-audit (push) Failing after 1m46s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m46s
2026-08-04 14:32:06 +02:00
zemion 9ffd46fe22 Make package publication retries hash-safe
Dependency Audit / dependency-audit (push) Failing after 1m48s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m35s
2026-08-04 14:18:49 +02:00
zemion be51a9c347 Document production target evidence handoff
Dependency Audit / dependency-audit (push) Failing after 1m46s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m40s
2026-08-04 14:00:45 +02:00
zemion e36a6573bf Harden package release workflows for Gitea 2026-08-04 14:00:32 +02:00
zemion 629bfec1f1 Cover datasource publication change events
Dependency Audit / dependency-audit (push) Failing after 1m40s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m48s
2026-08-04 12:05:48 +02:00
zemion 9e956eec6f Classify first-run bootstrap endpoints
Dependency Audit / dependency-audit (push) Failing after 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m52s
2026-08-04 10:07:24 +02:00
zemion 9bb2c808a6 Enforce declared platform interface inventory
Dependency Audit / dependency-audit (push) Failing after 1m36s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m2s
2026-08-04 05:20:47 +02:00
zemion f7590a7b8b Add registry-backed module package releases
Dependency Audit / dependency-audit (push) Failing after 1m37s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m21s
2026-08-04 04:14:28 +02:00
zemion 1a68565ba0 Reconcile encryption interface surfaces
Dependency Audit / dependency-audit (push) Failing after 1m40s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m33s
2026-08-04 01:28:03 +02:00
zemion d9f67b5c26 Reconcile implemented platform interface surfaces
Dependency Audit / dependency-audit (push) Failing after 1m39s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m32s
2026-08-04 01:04:40 +02:00
zemion 1cfdaec250 inventory: mark campaign archive UI reachable
Dependency Audit / dependency-audit (push) Successful in 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m54s
2026-08-03 20:39:16 +02:00
zemion 62501d399a ci: enforce endpoint inventory independently
Dependency Audit / dependency-audit (push) Successful in 1m46s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m43s
2026-08-03 20:29:14 +02:00
zemion ce5528e3b8 Record first verified runtime distribution 2026-08-03 20:17:29 +02:00
zemion 1f039dd39c Retain Caddy file capability at ingress boundary 2026-08-03 20:02:29 +02:00
zemion d107d94fec Add standalone managed ingress diagnostics 2026-08-03 19:59:12 +02:00
zemion 6163c5992f Fix ingress probe image selection 2026-08-03 19:49:35 +02:00
zemion 2f28f22fd1 Probe managed ingress across Docker namespaces 2026-08-03 19:41:04 +02:00
zemion 909862afdb Fix managed ingress loopback publication 2026-08-03 19:28:10 +02:00
zemion eb04804d36 Handle Redis under arm64 CI emulation 2026-08-03 19:18:01 +02:00
zemion 017aa7a702 Enable arm64 runtime smoke execution 2026-08-03 19:11:45 +02:00
zemion af27b9fbdf Resolve runtime dependency platform digests 2026-08-03 19:04:06 +02:00
zemion cb45251c59 Fix read-only Web runtime publication 2026-08-03 18:56:23 +02:00
zemion 3b3d5b3386 Redact runtime smoke diagnostics
Dependency Audit / dependency-audit (push) Successful in 1m44s
Deployment Installer / deployment-installer (push) Successful in 7s
2026-08-03 18:36:00 +02:00
zemion 313249b8fc Exercise packaged runtime topology
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
2026-08-03 18:26:04 +02:00
zemion 282c90c54b Make ingress drill Docker socket portable
Dependency Audit / dependency-audit (push) Successful in 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m56s
2026-08-03 17:55:48 +02:00
zemion 25424187a8 Package runtime migration scripts correctly
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m1s
2026-08-03 17:54:09 +02:00
zemion ff8ee991c3 Harden runtime distribution acceptance
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m49s
2026-08-03 17:32:52 +02:00
zemion a5a0731d20 Fix runtime distribution signing environment
Dependency Audit / dependency-audit (push) Successful in 1m48s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m50s
2026-08-03 17:09:50 +02:00
zemion a0f161041d Record Risk Compliance interface migration
Dependency Audit / dependency-audit (push) Successful in 1m46s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m42s
2026-08-03 15:47:41 +02:00
zemion cb85999a14 Record Notifications interface migration
Dependency Audit / dependency-audit (push) Successful in 1m52s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m42s
2026-08-03 15:34:18 +02:00
zemion cbfe8b03a7 Record Ops interface migration
Dependency Audit / dependency-audit (push) Successful in 1m57s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m5s
2026-08-03 15:19:07 +02:00
zemion 768e9a51c9 Record Calendar interface migration
Dependency Audit / dependency-audit (push) Successful in 1m59s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m5s
2026-08-03 15:09:01 +02:00
zemion 087561ee12 Record Cases interface migration
Dependency Audit / dependency-audit (push) Successful in 1m52s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 11m2s
2026-08-03 14:44:14 +02:00
zemion 11c1aa1815 Record Dashboard interface migration
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m49s
2026-08-03 14:25:45 +02:00
zemion 478ecb5d0e Record Dataflow interface migration
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m40s
2026-08-03 14:20:21 +02:00
zemion 32689d027a Repair missing packages during environment sync
Dependency Audit / dependency-audit (push) Successful in 1m50s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m50s
2026-08-03 14:06:41 +02:00
zemion b7cc2d2df4 Record Datasources interface migration
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m43s
2026-08-03 13:47:26 +02:00
zemion b70869e747 Record Addresses interface migration
Dependency Audit / dependency-audit (push) Successful in 1m43s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m43s
2026-08-03 13:41:04 +02:00
zemion 0c84afb158 Record Templates interface migration
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m56s
2026-08-03 13:35:11 +02:00
zemion 145aa58c11 Record Distribution Lists interface migration
Dependency Audit / dependency-audit (push) Successful in 1m58s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m58s
2026-08-03 13:28:33 +02:00
zemion a3566c9311 Record Voting interface migration
Dependency Audit / dependency-audit (push) Successful in 2m4s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m49s
2026-08-03 13:17:08 +02:00
zemion 3219460064 Record Forms interface migration
Dependency Audit / dependency-audit (push) Successful in 2m3s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m0s
2026-08-03 13:09:24 +02:00
zemion 4b2a15adb5 Record Forms Runtime interface migration
Dependency Audit / dependency-audit (push) Successful in 2m1s
Deployment Installer / deployment-installer (push) Successful in 7s
Security Audit / security-audit (push) Successful in 10m49s
2026-08-03 13:02:51 +02:00
zemion acc5ffc247 Record Approvals interface migration
Dependency Audit / dependency-audit (push) Successful in 2m0s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m0s
2026-08-03 12:56:07 +02:00
zemion f4f9836a09 Record Committee interface migration
Dependency Audit / dependency-audit (push) Successful in 1m57s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m53s
2026-08-03 12:49:58 +02:00
zemion 758fa1bba7 Record IDM interface migration
Dependency Audit / dependency-audit (push) Successful in 1m55s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m54s
2026-08-03 12:37:12 +02:00
zemion 0acc8cfc31 Record Postbox interface migration
Dependency Audit / dependency-audit (push) Successful in 1m51s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m52s
2026-08-03 12:26:52 +02:00
zemion 344bcaf1bc Record Organizations interface migration
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m59s
2026-08-03 12:10:13 +02:00
zemion 794622e4ed Record Views interface migration
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 7s
Security Audit / security-audit (push) Successful in 10m42s
2026-08-03 11:58:24 +02:00
zemion 7653e9851f Record Tenancy interface migration
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m56s
2026-08-03 11:43:28 +02:00
zemion 6f896d9c04 Record Admin interface migration
Dependency Audit / dependency-audit (push) Successful in 1m41s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m48s
2026-08-03 11:30:45 +02:00
zemion 8f5ac52b58 Record Access interface migration
Dependency Audit / dependency-audit (push) Successful in 1m42s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m47s
2026-08-03 11:01:57 +02:00
zemion 2bc9ad7f00 Record completed Audit interface migration
Dependency Audit / dependency-audit (push) Successful in 1m43s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m51s
2026-08-03 10:43:57 +02:00
zemion fa1a4bacfb Record completed Scheduling interface migration
Dependency Audit / dependency-audit (push) Successful in 1m43s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m43s
2026-08-03 10:36:39 +02:00
zemion 0c0669768d Record Policy interface migration
Dependency Audit / dependency-audit (push) Successful in 1m42s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m45s
2026-08-03 10:23:22 +02:00
zemion 7b6ceeb185 Record completed Core configuration patterns
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m37s
2026-08-03 10:17:18 +02:00
zemion ff12f676a1 Record Mail interface pattern migration
Dependency Audit / dependency-audit (push) Successful in 1m46s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 11m8s
2026-08-03 10:07:19 +02:00
zemion eb9ab9ef1c Record Files interface pattern migration
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m49s
2026-08-03 09:51:01 +02:00
zemion 935c1fe162 Track module interface migration work
Dependency Audit / dependency-audit (push) Successful in 1m38s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m19s
2026-08-03 08:31:34 +02:00
zemion c7d1cd0e8f Exercise recovery in datasource composition check
Dependency Audit / dependency-audit (push) Successful in 1m40s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m14s
2026-08-03 07:46:57 +02:00
zemion ac80d7e4e3 Record Campaign review pattern evidence
Dependency Audit / dependency-audit (push) Successful in 1m38s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m18s
2026-08-03 07:24:23 +02:00
zemion 5e449b0983 Record Core lifecycle recovery adoption
Dependency Audit / dependency-audit (push) Successful in 1m44s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m41s
2026-08-03 07:02:34 +02:00
zemion d4bf07b446 Record workflow recovery ledger adoption
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m35s
2026-08-03 06:37:45 +02:00
zemion adc4db9fdf Record Dataflow recovery adoption
Dependency Audit / dependency-audit (push) Successful in 1m45s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m16s
2026-08-03 06:09:53 +02:00
zemion 484f2af3ac Record Connectors recovery adoption
Dependency Audit / dependency-audit (push) Successful in 1m40s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m11s
2026-08-03 05:43:58 +02:00
zemion abf9564cee Record Mail recovery adoption
Dependency Audit / dependency-audit (push) Successful in 1m36s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m13s
2026-08-03 05:00:46 +02:00
zemion 5bef966119 Record Files recovery ledger adoption
Dependency Audit / dependency-audit (push) Successful in 2m0s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m13s
2026-08-03 04:22:28 +02:00
zemion 5e80b39bbd Record Campaign recovery ledger adoption
Dependency Audit / dependency-audit (push) Successful in 1m47s
Deployment Installer / deployment-installer (push) Successful in 5s
Security Audit / security-audit (push) Successful in 10m27s
2026-08-03 03:55:16 +02:00
zemion 9370f501a0 Inventory module recovery ledger adoption
Dependency Audit / dependency-audit (push) Successful in 1m36s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 9m54s
2026-08-03 03:02:53 +02:00
zemion e8f7e2c194 Repair release and interface CI gates
Dependency Audit / dependency-audit (push) Successful in 1m35s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Successful in 10m5s
2026-08-03 02:02:48 +02:00
zemion cbbe08d912 Enforce signed backup evidence before migrations 2026-08-03 02:01:13 +02:00
78 changed files with 9104 additions and 615 deletions
+2 -2
View File
@@ -55,9 +55,9 @@ jobs:
- name: Install WebUI release dependencies with test scripts
working-directory: govoplan
run: bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui
- name: Validate platform endpoint surface declarations
- name: Validate platform interface and endpoint declarations
working-directory: govoplan
run: .venv/bin/python tools/inventory/platform-interface-inventory.py --strict
run: .venv/bin/python tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints
- name: Validate Search against PostgreSQL
working-directory: govoplan
env:
@@ -0,0 +1,49 @@
name: Developer Meta-package Release
on:
push:
tags:
- "v*"
jobs:
publish-package:
runs-on: ubuntu-latest
env:
GITEA_REPOSITORY: ${{ gitea.repository }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- name: Validate protected release tag and package version
run: |
python - <<'PY'
import os
from pathlib import Path
import subprocess
import tomllib
tag = os.environ["GITEA_REF_NAME"]
project = tomllib.loads(Path("packages/govoplan-meta/pyproject.toml").read_text(encoding="utf-8"))["project"]
if tag != f"v{project['version']}":
raise SystemExit("meta-package version does not match the release tag")
if subprocess.run(["git", "merge-base", "--is-ancestor", "HEAD", "origin/main"]).returncode:
raise SystemExit("release tag is not contained in main")
PY
- name: Build and publish developer package
env:
PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
run: |
set -euo pipefail
test -n "$PACKAGE_USERNAME"
test -n "$PACKAGE_TOKEN"
python -m pip install --disable-pip-version-check build==1.5.0 twine==7.0.0
python -m build --wheel --outdir dist packages/govoplan-meta
python -m twine check dist/*.whl
TWINE_USERNAME="$PACKAGE_USERNAME" TWINE_PASSWORD="$PACKAGE_TOKEN" \
python -m twine upload --non-interactive \
--repository-url https://git.add-ideas.de/api/packages/GovOPlaN/pypi \
dist/*.whl
+6
View File
@@ -25,6 +25,12 @@ jobs:
- name: Bootstrap GovOPlaN repositories
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --reuse-checkout-auth --exclude-repo addideas-govoplan-website
- name: Validate package publication contracts
working-directory: govoplan
run: |
python tools/repo/sync-module-package-workflows.py --check
python tools/release/generate-developer-meta-package.py --check
python -m unittest tests.test_module_package_workflows tests.test_package_registry_release
- name: Install backend release integration dependencies
working-directory: govoplan
run: |
+139 -9
View File
@@ -39,6 +39,10 @@ on:
description: Digest-pinned GreenMail image
required: true
type: string
binfmt_image:
description: Digest-pinned tonistiigi/binfmt image for arm64 CI execution
required: true
type: string
jobs:
publish-runtime:
@@ -53,21 +57,71 @@ jobs:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: "22"
- name: Validate immutable release inputs
env:
VERSION: ${{ inputs.version }}
PYTHON_IMAGE: ${{ inputs.python_image }}
NGINX_IMAGE: ${{ inputs.nginx_image }}
POSTGRES_IMAGE: ${{ inputs.postgres_image }}
REDIS_IMAGE: ${{ inputs.redis_image }}
LOAD_BALANCER_IMAGE: ${{ inputs.load_balancer_image }}
MANAGED_INGRESS_IMAGE: ${{ inputs.managed_ingress_image }}
GARAGE_IMAGE: ${{ inputs.garage_image }}
TEST_MAIL_IMAGE: ${{ inputs.test_mail_image }}
BINFMT_IMAGE: ${{ inputs.binfmt_image }}
run: |
python - <<'PY'
import os
import re
version = os.environ["VERSION"]
if re.fullmatch(r"[0-9]+\.[0-9]+\.[0-9]+(?:[-+][A-Za-z0-9.-]+)?", version) is None:
raise SystemExit("version must be a SemVer value without a leading v")
image_pattern = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$")
for name in (
"PYTHON_IMAGE",
"NGINX_IMAGE",
"POSTGRES_IMAGE",
"REDIS_IMAGE",
"LOAD_BALANCER_IMAGE",
"MANAGED_INGRESS_IMAGE",
"GARAGE_IMAGE",
"TEST_MAIL_IMAGE",
"BINFMT_IMAGE",
):
if image_pattern.fullmatch(os.environ[name]) is None:
raise SystemExit(f"{name} must be an exact sha256 image reference")
PY
- name: Use HTTPS for GovOPlaN repositories
run: |
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "git@git.add-ideas.de:GovOPlaN/govoplan"
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "ssh://git@git.add-ideas.de/GovOPlaN/govoplan"
- name: Bootstrap release sources
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --exclude-repo addideas-govoplan-website
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --reuse-checkout-auth --exclude-repo addideas-govoplan-website
- name: Build release wheel roots and WebUI
working-directory: govoplan
env:
VERSION: ${{ inputs.version }}
GOVOPLAN_PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
GOVOPLAN_PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
run: |
python -m venv .runtime-build
.runtime-build/bin/python -m pip install --upgrade pip wheel cryptography
mkdir -p runtime-output/local-wheels
.runtime-build/bin/python -m pip wheel --no-deps --wheel-dir runtime-output/local-wheels --requirement requirements-release.txt
bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui
.runtime-build/bin/python -m pip install --upgrade pip cryptography
.runtime-build/bin/python tools/release/generate-release-package-set.py \
--version "$VERSION" \
--output runtime-output/release-packages.json
.runtime-build/bin/python tools/release/resolve-package-artifacts.py \
--package-set runtime-output/release-packages.json \
--wheelhouse runtime-output/local-wheels \
--webui-packages runtime-output/webui-packages \
--lock-output runtime-output/package-artifacts.lock.json \
--requirements-output runtime-output/requirements-release.packages.txt \
--python .runtime-build/bin/python
PYTHON=.runtime-build/bin/python \
GOVOPLAN_WEBUI_PACKAGE_LOCK="$PWD/runtime-output/package-artifacts.lock.json" \
GOVOPLAN_WEBUI_PACKAGE_DIR="$PWD/runtime-output/webui-packages" \
bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui
npm --prefix ../govoplan-core/webui run build
.runtime-build/bin/python tools/release/prepare-runtime-context.py \
--wheelhouse runtime-output/local-wheels \
@@ -162,6 +216,41 @@ jobs:
WEB_DIGEST="sha256:$(sha256sum runtime-output/web-index.json | cut -d' ' -f1)"
python tools/release/resolve-oci-platforms.py --repository git.add-ideas.de/govoplan/runtime-api --index-digest "$API_DIGEST" --index runtime-output/api-index.json --output runtime-output/api-metadata.json
python tools/release/resolve-oci-platforms.py --repository git.add-ideas.de/govoplan/runtime-web --index-digest "$WEB_DIGEST" --index runtime-output/web-index.json --output runtime-output/web-metadata.json
- name: Resolve managed dependency platform images
working-directory: govoplan
env:
POSTGRES_IMAGE: ${{ inputs.postgres_image }}
REDIS_IMAGE: ${{ inputs.redis_image }}
run: |
docker buildx imagetools inspect "$POSTGRES_IMAGE" --raw > runtime-output/postgres-index.json
docker buildx imagetools inspect "$REDIS_IMAGE" --raw > runtime-output/redis-index.json
python tools/release/resolve-oci-platforms.py \
--repository "${POSTGRES_IMAGE%@*}" \
--index-digest "${POSTGRES_IMAGE##*@}" \
--index runtime-output/postgres-index.json \
--output runtime-output/postgres-metadata.json
python tools/release/resolve-oci-platforms.py \
--repository "${REDIS_IMAGE%@*}" \
--index-digest "${REDIS_IMAGE##*@}" \
--index runtime-output/redis-index.json \
--output runtime-output/redis-metadata.json
- name: Register arm64 execution for runtime smoke
working-directory: govoplan
env:
BINFMT_IMAGE: ${{ inputs.binfmt_image }}
run: docker run --privileged --rm "$BINFMT_IMAGE" --install arm64
- name: Exercise amd64 and arm64 runtime images
working-directory: govoplan
run: |
for ARCH in amd64 arm64; do
.runtime-build/bin/python tools/checks/runtime-image-smoke.py \
--api-metadata runtime-output/api-metadata.json \
--web-metadata runtime-output/web-metadata.json \
--postgres-metadata runtime-output/postgres-metadata.json \
--redis-metadata runtime-output/redis-metadata.json \
--platform "linux/$ARCH" \
--output "runtime-output/evidence/runtime-smoke-$ARCH.json"
done
- name: Generate and sign distribution evidence
working-directory: govoplan
env:
@@ -190,6 +279,7 @@ jobs:
--web-metadata runtime-output/web-metadata.json \
--deployer runtime-output/govoplan-deploy.pyz \
--deployer-url "$ARTIFACT_BASE/govoplan-deploy.pyz" \
--package-lock runtime-output/package-artifacts.lock.json \
--artifact-base-url "$ARTIFACT_BASE" \
--source-commit "$SOURCE_COMMIT" \
--version "$VERSION" \
@@ -202,21 +292,53 @@ jobs:
--dependency "test_mail=$TEST_MAIL_IMAGE" \
--output-directory runtime-output/evidence \
--descriptor runtime-output/distribution-descriptor.json
python tools/release/generate-runtime-distribution.py \
.runtime-build/bin/python tools/release/generate-runtime-distribution.py \
--descriptor runtime-output/distribution-descriptor.json \
--signing-key "$SIGNING_KEY_ID=runtime-output/signing-key.pem" \
--output runtime-output/distribution-manifest.json
openssl pkeyutl -sign -inkey runtime-output/signing-key.pem -rawin \
-in runtime-output/govoplan-deploy.pyz \
-out runtime-output/govoplan-deploy.pyz.sig
sha256sum runtime-output/govoplan-deploy.pyz > runtime-output/govoplan-deploy.pyz.sha256
sha256sum runtime-output/distribution-manifest.json > runtime-output/distribution-manifest.json.sha256
(cd runtime-output && sha256sum govoplan-deploy.pyz > govoplan-deploy.pyz.sha256)
(cd runtime-output && sha256sum distribution-manifest.json > distribution-manifest.json.sha256)
rm runtime-output/signing-key.pem
- name: Verify the published bundle contract with the zipapp
working-directory: govoplan
env:
VERSION: ${{ inputs.version }}
SIGNING_KEY_ID: ${{ secrets.RUNTIME_DISTRIBUTION_SIGNING_KEY_ID }}
run: |
(cd runtime-output && sha256sum --check govoplan-deploy.pyz.sha256)
(cd runtime-output && sha256sum --check distribution-manifest.json.sha256)
.runtime-build/bin/python - <<'PY'
import json
import os
from pathlib import Path
keyring = json.loads(
Path("runtime-output/distribution-keyring.json").read_text(encoding="utf-8")
)
key_id = os.environ["SIGNING_KEY_ID"]
matches = [item for item in keyring["keys"] if item.get("key_id") == key_id]
if len(matches) != 1 or matches[0].get("status") != "active":
raise SystemExit("runtime signing key is not uniquely active in the keyring")
Path("runtime-output/runtime-release-public.pem").write_text(
matches[0]["public_key_pem"], encoding="utf-8"
)
PY
openssl pkeyutl -verify -pubin \
-inkey runtime-output/runtime-release-public.pem -rawin \
-in runtime-output/govoplan-deploy.pyz \
-sigfile runtime-output/govoplan-deploy.pyz.sig
cp runtime-output/govoplan-deploy.pyz runtime-output/govoplan-deploy.tampered.pyz
printf '\0' >> runtime-output/govoplan-deploy.tampered.pyz
if openssl pkeyutl -verify -pubin \
-inkey runtime-output/runtime-release-public.pem -rawin \
-in runtime-output/govoplan-deploy.tampered.pyz \
-sigfile runtime-output/govoplan-deploy.pyz.sig >/dev/null 2>&1; then
echo "Tampered deployment bootstrap unexpectedly verified" >&2
exit 1
fi
MANIFEST_SHA256="$(cut -d' ' -f1 runtime-output/distribution-manifest.json.sha256)"
python runtime-output/govoplan-deploy.pyz init \
--directory runtime-output/acceptance-install \
@@ -236,14 +358,17 @@ jobs:
python tools/checks/managed-ingress-drill.py
--caddy-image "$MANAGED_INGRESS_IMAGE"
--load-balancer-image "$LOAD_BALANCER_IMAGE"
--probe-image "$(jq -r '.platforms["linux/amd64"]' runtime-output/api-metadata.json)"
- name: Publish immutable Gitea release assets
working-directory: govoplan
env:
VERSION: ${{ inputs.version }}
SOURCE_COMMIT: ${{ gitea.sha }}
GITEA_RELEASE_TOKEN: ${{ secrets.GOVOPLAN_RELEASE_TOKEN }}
run: |
python tools/release/publish-runtime-release.py \
--tag "v$VERSION" \
--target-commit "$SOURCE_COMMIT" \
--title "GovOPlaN v$VERSION runtime distribution" \
--asset runtime-output/govoplan-deploy.pyz \
--asset runtime-output/govoplan-deploy.pyz.sig \
@@ -252,7 +377,12 @@ jobs:
--asset runtime-output/distribution-manifest.json.sha256 \
--asset runtime-output/distribution-keyring.json \
--asset runtime-output/context-amd64/composition.json \
--asset runtime-output/release-packages.json \
--asset runtime-output/package-artifacts.lock.json \
--asset runtime-output/requirements-release.packages.txt \
--asset runtime-output/evidence/api-sbom.cdx.json \
--asset runtime-output/evidence/web-sbom.cdx.json \
--asset runtime-output/evidence/api-provenance.json \
--asset runtime-output/evidence/web-provenance.json
--asset runtime-output/evidence/web-provenance.json \
--asset runtime-output/evidence/runtime-smoke-amd64.json \
--asset runtime-output/evidence/runtime-smoke-arm64.json
@@ -0,0 +1,44 @@
name: Runtime Ingress Drill
on:
workflow_dispatch:
inputs:
caddy_image:
description: Digest-pinned Caddy image
required: true
type: string
load_balancer_image:
description: Digest-pinned HAProxy image
required: true
type: string
probe_image:
description: Digest-pinned amd64 GovOPlaN API image
required: true
type: string
jobs:
managed-ingress:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
path: govoplan
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- name: Authenticate runtime image pull
env:
REGISTRY_USERNAME: ${{ secrets.GOVOPLAN_REGISTRY_USERNAME }}
REGISTRY_TOKEN: ${{ secrets.GOVOPLAN_REGISTRY_TOKEN }}
run: echo "$REGISTRY_TOKEN" | docker login git.add-ideas.de --username "$REGISTRY_USERNAME" --password-stdin
- name: Exercise the managed ingress boundary
working-directory: govoplan
env:
CADDY_IMAGE: ${{ inputs.caddy_image }}
LOAD_BALANCER_IMAGE: ${{ inputs.load_balancer_image }}
PROBE_IMAGE: ${{ inputs.probe_image }}
run: >-
python tools/checks/managed-ingress-drill.py
--caddy-image "$CADDY_IMAGE"
--load-balancer-image "$LOAD_BALANCER_IMAGE"
--probe-image "$PROBE_IMAGE"
+3
View File
@@ -10,6 +10,9 @@ tools/release/runtime/*
!tools/release/runtime/Dockerfile.web
!tools/release/runtime/nginx.conf
__pycache__/
build/
dist/
*.egg-info/
audit-reports/
coverage/
htmlcov/
+12
View File
@@ -113,6 +113,18 @@ Generate the CycloneDX dependency inventory from a resolved release environment:
./.venv/bin/python tools/release/generate-release-sbom.py --python ./.venv/bin/python
```
Synchronize module package workflows and inspect the registry release contract:
```sh
./.venv/bin/python tools/repo/sync-module-package-workflows.py --check
./.venv/bin/python tools/release/generate-release-package-set.py \
--output /tmp/govoplan-release-packages.json
```
Package publication, exact artifact locking, and the optional `govoplan`
developer meta-package are documented in
[Package Registry Releases](docs/PACKAGE_REGISTRY_RELEASES.md).
For reproducible release artifacts, set `SOURCE_DATE_EPOCH` to the release
commit timestamp (or pass an explicit timezone-qualified `--timestamp`):
+133
View File
@@ -0,0 +1,133 @@
# Backup And Restore Evidence
## Boundary
`govoplan-deploy` verifies backup and restore evidence; it does not receive
database, object-store, KMS, or orchestrator administration credentials and it
does not create the backup. A provider-owned backup controller creates one
coordinated recovery point, a separate drill runner restores it into an
isolated target, and an evidence authority signs the resulting receipt.
The application containers receive only a sanitized projection: evidence,
recovery-point and drill identifiers, hashes, timestamps, component count, and
measured RPO/RTO. Artifact locations, provider credentials, encryption-key
references, the public trust keyring, and private signing keys remain in the
deployment/evidence boundary.
The machine-readable contracts are:
- [`backup-evidence.schema.json`](backup-evidence.schema.json);
- [`backup-evidence-keyring.schema.json`](backup-evidence-keyring.schema.json).
One evidence document is bound to the installation id, deployment profile,
topology subject, exact signed release manifest, image digests, and composition
digest. It covers PostgreSQL, objects, protected configuration, and recoverable
key custody at one recovery point. It contains references, never key material.
## Production Sequence
1. Establish the provider snapshot, application quiesce, or transaction
boundary and retain a hash of its fencing token.
2. Capture PostgreSQL, object storage, protected deployment configuration, and
key-custody state within five minutes of that recovery point.
3. Restore all four components into a target isolated from production write
endpoints and production queues.
4. Start the exact immutable release named in the evidence, verify migration
heads, verify a deterministic manifest of representative object hashes, and
execute the documented semantic journey checks.
5. Record actual data loss and elapsed recovery as measured RPO and RTO. A
measured RPO above the declared objective invalidates the evidence.
6. Sign the canonical receipt using an evidence-authority Ed25519 key held
outside the application and deployment host. During key rotation, include
both accepted signatures.
7. Transfer the evidence SHA-256 through an independent approved channel, then
verify and adopt it on the deployment host.
Provider automation can sign and validate an unsigned receipt with:
```sh
python tools/deployment/sign-backup-evidence.py \
--input unsigned-backup-evidence.json \
--output backup-evidence.json \
--trusted-keyring backup-evidence-keyring.json \
--signing-key backup-authority-2026=/run/keys/backup-authority.pem
```
The private key file must be owner-only. The tool refuses an unexpected key
type, an inactive/untrusted signer, malformed or partial evidence, stale
recovery points, failed drill checks, mismatched releases, and non-canonical
output.
Adopt the result using the independently obtained digest:
```sh
python3 govoplan-deploy.pyz verify-backup \
--directory /srv/govoplan/default \
--evidence ./backup-evidence.json \
--evidence-sha256 "$APPROVED_BACKUP_EVIDENCE_SHA256" \
--trusted-keyring ./backup-evidence-keyring.json \
--adopt
```
Evidence is fresh for at most 24 hours and may declare an earlier expiry. Every
self-hosted release identity change is conservatively treated as a migration
boundary. `doctor`, Compose `apply`, and `render-kubernetes` fail closed when
fresh evidence for the previously applied immutable release is unavailable.
Compose verifies once before changing runtime state and again after API/worker
quiescing immediately before migration. The exported Kubernetes migration Job
is generated only after verification and is annotated with the sanitized
evidence digest, recovery-point id, and drill id.
## Provider Runbooks
### PostgreSQL
Use a managed transaction-consistent snapshot or a base backup plus retained
WAL sufficient to reconstruct the declared point. Record the provider,
protected artifact reference and digest, snapshot identity, and PostgreSQL LSN.
The restore drill must connect only to the isolated database and must compare
the resulting migration-head digest with the release expectation.
### Object Storage
Use provider snapshots/versioning or an immutable object copy. Build a sorted
manifest containing object key, version, size, and content digest, then record
its digest, object count, total bytes, provider version identity, and protected
artifact reference. Verify representative objects from every owning module
after restore. Single-node managed Garage is persistent but not highly
available; copy its coordinated recovery material to an independent failure
domain.
### Configuration And Key Custody
Back up the private installation bundle and external secret-manager bindings as
an encrypted artifact. Record only its reference and digest. For KMS/HSM/vault
state, record the provider keyset reference, version, and a successful
recoverability assertion. Never put a key, recovery share, token, password, or
credential-bearing URL in evidence. The isolated drill must prove that the
restored release can decrypt representative protected content without
exporting the key material into the report.
## Ownership And Retention
The deployment owner approves the RPO/RTO objectives. State-service owners
operate backup capture and restoration. Module owners define representative
objects and semantic checks. Security owns evidence-authority keys and
revocation. Operations schedules drills and retains sanitized status.
Retain backup artifacts for the approved legal/operational period and at least
through the release's rollback window. Retain signed evidence, drill reports,
and deletion receipts for the audit period. Disposal must remove every backup
copy and provider version according to policy, then revoke or retire references
without deleting the audit receipt. Cryptographic erasure is valid only when
key-destruction evidence and provider-copy coverage are independently proven.
## Failure Handling
Missing components, component-time skew, stale or expired evidence, revocation,
signature/key mismatch, changed stored files, release mismatch, failed semantic
checks, or an RPO breach block migration. The deployment journal records the
rejection without private provider details. If migration has not started, the
operator may supply fresh evidence and retry. Once migration starts, recovery
is explicitly forward-only until the verified coordinated recovery point is
restored with its matching release.
+100 -21
View File
@@ -97,6 +97,9 @@ The private installation directory contains:
| `receipt.json` | Last successfully applied immutable identities |
| `distribution-manifest.json` | Canonical signed runtime/image selection adopted by the installer |
| `distribution-keyring.json` | Explicitly installed public trust anchor for runtime releases |
| `backup-evidence.json` | Signed provider-neutral coordinated backup and isolated-restore receipt |
| `backup-keyring.json` | Explicit public trust anchor for backup evidence authorities |
| `backup-verification.json` | Sanitized local verification/adoption receipt |
| `applied-state/` | Checksum-verified snapshot of the last healthy deployment bundle |
| `operations/<id>/` | Private hash-chained deployment progress and recovery evidence |
| `kubernetes.json` | Optional stateless multi-host Kubernetes export |
@@ -142,7 +145,54 @@ installation. Separate amd64/arm64 API and WebUI images are joined into OCI
indexes and run as non-root identities. The release assets include CycloneDX
application SBOMs, SLSA-style provenance, exact composition evidence, the
single-file deployer, its detached Ed25519 signature, and a signed, expiring
distribution manifest.
distribution manifest. Evidence generation and signing run through the
workflow's isolated release Python environment so their cryptographic tooling
is explicit and independent of packages preinstalled in the Actions runner.
The API image points Core at the migration scripts installed from the verified
wheel under `/opt/govoplan/runtime/govoplan_core_runtime`; migrations therefore
do not depend on a source checkout or the build host's Python installation
scheme.
Before publication, the exact amd64 and arm64 image manifests each run release
migrations against the pinned PostgreSQL image, reach API and WebUI readiness
as non-root/read-only processes, and complete a task through the pinned Redis
image and packaged worker. Sanitized per-platform smoke receipts are retained
as immutable release assets.
PostgreSQL and Redis indexes are resolved to untagged platform-child digests
before each smoke run. This keeps the evidence architecture-specific and
avoids retargeting one local Docker tag between incompatible platforms.
The CI host registers arm64 execution with an explicitly supplied,
digest-pinned `tonistiigi/binfmt` image immediately before the smoke. This
privileged helper is confined to the release runner and is never part of a
GovOPlaN target deployment or its runtime image set.
Because QEMU user-mode execution triggers Redis's arm64 host-kernel COW guard,
the arm64 smoke suppresses only `ARM64-COW-BUG` while persistence, snapshots,
and append-only files are disabled. Target Redis services never inherit this
test-only option.
The smoke also proves a bounded post-migration table contract and aborts as
soon as a required container exits, rather than allowing a dead process to
consume the full readiness timeout.
Ingress acceptance streams generated configuration into Docker-managed
volumes before starting the read-only containers. It therefore also works when
an Actions job reaches a host or remote Docker daemon through a mounted socket;
the drill never assumes that a job-container path is visible to that daemon.
The drill allocates explicit loopback-only host ports and verifies Docker's
host binding configuration, avoiding daemon-specific random-port shorthand
behavior. Because an Actions job and deployment containers may be Docker
siblings, functional HTTP/TLS checks run from the digest-pinned API image on
the deployment network instead of assuming the Docker host is job-local.
The dispatch-only `Runtime Ingress Drill` workflow exposes the same bounded
check independently so ingress changes can be diagnosed before an immutable
runtime publication; it accepts only digest-pinned Caddy, HAProxy, and API
images and has no push trigger.
The official Caddy binary carries the `NET_BIND_SERVICE` file capability. The
managed-ingress container therefore drops every capability and adds back only
`NET_BIND_SERVICE`; otherwise Linux rejects the binary at `execve` before its
high-port configuration can start. `no-new-privileges`, a read-only root
filesystem, and non-privileged container ports remain enforced.
The bounded setup helper writes only generated public configuration as root so
it can initialize a new volume; the actual HAProxy process retains the image's
non-root identity and runs read-only with all capabilities dropped.
The manifest contract is
[`runtime-distribution-manifest.schema.json`](runtime-distribution-manifest.schema.json),
@@ -171,26 +221,39 @@ references and archive hashes; mutable tags or incomplete bundles are rejected.
## Current Production Gates
The tool deliberately reports blockers instead of pretending the source tree is
a production distribution:
The first immutable production-distribution baseline is published as
[`v0.1.14`](https://git.add-ideas.de/GovOPlaN/govoplan/releases/tag/v0.1.14)
from source commit `1f039dd39c1ce2672f4978c8abc6dff862ef1445`. Runtime
Distribution [run #459](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/459)
proved migrations, schema compatibility, non-root API/Web readiness, and worker
delivery/shutdown on both `linux/amd64` and `linux/arm64`. Its signed manifest
has SHA-256
`d703267e01855dee63200cb20921c91c3f95fbff550c8ca76e9a35cba3f69109`
and pins these runtime indexes:
1. **First publication.** The protected workflow and fail-closed artifact
contracts are implemented, but a release operator must configure the Gitea
registry/release tokens and runtime Ed25519 key, publish the first pinned
release, and retain its amd64/arm64 readiness evidence.
3. **First administrator.** Production needs a one-time, restricted enrollment
- API: `git.add-ideas.de/govoplan/runtime-api@sha256:197ed01790986f2bc927eaa5d8348fa118702e5d2dc05feb851fc2643c23764a`
- WebUI: `git.add-ideas.de/govoplan/runtime-web@sha256:e936cca124f1fad29a067834cf17627d4c236410fdc3fa129e0ccb26b8193812`
The signed bootstrap has SHA-256
`1ff946fba82b0895d153b23352d06e30fe18388450dfd37fed6fb9912310efc5`
and key id `runtime-distribution-2026-01`. The managed-ingress boundary passed
the same publication run and the independently dispatchable Runtime Ingress
Drill [run #458](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/458).
Every later release must renew this evidence; the following target-specific
gates remain:
1. **First administrator.** Production needs a one-time, restricted enrollment
identity. The development bootstrap must not be enabled in production.
4. **Image/module composition.** The deployer now enforces the signed
2. **Image/module composition.** The deployer enforces the signed
composition. A selected module not shipped by that release cannot be
enabled.
5. **Deployment agent.** Web updates need a separate privileged reconciler with
3. **Deployment agent.** Web updates need a separate privileged reconciler with
a typed command allowlist. The API and browser must never receive the Docker
socket or arbitrary shell access.
6. **Ingress reachability evidence.** Managed Caddy ingress and the
4. **Target reachability evidence.** Managed Caddy ingress and the
existing-proxy contract are implemented. A production claim still requires
running `doctor` from the target host after public DNS/firewall changes and
retaining the first successful container drill and public TLS/readiness
evidence.
retaining public TLS/readiness evidence for that deployment.
`apply --allow-unverified-images` is therefore restricted to the evaluation
profile. It explicitly acknowledges both mutable image identities and
@@ -369,10 +432,12 @@ starts, recovery is forward-only unless an independently verified database
backup is restored. See
[Recovery And Rollback Guarantees](RECOVERY_AND_ROLLBACK_GUARANTEES.md).
Production updates still need an operator-provided database backup/restore
gate, database compatibility declaration, image signature verification, and
deployment-specific drain policy. The deployment journal proves its own
actions; it does not manufacture backup evidence.
Production updates still need operator/provider-created coordinated backup and
restore evidence, a database compatibility declaration, and a
deployment-specific drain policy. The deployer now verifies and enforces the
signed evidence before migration, but does not manufacture backups or receive
provider administration credentials. See
[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md).
## Stateless Kubernetes Runtime
@@ -391,7 +456,9 @@ The output includes a release-specific migration Job, database-head wait init
containers, API readiness/liveness probes, rolling Deployments, Services, Pod
disruption budgets, a tokenless ServiceAccount, and one fenced scheduler. Apply
the named Secret through the cluster's secret manager and review ingress proxy
CIDRs before deployment. Detailed rollout and scaling rules live in
CIDRs before deployment. A release-changing export requires adopted backup
evidence and carries only its sanitized digest and identifiers as Job
annotations. Detailed rollout and scaling rules live in
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
## Recovery Commands
@@ -437,15 +504,27 @@ of the reviewed update recipe instead of a non-functional update button.
## Distribution Workflow
The downloadable entry point is a release asset. Obtain the zipapp, detached
signature, checksum, and trusted public keyring through independently
authenticated paths before execution:
The downloadable entry point is a reproducible release asset: sorted source
paths, fixed ZIP metadata, fixed compression settings, and identical source
bytes produce an identical zipapp regardless of checkout timestamps. Obtain the
zipapp, detached signature, checksum, and trusted public keyring through
independently authenticated paths before execution:
```sh
curl --proto '=https' --tlsv1.2 --fail --location \
https://git.add-ideas.de/GovOPlaN/govoplan/releases/download/vX.Y.Z/govoplan-deploy.pyz \
--output govoplan-deploy.pyz
sha256sum --check govoplan-deploy.pyz.sha256
python3 - <<'PY'
import json
from pathlib import Path
keyring = json.loads(Path("distribution-keyring.json").read_text())
active = [key for key in keyring["keys"] if key["status"] == "active"]
if len(active) != 1:
raise SystemExit("expected exactly one active runtime release key")
Path("runtime-release-public.pem").write_text(active[0]["public_key_pem"])
PY
openssl pkeyutl -verify -pubin -inkey runtime-release-public.pem -rawin \
-in govoplan-deploy.pyz -sigfile govoplan-deploy.pyz.sig
python3 govoplan-deploy.pyz init
+228 -102
View File
@@ -15,7 +15,14 @@ machine-readable field, label, translation, route, API-reference, and module
manifest evidence. This hand-maintained document remains the reviewed product
interpretation and rollout ledger; generated evidence does not replace it.
Snapshot refreshed: 2026-07-22.
Snapshot refreshed: 2026-08-03.
The generated snapshot contains 65 module manifests, 35 WebUI-contributing
repositories, 40 statically declared module routes, 1,156 UI fields, and 836
backend endpoints. All backend endpoints are classified and no stale endpoint
declarations were found. The 234 endpoints without a static WebUI reference are
kept visible as review evidence; they may intentionally serve workers, public
clients, connectors, or external integrations.
Evidence was read from tracked Git `HEAD` in the local GovOPlaN checkouts:
@@ -51,37 +58,73 @@ Inventory states:
| Surface | Owner and code evidence | Audience/access evidence | Primary task and target archetype | Audit / rollout |
| --- | --- | --- | --- | --- |
| Public landing and login | `govoplan-core` `PublicLandingPage`; rendered while no authenticated principal exists | Unauthenticated; maintenance and backend-reachability context are shell inputs | Understand the service and authenticate; public entry | Unreviewed; later public-entry audit |
| Session/bootstrap state | `govoplan-core` `App.tsx` and `AppShell` | All browser sessions during bootstrap | Understand that session/platform state is loading; state contract | Unreviewed; core shell |
| `/` authenticated redirect | `govoplan-core` chooses the first visible navigation destination | Authenticated; result depends on visible nav contributions | Enter the actor's first accessible service area; navigation behavior, not a content page | Unreviewed; focused-view/default-route work must preserve this fallback |
| `/dashboard` fallback | `govoplan-core` `DashboardPage` only when the Dashboard module is absent | Authenticated; no route-specific scope in core | Cross-module starting point; dashboard | Unreviewed; compare with module dashboard before shared changes |
| `/settings` | `govoplan-core` `SettingsPage` | Authenticated; contributed sections and integrations filter internally | Profile, UI/workspace preference, local connection, and user-scoped integration settings; configuration | Unreviewed; Core #225 program |
| Shell chrome | `AppShell`, `Titlebar`, `IconRail`, `BreadcrumbBar`, `HelpMenu`, language menu, unsaved-change provider | Public/authenticated variants; nav filtered later | Tenant/actor context, global navigation, help, language, session and maintenance state | Unreviewed; platform-owned prerequisite for focused views |
| Public landing and login | `govoplan-core` `PublicLandingPage`; rendered while no authenticated principal exists | Unauthenticated; maintenance and backend-reachability context are shell inputs | Understand the service and authenticate; public entry | Core shell contract complete under Core #227: semantic entry/login, uniform reachable/offline/maintenance feedback, keyboard focus, responsive layout and privacy-safe pre-authentication state |
| Session/bootstrap state | `govoplan-core` `App.tsx` and `AppShell` | All browser sessions during bootstrap | Understand that session/platform state is loading; state contract | Core shell contract complete: loading, unreachable, maintenance, authentication-required and module-load failure states use shared status/alert boundaries without erasing the shell |
| `/` authenticated redirect | `govoplan-core` chooses the first visible navigation destination | Authenticated; result depends on visible nav contributions | Enter the actor's first accessible service area; navigation behavior, not a content page | Core route/module-permutation contract complete; permission, module, View and fallback filtering precede navigation and do not execute a domain action |
| `/dashboard` fallback | `govoplan-core` `DashboardPage` only when the Dashboard module is absent | Authenticated; no route-specific scope in core | Cross-module starting point; dashboard | Core fallback and Dashboard module permutations complete; fallback remains usable without the optional Dashboard module |
| `/settings` | `govoplan-core` `SettingsPage` | Authenticated; contributed sections and integrations filter internally | Profile, UI/workspace preference, local connection, and user-scoped integration settings; configuration | Core-owned pattern migration complete in [Core #225](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/225), commit `fa32cca` |
| Shell chrome | `AppShell`, `Titlebar`, `IconRail`, `BreadcrumbBar`, `HelpMenu`, language menu, unsaved-change provider | Public/authenticated variants; nav filtered later | Tenant/actor context, global navigation, help, language, session and maintenance state | Core shell contract complete under Core #227/#225 and Views #2: semantic global controls, scroll-safe rail, visible maintenance state, guarded navigation, configured Docs fallback, optional Search, responsive/theme/i18n checks and module permutations |
## Direct Module Route Contributions
The access guard column reports only the route-level declaration in
`module.ts`. Inner APIs and controls may impose additional checks.
The access column summarizes only the route-level declaration in `module.ts`.
Inner APIs and controls may impose additional checks. Public and compatibility
routes are called out explicitly because they do not have the same manifest
semantics as authenticated navigation routes.
| Route | Owner / render evidence | Route-level access evidence | Primary task | Target archetype | Status / priority |
| --- | --- | --- | --- | --- | --- |
| `/admin` | `govoplan-access` `AdminPage` | Any core `adminReadScopes` | Administer system and tenant concerns assembled from module sections | Administration/configuration | Contributed; unreviewed; P1 under [Core #225](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/225) |
| `/address-book` | `govoplan-addresses` `AddressBookPage` | `addresses:contact:read` | Browse and manage contacts, address books, and lists | Directory/list-detail | Contributed; unreviewed; P2 after Campaign |
| `/calendar` | `govoplan-calendar` `CalendarPage` | `calendar:event:read` | Browse calendars/events and act on calendar data | Directory/list-detail | Contributed; metadata gap; unreviewed; P2 after Campaign |
| `/campaigns` | `govoplan-campaign` `CampaignListPage` | `campaigns:campaign:read` | Find, compare, create, and open campaigns | List-detail entry | Pilot; P1 [Campaign #74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74) |
| `/campaigns/:campaignId/*` | `govoplan-campaign` `CampaignResourceRoute` and `CampaignWorkspace` | `campaigns:campaign:read`, plus resource probe | Configure, review, send, and inspect one campaign/version | List-detail workspace containing edit, review, monitoring, and evidence surfaces | Pilot; P1 Campaign #74 |
| `/operator` | `govoplan-campaign` `OperatorQueuePage` | `campaigns:campaign:read` and any of queue, control, retry, or reconcile | Monitor and intervene in campaign jobs through authority-specific controls | Monitoring/work queue | Pilot; durable queue controls delivered in [Campaign #78](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/78); #74 audit remains |
| `/reports` | `govoplan-campaign` `AggregateReportsPage` | `campaigns:report:read` | Compare privacy-protected cross-campaign outcome totals without recipient detail, diagnostics, export, or drill-down | Aggregate reporting | Pilot; aggregate-reader surface delivered in [Campaign #80](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/80); #74 audit remains |
| `/templates` | `govoplan-campaign` `TemplatesPage` | No route guard declared in `module.ts` | Browse/manage campaign templates | Directory/list-detail | Pilot audit; permission intent must be verified; P2 |
| `/dashboard` | `govoplan-dashboard` `DashboardPage` | No route-specific scope | Assemble module-provided actionable widgets | Dashboard | Contributed; unreviewed; P2 |
| `/docs` | `govoplan-docs` `DocsPage` | Docs read or system/tenant settings read scopes | Read configured, available, and evidence-aware documentation | Documentation directory/reference | Contributed; unreviewed; P1 [Docs #15](https://git.add-ideas.de/GovOPlaN/govoplan-docs/issues/15) after initial pattern content |
| `/files` | `govoplan-files` `FilesPage` | `files:file:read` | Browse folders/files and perform managed-file work | Directory/explorer | Contributed; metadata gap; unreviewed; P2 after Campaign |
| `/idm` | `govoplan-idm` `IdmPage` | Any IDM assignment/write or organization function-assign scope | Inspect and govern identity/function assignments | List-detail/configuration | Contributed; unreviewed; P2 |
| `/mail` | `govoplan-mail` `MailboxPage` | `mail:mailbox:read` | Browse mailboxes and messages | Directory/list-detail | Contributed; metadata gap; unreviewed; P2 after Campaign |
| `/notifications` | `govoplan-notifications` `NotificationCenterPage` | `notifications:notification:read` | Inspect and acknowledge notification state | List-detail/inbox | Contributed without a nav item or backend frontend metadata; navigation intent unknown; P2 discovery |
| `/ops` | `govoplan-ops` `OpsPage` | Ops read or system/tenant settings read scopes | Inspect runtime health and readiness | Monitoring | Contributed; unreviewed; P2 |
| `/organizations` | `govoplan-organizations` `OrganizationsPage` | Organization model/unit/function or admin settings read scopes | Model and inspect organizational structures/functions | Directory/list-detail | Contributed; unreviewed; P2 |
| `/scheduling` | `govoplan-scheduling` `SchedulingPage` | `scheduling:schedule:read` | Plan and decide scheduling requests and availability | List-detail/guided decision | Contributed; metadata gap; unreviewed; P2 |
| Routes | Owner | Route-level access | Primary archetype | Migration issue |
| --- | --- | --- | --- | --- |
| `/admin` | Access | Any declared administration/read scope | Administration/configuration host | Access pattern migration complete in [Access #19](https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/19), commit `1409dbf`; shared host contract complete in [Core #225](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/225) |
| `/address-book` | Addresses | `addresses:contact:read` | Governed source directory, contact/list detail, external-provider operation, governance facts, and reversible correction | Addresses pattern migration complete in [Addresses #23](https://git.add-ideas.de/GovOPlaN/govoplan-addresses/issues/23), commit `f9a7185` |
| `/approvals` | Approvals | `approvals:workspace:read` | Work queue/guided decision | Approvals pattern migration complete in [Approvals #3](https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/3), commit `24e9559` |
| `/calendar` | Calendar | `calendar:event:read` | Full-height calendar workspace with filterable collection/agenda sidebar, continuous and bounded date views, guarded VEVENT and source editors, synchronized-source status, durable outbox recovery, and destructive remote-move evidence | Calendar pattern migration complete in [Calendar #22](https://git.add-ideas.de/GovOPlaN/govoplan-calendar/issues/22), commit `d7fd944` |
| `/campaigns`, `/campaigns/:campaignId/*`, `/campaigns/queue`, `/campaigns/reports` | Campaign | Campaign read/report/control scopes | List-detail, guided review, monitoring, reporting | Campaign pattern pilot complete in [Campaign #74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74); bounded product features such as watched-folder policy remain independently tracked |
| `/operator` | Campaign | Campaign read plus queue/control scope | Compatibility redirect to `/campaigns/queue` | Campaign #74 complete; redirect remains declared for saved links and is retired under the compatibility policy rather than through the UI migration |
| `/cases`, `/cases/:caseId` | Cases | `cases:case:read` | Governed case directory and detail workspace with guarded OCC lifecycle editor, provider-owned references, immutable timeline/history, and confirmed object-access editor | Cases pattern migration complete in [Cases #4](https://git.add-ideas.de/GovOPlaN/govoplan-cases/issues/4), commit `43b4cc8` |
| `/committee` | Committee | `committee:workspace:read` | Governed workspace | Committee pattern migration complete in [Committee #2](https://git.add-ideas.de/GovOPlaN/govoplan-committee/issues/2), commit `e64af30` |
| `/dashboard` | Dashboard | No route-specific scope | View-specific personal workspace with module/permission-filtered widget library, guarded four-column composition, nested widget settings, server/browser fallback, and optimistic layout persistence | Dashboard pattern migration complete in [Dashboard #3](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard/issues/3), commit `da3947f` |
| `/dataflow` | Dataflow | Pipeline read/admin | Governed library, guarded graph/constrained-SQL definition editor, typed node inspector, bounded intermediate preview, automation triggers, and durable run/deployment evidence | Dataflow pattern migration complete in [Dataflow #20](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/20), commit `109ddcd` |
| `/datasources` | Datasources | Catalogue read/source admin | Governed catalogue, staging preflight, optional-origin directory, authority editor, and immutable evidence | Datasources pattern migration complete in [Datasources #7](https://git.add-ideas.de/GovOPlaN/govoplan-datasources/issues/7), commit `6406ce7` |
| `/distribution-lists` | Distribution Lists | List read/write/admin | Governed directory, immutable-revision editor, expansion preview, and evidence register | Distribution Lists pattern migration complete in [Distribution Lists #8](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists/issues/8), commit `6cdd804` |
| `/docs` | Docs | Documentation or settings read | Documentation/reference | Configured-system workflow/reference/pattern help complete in [Docs #15](https://git.add-ideas.de/GovOPlaN/govoplan-docs/issues/15), commit `abe2f78` |
| `/files` | Files | `files:file:read` | Directory/explorer | Files pattern migration complete in [Files #42](https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/42), commit `d8ae506` |
| `/forms` | Forms | `forms:definition:read` | Definition library/editor | Forms pattern migration complete in [Forms #4](https://git.add-ideas.de/GovOPlaN/govoplan-forms/issues/4), commit `e505536` |
| `/forms-runtime`, `/forms-runtime/:instanceId` | Forms Runtime | Participate or workspace read | Guided form execution | Forms Runtime pattern migration complete in [Forms Runtime #5](https://git.add-ideas.de/GovOPlaN/govoplan-forms-runtime/issues/5), commit `07dd35b` |
| `/idm` | IDM | Assignment, function-change, relationship, or organization scopes | Directory/governed change | IDM pattern migration complete in [IDM #12](https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/12), commit `d864317` |
| `/mail`, `/mail/bounces` | Mail | Mailbox or bounce read/manage | Directory/explorer, operational evidence | Mail pattern migration complete in [Mail #20](https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/20), commit `7844d9c` |
| `/notifications` | Notifications | `notifications:notification:read` | Inbox/list-detail with guarded recipient state, confirmed local cancellation/dispatch, and sanitized delivery evidence | Notifications pattern migration complete in [Notifications #4](https://git.add-ideas.de/GovOPlaN/govoplan-notifications/issues/4), commit `ad6a31f` |
| `/ops` | Ops | Operations or settings read | Monitoring/evidence with contextual run, drain, readiness-blocker, and recovery guidance | Ops pattern migration complete in [Ops #4](https://git.add-ideas.de/GovOPlaN/govoplan-ops/issues/4), commit `2b32643` |
| `/organizations` | Organizations | Model/unit/function or settings read | Directory/hierarchy editor | Organizations pattern migration complete in [Organizations #7](https://git.add-ideas.de/GovOPlaN/govoplan-organizations/issues/7), commit `97acfcb` |
| `/portal` | Portal | `portal:service:read` | Explained service directory and governed handoff | Portal pattern migration complete in [Portal #2](https://git.add-ideas.de/GovOPlaN/govoplan-portal/issues/2); durable evidence in `govoplan-portal/docs/INTERFACE_PATTERN_MIGRATION.md` |
| `/postbox` | Postbox | `postbox:postbox:read` | Inbox/list-detail | Postbox pattern migration complete in [Postbox #26](https://git.add-ideas.de/GovOPlaN/govoplan-postbox/issues/26), commit `a97eb3b` |
| `/projects` | Projects | `projects:project:read` | Revisioned list-detail/project workspace | Projects pattern migration complete in [Projects #2](https://git.add-ideas.de/GovOPlaN/govoplan-projects/issues/2); durable evidence in `govoplan-projects/docs/INTERFACE_PATTERN_MIGRATION.md` |
| `/reporting`, `/reports` | Reporting | `reporting:definition:read` | Governed report catalogue, analytical workspace and evidence | Reporting pattern migration complete in [Reporting #8](https://git.add-ideas.de/GovOPlaN/govoplan-reporting/issues/8); durable evidence in `govoplan-reporting/docs/INTERFACE_PATTERN_MIGRATION.md` |
| `/risk-compliance` | Risk Compliance | Workspace or sanctions read | Immutable source evidence, version-pinned screening, list-detail review, and revisioned assurance graph with explicit blockers and consequences | Risk Compliance pattern migration complete in [Risk Compliance #8](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance/issues/8), commit `24d80a6` |
| `/scheduling` | Scheduling | `scheduling:schedule:read` | List-detail/guided decision | Scheduling pattern migration complete in [Scheduling #8](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling/issues/8), commit `c17cbda` |
| `/scheduling/public/:requestId/:token` | Scheduling | Public signed token | Public participation | Scheduling #8 complete in `c17cbda` |
| `/search` | Search | `search:result:read` | Keyboard-first global/context overlay and full results fallback | Search pattern migration complete in [Search #4](https://git.add-ideas.de/GovOPlaN/govoplan-search/issues/4); durable evidence in `govoplan-search/docs/INTERFACE_PATTERN_MIGRATION.md` |
| `/templates` | Templates | Template read/write/publish/render/admin | Governed library, immutable-revision editor, compatibility preview, and render evidence | Templates pattern migration complete in [Templates #5](https://git.add-ideas.de/GovOPlaN/govoplan-templates/issues/5), commit `72fafa2` |
| `/voting` | Voting | `voting:ballot:read` | Governed ballot workspace | Voting pattern migration complete in [Voting #1](https://git.add-ideas.de/GovOPlaN/govoplan-voting/issues/1), commit `2625990` |
| `/workflow` | Workflow | Definition read or instance admin | Native BPMN editor, governed revision actions and execution evidence | Workflow pattern migration complete in [Workflow #15](https://git.add-ideas.de/GovOPlaN/govoplan-workflow/issues/15); durable evidence in `govoplan-workflow/docs/INTERFACE_PATTERN_MIGRATION.md` |
## Final Module Closure Evidence
The final five module-owned work packages complete the 2026-08-03 rollout
snapshot. Their module documents are the durable detailed inventories; the
table below records the cross-product closure evidence.
| Owner | Dominant archetype and consequential boundary | Focused evidence |
| --- | --- | --- |
| Workflow | Definition list-detail plus specialized native BPMN editor; save/activate/archive/delete/reset and instance transitions remain revisioned, confirmed, and Engine-owned | Shared dialogs/status/alerts/help, dirty-navigation guard, keyboard palette insertion, edge inspector alternative, responsive/reduced-motion contract, TypeScript and focused structure test |
| Search | Focus-contained global/context overlay plus URL-stable full results; filters only narrow permission-aware source results | F3/Ctrl/Cmd+K, listbox keyboard navigation, provider-partial diagnostics, shared controls/help, narrow layout and focused overlay/interface tests |
| Reporting | Three-region governed analytical workspace; runs, schedules, exports and publications retain purpose, permission, source and policy provenance | Shared grid/dialog/status/help, keyboard-explainable Run blockers, responsive task order, provider/semantic backend tests and focused interface test |
| Projects | Revisioned list-detail planning workspace; visibility and saves are ACL/OCC-governed and retain a change reason | Shared dialog/status/help/field labels, save errors attached to the editor, semantic list controls, responsive/focus contract and focused interface test |
| Portal | Explained service directory and exact-revision provider handoff; Portal never owns the launched case/form/workflow effect | Shared status/alert/toggle/help/blocker controls, stable disabled Open action with actor/action/destination, guarded navigation, responsive layout and focused interface test |
Future WebUI modules and newly added routes are not grandfathered by this
snapshot. They must meet the same surface definition of done in their owning
feature issue and pass the source/runtime inventory gates; they do not reopen
this finite migration program unless the pattern contract itself changes.
## Manifest And Runtime Route Alignment
@@ -91,28 +134,27 @@ loading reason about the configured interface without executing module UI code.
is recorded here as an evidence gap; this inventory does not infer whether each
gap is intentional.
| Module | `module.ts` routes | Backend manifest frontend routes | Backend nav alignment | Result |
| --- | --- | --- | --- | --- |
| Access | `/admin` | `/admin` | Aligned | Described |
| Addresses | `/address-book` | `/address-book` | Aligned | Described |
| Admin | No direct route; `admin.sections` | None | Not applicable | Composed surface |
| Audit | No direct route; `admin.sections` | None | Not applicable | Composed surface |
| Calendar | `/calendar` | None | `/calendar` nav exists | Metadata gap |
| Campaign | Five routes | None | Four top-level nav items exist | Metadata gap; wildcard resource route is also undescribed |
| Dashboard | `/dashboard` | `/dashboard` | Aligned | Described |
| Docs | `/docs` | `/docs` | Aligned | Described |
| Files | `/files` | None | `/files` nav exists | Metadata gap |
| IDM | `/idm` | `/idm` | Aligned | Described |
| Mail | `/mail` | None | `/mail` nav exists | Metadata gap |
| Notifications | `/notifications` | No frontend metadata | No nav item | Metadata and discovery gap |
| Ops | `/ops` | `/ops` | Aligned | Described |
| Organizations | `/organizations` | `/organizations` | Aligned | Described |
| Policy | No direct route; `admin.sections` | None | Not applicable | Composed surface |
| Scheduling | `/scheduling` | None | `/scheduling` nav exists | Metadata gap |
The generated comparison is aligned for all authenticated canonical routes.
Two deliberate exceptions remain visible:
Before a release claims a complete configured-system route inventory, add a
contract check or explicit exceptions so executable routes and manifest
metadata cannot silently diverge.
- Campaign contributes `/operator` as a compatibility redirect for saved View
projections; its canonical and manifest-declared destination is
`/campaigns/queue`.
- Scheduling contributes `/scheduling/public/:requestId/:token` through the
separate `publicRoutes` contract. Authenticated manifest routes intentionally
do not describe public signed-token entry points yet.
Admin, Audit, Policy, Tenancy, and Views contribute composed administration or
settings surfaces rather than direct routes. Their migration issues are
[Admin #8](https://git.add-ideas.de/GovOPlaN/govoplan-admin/issues/8),
[Audit #8](https://git.add-ideas.de/GovOPlaN/govoplan-audit/issues/8),
[Policy #11](https://git.add-ideas.de/GovOPlaN/govoplan-policy/issues/11),
[Tenancy #6](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy/issues/6), and
[Views #2](https://git.add-ideas.de/GovOPlaN/govoplan-views/issues/2).
Release evidence must continue to run the generated inventory and manifest
shape checks so new executable routes, public routes, aliases, and composed
surfaces cannot silently diverge from their declared metadata.
## Composed Surfaces And Extension Points
@@ -121,25 +163,108 @@ enabled and the actor passes the declared filters.
| Host surface | Contributor and evidence | Contributed regions/actions | Pattern implication | Audit |
| --- | --- | --- | --- | --- |
| `/admin` | Access host (`AdminPage`) | System tenants/users/roles, tenant users/groups/roles/API keys/settings, function-role mappings, user/group mail and file connector scopes | One stable admin information architecture must contain both host-owned and contributed sections | Unreviewed; P1 Core #225 |
| `/admin` | `govoplan-admin` `admin.sections` | Overview; system settings; configuration changes; configuration packages; role/group templates; module management | Configuration, guided operations, review/preflight, consequence | In progress under Core #225; surface-level evidence still needed |
| `/admin` | `govoplan-audit` `admin.sections` | System audit; tenant audit | Evidence/provenance and reporting | Unreviewed |
| `/admin` | `govoplan-files` `admin.sections` and `files.connectors` | System and tenant file connections plus scoped connector managers used by Access | Adaptive configuration, discovery/test, policy and credentials | First migration family in Core #225; verification incomplete in this inventory |
| `/admin` | `govoplan-organizations` `admin.sections` | Tenant organization settings | Configuration/list-detail | Unreviewed |
| `/admin` | `govoplan-policy` `admin.sections` | System, tenant, group, and user retention | Effective value, source/provenance, consequential configuration | Unreviewed; Core #225 phase 4 |
| `/admin` and `/settings` | `govoplan-mail` `mail.profiles` | System/tenant/group/user mail profile and policy managers | Same server/credential/policy grammar as file connectors | Unreviewed; Core #225 mail migration |
| `/settings` | Core host | Profile; interface; workspace; local connection | Personal configuration with adaptive forms and immediate feedback | Unreviewed |
| `/settings` | Files and Mail named capabilities | User-scoped file connections and mail profiles/policy | Optional integration regions disappear cleanly when capability absent | Unreviewed |
| `/settings` | `govoplan-notifications` `settings.sections` | Notification preferences | Personal configuration | Unreviewed |
| `/dashboard` | Dashboard host and `dashboard.widgets` | Installed-modules widget; Ops health widget when Ops contributes it | Widget ordering, staleness, permissions, destination behavior | Unreviewed |
| `/organizations` | IDM `organizations.functionActions` | Action leading to assignment view filtered by IDM scopes | Cross-module context action through explicit capability | Unreviewed |
| Campaign attachments/import | Files `files.fileExplorer` | Folder tree, managed chooser, file listing/pattern resolution/sharing | Optional domain composition without sibling-private imports | Pilot audit under Campaign #74 |
| Campaign review/send | Mail runtime `mail.devMailbox` | Mock-mail verification when backend advertises runtime capability | Optional review stage with unavailable/optional states | Pilot audit under Campaign #63/#62 |
| `/admin` | Access host (`AdminPage`) | System tenants/users/roles, tenant users/groups/roles/API keys/settings, function-role mappings, user/group mail and file connector scopes | One stable admin information architecture must contain both host-owned and contributed sections | Pattern migration, contextual help, explained permission/protection states, optional-module blockers, localization, and focused evidence complete in Access #19 (`1409dbf`); Core #225 shared host contract complete |
| `/admin` | `govoplan-admin` `admin.sections` | Overview; system settings; configuration changes; configuration packages; role/group templates; module management | Configuration, guided operations, review/preflight, consequence | Pattern migration, contextual help, explained permission/protection/applicability states, guarded consequential actions, localization, and focused evidence complete in Admin #8 (`d428f33`) |
| `/admin` | `govoplan-tenancy` `admin.sections` | System tenant registry and active-tenant settings | Administration directory, effective configuration, lifecycle consequence | Pattern migration, contextual help, explained permission/lifecycle/system-policy states, dirty-state guards, localization, and focused evidence complete in Tenancy #6 (`e76fe16`) |
| `/admin` | `govoplan-audit` `admin.sections` | System audit; tenant audit | Evidence/provenance and reporting | Pattern migration, localized evidence projection, contextual help, and focused tests complete in Audit #8 (`6d3fcc1`) |
| `/admin` | `govoplan-files` `admin.sections` and `files.connectors` | System and tenant file connections plus scoped connector managers used by Access | Adaptive configuration, discovery/test, policy and credentials | Pattern migration, contextual help, blocker explanations and focused evidence complete in Files #42 (`d8ae506`) |
| `/admin` | `govoplan-organizations` `admin.sections` | Tenant organization settings | Configuration/list-detail | Pattern migration, tenant-owned provenance, contextual help, guarded settings/editor drafts, explained permission states, localization and focused evidence complete in Organizations #7 (`97acfcb`) |
| `/admin` | `govoplan-policy` `admin.sections` | System, tenant, group, and user retention | Effective value, source/provenance, consequential configuration | Pattern migration complete in Policy #11 (`f964ed7`) with Core editor contract `fa32cca` |
| `/admin` and `/settings` | `govoplan-mail` `mail.profiles` | System/tenant/group/user mail profile and policy managers | Same server/credential/policy grammar as file connectors | Pattern migration, contextual help, policy/target/permission blockers and focused evidence complete in Mail #20 (`7844d9c`; shared test-reason contract Core `2d0551a`) |
| `/settings` | Core host | Profile; interface; workspace; local connection | Personal configuration with adaptive forms and immediate feedback | Pattern migration complete in Core #225 (`fa32cca`) |
| `/settings` | Files and Mail named capabilities | User-scoped file connections and mail profiles/policy | Optional integration regions disappear cleanly when capability absent | Files #42, Mail #20 and Core #225 complete |
| `/admin` and `/settings` | `govoplan-views` `admin.sections`, `settings.sections`, and `views.runtime` | System/tenant definition and assignment editors, personal/group editors, global selector | Versioned presentation projection with inheritance, lockout safeguards, optional directory targets, and no authorization effect | Pattern migration, contextual help, localized selector/editor, guarded drafts, explained inherited/permission/capability states, and focused evidence complete in Views #2 (`c125f33`) |
| `/settings` | `govoplan-notifications` `settings.sections` | Notification preferences | Personal configuration | Pattern migration, contextual help, permission/target explanation, typed toggles and focused evidence complete in Notifications #4 (`ad6a31f`) |
| `/dashboard` | Dashboard host and `dashboard.widgets` | Installed-modules widget; Ops health widget when Ops contributes it | Widget ordering, staleness, permissions, destination behavior | Pattern migration, view-aware composition, keyboard/drag alternatives, responsive packing, module filtering and focused evidence complete in Dashboard #3 (`da3947f`) |
| `/organizations` | IDM `organizations.functionActions` | Action leading to assignment view filtered by IDM scopes | Cross-module context action through explicit capability | IDM pattern migration complete in IDM #12 (`d864317`) |
| Campaign attachments/import | Files `files.fileExplorer` | Folder tree, managed chooser, file listing/pattern resolution/sharing | Optional domain composition without sibling-private imports | Campaign #74 pilot complete; watched-folder and duplicate-attachment product policy remain independent Campaign #60/#61 features |
| Campaign review/send | Mail runtime `mail.devMailbox` | Mock-mail verification when backend advertises runtime capability | Optional review stage with unavailable/optional states | Explicit intervention and review-progress vocabulary delivered in Campaign #63; send modes/progress delivered in #62/#79 |
Other named capability exports (`files.connectors`, `organizations.functionPicker`,
and mail profile validation) are contracts consumed inside the composed surfaces
above; they are not independent routes.
## Core Configuration Surface Map
Core #225 now supplies and verifies the platform-owned configuration contract.
The durable Core inventory is
`govoplan-core/docs/INTERFACE_PATTERN_MIGRATION.md`.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Completion evidence |
| --- | --- | --- | --- | --- |
| `/settings` (`SettingsPage`) | Change personal profile, interface/workspace preferences, or local development connection | Two-zone typed settings workspace | Changes are user-scoped; save and test actions distinguish clean, busy, and active states | Contextual help, unsaved guard, typed controls and keyboard-explainable disabled actions in Core `fa32cca` |
| Reusable credentials (`CredentialEnvelopeManager`) | Compare and configure scoped reusable authentication material | Repeated administration plus adaptive create/edit | Secret values are write-only; permission and missing-owner states block mutation explicitly; deletion can break dependent connections | Actionable blocker, stable row actions, typed references, unsaved guard and shared destructive confirmation |
| Retention (`RetentionPolicyManagement`) | Inspect effective retention and narrow permitted local values | Effective-policy editor | Parent locks, source paths and write authority control whether sensitive evidence can be retained | Typed narrowing controls, source-path help, lock/target/permission blockers and clean/loading/save reasons |
| Shared configuration primitives | Compose module-owned settings without sibling-private imports | Platform behavior contract | Consequence, focus, help, async, confirmation and permission semantics remain consistent | Core component suites, 121 module-system tests and full-product type/build/bundle gates |
No primary Core configuration flow requires raw JSON. Expert JSON remains
limited to diagnostics, interchange, conflict evidence, or read-only inspection.
## Policy Surface Map
Policy #11 verifies the four composed retention sections. The durable
module-level inventory is
`govoplan-policy/docs/INTERFACE_PATTERN_MIGRATION.md`.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Completion evidence |
| --- | --- | --- | --- | --- |
| System retention | Set the instance ceiling and run retention | Effective-policy editor plus destructive operation | An applied run can irreversibly redact/delete retained content; dry-run and applied evidence remain distinct | Core source-path/lock contract, permission and busy reasons, shared confirmation, typed/filterable outcome grid and audit-oriented wording |
| Tenant retention | Narrow the inherited system ceiling | Effective-policy editor | Tenant policy cannot silently loosen its parent | Core typed controls, effective path and parent-lock explanation |
| Group and user retention | Select an authorized target and narrow inherited policy | Targeted effective-policy editor | Selection exposes only bounded account/group labels; no retained content is returned | Delta-backed target loading, retry, missing-target blocker and responsive shared admin composition |
Automated evidence for Policy `f964ed7` comprises 50 backend/manifest tests,
the Policy interface structural gate, 65 manifest-shape checks, and the
full-product TypeScript/Vite build with structural localization, theme and
bundle-budget gates. Policy uses no sibling-private imports.
## Files Surface Map
Files #42 classifies and verifies the complete Files-owned route and composition
boundary. The durable module-level inventory is
`govoplan-files/docs/INTERFACE_PATTERN_MIGRATION.md`.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Completion evidence |
| --- | --- | --- | --- | --- |
| `/files` (`FilesPage`) | Browse spaces/folders and repeatedly act on current content | Full-height directory/explorer | Navigation is low consequence; upload, synchronize, move, copy and share are medium; delete is high | Stable two-pane composition, contextual help, selection/permission/state-specific disabled reasons, shared confirmation and responsive collapse |
| Upload/archive, transfer, rename and connector-import dialogs | Supply, validate and review one bounded change | Adaptive create/edit or guided import | Writes managed content and may resolve conflicts or import untrusted bytes | Shared dialogs/drop zone, bounded archive preflight, conflict review, explicit confirmation and no browser-native confirmation |
| Share/access explanation | Inspect or change who can use a resource | Review/decision | Grants can disclose content; delete/revoke changes access | Shared access explanation, action components and destructive confirmation; backend redaction remains authoritative |
| File connector tree and connection/credential dialogs | Compare and configure external endpoints and reusable credentials | Administration plus adaptive create/edit | Endpoint, secret and capability changes can enable remote access | Shared connection tree/forms/advanced panel, endpoint discovery and login test, unsaved-change guard, read-only deployment provenance and actionable disabled reasons |
| Connector policy card | Narrow effective connector use | Effective-policy editor | Inherited deny/allow rules affect lower scopes | Typed selectors, deny-precedence warning, effective sources, contextual admin help and permission blocker |
| `files.widget.spaces` | See available spaces and enter Files | Dashboard widget | Space/provider names remain permission-filtered | Shared loading, alert and status components; bounded configuration and refresh |
| `files.fileExplorer` capability | Select a governed managed snapshot for another module | Directory chooser | Exact file/version becomes another module's governed input | Capability-only composition, no sibling-private import, stable chooser/confirmation and exact snapshot evidence |
Automated evidence for commit `d8ae506` comprises 104 Files backend tests,
three focused Files WebUI structure tests, the full-product TypeScript/Vite
build, structural localization audit, theme contract and bundle budget. Shared
Dialog and disabled-tooltip behavior provide focus entry/return and
keyboard-reachable explanations; responsive source order is guarded at 1050 px
and 760 px. Secrets are not returned to the WebUI, and JSON remains only an
advanced provider-compatibility escape hatch rather than the primary editor.
## Mail Surface Map
Mail #20 classifies and verifies the complete Mail-owned route and composition
boundary. The durable module-level inventory is
`govoplan-mail/docs/INTERFACE_PATTERN_MIGRATION.md`.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Completion evidence |
| --- | --- | --- | --- | --- |
| `/mail` (`MailboxPage`) | Browse an authorized provider mailbox without changing it | Full-height directory/explorer | Message metadata and content are private; every provider read is bounded and non-mutating | Stable three-pane composition, contextual help, explicit no-profile blocker, refresh reasons, keyboard rows, paging and responsive collapse |
| Mail profile tree and profile/server/credential dialogs | Compare and configure reusable transport identities | Administration plus guided/adaptive create/edit | Endpoint and credential changes can enable external effects | Shared connection tree/dialog/stage rail/forms, focused hierarchy editors, unsaved guard, connection tests, permission/target blockers and disabled-save reasons |
| Mail policy card | Narrow profile visibility, lower-scope definitions and transport/address patterns | Effective-policy editor | Inherited allow/deny rules affect delivery and lower scopes | Typed selectors and controls, effective source path, lock/read-only blocker, dirty-save state and contextual admin help |
| `/mail/bounces` watcher table | Configure and explicitly scan bounded IMAP evidence sources | Operational administration | Provider access changes durable source cursors and evidence | Shared grid/status/loading/alerts, actionable no-profile and busy states, field help and stable row actions |
| `/mail/bounces` observations and watcher removal | Review sanitized delivery outcomes or stop future scans | Evidence/reporting plus destructive confirmation | Recipient diagnostics are sensitive; watcher removal retains existing evidence | Bounded sanitized rows and shared confirmation with retained-evidence consequence |
| `mail.profiles` and reference-selector capabilities | Select/validate Mail-owned transport from another module | Governed capability composition | A selected identity can perform external effects | Stable references, Mail-owned authorization/secret resolution, no sibling-private imports and clean optional absence |
Automated evidence for Mail commit `7844d9c` and Core commit `2d0551a`
comprises 114 Mail backend tests, Mail's focused UI/model/structure suite, the
Core shared mail-component suite, 65 manifest-shape checks and the full-product
TypeScript/Vite build with structural localization, theme and bundle-budget
gates. Shared Dialog and disabled-tooltip behavior provides focus containment,
return and keyboard-reachable explanations. Responsive source order is guarded
at 1250 px, 900 px and 760 px. Passwords remain write-only, mailbox responses
are bounded, and bounce evidence excludes raw provider messages.
## Campaign Pilot Surface Map
Campaign is detailed first because it exercises almost every archetype. The
@@ -156,70 +281,71 @@ prove that the composition or states satisfy the pattern.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Known issue / rollout |
| --- | --- | --- | --- | --- |
| Campaign list (`CampaignListPage`) | Find, compare, create, open | List-detail entry | Campaign lifecycle/status and creation | Audit in [#74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74); guided entry [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35) |
| Overview (`CampaignOverviewPage`) | Understand/edit campaign identity, version, access, lifecycle | Object overview plus adaptive edit | Lock/archive/delete/access changes need real consequence and reversibility wording | #74 remaining audit |
| Fields (`CampaignFieldsPage`) | Define recipient/template field schema | Structured editor | Schema changes can invalidate recipient/template data | #74 audit |
| Attachments/files (`AttachmentsDataPage`, `AttachmentRulesOverlay`) | Select sources and attachment/ZIP rules | Directory chooser plus adaptive rule editor | Missing or mismatched files affect built messages | #74; attachment-detail [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) |
| Recipients (`RecipientDataPage`) | Select/import/map/edit recipients, address fields and per-recipient values/files | Import/mapping plus list-detail editor | Personal data, validation, bulk activation, file links | Consolidated editor delivered in #67; #74 remaining audit and guided entry #35 |
| Template (`TemplateDataPage`, placeholder/expression dialogs) | Author subject/body and preview substitutions | Adaptive editor plus stable preview | Generated communication content and unresolved expressions | #74; stable overlay [#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73) |
| Mail settings (`MailSettingsPage` settings view) | Select/configure campaign mail transport | Adaptive configuration | Credentials, SMTP/IMAP destinations, test outcomes | #74; align with Core #225 mail pattern |
| Campaign settings (`GlobalSettingsPage` settings view) | Configure campaign behavior | Adaptive configuration | Can alter validation/build/send behavior | #74 audit |
| Mail policy (`MailSettingsPage` policy view) | Inspect/override effective mail policy | Effective policy/provenance editor | Inheritance and locks affect allowed delivery | #74; Core #225 policy pattern |
| Campaign policy (`GlobalSettingsPage` policy view) | Inspect/override campaign policy | Effective policy/provenance editor | Inheritance, actor authority, and blocked edits | #74; Core #225 policy pattern |
| Review/send (`ReviewSendPage`) | Validate, build, mock-test, confirm/send, inspect results | Guided review/decision plus durable progress | External communication, bounded synchronous execution, persisted queue mode, partial effects, retries, evidence | Bounded synchronous and explicit/persisted queued modes delivered in [#62](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/62) and [#79](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/79); [#63](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/63) wording and #74 audit remain |
| Campaign list (`CampaignListPage`) | Find, compare, create, open | List-detail entry | Campaign lifecycle/status and creation | #74 and guided entry [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35) complete |
| Overview (`CampaignOverviewPage`) | Understand/edit campaign identity, version, access, lifecycle | Object overview plus adaptive edit | Lock/archive/delete/access changes expose consequence, reversibility, owner/access and lifecycle evidence | #74 complete; lifecycle policy is independently extended in Campaign #26 |
| Fields (`CampaignFieldsPage`) | Define recipient/template field schema | Structured editor | Schema changes can invalidate recipient/template data | #74 complete |
| Attachments/files (`AttachmentsDataPage`, `AttachmentRulesOverlay`) | Select sources and attachment/ZIP rules | Directory chooser plus adaptive rule editor | Missing or mismatched files affect built messages | #74 and attachment-detail [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) complete |
| Recipients (`RecipientDataPage`) | Select/import/map/edit recipients, address fields and per-recipient values/files | Import/mapping plus list-detail editor | Personal data, validation, bulk activation, file links | Consolidated editor #67, guided entry #35 and #74 audit complete; independent bulk action #68 remains product scope |
| Template (`TemplateDataPage`, placeholder/expression dialogs) | Author subject/body and preview substitutions | Adaptive editor plus stable preview | Generated communication content and unresolved expressions | #74 and stable overlay [#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73) complete |
| Mail settings (`MailSettingsPage` settings view) | Select/configure campaign mail transport | Adaptive configuration | Credentials, SMTP/IMAP destinations, test outcomes | #74 and Core #225 shared mail pattern complete; final credential hierarchy remains Mail #10 |
| Campaign settings (`GlobalSettingsPage` settings view) | Configure campaign behavior | Adaptive configuration | Can alter validation/build/send behavior | #74 complete |
| Mail policy (`MailSettingsPage` policy view) | Inspect/override effective mail policy | Effective policy/provenance editor | Inheritance and locks affect allowed delivery | #74 and Core #225 effective-policy pattern complete |
| Campaign policy (`GlobalSettingsPage` policy view) | Inspect/override campaign policy | Effective policy/provenance editor | Inheritance, actor authority, and blocked edits | #74 and Core #225 effective-policy pattern complete |
| Review/send (`ReviewSendPage`) | Validate, build, mock-test, confirm/send, inspect results | Guided review/decision plus durable progress | External communication, bounded synchronous execution, persisted queue mode, partial effects, retries, evidence | Interventions #63, send/progress #62/#79 and #74 wording/accessibility audit complete |
| Message and attachment detail overlays | Inspect one built/mock message and its attachment links | Stable detail/review dialog | Personal data, exact outbound content, reviewed state | Delivered and verified in [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) and [#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73) |
| Campaign report (`CampaignReportPage`) | Filter and inspect delivery outcomes | Reporting/list-detail | Partial, failed, explicitly excluded/skipped, SMTP/IMAP outcomes and retries | Server-owned filtering and counts delivered in [#65](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/65) with the full-result DataGrid contract from [Core #263](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/263); excluded semantics in [#66](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/66) |
| Audit (`CampaignAuditPage`) | Inspect campaign evidence/history | Provenance timeline/report | Actor/action/effect trace | #74 audit |
| JSON (`CampaignJsonView`) | Inspect expert representation | Advanced diagnostics/reference | Raw data may contain personal/configuration values; not a primary editor | #74 privacy/redaction audit |
| Audit (`CampaignAuditPage`) | Reach campaign evidence/history | Explained provenance handoff | Campaign emits platform evidence; Audit owns reading, retention and bundles | #74 complete as an explicit Audit handoff; object-scoped projection may follow Audit #3 without a sibling-private import |
| JSON (`CampaignJsonView`) | Inspect/download expert representation | Advanced diagnostics/reference | Full authorized configuration may contain personal data but no inline transport secrets | #74 privacy audit complete with explicit sensitivity warning and campaign-read boundary |
| Create wizard (`CreateWizard`) | Seed a campaign through basics, sender, fields, recipients, template, attachments, review, send | Guided setup | Current steps mix creation and later consequential delivery; completion semantics need audit | Guided first campaign [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35) |
| Review/send wizard routes | Alternate guided review/send shells | Guided review | Tracked routes exist; implementation relationship to `ReviewSendPage` must be established, not guessed | #74 inventory decision |
| Operator queue (`OperatorQueuePage`) | Monitor jobs and intervene | Monitoring/work queue | Campaign/version/job identity, historical active-version discovery, fixed action positions, authority-aware disabled states, exact non-overlapping queue counts, server-paged jobs, bounded refresh, retry/queue/reconcile per version, campaign-wide pause/resume/cancel, and leave/return progress | Durable operator controls delivered in [#78](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/78); #74 wording/accessibility audit remains |
| Review/send wizard routes | Focus the canonical review or send stage | Guided review | Thin wrappers render the same `ReviewSendPage` with a stable initial stage; no parallel workflow state exists | #74 inventory decision complete |
| Operator queue (`OperatorQueuePage`) | Monitor jobs and intervene | Monitoring/work queue | Campaign/version/job identity, historical active-version discovery, fixed action positions, authority-aware disabled states, exact non-overlapping queue counts, server-paged jobs, bounded refresh, retry/queue/reconcile per version, campaign-wide pause/resume/cancel, and leave/return progress | Durable controls #78 and #74 wording/accessibility audit complete |
| Aggregate reports (`AggregateReportsPage`) | Compare cross-campaign delivery outcomes | Privacy-preserving aggregate reporting | Tenant/campaign ACL, deployment/tenant small-cell policy, complementary and overlapping-cell suppression, explicit denominator, and no recipient detail/diagnostics/export/drill-down | Separate aggregate-reader surface delivered in [#80](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/80); not parity with the permission-gated per-campaign detail report |
| Templates route (`TemplatesPage`) | Browse template records | Directory/list-detail | Template availability and later generated outputs | #74 audit; verify missing route guard intent |
The five review stages currently named in code are `Validate and inspect`,
`Build and review`, `Mock send and verify`, `Confirm and send`, and `Delivery
results`. Campaign #63 owns the intervention and status vocabulary; Workflow is
not required to define or implement it.
## Repositories Without A WebUI Route Contribution
## Repositories Without A WebUI Package
The following local repositories contain a backend manifest but no
`webui/src/module.ts` at this snapshot:
The generated manifest snapshot reports no WebUI package for:
`govoplan-approvals`, `govoplan-assets`, `govoplan-booking`,
`govoplan-certificates`, `govoplan-committee`, `govoplan-consultation`,
`govoplan-contracts`, `govoplan-dist-lists`, `govoplan-evaluation`,
`govoplan-facilities`, `govoplan-forms-runtime`, `govoplan-grants`,
`govoplan-helpdesk`, `govoplan-identity`, `govoplan-inspections`,
`govoplan-tickets`, `govoplan-learning`, `govoplan-permits`,
`govoplan-poll`, `govoplan-procurement`, `govoplan-records`,
`govoplan-resources`, `govoplan-rest`, `govoplan-risk-compliance`,
`govoplan-soap`, `govoplan-tenancy`, and `govoplan-transparency`.
`govoplan-assets`, `govoplan-booking`, `govoplan-certificates`,
`govoplan-connectors`, `govoplan-consultation`, `govoplan-contracts`,
`govoplan-decisions`, `govoplan-encryption`, `govoplan-evaluation`,
`govoplan-facilities`, `govoplan-grants`, `govoplan-helpdesk`,
`govoplan-identity`, `govoplan-identity-trust`, `govoplan-inspections`,
`govoplan-learning`, `govoplan-mandates`, `govoplan-parties`,
`govoplan-permits`, `govoplan-poll`, `govoplan-procurement`,
`govoplan-records`, `govoplan-resources`, `govoplan-rest`,
`govoplan-services`, `govoplan-soap`, `govoplan-tickets`,
`govoplan-transparency`, `govoplan-wiki`, and `govoplan-workflow-engine`.
This is only negative route evidence. It does not classify the backend module's
maturity or decide that it needs a WebUI. Connector-only, capability-only, or
backend-only modules may remain intentionally headless.
Tenancy does provide composed administration surfaces despite having no direct
route. This section is only negative package evidence; connector-only,
capability-only, runtime-only, and backend-only modules may intentionally remain
headless. A new WebUI should be created only for a concrete user task, not to
make every module symmetrical.
## Rollout Matrix
| Order | Scope | Current evidence | Target | Owner / issue | Verification gate | Status |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | Product grammar and route inventory | Doctrine, ledger, layout rules, module contract, current route sources | One reconciled pattern language and evidence inventory | Meta [#11](https://git.add-ideas.de/GovOPlaN/govoplan/issues/11) | Docs links/diff checks; issue/wiki sync after integration | Initial slice in this document |
| 0 | Product grammar and route inventory | Doctrine, ledger, layout rules, module contract, current route sources | One reconciled pattern language and evidence inventory | Meta [#11](https://git.add-ideas.de/GovOPlaN/govoplan/issues/11) | Reviewed route/component inventory, module documents, manifest shapes and focused contracts | Complete 2026-08-03 |
| 1 | Campaign baseline integration | Recipient-editor WIP and tracker state have been reconciled with remote `main` | Integrated, testable baseline before migration claims | Campaign #67 and tracker cleanup | Backend and focused WebUI suites; issue evidence | Complete 2026-07-22 |
| 2 | Campaign previews/details | Stable shared dialog with bounded scrolling and fixed responsive preview workspace | Stable header/body/footer, accessible long-content detail | Campaign #59 and #73 | Review-preview and overlay structure tests | Complete 2026-07-22 |
| 3 | Campaign review/interventions | Five domain-owned stages with unresolved intervention language | Clear stages, outcomes, blockers, next actor/action, reviewed evidence | Campaign #63 | State matrix behavior/accessibility tests and agreed vocabulary | P1 needs product wording decision |
| 3 | Campaign review/interventions | Five domain-owned stages use central blocker and guided-review primitives; validation/build warnings name action, actor, and destination; hard blockers, individual review, and group review remain distinct; reviewed/remaining counts survive reload through build-bound review evidence | Clear stages, outcomes, blockers, next actor/action, reviewed evidence | Campaign #63 | `reviewProgress` state tests, shared-component structure contract, TypeScript build, configured-system help topic, and Campaign documentation tests | Complete 2026-08-03 (`d635f3a`; Core primitives and contextual help `b823a22`) |
| 4 | Campaign send/progress | A hard deployment ceiling bounds synchronous delivery; the selected synchronous, worker-queue, or database-queue mode is explicit and persisted; progress and recovery survive navigation; immediate-send response and audit evidence are allowlisted | Pre-send mode/consequence plus durable leave/return progress, retry and reconciliation without recipient/provider leakage | Campaign #62 and #79 | Boundary/concurrency/preflight, async selection, persisted mode, sanitized response/audit, partial/failure/retry and reload/return tests | Complete 2026-07-22 (`7e16603`, `60efd1c`, `62a6879`, `b0282eb`, `f095a3e`) |
| 5 | Campaign report filtering | Core DataGrid distinguishes client/full-result from server-owned queries; Campaign applies filter/sort/count before pagination and synchronizes count shortcuts with the grid query | One shared server-owned status/list/filter/count model | Campaign #65 and Core #263 | DataGrid contract/build tests plus exact shortcut/query/filter/count and large-result behavior | Complete 2026-07-22 (`e6062fe`, `cece71d`, `aa4ec66`, `4eb651c`) |
| 6 | Campaign operator recovery | A durable campaign/version queue page exposes historical work, exact non-overlapping state counts, persisted mode, permission-safe controls, server-paged job evidence, bounded refresh and active-state recovery | Fixed-position actions, disabled explanations, leave/return state, version-scoped retry/queue/reconcile and explicit campaign-wide pause/resume/cancel | Campaign #78 | Queue model/structure, historical-version, permission, paging, recovery-control, stale-response and delta tests | Complete 2026-07-22 (`21f3014`, `99d44ee`, `735e874`) |
| 7 | Campaign aggregate reports | A separate aggregate-reader projection and UI expose only policy-suppressed business totals with a stable status domain | Explicit denominator and exclusions, deployment floor plus tenant-strengthened small-cell threshold, complementary and overlapping-cell suppression, no detail/export/diagnostics | Campaign #80 | Aggregate query, cross-metric suppression, route/role/ACL, stable filter and UI structure tests | Complete 2026-07-22 (`06125cc`, `fc36aee`, `8ee87b7`, `ac3329c`, `1225802`) |
| 8 | Campaign excluded outcomes | Excluded build rows become explicit skipped transport outcomes and remain protected from queue/cancel/retry ambiguity | One durable source-to-job-to-report meaning with guarded historical normalization | Campaign #66 | Builder/persistence, migration, query/count, queue-control and report-explanation tests | Complete 2026-07-22 (`7229fb8`) |
| 9 | Guided first campaign | Existing wizard routes and ordinary workspace overlap | Task-oriented entry that hands off clearly to normal editing/review | Campaign #35 | First-run flow, resume/back, validation, optional modules, no implicit send | P1 after core pilot patterns stabilize |
| 10 | Prove/extract generic primitives | Core already exports many primitives; Campaign composition still unreviewed | Extract only contracts with a second consumer or clear platform ownership | Core #225 plus bounded follow-ups | Core behavior/accessibility tests and module-permutation tests | After Campaign proof |
| 11 | Configured-system pattern help | Docs route and classification exist | Role/config-aware pattern and route/field/blocker help | Docs #15 | Topic grouping, audience filtering, stable links/anchors | P1 after initial pattern IDs stabilize |
| 12 | Admin/configuration family | Phase inventory and connector primitives exist in the ledger | Apply the pattern to files, mail, policy, retention, packages, modules, API keys, settings | Core #225 and module children | Per-surface state/accessibility/consequence evidence | Parallel where independent of Campaign shared decisions |
| 13 | Remaining direct routes | Routes are contributed; most are unreviewed | Per-module bounded audit and migration plan | New module issues derived from this inventory | Applicable definition-of-done gates | P2 after Campaign, not a bulk rewrite |
| 14 | Manifest/runtime alignment | Several executable routes are absent from manifest metadata | Declared alignment or explicit validated exception | Core contract issue to create | Automated manifest/module route check and configured Docs verification | Discovery follow-up |
| 9 | Guided first campaign | Eight-stage creation flow persists current step/draft and hands off to ordinary review/delivery preparation | Task-oriented entry that hands off clearly to normal editing/review | Campaign #35 | First-run flow, resume/back, partial validation, immutable-history and optional-module behavior, no implicit send | Complete 2026-07-30 |
| 10 | Prove/extract generic primitives | Shared consequence, focus, help, blocker, unsaved-change, confirmation, connection-tree and effective-policy contracts now have Core and multiple module consumers | Keep Core behavior-only and leave domain composition in owning modules | Core #225 plus bounded follow-ups | Core behavior/accessibility tests and module-permutation tests | Complete 2026-08-03 (`fa32cca`; Files `d8ae506`; Mail `7844d9c`) |
| 11 | Configured-system pattern help | Role/config-aware workflow, reference, pattern, and system topics are projected by Docs; shared route, field, blocker, and action links resolve to configured Docs or the hosted fallback | Stable configured-system guidance without feature-to-Docs imports | Docs #15 | Docs suite, shared component tests, Campaign review tests, 46 module permutations, full-product bundle budget | Complete 2026-08-03 (Docs `abe2f78`; Core `b823a22`; Campaign `d635f3a`) |
| 12 | Admin/configuration family | Core host/settings/credential/retention contracts, shared primitives, module lifecycle, Files, Mail, Policy, Access, Admin, Tenancy, Views, and Organizations are integrated and verified | Continue the same consequence/provenance grammar only through bounded module-owned migrations | Core #225 and module children | Per-surface state/accessibility/consequence evidence | Core #225 complete `fa32cca`; Access `1409dbf`; Files `d8ae506`; Mail `7844d9c`; Policy `f964ed7`; Admin `d428f33`; Tenancy `e76fe16`; Views `c125f33`; Organizations `97acfcb` |
| 13 | Remaining module surfaces | 33 bounded module-owned issues cover every WebUI contributor not already tracked by Campaign #74 or completed Docs #15 | Per-module audit and migration, ordered by user task and consequence rather than a bulk rewrite | Issues linked in the direct-route and composed-surface sections | Module-focused tests, manifest shapes, contextual Docs, and applicable definition-of-done gates | Complete: prior 28 recorded commits plus Workflow #15, Search #4, Reporting #8, Projects #2 and Portal #2 verified 2026-08-03 |
| 14 | Manifest/runtime alignment | Authenticated canonical routes align; public signed-token and compatibility routes are explicit exceptions | Stable declarations reconcile with source and any effective runtime module combination | [Meta #25](https://git.add-ideas.de/GovOPlaN/govoplan/issues/25) | Strict duplicate/stale/undeclared declaration CI, per-module digests, and authorized read-only runtime inventory | Complete 2026-08-04 |
Workflow remains outside this rollout matrix because it has its own runtime and
editor workstream, not because it is postponed. Focused views can be specified,
+184
View File
@@ -0,0 +1,184 @@
# Package Registry Releases
GovOPlaN publishes reusable module artifacts through Gitea's native PyPI and
npm registries. These packages improve developer installation, release
resolution, cacheability, and artifact inspection. They do not replace the
signed runtime distribution: the signed manifest and digest-pinned OCI images
remain the production deployment authority.
## Publication boundary
Every repository with a `pyproject.toml` contains
`.gitea/workflows/module-package-release.yml`. The meta repository owns the
canonical template and installs it with:
```bash
python tools/repo/sync-module-package-workflows.py --write
python tools/repo/sync-module-package-workflows.py --check
```
The workflow runs for `v*` tags and may be dispatched manually for an existing
tag. The organization preflight verifies that every package repository protects
the `v*` namespace. Before building, the workflow itself verifies that:
- the tagged commit is contained in `main`;
- the tag, Python project version, and optional WebUI package version agree;
- package names remain in the `govoplan-*` and `@govoplan/*-webui` namespaces.
The workflow binds the repository explicitly from the Gitea Actions context.
Do not rely on GitHub-compatible environment variables being injected by the
runner image; Gitea runners may expose only the context values. Gitea 1.24 job
tokens cannot read repository tag-protection settings, so package jobs must not
receive a broad administrator token merely to repeat the organization preflight.
Run the following before the first publication and after repository or tag-rule
changes:
```bash
python tools/gitea/gitea-configure-package-releases.py
```
Preview and dispatch the exact wheel/WebUI versions selected by the developer
meta-package with:
```bash
python tools/gitea/gitea-dispatch-package-set.py \
--env-file ~/.config/gitea/gitea.env
python tools/gitea/gitea-dispatch-package-set.py \
--env-file ~/.config/gitea/gitea.env \
--apply
```
The dispatcher reads exact versions from `packages/govoplan-meta/pyproject.toml`,
inspects the selected tag to determine whether a WebUI package is expected,
skips complete registry pairs and does not duplicate an active workflow. Use
`--repository govoplan-core` for a bounded dispatch or `--verify-existing` to
rebuild and hash-verify versions already present in both registries.
For coordinated lockstep tags, `push-release-tag.sh` pushes module tags first,
Core next, and the meta tag last. This is a dependency guarantee for a
single-capacity Actions runner: the developer package cannot run before its
exact Core and module versions have entered the queue.
The same release entry point first validates the migration graph, then records
the reviewed current Alembic heads under the target release version and reruns
the strict migration audit before it changes package versions, commits, or
tags. The default preflight intentionally does not require those heads to exist
in the previous release baseline. A failed candidate-baseline check therefore
cannot produce a protected package release.
The source gate validates `pyproject.toml`, the module version declaration
(`MODULE_VERSION` or the top-level `ModuleManifest.version`), public package
`__version__`, and WebUI metadata before creating tags. Release-tag artifact
checks run only after the candidate tags and immutable WebUI lock have been
created locally.
Release-lock regeneration resolves a fresh immutable lock from the reviewed
candidate manifests; it does not seed resolution from the previous release
lock. This prevents removed transitive packages and stale peer metadata from
blocking or contaminating the new release. Candidate resolution also uses an
isolated temporary npm cache, so a locally replaced tag cannot reuse metadata
from a failed, unpushed release attempt.
Modules that retain the same WebUI package identity in both a root publish
manifest and `webui/package.json` use the WebUI manifest as the canonical peer
contract. The coordinated release synchronizes `peerDependencies` and
`peerDependenciesMeta` into the publish manifest before creating the module
tag, then synchronizes each lockfile root from the final package metadata. A
distinct root package remains independent.
It builds one wheel and, where applicable, one npm tarball. The workflow records
the source tag, source commit, filename, size, and SHA-256 in
`package-artifacts.json` before publishing. Gitea rejects a second upload of the
same package version, so correction requires a new version rather than artifact
replacement.
A retry after partial publication is safe. Before upload, the workflow reads the
native package registry file record and compares its SHA-256 with the artifact
rebuilt from the protected tag. An exact existing artifact is skipped; a
same-version artifact with another digest or an unexpected file set fails
closed. This permits a failed npm publication to resume without weakening
package immutability or accepting `--skip-existing` blindly.
The npm tarball is always published through an explicit local `./dist/...`
path. Without that prefix, npm may interpret a relative tarball name as a Git
package shorthand before it ever contacts the configured registry.
Published WebUI packages contain registry-compatible dependencies only. The
workflow converts an internal dependency pinned to a protected `vX.Y.Z` Git tag
into the exact `X.Y.Z` registry version and rejects unresolved `file:` or Git
dependencies. Repository development metadata may therefore keep local or Git
references without leaking them into the published package contract.
Historical `add-ideas` and current `GovOPlaN` organization URLs are accepted
for immutable tagged releases; both normalize to the same exact registry
dependency and no branch or unversioned Git reference is accepted.
## One-time Gitea setup
Protect `v*` tags in every package repository and the meta repository. Allow
only the `Owners` team to create or delete those tags.
```bash
set -a
. ~/.config/gitea/gitea.env
set +a
python tools/gitea/gitea-configure-package-releases.py --apply
```
Create a dedicated personal access token with only `write:package` scope and
store these organization-level Actions secrets on `GovOPlaN`:
- `GOVOPLAN_PACKAGE_USERNAME`: account owning the package token;
- `GOVOPLAN_PACKAGE_TOKEN`: dedicated package-write token.
Do not use an administrator or general release token. Gitea 1.24 does not grant
package publication to the automatic Actions job token. Organization secrets
allow the same least-privilege credential to serve every module workflow.
## Exact release consumption
`tools/release/generate-release-package-set.py` translates the reviewed Git
source refs in `requirements-release.txt` into an exact registry package set.
It resolves each version tag to its commit and verifies the package metadata in
that tag.
`tools/release/resolve-package-artifacts.py` then downloads exactly those wheel
and WebUI versions from Gitea. It reads the identity embedded in every wheel and
npm tarball, rejects missing, duplicate, unexpected, or oversized artifacts,
and writes `package-artifacts.lock.json` with SHA-256 values and npm integrity
values. Credentials are accepted only through environment variables and are
never written to the lock. Python resolution ignores ambient pip configuration
and extra indexes for GovOPlaN roots, preventing an internal package name from
being selected from an undeclared registry.
The runtime distribution workflow uses the verified wheelhouse directly and
installs module WebUI tarballs only after matching them to the lock. It publishes
the package set, package lock, and hash-locked requirements as release assets.
The package-lock SHA-256 is part of the signed distribution manifest. Runtime
finalization also requires the lock's package versions and hashes to match the
wheel composition embedded in the images. OCI assembly remains network-free
after package and third-party dependency resolution.
The source refs remain in the module catalog for source provenance and release
planning. Production installation consumes the signed runtime images rather
than invoking `pip`, `npm`, or Git on the target host.
## Developer meta-package
`packages/govoplan-meta` builds the optional `govoplan` package. Its default
dependencies mirror the reviewed runtime roots; `govoplan[full]` adds all
currently packageable workspace modules. Regenerate it after changing release
requirements or package versions:
```bash
python tools/release/generate-developer-meta-package.py
python tools/release/generate-developer-meta-package.py --check
```
`push-release-tag.sh` performs this synchronization before release commits and
tags. The meta-package is for editable/developer setup and composition tests. It
does not enable modules, apply migrations, provision services, or establish
backup and recovery evidence.
Generic Packages are intentionally not used. Add that transport only when a
consumer needs an artifact format unsupported by PyPI, npm, Gitea Releases, or
the OCI registry.
+68 -15
View File
@@ -25,6 +25,7 @@ releases, module boundaries, migrations, and security controls.
| Labels and translations | Generated translation catalogs plus source usage |
| Fields and help coverage | Shared form components plus generated TypeScript AST inventory |
| API use by the WebUI | Typed API clients plus generated static reference inventory |
| Stable platform interface IDs | Typed manifest/WebUI declarations plus line-independent source anchors for low-level controls |
| Effective configuration | Owning module data plus Policy provenance |
Runtime introspection is authoritative for an installed system. Static source
@@ -45,9 +46,13 @@ The command writes:
- `audit-reports/platform-inventory/platform-interface-inventory.json`
- `audit-reports/platform-inventory/platform-interface-inventory.md`
Use `--strict` in CI. In addition to translation coverage, strict mode requires
every backend endpoint without a statically visible WebUI path to have an exact
entry in
Use `--strict` for the combined translation, endpoint, and declaration audit.
Use `--strict-declarations` for duplicate/stale/undeclared interface checks
without making existing translation coverage a release blocker. Use
`--strict-endpoints` in the endpoint-surface CI gate so unrelated translation
catalog work cannot disable route classification enforcement. Both strict modes
require every backend endpoint without a statically visible WebUI path to have
an exact entry in
`tools/inventory/endpoint-surface-declarations.json`. The registry is keyed by
repository, HTTP method, and canonical version-independent path. It accepts:
@@ -73,6 +78,16 @@ It combines:
2. TypeScript AST extraction of fields, label attributes, visible text,
translations, frontend routes, navigation, capabilities, and API references
3. Python AST extraction of FastAPI route decorators and router prefixes
4. normalized runtime declarations from every loaded `ModuleManifest`
The declaration set covers routes, navigation, View surfaces, fields, actions,
help references, translations, admin/settings sections, widgets, search
objects, permissions, provided interfaces, and backend capabilities. Typed
module contributions keep their declared IDs. Shared controls may declare
`interfaceId` and `helpTopicId`; otherwise the extractor assigns a deterministic
source anchor based on repository, file, component context, control type, and
semantic label rather than a line number. The JSON records which identity
source was used.
The JSON includes exact repository, file, and line evidence. A missing-help
entry is a review candidate because dynamic parent components may supply help.
@@ -80,9 +95,40 @@ A backend route without a static frontend reference is also a review candidate:
public APIs, workers, callbacks, health checks, connectors, and dynamic URL
assembly are valid explanations.
`--strict` currently enforces only translation-catalog completeness. Endpoint
and help classifications need narrow reviewed baselines before they can become
release gates.
The module matrix enforces endpoint and interface declarations with
`--strict-endpoints --strict-declarations`.
Combined `--strict` additionally fails when used translation keys are absent
from generated locale catalogs. Help-text findings remain review candidates
rather than a release gate because dynamic parent components can supply help.
## Runtime Comparison
Core exposes a sanitized read-only catalog at
`GET /api/v1/platform/interface-catalog`. Access requires
`admin:module:read` or `system:settings:read`. Tenant module entitlements are
applied before serialization, so the response describes only the effective
installed combination. It contains IDs, paths, authorization metadata,
versions, counts, and canonical digests; it excludes factories, callbacks,
credentials, and mutable runtime state.
Capture and compare a running installation:
```bash
curl --fail --silent \
-H "Authorization: Bearer $GOVOPLAN_ACCESS_TOKEN" \
"$GOVOPLAN_URL/api/v1/platform/interface-catalog" \
> /tmp/govoplan-runtime-interface.json
./.venv/bin/python tools/inventory/platform-interface-inventory.py \
--runtime-snapshot /tmp/govoplan-runtime-interface.json \
--strict-declarations \
--strict-endpoints
```
The comparison accepts any installed subset. Every module present in the
runtime response must have the same contract version, module version, and
declaration digest as the static release inventory. Unknown, duplicate, or
mismatched runtime modules fail strict declaration mode.
## Admin Information Architecture
@@ -136,13 +182,20 @@ Custom code, new routes, arbitrary SQL, and executable workflow nodes remain
release artifacts. Modeling them as ordinary configuration would create an
unreviewed code-execution and migration channel.
## Next Enforcement Slices
## Enforced Contract
1. Require every WebUI module route and admin/settings contribution to have
matching manifest metadata or a reviewed exception.
2. Add stable field IDs and optional help-topic IDs to shared field components.
3. Classify each statically unreferenced backend endpoint by consumer type.
4. Compare a running installation's OpenAPI and module registry against the
release inventory.
5. Publish the sanitized installed-system structure through Ops/Docs for
authorized administrators.
1. Public WebUI routes and View surfaces must reconcile with runtime manifest
metadata; stale runtime routes and source-only public surfaces fail CI.
2. Duplicate stable IDs fail CI. Shared controls support explicit field/action
and help-topic identities; fallback anchors remain visible review evidence.
3. Every statically unreferenced backend endpoint has an exact reviewed
consumer classification, and stale classifications fail CI.
4. Runtime module combinations can be compared exactly with static release
evidence through versioned per-module digests.
5. Runtime introspection is authorized, tenant-filtered, and read-only. It is
safe for Ops/Docs projection but is not a generic configuration or code
mutation channel.
Generated JSON and Markdown remain build/audit artifacts. Do not hand-edit or
use them as a backlog; change the owning manifest, typed WebUI contribution,
translation/help declaration, or exact endpoint classification instead.
+182
View File
@@ -0,0 +1,182 @@
# Production Target And Independent Evidence Handoff
This runbook identifies the external inputs needed to finish
[GovOPlaN #27](https://git.add-ideas.de/GovOPlaN/govoplan/issues/27) and
[GovOPlaN #37](https://git.add-ideas.de/GovOPlaN/govoplan/issues/37). The
repository can render, inspect and sign evidence for a target, but it cannot
manufacture an independent failure domain or an independent approval authority.
## GovOPlaN #27: real two-node target
The bounded acceptance target is two independently schedulable worker nodes.
The API and WebUI must each have ready replicas on both nodes, all Deployments
must be available, the active module composition and software versions must be
consistent, every configured queue must have a worker, and the database
connection budget must pass. The validation then deletes one ready API pod and
requires replacement without an observed readiness outage.
Two virtual machines on different physical hosts or availability zones meet the
failure-domain intent. Two containers, VMs or Kubernetes nodes on one physical
host are useful development targets but do not close #27. A two-worker cluster
also does not prove control-plane high availability. For a self-managed
production cluster, use three control-plane nodes plus at least two workers; a
managed control plane plus two workers is the shorter path.
### What the target owner must provide
Provide these through a secure handoff, not an issue, chat message or Git:
1. A kubeconfig path with access to the target, for example
`~/.config/govoplan/targets/<target>.kubeconfig`, mode `0600`.
2. A stable installation ID, public HTTPS hostname, namespace, ingress class and
TLS-secret or certificate-manager arrangement.
3. Two independently schedulable workers and permission to place API and WebUI
replicas on both.
4. External, logically shared PostgreSQL, Redis and S3 endpoints with trusted
CA material and network reachability from every worker. Do not co-locate the
only copies of these services on the two workers used for the failure drill.
5. The six runtime secret values required by the generated manifest:
`MASTER_KEY_B64`, `DATABASE_URL`, `GOVOPLAN_DATABASE_URL_PGTOOLS`,
`REDIS_URL`, `FILE_STORAGE_S3_ACCESS_KEY_ID` and
`FILE_STORAGE_S3_SECRET_ACCESS_KEY`.
6. A short-lived GovOPlaN API key limited to `ops:operations:read`, supplied in
`GOVOPLAN_OPS_API_KEY` only for evidence collection.
7. An approved drill window and permission to delete one API pod.
If no Kubernetes target exists, provide hostnames/IP addresses for the machines,
an SSH user and key path, the internal/external DNS plan, and the permitted
firewall ports. Those inputs are sufficient to provision a k3s target. They are
not sufficient to claim control-plane HA unless three control-plane failure
domains are present.
### Separate deployment and evidence authorities
The deployment identity may create and update the namespace, Secret,
ConfigMap, Deployments, Services, Jobs, PodDisruptionBudgets and Ingress. The
evidence collector only needs:
- cluster scope: `get` and `list` for `nodes`;
- target namespace: `get` and `list` for `pods` and `deployments`;
- target namespace during the approved drill: `delete` for `pods`.
Use separate kubeconfig contexts or service accounts when the same person does
not hold both roles.
### Render, apply and verify
Use the signed, digest-pinned installation bundle selected for the target:
```bash
export KUBECONFIG="$HOME/.config/govoplan/targets/<target>.kubeconfig"
python tools/deployment/govoplan-deploy.py render-kubernetes \
--directory /srv/govoplan/<installation-id> \
--namespace govoplan \
--secret-name govoplan-runtime \
--tls-secret-name govoplan-tls \
--ingress-class-name nginx \
--output /srv/govoplan/<installation-id>/kubernetes.json
kubectl apply -f /srv/govoplan/<installation-id>/kubernetes.json
kubectl -n govoplan wait --for=condition=available deployment --all --timeout=10m
export GOVOPLAN_OPS_API_KEY="$(cat /run/secrets/govoplan-ops-evidence-key)"
python tools/deployment/govoplan-deploy.py verify-kubernetes \
--directory /srv/govoplan/<installation-id> \
--namespace govoplan \
--exercise-api-pod-loss \
--output /srv/govoplan/<installation-id>/evidence/kubernetes-multi-host.json
unset GOVOPLAN_OPS_API_KEY
```
The verifier emits sanitized JSON and exits nonzero if the topology, runtime,
queue, connection-budget or pod-loss checks fail. Preserve the private cluster
logs and manifest alongside the sanitized result in the controlled evidence
store.
## GovOPlaN #37: controlled signed target evidence
Yes, collection, review and signing can run in containers. A container provides
repeatability and process isolation; it does not create independent authority.
The production approver must control a different private key from the target
operator/assessor and must review the evidence before signing the
`production_approval` scope.
Use at least these three key boundaries:
1. **Installer authority:** signs installed-release-origin receipts only.
2. **Target assessment authority:** signs the permitted target, accessibility,
privacy, security, operations and recovery scopes.
3. **Production approval authority:** independently signs only
`production_approval` after reviewing the other evidence.
Do not reuse release-catalog keys for any of these roles. Keep private Ed25519
keys outside Git, Gitea, GovOPlaN application storage and chat. Publish only the
public keyrings. The proof issuer already rejects key reuse across release,
installer and proof trust domains.
### Generate independently held keys
Each authority runs this command in its own `0700` directory. The generator
refuses existing output paths and writes both files as `0600`:
```bash
install -d -m 0700 "$HOME/.config/govoplan/authority-keys"
python tools/assessments/generate-authority-keypair.py \
--purpose proof \
--key-id authority:target-2026 \
--scope target_environment \
--scope accessibility \
--scope privacy \
--scope security \
--scope operations \
--scope recovery \
--private-key "$HOME/.config/govoplan/authority-keys/target-2026.pem" \
--keyring "$HOME/.config/govoplan/authority-keys/target-2026-public.json"
```
The independent production approver generates another key with only
`--scope production_approval`. An installer authority uses `--purpose installer`
and no `--scope`. Merge public key entries into the separately controlled
keyrings only after the responsible authorities verify fingerprints out of
band.
### Container boundary
Use two one-shot jobs or containers:
- **Collector/assessor:** network access, read-only source and trust mounts,
read/write private evidence output, and the narrowly scoped kubeconfig. It
must not receive the production-approval private key.
- **Production approver:** `--network none`, read-only assessment/evidence/trust
mounts, a read-only secret mount containing only the approval key, and a
separate output mount. It must not receive deployment credentials.
Build or select the assessment image by digest and record that digest in the
evidence log. A representative runtime shape is:
```bash
docker run --rm --network none --read-only --tmpfs /tmp \
--user "$(id -u):$(id -g)" \
--mount type=bind,src="$PWD/evidence",dst=/evidence,readonly \
--mount type=bind,src="$PWD/trust",dst=/trust,readonly \
--mount type=bind,src="$HOME/.config/govoplan/authority-keys",dst=/run/keys,readonly \
--mount type=bind,src="$PWD/approved",dst=/output \
<assessment-image>@sha256:<digest> \
<assessment command>
```
The current evidence commands and required scopes are documented in
[`TARGET_MATURITY_EVIDENCE_RUNBOOK.md`](TARGET_MATURITY_EVIDENCE_RUNBOOK.md).
The final proof must cover `target_environment`, `accessibility`, `privacy`,
`security`, `operations`, `recovery` and independent `production_approval`, and
must bind to the verified installed composition and installer receipt.
## Completion boundary
#27 can close after the real target produces a passing pod-loss result. #37 can
close after an independently approved, schema-valid proof is generated for that
same installed composition and the public authority keyrings, proof and private
evidence custody references are recorded. Neither issue should close from a
single-host simulation or a self-approved signature.
+13 -5
View File
@@ -44,6 +44,10 @@ least one check. The ledger verifies its hash chain before evidence is trusted.
This is a platform contract, not an assertion that every existing module
operation has adopted it. Module operations with external or multi-resource
effects must be migrated to the ledger before claiming these guarantees.
The owning-module inventory and adoption state are maintained in
[Recovery Ledger Adoption](RECOVERY_LEDGER_ADOPTION.md); CI validates the
machine-readable inventory so newly identified boundaries cannot disappear from
the backlog silently.
## Deployment Journal
@@ -98,11 +102,15 @@ old code may not understand the new schema. Recovery then means one of:
3. restore a separately verified, coordinated database/object/key backup and
then deploy the matching release.
The deployment tool does not create or validate that database backup. A
`backup-required` annotation on the Kubernetes migration Job is an operator
gate, not backup evidence. Production automation must provide a backup hook or
external backup controller whose artifact, timestamp, scope, encryption key,
and restore test can be referenced from the recovery record.
The deployment tool does not create that backup. It does verify an externally
produced, signed evidence contract covering PostgreSQL, objects, protected
configuration, and key custody at one recovery point plus an isolated restore
drill. A self-hosted release change cannot reach the migration command or be
exported as a Kubernetes migration Job until fresh evidence bound to the
previous immutable release has been adopted. Compose verifies it again after
runtime quiescing. See
[Backup And Restore Evidence](BACKUP_AND_RESTORE_EVIDENCE.md) for the contract,
provider runbooks, RPO/RTO ownership, retention, and disposal rules.
## Scaled Nodes
+89
View File
@@ -0,0 +1,89 @@
# Recovery Ledger Adoption
The Core recovery ledger is a platform primitive, not automatic protection for
module-owned effects. The canonical, machine-checked inventory is
[`recovery-operation-inventory.json`](recovery-operation-inventory.json).
## Classification Rules
- Use `atomic` only when every mutation commits in one database transaction and
no external effect occurs.
- Use `compensation` when every completed effect has a bounded, verifiable
inverse action. A best-effort delete is not proof of compensation.
- Use `snapshot_restore` only with fresh, signed backup evidence that covers all
affected state services at one recovery point.
- Use `forward_recovery` for provider acceptance, queue publication, cursor
advancement, and other effects that may be resumable but cannot safely be
undone.
- Use `irreversible` for approved purge or destruction where no automated
recovery is claimed.
One feature may cross more than one boundary. Module installation is
compensatable before schema migration, forward-only after migration starts, and
snapshot-restorable for an approved destructive retirement. Mail submission is
forward recovery because losing the response after provider acceptance must not
cause an automatic resend.
## Adoption Order
1. Campaign build is the reference implementation for a database plus object
storage operation. Its operation reserves a build-specific object prefix,
persists request and precondition evidence before writes, records the final
object manifest, and verifies database/object state before success.
2. Campaign delivery and Mail provider effects adopt outcome-unknown semantics
without weakening their existing provider-specific idempotency records.
3. Files applies the same contract to uploads, purge, integrity reconciliation,
and writable connector synchronization.
4. Connectors, Dataflow, and Workflow Engine consume the contract at their
registry/capability boundaries so optional providers remain optional.
5. Core module lifecycle uses the ledger in addition to, not instead of, signed
deployment and backup evidence.
Every fenced operation uses a process incarnation and distributed lease. A
stale process cannot append a checkpoint or report success. An expired operation
is claimed for recovery through an explicit takeover that preserves the prior
fence in the checkpoint chain; it is never resumed as a normal retry.
Connectors read-only sanctions and feed acquisitions are adopted: source
revision/cursor and dry-run evidence are recorded before provider I/O, while
the immutable snapshot and terminal checkpoint commit atomically. The generic
external-mutation contract is conformance-tested but remains `planned` until a
production connector actually publishes, updates, or deletes provider state.
Dataflow runs are adopted. Database-only execution uses one atomic terminal
commit for the run projection and recovery checkpoint. Output publication uses
forward recovery: source and output digests are checkpointed before dispatch,
a conclusive provider result commits with the run projection, and an expired
or failed attempt after dispatch becomes `outcome_unknown`. A stale attempt may
be retried only when its durable boundary proves dispatch had not started.
Workflow Engine is adopted at both declared boundaries. Instance workers,
trigger deliveries, and timer resumptions use process-bound distributed fences.
Every module-action invocation records the pinned definition, input, preview,
authority, provider-idempotency, and action-contract hashes before dispatch.
Conclusive results commit with the Workflow projection. A lost acknowledgement,
invalid result, or unannounced non-atomic effect becomes `outcome_unknown` and
cannot be retried until evidence confirms either that the effect occurred or is
absent. Linked Dataflow uncertainty blocks the Workflow without duplicating
Dataflow's recovery authority.
Core module lifecycle is adopted at four boundaries. Installer recovery is
prepared before snapshots so a full database restore preserves the attempted
operation. Pre-migration package changes use compensation, migrated changes use
forward recovery, destructive retirement requires a hashed and restore-checked
snapshot, and live graph changes restore the prior registry when no migration
ran. A deployment-wide database fence serializes these effects; any unresolved
predecessor blocks a differently keyed retry until explicit reconciliation.
Supervised installs become successful only after restart and health evidence is
recorded.
## Operator Contract
Ops lists non-terminal and manual-intervention operations. Operators must verify
the checkpoint chain before trusting evidence, distinguish `outcome_unknown`
from rejection, and use the owning module's documented reconciliation action.
No evidence payload may contain credentials or resolved secrets.
The parent adoption issue remains open until all inventory rows are adopted and
the module matrix proves crash, retry, stale-fence, tamper, and optional-module
behavior for each consequential path.
+7
View File
@@ -76,6 +76,13 @@ one place. If a deployment profile later needs pinned SHAs for every repository,
generate that lock as a release artifact instead of making day-to-day
development depend on submodule updates.
Module release tags also publish wheels and WebUI tarballs to the organization
PyPI/npm registries. The meta release resolves exact versions into a hash-bound
package lock before producing the signed OCI runtime. See
`docs/PACKAGE_REGISTRY_RELEASES.md`. Git tags remain source provenance; package
registries are reusable artifact transport; the signed runtime manifest and
digest-pinned images remain production authority.
## Docker Placement
Whole-product Docker and production-like deployment composition belongs in
+8 -2
View File
@@ -1,5 +1,9 @@
# Scaling And Multi-Host Deployment
For the exact external handoff, least-privilege collector permissions and live
two-node acceptance procedure, see
[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
## Implemented Contract
GovOPlaN now supports a stateless application tier backed by logically shared
@@ -190,14 +194,16 @@ access, node visibility, drain controls, migration serialization, and scheduler
fencing. It does not by itself provide:
- a highly available PostgreSQL, Redis, or object-store deployment;
- automatic PostgreSQL backup, point-in-time recovery, or restore verification;
- automatic PostgreSQL/object backup creation or point-in-time recovery;
- autoscaling policy;
- central logs, metrics, traces, or alert routing;
- certificate portability between independently managed ingress providers;
- automatic reconciliation of every possible module side effect;
- a service-level availability guarantee.
Those are deployment and module-adoption requirements. Before claiming high
The deployer verifies and gates migrations on signed coordinated backup and
isolated-restore evidence, but backup capture and restoration remain owned by
the selected state-service providers. Before claiming high
availability, drill replica loss, rolling replacement, session continuity, job
redelivery, scheduler failover, migration exclusion, object-store outage, and a
coordinated database/object/key restore. Recovery rules and evidence are
@@ -141,12 +141,16 @@ The canonical backlog item is
Implementation status as of the current source tree:
- Slice 1 now has the source-controlled production artifact boundary: offline
per-architecture wheel resolution, non-root API/Web image definitions,
multi-architecture OCI publication, signed composition/SBOM/provenance,
immutable Gitea assets, a signed one-file deployer, and fail-closed manifest
adoption. The first real published release and cross-architecture runtime
evidence remain release-operator work rather than source-code claims.
- Slice 1 has a published production-artifact baseline. Immutable
[`v0.1.14`](https://git.add-ideas.de/GovOPlaN/govoplan/releases/tag/v0.1.14)
binds source commit `1f039dd39c1ce2672f4978c8abc6dff862ef1445`, a signed
one-file deployer, exact API/Web and managed-dependency image digests,
composition, SBOMs, and provenance. Runtime Distribution
[run #459](https://git.add-ideas.de/GovOPlaN/govoplan/actions/runs/459)
passed migrations, schema checks, non-root API/Web readiness, and worker
delivery/shutdown on both amd64 and arm64. Each future release must renew the
evidence, and a real installation must still produce topology-specific
ingress, failover, backup, and recovery receipts.
- Slice 6 has a working application-tier foundation: state profiles, shared
object storage, runtime node registration/heartbeats/drain, fenced scheduler,
migration serialization, exact-head startup waiting, Ops visibility, and a
+4
View File
@@ -1,5 +1,9 @@
# Target Maturity Evidence Runbook
For authority-key generation, container isolation and the concrete inputs that
must be supplied by the target owner and independent production approver, see
[`PRODUCTION_TARGET_HANDOFF.md`](PRODUCTION_TARGET_HANDOFF.md).
This runbook turns retained target-environment results into a sanitized,
signed GovOPlaN capability-fit proof. It does not make a deployment suitable,
certified, supported, or production-approved by itself. The proof records what
+37
View File
@@ -0,0 +1,37 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-keyring-v1.json",
"title": "GovOPlaN backup evidence trust keyring",
"type": "object",
"additionalProperties": false,
"required": ["schema_version", "purpose", "keys"],
"properties": {
"schema_version": { "const": "1" },
"purpose": { "const": "govoplan-backup-evidence" },
"keys": {
"type": "array",
"minItems": 1,
"maxItems": 64,
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"key_id",
"algorithm",
"status",
"public_key_pem",
"not_before",
"expires_at"
],
"properties": {
"key_id": { "type": "string", "minLength": 1, "maxLength": 128 },
"algorithm": { "const": "ed25519" },
"status": { "enum": ["active", "retired", "revoked"] },
"public_key_pem": { "type": "string", "minLength": 1, "maxLength": 8192 },
"not_before": { "type": "string", "format": "date-time" },
"expires_at": { "type": "string", "format": "date-time" }
}
}
}
}
}
+255
View File
@@ -0,0 +1,255 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://govoplan.add-ideas.de/schemas/backup-evidence-v1.json",
"title": "GovOPlaN coordinated backup and restore evidence",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"evidence_id",
"installation_id",
"deployment_subject",
"release",
"recovery_point",
"components",
"restore_drill",
"issued_at",
"expires_at",
"revoked",
"signatures"
],
"properties": {
"schema_version": { "const": "1" },
"evidence_id": { "$ref": "#/$defs/token" },
"installation_id": { "$ref": "#/$defs/token" },
"deployment_subject": {
"type": "object",
"additionalProperties": false,
"required": ["profile", "topology", "subject_ref"],
"properties": {
"profile": { "enum": ["evaluation", "self-hosted"] },
"topology": { "$ref": "#/$defs/token" },
"subject_ref": { "$ref": "#/$defs/reference" }
}
},
"release": {
"type": "object",
"additionalProperties": false,
"required": [
"channel",
"version",
"manifest_sha256",
"composition_sha256",
"api_image",
"web_image"
],
"properties": {
"channel": { "$ref": "#/$defs/token" },
"version": { "$ref": "#/$defs/token" },
"manifest_sha256": { "$ref": "#/$defs/sha256" },
"composition_sha256": { "$ref": "#/$defs/sha256" },
"api_image": { "$ref": "#/$defs/digest_image" },
"web_image": { "$ref": "#/$defs/digest_image" }
}
},
"recovery_point": {
"type": "object",
"additionalProperties": false,
"required": ["id", "captured_at", "consistency", "rpo_seconds", "write_fence"],
"properties": {
"id": { "$ref": "#/$defs/token" },
"captured_at": { "type": "string", "format": "date-time" },
"consistency": {
"enum": ["provider-atomic", "application-quiesced", "transaction-consistent"]
},
"rpo_seconds": { "$ref": "#/$defs/duration" },
"write_fence": {
"type": "object",
"additionalProperties": false,
"required": ["mode", "token_sha256", "established_at"],
"properties": {
"mode": {
"enum": ["provider-snapshot", "application-quiesce", "transaction-boundary"]
},
"token_sha256": { "$ref": "#/$defs/sha256" },
"established_at": { "type": "string", "format": "date-time" }
}
}
}
},
"components": {
"type": "object",
"additionalProperties": false,
"required": ["database", "objects", "configuration", "key_custody"],
"properties": {
"database": { "$ref": "#/$defs/database" },
"objects": { "$ref": "#/$defs/objects" },
"configuration": { "$ref": "#/$defs/configuration" },
"key_custody": { "$ref": "#/$defs/key_custody" }
}
},
"restore_drill": {
"type": "object",
"additionalProperties": false,
"required": [
"drill_id",
"recovery_point_id",
"started_at",
"completed_at",
"isolated_target_ref",
"release_manifest_sha256",
"migration_heads_sha256",
"representative_object_manifest_sha256",
"database_verified",
"objects_verified",
"configuration_verified",
"key_custody_verified",
"semantic_checks",
"measured_rpo_seconds",
"measured_rto_seconds",
"evidence_ref"
],
"properties": {
"drill_id": { "$ref": "#/$defs/token" },
"recovery_point_id": { "$ref": "#/$defs/token" },
"started_at": { "type": "string", "format": "date-time" },
"completed_at": { "type": "string", "format": "date-time" },
"isolated_target_ref": { "$ref": "#/$defs/reference" },
"release_manifest_sha256": { "$ref": "#/$defs/sha256" },
"migration_heads_sha256": { "$ref": "#/$defs/sha256" },
"representative_object_manifest_sha256": { "$ref": "#/$defs/sha256" },
"database_verified": { "const": true },
"objects_verified": { "const": true },
"configuration_verified": { "const": true },
"key_custody_verified": { "const": true },
"semantic_checks": {
"type": "array",
"minItems": 1,
"maxItems": 128,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["id", "status", "evidence_ref"],
"properties": {
"id": { "$ref": "#/$defs/token" },
"status": { "const": "passed" },
"evidence_ref": { "$ref": "#/$defs/reference" }
}
}
},
"measured_rpo_seconds": { "$ref": "#/$defs/duration" },
"measured_rto_seconds": { "$ref": "#/$defs/duration" },
"evidence_ref": { "$ref": "#/$defs/reference" }
}
},
"issued_at": { "type": "string", "format": "date-time" },
"expires_at": { "type": "string", "format": "date-time" },
"revoked": { "const": false },
"signatures": {
"type": "array",
"minItems": 1,
"maxItems": 16,
"items": { "$ref": "#/$defs/signature" }
}
},
"$defs": {
"token": {
"type": "string",
"minLength": 1,
"maxLength": 128,
"pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"
},
"sha256": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"digest_image": {
"type": "string",
"maxLength": 300,
"pattern": "^[^@\\s]+@sha256:[0-9a-f]{64}$"
},
"reference": {
"type": "string",
"minLength": 3,
"maxLength": 2048,
"pattern": "^[A-Za-z][A-Za-z0-9+.-]*:[^\\s]+$"
},
"duration": { "type": "integer", "minimum": 0, "maximum": 2592000 },
"protected_key": {
"type": "object",
"properties": {
"protected": { "const": true },
"encryption_key_ref": { "$ref": "#/$defs/reference" },
"captured_at": { "type": "string", "format": "date-time" }
}
},
"database": {
"type": "object",
"additionalProperties": false,
"required": [
"provider", "artifact_ref", "artifact_sha256", "snapshot_id", "lsn",
"protected", "encryption_key_ref", "captured_at"
],
"properties": {
"provider": { "$ref": "#/$defs/token" },
"artifact_ref": { "$ref": "#/$defs/reference" },
"artifact_sha256": { "$ref": "#/$defs/sha256" },
"snapshot_id": { "$ref": "#/$defs/token" },
"lsn": { "type": "string", "minLength": 1, "maxLength": 256 },
"protected": { "const": true },
"encryption_key_ref": { "$ref": "#/$defs/reference" },
"captured_at": { "type": "string", "format": "date-time" }
}
},
"objects": {
"type": "object",
"additionalProperties": false,
"required": [
"provider", "artifact_ref", "manifest_sha256", "version_id",
"object_count", "total_bytes", "protected", "encryption_key_ref", "captured_at"
],
"properties": {
"provider": { "$ref": "#/$defs/token" },
"artifact_ref": { "$ref": "#/$defs/reference" },
"manifest_sha256": { "$ref": "#/$defs/sha256" },
"version_id": { "$ref": "#/$defs/token" },
"object_count": { "type": "integer", "minimum": 0 },
"total_bytes": { "type": "integer", "minimum": 0 },
"protected": { "const": true },
"encryption_key_ref": { "$ref": "#/$defs/reference" },
"captured_at": { "type": "string", "format": "date-time" }
}
},
"configuration": {
"type": "object",
"additionalProperties": false,
"required": ["artifact_ref", "sha256", "protected", "encryption_key_ref", "captured_at"],
"properties": {
"artifact_ref": { "$ref": "#/$defs/reference" },
"sha256": { "$ref": "#/$defs/sha256" },
"protected": { "const": true },
"encryption_key_ref": { "$ref": "#/$defs/reference" },
"captured_at": { "type": "string", "format": "date-time" }
}
},
"key_custody": {
"type": "object",
"additionalProperties": false,
"required": ["provider", "keyset_ref", "keyset_version", "recoverable", "captured_at"],
"properties": {
"provider": { "$ref": "#/$defs/token" },
"keyset_ref": { "$ref": "#/$defs/reference" },
"keyset_version": { "$ref": "#/$defs/token" },
"recoverable": { "const": true },
"captured_at": { "type": "string", "format": "date-time" }
}
},
"signature": {
"type": "object",
"additionalProperties": false,
"required": ["key_id", "algorithm", "value"],
"properties": {
"key_id": { "$ref": "#/$defs/token" },
"algorithm": { "const": "ed25519" },
"value": { "type": "string", "minLength": 1, "maxLength": 256 }
}
}
}
}
+195
View File
@@ -0,0 +1,195 @@
{
"schema_version": 1,
"parent_issue": "https://git.add-ideas.de/GovOPlaN/govoplan/issues/36",
"operations": [
{
"id": "campaign.build.publish-artifacts",
"repository": "govoplan-campaign",
"resources": ["postgresql", "object-storage", "templates-capability", "files-capability"],
"mode": "compensation",
"fenced": true,
"adoption": "reference-implementation",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/92"
},
{
"id": "campaign.delivery.external-channels",
"repository": "govoplan-campaign",
"resources": ["postgresql", "queue", "smtp", "imap", "postbox", "print-provider"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/92"
},
{
"id": "campaign.retention.generated-artifacts",
"repository": "govoplan-campaign",
"resources": ["postgresql", "object-storage"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/92"
},
{
"id": "files.upload.finalize",
"repository": "govoplan-files",
"resources": ["postgresql", "object-storage", "filesystem-staging"],
"mode": "compensation",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/41"
},
{
"id": "files.retention.purge",
"repository": "govoplan-files",
"resources": ["postgresql", "object-storage", "encryption-key-custody"],
"mode": "irreversible",
"fenced": true,
"adoption": "planned",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/41"
},
{
"id": "files.integrity.reconcile",
"repository": "govoplan-files",
"resources": ["postgresql", "object-storage"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/41"
},
{
"id": "files.connector.write-sync",
"repository": "govoplan-files",
"resources": ["postgresql", "object-storage", "external-connector"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "planned",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/41"
},
{
"id": "mail.outbox.smtp-submit",
"repository": "govoplan-mail",
"resources": ["postgresql", "queue", "smtp"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/19"
},
{
"id": "mail.sent.imap-append",
"repository": "govoplan-mail",
"resources": ["postgresql", "imap"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/19"
},
{
"id": "mail.mailbox.imap-mutate",
"repository": "govoplan-mail",
"resources": ["postgresql", "imap"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "planned",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/19"
},
{
"id": "mail.mailbox.sync-cursor",
"repository": "govoplan-mail",
"resources": ["postgresql", "imap"],
"mode": "atomic",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-mail/issues/19"
},
{
"id": "connectors.sync.read-snapshot",
"repository": "govoplan-connectors",
"resources": ["postgresql", "external-provider"],
"mode": "atomic",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/15"
},
{
"id": "connectors.sync.external-mutation",
"repository": "govoplan-connectors",
"resources": ["postgresql", "queue", "external-provider"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "planned",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/15"
},
{
"id": "dataflow.run.database-only",
"repository": "govoplan-dataflow",
"resources": ["postgresql"],
"mode": "atomic",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/19"
},
{
"id": "dataflow.run.publish-output",
"repository": "govoplan-dataflow",
"resources": ["postgresql", "queue", "object-storage", "external-sink"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-dataflow/issues/19"
},
{
"id": "workflow-engine.instance.state-transition",
"repository": "govoplan-workflow-engine",
"resources": ["postgresql"],
"mode": "atomic",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-workflow-engine/issues/1"
},
{
"id": "workflow-engine.activity.external-effect",
"repository": "govoplan-workflow-engine",
"resources": ["postgresql", "queue", "module-capability", "external-provider"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-workflow-engine/issues/1"
},
{
"id": "core.module-lifecycle.pre-migration",
"repository": "govoplan-core",
"resources": ["postgresql", "package-environment", "webui-bundle", "filesystem"],
"mode": "compensation",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/281"
},
{
"id": "core.module-lifecycle.post-migration",
"repository": "govoplan-core",
"resources": ["postgresql", "package-environment", "webui-bundle", "runtime-nodes"],
"mode": "forward_recovery",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/281"
},
{
"id": "core.module-retirement.destroy-data",
"repository": "govoplan-core",
"resources": ["postgresql", "object-storage", "package-environment"],
"mode": "snapshot_restore",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/281"
},
{
"id": "core.module-runtime.apply-graph",
"repository": "govoplan-core",
"resources": ["postgresql", "runtime-nodes", "module-registry"],
"mode": "compensation",
"fenced": true,
"adoption": "adopted",
"issue": "https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/281"
}
]
}
@@ -13,6 +13,7 @@
"expires_at",
"revoked",
"deployer",
"package_lock",
"images",
"dependencies",
"composition",
@@ -27,6 +28,7 @@
"expires_at": { "type": "string", "format": "date-time" },
"revoked": { "const": false },
"deployer": { "$ref": "#/$defs/artifact" },
"package_lock": { "$ref": "#/$defs/artifact" },
"images": {
"type": "object",
"additionalProperties": false,
+11
View File
@@ -0,0 +1,11 @@
# GovOPlaN developer meta-package
`govoplan` is an optional convenience package for local development and
composition tests. The default dependency set matches the reviewed runtime
release roots; `govoplan[full]` adds every packageable module present in the
workspace at generation time.
This package is not a production deployment artifact. Production installations
consume the signed runtime distribution manifest and digest-pinned OCI images.
The package does not enable modules, apply migrations, choose infrastructure,
or replace installation and recovery evidence.
+90
View File
@@ -0,0 +1,90 @@
[build-system]
requires = ["setuptools>=69", "wheel"]
build-backend = "setuptools.build_meta"
[project]
name = "govoplan"
version = "0.1.15"
description = "Developer convenience package for a versioned GovOPlaN composition"
readme = "README.md"
requires-python = ">=3.12"
license = { text = "AGPL-3.0-or-later" }
dependencies = [
"govoplan-core[server]==0.1.15",
"govoplan-tenancy==0.1.15",
"govoplan-organizations==0.1.15",
"govoplan-identity==0.1.15",
"govoplan-idm==0.1.15",
"govoplan-access==0.1.15",
"govoplan-admin==0.1.15",
"govoplan-policy==0.1.15",
"govoplan-audit==0.1.15",
"govoplan-dashboard==0.1.15",
"govoplan-files==0.1.15",
"govoplan-mail==0.1.15",
"govoplan-campaign==0.1.15",
"govoplan-calendar==0.1.15",
"govoplan-docs==0.1.15",
"govoplan-ops==0.1.15",
]
[project.optional-dependencies]
full = [
"govoplan-addresses==0.1.15",
"govoplan-approvals==0.1.15",
"govoplan-assets==0.1.15",
"govoplan-booking==0.1.15",
"govoplan-cases==0.1.15",
"govoplan-certificates==0.1.15",
"govoplan-committee==0.1.15",
"govoplan-connectors==0.1.15",
"govoplan-consultation==0.1.15",
"govoplan-contracts==0.1.15",
"govoplan-dataflow==0.1.15",
"govoplan-datasources==0.1.15",
"govoplan-decisions==0.1.15",
"govoplan-dist-lists==0.1.15",
"govoplan-encryption==0.1.15",
"govoplan-evaluation==0.1.15",
"govoplan-facilities==0.1.15",
"govoplan-forms==0.1.15",
"govoplan-forms-runtime==0.1.15",
"govoplan-grants==0.1.15",
"govoplan-helpdesk==0.1.15",
"govoplan-identity-trust==0.1.15",
"govoplan-inspections==0.1.15",
"govoplan-learning==0.1.15",
"govoplan-mandates==0.1.15",
"govoplan-notifications==0.1.15",
"govoplan-parties==0.1.15",
"govoplan-permits==0.1.15",
"govoplan-poll==0.1.15",
"govoplan-portal==0.1.15",
"govoplan-postbox==0.1.15",
"govoplan-procurement==0.1.15",
"govoplan-projects==0.1.15",
"govoplan-records==0.1.15",
"govoplan-reporting==0.1.15",
"govoplan-resources==0.1.15",
"govoplan-rest==0.1.15",
"govoplan-risk-compliance==0.1.15",
"govoplan-scheduling==0.1.15",
"govoplan-search==0.1.15",
"govoplan-services==0.1.15",
"govoplan-soap==0.1.15",
"govoplan-templates==0.1.15",
"govoplan-tickets==0.1.15",
"govoplan-transparency==0.1.15",
"govoplan-views==0.1.15",
"govoplan-voting==0.1.15",
"govoplan-wiki==0.1.15",
"govoplan-workflow==0.1.15",
"govoplan-workflow-engine==0.1.15",
]
[project.urls]
Repository = "https://git.add-ideas.de/GovOPlaN/govoplan"
Documentation = "https://govoplan.add-ideas.de"
[tool.setuptools.packages.find]
where = ["src"]
@@ -0,0 +1,12 @@
"""Metadata helpers for the optional GovOPlaN developer composition."""
from importlib.metadata import PackageNotFoundError, version
try:
__version__ = version("govoplan")
except PackageNotFoundError: # pragma: no cover - source checkout only
__version__ = "0+unknown"
__all__ = ["__version__"]
+15 -15
View File
@@ -1,18 +1,18 @@
# Whole-product release install from immutable, independently versioned module tags.
# Only add a module after its referenced tag has been published.
../govoplan-core[server]
govoplan-tenancy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-tenancy.git@v0.1.8
govoplan-organizations @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-organizations.git@v0.1.8
govoplan-identity @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-identity.git@v0.1.8
govoplan-idm @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-idm.git@v0.1.8
govoplan-access @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-access.git@v0.1.8
govoplan-admin @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-admin.git@v0.1.8
govoplan-policy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-policy.git@v0.1.8
govoplan-audit @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-audit.git@v0.1.8
govoplan-dashboard @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-dashboard.git@v0.1.8
govoplan-files @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-files.git@v0.1.8
govoplan-mail @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-mail.git@v0.1.10
govoplan-campaign @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-campaign.git@v0.1.11
govoplan-calendar @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-calendar.git@v0.1.8
govoplan-docs @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-docs.git@v0.1.8
govoplan-ops @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-ops.git@v0.1.8
govoplan-tenancy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-tenancy.git@v0.1.15
govoplan-organizations @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-organizations.git@v0.1.15
govoplan-identity @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-identity.git@v0.1.15
govoplan-idm @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-idm.git@v0.1.15
govoplan-access @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-access.git@v0.1.15
govoplan-admin @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-admin.git@v0.1.15
govoplan-policy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-policy.git@v0.1.15
govoplan-audit @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-audit.git@v0.1.15
govoplan-dashboard @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-dashboard.git@v0.1.15
govoplan-files @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-files.git@v0.1.15
govoplan-mail @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-mail.git@v0.1.15
govoplan-campaign @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-campaign.git@v0.1.15
govoplan-calendar @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-calendar.git@v0.1.15
govoplan-docs @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-docs.git@v0.1.15
govoplan-ops @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-ops.git@v0.1.15
+120
View File
@@ -0,0 +1,120 @@
from __future__ import annotations
import base64
import json
from pathlib import Path
import stat
import subprocess
import sys
import tempfile
import unittest
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from jsonschema import Draft202012Validator, FormatChecker
META_ROOT = Path(__file__).resolve().parents[1]
GENERATOR = META_ROOT / "tools" / "assessments" / "generate-authority-keypair.py"
class AssessmentAuthorityKeypairTests(unittest.TestCase):
def test_generates_schema_valid_scoped_proof_authority(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
output_dir = Path(temp_dir)
output_dir.chmod(0o700)
private_path = output_dir / "target.pem"
keyring_path = output_dir / "target.json"
result = subprocess.run(
(
sys.executable,
str(GENERATOR),
"--purpose",
"proof",
"--key-id",
"authority:target-2026",
"--scope",
"target_environment",
"--scope",
"operations",
"--private-key",
str(private_path),
"--keyring",
str(keyring_path),
),
check=False,
capture_output=True,
text=True,
)
self.assertEqual(0, result.returncode, result.stderr)
self.assertEqual(0o600, stat.S_IMODE(private_path.stat().st_mode))
self.assertEqual(0o600, stat.S_IMODE(keyring_path.stat().st_mode))
keyring = json.loads(keyring_path.read_text(encoding="utf-8"))
schema = json.loads(
(
META_ROOT
/ "docs"
/ "capability-fit-proof-authority-keyring.schema.json"
).read_text(encoding="utf-8")
)
errors = tuple(
Draft202012Validator(
schema, format_checker=FormatChecker()
).iter_errors(keyring)
)
self.assertEqual((), errors)
self.assertEqual(
["target_environment", "operations"],
keyring["keys"][0]["allowed_scopes"],
)
private_key = serialization.load_pem_private_key(
private_path.read_bytes(), password=None
)
self.assertIsInstance(private_key, Ed25519PrivateKey)
public_key = base64.b64encode(
private_key.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
).decode("ascii")
self.assertEqual(public_key, keyring["keys"][0]["public_key"])
def test_installer_authority_uses_fixed_scope_and_refuses_overwrite(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
output_dir = Path(temp_dir)
output_dir.chmod(0o700)
private_path = output_dir / "installer.pem"
keyring_path = output_dir / "installer.json"
command = (
sys.executable,
str(GENERATOR),
"--purpose",
"installer",
"--key-id",
"authority:installer-2026",
"--private-key",
str(private_path),
"--keyring",
str(keyring_path),
)
first = subprocess.run(
command, check=False, capture_output=True, text=True
)
second = subprocess.run(
command, check=False, capture_output=True, text=True
)
self.assertEqual(0, first.returncode, first.stderr)
self.assertNotEqual(0, second.returncode)
keyring = json.loads(keyring_path.read_text(encoding="utf-8"))
self.assertEqual(
["installed_release_origin"],
keyring["keys"][0]["allowed_scopes"],
)
if __name__ == "__main__":
unittest.main()
+430
View File
@@ -0,0 +1,430 @@
from __future__ import annotations
import base64
from contextlib import redirect_stderr, redirect_stdout
from datetime import UTC, datetime, timedelta
import hashlib
import io
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
META_ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(META_ROOT / "tools" / "deployment"))
from govoplan_deploy.backup_evidence import verify_backup_evidence # noqa: E402
from govoplan_deploy.bundle import ( # noqa: E402
atomic_write,
bundle_paths,
canonical_json,
read_env,
)
from govoplan_deploy.cli import main as deploy_main # noqa: E402
from govoplan_deploy.distribution import ( # noqa: E402
DistributionError,
canonical_signed_payload,
canonical_json as canonical_distribution_json,
)
from govoplan_deploy.model import default_spec, parse_spec # noqa: E402
from govoplan_deploy.planning import ( # noqa: E402
release_change_requires_backup,
verify_stored_backup_evidence,
)
class BackupEvidenceTests(unittest.TestCase):
def setUp(self) -> None:
self.now = datetime(2026, 8, 3, 12, tzinfo=UTC)
self.private = Ed25519PrivateKey.generate()
public = (
self.private.public_key()
.public_bytes(
serialization.Encoding.PEM,
serialization.PublicFormat.SubjectPublicKeyInfo,
)
.decode("ascii")
)
self.keyring = {
"schema_version": "1",
"purpose": "govoplan-backup-evidence",
"keys": [
{
"key_id": "backup-controller-1",
"algorithm": "ed25519",
"status": "active",
"public_key_pem": public,
"not_before": (self.now - timedelta(days=1)).isoformat(),
"expires_at": (self.now + timedelta(days=365)).isoformat(),
}
],
}
self.release = {
"channel": "stable",
"version": "1.2.3",
"manifest_sha256": "a" * 64,
"composition_sha256": "b" * 64,
"api_image": "registry.example/api@sha256:" + "c" * 64,
"web_image": "registry.example/web@sha256:" + "d" * 64,
}
def test_verifies_coordinated_restore_drill_and_release_binding(self) -> None:
summary = verify_backup_evidence(
self._evidence(),
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
self.assertEqual("recovery-1", summary["recovery_point_id"])
self.assertEqual("restore-1", summary["restore_drill_id"])
self.assertEqual("backup-controller-1", summary["signature_key_id"])
def test_tampering_staleness_and_partial_restore_fail_closed(self) -> None:
tampered = self._evidence()
tampered["components"]["objects"]["object_count"] = 999
with self.assertRaisesRegex(DistributionError, "signature verification"):
verify_backup_evidence(
tampered,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
stale = self._evidence(captured=self.now - timedelta(days=2))
with self.assertRaisesRegex(DistributionError, "stale"):
verify_backup_evidence(
stale,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
partial = self._evidence()
partial["restore_drill"]["objects_verified"] = False
partial["signatures"] = [self._signature(partial)]
with self.assertRaisesRegex(DistributionError, "objects_verified"):
verify_backup_evidence(
partial,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
def test_wrong_release_key_purpose_and_component_skew_fail_closed(self) -> None:
wrong_release = dict(self.release)
wrong_release["version"] = "1.2.4"
with self.assertRaisesRegex(DistributionError, "release field"):
verify_backup_evidence(
self._evidence(),
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=wrong_release,
now=self.now,
)
wrong_keyring = dict(self.keyring)
wrong_keyring["purpose"] = "govoplan-runtime-distribution"
with self.assertRaisesRegex(DistributionError, "wrong purpose"):
verify_backup_evidence(
self._evidence(),
wrong_keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
skewed = self._evidence()
skewed["components"]["database"]["captured_at"] = (
self.now - timedelta(hours=1)
).isoformat()
skewed["signatures"] = [self._signature(skewed)]
with self.assertRaisesRegex(DistributionError, "one recovery point"):
verify_backup_evidence(
skewed,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
false_rto = self._evidence()
false_rto["restore_drill"]["measured_rto_seconds"] = 1
false_rto["signatures"] = [self._signature(false_rto)]
with self.assertRaisesRegex(DistributionError, "RTO"):
verify_backup_evidence(
false_rto,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
now=self.now,
)
def test_provider_signing_tool_emits_canonical_verified_evidence(self) -> None:
self.now = datetime.now(UTC)
self.keyring["keys"][0]["not_before"] = (
self.now - timedelta(days=1)
).isoformat()
self.keyring["keys"][0]["expires_at"] = (
self.now + timedelta(days=365)
).isoformat()
evidence = self._evidence()
evidence["signatures"] = []
with tempfile.TemporaryDirectory(prefix="govoplan-backup-signer-") as value:
root = Path(value)
source = root / "unsigned.json"
output = root / "signed.json"
keyring = root / "keyring.json"
private_key = root / "private.pem"
atomic_write(source, canonical_json(evidence), mode=0o600)
atomic_write(keyring, canonical_json(self.keyring), mode=0o600)
atomic_write(
private_key,
self.private.private_bytes(
serialization.Encoding.PEM,
serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
),
mode=0o600,
)
result = subprocess.run(
[
sys.executable,
str(META_ROOT / "tools/deployment/sign-backup-evidence.py"),
"--input",
str(source),
"--output",
str(output),
"--trusted-keyring",
str(keyring),
"--signing-key",
f"backup-controller-1={private_key}",
],
cwd=META_ROOT,
check=False,
capture_output=True,
text=True,
)
self.assertEqual(0, result.returncode, result.stderr)
encoded = output.read_bytes()
signed = json.loads(encoded)
self.assertEqual(canonical_distribution_json(signed), encoded)
summary = verify_backup_evidence(
signed,
self.keyring,
installation_id="govoplan-test",
profile="self-hosted",
release=self.release,
)
self.assertEqual("backup-controller-1", summary["signature_key_id"])
def test_cli_adoption_gates_the_next_release_against_previous_receipt(self) -> None:
self.now = datetime.now(UTC)
self.keyring["keys"][0]["not_before"] = (
self.now - timedelta(days=1)
).isoformat()
self.keyring["keys"][0]["expires_at"] = (
self.now + timedelta(days=365)
).isoformat()
evidence = self._evidence()
encoded_evidence = canonical_distribution_json(evidence)
encoded_keyring = canonical_distribution_json(self.keyring)
with tempfile.TemporaryDirectory(prefix="govoplan-backup-evidence-") as value:
paths = bundle_paths(Path(value))
paths.root.chmod(0o700)
raw = default_spec(
installation_id="govoplan-test",
profile="self-hosted",
public_url="https://govoplan.example.test",
ingress_mode="existing-proxy",
trusted_proxy_cidrs=("127.0.0.1/32",),
).to_dict()
raw["release"] = {
**raw["release"],
**self.release,
}
current = parse_spec(raw)
atomic_write(paths.spec, canonical_json(current.to_dict()), mode=0o600)
source_evidence = paths.root / "source-backup.json"
source_keyring = paths.root / "source-keyring.json"
atomic_write(source_evidence, encoded_evidence, mode=0o600)
atomic_write(source_keyring, encoded_keyring, mode=0o600)
output = io.StringIO()
with redirect_stdout(output), redirect_stderr(output):
result = deploy_main(
[
"verify-backup",
"--directory",
str(paths.root),
"--evidence",
str(source_evidence),
"--evidence-sha256",
hashlib.sha256(encoded_evidence).hexdigest(),
"--trusted-keyring",
str(source_keyring),
"--adopt",
]
)
self.assertEqual(0, result, output.getvalue())
runtime_environment = read_env(paths.env)
self.assertEqual(
"verified",
runtime_environment["GOVOPLAN_BACKUP_EVIDENCE_STATE"],
)
self.assertEqual(
"recovery-1",
runtime_environment["GOVOPLAN_BACKUP_RECOVERY_POINT_ID"],
)
self.assertNotIn("snapshot:postgres", str(runtime_environment))
self.assertNotIn("urn:kms", str(runtime_environment))
receipt = {
"installation_id": current.installation_id,
"profile": current.profile,
"release": dict(self.release),
}
atomic_write(paths.receipt, canonical_json(receipt), mode=0o600)
target_raw = current.to_dict()
target_raw["release"]["version"] = "1.2.4"
target_raw["release"]["manifest_sha256"] = "9" * 64
target = parse_spec(target_raw)
self.assertTrue(release_change_requires_backup(target, receipt))
summary = verify_stored_backup_evidence(
target,
paths,
receipt=receipt,
)
self.assertEqual("recovery-1", summary["recovery_point_id"])
def _evidence(self, *, captured: datetime | None = None) -> dict[str, object]:
captured = captured or self.now - timedelta(hours=2)
started = captured + timedelta(minutes=15)
completed = captured + timedelta(minutes=30)
issued = completed + timedelta(minutes=10)
artifact_time = captured.isoformat()
payload: dict[str, object] = {
"schema_version": "1",
"evidence_id": "backup-1",
"installation_id": "govoplan-test",
"deployment_subject": {
"profile": "self-hosted",
"topology": "compose",
"subject_ref": "urn:govoplan:installation:govoplan-test",
},
"release": dict(self.release),
"recovery_point": {
"id": "recovery-1",
"captured_at": captured.isoformat(),
"consistency": "application-quiesced",
"rpo_seconds": 300,
"write_fence": {
"mode": "application-quiesce",
"token_sha256": "e" * 64,
"established_at": captured.isoformat(),
},
},
"components": {
"database": {
"provider": "postgres",
"artifact_ref": "snapshot:postgres:backup-1",
"artifact_sha256": "1" * 64,
"snapshot_id": "pg-snapshot-1",
"lsn": "0/16B6C50",
"protected": True,
"encryption_key_ref": "urn:kms:key:database-backup",
"captured_at": artifact_time,
},
"objects": {
"provider": "s3",
"artifact_ref": "s3://backup/govoplan-test/recovery-1",
"manifest_sha256": "2" * 64,
"version_id": "object-snapshot-1",
"object_count": 4,
"total_bytes": 1024,
"protected": True,
"encryption_key_ref": "urn:kms:key:object-backup",
"captured_at": artifact_time,
},
"configuration": {
"artifact_ref": "backup:configuration:recovery-1",
"sha256": "3" * 64,
"protected": True,
"encryption_key_ref": "urn:kms:key:configuration-backup",
"captured_at": artifact_time,
},
"key_custody": {
"provider": "kms",
"keyset_ref": "urn:kms:keyset:govoplan-test",
"keyset_version": "version-4",
"recoverable": True,
"captured_at": artifact_time,
},
},
"restore_drill": {
"drill_id": "restore-1",
"recovery_point_id": "recovery-1",
"started_at": started.isoformat(),
"completed_at": completed.isoformat(),
"isolated_target_ref": "urn:govoplan:restore-target:restore-1",
"release_manifest_sha256": self.release["manifest_sha256"],
"migration_heads_sha256": "4" * 64,
"representative_object_manifest_sha256": "2" * 64,
"database_verified": True,
"objects_verified": True,
"configuration_verified": True,
"key_custody_verified": True,
"semantic_checks": [
{
"id": "institutional-journey",
"status": "passed",
"evidence_ref": "evidence:journey:institutional-1",
}
],
"measured_rpo_seconds": 120,
"measured_rto_seconds": 900,
"evidence_ref": "evidence:restore:restore-1",
},
"issued_at": issued.isoformat(),
"expires_at": (self.now + timedelta(days=7)).isoformat(),
"revoked": False,
"signatures": [],
}
payload["signatures"] = [self._signature(payload)]
return payload
def _signature(self, payload: dict[str, object]) -> dict[str, str]:
return {
"key_id": "backup-controller-1",
"algorithm": "ed25519",
"value": base64.b64encode(
self.private.sign(canonical_signed_payload(payload))
).decode("ascii"),
}
if __name__ == "__main__":
unittest.main()
+25 -1
View File
@@ -318,7 +318,16 @@ class DeploymentInstallerTests(unittest.TestCase):
},
)
manifest = render_kubernetes(spec, environment)
manifest = render_kubernetes(
spec,
environment,
backup_required=True,
backup_evidence={
"evidence_sha256": "c" * 64,
"recovery_point_id": "recovery-1",
"restore_drill_id": "drill-1",
},
)
rendered = json.dumps(manifest, sort_keys=True)
kinds = [item["kind"] for item in manifest["items"]]
deployments = {
@@ -351,6 +360,18 @@ class DeploymentInstallerTests(unittest.TestCase):
"forward-recovery",
migration["metadata"]["annotations"]["govoplan.add-ideas.de/recovery-mode"],
)
self.assertEqual(
"c" * 64,
migration["metadata"]["annotations"][
"govoplan.add-ideas.de/backup-evidence-sha256"
],
)
self.assertEqual(
"recovery-1",
migration["metadata"]["annotations"][
"govoplan.add-ideas.de/recovery-point"
],
)
config = next(item for item in manifest["items"] if item["kind"] == "ConfigMap")
self.assertEqual("shared", config["data"]["GOVOPLAN_STATE_PROFILE"])
self.assertEqual("3", config["data"]["GOVOPLAN_EXPECTED_API_REPLICAS"])
@@ -530,6 +551,9 @@ class DeploymentInstallerTests(unittest.TestCase):
)
self.assertIn("caddy-data:/data", ingress["volumes"])
self.assertIn("caddy-config:/config", ingress["volumes"])
self.assertEqual(["ALL"], ingress["cap_drop"])
self.assertEqual(["NET_BIND_SERVICE"], ingress["cap_add"])
self.assertEqual(["no-new-privileges:true"], ingress["security_opt"])
self.assertIn("reverse_proxy load-balancer:8080", render_caddy_config(spec))
self.assertNotIn("operator@example.test", json.dumps(compose))
+158
View File
@@ -0,0 +1,158 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import json
import subprocess
import sys
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
def _load_module():
path = ROOT / "tools/checks/managed-ingress-drill.py"
spec = importlib.util.spec_from_file_location("managed_ingress_drill", path)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
INGRESS = _load_module()
class ManagedIngressDrillTests(unittest.TestCase):
def test_config_is_streamed_into_a_daemon_visible_volume(self) -> None:
completed = subprocess.CompletedProcess([], 0, "", "")
with patch.object(INGRESS, "_run", return_value=completed) as run:
INGRESS._write_volume_file(
image="registry.example/caddy@sha256:" + "1" * 64,
volume="config-volume",
filename="Caddyfile",
content=":8080 { respond /health 200 }\n",
)
argv = run.call_args.args[0]
self.assertIn("type=volume,src=config-volume,dst=/govoplan-config", argv)
self.assertIn("0:0", argv)
self.assertNotIn("type=bind", " ".join(argv))
self.assertEqual(
":8080 { respond /health 200 }\n",
run.call_args.kwargs["input_text"],
)
def test_config_filename_cannot_escape_the_volume(self) -> None:
with self.assertRaisesRegex(ValueError, "invalid config filename"):
INGRESS._write_volume_file(
image="registry.example/caddy@sha256:" + "1" * 64,
volume="config-volume",
filename="../Caddyfile",
content="",
)
def test_drill_has_no_runner_local_bind_mounts(self) -> None:
source = (ROOT / "tools/checks/managed-ingress-drill.py").read_text(
encoding="utf-8"
)
self.assertNotIn("type=bind", source)
self.assertIn('"--network-alias",\n "load-balancer"', source)
self.assertNotIn('"127.0.0.1::8080"', source)
self.assertIn("requested_http_port", source)
self.assertIn("requested_https_port", source)
self.assertIn('"--cap-add",\n "NET_BIND_SERVICE"', source)
def test_published_port_reads_the_docker_mapping(self) -> None:
completed = subprocess.CompletedProcess(
[],
0,
json.dumps(
{
"8443/tcp": [
{"HostIp": "127.0.0.1", "HostPort": "49152"}
]
}
),
"",
)
with patch.object(INGRESS, "_run", return_value=completed) as run:
port = INGRESS._published_port("ingress", 8443)
self.assertEqual(49152, port)
self.assertEqual(
[
"docker",
"inspect",
"--format",
"{{json .HostConfig.PortBindings}}",
"ingress",
],
run.call_args.args[0],
)
def test_published_port_rejects_non_loopback_binding(self) -> None:
completed = subprocess.CompletedProcess(
[],
0,
'{"8443/tcp":[{"HostIp":"0.0.0.0","HostPort":"49152"}]}',
"",
)
with patch.object(INGRESS, "_run", return_value=completed):
with self.assertRaisesRegex(RuntimeError, "loopback binding"):
INGRESS._published_port("ingress", 8443)
def test_probe_runs_as_a_network_sibling_from_a_digest_image(self) -> None:
completed = subprocess.CompletedProcess([], 0, "", "")
image = "registry.example/runtime-api@sha256:" + "1" * 64
with patch.object(INGRESS, "_run", return_value=completed) as run:
INGRESS._probe_ingress(
image=image,
network="deployment-network",
container="ingress",
)
argv = run.call_args.args[0]
self.assertEqual("docker", argv[0])
self.assertIn("deployment-network", argv)
self.assertIn(image, argv)
self.assertIn('(\"ingress\", port)', argv[-1])
self.assertIn("server_hostname=\"localhost\"", argv[-1])
self.assertNotIn("localhost:49152", argv[-1])
def test_probe_diagnostics_include_container_stderr(self) -> None:
probe_failure = subprocess.CalledProcessError(1, ["docker", "run"])
state = subprocess.CompletedProcess([], 0, '{"Running":false}', "")
logs = subprocess.CompletedProcess([], 0, "", "caddy startup failed")
with patch.object(
INGRESS,
"_run",
side_effect=[probe_failure, state, logs],
), patch.object(INGRESS.sys, "stderr") as stderr:
with self.assertRaises(subprocess.CalledProcessError):
INGRESS._probe_ingress(
image="registry.example/runtime-api@sha256:" + "1" * 64,
network="deployment-network",
container="ingress",
)
rendered = "".join(call.args[0] for call in stderr.write.call_args_list)
self.assertIn('"Running":false', rendered)
self.assertIn("caddy startup failed", rendered)
def test_standalone_workflow_is_dispatch_only_and_digest_bounded(self) -> None:
workflow = (
ROOT / ".gitea/workflows/runtime-ingress-drill.yml"
).read_text(encoding="utf-8")
self.assertIn("workflow_dispatch:", workflow)
self.assertNotIn("\n push:", workflow)
self.assertIn("--probe-image \"$PROBE_IMAGE\"", workflow)
self.assertIn("GOVOPLAN_REGISTRY_TOKEN", workflow)
if __name__ == "__main__":
unittest.main()
+113
View File
@@ -0,0 +1,113 @@
from __future__ import annotations
import json
from pathlib import Path
import shutil
import subprocess
import tempfile
import unittest
META_ROOT = Path(__file__).resolve().parents[1]
class ModulePackageWorkflowTests(unittest.TestCase):
def test_template_enforces_tag_version_hash_and_registry_contract(self) -> None:
workflow = (
META_ROOT / "tools/repo/templates/module-package-release.yml"
).read_text(encoding="utf-8")
self.assertIn("GITEA_REPOSITORY: ${{ gitea.repository }}", workflow)
self.assertNotIn("tag_protections", workflow)
self.assertNotIn("secrets.GITEA_TOKEN", workflow)
self.assertIn("git merge-base --is-ancestor", workflow)
self.assertIn("does not match", workflow)
self.assertIn("package-artifacts.json", workflow)
self.assertIn("api/packages/GovOPlaN/pypi", workflow)
self.assertIn("api/packages/GovOPlaN/npm", workflow)
self.assertIn('npm publish "./${webui_packages[0]}"', workflow)
self.assertIn("Check immutable registry state", workflow)
self.assertIn('files[0].get("sha256") != expected_sha256', workflow)
self.assertIn('if [[ "$PUBLISH_PYPI" == 1 ]]', workflow)
self.assertIn('[[ "$PUBLISH_NPM" == 1 ]]', workflow)
self.assertIn("GOVOPLAN_PACKAGE_TOKEN", workflow)
self.assertIn("must resolve to an exact registry version", workflow)
self.assertIn("git\\\\.add-ideas\\\\.de/(?:GovOPlaN|add-ideas)", workflow)
self.assertIn("release package identity does not match", workflow)
self.assertNotIn("Generic", workflow)
@unittest.skipUnless(shutil.which("node"), "Node.js is required")
def test_webui_publication_normalizes_internal_git_dependencies(self) -> None:
workflow = (
META_ROOT / "tools/repo/templates/module-package-release.yml"
).read_text(encoding="utf-8")
marker = " node <<'NODE'\n"
script = workflow.split(marker, 1)[1].split("\n NODE", 1)[0]
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
package_dir = root / ".package-webui"
package_dir.mkdir()
package_path = package_dir / "package.json"
package_path.write_text(
json.dumps(
{
"name": "@govoplan/core-webui",
"version": "0.1.14",
"private": True,
"dependencies": {
"@govoplan/access-webui": (
"git+ssh://git@git.add-ideas.de/GovOPlaN/"
"govoplan-access.git#v0.1.11"
),
"@govoplan/admin-webui": (
"git+ssh://git@git.add-ideas.de/add-ideas/"
"govoplan-admin.git#v0.1.8"
)
},
}
),
encoding="utf-8",
)
subprocess.run(
["node"],
input=script,
cwd=root,
check=True,
text=True,
capture_output=True,
)
package = json.loads(package_path.read_text(encoding="utf-8"))
self.assertNotIn("private", package)
self.assertEqual(
"0.1.11", package["dependencies"]["@govoplan/access-webui"]
)
self.assertEqual(
"0.1.8", package["dependencies"]["@govoplan/admin-webui"]
)
def test_sync_script_only_targets_packageable_govoplan_repositories(self) -> None:
namespace: dict[str, object] = {
"__file__": str(META_ROOT / "tools/repo/sync-module-package-workflows.py"),
"__name__": "test_sync_module_package_workflows",
}
script = (META_ROOT / "tools/repo/sync-module-package-workflows.py").read_text(
encoding="utf-8"
)
exec(compile(script, str(namespace["__file__"]), "exec"), namespace)
with tempfile.TemporaryDirectory() as temporary:
parent = Path(temporary)
package_repositories = namespace["package_repositories"]
# The production inventory is authoritative, so a temporary parent
# only exposes matching paths that are present in that inventory.
known = parent / "govoplan-core"
known.mkdir()
(known / "pyproject.toml").write_text("[project]\n", encoding="utf-8")
self.assertEqual(package_repositories(parent), (known,))
if __name__ == "__main__":
unittest.main()
+167
View File
@@ -0,0 +1,167 @@
from __future__ import annotations
from io import BytesIO
import importlib.util
import json
from pathlib import Path
import sys
import tarfile
import tempfile
import tomllib
import unittest
import zipfile
ROOT = Path(__file__).resolve().parents[1]
def _load(name: str, path: Path):
spec = importlib.util.spec_from_file_location(name, path)
assert spec is not None and spec.loader is not None
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return module
PACKAGE_SET = _load(
"generate_release_package_set",
ROOT / "tools/release/generate-release-package-set.py",
)
ARTIFACTS = _load(
"resolve_package_artifacts",
ROOT / "tools/release/resolve-package-artifacts.py",
)
class PackageRegistryReleaseTests(unittest.TestCase):
def test_current_release_sources_form_a_hash_bound_package_set(self) -> None:
core_version = tomllib.loads(
(ROOT.parent / "govoplan-core/pyproject.toml").read_text(encoding="utf-8")
)["project"]["version"]
payload = PACKAGE_SET.generate_package_set(
core_version=core_version,
requirements=ROOT / "requirements-release.txt",
workspace=ROOT.parent,
)
self.assertEqual("1", payload["schema_version"])
self.assertEqual("govoplan-core", payload["python"][0]["name"])
self.assertIn(
"@govoplan/core-webui",
{item["name"] for item in payload["webui"]},
)
unsigned = dict(payload)
digest = unsigned.pop("package_set_sha256")
self.assertEqual(ARTIFACTS._canonical_sha256(unsigned), digest)
def test_wheel_and_webui_artifacts_are_verified_by_embedded_identity(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-package-artifacts-") as value:
root = Path(value)
wheels = root / "wheels"
webui = root / "webui"
wheels.mkdir()
webui.mkdir()
wheel = wheels / "govoplan_demo-1.2.3-py3-none-any.whl"
with zipfile.ZipFile(wheel, "w") as archive:
archive.writestr(
"govoplan_demo-1.2.3.dist-info/METADATA",
"Metadata-Version: 2.1\nName: govoplan-demo\nVersion: 1.2.3\n",
)
package_json = json.dumps(
{"name": "@govoplan/demo-webui", "version": "1.2.3"}
).encode("utf-8")
npm = webui / "govoplan-demo-webui-1.2.3.tgz"
with tarfile.open(npm, "w:gz") as archive:
member = tarfile.TarInfo("package/package.json")
member.size = len(package_json)
archive.addfile(member, BytesIO(package_json))
source = {
"version": "1.2.3",
"repository": "govoplan-demo",
"tag": "v1.2.3",
"commit": "1" * 40,
}
python_rows = ARTIFACTS._verify_wheels(
({"name": "govoplan-demo", "extras": ["server"], **source},), wheels
)
webui_rows = ARTIFACTS._verify_webui(
({"name": "@govoplan/demo-webui", **source},), webui
)
self.assertEqual("govoplan-demo", python_rows[0]["name"])
self.assertEqual(["server"], python_rows[0]["extras"])
self.assertEqual("@govoplan/demo-webui", webui_rows[0]["name"])
self.assertTrue(str(webui_rows[0]["integrity"]).startswith("sha512-"))
def test_package_set_rejects_argument_shaped_package_names(self) -> None:
payload = {
"schema_version": "1",
"release_version": "1.2.3",
"registries": {
"python": "https://packages.example.test/pypi/simple",
"npm": "https://packages.example.test/npm/",
},
"python": [
{
"name": "--index-url",
"version": "1.2.3",
"repository": "govoplan-demo",
"extras": [],
"tag": "v1.2.3",
"commit": "1" * 40,
}
],
"webui": [
{
"name": "@govoplan/demo-webui",
"version": "1.2.3",
"repository": "govoplan-demo",
"tag": "v1.2.3",
"commit": "1" * 40,
}
],
}
payload["package_set_sha256"] = ARTIFACTS._canonical_sha256(payload)
with tempfile.TemporaryDirectory() as value:
path = Path(value) / "packages.json"
path.write_text(json.dumps(payload), encoding="utf-8")
with self.assertRaisesRegex(
ARTIFACTS.PackageArtifactError, "invalid identity"
):
ARTIFACTS._load_package_set(path)
def test_runtime_workflow_consumes_registry_artifacts_and_publishes_lock(self) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
self.assertIn("resolve-package-artifacts.py", workflow)
self.assertIn("package-artifacts.lock.json", workflow)
self.assertIn(
"--package-lock runtime-output/package-artifacts.lock.json",
workflow,
)
self.assertNotIn(
"pip wheel --no-deps --wheel-dir runtime-output/local-wheels",
workflow,
)
def test_developer_meta_package_matches_workspace_versions(self) -> None:
script = _load(
"generate_developer_meta_package",
ROOT / "tools/release/generate-developer-meta-package.py",
)
expected = script.render(
workspace=ROOT.parent,
requirements=ROOT / "requirements-release.txt",
)
actual = (ROOT / "packages/govoplan-meta/pyproject.toml").read_text(
encoding="utf-8"
)
self.assertEqual(expected, actual)
if __name__ == "__main__":
unittest.main()
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
import unittest
META_ROOT = Path(__file__).resolve().parents[1]
TOOLS_ROOT = META_ROOT / "tools" / "gitea"
if str(TOOLS_ROOT) not in sys.path:
sys.path.insert(0, str(TOOLS_ROOT))
SCRIPT = TOOLS_ROOT / "gitea-dispatch-package-set.py"
SPEC = importlib.util.spec_from_file_location("gitea_dispatch_package_set", SCRIPT)
assert SPEC is not None and SPEC.loader is not None
MODULE = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = MODULE
SPEC.loader.exec_module(MODULE)
class PackageSetDispatchTests(unittest.TestCase):
def test_meta_package_resolves_to_exact_tagged_repository_targets(self) -> None:
targets = MODULE.package_targets()
self.assertEqual(66, len(targets))
self.assertEqual(66, len({target.distribution for target in targets}))
by_name = {target.distribution: target for target in targets}
self.assertEqual("v0.1.14", by_name["govoplan-core"].tag)
self.assertEqual("v0.1.8", by_name["govoplan-access"].tag)
self.assertTrue(by_name["govoplan-core"].tag_exists)
self.assertTrue(by_name["govoplan-access"].has_webui)
self.assertEqual(
"@govoplan/access-webui",
by_name["govoplan-access"].webui_package,
)
if __name__ == "__main__":
unittest.main()
+136
View File
@@ -141,6 +141,34 @@ class PlatformInterfaceInventoryTests(unittest.TestCase):
self.assertEqual(1, result["summary"]["stale_endpoint_declarations"])
self.assertIsNone(result["api"]["backend_endpoints"][0]["surface"])
def test_endpoint_only_strict_mode_does_not_fail_on_translation_debt(
self,
) -> None:
result = {
"translation_health": {"missing_catalog_entries": ["missing.key"]},
"api": {
"unclassified_endpoints": [],
"stale_endpoint_declarations": [],
},
}
self.assertEqual(
[],
inventory._strict_failures(
result,
check_translations=False,
check_endpoints=True,
),
)
self.assertEqual(
["used translation keys are missing from generated catalogs"],
inventory._strict_failures(
result,
check_translations=True,
check_endpoints=True,
),
)
def test_fastapi_route_scanner_includes_router_prefix(self) -> None:
tree = ast.parse(
"""
@@ -171,6 +199,114 @@ def read_item(item_id: str):
},
)
def test_source_declarations_normalize_stable_control_and_contribution_ids(
self,
) -> None:
webui = {
"fields": [
{
"repository": "govoplan-example",
"file": "webui/src/Example.tsx",
"line": 12,
"column": 3,
"id": "govoplan-example.field.example.name.abc123",
"idSource": "source_anchor",
"explicitId": None,
"context": "Example",
"helpId": "govoplan-example.field.example.name.abc123.help",
"helpDynamic": False,
}
],
"actions": [],
"contributions": [
{
"repository": "govoplan-example",
"file": "webui/src/module.ts",
"line": 20,
"column": 5,
"kind": "frontend_route",
"id": "/examples/:exampleId",
"path": "/examples/:exampleId",
}
],
"translationCatalog": {"en": {}, "de": {}},
}
manifests = [{"repository": "govoplan-example", "id": "examples"}]
declarations = inventory._source_interface_declarations(webui, manifests)
keys = {item["key"] for item in declarations}
self.assertIn("field:examples.field.example.name.abc123", keys)
self.assertIn(
"help:examples.field.example.name.abc123.help",
keys,
)
self.assertIn(
"frontend_route:examples.route.examples.exampleid",
keys,
)
def test_declaration_health_rejects_duplicate_and_undeclared_source_ids(
self,
) -> None:
declaration = {
"key": "frontend_route:example.route.unlisted",
"id": "example.route.unlisted",
"module_id": "example",
"kind": "frontend_route",
"origin": "webui_contribution",
}
manifests = [
{
"id": "example",
"repository": "govoplan-example",
"interface_catalog": {"declarations": []},
}
]
health = inventory._declaration_health(
[declaration, dict(declaration)],
manifests,
)
self.assertEqual(1, len(health["duplicate_ids"]))
self.assertEqual(1, len(health["undeclared_source_surfaces"]))
def test_runtime_snapshot_comparison_accepts_an_installed_subset(self) -> None:
manifests = [
{
"id": "one",
"interface_catalog": {
"contract_version": "1",
"module_id": "one",
"module_version": "1.0.0",
"digest": "sha256:one",
},
},
{
"id": "two",
"interface_catalog": {
"contract_version": "1",
"module_id": "two",
"module_version": "1.0.0",
"digest": "sha256:two",
},
},
]
snapshot = {
"contract_version": "1",
"modules": [dict(manifests[1]["interface_catalog"])],
}
comparison = inventory._compare_runtime_snapshot(snapshot, manifests)
self.assertEqual(["two"], comparison["matched_modules"])
self.assertEqual([], comparison["mismatches"])
snapshot["modules"][0]["digest"] = "sha256:changed"
comparison = inventory._compare_runtime_snapshot(snapshot, manifests)
self.assertEqual("digest_mismatch", comparison["mismatches"][0]["reason"])
if __name__ == "__main__":
unittest.main()
+54
View File
@@ -113,6 +113,60 @@ class PythonEnvironmentSyncTests(unittest.TestCase):
self.assertEqual(plan.mode, "Selective Python environment repair")
self.assertEqual(plan.commands[0][-2:], ("-e", str(project_root)))
def test_metadata_sync_also_repairs_unrelated_missing_distribution(self) -> None:
sync = load_sync_module()
with tempfile.TemporaryDirectory(prefix="govoplan-python-sync-") as directory:
root = Path(directory)
requirements = root / "requirements-dev.txt"
requirements.write_text("-e ./govoplan-changed\n-e ./govoplan-missing\n", encoding="utf-8")
for project in ("govoplan-changed", "govoplan-missing"):
project_root = root / project
project_root.mkdir()
(project_root / "pyproject.toml").write_text(
f'[project]\nname = "{project}"\nversion = "0.1.10"\n',
encoding="utf-8",
)
entries = sync.local_requirement_entries(requirements)
fingerprint = sync.build_fingerprint(
requirements=requirements,
python="/test/venv/bin/python",
local_requirements=entries,
)
requirements_digest = hashlib.sha256(requirements.read_bytes()).hexdigest()
previous = {
"version": sync.STAMP_VERSION,
"python": "/test/venv/bin/python",
"inputs": [
{"path": str(requirements), "sha256": requirements_digest},
{"path": entries[0].pyproject, "sha256": "stale"},
{
"path": entries[1].pyproject,
"sha256": hashlib.sha256(Path(entries[1].pyproject).read_bytes()).hexdigest(),
},
],
"requirements_entries": [
entry.as_dict() for entry in sync.parse_requirement_entries(requirements)
],
}
plan = sync.build_install_plan(
previous=previous,
fingerprint=fingerprint,
requirements=requirements,
python="/test/venv/bin/python",
local_requirements=entries,
repair_requirements=(entries[1],),
force=False,
)
self.assertEqual(plan.mode, "Selective Python environment sync")
self.assertEqual(len(plan.commands), 1)
command = plan.commands[0]
self.assertEqual(command.count("-e"), 2)
self.assertIn(str(root / "govoplan-changed"), command)
self.assertIn(str(root / "govoplan-missing"), command)
def test_declared_module_entry_points_are_part_of_environment_validation(self) -> None:
sync = load_sync_module()
with tempfile.TemporaryDirectory(prefix="govoplan-python-sync-") as directory:
@@ -0,0 +1,61 @@
from __future__ import annotations
import json
from pathlib import Path
from urllib.parse import urlparse
ROOT = Path(__file__).resolve().parents[1]
INVENTORY = ROOT / "docs" / "recovery-operation-inventory.json"
MODES = {
"atomic",
"compensation",
"snapshot_restore",
"forward_recovery",
"irreversible",
}
ADOPTION_STATES = {"planned", "reference-implementation", "adopted"}
REQUIRED_PREFIXES = {
"campaign.",
"files.",
"mail.",
"connectors.",
"dataflow.",
"workflow-engine.",
"core.module-lifecycle.",
"core.module-runtime.",
}
ATOMIC_EXTERNAL_READS = {
"connectors.sync.read-snapshot",
"mail.mailbox.sync-cursor",
}
def test_recovery_operation_inventory_is_complete_and_actionable() -> None:
payload = json.loads(INVENTORY.read_text(encoding="utf-8"))
assert payload["schema_version"] == 1
operations = payload["operations"]
ids = [item["id"] for item in operations]
assert len(ids) == len(set(ids))
assert all(any(item.startswith(prefix) for item in ids) for prefix in REQUIRED_PREFIXES)
for item in operations:
assert item["mode"] in MODES
assert item["adoption"] in ADOPTION_STATES
assert item["repository"].startswith("govoplan-")
assert item["resources"]
assert item["fenced"] is True
issue = urlparse(item["issue"])
assert issue.scheme == "https"
assert issue.netloc == "git.add-ideas.de"
assert issue.path.startswith(f"/GovOPlaN/{item['repository']}/issues/")
def test_non_atomic_operations_do_not_claim_plain_database_rollback() -> None:
operations = json.loads(INVENTORY.read_text(encoding="utf-8"))["operations"]
for item in operations:
if item["mode"] == "atomic":
assert (
item["resources"] == ["postgresql"]
or item["id"] in ATOMIC_EXTERNAL_READS
)
+55
View File
@@ -29,17 +29,72 @@ class ReleaseEntrypointGateTests(unittest.TestCase):
workflow = script[confirm:]
source_gate = workflow.index("run_version_alignment_gate source")
baseline = workflow.index("record_migration_release_baseline")
first_commit = workflow.index('run git -C "$repo" commit')
lock_generation = workflow.index("generate_release_lock")
full_gate = workflow.index("run_version_alignment_gate", source_gate + 1)
first_push = workflow.index('run git -C "$repo" push')
self.assertLess(baseline, source_gate)
self.assertLess(source_gate, first_commit)
self.assertLess(first_commit, lock_generation)
self.assertLess(lock_generation, full_gate)
self.assertLess(full_gate, first_push)
self.assertLess(manifest_gate, confirm)
def test_lockstep_release_pushes_meta_package_after_core(self) -> None:
script = (META_ROOT / "tools" / "release" / "push-release-tag.sh").read_text()
module_push = script.index('for repo in "${MODULE_REPOS[@]}"; do\n run git -C "$repo" push')
core_push = script.index('run git -C "$ROOT" push', module_push)
support_push = script.index('for repo in "${SUPPORT_REPOS[@]}"; do\n run git -C "$repo" push', core_push)
self.assertLess(module_push, core_push)
self.assertLess(core_push, support_push)
def test_default_migration_preflight_accepts_new_release_heads(self) -> None:
script = (META_ROOT / "tools" / "release" / "push-release-tag.sh").read_text()
audit_function = script[
script.index("run_migration_release_audit()") :
script.index("record_migration_release_baseline()")
]
self.assertNotIn("--strict-if-baseline", audit_function)
self.assertIn('command+=("--strict")', audit_function)
def test_source_gate_does_not_require_tags_before_they_are_created(self) -> None:
script = (META_ROOT / "tools" / "release" / "push-release-tag.sh").read_text()
gate = script[
script.index("run_version_alignment_gate()") :
script.index("run_manifest_shape_gate()")
]
self.assertIn('command+=(--source-metadata-only)', gate)
self.assertIn('else\n command+=(--release-composition)', gate)
def test_version_updater_targets_canonical_runtime_declarations(self) -> None:
script = (META_ROOT / "tools" / "release" / "push-release-tag.sh").read_text()
self.assertIn("^manifest\\s*=\\s*ModuleManifest", script)
self.assertIn("could not update module version declaration", script)
self.assertIn("update_package_init_versions", script)
self.assertIn("synchronize-webui-package-metadata.py", script)
self.assertIn('"peerDependenciesMeta",', script)
self.assertLess(
script.index('synchronize-webui-package-metadata.py" --repo "$repo"'),
script.index('synchronize_lockfile_root "$package_path"', script.index('synchronize-webui-package-metadata.py" --repo "$repo"')),
)
self.assertNotIn("could not update ModuleManifest.version", script)
def test_release_lock_refreshes_candidate_govoplan_metadata(self) -> None:
script = (META_ROOT / "tools" / "release" / "generate-release-lock.sh").read_text()
self.assertEqual(2, script.count('"npm_config_cache=$TMP_DIR/npm-cache"'))
self.assertNotIn(
'cp "$WEBUI/package-lock.release.json" "$TMP_DIR/package-lock.json"',
script,
)
self.assertIn('cp "$WEBUI/package.release.json" "$TMP_DIR/package.json"', script)
def test_source_catalog_generator_enforces_explicit_repo_versions(self) -> None:
script = (META_ROOT / "tools" / "release" / "generate-release-catalog.py").read_text()
+30
View File
@@ -340,6 +340,7 @@ def add_scoped_workflow_manifest(repo: Path) -> None:
(backend / "__init__.py").write_text("", encoding="utf-8")
(backend / "manifest.py").write_text(
"""from govoplan_core.core.modules import DocumentationCondition, DocumentationTopic, ModuleManifest, PermissionDefinition
from govoplan_core.core.provider_governance import declared_module_architecture
def get_manifest():
@@ -368,6 +369,20 @@ def get_manifest():
conditions=(DocumentationCondition(required_scopes=("access:item:read",)),),
metadata={"kind": "workflow"},
),
DocumentationTopic(
id="access.admin.reference",
title="Administer access",
summary="Static administrator documentation for the release fixture.",
documentation_types=("admin",),
metadata={"kind": "reference"},
),
),
architecture=declared_module_architecture(
layer="institutional_foundation",
kind="foundation",
maturity="scaffold",
documentation_ref="pyproject.toml",
known_limits=("Release-test fixture only.",),
),
)
""",
@@ -387,6 +402,7 @@ def replace_with_unscoped_workflow_manifest(repo: Path) -> None:
manifest = repo / "src" / "govoplan_access" / "backend" / "manifest.py"
manifest.write_text(
"""from govoplan_core.core.modules import DocumentationTopic, ModuleManifest
from govoplan_core.core.provider_governance import declared_module_architecture
def get_manifest():
@@ -402,6 +418,20 @@ def get_manifest():
documentation_types=("user",),
metadata={"kind": "workflow"},
),
DocumentationTopic(
id="access.admin.reference",
title="Administer access",
summary="Static administrator documentation for the release fixture.",
documentation_types=("admin",),
metadata={"kind": "reference"},
),
),
architecture=declared_module_architecture(
layer="institutional_foundation",
kind="foundation",
maturity="scaffold",
documentation_ref="pyproject.toml",
known_limits=("Release-test fixture only.",),
),
)
""",
+8
View File
@@ -93,6 +93,14 @@ class RuntimeDistributionTests(unittest.TestCase):
with self.assertRaisesRegex(DistributionError, "active trusted key"):
verify_manifest(unknown, self.keyring, now=self.now)
with self.assertRaisesRegex(DistributionError, "expected 'candidate'"):
verify_manifest(
self._manifest(),
self.keyring,
expected_channel="candidate",
now=self.now,
)
def test_offline_image_index_is_complete_and_digest_bound(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-offline-images-") as value:
root = Path(value)
+245
View File
@@ -1,12 +1,17 @@
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import shutil
import sys
import tempfile
import unittest
from unittest.mock import patch
from urllib.error import HTTPError
ROOT = Path(__file__).resolve().parents[1]
@@ -26,9 +31,211 @@ FINALIZE = _load(
"finalize_runtime_distribution",
ROOT / "tools/release/finalize-runtime-distribution.py",
)
DEPLOYER_BUILD = _load(
"build_deployer_zipapp",
ROOT / "tools/deployment/build-deployer-zipapp.py",
)
PUBLISH = _load(
"publish_runtime_release",
ROOT / "tools/release/publish-runtime-release.py",
)
class RuntimeDistributionBuildTests(unittest.TestCase):
def test_deployment_zipapp_is_reproducible_across_source_mtimes(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-reproducible-zipapp-") as value:
root = Path(value)
source = root / "source"
shutil.copytree(ROOT / "tools/deployment", source)
first = root / "first.pyz"
second = root / "second.pyz"
original_root = DEPLOYER_BUILD.ROOT
try:
DEPLOYER_BUILD.ROOT = source
self.assertEqual(0, DEPLOYER_BUILD.main(["--output", str(first)]))
for path in source.rglob("*.py"):
os.utime(path, (2_000_000_000, 2_000_000_000))
self.assertEqual(0, DEPLOYER_BUILD.main(["--output", str(second)]))
finally:
DEPLOYER_BUILD.ROOT = original_root
self.assertEqual(first.read_bytes(), second.read_bytes())
def test_workflow_signs_with_the_release_environment(self) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
self.assertIn(
".runtime-build/bin/python tools/release/generate-runtime-distribution.py",
workflow,
)
self.assertNotIn(
"\n python tools/release/generate-runtime-distribution.py",
workflow,
)
def test_workflow_rejects_missing_or_mutable_image_inputs_before_build(self) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
validation = workflow.index("- name: Validate immutable release inputs")
bootstrap = workflow.index("- name: Bootstrap release sources")
self.assertLess(validation, bootstrap)
self.assertIn('image_pattern = re.compile(r"^[^@\\s]+@sha256:', workflow)
for input_name in (
"python_image",
"nginx_image",
"postgres_image",
"redis_image",
"load_balancer_image",
"managed_ingress_image",
"garage_image",
"test_mail_image",
"binfmt_image",
):
self.assertIn(f"inputs.{input_name}", workflow)
def test_api_runtime_points_core_at_packaged_migration_scripts(self) -> None:
dockerfile = (ROOT / "tools/release/runtime/Dockerfile.api").read_text(
encoding="utf-8"
)
self.assertIn(
"GOVOPLAN_CORE_SOURCE_ROOT=/opt/govoplan/runtime/govoplan_core_runtime",
dockerfile,
)
def test_web_runtime_uses_only_writable_tmpfs_for_nginx_temp_files(self) -> None:
nginx = (ROOT / "tools/release/runtime/nginx.conf").read_text(
encoding="utf-8"
)
for temporary_path in (
"client_body_temp_path /tmp/client_temp;",
"fastcgi_temp_path /tmp/fastcgi_temp;",
"proxy_temp_path /tmp/proxy_temp;",
"scgi_temp_path /tmp/scgi_temp;",
"uwsgi_temp_path /tmp/uwsgi_temp;",
):
self.assertIn(temporary_path, nginx)
def test_workflow_verifies_portable_bootstrap_artifacts_before_execution(
self,
) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
self.assertIn(
"(cd runtime-output && sha256sum govoplan-deploy.pyz > "
"govoplan-deploy.pyz.sha256)",
workflow,
)
self.assertIn("openssl pkeyutl -verify -pubin", workflow)
self.assertIn("govoplan-deploy.tampered.pyz", workflow)
self.assertLess(
workflow.index("openssl pkeyutl -verify -pubin"),
workflow.index("python runtime-output/govoplan-deploy.pyz init"),
)
def test_workflow_retains_both_platform_runtime_smoke_receipts(self) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
self.assertIn('for ARCH in amd64 arm64; do', workflow)
self.assertIn("tools/checks/runtime-image-smoke.py", workflow)
self.assertIn("Resolve managed dependency platform images", workflow)
self.assertIn("--postgres-metadata", workflow)
self.assertIn("--redis-metadata", workflow)
self.assertIn("Register arm64 execution for runtime smoke", workflow)
self.assertIn(
'docker run --privileged --rm "$BINFMT_IMAGE" --install arm64',
workflow,
)
self.assertIn("runtime-smoke-amd64.json", workflow)
self.assertIn("runtime-smoke-arm64.json", workflow)
self.assertIn(
"jq -r '.platforms[\"linux/amd64\"]' runtime-output/api-metadata.json",
workflow,
)
self.assertNotIn(".platforms[\\\"linux/amd64\\\"]", workflow)
def test_workflow_binds_the_release_tag_to_the_workflow_commit(self) -> None:
workflow = (ROOT / ".gitea/workflows/runtime-distribution.yml").read_text(
encoding="utf-8"
)
publisher = (ROOT / "tools/release/publish-runtime-release.py").read_text(
encoding="utf-8"
)
self.assertIn("SOURCE_COMMIT: ${{ gitea.sha }}", workflow)
self.assertIn('--target-commit "$SOURCE_COMMIT"', workflow)
self.assertIn('"target_commitish": target_commit', publisher)
self.assertIn("self._resolve_commit(tag) != target_commit", publisher)
def test_runtime_publisher_rejects_a_tag_on_another_commit(self) -> None:
publisher = PUBLISH.GiteaReleasePublisher(
base_url="https://git.example.test",
owner="GovOPlaN",
repo="govoplan",
token="secret",
)
target = "1" * 40
with (
patch.object(
publisher,
"_resolve_commit",
side_effect=(target, "2" * 40),
),
self.assertRaisesRegex(PUBLISH.PublishError, "another commit"),
):
publisher.release(
tag="v1.2.3",
target_commit=target,
title="Release",
body="Body",
)
def test_runtime_publisher_creates_the_tag_at_the_exact_commit(self) -> None:
publisher = PUBLISH.GiteaReleasePublisher(
base_url="https://git.example.test",
owner="GovOPlaN",
repo="govoplan",
token="secret",
)
target = "1" * 40
requests: list[tuple[str, dict[str, object] | None]] = []
def request(method: str, _url: str, **kwargs):
payload = kwargs.get("payload")
requests.append((method, payload))
if method == "GET":
raise HTTPError(_url, 404, "not found", {}, None)
return {"id": 1}
with (
patch.object(
publisher,
"_resolve_commit",
side_effect=(target, None, target),
),
patch.object(publisher, "_json", side_effect=request),
):
release = publisher.release(
tag="v1.2.3",
target_commit=target,
title="Release",
body="Body",
)
self.assertEqual({"id": 1}, release)
self.assertEqual("POST", requests[-1][0])
assert requests[-1][1] is not None
self.assertEqual(target, requests[-1][1]["target_commitish"])
def test_resolves_platforms_and_builds_evidence_descriptor(self) -> None:
index = {
"schemaVersion": 2,
@@ -52,6 +259,15 @@ class RuntimeDistributionBuildTests(unittest.TestCase):
"registry.example/govoplan/api@sha256:" + "1" * 64,
metadata["platforms"]["linux/amd64"],
)
dependency_metadata = OCI.resolve_platforms(
index,
repository="registry.example:5000/library/postgres:16-alpine",
index_digest="sha256:" + "a" * 64,
)
self.assertEqual(
"registry.example:5000/library/postgres@sha256:" + "2" * 64,
dependency_metadata["platforms"]["linux/arm64"],
)
with tempfile.TemporaryDirectory(prefix="govoplan-runtime-finalize-") as value:
root = Path(value)
@@ -83,12 +299,37 @@ class RuntimeDistributionBuildTests(unittest.TestCase):
(root / "web.json").write_text(json.dumps(web_metadata))
deployer = root / "govoplan-deploy.pyz"
deployer.write_bytes(b"zipapp")
package_lock = root / "package-artifacts.lock.json"
package_lock_value = {
"schema_version": "1",
"release_version": "1.2.3",
"python": [
{
"name": "govoplan-core",
"version": "1.2.3",
"sha256": "8" * 64,
}
],
"webui": [],
}
package_lock_value["lock_sha256"] = hashlib.sha256(
json.dumps(
package_lock_value,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
).hexdigest()
package_lock.write_text(
json.dumps(package_lock_value) + "\n",
encoding="utf-8",
)
args = argparse.Namespace(
composition=root / "composition.json",
api_metadata=root / "api.json",
web_metadata=root / "web.json",
deployer=deployer,
deployer_url="https://downloads.example/govoplan-deploy.pyz",
package_lock=package_lock,
artifact_base_url="https://downloads.example/runtime/v1.2.3",
source_commit="f" * 40,
version="1.2.3",
@@ -112,6 +353,10 @@ class RuntimeDistributionBuildTests(unittest.TestCase):
)
self.assertTrue((root / "evidence/api-sbom.cdx.json").is_file())
self.assertTrue((root / "evidence/web-provenance.json").is_file())
self.assertEqual(
hashlib.sha256(package_lock.read_bytes()).hexdigest(),
descriptor["package_lock"]["sha256"],
)
def test_rejects_incomplete_oci_index(self) -> None:
with self.assertRaisesRegex(ValueError, "linux/amd64 and linux/arm64"):
+86
View File
@@ -0,0 +1,86 @@
from __future__ import annotations
import importlib.util
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
ROOT = Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "tools/checks/runtime-image-smoke.py"
SPEC = importlib.util.spec_from_file_location("runtime_image_smoke", SCRIPT)
assert SPEC is not None and SPEC.loader is not None
MODULE = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = MODULE
SPEC.loader.exec_module(MODULE)
class RuntimeImageSmokeTests(unittest.TestCase):
def test_selects_the_exact_platform_digest(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-runtime-smoke-") as value:
path = Path(value) / "metadata.json"
path.write_text(
json.dumps(
{
"index": "registry.example/api@sha256:" + "a" * 64,
"platforms": {
"linux/amd64": "registry.example/api@sha256:" + "1" * 64,
"linux/arm64": "registry.example/api@sha256:" + "2" * 64,
},
}
),
encoding="utf-8",
)
self.assertEqual(
"registry.example/api@sha256:" + "2" * 64,
MODULE.platform_image(path, "linux/arm64", "API"),
)
def test_rejects_mutable_or_missing_platform_images(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-runtime-smoke-") as value:
path = Path(value) / "metadata.json"
path.write_text(
json.dumps({"platforms": {"linux/amd64": "registry.example/api:latest"}}),
encoding="utf-8",
)
with self.assertRaisesRegex(MODULE.SmokeError, "exact sha256"):
MODULE.platform_image(path, "linux/amd64", "API")
with self.assertRaisesRegex(MODULE.SmokeError, "exact sha256"):
MODULE.platform_image(path, "linux/arm64", "API")
def test_readiness_fails_immediately_when_container_exits(self) -> None:
exited = subprocess.CompletedProcess([], 0, "false\n", "")
logs = subprocess.CompletedProcess(
[], 0, "fatal startup error db-secret\n", ""
)
with patch.object(MODULE, "_run", side_effect=(exited, logs)):
with self.assertRaisesRegex(
MODULE.SmokeError,
r"container exited before readiness: fatal startup error \[redacted\]",
):
MODULE._wait_for(
"WebUI",
lambda: self.fail("probe must not run for an exited container"),
timeout=60,
container="web",
redactions=("db-secret",),
)
def test_smoke_supplies_the_packaged_web_upstream_and_schema_contract(self) -> None:
source = SCRIPT.read_text(encoding="utf-8")
self.assertIn('"--network-alias",\n "load-balancer"', source)
self.assertIn("'core_system_settings'", source)
self.assertIn("'core_runtime_nodes'", source)
self.assertIn('if platform == "linux/arm64"', source)
self.assertIn('"ARM64-COW-BUG"', source)
if __name__ == "__main__":
unittest.main()
+71
View File
@@ -0,0 +1,71 @@
from __future__ import annotations
import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest
META_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = META_ROOT / "tools" / "release" / "synchronize-webui-package-metadata.py"
class SynchronizeWebuiPackageMetadataTests(unittest.TestCase):
def test_copies_peer_contract_without_changing_publish_paths(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repo = Path(directory)
(repo / "webui").mkdir()
(repo / "package.json").write_text(
json.dumps(
{
"name": "@govoplan/example-webui",
"exports": {".": "./webui/src/index.ts"},
"peerDependencies": {"vite": "^6"},
}
)
)
(repo / "webui" / "package.json").write_text(
json.dumps(
{
"name": "@govoplan/example-webui",
"peerDependencies": {"vite": "^7"},
"peerDependenciesMeta": {"vite": {"optional": True}},
}
)
)
subprocess.run(
[sys.executable, str(SCRIPT), "--repo", str(repo)],
check=True,
capture_output=True,
text=True,
)
package = json.loads((repo / "package.json").read_text())
self.assertEqual({"vite": "^7"}, package["peerDependencies"])
self.assertEqual({"vite": {"optional": True}}, package["peerDependenciesMeta"])
self.assertEqual({".": "./webui/src/index.ts"}, package["exports"])
def test_leaves_distinct_root_and_webui_packages_separate(self) -> None:
with tempfile.TemporaryDirectory() as directory:
repo = Path(directory)
(repo / "webui").mkdir()
(repo / "package.json").write_text(json.dumps({"name": "@govoplan/one"}))
(repo / "webui" / "package.json").write_text(json.dumps({"name": "@govoplan/two"}))
subprocess.run(
[sys.executable, str(SCRIPT), "--repo", str(repo)],
check=True,
capture_output=True,
text=True,
)
root = json.loads((repo / "package.json").read_text())
self.assertEqual("@govoplan/one", root["name"])
self.assertNotIn("peerDependencies", root)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,194 @@
#!/usr/bin/env python3
"""Generate an independently held Ed25519 assessment-authority keypair."""
from __future__ import annotations
import argparse
import base64
from datetime import UTC, datetime, timedelta
import json
import os
from pathlib import Path
import re
import stat
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
KEY_ID_PATTERN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$")
PROOF_SCOPES = (
"target_environment",
"external_providers",
"accessibility",
"privacy",
"security",
"operations",
"recovery",
"production_approval",
)
PURPOSES = {
"proof": (
"govoplan.capability-fit-proof-authorities",
"./capability-fit-proof-authority-keyring.schema.json",
),
"installer": (
"govoplan.installer-receipt-authorities",
"./installer-receipt-authority-keyring.schema.json",
),
}
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--purpose", choices=tuple(PURPOSES), required=True)
parser.add_argument("--key-id", required=True)
parser.add_argument(
"--scope",
action="append",
choices=PROOF_SCOPES,
default=[],
help="Authorized proof scope; repeat as needed. Not used for installer keys.",
)
parser.add_argument("--private-key", type=Path, required=True)
parser.add_argument("--keyring", type=Path, required=True)
parser.add_argument(
"--valid-days",
type=int,
default=365,
help="Validity from generation time (default: 365 days).",
)
parser.add_argument(
"--status",
choices=("active", "next"),
default="active",
)
args = parser.parse_args(argv)
if not KEY_ID_PATTERN.fullmatch(args.key_id):
parser.error("--key-id must be a valid opaque identifier")
if args.valid_days < 1 or args.valid_days > 3660:
parser.error("--valid-days must be between 1 and 3660")
scopes = _resolve_scopes(parser, purpose=args.purpose, scopes=args.scope)
private_path = args.private_key.expanduser().resolve()
keyring_path = args.keyring.expanduser().resolve()
_require_fresh_output(parser, private_path, label="private key")
_require_fresh_output(parser, keyring_path, label="keyring")
_require_private_directory(parser, private_path.parent)
_require_output_directory(parser, keyring_path.parent)
private_key = Ed25519PrivateKey.generate()
private_bytes = private_key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.PKCS8,
encryption_algorithm=serialization.NoEncryption(),
)
public_bytes = private_key.public_key().public_bytes(
encoding=serialization.Encoding.Raw,
format=serialization.PublicFormat.Raw,
)
public_base64 = base64.b64encode(public_bytes).decode("ascii")
now = datetime.now(UTC).replace(microsecond=0)
not_after = now + timedelta(days=args.valid_days)
purpose, schema = PURPOSES[args.purpose]
keyring = {
"$schema": schema,
"schema_version": "0.1.0",
"purpose": purpose,
"keys": [
{
"key_id": args.key_id,
"status": args.status,
"public_key": public_base64,
"allowed_scopes": scopes,
"not_before": _rfc3339(now),
"not_after": _rfc3339(not_after),
}
],
}
_write_new_private_file(private_path, private_bytes)
try:
_write_new_private_file(
keyring_path,
(json.dumps(keyring, indent=2, sort_keys=True) + "\n").encode("utf-8"),
)
except BaseException:
private_path.unlink(missing_ok=True)
keyring_path.unlink(missing_ok=True)
raise
print(f"private_key={private_path}")
print(f"keyring={keyring_path}")
print(f"key_id={args.key_id}")
print(f"allowed_scopes={','.join(scopes)}")
return 0
def _resolve_scopes(
parser: argparse.ArgumentParser, *, purpose: str, scopes: list[str]
) -> list[str]:
if purpose == "installer":
if scopes:
parser.error("installer authorities do not accept --scope")
return ["installed_release_origin"]
unique = list(dict.fromkeys(scopes))
if not unique:
parser.error("proof authorities require at least one --scope")
return unique
def _require_fresh_output(
parser: argparse.ArgumentParser, path: Path, *, label: str
) -> None:
if path.exists() or path.is_symlink():
parser.error(f"{label.capitalize()} output already exists: {path}")
def _require_private_directory(
parser: argparse.ArgumentParser, directory: Path
) -> None:
_require_output_directory(parser, directory)
mode = stat.S_IMODE(directory.stat().st_mode)
if mode & (stat.S_IRWXG | stat.S_IRWXO):
parser.error(
"Private-key parent directory must not be accessible by group or others"
)
def _require_output_directory(
parser: argparse.ArgumentParser, directory: Path
) -> None:
try:
metadata = directory.lstat()
except OSError as exc:
parser.error(f"Output parent directory is unavailable: {directory}")
raise AssertionError from exc
if stat.S_ISLNK(metadata.st_mode) or not stat.S_ISDIR(metadata.st_mode):
parser.error(f"Output parent must be a real directory: {directory}")
def _write_new_private_file(path: Path, payload: bytes) -> None:
flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL
if hasattr(os, "O_NOFOLLOW"):
flags |= os.O_NOFOLLOW
descriptor = os.open(path, flags, 0o600)
try:
with os.fdopen(descriptor, "wb", closefd=False) as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
metadata = os.fstat(descriptor)
if not stat.S_ISREG(metadata.st_mode) or stat.S_IMODE(metadata.st_mode) != 0o600:
raise OSError("Authority output could not be secured")
finally:
os.close(descriptor)
def _rfc3339(value: datetime) -> str:
return value.isoformat().replace("+00:00", "Z")
if __name__ == "__main__":
raise SystemExit(main())
+176 -139
View File
@@ -15,6 +15,7 @@ from govoplan_core.core.dataflows import (
dataflow_run_lifecycle,
)
from govoplan_core.core.automation import AutomationPrincipalResolution
from govoplan_core.core.change_sequence import ChangeSequenceEntry
from govoplan_core.core.access import (
CAPABILITY_AUTH_AUTOMATION_PRINCIPAL_PROVIDER,
)
@@ -25,6 +26,12 @@ from govoplan_core.core.datasources import (
datasource_publication,
)
from govoplan_core.core.modules import ModuleContext
from govoplan_core.core.recovery import RecoveryCheckpoint, RecoveryOperation
from govoplan_core.core.runtime_coordination import (
DistributedLease,
RuntimeIdentity,
bind_process_runtime_identity,
)
from govoplan_core.core.tabular_sources import (
TabularSnapshotInput,
tabular_snapshot_writer,
@@ -73,6 +80,10 @@ def main() -> int:
Base.metadata.create_all(
engine,
tables=[
ChangeSequenceEntry.__table__,
DistributedLease.__table__,
RecoveryOperation.__table__,
RecoveryCheckpoint.__table__,
ConnectorTabularSource.__table__,
DatasourceRecord.__table__,
DatasourcePayloadRecord.__table__,
@@ -86,153 +97,168 @@ def main() -> int:
],
)
session_factory = sessionmaker(bind=engine)
with session_factory() as session:
principal = _principal()
writer = tabular_snapshot_writer(registry)
lifecycle = datasource_lifecycle(registry)
catalogue = datasource_catalogue(registry)
publisher = datasource_publication(registry)
runner = dataflow_run_lifecycle(registry)
if (
writer is None
or lifecycle is None
or catalogue is None
or publisher is None
or runner is None
):
raise RuntimeError("Datasource composition capabilities are incomplete.")
bind_process_runtime_identity(_runtime_identity())
try:
with session_factory() as session:
principal = _principal()
writer = tabular_snapshot_writer(registry)
lifecycle = datasource_lifecycle(registry)
catalogue = datasource_catalogue(registry)
publisher = datasource_publication(registry)
runner = dataflow_run_lifecycle(registry)
if (
writer is None
or lifecycle is None
or catalogue is None
or publisher is None
or runner is None
):
raise RuntimeError(
"Datasource composition capabilities are incomplete."
)
origin = writer.create_snapshot(
session,
principal,
snapshot=TabularSnapshotInput(
name="Monthly cases",
source_name="connector_monthly_cases",
rows=(
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
origin = writer.create_snapshot(
session,
principal,
snapshot=TabularSnapshotInput(
name="Monthly cases",
source_name="connector_monthly_cases",
rows=(
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
),
),
),
)
datasource = lifecycle.register_origin(
session,
principal,
origin_ref=origin.ref,
name="Monthly cases cache",
source_name="monthly_cases",
mode="cached",
)
result = preview_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelinePreviewRequest(
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
)
datasource = lifecycle.register_origin(
session,
principal,
origin_ref=origin.ref,
name="Monthly cases cache",
source_name="monthly_cases",
mode="cached",
)
result = preview_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelinePreviewRequest(
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
),
row_limit=100,
),
row_limit=100,
),
principal=principal,
registry=registry,
)
expected_rows = [
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
]
if result.status != "succeeded":
raise RuntimeError(f"Dataflow preview failed: {result.diagnostics}")
if result.rows != expected_rows:
raise RuntimeError(f"Unexpected Dataflow rows: {result.rows!r}")
if result.source_fingerprints[0]["source_ref"] != datasource.ref:
raise RuntimeError("Dataflow lineage did not retain the datasource reference.")
pipeline = create_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelineCreateRequest(
name="Monthly case output",
status="active",
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
principal=principal,
registry=registry,
)
expected_rows = [
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
]
if result.status != "succeeded":
raise RuntimeError(
f"Dataflow preview failed: {result.diagnostics}"
)
if result.rows != expected_rows:
raise RuntimeError(f"Unexpected Dataflow rows: {result.rows!r}")
if result.source_fingerprints[0]["source_ref"] != datasource.ref:
raise RuntimeError(
"Dataflow lineage did not retain the datasource reference."
)
pipeline = create_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelineCreateRequest(
name="Monthly case output",
status="active",
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
),
editor_mode="graph",
),
editor_mode="graph",
),
)
run_request = DataflowRunRequest(
pipeline_ref=f"pipeline:{pipeline.id}",
revision=1,
idempotency_key="composition-run-1",
publication=DataflowPublicationTarget(
name="Monthly case result",
source_name="monthly_case_result",
freeze=True,
frozen_label="Composition evidence",
),
)
published = runner.start_run(
session,
principal,
request=run_request,
)
replayed = runner.start_run(
session,
principal,
request=run_request,
)
if published.status != "queued":
raise RuntimeError(
f"Dataflow run was not queued: {published.status}"
)
worker = SqlDataflowRunWorker(
registry=_AutomationRegistry(registry, principal)
)
worker_result = worker.dispatch_pending(
session,
worker_id="composition-worker",
)
if worker_result["succeeded"] != 1:
raise RuntimeError(
f"Dataflow worker failed: {worker_result!r}"
run_request = DataflowRunRequest(
pipeline_ref=f"pipeline:{pipeline.id}",
revision=1,
idempotency_key="composition-run-1",
publication=DataflowPublicationTarget(
name="Monthly case result",
source_name="monthly_case_result",
freeze=True,
frozen_label="Composition evidence",
),
)
completed = runner.get_run(
session,
principal,
run_ref=published.ref,
)
if completed is None:
raise RuntimeError("Dataflow run evidence disappeared.")
published = completed
if published.status != "succeeded":
raise RuntimeError(f"Dataflow publication failed: {published.error}")
if replayed.ref != published.ref or not replayed.replayed:
raise RuntimeError("Dataflow run idempotency did not replay the prior run.")
if (
not published.output_datasource_ref
or not published.output_materialization_ref
):
raise RuntimeError("Dataflow publication did not retain output references.")
output = catalogue.read_datasource(
session,
principal,
request=DatasourceReadRequest(
datasource_ref=published.output_datasource_ref,
),
)
if list(output.rows) != expected_rows:
raise RuntimeError(
f"Unexpected published Dataflow rows: {list(output.rows)!r}"
published = runner.start_run(
session,
principal,
request=run_request,
)
if (
output.materialization is None
or output.materialization.ref != published.output_materialization_ref
or output.materialization.frozen_at is None
):
raise RuntimeError(
"Published Datasource materialization is not pinned and frozen."
replayed = runner.start_run(
session,
principal,
request=run_request,
)
engine.dispose()
if published.status != "queued":
raise RuntimeError(
f"Dataflow run was not queued: {published.status}"
)
worker = SqlDataflowRunWorker(
registry=_AutomationRegistry(registry, principal)
)
worker_result = worker.dispatch_pending(
session,
worker_id="composition-worker",
)
if worker_result["succeeded"] != 1:
raise RuntimeError(f"Dataflow worker failed: {worker_result!r}")
completed = runner.get_run(
session,
principal,
run_ref=published.ref,
)
if completed is None:
raise RuntimeError("Dataflow run evidence disappeared.")
published = completed
if published.status != "succeeded":
raise RuntimeError(
f"Dataflow publication failed: {published.error}"
)
if replayed.ref != published.ref or not replayed.replayed:
raise RuntimeError(
"Dataflow run idempotency did not replay the prior run."
)
if (
not published.output_datasource_ref
or not published.output_materialization_ref
):
raise RuntimeError(
"Dataflow publication did not retain output references."
)
output = catalogue.read_datasource(
session,
principal,
request=DatasourceReadRequest(
datasource_ref=published.output_datasource_ref,
),
)
if list(output.rows) != expected_rows:
raise RuntimeError(
f"Unexpected published Dataflow rows: {list(output.rows)!r}"
)
if (
output.materialization is None
or output.materialization.ref
!= published.output_materialization_ref
or output.materialization.frozen_at is None
):
raise RuntimeError(
"Published Datasource materialization is not pinned and frozen."
)
finally:
bind_process_runtime_identity(None)
engine.dispose()
print(
"Connector -> Datasources -> pinned Dataflow publication composition passed."
)
@@ -252,6 +278,17 @@ class _AutomationProvider:
)
def _runtime_identity() -> RuntimeIdentity:
return RuntimeIdentity(
installation_id="datasource-composition-check",
node_id="composition-worker",
incarnation="composition-worker-incarnation",
role="worker",
software_version="test",
composition_hash="c" * 64,
)
class _AutomationRegistry:
def __init__(self, registry, principal: ApiPrincipal) -> None:
self.registry = registry
+4
View File
@@ -40,6 +40,10 @@ GOVOPLAN_CORE_ROOT="$ROOT" PYTHON="$PYTHON" CHECK_TESTCLIENT_DEPRECATIONS=1 bash
PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-manifest-shapes.py" --require-architecture
cd "$META_ROOT"
"$PYTHON" tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints
"$PYTHON" tools/repo/sync-module-package-workflows.py --check
"$PYTHON" tools/release/generate-developer-meta-package.py --check
"$PYTHON" -m unittest tests.test_module_package_workflows tests.test_package_registry_release
"$PYTHON" -m unittest tests.test_deployment_installer
"$PYTHON" -m unittest tests.test_capability_fit_evidence
"$PYTHON" -m unittest tests.test_configuration_package_artifacts
@@ -182,6 +182,11 @@ run_step "Validate installed module manifests and registry"
"$PYTHON" "$META_ROOT/tools/checks/release_integration.py" artifacts \
--requirements "$META_ROOT/requirements-release.txt"
run_step "Validate platform interface and endpoint declarations"
"$PYTHON" "$META_ROOT/tools/inventory/platform-interface-inventory.py" \
--strict-declarations \
--strict-endpoints
run_step "Generate release dependency provenance"
"$PYTHON" "$META_ROOT/tools/release/generate-release-sbom.py" \
--python "$PYTHON" \
+270 -50
View File
@@ -4,13 +4,14 @@
from __future__ import annotations
import argparse
import json
from pathlib import Path
import re
import shutil
import socket
import subprocess
import sys
import tempfile
import time
from uuid import uuid4
@@ -27,57 +28,211 @@ from govoplan_deploy.model import default_spec # noqa: E402
DIGEST_IMAGE = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$")
def _run(argv: list[str], *, check: bool = True) -> subprocess.CompletedProcess[str]:
return subprocess.run(
argv,
check=check,
capture_output=True,
text=True,
timeout=60,
def _run(
argv: list[str],
*,
check: bool = True,
input_text: str | None = None,
) -> subprocess.CompletedProcess[str]:
try:
return subprocess.run(
argv,
check=check,
capture_output=True,
input=input_text,
text=True,
timeout=60,
)
except subprocess.CalledProcessError as exc:
stderr = exc.stderr.strip()
if stderr:
print(stderr, file=sys.stderr)
raise
def _write_volume_file(
*,
image: str,
volume: str,
filename: str,
content: str,
) -> None:
if not re.fullmatch(r"[A-Za-z0-9_.-]+", filename):
raise ValueError(f"invalid config filename: {filename!r}")
_run(
[
"docker",
"run",
"--rm",
"--interactive",
"--user",
"0:0",
"--mount",
f"type=volume,src={volume},dst=/govoplan-config",
"--entrypoint",
"sh",
image,
"-c",
f"umask 022; cat > /govoplan-config/{filename}",
],
input_text=content,
)
def _published_port(container: str, target: int) -> int:
output = _run(["docker", "port", container, f"{target}/tcp"]).stdout.strip()
output = _run(
[
"docker",
"inspect",
"--format",
"{{json .HostConfig.PortBindings}}",
container,
]
).stdout.strip()
try:
return int(output.rsplit(":", 1)[1])
except (IndexError, ValueError) as exc:
raise RuntimeError(f"cannot determine published port from {output!r}") from exc
bindings = json.loads(output)[f"{target}/tcp"]
if not isinstance(bindings, list) or len(bindings) != 1:
raise ValueError("expected exactly one published binding")
binding = bindings[0]
if binding.get("HostIp") != "127.0.0.1":
raise ValueError("published binding is not loopback-only")
return int(binding["HostPort"])
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
raise RuntimeError(
f"cannot determine loopback binding for {target}/tcp from {output!r}"
) from exc
def _curl(url: str, *, headers: bool = False) -> str:
argv = ["curl", "--silent", "--show-error", "--insecure"]
if headers:
argv.extend(["--head"])
argv.append(url)
return _run(argv).stdout
def _available_loopback_port(*, exclude: frozenset[int] = frozenset()) -> int:
for _attempt in range(10):
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as listener:
listener.bind(("127.0.0.1", 0))
port = int(listener.getsockname()[1])
if port not in exclude:
return port
raise RuntimeError("cannot allocate distinct loopback ports for ingress drill")
def _wait_for_https(port: int) -> str:
deadline = time.monotonic() + 30
last_error = ""
while time.monotonic() < deadline:
try:
return _curl(f"https://localhost:{port}/health")
except subprocess.CalledProcessError as exc:
last_error = exc.stderr.strip()
time.sleep(0.5)
raise RuntimeError(f"managed ingress did not become ready: {last_error}")
def _probe_ingress(*, image: str, network: str, container: str) -> None:
probe = r'''
import socket
import ssl
import time
def request(port, payload, *, tls):
connection = socket.create_connection(("ingress", port), timeout=3)
if tls:
connection = ssl._create_unverified_context().wrap_socket(
connection, server_hostname="localhost"
)
with connection:
connection.sendall(payload)
chunks = []
while True:
chunk = connection.recv(65536)
if not chunk:
break
chunks.append(chunk)
return b"".join(chunks)
deadline = time.monotonic() + 30
last_error = ""
while time.monotonic() < deadline:
try:
response = request(
8443,
b"GET /health HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n",
tls=True,
)
head, body_bytes = response.split(b"\r\n\r\n", 1)
status = int(head.split(b" ", 2)[1])
if status != 200:
raise RuntimeError(f"HTTPS returned {status}, expected 200")
body = body_bytes.decode("utf-8").strip()
if body != "proto=https":
raise RuntimeError(f"forwarded protocol was not normalized: {body!r}")
break
except Exception as exc:
last_error = f"{type(exc).__name__}: {exc}"
time.sleep(0.5)
else:
raise SystemExit(f"managed ingress did not become ready: {last_error}")
response = request(
8080,
b"HEAD /health HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n",
tls=False,
)
head = response.split(b"\r\n\r\n", 1)[0].decode("iso-8859-1")
lines = head.split("\r\n")
status = int(lines[0].split(" ", 2)[1])
if status != 308:
raise SystemExit(f"HTTP returned {status}, expected redirect 308")
headers = {
key.lower(): value.strip()
for key, separator, value in (line.partition(":") for line in lines[1:])
if separator
}
location = headers.get("location", "")
if not location.startswith("https://localhost"):
raise SystemExit(f"HTTP redirect had unexpected location: {location!r}")
'''
try:
_run(
[
"docker",
"run",
"--rm",
"--network",
network,
"--read-only",
"--security-opt",
"no-new-privileges",
"--cap-drop",
"ALL",
"--entrypoint",
"python",
image,
"-c",
probe,
]
)
except subprocess.CalledProcessError:
_print_container_diagnostics(container)
raise
def _print_container_diagnostics(container: str) -> None:
state = _run(
["docker", "inspect", "--format", "{{json .State}}", container],
check=False,
)
state_detail = (state.stdout + state.stderr).strip()
if state_detail:
print(f"managed ingress state:\n{state_detail}", file=sys.stderr)
logs = _run(["docker", "logs", container], check=False)
log_detail = (logs.stdout + logs.stderr).strip()
if log_detail:
print(f"managed ingress logs:\n{log_detail}", file=sys.stderr)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--caddy-image", required=True)
parser.add_argument("--load-balancer-image", required=True)
parser.add_argument("--probe-image", required=True)
args = parser.parse_args()
for label, image in (
("--caddy-image", args.caddy_image),
("--load-balancer-image", args.load_balancer_image),
("--probe-image", args.probe_image),
):
if DIGEST_IMAGE.fullmatch(image) is None:
parser.error(f"{label} must be pinned by sha256 digest")
if shutil.which("docker") is None or shutil.which("curl") is None:
parser.error("docker and curl are required")
if shutil.which("docker") is None:
parser.error("docker is required")
suffix = uuid4().hex[:10]
network = f"govoplan-ingress-drill-{suffix}"
@@ -85,15 +240,33 @@ def main() -> int:
backend = f"govoplan-ingress-backend-{suffix}"
data_volume = f"govoplan-ingress-data-{suffix}"
config_volume = f"govoplan-ingress-config-{suffix}"
backend_config_volume = f"govoplan-ingress-backend-config-{suffix}"
ingress_config_volume = f"govoplan-ingress-caddy-config-{suffix}"
load_balancer_config_volume = f"govoplan-ingress-haproxy-config-{suffix}"
cleanup = [
["docker", "rm", "--force", ingress, backend],
["docker", "network", "rm", network],
["docker", "volume", "rm", data_volume, config_volume],
[
"docker",
"volume",
"rm",
data_volume,
config_volume,
backend_config_volume,
ingress_config_volume,
load_balancer_config_volume,
],
]
try:
_run(["docker", "network", "create", network])
_run(["docker", "volume", "create", data_volume])
_run(["docker", "volume", "create", config_volume])
for volume in (
data_volume,
config_volume,
backend_config_volume,
ingress_config_volume,
load_balancer_config_volume,
):
_run(["docker", "volume", "create", volume])
with tempfile.TemporaryDirectory(prefix="govoplan-ingress-") as directory:
root = Path(directory)
backend_config = root / "backend.Caddyfile"
@@ -127,6 +300,24 @@ def main() -> int:
encoding="utf-8",
)
load_balancer_config.chmod(0o644)
_write_volume_file(
image=args.load_balancer_image,
volume=load_balancer_config_volume,
filename="haproxy.cfg",
content=load_balancer_config.read_text(encoding="utf-8"),
)
_write_volume_file(
image=args.caddy_image,
volume=backend_config_volume,
filename="Caddyfile",
content=backend_config.read_text(encoding="utf-8"),
)
_write_volume_file(
image=args.caddy_image,
volume=ingress_config_volume,
filename="Caddyfile",
content=ingress_config.read_text(encoding="utf-8"),
)
_run(
[
"docker",
@@ -136,7 +327,11 @@ def main() -> int:
"--cap-drop",
"ALL",
"--mount",
f"type=bind,src={load_balancer_config},dst=/usr/local/etc/haproxy/haproxy.cfg,readonly",
(
"type=volume,"
f"src={load_balancer_config_volume},"
"dst=/usr/local/etc/haproxy,readonly"
),
args.load_balancer_image,
"haproxy",
"-c",
@@ -154,18 +349,28 @@ def main() -> int:
backend,
"--network",
network,
"--network-alias",
"load-balancer",
"--read-only",
"--tmpfs",
"/tmp:rw,noexec,nosuid,size=16m",
"--mount",
f"type=bind,src={backend_config},dst=/etc/caddy/Caddyfile,readonly",
(
"type=volume,"
f"src={backend_config_volume},"
"dst=/govoplan-config,readonly"
),
args.caddy_image,
"caddy",
"run",
"--config",
"/etc/caddy/Caddyfile",
"/govoplan-config/Caddyfile",
]
)
requested_http_port = _available_loopback_port()
requested_https_port = _available_loopback_port(
exclude=frozenset({requested_http_port})
)
ingress_command = [
"docker",
"run",
@@ -174,6 +379,8 @@ def main() -> int:
ingress,
"--network",
network,
"--network-alias",
"ingress",
"--read-only",
"--tmpfs",
"/tmp:rw,noexec,nosuid,size=16m",
@@ -181,12 +388,18 @@ def main() -> int:
"no-new-privileges",
"--cap-drop",
"ALL",
"--cap-add",
"NET_BIND_SERVICE",
"--publish",
"127.0.0.1::8080",
f"127.0.0.1:{requested_http_port}:8080/tcp",
"--publish",
"127.0.0.1::8443",
f"127.0.0.1:{requested_https_port}:8443/tcp",
"--mount",
f"type=bind,src={ingress_config},dst=/etc/caddy/Caddyfile,readonly",
(
"type=volume,"
f"src={ingress_config_volume},"
"dst=/govoplan-config,readonly"
),
"--mount",
f"type=volume,src={data_volume},dst=/data",
"--mount",
@@ -195,25 +408,32 @@ def main() -> int:
"caddy",
"run",
"--config",
"/etc/caddy/Caddyfile",
"/govoplan-config/Caddyfile",
]
_run(ingress_command)
http_port = _published_port(ingress, 8080)
https_port = _published_port(ingress, 8443)
body = _wait_for_https(https_port)
if body.strip() != "proto=https":
raise RuntimeError(f"forwarded protocol was not normalized: {body!r}")
redirect = _curl(f"http://localhost:{http_port}/health", headers=True)
if not redirect.startswith("HTTP/1.1 308"):
raise RuntimeError(f"HTTP was not redirected to HTTPS: {redirect!r}")
if (http_port, https_port) != (
requested_http_port,
requested_https_port,
):
raise RuntimeError("Docker published unexpected ingress ports")
_probe_ingress(
image=args.probe_image,
network=network,
container=ingress,
)
_run(["docker", "rm", "--force", ingress])
_run(ingress_command)
https_port = _published_port(ingress, 8443)
if _wait_for_https(https_port).strip() != "proto=https":
raise RuntimeError(
"managed ingress did not recover with persistent state"
)
if https_port != requested_https_port:
raise RuntimeError("Docker changed the ingress TLS binding on restart")
_probe_ingress(
image=args.probe_image,
network=network,
container=ingress,
)
_run(
[
"docker",
+656
View File
@@ -0,0 +1,656 @@
#!/usr/bin/env python3
"""Exercise a pinned GovOPlaN runtime image pair on one OCI platform."""
from __future__ import annotations
import argparse
import base64
from datetime import UTC, datetime
import json
import os
from pathlib import Path
import re
import secrets
import subprocess
import time
from typing import Callable, Sequence
PLATFORMS = frozenset({"linux/amd64", "linux/arm64"})
DIGEST_IMAGE = re.compile(r"^[^\s@]+@sha256:[0-9a-f]{64}$")
BASE_MODULES = (
"tenancy",
"organizations",
"identity",
"idm",
"access",
"admin",
"dashboard",
"policy",
"audit",
"docs",
"ops",
)
class SmokeError(RuntimeError):
"""A runtime image failed its bounded acceptance drill."""
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--api-metadata", type=Path, required=True)
parser.add_argument("--web-metadata", type=Path, required=True)
parser.add_argument("--postgres-metadata", type=Path, required=True)
parser.add_argument("--redis-metadata", type=Path, required=True)
parser.add_argument("--platform", choices=sorted(PLATFORMS), required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--timeout-seconds", type=float, default=600.0)
return parser
def _utc_now() -> str:
return datetime.now(UTC).isoformat().replace("+00:00", "Z")
def _digest_image(value: object, label: str) -> str:
if not isinstance(value, str) or DIGEST_IMAGE.fullmatch(value) is None:
raise SmokeError(f"{label} must be an exact sha256 image reference")
return value
def platform_image(path: Path, platform: str, label: str) -> str:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
raise SmokeError(f"cannot read {label} OCI metadata") from exc
if not isinstance(payload, dict) or not isinstance(payload.get("platforms"), dict):
raise SmokeError(f"{label} OCI metadata has no platform map")
return _digest_image(payload["platforms"].get(platform), f"{label} {platform}")
def _tail(value: str, *, limit: int = 4000) -> str:
return value[-limit:].strip()
def _run(
arguments: Sequence[str],
*,
check: bool = True,
timeout: float = 600.0,
redactions: Sequence[str] = (),
) -> subprocess.CompletedProcess[str]:
try:
result = subprocess.run(
list(arguments),
check=False,
capture_output=True,
text=True,
timeout=timeout,
)
except (OSError, subprocess.TimeoutExpired) as exc:
raise SmokeError(f"container command could not complete: {type(exc).__name__}") from exc
if check and result.returncode != 0:
detail = _tail(result.stderr or result.stdout or "no diagnostic output")
for secret in redactions:
if secret:
detail = detail.replace(secret, "[redacted]")
raise SmokeError(f"container command failed: {detail}")
return result
def _wait_for(
label: str,
probe: Callable[[], subprocess.CompletedProcess[str]],
*,
timeout: float,
container: str | None = None,
redactions: Sequence[str] = (),
) -> None:
deadline = time.monotonic() + timeout
last = ""
while time.monotonic() < deadline:
if container is not None:
state = _run(
("docker", "inspect", "--format", "{{.State.Running}}", container),
check=False,
timeout=30,
)
if state.returncode != 0 or state.stdout.strip() != "true":
logs = _run(
("docker", "logs", "--tail", "100", container),
check=False,
timeout=30,
)
detail = _tail(logs.stdout + logs.stderr, limit=8000)
for secret in redactions:
if secret:
detail = detail.replace(secret, "[redacted]")
raise SmokeError(
f"{label} container exited before readiness: "
f"{detail or 'no diagnostic output'}"
)
result = probe()
if result.returncode == 0:
return
last = _tail(result.stderr or result.stdout)
time.sleep(2.0)
raise SmokeError(f"{label} did not become ready: {last or 'probe failed'}")
def _environment(
*,
database_password: str,
master_key: str,
platform_slug: str,
) -> dict[str, str]:
database = (
f"postgresql+psycopg://govoplan:{database_password}@postgres:5432/govoplan"
)
return {
"APP_ENV": "dev",
"GOVOPLAN_INSTALL_PROFILE": "evaluation",
"GOVOPLAN_INSTALLATION_ID": f"runtime-smoke-{platform_slug}",
"GOVOPLAN_STATE_PROFILE": "host-shared",
"GOVOPLAN_RUNTIME_HEARTBEAT_SECONDS": "5",
"GOVOPLAN_RUNTIME_STALE_AFTER_SECONDS": "30",
"GOVOPLAN_EXPECTED_API_REPLICAS": "1",
"GOVOPLAN_EXPECTED_WORKER_REPLICAS": "1",
"DATABASE_URL": database,
"GOVOPLAN_DATABASE_URL_PGTOOLS": (
f"postgresql://govoplan:{database_password}@postgres:5432/govoplan"
),
"GOVOPLAN_DB_CONNECTION_LIMIT": "100",
"GOVOPLAN_DB_CONNECTION_RESERVE": "10",
"REDIS_URL": "redis://redis:6379/0",
"CELERY_ENABLED": "true",
"CELERY_QUEUES": "default",
"CELERY_WORKER_CONCURRENCY": "1",
"ENABLED_MODULES": ",".join(BASE_MODULES),
"GOVOPLAN_MIGRATION_TRACK": "release",
"DEV_AUTO_MIGRATE_ENABLED": "false",
"DEV_BOOTSTRAP_ENABLED": "false",
"AUTH_LOGIN_THROTTLE_ENABLED": "true",
"AUTH_COOKIE_SECURE": "false",
"CORS_ORIGINS": "http://localhost",
"GOVOPLAN_TRUSTED_HOSTS": "127.0.0.1,localhost,api",
"FORWARDED_ALLOW_IPS": "127.0.0.1",
"MASTER_KEY_B64": master_key,
"FILE_STORAGE_BACKEND": "local",
"FILE_STORAGE_LOCAL_ROOT": "/var/lib/govoplan/files",
"GOVOPLAN_MODULE_LIVE_APPLY_ENABLED": "false",
}
def _env_arguments(values: dict[str, str], *, role: str, node_id: str) -> list[str]:
arguments: list[str] = []
for key, value in sorted(
{**values, "GOVOPLAN_RUNTIME_ROLE": role, "GOVOPLAN_NODE_ID": node_id}.items()
):
arguments.extend(("--env", f"{key}={value}"))
return arguments
def run_smoke(
*,
api_image: str,
web_image: str,
postgres_image: str,
redis_image: str,
platform: str,
timeout: float,
) -> dict[str, object]:
for label, value in (
("API image", api_image),
("Web image", web_image),
("PostgreSQL image", postgres_image),
("Redis image", redis_image),
):
_digest_image(value, label)
if platform not in PLATFORMS:
raise SmokeError(f"unsupported runtime smoke platform: {platform}")
slug = platform.replace("linux/", "").replace("/", "-")
suffix = secrets.token_hex(4)
prefix = f"govoplan-runtime-{slug}-{suffix}"
names = {
"network": f"{prefix}-network",
"volume": f"{prefix}-data",
"postgres": f"{prefix}-postgres",
"redis": f"{prefix}-redis",
"api": f"{prefix}-api",
"web": f"{prefix}-web",
"worker": f"{prefix}-worker",
}
database_password = secrets.token_hex(20)
master_key = base64.urlsafe_b64encode(os.urandom(32)).decode("ascii")
redactions = (database_password, master_key)
environment = _environment(
database_password=database_password,
master_key=master_key,
platform_slug=slug,
)
checks: list[dict[str, object]] = []
started = time.monotonic()
def record(check_id: str, began: float) -> None:
duration = round(time.monotonic() - began, 3)
checks.append(
{
"id": check_id,
"state": "passed",
"duration_seconds": duration,
}
)
print(f"PASS {platform} {check_id} ({duration}s)", flush=True)
common_runtime = [
"--platform",
platform,
"--network",
names["network"],
"--read-only",
"--tmpfs",
"/tmp:rw,noexec,nosuid,size=64m",
"--security-opt",
"no-new-privileges:true",
"--cap-drop",
"ALL",
"--mount",
f"type=volume,source={names['volume']},target=/var/lib/govoplan",
]
redis_command = ["redis-server", "--save", "", "--appendonly", "no"]
if platform == "linux/arm64":
# QEMU user-mode execution triggers Redis's host-kernel COW guard even
# though this isolated smoke disables every persistence mechanism.
redis_command.extend(("--ignore-warnings", "ARM64-COW-BUG"))
try:
_run(("docker", "network", "create", names["network"]), timeout=timeout)
_run(("docker", "volume", "create", names["volume"]), timeout=timeout)
began = time.monotonic()
_run(
(
"docker",
"run",
"--detach",
"--platform",
platform,
"--name",
names["postgres"],
"--network",
names["network"],
"--network-alias",
"postgres",
"--env",
"POSTGRES_DB=govoplan",
"--env",
"POSTGRES_USER=govoplan",
"--env",
f"POSTGRES_PASSWORD={database_password}",
"--tmpfs",
"/var/lib/postgresql/data:rw,noexec,nosuid,size=384m",
postgres_image,
),
timeout=timeout,
redactions=redactions,
)
_run(
(
"docker",
"run",
"--detach",
"--platform",
platform,
"--name",
names["redis"],
"--network",
names["network"],
"--network-alias",
"redis",
"--read-only",
"--tmpfs",
"/data:rw,noexec,nosuid,size=64m",
redis_image,
*redis_command,
),
timeout=timeout,
)
_wait_for(
"PostgreSQL",
lambda: _run(
(
"docker",
"exec",
names["postgres"],
"pg_isready",
"--username",
"govoplan",
"--dbname",
"govoplan",
),
check=False,
timeout=30,
),
timeout=timeout,
container=names["postgres"],
redactions=redactions,
)
_wait_for(
"Redis",
lambda: _run(
("docker", "exec", names["redis"], "redis-cli", "ping"),
check=False,
timeout=30,
),
timeout=timeout,
container=names["redis"],
redactions=redactions,
)
record("managed_dependencies_ready", began)
began = time.monotonic()
_run(
(
"docker",
"run",
"--rm",
"--name",
f"{prefix}-migrate",
*common_runtime,
*_env_arguments(
environment,
role="migration",
node_id=f"runtime-smoke-{slug}-migration",
),
api_image,
"python",
"-m",
"govoplan_core.commands.init_db",
"--migration-track",
"release",
),
timeout=timeout,
redactions=redactions,
)
record("release_migrations", began)
began = time.monotonic()
_run(
(
"docker",
"run",
"--rm",
"--name",
f"{prefix}-schema",
*common_runtime,
*_env_arguments(
environment,
role="migration",
node_id=f"runtime-smoke-{slug}-schema",
),
api_image,
"python",
"-c",
(
"import os;"
"from sqlalchemy import create_engine,inspect;"
"engine=create_engine(os.environ['DATABASE_URL']);"
"tables=set(inspect(engine).get_table_names());"
"required={'alembic_version','core_scopes','core_system_settings',"
"'core_runtime_nodes'};"
"missing=required-tables;"
"assert not missing, f'missing release tables: {sorted(missing)}';"
"engine.dispose()"
),
),
timeout=timeout,
redactions=redactions,
)
record("release_schema_contract", began)
began = time.monotonic()
_run(
(
"docker",
"run",
"--detach",
"--name",
names["api"],
"--network-alias",
"api",
"--network-alias",
"load-balancer",
*common_runtime,
*_env_arguments(
environment,
role="api",
node_id=f"runtime-smoke-{slug}-api",
),
api_image,
),
timeout=timeout,
redactions=redactions,
)
_wait_for(
"GovOPlaN API",
lambda: _run(
(
"docker",
"exec",
names["api"],
"python",
"-c",
(
"import urllib.request;"
"r=urllib.request.Request('http://127.0.0.1:8000/health/ready',"
"headers={'Host':'127.0.0.1'});"
"assert urllib.request.urlopen(r,timeout=3).status==200"
),
),
check=False,
timeout=30,
),
timeout=timeout,
container=names["api"],
redactions=redactions,
)
_run(
(
"docker",
"exec",
names["api"],
"python",
"-c",
"import os; assert os.getuid() == 10001",
),
timeout=30,
)
record("api_non_root_readiness", began)
began = time.monotonic()
_run(
(
"docker",
"run",
"--detach",
"--platform",
platform,
"--name",
names["web"],
"--network",
names["network"],
"--network-alias",
"web",
"--read-only",
"--tmpfs",
"/tmp:rw,noexec,nosuid,size=64m",
"--security-opt",
"no-new-privileges:true",
"--cap-drop",
"ALL",
web_image,
),
timeout=timeout,
)
_wait_for(
"GovOPlaN WebUI",
lambda: _run(
(
"docker",
"exec",
names["api"],
"python",
"-c",
(
"import urllib.request;"
"assert urllib.request.urlopen('http://web:8080/health',timeout=3).status==200;"
"assert urllib.request.urlopen('http://web:8080/',timeout=3).status==200"
),
),
check=False,
timeout=30,
),
timeout=timeout,
container=names["web"],
redactions=redactions,
)
_run(
("docker", "exec", names["web"], "sh", "-c", "test \"$(id -u)\" = 101"),
timeout=30,
)
record("web_non_root_readiness", began)
began = time.monotonic()
_run(
(
"docker",
"run",
"--detach",
"--name",
names["worker"],
*common_runtime,
*_env_arguments(
environment,
role="worker",
node_id=f"runtime-smoke-{slug}-worker",
),
api_image,
"python",
"-m",
"celery",
"-A",
"govoplan_core.celery_app:celery",
"worker",
"--queues",
"default",
"--pool",
"solo",
"--concurrency",
"1",
"--hostname",
f"runtime-smoke-{slug}@%h",
"--loglevel",
"WARNING",
),
timeout=timeout,
redactions=redactions,
)
_wait_for(
"GovOPlaN worker",
lambda: _run(
(
"docker",
"exec",
names["api"],
"python",
"-c",
(
"from govoplan_core.celery_app import celery;"
"result=celery.send_task('govoplan.ping',queue='default');"
"assert result.get(timeout=10)=='pong'"
),
),
check=False,
timeout=30,
),
timeout=timeout,
container=names["worker"],
redactions=redactions,
)
_run(("docker", "stop", "--time", "20", names["worker"]), timeout=30)
record("worker_delivery_and_shutdown", began)
except SmokeError as exc:
for role in ("api", "web", "worker", "postgres", "redis"):
result = _run(
("docker", "logs", "--tail", "100", names[role]),
check=False,
timeout=30,
)
if result.stdout or result.stderr:
detail = _tail(result.stdout + result.stderr, limit=8000)
for secret in redactions:
detail = detail.replace(secret, "[redacted]")
print(f"--- {role} logs ---\n{detail}")
raise exc
finally:
for role in ("worker", "web", "api", "redis", "postgres"):
_run(
("docker", "rm", "--force", names[role]),
check=False,
timeout=30,
)
_run(("docker", "volume", "rm", "--force", names["volume"]), check=False)
_run(("docker", "network", "rm", names["network"]), check=False)
return {
"schema_version": "1",
"evidence_kind": "govoplan.runtime-image-smoke",
"captured_at": _utc_now(),
"platform": platform,
"images": {
"api": api_image,
"web": web_image,
"postgres": postgres_image,
"redis": redis_image,
},
"result": {"state": "passed"},
"checks": checks,
"duration_seconds": round(time.monotonic() - started, 3),
}
def main() -> int:
args = build_parser().parse_args()
try:
api_image = platform_image(args.api_metadata, args.platform, "API")
web_image = platform_image(args.web_metadata, args.platform, "Web")
postgres_image = platform_image(
args.postgres_metadata,
args.platform,
"PostgreSQL",
)
redis_image = platform_image(args.redis_metadata, args.platform, "Redis")
evidence = run_smoke(
api_image=api_image,
web_image=web_image,
postgres_image=postgres_image,
redis_image=redis_image,
platform=args.platform,
timeout=args.timeout_seconds,
)
except (OSError, SmokeError, ValueError) as exc:
print(f"runtime image smoke failed: {exc}")
return 1
args.output.parent.mkdir(parents=True, exist_ok=True)
temporary = args.output.with_suffix(args.output.suffix + ".tmp")
temporary.write_text(json.dumps(evidence, indent=2, sort_keys=True) + "\n", encoding="utf-8")
temporary.chmod(0o644)
temporary.replace(args.output)
print(f"Runtime image smoke evidence written to {args.output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+40 -8
View File
@@ -6,11 +6,13 @@ from __future__ import annotations
import argparse
from hashlib import sha256
from pathlib import Path
import zipapp
from zipfile import ZIP_DEFLATED, ZipFile, ZipInfo
ROOT = Path(__file__).resolve().parent
DEFAULT_OUTPUT = ROOT.parent.parent / "runtime" / "deployment" / "govoplan-deploy.pyz"
ZIP_TIMESTAMP = (1980, 1, 1, 0, 0, 0)
PYTHON_FILE_MODE = 0o100644
def main(argv: list[str] | None = None) -> int:
@@ -25,13 +27,7 @@ def main(argv: list[str] | None = None) -> int:
temporary = output.with_name(f".{output.name}.tmp")
if temporary.exists():
temporary.unlink()
zipapp.create_archive(
ROOT,
target=temporary,
interpreter="/usr/bin/env python3",
compressed=True,
filter=_include_source,
)
_write_reproducible_zipapp(temporary)
temporary.chmod(0o755)
temporary.replace(output)
digest = sha256(output.read_bytes()).hexdigest()
@@ -39,6 +35,42 @@ def main(argv: list[str] | None = None) -> int:
return 0
def _write_reproducible_zipapp(target: Path) -> None:
sources = tuple(
path
for path in sorted(ROOT.rglob("*"), key=lambda item: item.as_posix())
if path.is_file() and _include_source(path.relative_to(ROOT))
)
if not any(path.relative_to(ROOT).as_posix() == "__main__.py" for path in sources):
raise ValueError("deployment source has no __main__.py")
for path in sources:
if path.is_symlink():
raise ValueError(f"deployment source must not contain symlinks: {path}")
with target.open("wb") as handle:
handle.write(b"#!/usr/bin/env python3\n")
with ZipFile(
handle,
mode="w",
compression=ZIP_DEFLATED,
compresslevel=9,
strict_timestamps=True,
) as archive:
for source in sources:
relative = source.relative_to(ROOT).as_posix()
info = ZipInfo(relative, date_time=ZIP_TIMESTAMP)
info.compress_type = ZIP_DEFLATED
info.create_system = 3
info.external_attr = PYTHON_FILE_MODE << 16
info.flag_bits |= 0x800
archive.writestr(
info,
source.read_bytes(),
compress_type=ZIP_DEFLATED,
compresslevel=9,
)
def _include_source(path: Path) -> bool:
return (
"__pycache__" not in path.parts
@@ -0,0 +1,500 @@
"""Signed, provider-neutral backup and isolated-restore evidence."""
from __future__ import annotations
from datetime import UTC, datetime
from pathlib import Path
import re
from typing import Any, Mapping
from urllib.parse import urlsplit
from .distribution import (
DistributionError,
load_bounded_json,
verify_signed_document,
)
MAX_BACKUP_EVIDENCE_BYTES = 1024 * 1024
MAX_BACKUP_KEYRING_BYTES = 1024 * 1024
DEFAULT_MAX_BACKUP_AGE_SECONDS = 24 * 60 * 60
MAX_COORDINATION_SKEW_SECONDS = 5 * 60
SHA256 = re.compile(r"^[0-9a-f]{64}$")
TOKEN = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
IMAGE = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$")
REFERENCE = re.compile(r"^[A-Za-z][A-Za-z0-9+.-]*:[^\s]{1,2040}$")
def load_backup_evidence(path: Path) -> dict[str, Any]:
return load_bounded_json(path, maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES)
def load_backup_keyring(path: Path) -> dict[str, Any]:
return load_bounded_json(path, maximum_bytes=MAX_BACKUP_KEYRING_BYTES)
def verify_backup_evidence(
payload: Mapping[str, Any],
keyring: Mapping[str, Any],
*,
installation_id: str,
profile: str,
release: Mapping[str, object],
now: datetime | None = None,
max_age_seconds: int = DEFAULT_MAX_BACKUP_AGE_SECONDS,
openssl: str = "openssl",
) -> dict[str, object]:
current = (now or datetime.now(UTC)).astimezone(UTC)
values = _validate_payload(payload, now=current, max_age_seconds=max_age_seconds)
if payload.get("installation_id") != installation_id:
raise DistributionError("backup evidence belongs to another installation")
subject = _object(payload.get("deployment_subject"), "deployment_subject")
if subject.get("profile") != profile:
raise DistributionError("backup evidence belongs to another deployment profile")
evidence_release = _object(payload.get("release"), "release")
for field in (
"channel",
"version",
"manifest_sha256",
"composition_sha256",
"api_image",
"web_image",
):
if evidence_release.get(field) != release.get(field):
raise DistributionError(
f"backup evidence does not match release field {field!r}"
)
key_id = verify_signed_document(
payload,
keyring,
purpose="govoplan-backup-evidence",
label="backup evidence",
now=current,
openssl=openssl,
)
recovery_point = _object(payload.get("recovery_point"), "recovery_point")
restore = _object(payload.get("restore_drill"), "restore_drill")
return {
"evidence_id": payload["evidence_id"],
"recovery_point_id": recovery_point["id"],
"captured_at": recovery_point["captured_at"],
"expires_at": payload["expires_at"],
"restore_drill_id": restore["drill_id"],
"restore_started_at": restore["started_at"],
"restore_completed_at": restore["completed_at"],
"measured_rpo_seconds": restore["measured_rpo_seconds"],
"measured_rto_seconds": restore["measured_rto_seconds"],
"signature_key_id": key_id,
"component_count": values["component_count"],
}
def _validate_payload(
payload: Mapping[str, Any],
*,
now: datetime,
max_age_seconds: int,
) -> dict[str, int]:
if max_age_seconds < 60 or max_age_seconds > 30 * 24 * 60 * 60:
raise DistributionError("backup maximum age is out of bounds")
_exact_keys(
payload,
{
"schema_version",
"evidence_id",
"installation_id",
"deployment_subject",
"release",
"recovery_point",
"components",
"restore_drill",
"issued_at",
"expires_at",
"revoked",
"signatures",
},
"backup evidence",
)
if payload.get("schema_version") != "1":
raise DistributionError("unsupported backup evidence schema_version")
_token(payload.get("evidence_id"), "evidence_id")
_token(payload.get("installation_id"), "installation_id")
issued = _timestamp(payload.get("issued_at"), "issued_at")
expires = _timestamp(payload.get("expires_at"), "expires_at")
if issued > now or expires <= issued or expires <= now:
raise DistributionError("backup evidence is not currently valid")
if payload.get("revoked") is not False:
raise DistributionError("backup evidence is revoked")
subject = _object(payload.get("deployment_subject"), "deployment_subject")
_exact_keys(subject, {"profile", "topology", "subject_ref"}, "deployment_subject")
if subject.get("profile") not in {"evaluation", "self-hosted"}:
raise DistributionError("deployment_subject.profile is invalid")
_token(subject.get("topology"), "deployment_subject.topology")
_reference(subject.get("subject_ref"), "deployment_subject.subject_ref")
release = _object(payload.get("release"), "release")
_exact_keys(
release,
{
"channel",
"version",
"manifest_sha256",
"composition_sha256",
"api_image",
"web_image",
},
"release",
)
_token(release.get("channel"), "release.channel")
_token(release.get("version"), "release.version")
_sha256(release.get("manifest_sha256"), "release.manifest_sha256")
_sha256(release.get("composition_sha256"), "release.composition_sha256")
_image(release.get("api_image"), "release.api_image")
_image(release.get("web_image"), "release.web_image")
recovery = _object(payload.get("recovery_point"), "recovery_point")
_exact_keys(
recovery,
{"id", "captured_at", "consistency", "rpo_seconds", "write_fence"},
"recovery_point",
)
recovery_id = _token(recovery.get("id"), "recovery_point.id")
captured = _timestamp(recovery.get("captured_at"), "recovery_point.captured_at")
age = (now - captured).total_seconds()
if age < 0 or age > max_age_seconds:
raise DistributionError("backup recovery point is stale or in the future")
if recovery.get("consistency") not in {
"provider-atomic",
"application-quiesced",
"transaction-consistent",
}:
raise DistributionError("recovery_point.consistency is invalid")
declared_rpo = _bounded_integer(
recovery.get("rpo_seconds"),
"recovery_point.rpo_seconds",
maximum=30 * 24 * 60 * 60,
)
fence = _object(recovery.get("write_fence"), "recovery_point.write_fence")
_exact_keys(
fence,
{"mode", "token_sha256", "established_at"},
"recovery_point.write_fence",
)
if fence.get("mode") not in {
"provider-snapshot",
"application-quiesce",
"transaction-boundary",
}:
raise DistributionError("recovery_point.write_fence.mode is invalid")
_sha256(fence.get("token_sha256"), "recovery_point.write_fence.token_sha256")
established = _timestamp(
fence.get("established_at"),
"recovery_point.write_fence.established_at",
)
if abs((captured - established).total_seconds()) > MAX_COORDINATION_SKEW_SECONDS:
raise DistributionError(
"backup write fence is not coordinated with recovery point"
)
components = _object(payload.get("components"), "components")
_exact_keys(
components,
{"database", "objects", "configuration", "key_custody"},
"components",
)
captured_components = [
_database_component(components.get("database")),
_objects_component(components.get("objects")),
_configuration_component(components.get("configuration")),
_key_custody_component(components.get("key_custody")),
]
if any(
abs((component_time - captured).total_seconds()) > MAX_COORDINATION_SKEW_SECONDS
for component_time in captured_components
):
raise DistributionError("backup components do not share one recovery point")
restore = _object(payload.get("restore_drill"), "restore_drill")
_exact_keys(
restore,
{
"drill_id",
"recovery_point_id",
"started_at",
"completed_at",
"isolated_target_ref",
"release_manifest_sha256",
"migration_heads_sha256",
"representative_object_manifest_sha256",
"database_verified",
"objects_verified",
"configuration_verified",
"key_custody_verified",
"semantic_checks",
"measured_rpo_seconds",
"measured_rto_seconds",
"evidence_ref",
},
"restore_drill",
)
_token(restore.get("drill_id"), "restore_drill.drill_id")
if restore.get("recovery_point_id") != recovery_id:
raise DistributionError("restore drill used another recovery point")
started = _timestamp(restore.get("started_at"), "restore_drill.started_at")
completed = _timestamp(restore.get("completed_at"), "restore_drill.completed_at")
if started < captured or completed < started or completed > issued:
raise DistributionError(
"restore drill completion is outside evidence chronology"
)
_reference(restore.get("isolated_target_ref"), "restore_drill.isolated_target_ref")
_reference(restore.get("evidence_ref"), "restore_drill.evidence_ref")
for field in (
"release_manifest_sha256",
"migration_heads_sha256",
"representative_object_manifest_sha256",
):
_sha256(restore.get(field), f"restore_drill.{field}")
if restore.get("release_manifest_sha256") != release.get("manifest_sha256"):
raise DistributionError("restore drill used another immutable release")
for field in (
"database_verified",
"objects_verified",
"configuration_verified",
"key_custody_verified",
):
if restore.get(field) is not True:
raise DistributionError(f"restore_drill.{field} must be true")
semantic = restore.get("semantic_checks")
if not isinstance(semantic, list) or not semantic or len(semantic) > 128:
raise DistributionError("restore_drill.semantic_checks must not be empty")
seen_checks: set[str] = set()
for index, raw in enumerate(semantic):
check = _object(raw, f"restore_drill.semantic_checks[{index}]")
_exact_keys(
check,
{"id", "status", "evidence_ref"},
f"restore_drill.semantic_checks[{index}]",
)
check_id = _token(check.get("id"), f"semantic_checks[{index}].id")
if check_id in seen_checks or check.get("status") != "passed":
raise DistributionError("restore drill semantic checks are invalid")
seen_checks.add(check_id)
_reference(check.get("evidence_ref"), f"semantic_checks[{index}].evidence_ref")
measured_rpo = _bounded_integer(
restore.get("measured_rpo_seconds"),
"restore_drill.measured_rpo_seconds",
maximum=30 * 24 * 60 * 60,
)
measured_rto = _bounded_integer(
restore.get("measured_rto_seconds"),
"restore_drill.measured_rto_seconds",
maximum=30 * 24 * 60 * 60,
)
if abs((completed - started).total_seconds() - measured_rto) > 5:
raise DistributionError("restore drill RTO does not match its timestamps")
if measured_rpo > declared_rpo:
raise DistributionError(
"restore drill exceeds the declared recovery point objective"
)
_validate_signatures(payload.get("signatures"))
return {"component_count": len(captured_components)}
def _database_component(raw: object) -> datetime:
value = _object(raw, "components.database")
_exact_keys(
value,
{
"provider",
"artifact_ref",
"artifact_sha256",
"snapshot_id",
"lsn",
"protected",
"encryption_key_ref",
"captured_at",
},
"components.database",
)
_common_artifact(value, "components.database")
_token(value.get("snapshot_id"), "components.database.snapshot_id")
_bounded_text(value.get("lsn"), "components.database.lsn", maximum=256)
return _timestamp(value.get("captured_at"), "components.database.captured_at")
def _objects_component(raw: object) -> datetime:
value = _object(raw, "components.objects")
_exact_keys(
value,
{
"provider",
"artifact_ref",
"manifest_sha256",
"version_id",
"object_count",
"total_bytes",
"protected",
"encryption_key_ref",
"captured_at",
},
"components.objects",
)
_token(value.get("provider"), "components.objects.provider")
_reference(value.get("artifact_ref"), "components.objects.artifact_ref")
_sha256(value.get("manifest_sha256"), "components.objects.manifest_sha256")
_token(value.get("version_id"), "components.objects.version_id")
_bounded_integer(value.get("object_count"), "components.objects.object_count")
_bounded_integer(value.get("total_bytes"), "components.objects.total_bytes")
_protected_key_reference(value, "components.objects")
return _timestamp(value.get("captured_at"), "components.objects.captured_at")
def _configuration_component(raw: object) -> datetime:
value = _object(raw, "components.configuration")
_exact_keys(
value,
{
"artifact_ref",
"sha256",
"protected",
"encryption_key_ref",
"captured_at",
},
"components.configuration",
)
_reference(value.get("artifact_ref"), "components.configuration.artifact_ref")
_sha256(value.get("sha256"), "components.configuration.sha256")
_protected_key_reference(value, "components.configuration")
return _timestamp(value.get("captured_at"), "components.configuration.captured_at")
def _key_custody_component(raw: object) -> datetime:
value = _object(raw, "components.key_custody")
_exact_keys(
value,
{"provider", "keyset_ref", "keyset_version", "recoverable", "captured_at"},
"components.key_custody",
)
_token(value.get("provider"), "components.key_custody.provider")
_reference(value.get("keyset_ref"), "components.key_custody.keyset_ref")
_token(value.get("keyset_version"), "components.key_custody.keyset_version")
if value.get("recoverable") is not True:
raise DistributionError("components.key_custody.recoverable must be true")
return _timestamp(value.get("captured_at"), "components.key_custody.captured_at")
def _common_artifact(value: Mapping[str, Any], label: str) -> None:
_token(value.get("provider"), f"{label}.provider")
_reference(value.get("artifact_ref"), f"{label}.artifact_ref")
_sha256(value.get("artifact_sha256"), f"{label}.artifact_sha256")
_protected_key_reference(value, label)
def _protected_key_reference(value: Mapping[str, Any], label: str) -> None:
if value.get("protected") is not True:
raise DistributionError(f"{label}.protected must be true")
_reference(value.get("encryption_key_ref"), f"{label}.encryption_key_ref")
def _validate_signatures(raw: object) -> None:
if not isinstance(raw, list) or not raw or len(raw) > 16:
raise DistributionError("backup evidence signatures must not be empty")
seen: set[str] = set()
for index, item in enumerate(raw):
signature = _object(item, f"signatures[{index}]")
_exact_keys(signature, {"key_id", "algorithm", "value"}, f"signatures[{index}]")
key_id = _token(signature.get("key_id"), f"signatures[{index}].key_id")
if key_id in seen or signature.get("algorithm") != "ed25519":
raise DistributionError("backup evidence signatures are invalid")
seen.add(key_id)
encoded = signature.get("value")
if not isinstance(encoded, str) or len(encoded) > 256:
raise DistributionError("backup evidence signature value is invalid")
def _reference(raw: object, label: str) -> str:
value = _bounded_text(raw, label, maximum=2048)
if REFERENCE.fullmatch(value) is None or "BEGIN " in value.upper():
raise DistributionError(f"{label} must be an opaque provider reference")
parsed = urlsplit(value)
if parsed.username or parsed.password or parsed.query or parsed.fragment:
raise DistributionError(f"{label} must not contain credentials or query data")
return value
def _object(raw: object, label: str) -> dict[str, Any]:
if not isinstance(raw, dict) or not all(isinstance(key, str) for key in raw):
raise DistributionError(f"{label} must be an object")
return raw
def _exact_keys(value: Mapping[str, Any], keys: set[str], label: str) -> None:
if set(value) != keys:
missing = sorted(keys - set(value))
extra = sorted(set(value) - keys)
detail = []
if missing:
detail.append("missing " + ", ".join(missing))
if extra:
detail.append("unknown " + ", ".join(extra))
raise DistributionError(f"{label} has invalid fields: {'; '.join(detail)}")
def _timestamp(raw: object, label: str) -> datetime:
value = _bounded_text(raw, label, maximum=64)
try:
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError as exc:
raise DistributionError(f"{label} must be an RFC3339 timestamp") from exc
if parsed.tzinfo is None:
raise DistributionError(f"{label} must include a timezone")
return parsed.astimezone(UTC)
def _token(raw: object, label: str) -> str:
value = _bounded_text(raw, label, maximum=128)
if TOKEN.fullmatch(value) is None:
raise DistributionError(f"{label} is invalid")
return value
def _sha256(raw: object, label: str) -> str:
value = _bounded_text(raw, label, maximum=64)
if SHA256.fullmatch(value) is None:
raise DistributionError(f"{label} must be a lowercase SHA-256 digest")
return value
def _image(raw: object, label: str) -> str:
value = _bounded_text(raw, label, maximum=300)
if IMAGE.fullmatch(value) is None:
raise DistributionError(f"{label} must be an OCI image pinned by sha256")
return value
def _bounded_text(raw: object, label: str, *, maximum: int) -> str:
if not isinstance(raw, str) or not raw or len(raw) > maximum or "\n" in raw:
raise DistributionError(f"{label} is invalid")
return raw
def _bounded_integer(
raw: object,
label: str,
*,
maximum: int = 2**63 - 1,
) -> int:
if isinstance(raw, bool) or not isinstance(raw, int) or raw < 0 or raw > maximum:
raise DistributionError(f"{label} is out of bounds")
return raw
__all__ = [
"DEFAULT_MAX_BACKUP_AGE_SECONDS",
"MAX_BACKUP_EVIDENCE_BYTES",
"MAX_BACKUP_KEYRING_BYTES",
"load_backup_evidence",
"load_backup_keyring",
"verify_backup_evidence",
]
@@ -28,7 +28,26 @@ PLAN_FILENAME = "plan.json"
RECEIPT_FILENAME = "receipt.json"
MANIFEST_FILENAME = "distribution-manifest.json"
KEYRING_FILENAME = "distribution-keyring.json"
BACKUP_EVIDENCE_FILENAME = "backup-evidence.json"
BACKUP_KEYRING_FILENAME = "backup-keyring.json"
BACKUP_VERIFICATION_FILENAME = "backup-verification.json"
LOCK_FILENAME = ".deployment.lock"
BACKUP_RUNTIME_ENV_KEYS = (
"GOVOPLAN_BACKUP_EVIDENCE_STATE",
"GOVOPLAN_BACKUP_EVIDENCE_ID",
"GOVOPLAN_BACKUP_RECOVERY_POINT_ID",
"GOVOPLAN_BACKUP_RESTORE_DRILL_ID",
"GOVOPLAN_BACKUP_EVIDENCE_SHA256",
"GOVOPLAN_BACKUP_RELEASE_MANIFEST_SHA256",
"GOVOPLAN_BACKUP_CAPTURED_AT",
"GOVOPLAN_BACKUP_EXPIRES_AT",
"GOVOPLAN_BACKUP_RESTORE_STARTED_AT",
"GOVOPLAN_BACKUP_RESTORE_COMPLETED_AT",
"GOVOPLAN_BACKUP_VERIFIED_AT",
"GOVOPLAN_BACKUP_MEASURED_RPO_SECONDS",
"GOVOPLAN_BACKUP_MEASURED_RTO_SECONDS",
"GOVOPLAN_BACKUP_COMPONENT_COUNT",
)
RUNTIME_ENV_KEYS = (
"APP_ENV",
"GOVOPLAN_INSTALL_PROFILE",
@@ -78,6 +97,7 @@ RUNTIME_ENV_KEYS = (
"FILE_STORAGE_S3_BUCKET",
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
*BACKUP_RUNTIME_ENV_KEYS,
)
@@ -95,6 +115,9 @@ class BundlePaths:
receipt: Path
manifest: Path
keyring: Path
backup_evidence: Path
backup_keyring: Path
backup_verification: Path
lock: Path
@@ -116,6 +139,9 @@ def bundle_paths(root: Path) -> BundlePaths:
receipt=resolved / RECEIPT_FILENAME,
manifest=resolved / MANIFEST_FILENAME,
keyring=resolved / KEYRING_FILENAME,
backup_evidence=resolved / BACKUP_EVIDENCE_FILENAME,
backup_keyring=resolved / BACKUP_KEYRING_FILENAME,
backup_verification=resolved / BACKUP_VERIFICATION_FILENAME,
lock=resolved / LOCK_FILENAME,
)
@@ -601,6 +627,7 @@ def render_compose(spec: InstallationSpec) -> dict[str, object]:
"tmpfs": ["/tmp:rw,noexec,nosuid,size=64m"],
"security_opt": ["no-new-privileges:true"],
"cap_drop": ["ALL"],
"cap_add": ["NET_BIND_SERVICE"],
"volumes": [
f"./{CADDY_CONFIG_FILENAME}:/etc/caddy/Caddyfile:ro",
"caddy-data:/data",
+291 -9
View File
@@ -20,7 +20,14 @@ from typing import Iterator, Mapping, Sequence
from urllib.error import URLError
from urllib.request import urlopen
from .backup_evidence import (
DEFAULT_MAX_BACKUP_AGE_SECONDS,
MAX_BACKUP_EVIDENCE_BYTES,
MAX_BACKUP_KEYRING_BYTES,
verify_backup_evidence,
)
from .bundle import (
BACKUP_RUNTIME_ENV_KEYS,
atomic_write,
bundle_paths,
canonical_json,
@@ -72,7 +79,12 @@ from .kubernetes import (
render_kubernetes,
write_secret_creation_hint,
)
from .planning import DeploymentPlan, build_plan
from .planning import (
DeploymentPlan,
build_plan,
release_change_requires_backup,
verify_stored_backup_evidence,
)
from .recovery import (
DeploymentOperationJournal,
list_operations,
@@ -183,6 +195,22 @@ def build_parser() -> argparse.ArgumentParser:
help="Load verified archives into Docker using fixed image-load commands.",
)
verify_backup = subparsers.add_parser(
"verify-backup",
help="Verify and optionally adopt signed coordinated backup evidence.",
)
_directory_argument(verify_backup)
evidence_source = verify_backup.add_mutually_exclusive_group(required=True)
evidence_source.add_argument("--evidence", type=Path)
evidence_source.add_argument("--evidence-url")
verify_backup.add_argument("--evidence-sha256", required=True)
verify_backup.add_argument("--trusted-keyring", type=Path, required=True)
verify_backup.add_argument(
"--allow-private-evidence-host",
action="store_true",
)
verify_backup.add_argument("--adopt", action="store_true")
kubernetes = subparsers.add_parser(
"render-kubernetes",
help="Export the stateless multi-host runtime for Kubernetes.",
@@ -392,6 +420,8 @@ def main(argv: Sequence[str] | None = None) -> int:
return _verify_release(args)
if args.command == "verify-offline-images":
return _verify_offline_images(args)
if args.command == "verify-backup":
return _verify_backup(args)
if args.command == "render-kubernetes":
return _render_kubernetes(args)
if args.command == "verify-kubernetes":
@@ -534,15 +564,17 @@ def _render_or_doctor(args: argparse.Namespace) -> int:
def _apply(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
spec = load_spec(paths.spec)
if args.allow_unverified_images and spec.profile != "evaluation":
raise ValueError(
"--allow-unverified-images is restricted to evaluation installations"
)
ensure_private_directory(paths.root)
with _deployment_lock(paths.lock):
spec = load_spec(paths.spec)
previous_receipt = _read_json_object(paths.receipt)
backup_required = release_change_requires_backup(spec, previous_receipt)
if args.allow_unverified_images and spec.profile != "evaluation":
raise ValueError(
"--allow-unverified-images is restricted to evaluation installations"
)
secrets = reconcile_runtime_environment(spec, read_env(paths.env))
_write_bundle(spec, paths, secrets)
secrets = _write_bundle(spec, paths, secrets)
plan = build_plan(spec, paths, include_host_checks=True)
_write_plan(paths.plan, plan)
effective_errors = [
@@ -577,6 +609,36 @@ def _apply(args: argparse.Namespace) -> int:
paths,
plan=plan.to_dict(),
)
try:
if backup_required:
backup_summary = verify_stored_backup_evidence(
spec,
paths,
receipt=previous_receipt,
)
journal.record(
"backup-evidence-verified",
"succeeded",
dict(backup_summary),
)
else:
journal.record(
"backup-evidence-not-required",
"succeeded",
{"release_change": False},
)
except BaseException as exc:
journal.record(
"backup-evidence-rejected",
"blocked",
{
"phase": "preflight",
"exception_type": type(exc).__name__,
"migration_started": False,
},
)
journal.failed(exc)
raise
compose = [
docker,
"compose",
@@ -624,6 +686,29 @@ def _apply(args: argparse.Namespace) -> int:
"succeeded",
{"services": mutable_runtime_services, "timeout_seconds": 120},
)
if backup_required:
try:
backup_summary = verify_stored_backup_evidence(
spec,
paths,
receipt=previous_receipt,
)
except BaseException as exc:
journal.record(
"backup-evidence-rejected",
"blocked",
{
"phase": "migration-boundary",
"exception_type": type(exc).__name__,
"migration_started": False,
},
)
raise
journal.record(
"backup-evidence-reverified",
"succeeded",
dict(backup_summary),
)
journal.migration_started()
_run([*compose, "run", "--rm", "migrate"], cwd=paths.root)
journal.migration_completed()
@@ -898,6 +983,108 @@ def _verify_offline_images(args: argparse.Namespace) -> int:
return 0
def _verify_backup(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
ensure_private_directory(paths.root)
with _deployment_lock(paths.lock):
return _verify_backup_locked(args, paths)
def _verify_backup_locked(args: argparse.Namespace, paths) -> int:
spec = load_spec(paths.spec)
expected_digest = str(args.evidence_sha256 or "").strip().lower()
if len(expected_digest) != 64 or any(
character not in "0123456789abcdef" for character in expected_digest
):
raise ValueError("--evidence-sha256 must be a lowercase SHA-256 digest")
if args.evidence_url:
encoded_evidence = fetch_bounded_https(
str(args.evidence_url),
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
allow_private_host=args.allow_private_evidence_host,
)
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
else:
evidence_path = args.evidence.expanduser().resolve()
encoded_evidence = read_bounded_bytes(
evidence_path,
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
)
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
if encoded_evidence != canonical_distribution_json(evidence):
raise DistributionError("backup evidence is not canonical JSON")
if hashlib.sha256(encoded_evidence).hexdigest() != expected_digest:
raise DistributionError("backup evidence SHA-256 does not match")
keyring_path = args.trusted_keyring.expanduser().resolve()
keyring = load_bounded_json(
keyring_path,
maximum_bytes=MAX_BACKUP_KEYRING_BYTES,
)
encoded_keyring = canonical_distribution_json(keyring)
receipt = _read_json_object(paths.receipt)
previous_release = receipt.get("release") if receipt else None
release: Mapping[str, object] = (
previous_release
if isinstance(previous_release, Mapping)
else {
"channel": spec.release.channel,
"version": spec.release.version,
"manifest_sha256": spec.release.manifest_sha256,
"composition_sha256": spec.release.composition_sha256,
"api_image": spec.release.api_image,
"web_image": spec.release.web_image,
}
)
summary = verify_backup_evidence(
evidence,
keyring,
installation_id=spec.installation_id,
profile=spec.profile,
release=release,
max_age_seconds=DEFAULT_MAX_BACKUP_AGE_SECONDS,
)
print(
"Verified coordinated recovery point "
f"{summary['recovery_point_id']} with restore drill "
f"{summary['restore_drill_id']} and trusted key "
f"{summary['signature_key_id']}."
)
if not args.adopt:
return 0
verification = {
"schema_version": 1,
"evidence_sha256": expected_digest,
"keyring_sha256": hashlib.sha256(encoded_keyring).hexdigest(),
"signature_key_id": summary["signature_key_id"],
"verified_at": _now(),
"evidence_id": summary["evidence_id"],
"recovery_point_id": summary["recovery_point_id"],
"restore_drill_id": summary["restore_drill_id"],
"release_manifest_sha256": release.get("manifest_sha256"),
"captured_at": summary["captured_at"],
"expires_at": summary["expires_at"],
"restore_started_at": summary["restore_started_at"],
"restore_completed_at": summary["restore_completed_at"],
"measured_rpo_seconds": summary["measured_rpo_seconds"],
"measured_rto_seconds": summary["measured_rto_seconds"],
"component_count": summary["component_count"],
}
atomic_write(paths.backup_evidence, encoded_evidence, mode=0o600)
atomic_write(paths.backup_keyring, encoded_keyring, mode=0o600)
atomic_write(
paths.backup_verification,
canonical_json(verification),
mode=0o600,
)
_write_bundle(
spec,
paths,
reconcile_runtime_environment(spec, read_env(paths.env)),
)
print(f"Adopted signed backup evidence in {paths.root}.")
return 0
def _selected_dependency_images(
spec: InstallationSpec,
*,
@@ -929,6 +1116,30 @@ def _render_kubernetes(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
spec = load_spec(paths.spec)
environment = reconcile_runtime_environment(spec, read_env(paths.env))
environment.update(_backup_runtime_environment(spec, paths))
receipt = _read_json_object(paths.receipt)
backup_required = release_change_requires_backup(spec, receipt)
backup_summary: Mapping[str, object] | None = None
evidence_files = (
paths.backup_evidence,
paths.backup_keyring,
paths.backup_verification,
)
if backup_required:
backup_summary = verify_stored_backup_evidence(
spec,
paths,
receipt=receipt,
)
elif all(path.is_file() for path in evidence_files):
try:
backup_summary = verify_stored_backup_evidence(
spec,
paths,
receipt=receipt,
)
except (DistributionError, OSError):
backup_summary = None
manifest = render_kubernetes(
spec,
environment,
@@ -936,6 +1147,8 @@ def _render_kubernetes(args: argparse.Namespace) -> int:
secret_name=args.secret_name,
tls_secret_name=args.tls_secret_name,
ingress_class_name=args.ingress_class_name,
backup_required=backup_required,
backup_evidence=backup_summary,
)
output = (args.output or (paths.root / "kubernetes.json")).expanduser().resolve()
atomic_write(output, canonical_json(manifest), mode=0o600)
@@ -1027,6 +1240,7 @@ def _deployment_receipt(
return {
"schema_version": 1,
"installation_id": spec.installation_id,
"profile": spec.profile,
"applied_at": _now(),
"spec_sha256": digest_json(spec.to_dict()),
"compose_sha256": digest_json(render_compose(spec)),
@@ -1064,6 +1278,16 @@ def _deployment_receipt(
}
def _read_json_object(path: Path) -> dict[str, object]:
if not path.is_file():
return {}
try:
value = load_bounded_json(path, maximum_bytes=64 * 1024)
except (DistributionError, OSError):
return {}
return value
def _updated_spec(
current: InstallationSpec, args: argparse.Namespace
) -> InstallationSpec:
@@ -1222,10 +1446,12 @@ def _write_bundle(
spec: InstallationSpec,
paths,
secrets: Mapping[str, str],
) -> None:
) -> dict[str, str]:
ensure_private_directory(paths.root)
runtime_environment = dict(secrets)
runtime_environment.update(_backup_runtime_environment(spec, paths))
atomic_write(paths.spec, canonical_json(spec.to_dict()), mode=0o600)
write_env(paths.env, secrets)
write_env(paths.env, runtime_environment)
atomic_write(paths.compose, canonical_json(render_compose(spec)), mode=0o600)
atomic_write(
paths.load_balancer_config,
@@ -1247,6 +1473,62 @@ def _write_bundle(
render_garage_config().encode("utf-8"),
mode=0o644,
)
return dict(sorted(runtime_environment.items()))
def _backup_runtime_environment(
spec: InstallationSpec,
paths,
) -> dict[str, str]:
values = {key: "" for key in BACKUP_RUNTIME_ENV_KEYS}
evidence_files = (
paths.backup_evidence,
paths.backup_keyring,
paths.backup_verification,
)
if not any(path.is_file() for path in evidence_files):
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "absent"
return values
if not all(path.is_file() for path in evidence_files):
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "invalid"
return values
verification = _read_json_object(paths.backup_verification)
try:
summary = verify_stored_backup_evidence(
spec,
paths,
receipt=_read_json_object(paths.receipt),
)
except (DistributionError, OSError):
values["GOVOPLAN_BACKUP_EVIDENCE_STATE"] = "invalid"
return values
values.update(
{
"GOVOPLAN_BACKUP_EVIDENCE_STATE": "verified",
"GOVOPLAN_BACKUP_EVIDENCE_ID": str(summary["evidence_id"]),
"GOVOPLAN_BACKUP_RECOVERY_POINT_ID": str(summary["recovery_point_id"]),
"GOVOPLAN_BACKUP_RESTORE_DRILL_ID": str(summary["restore_drill_id"]),
"GOVOPLAN_BACKUP_EVIDENCE_SHA256": str(summary["evidence_sha256"]),
"GOVOPLAN_BACKUP_RELEASE_MANIFEST_SHA256": str(
verification["release_manifest_sha256"]
),
"GOVOPLAN_BACKUP_CAPTURED_AT": str(summary["captured_at"]),
"GOVOPLAN_BACKUP_EXPIRES_AT": str(summary["expires_at"]),
"GOVOPLAN_BACKUP_RESTORE_STARTED_AT": str(summary["restore_started_at"]),
"GOVOPLAN_BACKUP_RESTORE_COMPLETED_AT": str(
summary["restore_completed_at"]
),
"GOVOPLAN_BACKUP_VERIFIED_AT": str(verification["verified_at"]),
"GOVOPLAN_BACKUP_MEASURED_RPO_SECONDS": str(
summary["measured_rpo_seconds"]
),
"GOVOPLAN_BACKUP_MEASURED_RTO_SECONDS": str(
summary["measured_rto_seconds"]
),
"GOVOPLAN_BACKUP_COMPONENT_COUNT": str(summary["component_count"]),
}
)
return values
def _write_plan(path: Path, plan: DeploymentPlan) -> None:
+100 -30
View File
@@ -74,7 +74,9 @@ def read_bounded_bytes(path: Path, *, maximum_bytes: int) -> bytes:
try:
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
raise DistributionError(f"trusted JSON file is invalid or too large: {path}")
raise DistributionError(
f"trusted JSON file is invalid or too large: {path}"
)
chunks: list[bytes] = []
total = 0
while True:
@@ -109,7 +111,9 @@ def fetch_bounded_https(
if parsed.scheme != "https" or not parsed.hostname:
raise DistributionError("distribution downloads require an absolute HTTPS URL")
if parsed.username or parsed.password or parsed.fragment:
raise DistributionError("distribution URL must not contain credentials or a fragment")
raise DistributionError(
"distribution URL must not contain credentials or a fragment"
)
if not allow_private_host:
_require_public_host(parsed.hostname)
request = Request(url, headers={"Accept": "application/json"})
@@ -163,6 +167,7 @@ def validate_manifest(
"composition",
"signatures",
},
optional={"package_lock"},
label="distribution manifest",
)
if payload.get("schema_version") != "1":
@@ -172,9 +177,11 @@ def validate_manifest(
raise DistributionError(
f"distribution channel is {channel!r}, expected {expected_channel!r}"
)
if isinstance(payload.get("sequence"), bool) or not isinstance(
payload.get("sequence"), int
) or int(payload["sequence"]) < 1:
if (
isinstance(payload.get("sequence"), bool)
or not isinstance(payload.get("sequence"), int)
or int(payload["sequence"]) < 1
):
raise DistributionError("distribution sequence must be a positive integer")
_token(payload.get("version"), "version", maximum=128, pattern=TOKEN)
issued = _datetime(payload.get("issued_at"), "issued_at")
@@ -194,6 +201,12 @@ def validate_manifest(
_https_url(deployer.get("url"), "deployer.url")
_sha256(deployer.get("sha256"), "deployer.sha256")
if "package_lock" in payload:
package_lock = _object(payload.get("package_lock"), "package_lock")
_exact_keys(package_lock, required={"url", "sha256"}, label="package_lock")
_https_url(package_lock.get("url"), "package_lock.url")
_sha256(package_lock.get("sha256"), "package_lock.sha256")
images = _object(payload.get("images"), "images")
if set(images) != {"api", "web"}:
raise DistributionError("images must contain exactly api and web")
@@ -283,11 +296,41 @@ def verify_manifest(
) -> str:
current = (now or datetime.now(UTC)).astimezone(UTC)
validate_manifest(payload, expected_channel=expected_channel, now=current)
keys = _trusted_keys(keyring, now=current)
return verify_signed_document(
payload,
keyring,
purpose="govoplan-runtime-distribution",
label="distribution",
now=current,
openssl=openssl,
)
def verify_signed_document(
payload: Mapping[str, Any],
keyring: Mapping[str, Any],
*,
purpose: str,
label: str,
now: datetime,
openssl: str = "openssl",
) -> str:
keys = _trusted_keys(keyring, now=now, purpose=purpose, label=label)
signed = canonical_signed_payload(payload)
failures: list[str] = []
for item in payload["signatures"]:
signatures = payload.get("signatures")
if not isinstance(signatures, list) or not signatures:
raise DistributionError(f"{label} has no signatures")
for index, raw in enumerate(signatures):
item = _object(raw, f"{label}.signatures[{index}]")
_exact_keys(
item,
required={"key_id", "algorithm", "value"},
label=f"{label}.signatures[{index}]",
)
key_id = str(item["key_id"])
if KEY_ID.fullmatch(key_id) is None or item.get("algorithm") != "ed25519":
raise DistributionError(f"{label} signature is invalid")
public_key = keys.get(key_id)
if public_key is None:
continue
@@ -303,8 +346,10 @@ def verify_manifest(
failures.append(f"{key_id}: {exc}")
continue
return key_id
detail = "; ".join(failures) if failures else "no signature used an active trusted key"
raise DistributionError(f"distribution signature verification failed: {detail}")
detail = (
"; ".join(failures) if failures else "no signature used an active trusted key"
)
raise DistributionError(f"{label} signature verification failed: {detail}")
def verify_manifest_binding(
@@ -319,7 +364,9 @@ def verify_manifest_binding(
dependencies: Mapping[str, str],
) -> None:
if payload.get("channel") != channel or payload.get("version") != version:
raise DistributionError("stored manifest does not match release channel/version")
raise DistributionError(
"stored manifest does not match release channel/version"
)
images = _object(payload.get("images"), "images")
if _object(images.get("api"), "images.api").get("index") != api_image:
raise DistributionError("stored manifest does not match API image")
@@ -372,9 +419,9 @@ def verify_offline_image_index(
archive = root / archive_relative
if reference in references:
raise DistributionError("offline image index contains duplicate references")
if _sha256_regular_file(archive, maximum_bytes=MAX_OFFLINE_IMAGE_BYTES) != _sha256(
value.get("sha256"), "offline image sha256"
):
if _sha256_regular_file(
archive, maximum_bytes=MAX_OFFLINE_IMAGE_BYTES
) != _sha256(value.get("sha256"), "offline image sha256"):
raise DistributionError(f"offline image archive digest mismatch: {archive}")
references[reference] = archive
missing = sorted(set(expected_references) - set(references))
@@ -418,19 +465,21 @@ def _trusted_keys(
keyring: Mapping[str, Any],
*,
now: datetime,
purpose: str,
label: str,
) -> dict[str, str]:
_exact_keys(
keyring,
required={"schema_version", "purpose", "keys"},
label="distribution keyring",
label=f"{label} keyring",
)
if keyring.get("schema_version") != "1":
raise DistributionError("unsupported distribution keyring schema_version")
if keyring.get("purpose") != "govoplan-runtime-distribution":
raise DistributionError("distribution keyring has the wrong purpose")
raise DistributionError(f"unsupported {label} keyring schema_version")
if keyring.get("purpose") != purpose:
raise DistributionError(f"{label} keyring has the wrong purpose")
values = keyring.get("keys")
if not isinstance(values, list) or not values:
raise DistributionError("distribution keyring contains no keys")
raise DistributionError(f"{label} keyring contains no keys")
trusted: dict[str, str] = {}
for index, item in enumerate(values):
key = _object(item, f"keyring.keys[{index}]")
@@ -453,11 +502,11 @@ def _trusted_keys(
pattern=KEY_ID,
)
if key_id in trusted:
raise DistributionError("distribution keyring contains duplicate key ids")
raise DistributionError(f"{label} keyring contains duplicate key ids")
if key.get("algorithm") != "ed25519":
raise DistributionError("distribution key must use ed25519")
raise DistributionError(f"{label} key must use ed25519")
if key.get("status") not in {"active", "retired", "revoked"}:
raise DistributionError("distribution key has an invalid status")
raise DistributionError(f"{label} key has an invalid status")
not_before = _datetime(key.get("not_before"), "key.not_before")
expires = _datetime(key.get("expires_at"), "key.expires_at")
public_key = key.get("public_key_pem")
@@ -466,11 +515,11 @@ def _trusted_keys(
or len(public_key.encode("utf-8")) > 8192
or "BEGIN PUBLIC KEY" not in public_key
):
raise DistributionError("distribution key has an invalid public key")
raise DistributionError(f"{label} key has an invalid public key")
if key.get("status") == "active" and not_before <= now < expires:
trusted[key_id] = public_key
if not trusted:
raise DistributionError("distribution keyring has no currently active keys")
raise DistributionError(f"{label} keyring has no currently active keys")
return trusted
@@ -534,13 +583,17 @@ def _require_public_host(hostname: str) -> None:
for value in socket.getaddrinfo(hostname, 443, type=socket.SOCK_STREAM)
}
except OSError as exc:
raise DistributionError(f"distribution host cannot be resolved: {hostname}") from exc
raise DistributionError(
f"distribution host cannot be resolved: {hostname}"
) from exc
if not addresses:
raise DistributionError("distribution host resolved to no addresses")
for value in addresses:
address = ipaddress.ip_address(value)
if not address.is_global:
raise DistributionError("distribution host resolves to a non-public address")
raise DistributionError(
"distribution host resolves to a non-public address"
)
def _sha256_regular_file(path: Path, *, maximum_bytes: int) -> str:
@@ -553,7 +606,9 @@ def _sha256_regular_file(path: Path, *, maximum_bytes: int) -> str:
try:
opened = os.fstat(descriptor)
if not stat.S_ISREG(opened.st_mode) or opened.st_size > maximum_bytes:
raise DistributionError(f"immutable artifact is invalid or too large: {path}")
raise DistributionError(
f"immutable artifact is invalid or too large: {path}"
)
while True:
chunk = os.read(descriptor, 1024 * 1024)
if not chunk:
@@ -582,10 +637,12 @@ def _exact_keys(
value: Mapping[str, Any],
*,
required: set[str],
optional: set[str] | None = None,
label: str,
) -> None:
optional = optional or set()
missing = sorted(required - set(value))
extra = sorted(set(value) - required)
extra = sorted(set(value) - required - optional)
if missing or extra:
detail = []
if missing:
@@ -602,7 +659,11 @@ def _token(
maximum: int,
pattern: re.Pattern[str],
) -> str:
if not isinstance(value, str) or len(value) > maximum or pattern.fullmatch(value) is None:
if (
not isinstance(value, str)
or len(value) > maximum
or pattern.fullmatch(value) is None
):
raise DistributionError(f"{label} is invalid")
return value
@@ -626,7 +687,11 @@ def _sha256(value: object, label: str) -> str:
def _digest_image(value: object, label: str) -> str:
if not isinstance(value, str) or len(value) > 300 or DIGEST_IMAGE.fullmatch(value) is None:
if (
not isinstance(value, str)
or len(value) > 300
or DIGEST_IMAGE.fullmatch(value) is None
):
raise DistributionError(f"{label} must be an OCI image pinned by sha256")
return value
@@ -635,7 +700,12 @@ def _https_url(value: object, label: str) -> str:
if not isinstance(value, str) or len(value) > 2048:
raise DistributionError(f"{label} must be an HTTPS URL")
parsed = urlsplit(value)
if parsed.scheme != "https" or not parsed.netloc or parsed.username or parsed.password:
if (
parsed.scheme != "https"
or not parsed.netloc
or parsed.username
or parsed.password
):
raise DistributionError(f"{label} must be an HTTPS URL without credentials")
return value
+26 -1
View File
@@ -10,6 +10,7 @@ import re
from typing import Any, Mapping
from urllib.parse import urlsplit
from .bundle import BACKUP_RUNTIME_ENV_KEYS
from .model import InstallationSpec, image_is_digest_pinned
@@ -55,6 +56,7 @@ _CONFIG_KEYS = (
"FILE_STORAGE_S3_BUCKET",
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
"FILE_STORAGE_S3_ENDPOINT_TRUSTED",
*BACKUP_RUNTIME_ENV_KEYS,
)
_QUEUE_NAME = re.compile(r"^[a-z][a-z0-9_.-]{0,63}$")
@@ -75,6 +77,8 @@ def render_kubernetes(
secret_name: str = "govoplan-runtime",
tls_secret_name: str = "govoplan-tls",
ingress_class_name: str | None = None,
backup_required: bool = True,
backup_evidence: Mapping[str, object] | None = None,
) -> dict[str, Any]:
"""Render runtime roles only; shared state services stay externally managed."""
@@ -199,6 +203,8 @@ def render_kubernetes(
environment,
"MIGRATION",
),
backup_required=backup_required,
backup_evidence=backup_evidence,
),
]
for pool in worker_pools:
@@ -765,9 +771,28 @@ def _migration_job(
secret_name: str,
service_account: str,
database_environment: Mapping[str, str],
backup_required: bool,
backup_evidence: Mapping[str, object] | None,
) -> dict[str, Any]:
job_labels = {**labels, "app.kubernetes.io/component": "migration"}
job_name = _name_with_suffix(name, f"migrate-{release_key}")
backup_annotations = {
"govoplan.add-ideas.de/backup-required": str(backup_required).lower(),
}
if backup_evidence is not None:
backup_annotations.update(
{
"govoplan.add-ideas.de/backup-evidence-sha256": str(
backup_evidence["evidence_sha256"]
),
"govoplan.add-ideas.de/recovery-point": str(
backup_evidence["recovery_point_id"]
),
"govoplan.add-ideas.de/restore-drill": str(
backup_evidence["restore_drill_id"]
),
}
)
return {
"apiVersion": "batch/v1",
"kind": "Job",
@@ -777,7 +802,7 @@ def _migration_job(
"labels": job_labels,
"annotations": {
"govoplan.add-ideas.de/recovery-mode": "forward-recovery",
"govoplan.add-ideas.de/backup-required": "true",
**backup_annotations,
"argocd.argoproj.io/sync-wave": "-1",
},
},
@@ -3,6 +3,7 @@
from __future__ import annotations
from dataclasses import asdict, dataclass
import hashlib
import json
import os
from pathlib import Path
@@ -18,6 +19,11 @@ from urllib.error import HTTPError, URLError
from urllib.parse import urlsplit
from urllib.request import Request, urlopen
from .backup_evidence import (
MAX_BACKUP_EVIDENCE_BYTES,
MAX_BACKUP_KEYRING_BYTES,
verify_backup_evidence,
)
from .bundle import (
BundlePaths,
canonical_json,
@@ -33,8 +39,11 @@ from .distribution import (
MAX_KEYRING_BYTES,
MAX_MANIFEST_BYTES,
DistributionError,
canonical_json as canonical_distribution_json,
decode_json_bytes,
file_sha256,
load_bounded_json,
read_bounded_bytes,
verify_manifest,
verify_manifest_binding,
)
@@ -232,6 +241,9 @@ def static_checks(spec: InstallationSpec, paths: BundlePaths) -> tuple[Check, ..
)
checks.extend(_distribution_checks(spec, paths))
checks.extend(
_backup_evidence_checks(spec, paths, receipt=_read_receipt(paths.receipt))
)
values = read_env(paths.env)
required = {"MASTER_KEY_B64", "DATABASE_URL"}
@@ -350,6 +362,189 @@ def static_checks(spec: InstallationSpec, paths: BundlePaths) -> tuple[Check, ..
return tuple(checks)
def release_change_requires_backup(
spec: InstallationSpec,
receipt: Mapping[str, object],
) -> bool:
if spec.profile != "self-hosted" or not receipt:
return False
previous = receipt.get("release")
if not isinstance(previous, Mapping):
return True
desired = {
"channel": spec.release.channel,
"version": spec.release.version,
"manifest_sha256": spec.release.manifest_sha256,
"composition_sha256": spec.release.composition_sha256,
"api_image": spec.release.api_image,
"web_image": spec.release.web_image,
}
return any(previous.get(key) != value for key, value in desired.items())
def verify_stored_backup_evidence(
spec: InstallationSpec,
paths: BundlePaths,
*,
receipt: Mapping[str, object],
) -> dict[str, object]:
verification = load_bounded_json(
paths.backup_verification,
maximum_bytes=64 * 1024,
)
expected_fields = {
"schema_version",
"evidence_sha256",
"keyring_sha256",
"signature_key_id",
"verified_at",
"evidence_id",
"recovery_point_id",
"restore_drill_id",
"release_manifest_sha256",
"captured_at",
"expires_at",
"restore_started_at",
"restore_completed_at",
"measured_rpo_seconds",
"measured_rto_seconds",
"component_count",
}
if set(verification) != expected_fields or verification.get("schema_version") != 1:
raise DistributionError("backup verification receipt is malformed")
encoded_evidence = read_bounded_bytes(
paths.backup_evidence,
maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES,
)
encoded_keyring = read_bounded_bytes(
paths.backup_keyring,
maximum_bytes=MAX_BACKUP_KEYRING_BYTES,
)
evidence = decode_json_bytes(encoded_evidence, label="backup evidence")
keyring = decode_json_bytes(encoded_keyring, label="backup keyring")
if encoded_evidence != canonical_distribution_json(evidence):
raise DistributionError("stored backup evidence is not canonical JSON")
if encoded_keyring != canonical_distribution_json(keyring):
raise DistributionError("stored backup keyring is not canonical JSON")
evidence_digest = hashlib.sha256(encoded_evidence).hexdigest()
keyring_digest = hashlib.sha256(encoded_keyring).hexdigest()
if evidence_digest != verification.get("evidence_sha256"):
raise DistributionError("stored backup evidence digest has changed")
if keyring_digest != verification.get("keyring_sha256"):
raise DistributionError("stored backup keyring digest has changed")
previous_release = receipt.get("release") if receipt else None
expected_release: Mapping[str, object] = (
previous_release
if isinstance(previous_release, Mapping)
else {
"channel": spec.release.channel,
"version": spec.release.version,
"manifest_sha256": spec.release.manifest_sha256,
"composition_sha256": spec.release.composition_sha256,
"api_image": spec.release.api_image,
"web_image": spec.release.web_image,
}
)
summary = verify_backup_evidence(
evidence,
keyring,
installation_id=spec.installation_id,
profile=spec.profile,
release=expected_release,
)
expected_summary = {
"signature_key_id": verification.get("signature_key_id"),
"evidence_id": verification.get("evidence_id"),
"recovery_point_id": verification.get("recovery_point_id"),
"restore_drill_id": verification.get("restore_drill_id"),
"captured_at": verification.get("captured_at"),
"expires_at": verification.get("expires_at"),
"restore_started_at": verification.get("restore_started_at"),
"restore_completed_at": verification.get("restore_completed_at"),
"measured_rpo_seconds": verification.get("measured_rpo_seconds"),
"measured_rto_seconds": verification.get("measured_rto_seconds"),
"component_count": verification.get("component_count"),
}
for field, expected in expected_summary.items():
if summary.get(field) != expected:
raise DistributionError(
f"backup verification receipt does not match {field!r}"
)
if expected_release.get("manifest_sha256") != verification.get(
"release_manifest_sha256"
):
raise DistributionError("backup verification receipt has another release")
return {
**summary,
"evidence_sha256": evidence_digest,
"keyring_sha256": keyring_digest,
}
def _backup_evidence_checks(
spec: InstallationSpec,
paths: BundlePaths,
*,
receipt: Mapping[str, object],
) -> tuple[Check, ...]:
required = release_change_requires_backup(spec, receipt)
available = all(
path.is_file()
for path in (
paths.backup_evidence,
paths.backup_keyring,
paths.backup_verification,
)
)
if not available:
return (
Check(
"backup.migration_gate",
"error"
if required
else "warning"
if spec.profile == "self-hosted"
else "ok",
(
"A release-changing migration has no verified coordinated backup evidence."
if required
else "No current coordinated backup evidence is adopted."
),
(
"Run verify-backup --adopt after an isolated restore drill."
if spec.profile == "self-hosted"
else ""
),
),
)
try:
summary = verify_stored_backup_evidence(
spec,
paths,
receipt=receipt,
)
except (DistributionError, OSError) as exc:
return (
Check(
"backup.migration_gate",
"error" if required else "warning",
f"Coordinated backup evidence is invalid: {exc}",
"Adopt fresh signed evidence for the currently applied release.",
),
)
return (
Check(
"backup.migration_gate",
"ok",
(
"Release migration is backed by recovery point "
f"{summary['recovery_point_id']} and restore drill "
f"{summary['restore_drill_id']}."
),
),
)
def _ingress_configuration_checks(
spec: InstallationSpec,
paths: BundlePaths,
@@ -29,6 +29,9 @@ _BUNDLE_FILES = (
"existing-proxy.json",
"distribution-manifest.json",
"distribution-keyring.json",
"backup-evidence.json",
"backup-keyring.json",
"backup-verification.json",
"receipt.json",
)
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""Sign and validate provider-produced GovOPlaN backup evidence."""
from __future__ import annotations
import argparse
import base64
import hashlib
from pathlib import Path
import re
import stat
from typing import Any
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
from govoplan_deploy.backup_evidence import (
MAX_BACKUP_EVIDENCE_BYTES,
load_backup_keyring,
verify_backup_evidence,
)
from govoplan_deploy.bundle import atomic_write
from govoplan_deploy.distribution import (
canonical_json,
canonical_signed_payload,
load_bounded_json,
)
KEY_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
def main() -> int:
parser = argparse.ArgumentParser(
description=(
"Sign a provider-produced backup/restore evidence document and "
"validate it against an independently managed public keyring."
)
)
parser.add_argument("--input", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
parser.add_argument("--trusted-keyring", type=Path, required=True)
parser.add_argument(
"--signing-key",
action="append",
required=True,
metavar="KEY_ID=PRIVATE_PEM",
help="Ed25519 signer; may be repeated during key rotation.",
)
parser.add_argument(
"--replace-signatures",
action="store_true",
help="Replace existing signatures instead of rejecting the input.",
)
args = parser.parse_args()
source = args.input.expanduser().resolve()
payload = load_bounded_json(source, maximum_bytes=MAX_BACKUP_EVIDENCE_BYTES)
existing = payload.get("signatures")
if existing not in (None, []) and not args.replace_signatures:
raise SystemExit("input already contains signatures; use --replace-signatures")
signers = [_load_signer(value) for value in args.signing_key]
if len({key_id for key_id, _ in signers}) != len(signers):
raise SystemExit("duplicate signing key id")
payload["signatures"] = []
signed = canonical_signed_payload(payload)
payload["signatures"] = [
{
"key_id": key_id,
"algorithm": "ed25519",
"value": base64.b64encode(private_key.sign(signed)).decode("ascii"),
}
for key_id, private_key in signers
]
keyring = load_backup_keyring(args.trusted_keyring.expanduser().resolve())
release = payload.get("release")
if not isinstance(release, dict):
raise SystemExit("input release must be an object")
verify_backup_evidence(
payload,
keyring,
installation_id=str(payload.get("installation_id") or ""),
profile=str(
_object(payload.get("deployment_subject"), "deployment_subject").get(
"profile"
)
or ""
),
release=release,
)
encoded = canonical_json(payload)
output = args.output.expanduser().resolve()
atomic_write(output, encoded, mode=0o600)
print(f"Wrote {output}")
print(f"SHA256 {hashlib.sha256(encoded).hexdigest()}")
return 0
def _load_signer(value: str) -> tuple[str, Ed25519PrivateKey]:
key_id, separator, raw_path = value.partition("=")
if not separator or KEY_ID.fullmatch(key_id) is None or not raw_path:
raise SystemExit("--signing-key must use KEY_ID=/path/to/private.pem")
path = Path(raw_path).expanduser().resolve()
mode = stat.S_IMODE(path.stat().st_mode)
if mode & 0o077:
raise SystemExit(
f"private signing key must not be group/world accessible: {path}"
)
private_key = serialization.load_pem_private_key(path.read_bytes(), password=None)
if not isinstance(private_key, Ed25519PrivateKey):
raise SystemExit(f"signing key is not Ed25519: {path}")
return key_id, private_key
def _object(value: object, label: str) -> dict[str, Any]:
if not isinstance(value, dict):
raise SystemExit(f"input {label} must be an object")
return value
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Configure and verify protected GovOPlaN package-release boundaries."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import sys
from gitea_common import (
GiteaClient,
GiteaError,
RepoTarget,
load_dotenv,
org_path,
quote_path,
repo_path,
require_token,
)
META_ROOT = Path(__file__).resolve().parents[2]
REQUIRED_SECRETS = {"GOVOPLAN_PACKAGE_USERNAME", "GOVOPLAN_PACKAGE_TOKEN"}
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--url", default="https://git.add-ideas.de")
parser.add_argument("--owner", default="GovOPlaN")
parser.add_argument("--team", default="Owners")
parser.add_argument("--pattern", default="v*")
parser.add_argument("--env-file", type=Path)
parser.add_argument("--apply", action="store_true")
return parser
def package_repositories() -> tuple[str, ...]:
inventory = json.loads((META_ROOT / "repositories.json").read_text(encoding="utf-8"))
values = ["govoplan"]
for item in inventory["repositories"]:
name = str(item["name"])
repository = META_ROOT.parent / str(item["path"])
if name.startswith("govoplan-") and (repository / "pyproject.toml").is_file():
values.append(name)
return tuple(sorted(set(values)))
def configure(
client: GiteaClient,
*,
owner: str,
team: str,
pattern: str,
apply: bool,
) -> tuple[str, ...]:
missing: list[str] = []
expected = {
"name_pattern": pattern,
"whitelist_teams": [team],
"whitelist_usernames": [],
}
for repository in package_repositories():
path = repo_path(owner, repository, "/tag_protections")
protections = client.request_json("GET", path)
matching = [
item
for item in protections
if isinstance(item, dict) and item.get("name_pattern") == pattern
]
if len(matching) == 1 and _matches(matching[0], expected):
print(f"protected {repository}:{pattern}")
continue
missing.append(repository)
if not apply:
print(f"would protect {repository}:{pattern}")
continue
if len(matching) == 1:
protection_id = matching[0].get("id")
client.request_json(
"PATCH",
f"{path}/{quote_path(str(protection_id))}",
body=expected,
)
print(f"updated {repository}:{pattern}")
elif not matching:
client.request_json("POST", path, body=expected)
print(f"created {repository}:{pattern}")
else:
raise GiteaError(f"{repository} has duplicate {pattern!r} tag protections")
return tuple(missing)
def _matches(value: dict[str, object], expected: dict[str, object]) -> bool:
return (
value.get("name_pattern") == expected["name_pattern"]
and sorted(value.get("whitelist_teams") or []) == expected["whitelist_teams"]
and sorted(value.get("whitelist_usernames") or []) == expected["whitelist_usernames"]
)
def main() -> int:
args = build_parser().parse_args()
try:
load_dotenv(args.env_file)
token = require_token()
target = RepoTarget(base_url=args.url, owner=args.owner, repo="govoplan")
with GiteaClient(target, token) as client:
mismatches = configure(
client,
owner=args.owner,
team=args.team,
pattern=args.pattern,
apply=args.apply,
)
secrets = client.request_json(
"GET", org_path(args.owner, "/actions/secrets"), query={"limit": 50}
)
names = {
str(item.get("name") or "")
for item in secrets
if isinstance(item, dict)
}
missing_secrets = sorted(REQUIRED_SECRETS - names)
if missing_secrets:
print(
"Missing organization Actions secrets: " + ", ".join(missing_secrets),
file=sys.stderr,
)
unresolved = (bool(mismatches) and not args.apply) or bool(missing_secrets)
if unresolved:
return 1
print("Package release protection and credential names are configured.")
return 0
except (GiteaError, OSError, ValueError, json.JSONDecodeError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
+302
View File
@@ -0,0 +1,302 @@
#!/usr/bin/env python3
"""Dispatch protected package releases required by the govoplan meta-package."""
from __future__ import annotations
import argparse
from dataclasses import dataclass
import json
from pathlib import Path
import re
import subprocess
import sys
import tomllib
from gitea_common import (
GiteaClient,
GiteaError,
RepoTarget,
load_dotenv,
org_path,
quote_path,
repo_path,
require_token,
)
META_ROOT = Path(__file__).resolve().parents[2]
META_PROJECT = META_ROOT / "packages" / "govoplan-meta" / "pyproject.toml"
WORKFLOW_ID = "module-package-release.yml"
EXACT_REQUIREMENT = re.compile(
r"^(?P<name>govoplan-[a-z0-9-]+)(?:\[[a-z0-9_,.-]+\])?==(?P<version>[0-9]+\.[0-9]+\.[0-9]+)$"
)
ACTIVE_STATES = {"queued", "waiting", "in_progress", "running"}
@dataclass(frozen=True, slots=True)
class PackageTarget:
distribution: str
version: str
repository: str
tag_exists: bool
has_webui: bool
@property
def tag(self) -> str:
return f"v{self.version}"
@property
def webui_package(self) -> str | None:
if not self.has_webui:
return None
return f"@govoplan/{self.distribution.removeprefix('govoplan-')}-webui"
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--url", default="https://git.add-ideas.de")
parser.add_argument("--owner", default="GovOPlaN")
parser.add_argument("--env-file", type=Path)
parser.add_argument(
"--repository",
action="append",
default=[],
help="Limit dispatch to one repository; repeat as needed.",
)
parser.add_argument(
"--verify-existing",
action="store_true",
help="Also rerun exact versions already present in both registries.",
)
parser.add_argument("--apply", action="store_true")
return parser
def package_targets(project_path: Path = META_PROJECT) -> tuple[PackageTarget, ...]:
project = tomllib.loads(project_path.read_text(encoding="utf-8"))["project"]
requirements = list(project.get("dependencies") or [])
requirements.extend(project.get("optional-dependencies", {}).get("full") or [])
parsed: dict[str, str] = {}
for requirement in requirements:
match = EXACT_REQUIREMENT.fullmatch(str(requirement))
if match is None:
raise ValueError(
f"Meta-package requirement is not an exact GovOPlaN version: {requirement!r}"
)
name = match.group("name")
version = match.group("version")
previous = parsed.setdefault(name, version)
if previous != version:
raise ValueError(f"Meta-package selects conflicting versions for {name}")
targets: list[PackageTarget] = []
for distribution, version in sorted(parsed.items()):
repository = distribution
repository_root = META_ROOT.parent / repository
if not (repository_root / ".git").is_dir():
raise ValueError(f"Package repository is not checked out: {repository}")
tag = f"v{version}"
tag_exists = _tag_exists(repository_root, tag)
has_webui = (
_tag_has_path(repository_root, tag, "webui/package.json")
if tag_exists
else False
)
targets.append(
PackageTarget(
distribution=distribution,
version=version,
repository=repository,
tag_exists=tag_exists,
has_webui=has_webui,
)
)
return tuple(targets)
def _tag_exists(repository: Path, tag: str) -> bool:
result = subprocess.run(
(
"git",
"-C",
str(repository),
"rev-parse",
"--verify",
"--quiet",
f"refs/tags/{tag}",
),
check=False,
capture_output=True,
text=True,
)
if result.returncode not in {0, 1}:
raise ValueError(
f"Could not inspect {repository.name}:{tag}: {result.stderr.strip()}"
)
return result.returncode == 0
def _tag_has_path(repository: Path, tag: str, path: str) -> bool:
result = subprocess.run(
("git", "-C", str(repository), "cat-file", "-e", f"{tag}:{path}"),
check=False,
capture_output=True,
text=True,
)
if result.returncode not in {0, 128}:
raise ValueError(
f"Could not inspect {repository.name}:{tag}:{path}: {result.stderr.strip()}"
)
return result.returncode == 0
def _published_packages(
client: GiteaClient, *, owner: str, package_type: str
) -> set[tuple[str, str]]:
values = client.paginate(
f"/packages/{quote_path(owner)}",
query={"type": package_type, "q": "govoplan"},
)
return {
(str(item.get("name") or ""), str(item.get("version") or ""))
for item in values
if item.get("type") == package_type
}
def _has_active_run(
client: GiteaClient, *, owner: str, repository: str
) -> bool:
payload = client.request_json(
"GET",
repo_path(
owner,
repository,
f"/actions/workflows/{quote_path(WORKFLOW_ID)}/runs",
),
query={"limit": 10},
)
runs = payload.get("workflow_runs") if isinstance(payload, dict) else None
return isinstance(runs, list) and any(
isinstance(run, dict) and str(run.get("status") or "") in ACTIVE_STATES
for run in runs
)
def dispatch(
client: GiteaClient,
*,
owner: str,
targets: tuple[PackageTarget, ...],
published_pypi: set[tuple[str, str]],
published_npm: set[tuple[str, str]],
verify_existing: bool,
apply: bool,
) -> tuple[int, int, int]:
dispatched = 0
active = 0
complete = 0
for target in targets:
wheel_exists = (target.distribution, target.version) in published_pypi
npm_exists = target.webui_package is None or (
target.webui_package,
target.version,
) in published_npm
if wheel_exists and npm_exists and not verify_existing:
complete += 1
print(f"complete {target.repository}:{target.tag}")
continue
if _has_active_run(client, owner=owner, repository=target.repository):
active += 1
print(f"active {target.repository}:{target.tag}")
continue
action = "dispatching" if apply else "would dispatch"
print(
f"{action} {target.repository}:{target.tag} "
f"(wheel={'present' if wheel_exists else 'missing'}, "
f"webui={'present' if npm_exists else 'missing'})"
)
if apply:
client.request_json(
"POST",
repo_path(
owner,
target.repository,
f"/actions/workflows/{quote_path(WORKFLOW_ID)}/dispatches",
),
body={"ref": "main", "inputs": {"release_tag": target.tag}},
)
dispatched += 1
return dispatched, active, complete
def main() -> int:
args = build_parser().parse_args()
try:
load_dotenv(args.env_file)
token = require_token()
targets = package_targets()
selected = set(args.repository)
if selected:
known = {target.repository for target in targets}
unknown = sorted(selected - known)
if unknown:
raise ValueError(
"Unknown meta-package repositories: " + ", ".join(unknown)
)
targets = tuple(
target for target in targets if target.repository in selected
)
missing_tags = [
f"{target.repository}:{target.tag}"
for target in targets
if not target.tag_exists
]
if missing_tags:
raise ValueError(
"Meta-package release tags are missing: " + ", ".join(missing_tags)
)
target = RepoTarget(base_url=args.url, owner=args.owner, repo="govoplan")
with GiteaClient(target, token) as client:
secrets = client.request_json(
"GET", org_path(args.owner, "/actions/secrets"), query={"limit": 50}
)
secret_names = {
str(item.get("name") or "")
for item in secrets
if isinstance(item, dict)
}
required = {"GOVOPLAN_PACKAGE_USERNAME", "GOVOPLAN_PACKAGE_TOKEN"}
if not required <= secret_names:
raise ValueError(
"Organization package publisher secrets are not configured"
)
published_pypi = _published_packages(
client, owner=args.owner, package_type="pypi"
)
published_npm = _published_packages(
client, owner=args.owner, package_type="npm"
)
counts = dispatch(
client,
owner=args.owner,
targets=targets,
published_pypi=published_pypi,
published_npm=published_npm,
verify_existing=args.verify_existing,
apply=args.apply,
)
action = "dispatched" if args.apply else "planned"
print(
f"Package set {action}: {counts[0]}; active: {counts[1]}; "
f"already complete: {counts[2]}."
)
return 0
except (GiteaError, OSError, ValueError, json.JSONDecodeError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
+302 -148
View File
@@ -155,44 +155,39 @@
"repository": "govoplan-access"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/admin/service-accounts",
"rationale": "Service-account lifecycle is implemented but its administration UI is tracked separately.",
"repository": "govoplan-access",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/18"
"rationale": "Access administration lists service accounts and opens their lifecycle and credential manager.",
"repository": "govoplan-access"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/admin/service-accounts",
"rationale": "Service-account lifecycle is implemented but its administration UI is tracked separately.",
"repository": "govoplan-access",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/18"
"rationale": "Access administration creates service accounts through the governed editor.",
"repository": "govoplan-access"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/admin/service-accounts/{}",
"rationale": "Service-account lifecycle is implemented but its administration UI is tracked separately.",
"repository": "govoplan-access",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/18"
"rationale": "Access administration refreshes service-account state before governed mutations.",
"repository": "govoplan-access"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "PATCH",
"path": "/admin/service-accounts/{}",
"rationale": "Service-account lifecycle is implemented but its administration UI is tracked separately.",
"repository": "govoplan-access",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/18"
"rationale": "Access administration edits, activates, and deactivates service accounts with revision checks.",
"repository": "govoplan-access"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/admin/service-accounts/{}/retire",
"rationale": "Service-account lifecycle is implemented but its administration UI is tracked separately.",
"repository": "govoplan-access",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-access/issues/18"
"rationale": "Access administration exposes separately confirmed retirement and credential revocation.",
"repository": "govoplan-access"
},
{
"category": "intentionally_headless",
@@ -237,44 +232,39 @@
"repository": "govoplan-admin"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/approvals/templates",
"rationale": "Approval-template and escalation administration UI is tracked separately.",
"repository": "govoplan-approvals",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/2"
"rationale": "Approval administration lists immutable template revisions.",
"repository": "govoplan-approvals"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/approvals/templates",
"rationale": "Approval-template and escalation administration UI is tracked separately.",
"repository": "govoplan-approvals",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/2"
"rationale": "Approval administration creates validated draft templates.",
"repository": "govoplan-approvals"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "PUT",
"path": "/approvals/templates/{}",
"rationale": "Approval-template and escalation administration UI is tracked separately.",
"repository": "govoplan-approvals",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/2"
"rationale": "Approval administration revises templates through optimistic immutable revisions.",
"repository": "govoplan-approvals"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/approvals/templates/{}/publish",
"rationale": "Approval-template and escalation administration UI is tracked separately.",
"repository": "govoplan-approvals",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/2"
"rationale": "Approval administration publishes a draft only after an explicit confirmation.",
"repository": "govoplan-approvals"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/approvals/{}/escalate",
"rationale": "Approval-template and escalation administration UI is tracked separately.",
"repository": "govoplan-approvals",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-approvals/issues/2"
"rationale": "The request dialog exposes escalation only for due pending requests and authorized administrators.",
"repository": "govoplan-approvals"
},
{
"category": "ui_reachable",
@@ -353,6 +343,20 @@
"rationale": "Published integration, interoperability, public-participant, or health endpoint.",
"repository": "govoplan-calendar"
},
{
"category": "intentionally_headless",
"method": "POST",
"path": "/bootstrap/first-admin",
"rationale": "The restricted first-run endpoint is consumed by the local bootstrap handoff and can only create the first durable administrator.",
"repository": "govoplan-core"
},
{
"category": "intentionally_headless",
"method": "GET",
"path": "/bootstrap/status",
"rationale": "The local bootstrap handoff reads only minimum first-run readiness before a normal authenticated shell exists.",
"repository": "govoplan-core"
},
{
"category": "public_integration",
"method": "GET",
@@ -368,12 +372,11 @@
"repository": "govoplan-calendar"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/campaigns/{}/archive",
"rationale": "Campaign archive lifecycle UI remains tracked by the archive policy issue.",
"repository": "govoplan-campaign",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/26"
"rationale": "The Campaign overview exposes the governed archive action with permission checks and an evidence-retention confirmation.",
"repository": "govoplan-campaign"
},
{
"category": "intentionally_headless",
@@ -389,6 +392,13 @@
"rationale": "Campaign delivery diagnostic/status API is retained for bounded support and automation consumers.",
"repository": "govoplan-campaign"
},
{
"category": "worker_internal",
"method": "POST",
"path": "/campaigns/operations/artifacts/reconcile",
"rationale": "Privileged, bounded artifact recovery operation used by operators and recovery automation rather than an end-user surface.",
"repository": "govoplan-campaign"
},
{
"category": "ui_reachable",
"method": "PUT",
@@ -571,6 +581,13 @@
"rationale": "The shared ownership UI uses a dynamic transfer-action path that the static string scan cannot resolve.",
"repository": "govoplan-core"
},
{
"category": "intentionally_headless",
"method": "GET",
"path": "/platform/interface-catalog",
"rationale": "Authorized module and system administrators consume the read-only control-plane inventory directly or through Ops/Docs projections.",
"repository": "govoplan-core"
},
{
"category": "compatibility",
"method": "GET",
@@ -656,156 +673,172 @@
"repository": "govoplan-docs"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/encryption/disable-preflight",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The Encryption administration panel displays disable readiness and bounded blocking envelope references.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/encryption/envelopes",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Feature modules register envelopes while retaining content ownership; the operator UI must not construct feature content envelopes.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/encryption/envelopes",
"rationale": "The Encryption administration panel consumes the bounded, secret-free envelope summary contract.",
"repository": "govoplan-encryption"
},
{
"category": "intentionally_headless",
"method": "GET",
"path": "/encryption/envelopes/{}",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Owning modules resolve a specific full envelope through the capability/API contract; the operator UI uses the secret-free summary projection.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/encryption/migrations",
"rationale": "The Encryption administration panel displays bounded migration state and evidence counts.",
"repository": "govoplan-encryption"
},
{
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/migrations",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Encryption custodians can authorize a two-phase migration from a bounded envelope summary.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/encryption/migrations/{}/outcome",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Only the owning module or governed worker can attest the durable content outcome and exact target envelope.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/migrations/{}/reconcile",
"rationale": "Encryption custodians can re-read recorded provider migration state without declaring an outcome.",
"repository": "govoplan-encryption"
},
{
"category": "ui_reachable",
"method": "GET",
"path": "/encryption/recoveries",
"rationale": "The Encryption administration panel displays bounded recovery requests, quorum, expiry, and state.",
"repository": "govoplan-encryption"
},
{
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/recoveries",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Authorized custodians can request an expiring high-assurance recovery ceremony.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/recoveries/{}/decision",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The recovery table exposes explicit approve/reject decisions with assurance, reason, and optimistic revision.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/encryption/vaults",
"rationale": "The Encryption administration panel consumes the bounded, secret-free vault summary contract.",
"repository": "govoplan-encryption"
},
{
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/vaults",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Encryption custodians can create a governed vault without entering or receiving raw key material.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "GET",
"path": "/encryption/vaults/{}",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "Capability consumers may resolve the complete vault reference; the operator UI uses the secret-free summary projection.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/vaults/{}/destruction",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The vault action group schedules destructive key lifecycle operations with explicit consequences.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/vaults/{}/reconcile",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The vault action group reconciles an outcome-unknown provider operation.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/vaults/{}/revoke",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The vault action group revokes the current key with policy, assurance, reason, and revision evidence.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/encryption/vaults/{}/rotate",
"rationale": "Encryption custodian, lifecycle, and recovery administration UI is tracked separately.",
"repository": "govoplan-encryption",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-encryption/issues/4"
"rationale": "The vault action group rotates the current key with policy, assurance, reason, and revision evidence.",
"repository": "govoplan-encryption"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/files/integrity/findings/{}/cleanup",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "The Files integrity panel requires a dry-run preview and separate confirmation before cleanup.",
"repository": "govoplan-files"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/files/integrity/findings/{}/recheck",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "The Files integrity panel rechecks findings against their displayed revision.",
"repository": "govoplan-files"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/files/integrity/scans",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "The Files integrity administration panel lists bounded reconciliation scans.",
"repository": "govoplan-files"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/files/integrity/scans",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "Authorized Files operators can create a bounded integrity scan from administration.",
"repository": "govoplan-files"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/files/integrity/scans/{}/findings",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "The Files integrity panel displays findings and blocker evidence for the selected scan.",
"repository": "govoplan-files"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/files/integrity/scans/{}/run",
"rationale": "File-integrity operations UI is tracked separately.",
"repository": "govoplan-files",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-files/issues/40"
"rationale": "The Files integrity panel runs and resumes bounded batches with stale-action protection.",
"repository": "govoplan-files"
},
{
"category": "compatibility",
@@ -857,60 +890,53 @@
"repository": "govoplan-identity"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/identity-trust/assurance/check",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Assurance checks are capability operations performed by protected module workflows rather than direct user commands.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/identity-trust/assurance/evidence",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Trusted assurance providers record evidence through this capability endpoint; users inspect the resulting projection.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "GET",
"path": "/identity-trust/device-keys",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Identity Trust settings and administration list permission-filtered public device keys.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/identity-trust/device-keys",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Device onboarding registers public key material through the trust capability; the UI manages registered keys without handling private material.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/identity-trust/device-keys/{}/revoke",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Authorized users and trust officers revoke public device keys through a consequence-aware confirmation.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "ui_reachable",
"method": "POST",
"path": "/identity-trust/epochs/rotate",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Identity Trust administration exposes governed epoch rotation with history and consequence explanations.",
"repository": "govoplan-identity-trust"
},
{
"category": "missing_ui",
"category": "intentionally_headless",
"method": "POST",
"path": "/identity-trust/key-access/decide",
"rationale": "Identity-assurance and device-key administration UI is tracked separately.",
"repository": "govoplan-identity-trust",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust/issues/2"
"rationale": "Protected modules request immutable key-access decisions through the capability; trust officers inspect the resulting decision projection.",
"repository": "govoplan-identity-trust"
},
{
"category": "ui_reachable",
@@ -1581,6 +1607,134 @@
"path": "/workflow/definitions/{}/triggers",
"rationale": "This capability-first module intentionally exposes a headless API for other modules and integrations.",
"repository": "govoplan-workflow-engine"
},
{
"category": "ui_reachable",
"method": "POST",
"path": "/addresses/contact-merges/{}/split",
"rationale": "The Addresses WebUI constructs the selected merge id and recovery action dynamically.",
"repository": "govoplan-addresses"
},
{
"category": "ui_reachable",
"method": "POST",
"path": "/addresses/contact-merges/{}/undo",
"rationale": "The Addresses WebUI constructs the selected merge id and recovery action dynamically.",
"repository": "govoplan-addresses"
},
{
"category": "intentionally_headless",
"method": "POST",
"path": "/addresses/contact-point-snapshots",
"rationale": "Campaign and other modules consume the governed contact-point snapshot capability without a direct Addresses screen.",
"repository": "govoplan-addresses"
},
{
"category": "intentionally_headless",
"method": "GET",
"path": "/addresses/contact-point-snapshots/{}",
"rationale": "Campaign and other modules consume the governed contact-point snapshot capability without a direct Addresses screen.",
"repository": "govoplan-addresses"
},
{
"category": "intentionally_headless",
"method": "POST",
"path": "/addresses/contact-point-sources/preview",
"rationale": "This capability endpoint previews a module-supplied contact source and is consumed by integrations rather than a direct screen.",
"repository": "govoplan-addresses"
},
{
"category": "intentionally_headless",
"method": "POST",
"path": "/addresses/contact-points/resolve",
"rationale": "This governed recipient-resolution endpoint is consumed by Campaign and other modules.",
"repository": "govoplan-addresses"
},
{
"category": "intentionally_headless",
"method": "GET",
"path": "/addresses/contacts/{}/redirect",
"rationale": "Stored references and module capabilities resolve merged-contact redirects without a direct user action.",
"repository": "govoplan-addresses"
},
{
"category": "missing_ui",
"method": "GET",
"path": "/addresses/imports/{}",
"rationale": "The import flow can create, apply, and roll back a run, but the WebUI does not yet resume a saved run by id after navigation or reload.",
"repository": "govoplan-addresses",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-addresses/issues/22"
},
{
"category": "ui_reachable",
"method": "GET",
"path": "/campaigns/{}/versions/{}/print-output/download",
"rationale": "The Campaign WebUI validates and follows the build artifact's server-provided download path.",
"repository": "govoplan-campaign"
},
{
"category": "missing_ui",
"method": "GET",
"path": "/relationships",
"rationale": "IDM documents typed relationships but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "POST",
"path": "/relationships",
"rationale": "IDM documents typed relationships but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "PATCH",
"path": "/relationships/{}",
"rationale": "IDM documents typed relationships but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "POST",
"path": "/relationships/{}/revoke",
"rationale": "IDM documents typed relationships but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "GET",
"path": "/typed-groups",
"rationale": "IDM documents typed groups but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "POST",
"path": "/typed-groups",
"rationale": "IDM documents typed groups but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "PATCH",
"path": "/typed-groups/{}",
"rationale": "IDM documents typed groups but its current administration page exposes function assignments only.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
},
{
"category": "missing_ui",
"method": "GET",
"path": "/typed-groups/{}/memberships",
"rationale": "IDM lacks the typed-group membership explanation surface for this existing API.",
"repository": "govoplan-idm",
"tracking_issue": "https://git.add-ideas.de/GovOPlaN/govoplan-idm/issues/11"
}
],
"schema_version": 1
+230 -7
View File
@@ -2,6 +2,7 @@
import fs from "node:fs";
import path from "node:path";
import { createHash } from "node:crypto";
import { pathToFileURL } from "node:url";
const [metaRootArgument] = process.argv.slice(2);
@@ -60,9 +61,19 @@ const helpAttributes = new Set([
"helperText",
"helpText"
]);
const actionComponentPattern = /(?:Action|Button|Link)$/;
const contributionTypes = new Map([
["AdminSectionsUiCapability", "admin_section"],
["DashboardWidgetsUiCapability", "widget"],
["OrganizationFunctionActionsUiCapability", "action"],
["SearchContextsUiCapability", "search_object"],
["SettingsSectionsUiCapability", "setting"],
["WizardDirectoriesUiCapability", "workflow_directory"]
]);
const result = {
fields: [],
actions: [],
labels: [],
visibleText: [],
translationCatalog: {},
@@ -71,7 +82,8 @@ const result = {
routes: [],
navigation: [],
frontendApiReferences: [],
uiCapabilities: []
uiCapabilities: [],
contributions: []
};
for (const repository of repositoryCatalog.repositories) {
@@ -115,6 +127,7 @@ function inspectSource(repository, sourceRoot, sourcePath) {
sourcePath.endsWith(".tsx") ? ts.ScriptKind.TSX : ts.ScriptKind.TS
);
const relativeFile = path.relative(path.join(workspaceRoot, repository), sourcePath);
const identityCounters = new Map();
function location(node) {
const position = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile));
@@ -178,6 +191,7 @@ function inspectSource(repository, sourceRoot, sourcePath) {
const isField =
fieldComponents.has(component) ||
(fieldComponentPattern.test(component) && component !== "FormField");
inspectAction(node, component, attributes, locate);
if (!isField) return;
const parentFormField = nearestFormField(node);
@@ -191,8 +205,25 @@ function inspectSource(repository, sourceRoot, sourcePath) {
null;
const help = firstAttribute(attributes, helpAttributes) ??
firstAttribute(parentAttributes, helpAttributes);
const hasHelp = hasAnyAttribute(attributes, helpAttributes) ||
hasAnyAttribute(parentAttributes, helpAttributes);
const explicitId = firstAttribute(
attributes,
new Set(["interfaceId", "data-interface-id", "id", "name", "field"])
);
const context = nearestNamedContext(node);
const stableId = sourceIdentity(
"field",
node,
component,
explicitId ?? label ?? attributes.get("placeholder") ?? "field"
);
result.fields.push({
...locate(node),
id: stableId,
explicitId,
idSource: explicitId === null ? "source_anchor" : "explicit",
context,
component,
name:
attributes.get("name") ??
@@ -202,8 +233,102 @@ function inspectSource(repository, sourceRoot, sourcePath) {
label,
placeholder: attributes.get("placeholder") ?? null,
help: help ?? null,
helpCandidate: help === null
helpId: hasHelp ? `${stableId}.help` : null,
helpDynamic: hasHelp && help === null,
helpCandidate: !hasHelp
});
}
function inspectAction(node, component, attributes, locate) {
const lowerComponent = component.toLowerCase();
const inputType = attributes.get("type")?.toLowerCase();
const isAction = lowerComponent === "button" ||
lowerComponent === "a" ||
actionComponentPattern.test(component) ||
(lowerComponent === "input" && ["button", "reset", "submit"].includes(inputType));
if (!isAction) return;
const label = attributes.get("aria-label") ??
attributes.get("title") ??
attributes.get("label") ??
staticJsxChildText(node) ??
null;
const explicitId = firstAttribute(
attributes,
new Set(["interfaceId", "data-interface-id", "id", "name"])
);
const context = nearestNamedContext(node);
result.actions.push({
...locate(node),
id: sourceIdentity(
"action",
node,
component,
explicitId ?? label ?? "action"
),
explicitId,
idSource: explicitId === null ? "source_anchor" : "explicit",
context,
component,
label
});
}
function nearestNamedContext(node) {
let current = node.parent;
while (current) {
if (ts.isFunctionDeclaration(current) && current.name) {
return current.name.text;
}
if (ts.isMethodDeclaration(current) && current.name) {
return current.name.getText(sourceFile);
}
if (
(ts.isArrowFunction(current) || ts.isFunctionExpression(current)) &&
ts.isVariableDeclaration(current.parent) &&
ts.isIdentifier(current.parent.name)
) {
return current.parent.name.text;
}
if (
(ts.isArrowFunction(current) || ts.isFunctionExpression(current)) &&
ts.isPropertyAssignment(current.parent)
) {
return propertyNameText(current.parent.name) ?? "anonymous";
}
current = current.parent;
}
return path.basename(relativeFile).replace(/\.[^.]+$/, "");
}
function sourceIdentity(kind, node, component, semantic) {
const context = nearestNamedContext(node);
const normalizedSemantic = slug(String(semantic));
const counterKey = `${kind}:${context}:${component}:${normalizedSemantic}`;
const occurrence = (identityCounters.get(counterKey) ?? 0) + 1;
identityCounters.set(counterKey, occurrence);
const anchor = [relativeFile, context, component, normalizedSemantic, occurrence].join(":");
const digest = createHash("sha256").update(anchor).digest("hex").slice(0, 12);
return `${repository}.${kind}.${slug(context)}.${normalizedSemantic}.${digest}`;
}
function staticJsxChildText(node) {
if (!ts.isJsxOpeningElement(node) || !ts.isJsxElement(node.parent)) return null;
const values = [];
for (const child of node.parent.children) {
if (ts.isJsxText(child)) {
const value = child.getText(sourceFile).replace(/\s+/g, " ").trim();
if (value) values.push(value);
} else if (
ts.isJsxExpression(child) &&
child.expression &&
ts.isStringLiteralLike(child.expression)
) {
values.push(child.expression.text);
}
}
return values.length > 0 ? values.join(" ") : null;
}
function nearestFormField(node) {
@@ -275,22 +400,103 @@ function inspectSource(repository, sourceRoot, sourcePath) {
function inspectProperty(node, locate) {
const propertyName = propertyNameText(node.name);
if (isDirectUiCapabilityProperty(node) && typeof propertyName === "string") {
result.uiCapabilities.push({ ...locate(node), name: propertyName });
result.contributions.push({
...locate(node),
kind: "ui_capability",
id: propertyName
});
}
const value = staticExpressionText(node.initializer);
if (value === null) return;
if (propertyName === "path" && value.startsWith("/") && !value.includes("/api/")) {
result.routes.push({ ...locate(node), path: value });
const routeCollection = nearestCollectionProperty(node);
if (routeCollection === "routes" || routeCollection === "publicRoutes") {
result.contributions.push({
...locate(node),
kind: routeCollection === "publicRoutes" ? "public_route" : "frontend_route",
id: value,
path: value
});
}
}
if (propertyName === "to" && value.startsWith("/")) {
result.navigation.push({ ...locate(node), path: value });
if (nearestCollectionProperty(node) === "navItems") {
result.contributions.push({
...locate(node),
kind: "navigation",
id: value,
path: value
});
}
}
if (
ancestorPropertyName(node, "uiCapabilities") &&
typeof propertyName === "string"
) {
result.uiCapabilities.push({ ...locate(node), name: propertyName });
if (propertyName === "id") {
const contributionKind = contributionKindFor(node);
if (contributionKind !== null) {
result.contributions.push({
...locate(node),
kind: contributionKind,
id: value
});
}
}
}
function contributionKindFor(node) {
const collection = nearestCollectionProperty(node);
if (collection === "viewSurfaces") return "view_surface";
if (collection === "widgets") return "widget";
if (collection === "contexts") return "search_object";
if (collection === "actions") return "action";
if (collection === "directories") return "workflow_directory";
if (collection !== "sections") return null;
const variableType = nearestVariableType(node);
for (const [typeName, kind] of contributionTypes) {
if (variableType.includes(typeName)) return kind;
}
return ancestorPropertyName(node, "admin.sections")
? "admin_section"
: ancestorPropertyName(node, "settings.sections")
? "setting"
: "section";
}
function nearestCollectionProperty(node) {
let current = node.parent;
while (current) {
if (
ts.isArrayLiteralExpression(current) &&
ts.isPropertyAssignment(current.parent)
) {
return propertyNameText(current.parent.name);
}
current = current.parent;
}
return null;
}
function nearestVariableType(node) {
let current = node.parent;
while (current) {
if (ts.isVariableDeclaration(current)) {
return current.type?.getText(sourceFile) ?? "";
}
current = current.parent;
}
return "";
}
function isDirectUiCapabilityProperty(node) {
const parent = node.parent;
const uiCapabilities = parent?.parent;
return ts.isObjectLiteralExpression(parent) &&
ts.isPropertyAssignment(uiCapabilities) &&
propertyNameText(uiCapabilities.name) === "uiCapabilities";
}
function inspectTranslationProperty(node, locate) {
const key = propertyNameText(node.name);
if (!key?.startsWith("i18n:")) return;
@@ -343,6 +549,23 @@ function firstAttribute(attributes, names) {
return null;
}
function hasAnyAttribute(attributes, names) {
for (const name of names) {
if (attributes.has(name)) return true;
}
return false;
}
function slug(value) {
const normalized = value
.toLowerCase()
.replace(/^i18n:/, "")
.replace(/\.[a-f0-9]{8}$/, "")
.replace(/[^a-z0-9]+/g, ".")
.replace(/^\.+|\.+$/g, "");
return (normalized || "unnamed").slice(0, 72);
}
function propertyNameText(name) {
if (
ts.isIdentifier(name) ||
+437 -16
View File
@@ -45,6 +45,28 @@ def main() -> int:
action="store_true",
help="Fail on missing translations or incomplete endpoint-surface declarations.",
)
parser.add_argument(
"--strict-endpoints",
action="store_true",
help="Fail only on incomplete or stale endpoint-surface declarations.",
)
parser.add_argument(
"--strict-declarations",
action="store_true",
help=(
"Fail on duplicate stable IDs, WebUI surfaces absent from runtime "
"metadata, or stale runtime route declarations."
),
)
parser.add_argument(
"--runtime-snapshot",
type=Path,
help=(
"Compare a saved /api/v1/platform/interface-catalog response with "
"the static manifest inventory. Any installed module combination "
"is accepted; every module present in the snapshot must match."
),
)
parser.add_argument(
"--endpoint-declarations",
type=Path,
@@ -66,6 +88,11 @@ def main() -> int:
backend_endpoints=backend_endpoints,
manifests=manifests,
endpoint_declarations=endpoint_declarations,
runtime_snapshot=(
_load_runtime_snapshot(args.runtime_snapshot.resolve())
if args.runtime_snapshot is not None
else None
),
)
output_dir = args.output_dir.resolve()
@@ -80,20 +107,13 @@ def main() -> int:
print(f"Platform inventory JSON: {json_path}")
print(f"Platform inventory summary: {markdown_path}")
if args.strict:
failures: list[str] = []
if inventory["translation_health"]["missing_catalog_entries"]:
failures.append("used translation keys are missing from generated catalogs")
if inventory["api"]["unclassified_endpoints"]:
failures.append(
f"{len(inventory['api']['unclassified_endpoints'])} backend "
"endpoints have no WebUI evidence or surface declaration"
)
if inventory["api"]["stale_endpoint_declarations"]:
failures.append(
f"{len(inventory['api']['stale_endpoint_declarations'])} "
"endpoint declarations do not match a backend endpoint"
)
if args.strict or args.strict_endpoints or args.strict_declarations:
failures = _strict_failures(
inventory,
check_translations=args.strict,
check_endpoints=args.strict or args.strict_endpoints,
check_declarations=args.strict or args.strict_declarations,
)
if failures:
print(
"Strict platform inventory failed: " + "; ".join(failures) + ".",
@@ -103,6 +123,58 @@ def main() -> int:
return 0
def _strict_failures(
inventory: dict[str, Any],
*,
check_translations: bool,
check_endpoints: bool,
check_declarations: bool = False,
) -> list[str]:
failures: list[str] = []
if (
check_translations
and inventory["translation_health"]["missing_catalog_entries"]
):
failures.append("used translation keys are missing from generated catalogs")
if check_endpoints and inventory["api"]["unclassified_endpoints"]:
failures.append(
f"{len(inventory['api']['unclassified_endpoints'])} backend "
"endpoints have no WebUI evidence or surface declaration"
)
if check_endpoints and inventory["api"]["stale_endpoint_declarations"]:
failures.append(
f"{len(inventory['api']['stale_endpoint_declarations'])} "
"endpoint declarations do not match a backend endpoint"
)
declaration_health = inventory.get("declaration_health", {})
if check_declarations and declaration_health.get("duplicate_ids"):
failures.append(
f"{len(declaration_health['duplicate_ids'])} platform interface "
"IDs are declared more than once"
)
if check_declarations and declaration_health.get("undeclared_source_surfaces"):
failures.append(
f"{len(declaration_health['undeclared_source_surfaces'])} public "
"WebUI surfaces have no runtime manifest declaration"
)
if check_declarations and declaration_health.get("stale_runtime_routes"):
failures.append(
f"{len(declaration_health['stale_runtime_routes'])} runtime route "
"declarations have no WebUI implementation"
)
runtime_comparison = inventory.get("runtime_comparison")
if (
check_declarations
and runtime_comparison is not None
and runtime_comparison.get("mismatches")
):
failures.append(
f"{len(runtime_comparison['mismatches'])} runtime catalog entries "
"do not match the release inventory"
)
return failures
def _resolve_workspace_root(catalog: dict[str, Any]) -> Path:
sibling_root = META_ROOT.parent.resolve()
configured_root = Path(str(catalog["default_parent"])).expanduser().resolve()
@@ -243,6 +315,10 @@ def _extract_manifests(
if (workspace_root / repository["path"] / "src").is_dir()
]
sys.path[:0] = [str(path) for path in source_roots]
from govoplan_core.core.platform_interfaces import ( # noqa: PLC0415
manifest_interface_catalog,
)
manifests: list[dict[str, Any]] = []
for repository in catalog["repositories"]:
source_root = workspace_root / repository["path"] / "src"
@@ -277,15 +353,24 @@ def _extract_manifests(
}
for permission in manifest.permissions
],
"interface_catalog": manifest_interface_catalog(manifest),
"frontend": (
{
"package": frontend.package_name,
"routes": [
_plain_value(route) for route in frontend.routes
],
"public_routes": [
_plain_value(route)
for route in frontend.public_routes
],
"nav_items": [
_plain_value(item) for item in frontend.nav_items
],
"settings_routes": [
_plain_value(route)
for route in frontend.settings_routes
],
"view_surfaces": [
_plain_value(surface)
for surface in frontend.view_surfaces
@@ -305,6 +390,7 @@ def _assemble_inventory(
backend_endpoints: list[dict[str, Any]],
manifests: list[dict[str, Any]],
endpoint_declarations: dict[tuple[str, str, str], dict[str, Any]],
runtime_snapshot: dict[str, Any] | None = None,
) -> dict[str, Any]:
frontend_refs = webui["frontendApiReferences"]
frontend_paths = {
@@ -374,25 +460,40 @@ def _assemble_inventory(
]
fields = webui["fields"]
help_candidates = [field for field in fields if field["helpCandidate"]]
dynamic_help = [field for field in fields if field.get("helpDynamic")]
source_declarations = _source_interface_declarations(webui, manifests)
declaration_health = _declaration_health(source_declarations, manifests)
runtime_comparison = (
_compare_runtime_snapshot(runtime_snapshot, manifests)
if runtime_snapshot is not None
else None
)
return {
"schema_version": 1,
"schema_version": 2,
"scope": {
"source": "local GovOPlaN repository catalog",
"limitations": [
"Static extraction cannot resolve runtime-computed labels, routes, or API paths.",
"A backend endpoint without a static WebUI reference may intentionally serve public clients, workers, connectors, or external integrations.",
"A field marked as a help candidate may receive contextual help from a surrounding dynamic component.",
"Low-level field and action IDs use line-independent source anchors unless an explicit interfaceId, DOM id, or name is declared.",
],
},
"modules": manifests,
"interface_declarations": source_declarations,
"declaration_health": declaration_health,
"runtime_comparison": runtime_comparison,
"ui": {
"fields": fields,
"actions": webui.get("actions", []),
"labels": webui["labels"],
"visible_text": webui["visibleText"],
"routes": webui["routes"],
"navigation": webui["navigation"],
"ui_capabilities": webui["uiCapabilities"],
"help_candidates": help_candidates,
"dynamic_help": dynamic_help,
"contributions": webui.get("contributions", []),
},
"translations": {
"catalog": catalogs,
@@ -417,6 +518,16 @@ def _assemble_inventory(
"ui_fields": len(fields),
"ui_fields_with_static_help": len(fields) - len(help_candidates),
"help_review_candidates": len(help_candidates),
"dynamic_help_references": len(dynamic_help),
"ui_actions": len(webui.get("actions", [])),
"interface_declarations": len(source_declarations),
"duplicate_interface_ids": len(declaration_health["duplicate_ids"]),
"undeclared_source_surfaces": len(
declaration_health["undeclared_source_surfaces"]
),
"stale_runtime_routes": len(
declaration_health["stale_runtime_routes"]
),
"label_attributes": len(webui["labels"]),
"visible_text_nodes": len(webui["visibleText"]),
"frontend_routes": len(webui["routes"]),
@@ -429,6 +540,299 @@ def _assemble_inventory(
}
def _source_interface_declarations(
webui: dict[str, Any],
manifests: list[dict[str, Any]],
) -> list[dict[str, Any]]:
module_by_repository = {
str(manifest["repository"]): str(manifest["id"])
for manifest in manifests
}
declarations: list[dict[str, Any]] = []
def module_id(repository: str) -> str:
if repository in module_by_repository:
return module_by_repository[repository]
if repository == "govoplan-core":
return "core"
return repository.removeprefix("govoplan-").replace("-", "_")
def source_evidence(item: dict[str, Any]) -> dict[str, Any]:
return {
key: item[key]
for key in ("repository", "file", "line", "column")
if key in item
}
for kind, items in (
("field", webui["fields"]),
("action", webui.get("actions", [])),
):
for item in items:
repository = str(item["repository"])
owner = module_id(repository)
raw_id = str(item["id"])
stable_id = (
f"{owner}.{raw_id[len(repository) + 1:]}"
if raw_id.startswith(f"{repository}.")
else _namespaced_interface_id(owner, kind, raw_id)
)
declarations.append(
{
"key": f"{kind}:{stable_id}",
"id": stable_id,
"module_id": owner,
"kind": kind,
"origin": "webui_source",
"id_source": item.get("idSource", "source_anchor"),
"explicit_id": item.get("explicitId"),
"context": item.get("context"),
**source_evidence(item),
}
)
if kind == "field" and item.get("helpId"):
help_raw_id = str(item["helpId"])
help_id = (
f"{owner}.{help_raw_id[len(repository) + 1:]}"
if help_raw_id.startswith(f"{repository}.")
else _namespaced_interface_id(owner, "help", help_raw_id)
)
declarations.append(
{
"key": f"help:{help_id}",
"id": help_id,
"module_id": owner,
"kind": "help",
"origin": "webui_source",
"field_id": stable_id,
"dynamic": bool(item.get("helpDynamic")),
**source_evidence(item),
}
)
for item in webui.get("contributions", []):
repository = str(item["repository"])
owner = module_id(repository)
kind = str(item["kind"])
raw_id = str(item["id"])
path = item.get("path")
if kind == "frontend_route" and isinstance(path, str):
stable_id = f"{owner}.route.{_surface_slug(path)}"
elif kind == "public_route" and isinstance(path, str):
stable_id = f"{owner}.public.{_surface_slug(path)}"
elif kind == "navigation" and isinstance(path, str):
stable_id = f"{owner}.nav.{_surface_slug(path)}"
else:
stable_id = _namespaced_interface_id(owner, kind, raw_id)
declarations.append(
{
"key": f"{kind}:{stable_id}",
"id": stable_id,
"module_id": owner,
"kind": kind,
"origin": "webui_contribution",
"declared_value": raw_id,
**({"path": path} if isinstance(path, str) else {}),
**source_evidence(item),
}
)
translation_entries: dict[tuple[str, str], dict[str, Any]] = {}
for locale, entries in webui["translationCatalog"].items():
for key, item in entries.items():
repository = str(item["repository"])
owner = module_id(repository)
declaration_key = (owner, str(key))
declaration = translation_entries.setdefault(
declaration_key,
{
"key": f"translation:{key}",
"id": key,
"module_id": owner,
"kind": "translation",
"origin": "translation_catalog",
"locales": [],
**source_evidence(item),
},
)
declaration["locales"].append(locale)
declarations.extend(translation_entries.values())
return sorted(
declarations,
key=lambda item: (
item["module_id"],
item["kind"],
item["id"],
item.get("file", ""),
item.get("line", 0),
),
)
def _declaration_health(
source_declarations: list[dict[str, Any]],
manifests: list[dict[str, Any]],
) -> dict[str, Any]:
grouped: dict[tuple[str, str, str], list[dict[str, Any]]] = {}
for declaration in source_declarations:
key = (
str(declaration["module_id"]),
str(declaration["kind"]),
str(declaration["id"]),
)
grouped.setdefault(key, []).append(declaration)
duplicate_ids = [
{
"module_id": key[0],
"kind": key[1],
"id": key[2],
"evidence": values,
}
for key, values in sorted(grouped.items())
if len(values) > 1
]
comparable_kinds = {
"frontend_route",
"navigation",
"public_route",
"view_surface",
}
source_surfaces = {
(str(item["module_id"]), str(item["kind"]), str(item["id"])): item
for item in source_declarations
if item["origin"] == "webui_contribution"
and item["kind"] in comparable_kinds
}
runtime_surfaces: dict[tuple[str, str, str], dict[str, Any]] = {}
for manifest in manifests:
catalog = manifest["interface_catalog"]
for declaration in catalog["declarations"]:
if declaration["kind"] not in comparable_kinds:
continue
key = (
str(manifest["id"]),
str(declaration["kind"]),
str(declaration["id"]),
)
runtime_surfaces[key] = {
"repository": manifest["repository"],
**declaration,
}
surface_id = declaration.get("metadata", {}).get("surface_id")
if (
declaration["kind"]
in {"frontend_route", "navigation", "settings_route"}
and isinstance(surface_id, str)
and surface_id
):
runtime_surfaces[
(str(manifest["id"]), "view_surface", surface_id)
] = {
"repository": manifest["repository"],
"key": f"view_surface:{surface_id}",
"id": surface_id,
"module_id": manifest["id"],
"kind": "view_surface",
"metadata": {
"derived_from": declaration["kind"],
"path": declaration.get("path"),
},
}
undeclared_source_surfaces = [
source_surfaces[key]
for key in sorted(source_surfaces.keys() - runtime_surfaces.keys())
]
route_kinds = {"frontend_route", "public_route"}
stale_runtime_routes = [
runtime_surfaces[key]
for key in sorted(runtime_surfaces.keys() - source_surfaces.keys())
if key[1] in route_kinds
]
return {
"duplicate_ids": duplicate_ids,
"undeclared_source_surfaces": undeclared_source_surfaces,
"stale_runtime_routes": stale_runtime_routes,
"source_declaration_count": len(source_declarations),
"runtime_declaration_count": sum(
len(manifest["interface_catalog"]["declarations"])
for manifest in manifests
),
}
def _compare_runtime_snapshot(
snapshot: dict[str, Any],
manifests: list[dict[str, Any]],
) -> dict[str, Any]:
static_by_module = {
str(manifest["id"]): manifest["interface_catalog"]
for manifest in manifests
}
modules = snapshot.get("modules")
if not isinstance(modules, list):
raise ValueError("Runtime interface snapshot must contain a modules list.")
mismatches: list[dict[str, Any]] = []
seen: set[str] = set()
matched: list[str] = []
for item in modules:
if not isinstance(item, dict) or not isinstance(item.get("module_id"), str):
raise ValueError("Runtime interface snapshot has an invalid module entry.")
module_id = item["module_id"]
if module_id in seen:
mismatches.append({"module_id": module_id, "reason": "duplicate_module"})
continue
seen.add(module_id)
expected = static_by_module.get(module_id)
if expected is None:
mismatches.append({"module_id": module_id, "reason": "unknown_module"})
continue
for field in ("contract_version", "module_version", "digest"):
if item.get(field) != expected.get(field):
mismatches.append(
{
"module_id": module_id,
"reason": f"{field}_mismatch",
"expected": expected.get(field),
"actual": item.get(field),
}
)
if not any(
mismatch["module_id"] == module_id for mismatch in mismatches
):
matched.append(module_id)
return {
"contract_version": snapshot.get("contract_version"),
"matched_modules": sorted(matched),
"mismatches": mismatches,
}
def _load_runtime_snapshot(path: Path) -> dict[str, Any]:
try:
payload = json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise ValueError(f"Runtime interface snapshot does not exist: {path}") from exc
except json.JSONDecodeError as exc:
raise ValueError(f"Runtime interface snapshot is invalid JSON: {exc}") from exc
if not isinstance(payload, dict):
raise ValueError("Runtime interface snapshot must be a JSON object.")
return payload
def _namespaced_interface_id(module_id: str, kind: str, value: str) -> str:
if value.startswith(f"{module_id}."):
return value
return f"{module_id}.{kind}.{_surface_slug(value)}"
def _surface_slug(value: str) -> str:
normalized = re.sub(r"[^a-z0-9]+", ".", value.strip().lower()).strip(".")
return normalized or "root"
def _render_markdown(inventory: dict[str, Any]) -> str:
summary = inventory["summary"]
missing = inventory["translation_health"]["missing_catalog_entries"]
@@ -450,8 +854,14 @@ def _render_markdown(inventory: dict[str, Any]) -> str:
"",
f"- Modules: {summary['modules']}",
f"- UI fields: {summary['ui_fields']}",
f"- UI actions: {summary['ui_actions']}",
f"- Fields with statically associated help: {summary['ui_fields_with_static_help']}",
f"- Fields with dynamic help references: {summary['dynamic_help_references']}",
f"- Help review candidates: {summary['help_review_candidates']}",
f"- Stable interface declarations: {summary['interface_declarations']}",
f"- Duplicate interface IDs: {summary['duplicate_interface_ids']}",
f"- WebUI surfaces missing runtime declarations: {summary['undeclared_source_surfaces']}",
f"- Runtime routes missing WebUI implementations: {summary['stale_runtime_routes']}",
f"- Label attributes: {summary['label_attributes']}",
f"- Frontend routes: {summary['frontend_routes']}",
f"- Backend endpoints: {summary['backend_endpoints']}",
@@ -505,13 +915,24 @@ def _render_markdown(inventory: dict[str, Any]) -> str:
)
lines.extend(
[
"",
"## Declaration Reconciliation",
"",
"Routes, navigation, View surfaces, fields, actions, help references,",
"translations, settings, widgets, search objects, and backend",
"capabilities use normalized stable IDs. CI rejects duplicate IDs,",
"WebUI public surfaces absent from runtime metadata, and runtime routes",
"without a WebUI implementation.",
"",
"",
"## Interpretation",
"",
"Use the JSON artifact for exact file and line evidence. Missing help is",
"a triage list, not an automatic defect. Endpoint coverage requires an",
"owner classification before enforcement. Runtime-computed structures",
"need explicit manifest metadata to become canonically visible.",
"need explicit manifest or typed PlatformWebModule metadata to become",
"canonically visible. The generated files are release evidence, not an",
"editable source of platform behavior.",
"",
]
)
@@ -26,6 +26,7 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--web-metadata", type=Path, required=True)
parser.add_argument("--deployer", type=Path, required=True)
parser.add_argument("--deployer-url", required=True)
parser.add_argument("--package-lock", type=Path, required=True)
parser.add_argument("--artifact-base-url", required=True)
parser.add_argument("--source-commit", required=True)
parser.add_argument("--version", required=True)
@@ -45,6 +46,11 @@ def build_parser() -> argparse.ArgumentParser:
def finalize(args: argparse.Namespace) -> dict[str, Any]:
composition = _json_object(args.composition)
_validate_package_lock(
_json_object(args.package_lock),
version=args.version,
composition=composition,
)
api = _image_metadata(_json_object(args.api_metadata), "api")
web = _image_metadata(_json_object(args.web_metadata), "web")
dependencies = dict(_dependency(value) for value in args.dependency)
@@ -99,6 +105,10 @@ def finalize(args: argparse.Namespace) -> dict[str, Any]:
"url": args.deployer_url,
"sha256": _sha256_file(args.deployer),
},
"package_lock": {
"url": f"{artifact_base}/package-artifacts.lock.json",
"sha256": _sha256_file(args.package_lock),
},
"images": {
"api": {
**api,
@@ -247,6 +257,57 @@ def _json_object(path: Path) -> dict[str, Any]:
return value
def _validate_package_lock(
lock: dict[str, Any],
*,
version: str,
composition: dict[str, Any],
) -> None:
if lock.get("schema_version") != "1" or lock.get("release_version") != version:
raise ValueError("package lock schema or release version does not match")
unsigned = dict(lock)
expected_hash = unsigned.pop("lock_sha256", None)
actual_hash = hashlib.sha256(
json.dumps(unsigned, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
if expected_hash != actual_hash:
raise ValueError("package lock hash does not match its contents")
rows = lock.get("python")
if not isinstance(rows, list):
raise ValueError("package lock Python artifacts are missing")
locked = _package_identities(rows, name_key="name")
composed = _package_identities(
composition["python"]["packages"],
name_key="package",
)
if locked != composed:
raise ValueError("package lock does not match the runtime wheel composition")
def _package_identities(
rows: list[object],
*,
name_key: str,
) -> dict[str, tuple[str, str]]:
identities: dict[str, tuple[str, str]] = {}
for row in rows:
if not isinstance(row, dict):
raise ValueError("package artifact identity is malformed")
name = row.get(name_key)
version = row.get("version")
digest = row.get("sha256")
if (
not isinstance(name, str)
or not isinstance(version, str)
or not isinstance(digest, str)
or SHA256.fullmatch(digest) is None
or name in identities
):
raise ValueError("package artifact identity is malformed or duplicated")
identities[name] = (version, digest)
return identities
def _https_url(value: str, label: str) -> str:
parsed = urlsplit(value)
if parsed.scheme != "https" or not parsed.netloc or parsed.username or parsed.password:
@@ -0,0 +1,125 @@
#!/usr/bin/env python3
"""Generate the optional GovOPlaN developer convenience meta-package."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import re
import tomllib
META_ROOT = Path(__file__).resolve().parents[2]
DIRECT = re.compile(r"^(govoplan-[a-z0-9-]+)(?:\[([^]]+)\])?\s+@\s+.*@v([A-Za-z0-9._+!-]+)$")
LOCAL_CORE = re.compile(r"^(?:-e\s+)?\.\./govoplan-core(?:\[([^]]+)\])?$")
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--workspace", type=Path, default=META_ROOT.parent)
parser.add_argument(
"--requirements",
type=Path,
default=META_ROOT / "requirements-release.txt",
)
parser.add_argument(
"--output",
type=Path,
default=META_ROOT / "packages" / "govoplan-meta" / "pyproject.toml",
)
parser.add_argument("--check", action="store_true")
return parser
def render(*, workspace: Path, requirements: Path) -> str:
core = tomllib.loads((workspace / "govoplan-core/pyproject.toml").read_text(encoding="utf-8"))["project"]
version = str(core["version"])
base: list[str] = []
for raw in requirements.read_text(encoding="utf-8").splitlines():
line = raw.strip()
if not line or line.startswith("#"):
continue
local = LOCAL_CORE.fullmatch(line)
if local:
extra = f"[{local.group(1)}]" if local.group(1) else ""
base.append(f"govoplan-core{extra}=={version}")
continue
match = DIRECT.fullmatch(line)
if match is None:
raise ValueError(f"unsupported release requirement: {line!r}")
extra = f"[{match.group(2)}]" if match.group(2) else ""
base.append(f"{match.group(1)}{extra}=={match.group(3)}")
base_names = {_requirement_name(item) for item in base}
full: list[str] = []
for project_path in sorted(workspace.glob("govoplan-*/pyproject.toml")):
project = tomllib.loads(project_path.read_text(encoding="utf-8"))["project"]
name = str(project.get("name") or "")
package_version = str(project.get("version") or "")
if name.startswith("govoplan-") and name not in base_names:
full.append(f"{name}=={package_version}")
return "\n".join(
[
"[build-system]",
'requires = ["setuptools>=69", "wheel"]',
'build-backend = "setuptools.build_meta"',
"",
"[project]",
'name = "govoplan"',
f'version = {json.dumps(version)}',
'description = "Developer convenience package for a versioned GovOPlaN composition"',
'readme = "README.md"',
'requires-python = ">=3.12"',
'license = { text = "AGPL-3.0-or-later" }',
"dependencies = [",
*[f" {json.dumps(item)}," for item in base],
"]",
"",
"[project.optional-dependencies]",
"full = [",
*[f" {json.dumps(item)}," for item in full],
"]",
"",
"[project.urls]",
'Repository = "https://git.add-ideas.de/GovOPlaN/govoplan"',
'Documentation = "https://govoplan.add-ideas.de"',
"",
"[tool.setuptools.packages.find]",
'where = ["src"]',
"",
]
)
def _requirement_name(value: str) -> str:
return value.split("[", 1)[0].split("==", 1)[0]
def main() -> int:
args = build_parser().parse_args()
try:
expected = render(
workspace=args.workspace.expanduser().resolve(),
requirements=args.requirements.expanduser().resolve(),
)
except (OSError, KeyError, ValueError, tomllib.TOMLDecodeError) as exc:
print(f"error: {exc}")
return 1
output = args.output.expanduser()
current = output.read_text(encoding="utf-8") if output.is_file() else None
if args.check:
if current != expected:
print(f"error: developer meta-package is stale: {output}")
return 1
print("Developer meta-package is synchronized.")
return 0
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(expected, encoding="utf-8")
print(f"Developer meta-package written to {output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+8 -5
View File
@@ -94,9 +94,6 @@ cleanup() {
trap cleanup EXIT
cp "$WEBUI/package.release.json" "$TMP_DIR/package.json"
if [[ -f "$WEBUI/package-lock.release.json" ]]; then
cp "$WEBUI/package-lock.release.json" "$TMP_DIR/package-lock.json"
fi
echo "Generating release lockfile from $WEBUI/package.release.json"
echo "Temporary workspace: $TMP_DIR"
@@ -120,7 +117,10 @@ GIT_ENV+=("GIT_CONFIG_COUNT=$git_config_count")
(
cd "$TMP_DIR"
"${GIT_ENV[@]}" PATH="$(dirname "$NPM_BIN"):$PATH" "$NPM_BIN" install --package-lock-only --ignore-scripts
"${GIT_ENV[@]}" \
"npm_config_cache=$TMP_DIR/npm-cache" \
PATH="$(dirname "$NPM_BIN"):$PATH" \
"$NPM_BIN" install --package-lock-only --ignore-scripts
mapfile -t GIT_PACKAGES < <(
PATH="$(dirname "$NODE_BIN"):$PATH" "$NODE_BIN" <<'NODE'
const fs = require("fs");
@@ -136,7 +136,10 @@ NODE
)
if [[ "${#GIT_PACKAGES[@]}" -gt 0 ]]; then
echo "Refreshing git package lock entries: ${GIT_PACKAGES[*]}"
"${GIT_ENV[@]}" PATH="$(dirname "$NPM_BIN"):$PATH" "$NPM_BIN" update --package-lock-only --ignore-scripts "${GIT_PACKAGES[@]}"
"${GIT_ENV[@]}" \
"npm_config_cache=$TMP_DIR/npm-cache" \
PATH="$(dirname "$NPM_BIN"):$PATH" \
"$NPM_BIN" update --package-lock-only --ignore-scripts "${GIT_PACKAGES[@]}"
fi
)
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""Generate the exact registry package set for a GovOPlaN runtime release."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import tomllib
META_ROOT = Path(__file__).resolve().parents[2]
NAME = re.compile(r"^govoplan-[a-z0-9-]+$")
VERSION = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._+!-]{0,127}$")
GIT_REQUIREMENT = re.compile(
r"^(?P<package>govoplan-[a-z0-9-]+)(?:\[(?P<extras>[^]]+)\])?\s+@\s+"
r"(?P<url>git\+[^\s]+/GovOPlaN/(?P<repo>govoplan-[a-z0-9-]+)\.git@v"
r"(?P<version>[A-Za-z0-9._+!-]+))$"
)
LOCAL_CORE = re.compile(r"^(?:-e\s+)?\.\./govoplan-core(?:\[(?P<extras>[^]]+)\])?$")
class PackageSetError(ValueError):
"""Release source references cannot form an immutable package set."""
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--version",
help="Core/meta release version. Defaults to the workspace Core version.",
)
parser.add_argument(
"--requirements",
type=Path,
default=META_ROOT / "requirements-release.txt",
)
parser.add_argument("--workspace", type=Path, default=META_ROOT.parent)
parser.add_argument("--output", type=Path, required=True)
return parser
def parse_release_requirements(path: Path, *, core_version: str) -> tuple[dict[str, object], ...]:
values: list[dict[str, object]] = []
for line_number, raw in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
line = raw.strip()
if not line or line.startswith("#"):
continue
local = LOCAL_CORE.fullmatch(line)
if local:
values.append(
{
"name": "govoplan-core",
"version": core_version.removeprefix("v"),
"repository": "govoplan-core",
"extras": _extras(local.group("extras")),
}
)
continue
match = GIT_REQUIREMENT.fullmatch(line)
if match is None:
raise PackageSetError(
f"unsupported release requirement at {path}:{line_number}: {line!r}"
)
values.append(
{
"name": match.group("package"),
"version": match.group("version"),
"repository": match.group("repo"),
"extras": _extras(match.group("extras")),
}
)
names = [str(item["name"]) for item in values]
if not values or names.count("govoplan-core") != 1 or len(names) != len(set(names)):
raise PackageSetError("release requirements must contain one Core and unique packages")
return tuple(values)
def generate_package_set(
*,
core_version: str,
requirements: Path,
workspace: Path,
) -> dict[str, object]:
core_version = core_version.removeprefix("v")
if VERSION.fullmatch(core_version) is None:
raise PackageSetError("release version is invalid")
python_packages: list[dict[str, object]] = []
webui_packages: list[dict[str, object]] = []
seen_webui: set[str] = set()
for requirement in parse_release_requirements(requirements, core_version=core_version):
repository = workspace / str(requirement["repository"])
tag = f"v{requirement['version']}"
if not (repository / ".git").is_dir():
raise PackageSetError(f"release repository is missing: {repository}")
commit = _git(repository, "rev-list", "-n", "1", tag)
if not commit:
raise PackageSetError(f"release tag is missing: {repository.name}@{tag}")
project = tomllib.loads(_git(repository, "show", f"{tag}:pyproject.toml"))["project"]
if project.get("name") != requirement["name"] or project.get("version") != requirement["version"]:
raise PackageSetError(f"tag metadata does not match {repository.name}@{tag}")
entry = {
**requirement,
"tag": tag,
"commit": commit,
}
python_packages.append(entry)
try:
webui_raw = _git(repository, "show", f"{tag}:webui/package.json")
except subprocess.CalledProcessError:
continue
webui = json.loads(webui_raw)
webui_name = webui.get("name")
if (
not isinstance(webui_name, str)
or not webui_name.startswith("@govoplan/")
or webui.get("version") != requirement["version"]
or webui_name in seen_webui
):
raise PackageSetError(f"WebUI tag metadata does not match {repository.name}@{tag}")
seen_webui.add(webui_name)
webui_packages.append(
{
"name": webui_name,
"version": requirement["version"],
"repository": requirement["repository"],
"tag": tag,
"commit": commit,
}
)
payload: dict[str, object] = {
"schema_version": "1",
"release_version": core_version,
"registries": {
"python": "https://git.add-ideas.de/api/packages/GovOPlaN/pypi/simple",
"npm": "https://git.add-ideas.de/api/packages/GovOPlaN/npm/",
},
"python": python_packages,
"webui": webui_packages,
}
payload["package_set_sha256"] = _canonical_sha256(payload)
return payload
def _extras(value: str | None) -> list[str]:
if not value:
return []
extras = sorted({item.strip() for item in value.split(",") if item.strip()})
if any(re.fullmatch(r"[a-z][a-z0-9_-]*", item) is None for item in extras):
raise PackageSetError("release requirement contains an invalid extra")
return extras
def _git(repository: Path, *arguments: str) -> str:
return subprocess.check_output(
["git", "-C", str(repository), *arguments],
text=True,
stderr=subprocess.DEVNULL,
).strip()
def _canonical_sha256(value: object) -> str:
encoded = json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def main() -> int:
args = build_parser().parse_args()
try:
workspace = args.workspace.expanduser().resolve()
version = args.version
if not version:
core = tomllib.loads(
(workspace / "govoplan-core/pyproject.toml").read_text(encoding="utf-8")
)
version = str(core["project"]["version"])
payload = generate_package_set(
core_version=version,
requirements=args.requirements.expanduser().resolve(),
workspace=workspace,
)
except (PackageSetError, OSError, ValueError, subprocess.CalledProcessError) as exc:
print(f"error: {exc}")
return 1
output = args.output.expanduser()
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f"Release package set written to {output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -8,6 +8,9 @@ WEBUI_DIR="${1:-$CORE_ROOT/webui}"
WORK_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/govoplan-webui-release-deps.XXXXXXXX")"
GOVOPLAN_DEPS="$WORK_ROOT/govoplan-webui-deps.tsv"
export GOVOPLAN_DEPS
PACKAGE_LOCK="${GOVOPLAN_WEBUI_PACKAGE_LOCK:-}"
PACKAGE_DIR="${GOVOPLAN_WEBUI_PACKAGE_DIR:-}"
PYTHON_BIN="${PYTHON:-python3}"
trap 'rm -rf "$WORK_ROOT"' EXIT
@@ -55,9 +58,69 @@ rm -f package-lock.json
npm cache clean --force
retry npm install --prefer-online
if [[ -n "$PACKAGE_LOCK" || -n "$PACKAGE_DIR" ]]; then
[[ -n "$PACKAGE_LOCK" && -n "$PACKAGE_DIR" ]] || {
echo "GOVOPLAN_WEBUI_PACKAGE_LOCK and GOVOPLAN_WEBUI_PACKAGE_DIR must be set together" >&2
exit 1
}
"$PYTHON_BIN" - "$PACKAGE_LOCK" "$PACKAGE_DIR" "$GOVOPLAN_DEPS" <<'PY'
from __future__ import annotations
import hashlib
import json
from pathlib import Path
import sys
lock_path = Path(sys.argv[1]).resolve()
package_dir = Path(sys.argv[2]).resolve()
output = Path(sys.argv[3])
lock = json.loads(lock_path.read_text(encoding="utf-8"))
if lock.get("schema_version") != "1" or not isinstance(lock.get("webui"), list):
raise SystemExit("WebUI package lock is malformed")
unsigned = dict(lock)
expected_lock_hash = unsigned.pop("lock_sha256", None)
actual_lock_hash = hashlib.sha256(
json.dumps(unsigned, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
if expected_lock_hash != actual_lock_hash:
raise SystemExit("WebUI package lock hash does not match its contents")
rows = {}
for item in lock["webui"]:
if not isinstance(item, dict) or not isinstance(item.get("name"), str):
raise SystemExit("WebUI package lock contains a malformed artifact")
if item["name"] in rows:
raise SystemExit(f"WebUI package lock contains duplicate artifact {item['name']}")
rows[item["name"]] = item
requested = []
for line in output.read_text(encoding="utf-8").splitlines():
if not line:
continue
name, _source_ref = line.split("\t", 1)
row = rows.get(name)
if not isinstance(row, dict):
raise SystemExit(f"WebUI package lock has no artifact for {name}")
filename = row.get("filename")
if not isinstance(filename, str) or Path(filename).name != filename:
raise SystemExit(f"WebUI package lock has an invalid filename for {name}")
artifact = package_dir / filename
if artifact.is_symlink() or not artifact.is_file():
raise SystemExit(f"WebUI package artifact is missing for {name}")
encoded = artifact.read_bytes()
if len(encoded) != row.get("size") or hashlib.sha256(encoded).hexdigest() != row.get("sha256"):
raise SystemExit(f"WebUI package artifact hash does not match for {name}")
requested.append(f"{name}\tfile:{artifact}")
output.write_text("\n".join(requested) + "\n", encoding="utf-8")
PY
fi
module_paths=()
while IFS=$'\t' read -r package_name spec; do
[[ -n "${package_name:-}" ]] || continue
if [[ "$spec" == file:* ]]; then
echo "Installing $package_name from verified package artifact"
module_paths+=("$spec")
continue
fi
git_url="${spec%%#*}"
git_ref="${spec#*#}"
if [[ "$git_url" == "$spec" || -z "$git_ref" ]]; then
+62 -15
View File
@@ -9,6 +9,7 @@ import json
import mimetypes
import os
from pathlib import Path
import re
import secrets
import sys
from typing import Any
@@ -18,6 +19,7 @@ from urllib.request import Request, urlopen
MAX_ASSET_BYTES = 256 * 1024 * 1024
COMMIT_SHA = re.compile(r"^[0-9a-f]{40}$")
class PublishError(RuntimeError):
@@ -30,6 +32,7 @@ def build_parser() -> argparse.ArgumentParser:
parser.add_argument("--owner", default="GovOPlaN")
parser.add_argument("--repo", default="govoplan")
parser.add_argument("--tag", required=True)
parser.add_argument("--target-commit", required=True)
parser.add_argument("--title", required=True)
parser.add_argument("--body", default="Signed GovOPlaN runtime distribution.")
parser.add_argument("--asset", type=Path, action="append", default=[], required=True)
@@ -48,25 +51,49 @@ class GiteaReleasePublisher:
self.repo = repo
self.token = token
def release(self, *, tag: str, title: str, body: str) -> dict[str, Any]:
def release(
self,
*,
tag: str,
target_commit: str,
title: str,
body: str,
) -> dict[str, Any]:
if COMMIT_SHA.fullmatch(target_commit) is None:
raise PublishError("release target must be an exact lowercase commit SHA")
resolved_target = self._resolve_commit(target_commit)
if resolved_target != target_commit:
raise PublishError("release target did not resolve to the requested commit")
existing_tag = self._resolve_commit(tag, allow_missing=True)
if existing_tag is not None and existing_tag != target_commit:
raise PublishError(
f"release tag {tag!r} already points to another commit"
)
path = self._repo_path(f"/releases/tags/{quote(tag, safe='')}")
try:
return self._json("GET", path)
release = self._json("GET", path)
except HTTPError as exc:
if exc.code != 404:
raise
return self._json(
"POST",
self._repo_path("/releases"),
payload={
"tag_name": tag,
"name": title,
"body": body,
"draft": False,
"prerelease": False,
},
expected=201,
)
release = self._json(
"POST",
self._repo_path("/releases"),
payload={
"tag_name": tag,
"target_commitish": target_commit,
"name": title,
"body": body,
"draft": False,
"prerelease": False,
},
expected=201,
)
if self._resolve_commit(tag) != target_commit:
raise PublishError(
f"release tag {tag!r} does not resolve to the requested commit"
)
return release
def upload_assets(self, release: dict[str, Any], assets: tuple[Path, ...]) -> None:
release_id = release.get("id")
@@ -165,6 +192,21 @@ class GiteaReleasePublisher:
def _headers(self) -> dict[str, str]:
return {"Authorization": f"token {self.token}", "Accept": "application/json"}
def _resolve_commit(self, ref: str, *, allow_missing: bool = False) -> str | None:
path = self._repo_path(f"/git/commits/{quote(ref, safe='')}")
try:
commit = self._json("GET", path)
except HTTPError as exc:
if allow_missing and exc.code == 404:
return None
raise
if not isinstance(commit, dict):
raise PublishError(f"Gitea returned an invalid commit for {ref!r}")
sha = commit.get("sha")
if not isinstance(sha, str) or COMMIT_SHA.fullmatch(sha) is None:
raise PublishError(f"Gitea returned an invalid commit SHA for {ref!r}")
return sha
def _repo_path(self, suffix: str) -> str:
return (
f"{self.base_url}/api/v1/repos/{quote(self.owner, safe='')}/"
@@ -189,7 +231,12 @@ def main() -> int:
repo=args.repo,
token=os.environ.get(args.token_env, ""),
)
release = publisher.release(tag=args.tag, title=args.title, body=args.body)
release = publisher.release(
tag=args.tag,
target_commit=args.target_commit,
title=args.title,
body=args.body,
)
publisher.upload_assets(release, tuple(args.asset))
except (HTTPError, OSError, PublishError, ValueError) as exc:
print(f"error: {exc}", file=sys.stderr)
+86 -7
View File
@@ -333,20 +333,53 @@ path = pathlib.Path(sys.argv[1])
new_version = sys.argv[2]
text = path.read_text()
text, count = re.subn(
r'(?m)^(\s*version=)["\'][^"\']+["\'](,?\s*)$',
r'(?m)^(MODULE_VERSION\s*=\s*)["\'][^"\']+["\'](\s*)$',
rf'\1"{new_version}"\2',
text,
count=1,
)
if count == 0:
text, count = re.subn(
r'(?m)^(MODULE_VERSION\s*=\s*)["\'][^"\']+["\'](\s*)$',
r'(?ms)(^manifest\s*=\s*ModuleManifest\(.*?^\s*version\s*=\s*)["\'][^"\']+["\'](,?\s*)$',
rf'\1"{new_version}"\2',
text,
count=1,
)
if count != 1:
raise SystemExit(f"could not update ModuleManifest.version in {path}")
raise SystemExit(f"could not update module version declaration in {path}")
path.write_text(text)
PYCODE
done
}
update_package_init_versions() {
local repo="$1"
local version="$2"
local package_init=""
for package_init in "$repo"/src/*/__init__.py; do
[[ -f "$package_init" ]] || continue
if ! grep -q '^__version__\s*=' "$package_init"; then
continue
fi
"$PYTHON" - "$package_init" "$version" <<'PYCODE'
from __future__ import annotations
import pathlib
import re
import sys
path = pathlib.Path(sys.argv[1])
new_version = sys.argv[2]
text = path.read_text()
text, count = re.subn(
r'(?m)^(__version__\s*=\s*)["\'][^"\']+["\'](\s*)$',
rf'\1"{new_version}"\2',
text,
count=1,
)
if count != 1:
raise SystemExit(f"could not update __version__ in {path}")
path.write_text(text)
PYCODE
done
@@ -380,6 +413,11 @@ if project_name != "govoplan-core":
peers["@govoplan/core-webui"] = f"^{new_version}"
path.write_text(json.dumps(data, indent=2) + "\n")
PYCODE
done
"$PYTHON" "$META_ROOT/tools/release/synchronize-webui-package-metadata.py" --repo "$repo"
for package_path in "$repo/package.json" "$repo/webui/package.json"; do
[[ -f "$package_path" ]] || continue
synchronize_lockfile_root "$package_path" "${package_path%package.json}package-lock.json"
done
@@ -428,7 +466,19 @@ if not isinstance(version, str) or not version:
lock["version"] = version
packages = lock.get("packages")
if isinstance(packages, dict) and isinstance(packages.get(""), dict):
packages[""]["version"] = version
root = packages[""]
root["version"] = version
for group in (
"dependencies",
"devDependencies",
"optionalDependencies",
"peerDependencies",
"peerDependenciesMeta",
):
if group in package:
root[group] = package[group]
else:
root.pop(group, None)
lock_path.write_text(json.dumps(lock, indent=2) + "\n")
PYCODE
}
@@ -456,6 +506,13 @@ path.write_text(updated)
PYCODE
}
update_developer_meta_package() {
"$PYTHON" "$META_ROOT/tools/release/generate-developer-meta-package.py" \
--workspace "$PARENT" \
--requirements "$META_ROOT/requirements-release.txt" \
--output "$META_ROOT/packages/govoplan-meta/pyproject.toml"
}
update_version_files() {
local repo="$1"
local version="$2"
@@ -463,6 +520,7 @@ update_version_files() {
update_pyproject "$repo" "$version"
update_manifest_version "$repo" "$project_name" "$version"
update_package_init_versions "$repo" "$version"
update_webui_package "$repo" "$project_name" "$version"
}
@@ -493,10 +551,11 @@ run_version_alignment_gate() {
"$PYTHON"
"$META_ROOT/tools/checks/check-version-alignment.py"
--workspace-root "$PARENT"
--release-composition
)
if [[ "$mode" == "source" ]]; then
command+=(--source-metadata-only)
else
command+=(--release-composition)
fi
local repo
for repo in "${PACKAGE_REPOS[@]}"; do
@@ -526,7 +585,8 @@ run_migration_release_audit() {
command+=("--strict")
;;
auto)
command+=("--strict-if-baseline")
# A coordinated release creates a new baseline after confirmation. The
# preflight validates the graph; strictness applies to that new baseline.
;;
warn)
;;
@@ -539,6 +599,18 @@ run_migration_release_audit() {
run "${command[@]}"
}
record_migration_release_baseline() {
local audit_script="$META_ROOT/tools/release/release-migration-audit.py"
[[ -f "$audit_script" ]] || fail "missing migration audit helper: $audit_script"
run "$PYTHON" "$audit_script" \
--track release \
--record-release "$TARGET_VERSION"
if [[ "$DRY_RUN" -eq 0 ]]; then
"$PYTHON" "$audit_script" --track release --strict
fi
}
print_command() {
printf '+'
printf ' %q' "$@"
@@ -865,6 +937,8 @@ run_manifest_shape_gate
confirm_release
record_migration_release_baseline
for repo in "${PACKAGE_REPOS[@]}"; do
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "Would update version files in $repo to $TARGET_VERSION"
@@ -875,8 +949,10 @@ done
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "Would update $META_ROOT/requirements-release.txt to $TAG"
echo "Would synchronize packages/govoplan-meta/pyproject.toml"
else
update_release_requirements "$TARGET_VERSION"
update_developer_meta_package
fi
refresh_development_webui_lock
@@ -929,10 +1005,13 @@ fi
run git -C "$ROOT" commit -m "$COMMIT_MESSAGE"
run git -C "$ROOT" tag -a "$TAG" -m "$TAG_MESSAGE"
for repo in "${PRE_CORE_REPOS[@]}"; do
for repo in "${MODULE_REPOS[@]}"; do
run git -C "$repo" push --atomic "$REMOTE" "HEAD:refs/heads/${BRANCHES[$repo]}" "refs/tags/$TAG"
done
run git -C "$ROOT" push --atomic "$REMOTE" "HEAD:refs/heads/${BRANCHES[$ROOT]}" "refs/tags/$TAG"
for repo in "${SUPPORT_REPOS[@]}"; do
run git -C "$repo" push --atomic "$REMOTE" "HEAD:refs/heads/${BRANCHES[$repo]}" "refs/tags/$TAG"
done
if [[ "$PUBLISH_WEB_CATALOG" -eq 1 ]]; then
CATALOG_ARGS=(
+6
View File
@@ -21,6 +21,12 @@ def resolve_platforms(
) -> dict[str, object]:
if not repository or "@" in repository or any(value.isspace() for value in repository):
raise ValueError("repository must be an unpinned OCI repository name")
last_slash = repository.rfind("/")
last_colon = repository.rfind(":")
if last_colon > last_slash:
repository = repository[:last_colon]
if not repository:
raise ValueError("repository must be an unpinned OCI repository name")
if DIGEST.fullmatch(index_digest) is None:
raise ValueError("index digest must be sha256:<hex>")
if not isinstance(payload, dict) or not isinstance(payload.get("manifests"), list):
+361
View File
@@ -0,0 +1,361 @@
#!/usr/bin/env python3
"""Download, verify, and lock exact GovOPlaN registry package artifacts."""
from __future__ import annotations
import argparse
import base64
from email.parser import BytesParser
from email.policy import compat32
import hashlib
import json
import os
from pathlib import Path, PurePosixPath
import re
import shutil
import subprocess
import sys
import tarfile
import tempfile
from urllib.parse import quote, urlsplit, urlunsplit
import zipfile
NAME = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$")
WEBUI_NAME = re.compile(r"^@govoplan/[a-z0-9]+(?:-[a-z0-9]+)*-webui$")
VERSION = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._+!-]{0,127}$")
COMMIT = re.compile(r"^[0-9a-f]{40}$")
MAX_ARTIFACT_BYTES = 512 * 1024 * 1024
class PackageArtifactError(ValueError):
"""Registry artifacts do not match the selected package set."""
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--package-set", type=Path, required=True)
parser.add_argument("--wheelhouse", type=Path, required=True)
parser.add_argument("--webui-packages", type=Path, required=True)
parser.add_argument("--lock-output", type=Path, required=True)
parser.add_argument("--requirements-output", type=Path)
parser.add_argument("--python", default=sys.executable)
parser.add_argument("--npm", default="npm")
return parser
def resolve(args: argparse.Namespace) -> dict[str, object]:
package_set = _load_package_set(args.package_set)
wheelhouse = args.wheelhouse.expanduser().resolve()
webui_packages = args.webui_packages.expanduser().resolve()
_require_empty_destination(wheelhouse)
_require_empty_destination(webui_packages)
wheelhouse.parent.mkdir(parents=True, exist_ok=True)
webui_packages.parent.mkdir(parents=True, exist_ok=True)
with tempfile.TemporaryDirectory(prefix="govoplan-package-resolution-") as value:
temporary = Path(value)
wheels = temporary / "wheels"
webui = temporary / "webui"
wheels.mkdir()
webui.mkdir()
_download_wheels(
packages=tuple(package_set["python"]),
destination=wheels,
python=args.python,
index_url=str(package_set["registries"]["python"]),
)
_download_webui(
packages=tuple(package_set["webui"]),
destination=webui,
npm=args.npm,
registry=str(package_set["registries"]["npm"]),
)
python_rows = _verify_wheels(tuple(package_set["python"]), wheels)
webui_rows = _verify_webui(tuple(package_set["webui"]), webui)
lock: dict[str, object] = {
"schema_version": "1",
"release_version": package_set["release_version"],
"package_set_sha256": package_set["package_set_sha256"],
"registries": package_set["registries"],
"python": python_rows,
"webui": webui_rows,
}
lock["lock_sha256"] = _canonical_sha256(lock)
shutil.copytree(wheels, wheelhouse, dirs_exist_ok=True)
shutil.copytree(webui, webui_packages, dirs_exist_ok=True)
args.lock_output.parent.mkdir(parents=True, exist_ok=True)
args.lock_output.write_text(json.dumps(lock, indent=2, sort_keys=True) + "\n", encoding="utf-8")
if args.requirements_output is not None:
_write_requirements(args.requirements_output, python_rows)
return lock
def _load_package_set(path: Path) -> dict[str, object]:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict) or value.get("schema_version") != "1":
raise PackageArtifactError("package set has an unsupported shape")
expected_hash = value.get("package_set_sha256")
unsigned = dict(value)
unsigned.pop("package_set_sha256", None)
if expected_hash != _canonical_sha256(unsigned):
raise PackageArtifactError("package set hash does not match its contents")
registries = value.get("registries")
if not isinstance(registries, dict) or set(registries) != {"python", "npm"}:
raise PackageArtifactError("package set registries are invalid")
for registry in registries.values():
parsed = urlsplit(str(registry))
if parsed.scheme != "https" or not parsed.netloc or parsed.username or parsed.password:
raise PackageArtifactError("package registries must use credential-free HTTPS URLs")
for group in ("python", "webui"):
packages = value.get(group)
if not isinstance(packages, list) or not packages:
raise PackageArtifactError(f"package set {group} entries are missing")
_validate_package_entries(group, packages)
return value
def _validate_package_entries(group: str, packages: list[object]) -> None:
names: set[str] = set()
for raw in packages:
if not isinstance(raw, dict):
raise PackageArtifactError(f"package set {group} entry is malformed")
name = raw.get("name")
version = raw.get("version")
repository = raw.get("repository")
tag = raw.get("tag")
commit = raw.get("commit")
name_valid = (
isinstance(name, str)
and (NAME.fullmatch(name) if group == "python" else WEBUI_NAME.fullmatch(name))
)
if (
not name_valid
or name in names
or not isinstance(version, str)
or VERSION.fullmatch(version) is None
or not isinstance(repository, str)
or NAME.fullmatch(repository) is None
or tag != f"v{version}"
or not isinstance(commit, str)
or COMMIT.fullmatch(commit) is None
):
raise PackageArtifactError(f"package set {group} entry has an invalid identity")
names.add(name)
if group == "python":
extras = raw.get("extras")
if not isinstance(extras, list) or any(
not isinstance(item, str)
or re.fullmatch(r"[a-z][a-z0-9_-]*", item) is None
for item in extras
):
raise PackageArtifactError("package set Python extras are invalid")
def _download_wheels(
*, packages: tuple[dict[str, object], ...], destination: Path, python: str, index_url: str
) -> None:
requirements = [_python_requirement(item) for item in packages]
environment = dict(os.environ)
environment["PIP_INDEX_URL"] = _authenticated_url(index_url)
environment["PIP_EXTRA_INDEX_URL"] = ""
environment["PIP_CONFIG_FILE"] = os.devnull
environment["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
subprocess.run(
[python, "-m", "pip", "download", "--no-deps", "--only-binary=:all:", "--dest", str(destination), *requirements],
check=True,
env=environment,
)
def _download_webui(
*, packages: tuple[dict[str, object], ...], destination: Path, npm: str, registry: str
) -> None:
environment = dict(os.environ)
npmrc: tempfile.NamedTemporaryFile[bytes] | None = None
token = os.environ.get("GOVOPLAN_PACKAGE_TOKEN", "")
if token:
parsed = urlsplit(registry)
auth_path = f"//{parsed.netloc}{parsed.path}:_authToken={token}\n"
npmrc = tempfile.NamedTemporaryFile(prefix="govoplan-npmrc-", delete=False)
npmrc.write(f"@govoplan:registry={registry}\n{auth_path}".encode("utf-8"))
npmrc.close()
os.chmod(npmrc.name, 0o600)
environment["NPM_CONFIG_USERCONFIG"] = npmrc.name
try:
for item in packages:
subprocess.run(
[npm, "pack", f"{item['name']}@{item['version']}", "--ignore-scripts", "--pack-destination", str(destination), "--registry", registry],
check=True,
env=environment,
)
finally:
if npmrc is not None:
Path(npmrc.name).unlink(missing_ok=True)
def _verify_wheels(packages: tuple[dict[str, object], ...], root: Path) -> list[dict[str, object]]:
expected = {_normalize(str(item["name"])): item for item in packages}
rows: list[dict[str, object]] = []
seen: set[str] = set()
for path in sorted(root.glob("*.whl")):
identity = _wheel_identity(path)
name = str(identity["name"])
package = expected.get(name)
if package is None or identity["version"] != package["version"] or name in seen:
raise PackageArtifactError(f"unexpected wheel artifact: {path.name}")
seen.add(name)
rows.append(_artifact_row(path, package))
if seen != set(expected):
raise PackageArtifactError("registry did not return every selected Python wheel")
return sorted(rows, key=lambda item: str(item["name"]))
def _verify_webui(packages: tuple[dict[str, object], ...], root: Path) -> list[dict[str, object]]:
expected = {str(item["name"]): item for item in packages}
rows: list[dict[str, object]] = []
seen: set[str] = set()
for path in sorted(root.glob("*.tgz")):
identity = _npm_identity(path)
name = str(identity["name"])
package = expected.get(name)
if package is None or identity["version"] != package["version"] or name in seen:
raise PackageArtifactError(f"unexpected WebUI artifact: {path.name}")
seen.add(name)
row = _artifact_row(path, package)
row["integrity"] = "sha512-" + base64.b64encode(hashlib.sha512(path.read_bytes()).digest()).decode("ascii")
rows.append(row)
if seen != set(expected):
raise PackageArtifactError("registry did not return every selected WebUI package")
return sorted(rows, key=lambda item: str(item["name"]))
def _wheel_identity(path: Path) -> dict[str, str]:
_bounded(path)
with zipfile.ZipFile(path) as archive:
metadata = [
item for item in archive.infolist()
if PurePosixPath(item.filename).name == "METADATA"
and PurePosixPath(item.filename).parent.name.endswith(".dist-info")
]
if len(metadata) != 1 or metadata[0].file_size > 1024 * 1024:
raise PackageArtifactError(f"wheel metadata is invalid: {path.name}")
parsed = BytesParser(policy=compat32).parsebytes(archive.read(metadata[0]))
name = _normalize(str(parsed.get("Name") or ""))
version = str(parsed.get("Version") or "")
if NAME.fullmatch(name) is None or VERSION.fullmatch(version) is None:
raise PackageArtifactError(f"wheel identity is invalid: {path.name}")
return {"name": name, "version": version}
def _npm_identity(path: Path) -> dict[str, str]:
_bounded(path)
with tarfile.open(path, mode="r:gz") as archive:
try:
member = archive.getmember("package/package.json")
except KeyError as exc:
raise PackageArtifactError(f"npm package metadata is missing: {path.name}") from exc
if not member.isfile() or member.size > 1024 * 1024:
raise PackageArtifactError(f"npm package metadata is invalid: {path.name}")
extracted = archive.extractfile(member)
if extracted is None:
raise PackageArtifactError(f"npm package metadata cannot be read: {path.name}")
value = json.load(extracted)
name = value.get("name")
version = value.get("version")
if not isinstance(name, str) or not name.startswith("@govoplan/") or not isinstance(version, str) or VERSION.fullmatch(version) is None:
raise PackageArtifactError(f"npm package identity is invalid: {path.name}")
return {"name": name, "version": version}
def _artifact_row(path: Path, package: dict[str, object]) -> dict[str, object]:
row = {
"name": package["name"],
"version": package["version"],
"repository": package["repository"],
"tag": package["tag"],
"commit": package["commit"],
"filename": path.name,
"sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
"size": path.stat().st_size,
}
if "extras" in package:
row["extras"] = package["extras"]
return row
def _write_requirements(path: Path, rows: list[dict[str, object]]) -> None:
lines = ["--no-index", "--find-links ./local-wheels", "--require-hashes"]
for row in rows:
selected_extras = row.get("extras") or []
extras = (
f"[{','.join(str(value) for value in selected_extras)}]"
if selected_extras
else ""
)
lines.append(
f"{row['name']}{extras}=={row['version']} "
f"--hash=sha256:{row['sha256']}"
)
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
def _python_requirement(item: dict[str, object]) -> str:
extras = item.get("extras") or []
suffix = f"[{','.join(str(value) for value in extras)}]" if extras else ""
return f"{item['name']}{suffix}=={item['version']}"
def _authenticated_url(url: str) -> str:
token = os.environ.get("GOVOPLAN_PACKAGE_TOKEN", "")
username = os.environ.get("GOVOPLAN_PACKAGE_USERNAME", "")
if not token:
return url
if not username:
raise PackageArtifactError("GOVOPLAN_PACKAGE_USERNAME is required with a package token")
parsed = urlsplit(url)
return urlunsplit(
(
parsed.scheme,
f"{quote(username, safe='')}:{quote(token, safe='')}@{parsed.netloc}",
parsed.path,
parsed.query,
"",
)
)
def _require_empty_destination(path: Path) -> None:
if path.exists() and (not path.is_dir() or any(path.iterdir())):
raise PackageArtifactError(f"output directory must be absent or empty: {path}")
if path.is_symlink():
raise PackageArtifactError(f"output directory must not be a symlink: {path}")
def _bounded(path: Path) -> None:
if path.is_symlink() or not path.is_file() or path.stat().st_size > MAX_ARTIFACT_BYTES:
raise PackageArtifactError(f"package artifact is invalid or too large: {path.name}")
def _normalize(value: str) -> str:
return re.sub(r"[-_.]+", "-", value.strip().lower())
def _canonical_sha256(value: object) -> str:
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode("utf-8")).hexdigest()
def main() -> int:
args = build_parser().parse_args()
try:
lock = resolve(args)
except (PackageArtifactError, OSError, ValueError, subprocess.CalledProcessError, zipfile.BadZipFile, tarfile.TarError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
print(f"Resolved {len(lock['python'])} Python and {len(lock['webui'])} WebUI packages.")
print(f"Package artifact lock written to {args.lock_output}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+1
View File
@@ -12,6 +12,7 @@ ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PYTHONPATH=/opt/govoplan/runtime \
PATH=/opt/govoplan/runtime/bin:${PATH} \
GOVOPLAN_CORE_SOURCE_ROOT=/opt/govoplan/runtime/govoplan_core_runtime \
HOME=/var/lib/govoplan
COPY wheelhouse/ /opt/govoplan/wheels/
+3
View File
@@ -13,7 +13,10 @@ http {
sendfile on;
server_tokens off;
client_body_temp_path /tmp/client_temp;
fastcgi_temp_path /tmp/fastcgi_temp;
proxy_temp_path /tmp/proxy_temp;
scgi_temp_path /tmp/scgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
map $http_x_forwarded_proto $govoplan_forwarded_proto {
default $scheme;
@@ -0,0 +1,56 @@
#!/usr/bin/env python3
"""Synchronize duplicated publish and development WebUI package contracts."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
SYNCHRONIZED_KEYS = ("peerDependencies", "peerDependenciesMeta")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repo", type=Path, required=True)
args = parser.parse_args()
root_path = args.repo / "package.json"
webui_path = args.repo / "webui" / "package.json"
if not root_path.exists() or not webui_path.exists():
return 0
root = _load(root_path)
webui = _load(webui_path)
root_name = root.get("name")
webui_name = webui.get("name")
if not isinstance(root_name, str) or root_name != webui_name:
return 0
changed = False
for key in SYNCHRONIZED_KEYS:
if key in webui:
value = webui[key]
if root.get(key) != value:
root[key] = value
changed = True
elif key in root:
del root[key]
changed = True
if changed:
root_path.write_text(json.dumps(root, indent=2) + "\n")
print(f"Synchronized WebUI peer metadata in {root_path}")
return 0
def _load(path: Path) -> dict[str, object]:
payload = json.loads(path.read_text())
if not isinstance(payload, dict):
raise SystemExit(f"package metadata must be an object: {path}")
return payload
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,78 @@
#!/usr/bin/env python3
"""Install or verify the canonical package-release workflow in module repos."""
from __future__ import annotations
import argparse
import json
from pathlib import Path
import sys
META_ROOT = Path(__file__).resolve().parents[2]
TEMPLATE = META_ROOT / "tools" / "repo" / "templates" / "module-package-release.yml"
DESTINATION = Path(".gitea/workflows/module-package-release.yml")
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description=__doc__)
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--check", action="store_true")
mode.add_argument("--write", action="store_true")
parser.add_argument(
"--parent",
type=Path,
default=META_ROOT.parent,
help="Parent directory containing the repositories.",
)
return parser
def package_repositories(parent: Path) -> tuple[Path, ...]:
inventory = json.loads((META_ROOT / "repositories.json").read_text(encoding="utf-8"))
repositories: list[Path] = []
for item in inventory["repositories"]:
name = str(item["name"])
if not name.startswith("govoplan-"):
continue
repository = parent / str(item["path"])
if (repository / "pyproject.toml").is_file():
repositories.append(repository)
return tuple(sorted(repositories))
def synchronize(*, parent: Path, write: bool) -> tuple[str, ...]:
expected = TEMPLATE.read_bytes()
mismatches: list[str] = []
for repository in package_repositories(parent):
destination = repository / DESTINATION
current = destination.read_bytes() if destination.is_file() else None
if current == expected:
continue
mismatches.append(repository.name)
if write:
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_suffix(destination.suffix + ".tmp")
temporary.write_bytes(expected)
temporary.chmod(0o644)
temporary.replace(destination)
return tuple(mismatches)
def main() -> int:
args = build_parser().parse_args()
mismatches = synchronize(parent=args.parent.expanduser().resolve(), write=args.write)
if args.write:
print(f"Installed package-release workflow in {len(mismatches)} repositories.")
return 0
if mismatches:
print("Package-release workflow is missing or stale in:", file=sys.stderr)
for repository in mismatches:
print(f"- {repository}", file=sys.stderr)
return 1
print("Package-release workflows are synchronized.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+9 -2
View File
@@ -149,6 +149,7 @@ def main() -> int:
requirements=requirements,
python=python,
local_requirements=local_requirements,
repair_requirements=environment.stale_requirements,
force=args.force,
)
@@ -250,6 +251,7 @@ def build_install_plan(
python: str,
local_requirements: tuple[RequirementEntry, ...],
force: bool,
repair_requirements: tuple[RequirementEntry, ...] = (),
) -> InstallPlan:
full_command = (python, "-m", "pip", "install", "-r", str(requirements))
if force:
@@ -273,7 +275,12 @@ def build_install_plan(
removed_paths = set(previous_inputs) - set(current_inputs)
stale_requirements = requirements_key in changed_paths or requirements_key in removed_paths
installs: list[tuple[str, ...]] = []
# Metadata changes and installation drift can happen together. Keep both
# sets in one resolver transaction so a selective metadata sync also
# repairs local distributions omitted from the current environment.
installs: list[tuple[str, ...]] = [
requirement.install_args for requirement in repair_requirements
]
warnings: list[str] = []
if stale_requirements:
@@ -322,7 +329,7 @@ def build_install_plan(
)
return InstallPlan(
"Selective Python environment sync",
f"installing {len(deduped_installs)} stale local requirement(s) in one resolver transaction.",
f"installing {len(deduped_installs)} stale or missing local requirement(s) in one resolver transaction.",
(command,),
tuple(warnings),
)
@@ -0,0 +1,270 @@
name: Module Package Release
on:
push:
tags:
- "v*"
workflow_dispatch:
inputs:
release_tag:
description: Existing protected version tag to publish
required: true
type: string
jobs:
publish-packages:
runs-on: ubuntu-latest
env:
GITEA_REPOSITORY: ${{ gitea.repository }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
fetch-depth: 0
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: "22"
- name: Select and validate protected release tag
shell: bash
env:
REQUESTED_TAG: ${{ inputs.release_tag }}
TRIGGER_TAG: ${{ gitea.ref_name }}
run: |
set -euo pipefail
tag="${REQUESTED_TAG:-$TRIGGER_TAG}"
case "$tag" in
v[0-9]*.[0-9]*.[0-9]*) ;;
*) echo "Release tag must start with a SemVer-shaped vX.Y.Z value" >&2; exit 1 ;;
esac
git fetch --force origin "refs/tags/$tag:refs/tags/$tag" refs/heads/main:refs/remotes/origin/main
tag_commit="$(git rev-list -n 1 "$tag")"
git merge-base --is-ancestor "$tag_commit" refs/remotes/origin/main || {
echo "Release tag is not contained in main" >&2
exit 1
}
git checkout --detach "$tag"
printf 'RELEASE_TAG=%s\n' "$tag" >> "$GITEA_ENV"
printf 'SOURCE_DATE_EPOCH=%s\n' "$(git show -s --format=%ct HEAD)" >> "$GITEA_ENV"
- name: Validate package versions
run: |
python - <<'PY'
import json
from pathlib import Path
import os
import re
import tomllib
tag = os.environ["RELEASE_TAG"]
expected = tag.removeprefix("v")
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
if project.get("version") != expected:
raise SystemExit(f"pyproject version {project.get('version')!r} does not match {tag}")
if re.fullmatch(r"govoplan-[a-z0-9-]+", str(project.get("name", ""))) is None:
raise SystemExit("Python distribution name must use the govoplan-* namespace")
webui = Path("webui/package.json")
if webui.is_file():
package = json.loads(webui.read_text(encoding="utf-8"))
if package.get("version") != expected:
raise SystemExit(f"WebUI version {package.get('version')!r} does not match {tag}")
if re.fullmatch(r"@govoplan/[a-z0-9-]+-webui", str(package.get("name", ""))) is None:
raise SystemExit("WebUI package name must use the @govoplan/*-webui namespace")
release = Path("webui/package.release.json")
if release.is_file():
release_package = json.loads(release.read_text(encoding="utf-8"))
if (
release_package.get("name") != package.get("name")
or release_package.get("version") != expected
):
raise SystemExit("WebUI release package identity does not match package.json and the release tag")
PY
- name: Build immutable package artifacts
shell: bash
run: |
set -euo pipefail
python -m pip install --disable-pip-version-check build==1.5.0 twine==7.0.0
rm -rf dist .package-webui
python -m build --wheel --outdir dist
python -m twine check dist/*.whl
if [[ -f webui/package.json ]]; then
mkdir .package-webui
cp -a webui/. .package-webui/
rm -rf .package-webui/node_modules .package-webui/dist
if [[ -f .package-webui/package.release.json ]]; then
cp .package-webui/package.release.json .package-webui/package.json
fi
node <<'NODE'
const fs = require("node:fs");
const path = ".package-webui/package.json";
const packageJson = JSON.parse(fs.readFileSync(path, "utf8"));
const groups = ["dependencies", "optionalDependencies", "peerDependencies"];
for (const group of groups) {
for (const [name, specifier] of Object.entries(packageJson[group] || {})) {
if (!name.startsWith("@govoplan/")) continue;
if (typeof specifier !== "string") {
throw new Error(`${group}.${name} must use a string version`);
}
const packageSlug = name.slice("@govoplan/".length);
if (!packageSlug.endsWith("-webui")) {
throw new Error(`${group}.${name} is outside the WebUI package namespace`);
}
const repository = `govoplan-${packageSlug.slice(0, -"-webui".length)}`;
const escapedRepository = repository.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
const gitTag = specifier.match(
new RegExp(
`^git\\+(?:ssh://git@|https://)git\\.add-ideas\\.de/(?:GovOPlaN|add-ideas)/${escapedRepository}\\.git#v([0-9]+\\.[0-9]+\\.[0-9]+)$`,
),
);
if (gitTag) {
packageJson[group][name] = gitTag[1];
continue;
}
if (specifier.startsWith("file:") || specifier.startsWith("git+")) {
throw new Error(
`${group}.${name} must resolve to an exact registry version for publication`,
);
}
}
}
delete packageJson.private;
fs.writeFileSync(path, `${JSON.stringify(packageJson, null, 2)}\n`);
NODE
npm pkg delete private --prefix .package-webui
(cd .package-webui && npm pack --ignore-scripts --pack-destination ../dist)
fi
python - <<'PY'
import hashlib
import json
from pathlib import Path
import os
import subprocess
artifacts = []
for path in sorted(Path("dist").iterdir()):
if path.suffix not in {".whl", ".tgz"}:
continue
digest = hashlib.sha256(path.read_bytes()).hexdigest()
artifacts.append({"filename": path.name, "sha256": digest, "size": path.stat().st_size})
payload = {
"schema_version": "1",
"repository": os.environ["GITEA_REPOSITORY"],
"tag": os.environ["RELEASE_TAG"],
"commit": subprocess.check_output(["git", "rev-parse", "HEAD"], text=True).strip(),
"artifacts": artifacts,
}
Path("dist/package-artifacts.json").write_text(
json.dumps(payload, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
PY
- name: Retain package hash evidence
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
with:
name: module-packages-${{ gitea.ref_name }}
path: dist/package-artifacts.json
- name: Check immutable registry state
shell: bash
env:
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
run: |
set -euo pipefail
test -n "$PACKAGE_TOKEN"
python - <<'PY'
import hashlib
import json
import os
from pathlib import Path
import tomllib
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen
api_root = "https://git.add-ideas.de/api/v1/packages/GovOPlaN"
token = os.environ["PACKAGE_TOKEN"]
def should_publish(kind, name, version, path):
package_url = "/".join(
(api_root, kind, quote(name, safe=""), quote(version, safe=""), "files")
)
request = Request(
package_url,
headers={"Accept": "application/json", "Authorization": f"token {token}"},
)
try:
with urlopen(request, timeout=30) as response:
files = json.load(response)
except HTTPError as exc:
if exc.code == 404:
print(f"{kind} package {name}=={version} is not published yet")
return True
raise
if not isinstance(files, list) or len(files) != 1:
raise SystemExit(
f"immutable {kind} package {name}=={version} has an unexpected file set"
)
expected_sha256 = hashlib.sha256(path.read_bytes()).hexdigest()
if files[0].get("sha256") != expected_sha256:
raise SystemExit(
f"immutable {kind} package {name}=={version} already exists with a different SHA-256"
)
print(f"verified existing {kind} package {name}=={version} ({expected_sha256})")
return False
project = tomllib.loads(Path("pyproject.toml").read_text(encoding="utf-8"))["project"]
wheels = tuple(Path("dist").glob("*.whl"))
if len(wheels) != 1:
raise SystemExit("release build must contain exactly one wheel")
publish_pypi = should_publish(
"pypi", str(project["name"]), str(project["version"]), wheels[0]
)
tarballs = tuple(Path("dist").glob("*.tgz"))
if len(tarballs) > 1:
raise SystemExit("release build must contain at most one npm package")
publish_npm = False
if tarballs:
webui = json.loads(
Path(".package-webui/package.json").read_text(encoding="utf-8")
)
publish_npm = should_publish(
"npm", str(webui["name"]), str(webui["version"]), tarballs[0]
)
with Path(os.environ["GITEA_ENV"]).open("a", encoding="utf-8") as env_file:
env_file.write(f"PUBLISH_PYPI={int(publish_pypi)}\n")
env_file.write(f"PUBLISH_NPM={int(publish_npm)}\n")
PY
- name: Publish wheel and WebUI package
shell: bash
env:
PACKAGE_USERNAME: ${{ secrets.GOVOPLAN_PACKAGE_USERNAME }}
PACKAGE_TOKEN: ${{ secrets.GOVOPLAN_PACKAGE_TOKEN }}
run: |
set -euo pipefail
test -n "$PACKAGE_USERNAME"
test -n "$PACKAGE_TOKEN"
if [[ "$PUBLISH_PYPI" == 1 ]]; then
TWINE_USERNAME="$PACKAGE_USERNAME" TWINE_PASSWORD="$PACKAGE_TOKEN" \
python -m twine upload --non-interactive \
--repository-url https://git.add-ideas.de/api/packages/GovOPlaN/pypi \
dist/*.whl
else
echo "Exact wheel is already present; skipping immutable retry."
fi
shopt -s nullglob
webui_packages=(dist/*.tgz)
if (( ${#webui_packages[@]} )) && [[ "$PUBLISH_NPM" == 1 ]]; then
npmrc="$(mktemp)"
trap 'rm -f "$npmrc"' EXIT
chmod 600 "$npmrc"
printf '%s\n' \
'@govoplan:registry=https://git.add-ideas.de/api/packages/GovOPlaN/npm/' \
"//git.add-ideas.de/api/packages/GovOPlaN/npm/:_authToken=$PACKAGE_TOKEN" \
> "$npmrc"
NPM_CONFIG_USERCONFIG="$npmrc" npm publish "./${webui_packages[0]}" \
--ignore-scripts --access public \
--registry https://git.add-ideas.de/api/packages/GovOPlaN/npm/
elif (( ${#webui_packages[@]} )); then
echo "Exact WebUI package is already present; skipping immutable retry."
fi