feat(release): build verified full-registry catalog candidates

This commit is contained in:
2026-09-08 02:26:19 +02:00
parent 32fe4b7238
commit 6bcb75f577
13 changed files with 1338 additions and 11 deletions
+138
View File
@@ -0,0 +1,138 @@
# Full registry candidates / Vollständige Registry-Kandidaten
## Operator workflow (EN)
The canonical `release-catalog.py full-registry` command takes `--package-set`,
`--package-lock`, `--wheelhouse`, `--webui-packages`, `--output-dir`, and a
configured `--catalog-signing-key`. Generate the package set with
`generate-release-package-set.py --profile full` and download its exact artifacts
with `resolve-package-artifacts.py`; do not substitute locally rebuilt wheels.
The candidate compares the package set with the exact developer meta-package
pins, checks archive bytes and package metadata against the lock, and synthesizes
every entry from its immutable tagged manifest. Native package publication and
its CI authority remain trusted: this verifies the published artifact identity,
not independent reproducible-build equivalence to source.
Pass `--selected-repository` once for each newly released repository, including
Core when it changes. These selected units must have clean, version-aligned
named branches whose HEAD equals the annotated local and remote release tag.
Other full-profile packages retain their exact older annotated tags; a later
workflow-only commit on `main` does not relabel those package contents or force
a version bump. Every source fetch/push endpoint must match the registered
origin, and all entries bind their source commit and annotated tag object.
Git replacement objects, caller Git configuration, and executable-path
redirection cannot substitute another tagged manifest tree.
The fixed existing website catalog and keyring are authenticated before signing.
An older catalog without a signed keyring hash can be migrated only through this
complete rebuild, only when its signature verifies and its entire keyring
exactly matches the configured known signers. No old entries or artifact hashes
are reused. The new catalog signs the exact unchanged website keyring hash;
key rotation remains a separate reviewed operation. Selective candidates still
reject unpinned base keyrings. New candidate directories are private and
exclusive: a retry must choose a new directory, not overwrite a reviewed one.
Run these commands on the trusted host, with an operator-private source workspace
and artifact directory. `RELEASE_PYTHON` must select its private environment and
`RELEASE_NPM` an absolute npm executable with a trusted sibling Node 22 binary.
In Flatpak, execute host commands through `flatpak-spawn --host`; sandbox and
host UID mappings are not interchangeable. Do not weaken trust gates or change
system-wide permissions.
```sh
# These paths identify previously prepared private operator resources.
RELEASE_WORKSPACE=/path/to/private/workspace
RELEASE_PYTHON="$RELEASE_WORKSPACE/govoplan/.host-venv/bin/python"
RELEASE_NPM=/path/to/private/node22/bin/npm
ARTIFACT_ROOT=/path/to/private/artifacts
RELEASE_CANDIDATE=/path/to/private/new-candidate
RELEASE_VERSION=0.1.45
RELEASE_TOOLS="$RELEASE_WORKSPACE/govoplan/tools/release"
umask 077
"$RELEASE_PYTHON" "$RELEASE_TOOLS/generate-release-package-set.py" \
--version "$RELEASE_VERSION" --profile full \
--workspace "$RELEASE_WORKSPACE" --output "$ARTIFACT_ROOT/packages.json"
PATH="$(dirname "$RELEASE_NPM"):/usr/bin:/bin" \
"$RELEASE_PYTHON" "$RELEASE_TOOLS/resolve-package-artifacts.py" \
--package-set "$ARTIFACT_ROOT/packages.json" \
--wheelhouse "$ARTIFACT_ROOT/wheels" \
--webui-packages "$ARTIFACT_ROOT/webui" \
--lock-output "$ARTIFACT_ROOT/artifacts.lock.json" \
--python "$RELEASE_PYTHON" --npm "$RELEASE_NPM"
# Repeat --selected-repository for EVERY newly released unit, not just Core.
"$RELEASE_PYTHON" "$RELEASE_TOOLS/release-catalog.py" full-registry \
--workspace-root "$RELEASE_WORKSPACE" \
--package-set "$ARTIFACT_ROOT/packages.json" \
--package-lock "$ARTIFACT_ROOT/artifacts.lock.json" \
--wheelhouse "$ARTIFACT_ROOT/wheels" --webui-packages "$ARTIFACT_ROOT/webui" \
--output-dir "$RELEASE_CANDIDATE" --selected-repository govoplan-core \
--catalog-signing-key known-key=/path/to/private/known-key.pem --json
"$RELEASE_PYTHON" "$RELEASE_TOOLS/release-catalog.py" publish-candidate \
--workspace-root "$RELEASE_WORKSPACE" --candidate-dir "$RELEASE_CANDIDATE" \
--channel stable --npm "$RELEASE_NPM" --build-web \
--commit --tag --push --tag-name "catalog-v$RELEASE_VERSION" --json
```
The last command is a strict non-mutating preview because `--apply` is absent.
Review its output, then repeat it with `--apply` to publish. The website's locked
build dependencies must already be installed before `--build-web`. The publisher
sanitizes the build and Git environments and pushes the verified immutable
website commit/tag. Source tag publication and registry package availability
must be complete before candidate generation.
Source tags, registry packages, and a signed module catalog do not imply that a
new runtime distribution exists. While runtime images are held, leave the Meta
Gitea runtime Release held too: a normal source-only Release can replace Gitea's
`releases/latest` discovery result despite having no deployment assets. The
deployer still requires an explicit signed manifest, digest, and trusted
keyring; it does not deploy a tag or module catalog directly.
## Betriebsablauf (DE)
`release-catalog.py full-registry` übernimmt den vollständigen Paketbestand,
die Registry-Sperrdatei, das Wheel-Verzeichnis, die WebUI-Archive und den
konfigurierten Signaturschlüssel. Zuerst mit
`generate-release-package-set.py --profile full` die exakten Meta-Paketversionen
ermitteln und mit `resolve-package-artifacts.py` die veröffentlichten Artefakte
herunterladen. Lokal neu gebaute Wheels sind kein Ersatz. Der Kandidat prüft
Paketidentitäten, Dateigrößen und Hashes und erzeugt alle Einträge aus den
unveränderlichen getaggten Manifesten. Die Registry und ihre veröffentlichende
CI bleiben eine Vertrauensgrundlage; dies ist kein unabhängiger Nachweis eines
reproduzierbaren Builds aus dem Quellcode.
Jedes neu veröffentlichte Repository wird mit `--selected-repository`
angegeben. Nur diese Auswahl muss mit dem sauberen, versionsgleichen HEAD eines
benannten Branches und dem annotierten lokalen und entfernten Tag übereinstimmen.
Unveränderte Pakete behalten ihren ursprünglichen Tag, auch wenn auf `main`
bereits eine spätere Workflow-Korrektur liegt. Alle Quelladressen müssen dem
registrierten Ursprung entsprechen; Commit und annotiertes Tag-Objekt werden
für jeden Eintrag gebunden. Git-Ersetzungsobjekte oder fremde Git-Konfiguration
können dabei keinen anderen Manifestbaum unterschieben.
Vor dem Signieren werden der bestehende Website-Katalog und sein Schlüsselbund
geprüft. Ein alter Katalog ohne signierten Schlüsselbund-Hash darf ausschließlich
durch diesen vollständigen Neuaufbau migriert werden: Seine Signatur muss gültig
sein und der gesamte Schlüsselbund exakt den konfigurierten bekannten Signierern
entsprechen. Alte Einträge oder Artefakt-Hashes werden nicht übernommen. Der neue
Katalog bindet den unveränderten Schlüsselbund-Hash; ein Schlüsselwechsel bleibt
ein eigener geprüfter Vorgang. Selektive Kandidaten verlangen weiterhin einen
bereits gebundenen Schlüsselbund. Kandidaten werden nur in neuen privaten
Verzeichnissen erzeugt und niemals überschrieben.
Das obige Befehlsbeispiel wird auf dem vertrauenswürdigen Host ausgeführt. Dafür
die private Python-Umgebung und einen absoluten `--npm`-Pfad zu Node 22 verwenden;
unter Flatpak die Host-Werkzeuge über `flatpak-spawn --host` aufrufen. Die
gesperrten Website-Build-Abhängigkeiten vorher installieren. Keine
Vertrauensprüfung umgehen und keine globalen Rechte ändern. Die Artefaktordner
müssen privat und bei der Auflösung leer sein. Vor der Kandidatenerzeugung
müssen Quell-Tags und Registry-Pakete vollständig veröffentlicht sein.
Die Veröffentlichung zunächst mit `publish-candidate --commit --tag --push
--build-web` ohne `--apply` prüfen und erst nach Prüfung mit `--apply` ausführen.
Solange Laufzeit-Images zurückgestellt sind, bleibt auch das Meta-Gitea-Runtime-
Release zurückgestellt: Ein reines Quellcode-Release könnte sonst als neuestes
Release erscheinen. Eine Installation benötigt weiterhin ein signiertes
Laufzeitmanifest, dessen Digest und einen explizit vertrauenswürdigen Schlüsselbund.
+7 -2
View File
@@ -189,8 +189,13 @@ than invoking `pip`, `npm`, or Git on the target host.
## Public module directory
`tools/release/publish-release-catalog.sh` resolves the selected package set and
registry lock before it creates a catalog. Catalog entries are synthesized from
For an operator-reviewed full publication, use
`tools/release/release-catalog.py full-registry` followed by the same tool's
`publish-candidate` command. Resolve the package set and registry lock first;
the older direct-write shell wrapper is not the strict candidate publication
path. See [Full registry candidates / Vollständige Registry-Kandidaten](FULL_REGISTRY_CANDIDATES.md)
for the private host runtime, exact artifact checks, and legacy keyring transition.
Catalog entries are synthesized from
the exact tagged module manifests, never from a hand-maintained module list or
the current workspace. Each entry binds its Python wheel and optional WebUI
tarball to the registry URL, filename, size, SHA-256, package identity, source
+9
View File
@@ -517,6 +517,15 @@ tree and requires byte-for-byte equality with those validated objects. Tags and
remote branch updates then reference that exact commit SHA rather than the
mutable worktree `HEAD`.
For a full registry-backed release, first build a fresh private candidate using
`release-catalog.py full-registry`. Pass `--selected-repository` for newly
released HEAD-bound units, not every unchanged package in the full profile.
The command independently checks all full-profile registry bytes and annotated
tag provenance, then feeds this same strict `publish-candidate` transaction.
It does not create Gitea runtime Releases or dispatch image builds. See
[Full registry candidates / Vollständige Registry-Kandidaten](FULL_REGISTRY_CANDIDATES.md)
for the complete EN/DE workflow and the narrowly scoped legacy keyring transition.
Published channels are expected below the public catalog base URL:
- `https://govoplan.add-ideas.de/catalogs/v1/channels/stable.json`
+466
View File
@@ -0,0 +1,466 @@
from __future__ import annotations
import base64
from contextlib import ExitStack
import csv
from copy import deepcopy
import hashlib
import io
import json
import os
from pathlib import Path
import sys
import tarfile
import tempfile
import unittest
from unittest.mock import patch
from urllib.parse import quote
import zipfile
from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tools/release"))
from govoplan_release import full_catalog # noqa: E402
from govoplan_release.artifact_identity import selected_artifact_identity_issues # noqa: E402
from govoplan_release.catalog import canonical_hash # noqa: E402
from govoplan_release.model import RepositorySpec # noqa: E402
from govoplan_release.registry_reference import registry_entry_source # noqa: E402
from govoplan_release.selective_catalog import ( # noqa: E402
load_authenticated_catalog_base, public_key_base64, signature,
)
from govoplan_release.source_provenance import ( # noqa: E402
SourceTagProvenanceIssue, catalog_source_selection,
registered_source_origin_issues,
)
from govoplan_release.version_alignment import candidate_catalog_version_issues # noqa: E402
class FullRegistryCatalogTests(unittest.TestCase):
def setUp(self) -> None:
self.temp = tempfile.TemporaryDirectory()
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.web = self.root / "addideas-govoplan-website"
self.wheels = self.root / "wheels"
self.npm = self.root / "npm"
self.wheels.mkdir(mode=0o700)
self.npm.mkdir(mode=0o700)
self.key = Ed25519PrivateKey.generate()
self.keypath = self.root / "key.pem"
self.keypath.write_bytes(self.key.private_bytes(
serialization.Encoding.PEM, serialization.PrivateFormat.PKCS8,
serialization.NoEncryption(),
))
self.keypath.chmod(0o600)
self.keyring = {
"keyring_version": "1", "keys": [{
"key_id": "known-key", "public_key": public_key_base64(self.key),
"status": "active",
}],
}
self.base = {
"catalog_version": "1", "channel": "stable", "sequence": 1,
"core_release": {"version": "1.0.0"}, "modules": [],
"release": {},
}
self.write_base()
self.package_set = {
"schema_version": "1", "release_version": "1.2.3", "profile": "full",
"registries": {
"python": "https://git.add-ideas.de/api/packages/GovOPlaN/pypi/simple",
"npm": "https://git.add-ideas.de/api/packages/GovOPlaN/npm/",
},
"python": [self.package("govoplan-core"), self.package("govoplan-demo")],
"webui": [self.package("govoplan-core", webui=True)],
}
self.seal(self.package_set, "package_set_sha256")
self.lock = {
"schema_version": "1", "release_version": "1.2.3", "profile": "full",
"registries": self.package_set["registries"],
"package_set_sha256": self.package_set["package_set_sha256"],
"python": [], "webui": [],
}
for row in self.package_set["python"]:
path = self.wheel(row["name"])
url = full_catalog._tool("resolve-package-artifacts")["_python_artifact_url"](
self.package_set["registries"]["python"], package=row, filename=path.name,
)
self.lock["python"].append(self.artifact(row, path, url))
npm_package = self.package_set["webui"][0]
npm_path = self.npm / "govoplan-core-webui-1.2.3.tgz"
self.tarball(npm_path, "@govoplan/core-webui", "1.2.3")
url = self.package_set["registries"]["npm"] + quote(npm_package["name"], safe="") + "/-/1.2.3/core-webui-1.2.3.tgz"
row = self.artifact(npm_package, npm_path, url)
row["integrity"] = "sha512-" + base64.b64encode(hashlib.sha512(npm_path.read_bytes()).digest()).decode()
self.lock["webui"].append(row)
self.seal(self.lock, "lock_sha256")
self.set_path = self.root / "package-set.json"
self.lock_path = self.root / "package-lock.json"
self.write_inputs()
self.output = self.root / "candidate"
@staticmethod
def package(repo: str, *, webui: bool = False) -> dict:
row = {
"name": "@govoplan/core-webui" if webui else repo, "version": "1.2.3",
"repository": repo, "tag": "v1.2.3",
"commit": ("a" if repo == "govoplan-core" else "b") * 40,
}
if not webui:
row["extras"] = ["server"] if repo == "govoplan-core" else []
return row
@staticmethod
def artifact(package: dict, path: Path, url: str) -> dict:
encoded = path.read_bytes()
return {**package, "filename": path.name, "url": url,
"sha256": hashlib.sha256(encoded).hexdigest(), "size": len(encoded)}
@staticmethod
def seal(payload: dict, field: str) -> None:
payload.pop(field, None)
payload[field] = hashlib.sha256(json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
def write_inputs(self) -> None:
self.set_path.write_text(json.dumps(self.package_set))
self.lock_path.write_text(json.dumps(self.lock))
def write_base(self) -> None:
self.base.pop("signatures", None)
self.base["signatures"] = [signature(self.base, key_id="known-key", private_key=self.key)]
folder = self.web / "public/catalogs/v1"
(folder / "channels").mkdir(parents=True, exist_ok=True)
(folder / "channels/stable.json").write_text(json.dumps(self.base))
(folder / "keyring.json").write_text(json.dumps(self.keyring))
def wheel(self, package: str) -> Path:
stem = package.replace("-", "_")
info = f"{stem}-1.2.3.dist-info"
files = {
f"{stem}/__init__.py": b"VALUE = 1\n",
f"{info}/METADATA": f"Metadata-Version: 2.1\nName: {package}\nVersion: 1.2.3\n".encode(),
f"{info}/WHEEL": b"Wheel-Version: 1.0\nRoot-Is-Purelib: true\nTag: py3-none-any\n",
}
record = io.StringIO()
writer = csv.writer(record, lineterminator="\n")
for name, value in files.items():
writer.writerow((name, "", len(value)))
writer.writerow((f"{info}/RECORD", "", ""))
files[f"{info}/RECORD"] = record.getvalue().encode()
path = self.wheels / f"{stem}-1.2.3-py3-none-any.whl"
with zipfile.ZipFile(path, "w") as archive:
for name, value in files.items():
archive.writestr(name, value)
return path
@staticmethod
def tarball(path: Path, name: str, version: str, *, duplicate: bool = False) -> None:
encoded = json.dumps({"name": name, "version": version}).encode()
with tarfile.open(path, "w:gz") as archive:
for _ in range(2 if duplicate else 1):
member = tarfile.TarInfo("package/package.json")
member.size = len(encoded)
archive.addfile(member, io.BytesIO(encoded))
def build(self, *, provenance_errors=(), origin_errors=()) -> dict:
registry_generator = full_catalog._tool("generate-release-catalog")
tools = {name: dict(full_catalog._tool(name)) for name in (
"generate-release-catalog", "generate-release-package-set", "resolve-package-artifacts",
)}
tools["generate-release-package-set"]["generate_package_set"] = lambda **kwargs: self.package_set
self.provenance = {
row["repository"]: {"commit_sha": row["commit"], "tag_object_sha": str(index + 1) * 40}
for index, row in enumerate(self.package_set["python"])
}
entry = {
"module_id": "demo", "name": "Demo", "version": "1.2.3",
"python_package": "govoplan-demo",
"python_ref": "govoplan-demo @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-demo.git@v1.2.3",
}
with ExitStack() as stack:
stack.enter_context(patch.object(full_catalog, "_tool", side_effect=lambda name: tools[name]))
stack.enter_context(patch.dict(registry_generator["_catalog_payload"].__globals__, {
"synthesize_repository_catalog_entries": lambda **kwargs: (dict(entry),),
}))
stack.enter_context(patch.object(full_catalog, "enforce_selected_version_alignment"))
stack.enter_context(patch.object(full_catalog, "registered_source_origin_issues", return_value=origin_errors))
self.provenance_check = stack.enter_context(patch.object(full_catalog, "source_tag_provenance_issues", return_value=provenance_errors))
stack.enter_context(patch.object(full_catalog, "selected_source_provenance", return_value=self.provenance))
return full_catalog.build_full_registry_candidate(
package_set_path=self.set_path, package_lock_path=self.lock_path,
wheelhouse=self.wheels, webui_packages=self.npm, output_dir=self.output,
selected_repositories=("govoplan-core",),
signing_keys=(f"known-key={self.keypath}",), workspace_root=self.root,
)
def candidate(self) -> dict:
return json.loads((self.output / "channels/stable.json").read_text())
def test_full_candidate_uses_registry_bytes_and_preserves_unchanged_tag_provenance(self) -> None:
result = self.build()
candidate = self.candidate()
self.assertEqual("ready", result["status"])
self.assertEqual(2, result["package_count"])
self.assertEqual(1, result["selected_count"])
self.assertEqual(self.keyring, json.loads((self.output / "keyring.json").read_text()))
self.assertEqual(canonical_hash(self.keyring), candidate["release"]["keyring_sha256"])
self.assertEqual(2, len(candidate["release"]["artifacts"]))
self.assertIn("/pypi/files/", candidate["core_release"]["python_ref"])
self.assertEqual((), candidate_catalog_version_issues(candidate))
self.assertEqual((), selected_artifact_identity_issues(candidate))
sources = catalog_source_selection(candidate)
self.assertEqual((), sources.issues)
self.assertEqual({"govoplan-core": "1.2.3"}, sources.selected_versions)
self.assertEqual({"govoplan-core": "1.2.3", "govoplan-demo": "1.2.3"}, sources.all_versions)
self.assertEqual("b" * 40, sources.selected_commits["govoplan-demo"])
self.assertEqual("2" * 40, sources.selected_tag_objects["govoplan-demo"])
self.assertEqual(2, self.provenance_check.call_count)
for call in self.provenance_check.call_args_list:
self.assertEqual({"govoplan-core"}, call.kwargs["require_head_repos"])
for path in [self.output, *self.output.rglob("*")]:
self.assertEqual(0o700 if path.is_dir() else 0o600, path.stat().st_mode & 0o777)
def test_legacy_base_is_authenticated_but_remains_rejected_by_selective(self) -> None:
self.build()
with self.assertRaisesRegex(ValueError, "does not pin"):
load_authenticated_catalog_base(
base_catalog=None, base_keyring=None, web_root=self.web,
channel="stable", public_base_url="https://unused.example",
signer_public_keys={"known-key": public_key_base64(self.key)},
)
def test_injected_key_or_mismatched_pinned_keyring_is_rejected(self) -> None:
for mutation in ("extra-key", "bad-hash"):
with self.subTest(mutation=mutation):
if mutation == "extra-key":
self.keyring["keys"].append({"key_id": "injected", "status": "active", "public_key": public_key_base64(Ed25519PrivateKey.generate())})
else:
self.keyring["keys"] = self.keyring["keys"][:1]
self.base["release"]["keyring_sha256"] = "f" * 64
self.write_base()
with self.assertRaises(ValueError):
self.build()
self.assertFalse(self.output.exists())
def test_tampered_base_signature_is_rejected(self) -> None:
path = self.web / "public/catalogs/v1/channels/stable.json"
payload = json.loads(path.read_text())
payload["sequence"] = 999
path.write_text(json.dumps(payload))
with self.assertRaisesRegex(ValueError, "signature verification"):
self.build()
def test_wrong_registry_bytes_and_reused_candidate_fail_closed(self) -> None:
self.build()
original = (self.output / "channels/stable.json").read_bytes()
with self.assertRaisesRegex(ValueError, "must not already exist"):
self.build()
self.assertEqual(original, (self.output / "channels/stable.json").read_bytes())
self.output = self.root / "candidate-2"
wheel = self.wheels / self.lock["python"][0]["filename"]
with wheel.open("ab") as stream:
stream.write(b"tampered")
with self.assertRaisesRegex(ValueError, "bytes differ"):
self.build()
self.assertFalse(self.output.exists())
def test_webui_identity_and_url_changes_are_rejected(self) -> None:
row = self.lock["webui"][0]
original = row["url"]
for url in (
original + "?alternate=true", original.replace("/-/1.2.3/", "/-/9.9.9/"),
original.replace("/npm/", "/npm/../other/"),
original.replace("/npm/", "/npm/%2e%2e/other/"),
original.replace("/npm/", "/npm/%252e%252e/other/"),
):
with self.subTest(url=url):
row["url"] = url
self.seal(self.lock, "lock_sha256")
self.write_inputs()
with self.assertRaisesRegex(ValueError, "URL differs"):
self.build()
def test_duplicate_missing_and_symlinked_artifacts_are_rejected(self) -> None:
row = self.lock["webui"][0]
self.lock["webui"].append(dict(row))
with self.assertRaisesRegex(ValueError, "duplicate/missing"):
full_catalog.verify_registry_artifacts(package_set=self.package_set, lock=self.lock, wheelhouse=self.wheels, webui_packages=self.npm)
self.lock["webui"].pop()
path = self.npm / row["filename"]
moved = self.root / "moved.tgz"
path.rename(moved)
path.symlink_to(moved)
with self.assertRaises(OSError):
full_catalog.verify_registry_artifacts(package_set=self.package_set, lock=self.lock, wheelhouse=self.wheels, webui_packages=self.npm)
def test_archive_metadata_is_bounded_and_not_ambiguous(self) -> None:
path = self.npm / "duplicate.tgz"
self.tarball(path, "@govoplan/core-webui", "1.2.3", duplicate=True)
with self.assertRaisesRegex(ValueError, "duplicate"):
full_catalog._inspect_npm_metadata(path, name="@govoplan/core-webui", version="1.2.3")
def test_origin_or_tag_provenance_failure_prevents_output(self) -> None:
issue = SourceTagProvenanceIssue("govoplan-core", "v1.2.3", "wrong immutable identity")
for kwargs in ({"origin_errors": (issue,)}, {"provenance_errors": (issue,)}):
with self.subTest(kwargs=kwargs), self.assertRaisesRegex(ValueError, "gate failed"):
self.build(**kwargs)
self.assertFalse(self.output.exists())
def test_registered_source_origin_requires_exact_fetch_and_push_targets(self) -> None:
repo = self.root / "govoplan-core"
repo.mkdir()
spec = RepositorySpec("govoplan-core", "system", "kernel", "git@example.test:trusted/core.git", "govoplan-core")
with patch("govoplan_release.source_provenance.load_repository_specs", return_value=(spec,)):
for targets in ((spec.remote, spec.remote), ("git@evil.test:core.git", spec.remote), (spec.remote, "git@evil.test:core.git")):
with self.subTest(targets=targets), patch("govoplan_release.source_provenance.git_text", side_effect=targets):
issues = registered_source_origin_issues(repo_versions={"govoplan-core": "1.2.3"}, workspace=self.root, remote="origin")
self.assertEqual(targets != (spec.remote, spec.remote), bool(issues))
def test_registry_metadata_cannot_cross_wire_webui_or_archive_identity(self) -> None:
self.build()
candidate = self.candidate()
entry = candidate["core_release"]
entry["webui_package"] = "@govoplan/files-webui"
with self.assertRaisesRegex(ValueError, "another source repository"):
registry_entry_source(entry)
candidate = self.candidate()
candidate["release"]["artifacts"][0]["archive_sha256"] = "f" * 64
self.assertIn("matching inspected wheel", " ".join(selected_artifact_identity_issues(candidate)))
def test_registry_version_and_selected_source_identity_must_agree(self) -> None:
self.build()
for field in ("commit_sha", "tag_object_sha"):
candidate = self.candidate()
candidate["release"]["selected_units"][0][field] = "f" * 40
self.assertIn("differs", " ".join(issue.message for issue in catalog_source_selection(candidate).issues))
candidate = self.candidate()
candidate["modules"][0]["artifact_integrity"]["python"]["git_ref"] = "v9.9.9"
self.assertTrue(candidate_catalog_version_issues(candidate))
def test_repeated_module_projections_must_bind_identical_python_and_webui_bytes(self) -> None:
self.build()
for kind in ("python", "webui"):
for reversed_order in (False, True):
with self.subTest(kind=kind, reversed_order=reversed_order):
candidate = self.candidate()
entry = deepcopy(candidate["core_release"])
entry["module_id"] = "another-core-projection"
artifact = entry["artifact_integrity"][kind]
artifact["sha256"] = "f" * 64
if kind == "python":
entry["python_ref"] = artifact["ref"] = entry["python_ref"].split("#sha256=", 1)[0] + "#sha256=" + "f" * 64
if reversed_order:
original = candidate["core_release"]
candidate["core_release"] = entry
entry = original
candidate["modules"].append(entry)
self.assertIn(f"conflicting {kind}", " ".join(selected_artifact_identity_issues(candidate)))
self.assertTrue(candidate_catalog_version_issues(candidate))
candidate = self.candidate()
repeated = deepcopy(candidate["modules"][0])
repeated["module_id"] = "second-demo-projection"
candidate["modules"].append(repeated)
self.assertEqual((), selected_artifact_identity_issues(candidate))
self.assertEqual((), candidate_catalog_version_issues(candidate))
def test_metadata_inspection_uses_the_opened_archive_not_a_replaced_path(self) -> None:
path = self.npm / "original.tgz"
replacement = self.npm / "replacement.tgz"
saved = self.npm / "saved.tgz"
self.tarball(path, "@govoplan/incorrect-webui", "1.2.3")
self.tarball(replacement, "@govoplan/core-webui", "1.2.3")
real_open = tarfile.open
def replace_path(*args, **kwargs):
self.assertIn("fileobj", kwargs)
path.rename(saved)
replacement.rename(path)
return real_open(*args, **kwargs)
with patch("govoplan_release.full_catalog.tarfile.open", side_effect=replace_path):
with self.assertRaisesRegex(ValueError, "metadata differs"):
full_catalog._inspect_npm_metadata(path, name="@govoplan/core-webui", version="1.2.3")
def test_registry_url_provenance_rejects_package_version_and_traversal_mismatch(self) -> None:
self.build()
for old, new in (
("/govoplan-core/1.2.3/", "/govoplan-files/1.2.3/"),
("/govoplan-core/1.2.3/", "/govoplan-core/9.9.9/"),
("/pypi/files/", "/pypi/files/%2e%2e/"),
("/pypi/files/", "/pypi/files/%252e%252e/"),
):
with self.subTest(new=new):
entry = self.candidate()["core_release"]
artifact = entry["artifact_integrity"]["python"]
artifact["url"] = artifact["url"].replace(old, new)
artifact["ref"] = entry["python_ref"] = entry["python_ref"].replace(old, new)
with self.assertRaises(ValueError):
registry_entry_source(entry)
def test_package_set_must_match_fixed_meta_pins_not_just_its_own_hash(self) -> None:
payload = deepcopy(self.package_set)
payload["python"][1]["version"] = "9.9.9"
self.seal(payload, "package_set_sha256")
self.set_path.write_text(json.dumps(payload))
with self.assertRaisesRegex(ValueError, "exact Meta full pins"):
self.build()
self.assertFalse(self.output.exists())
def test_package_set_git_reads_ignore_caller_redirection(self) -> None:
tool = full_catalog._tool("generate-release-package-set")
with patch.dict(os.environ, {"GIT_DIR": "/outside", "GIT_CONFIG_GLOBAL": "/outside/config", "PATH": "/outside/bin"}):
with patch("subprocess.check_output", return_value="a" * 40 + "\n") as execute:
self.assertEqual("a" * 40, tool["_git"](self.root, "rev-parse", "HEAD"))
self.assertEqual("/usr/bin/git", execute.call_args.args[0][0])
self.assertNotIn("GIT_DIR", execute.call_args.kwargs["env"])
self.assertEqual(os.devnull, execute.call_args.kwargs["env"]["GIT_CONFIG_GLOBAL"])
self.assertEqual("/usr/bin:/bin", execute.call_args.kwargs["env"]["PATH"])
def test_unchanged_real_annotated_ancestor_is_valid_but_selecting_it_requires_head(self) -> None:
from tests.test_release_source_provenance import git, git_text, make_repo
from govoplan_release.source_provenance import source_tag_provenance_issues
workspace = self.root / "source-workspace"
workspace.mkdir()
repo, _remote = make_repo(workspace, self.root, "govoplan-access", "1.2.3")
git(repo, "tag", "-a", "v1.2.3", "-m", "reviewed immutable package")
git(repo, "push", "origin", "refs/tags/v1.2.3")
tagged_commit = git_text(repo, "rev-parse", "HEAD")
(repo / "workflow-only.txt").write_text("post-tag workflow repair\n")
git(repo, "add", "workflow-only.txt")
git(repo, "commit", "-m", "repair workflow without replacing package")
git(repo, "push", "origin", "main")
common = {
"repo_versions": {"govoplan-access": "1.2.3"}, "workspace": workspace,
"expected_commits": {"govoplan-access": tagged_commit},
"expected_tag_objects": {"govoplan-access": git_text(repo, "rev-parse", "refs/tags/v1.2.3")},
}
self.assertEqual((), source_tag_provenance_issues(**common))
issues = source_tag_provenance_issues(**common, require_head_repos=("govoplan-access",))
self.assertIn("not selected HEAD", " ".join(issue.message for issue in issues))
def test_tagged_manifest_synthesis_ignores_local_git_replacement_objects(self) -> None:
from tests.test_release_source_provenance import git, git_text, make_repo
from govoplan_release.catalog_entry_synthesis import materialized_source_tree
workspace = self.root / "materialization-workspace"
workspace.mkdir()
repo, _remote = make_repo(workspace, self.root, "govoplan-access", "1.2.3")
original = (repo / "pyproject.toml").read_text()
tagged_commit = git_text(repo, "rev-parse", "HEAD")
git(repo, "tag", "-a", "v1.2.3", "-m", "reviewed immutable package")
(repo / "pyproject.toml").write_text(original.replace("1.2.3", "9.9.9"))
git(repo, "add", "pyproject.toml")
git(repo, "commit", "-m", "unreviewed replacement tree")
git(repo, "replace", tagged_commit, git_text(repo, "rev-parse", "HEAD"))
with patch.dict(os.environ, {"GIT_DIR": str(self.root / "outside"), "PATH": "/outside/bin"}):
with materialized_source_tree(repo, source_ref="v1.2.3") as source:
self.assertEqual(original, (source / "pyproject.toml").read_text())
if __name__ == "__main__":
unittest.main()
@@ -9,6 +9,7 @@ import json
from pathlib import Path
import re
import subprocess
import sys
import tomllib
@@ -216,10 +217,17 @@ def _extras(value: str | None) -> list[str]:
def _git(repository: Path, *arguments: str) -> str:
release_root = str(Path(__file__).resolve().parent)
if release_root not in sys.path:
sys.path.insert(0, release_root)
from govoplan_release.git_state import sanitized_git_environment, scoped_git_command
return subprocess.check_output(
["git", "-C", str(repository), *arguments],
scoped_git_command(repository, "-C", str(repository), *arguments),
text=True,
stderr=subprocess.DEVNULL,
env=sanitized_git_environment(),
timeout=30,
).strip()
@@ -14,6 +14,8 @@ import re
import stat
import zipfile
from .registry_reference import registry_artifact_conflicts, registry_entry_source
WHEEL_PAYLOAD_ALGORITHM = "govoplan-wheel-declared-payload-v1"
INSTALLED_PAYLOAD_ALGORITHM = "govoplan-installed-record-payload-v1"
@@ -185,10 +187,23 @@ def selected_artifact_identity_issues(payload: object) -> tuple[str, ...]:
modules = payload.get("modules")
if isinstance(modules, list):
entries.extend(modules)
conflicts = registry_artifact_conflicts(entries)
if conflicts:
return conflicts
package_by_repo: dict[str, tuple[str, str]] = {}
registry_artifacts: dict[str, dict[str, object]] = {}
for entry in entries:
if not isinstance(entry, dict):
continue
try:
registry_source = registry_entry_source(entry)
except ValueError as exc:
return (str(exc),)
if registry_source is not None:
package = str(entry["python_package"])
package_by_repo[registry_source.repository] = (package, registry_source.version)
registry_artifacts[package] = entry["artifact_integrity"]["python"]
continue
python_ref = entry.get("python_ref")
match = _PYTHON_REF.search(python_ref) if isinstance(python_ref, str) else None
package_name = entry.get("python_package")
@@ -219,6 +234,14 @@ def selected_artifact_identity_issues(payload: object) -> tuple[str, ...]:
issues: list[str] = []
if malformed_artifacts:
issues.append("release.artifacts contains malformed or duplicate identities")
for package, registry_artifact in registry_artifacts.items():
artifact = artifacts_by_package.get(package)
if artifact is None or (
artifact.get("archive_sha256") != registry_artifact.get("sha256")
or artifact.get("archive_size") != registry_artifact.get("size")
or artifact.get("package_version") != registry_artifact["registry_identity"].rsplit("@", 1)[-1]
):
issues.append(f"registry artifact {package} has no matching inspected wheel byte identity")
seen_repos: set[str] = set()
for unit in selected_units:
if not isinstance(unit, dict):
@@ -19,6 +19,7 @@ from typing import Iterator
from govoplan_core.core.modules import ModuleManifest
from govoplan_core.core.versioning import version_satisfies_range
from .git_state import sanitized_git_environment, scoped_git_command
from .workspace import load_repository_specs, resolve_repo_path
@@ -255,20 +256,19 @@ def materialized_source_tree(root: Path, *, source_ref: str | None) -> Iterator[
source_root = temporary / "source"
source_root.mkdir()
result = subprocess.run(
[
"git",
"-C",
str(root),
scoped_git_command(
root, "-C", str(root),
"archive",
"--format=tar",
f"--output={archive_path}",
source_ref,
],
),
check=False,
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
text=True,
timeout=30,
env=sanitized_git_environment(),
)
if result.returncode != 0:
detail = result.stderr.strip() or "Git archive failed"
@@ -0,0 +1,389 @@
"""Build a private full-profile candidate from exact verified registry bytes."""
from __future__ import annotations
import base64
from datetime import UTC, datetime, timedelta
from functools import lru_cache
import hashlib
import json
import os
from pathlib import Path, PurePosixPath
import re
import runpy
import stat
import tarfile
from typing import Any
from urllib.parse import quote
from .artifact_identity import inspect_python_wheel, selected_artifact_identity_issues
from .candidate_artifact import (
ensure_private_candidate_root, harden_private_candidate_tree,
validate_release_channel,
)
from .catalog import canonical_hash
from .module_directory import write_module_directory
from .selective_catalog import (
authenticate_base_catalog_signatures, authenticate_base_keyring,
configured_signer_public_keys, enforce_selected_version_alignment,
next_sequence, parse_signing_key, read_bounded_json_source, signature,
validate_catalog_object,
)
from .source_provenance import (
registered_source_origin_issues, selected_source_provenance,
source_tag_provenance_issues,
)
from .version_alignment import candidate_catalog_version_issues
from .workspace import META_ROOT, resolve_workspace_root, website_root
MAX_ARTIFACT_BYTES = 512 * 1024 * 1024
@lru_cache(maxsize=3)
def _tool(name: str) -> dict[str, Any]:
# These are fixed, operator-controlled release modules, not caller paths.
if name not in {
"generate-release-package-set", "generate-release-catalog",
"resolve-package-artifacts",
}:
raise ValueError("unknown registry release tool")
return runpy.run_path(str(META_ROOT / "tools" / "release" / f"{name}.py"))
def authenticate_full_rebuild_base(
*, web_root: Path, channel: str, signer_public_keys: dict[str, str],
) -> tuple[dict[str, Any], dict[str, Any]]:
"""Authenticate the fixed website pair without reusing legacy entry data.
Only this full rebuild may migrate a legacy catalog without a keyring hash.
In that case *every* active website key must exactly equal a configured
signer; injecting an additional website key cannot extend trust. Selective
candidates retain their stricter existing hash-pinned-base requirement.
"""
root = web_root / "public" / "catalogs" / "v1"
catalog = read_bounded_json_source(root / "channels" / f"{channel}.json", label="published base catalog")
keyring = read_bounded_json_source(root / "keyring.json", label="published website keyring")
if not isinstance(catalog, dict) or not isinstance(keyring, dict):
raise ValueError("published website catalog/keyring must be objects")
trusted_keys = authenticate_base_keyring(keyring)
release = catalog.get("release")
pinned = release.get("keyring_sha256") if isinstance(release, dict) else None
if pinned is None:
if trusted_keys != signer_public_keys or len(keyring["keys"]) != len(trusted_keys):
raise ValueError("legacy full rebuild requires exactly the configured known website signers")
elif pinned != canonical_hash(keyring):
raise ValueError("published base catalog does not pin its exact website keyring")
if any(trusted_keys.get(key) != value for key, value in signer_public_keys.items()):
raise ValueError("full rebuild cannot introduce or replace a website signer")
authenticate_base_catalog_signatures(
catalog, base_trusted_keys=trusted_keys, configured_signers=signer_public_keys,
)
validation = validate_catalog_object(
catalog, approved_channel=channel, signer_public_keys=signer_public_keys,
)
if validation.get("valid") is not True:
raise ValueError(f"published base catalog failed validation: {validation.get('error')}")
return catalog, keyring
def build_full_registry_candidate(
*, package_set_path: Path, package_lock_path: Path,
wheelhouse: Path, webui_packages: Path, output_dir: Path,
selected_repositories: tuple[str, ...], signing_keys: tuple[str, ...],
workspace_root: Path | str | None = None, channel: str = "stable",
source_remote: str = "origin", public_base_url: str = "https://govoplan.add-ideas.de",
expires_days: int = 90, sequence: int | None = None,
) -> dict[str, object]:
channel = validate_release_channel(channel)
if not isinstance(expires_days, int) or isinstance(expires_days, bool) or not 1 <= expires_days <= 365:
raise ValueError("catalog expiry must be between 1 and 365 days")
workspace = resolve_workspace_root(workspace_root)
ensure_private_candidate_root(workspace)
output = output_dir.expanduser().absolute()
ensure_private_candidate_root(output.parent, create=True)
if output.exists() or output.is_symlink():
raise ValueError("full candidate output must not already exist")
parsed_keys = tuple(parse_signing_key(value) for value in signing_keys)
if not parsed_keys:
raise ValueError("full candidate needs a configured signing key")
signer_keys = configured_signer_public_keys(parsed_keys)
base, keyring = authenticate_full_rebuild_base(
web_root=website_root(workspace), channel=channel, signer_public_keys=signer_keys,
)
package_set = _hashed_json(package_set_path, "package_set_sha256")
lock = _hashed_json(package_lock_path, "lock_sha256")
generator = _tool("generate-release-catalog")
version = package_set.get("release_version")
if package_set.get("profile") != "full" or not isinstance(version, str):
raise ValueError("full candidate requires the complete full-profile package set")
expected = _tool("generate-release-package-set")["generate_package_set"](
core_version=version, requirements=META_ROOT / "requirements-release.txt",
workspace=workspace, profile="full",
meta_package=META_ROOT / "packages/govoplan-meta/pyproject.toml",
)
if package_set != expected:
raise ValueError("package set differs from exact Meta full pins or immutable tag metadata")
generator["_validate_release_inputs"](package_set, lock, core_version=version)
if lock.get("registries") != package_set.get("registries"):
raise ValueError("artifact lock uses different package registries")
versions = {row["repository"]: row["version"] for row in package_set["python"]}
origin_failures = registered_source_origin_issues(
repo_versions=versions, workspace=workspace, remote=source_remote,
)
if origin_failures:
raise ValueError("Registered source origin gate failed: " + "; ".join(item.describe() for item in origin_failures))
selected = set(selected_repositories)
if not selected or len(selected) != len(selected_repositories) or not selected <= versions.keys():
raise ValueError("selected repositories must be unique members of the full package set")
selected_versions = {repo: versions[repo] for repo in sorted(selected)}
enforce_selected_version_alignment(repo_versions=selected_versions, workspace=workspace)
failures = source_tag_provenance_issues(
repo_versions=versions, workspace=workspace, remote=source_remote,
require_head_repos=selected,
)
if failures:
raise ValueError("Full source provenance gate failed: " + "; ".join(item.describe() for item in failures))
provenance = selected_source_provenance(repo_versions=versions, workspace=workspace)
wheel_identities = verify_registry_artifacts(
package_set=package_set, lock=lock, wheelhouse=wheelhouse,
webui_packages=webui_packages,
)
generated_at = datetime.now(tz=UTC)
resolved_sequence = sequence if sequence is not None else next_sequence(base, generated_at=generated_at)
if isinstance(resolved_sequence, bool) or not isinstance(resolved_sequence, int) or resolved_sequence <= int(base.get("sequence") or 0):
raise ValueError("full candidate sequence must advance the authenticated published channel")
# Fresh tagged manifests, including unchanged tagged ancestors; no legacy
# entry, registry hash, source URL or dependency contract is carried over.
candidate = generator["_catalog_payload"](
package_set=package_set, package_lock=lock, channel=channel,
sequence=resolved_sequence, generated_at=generated_at,
expires_at=generated_at + timedelta(days=expires_days), workspace=workspace,
public_base_url=public_base_url.rstrip("/"),
)
for entry in [candidate["core_release"], *candidate["modules"]]:
repo = entry["python_package"]
entry["source"] = {
"repository": repo, "tag": f"v{versions[repo]}",
"commit": provenance[repo]["commit_sha"],
"tag_object_sha": provenance[repo]["tag_object_sha"],
"repository_url": f"https://git.add-ideas.de/GovOPlaN/{repo}",
"revision_url": f"https://git.add-ideas.de/GovOPlaN/{repo}/commit/{provenance[repo]['commit_sha']}",
}
candidate["release"].update({
"selected_units": [
{"repo": repo, "version": versions[repo], "tag": f"v{versions[repo]}", **provenance[repo]}
for repo in sorted(selected)
],
"keyring_sha256": canonical_hash(keyring),
"artifacts": wheel_identities,
"base_catalog_sha256": canonical_hash(base),
})
# Recheck the exact objects used by synthesis, including unchanged source
# ancestors, before signing. No late tag movement can change this candidate.
origin_failures = registered_source_origin_issues(
repo_versions=versions, workspace=workspace, remote=source_remote,
)
if origin_failures:
raise ValueError("Registered source origin changed during synthesis: " + "; ".join(item.describe() for item in origin_failures))
failures = source_tag_provenance_issues(
repo_versions=versions, workspace=workspace, remote=source_remote,
require_head_repos=selected,
expected_commits={repo: row["commit_sha"] for repo, row in provenance.items()},
expected_tag_objects={repo: row["tag_object_sha"] for repo, row in provenance.items()},
)
if failures:
raise ValueError("Full source provenance changed during synthesis: " + "; ".join(item.describe() for item in failures))
failures = candidate_catalog_version_issues(candidate)
identity_failures = selected_artifact_identity_issues(candidate)
if failures or identity_failures:
raise ValueError("full candidate identity validation failed: " + "; ".join(
[item.message for item in failures] + list(identity_failures)
))
candidate["signatures"] = [signature(candidate, key_id=key, private_key=value) for key, value in parsed_keys]
validation = validate_catalog_object(candidate, approved_channel=channel, signer_public_keys=signer_keys)
if validation.get("valid") is not True:
raise ValueError(f"signed full candidate failed validation: {validation.get('error')}")
# Exclusive output creation preserves earlier reviewed candidates.
output.mkdir(mode=0o700)
(output / "channels").mkdir(mode=0o700)
catalog_path = output / "channels" / f"{channel}.json"
_write_private_json(catalog_path, candidate)
_write_private_json(output / "keyring.json", keyring)
write_module_directory(
catalog_payload=candidate, keyring_payload=keyring, output_root=output,
channel=channel, public_base_url=public_base_url,
)
result = {
"status": "ready", "candidate_dir": str(output), "catalog_path": str(catalog_path),
"channel": channel, "sequence": resolved_sequence,
"package_count": len(package_set["python"]), "webui_count": len(package_set["webui"]),
"selected_count": len(selected), "candidate_catalog_hash": canonical_hash(candidate),
"candidate_keyring_hash": canonical_hash(keyring), "validation_valid": True,
}
_write_private_json(output / "summary.json", result)
harden_private_candidate_tree(output)
return result
def _hashed_json(path: Path, field: str) -> dict[str, Any]:
payload = read_bounded_json_source(path, label=field)
if not isinstance(payload, dict):
raise ValueError(f"{field} input must be an object")
unsigned = dict(payload)
expected = unsigned.pop(field, None)
# Registry tools use their established ASCII-escaped canonical form.
encoded = json.dumps(unsigned, sort_keys=True, separators=(",", ":")).encode()
if expected != hashlib.sha256(encoded).hexdigest():
raise ValueError(f"{field} does not match its contents")
return payload
def verify_registry_artifacts(
*, package_set: dict[str, Any], lock: dict[str, Any],
wheelhouse: Path, webui_packages: Path,
) -> list[dict[str, object]]:
"""Compare exact registry bytes, metadata and source bindings without installs."""
identities = []
for group, root, suffix in (("python", wheelhouse, ".whl"), ("webui", webui_packages, ".tgz")):
ensure_private_candidate_root(root)
expected = {row["name"]: row for row in package_set[group]}
rows = lock[group]
if not isinstance(rows, list) or len(rows) != len(expected):
raise ValueError(f"artifact lock has duplicate/missing {group} rows")
filenames: set[str] = set()
seen: set[str] = set()
for row in rows:
selected = expected.get(row.get("name")) if isinstance(row, dict) else None
if selected is None or row["name"] in seen:
raise ValueError(f"artifact lock has unexpected/duplicate {group} identities")
seen.add(row["name"])
for key in ("name", "version", "repository", "tag", "commit"):
if row.get(key) != selected[key]:
raise ValueError("artifact lock differs from selected source identity")
filename = row.get("filename")
if not isinstance(filename, str) or re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,254}", filename) is None or not filename.endswith(suffix) or filename in filenames:
raise ValueError("artifact filename is invalid or duplicated")
filenames.add(filename)
path = root / filename
sha256, sha512, size = _hash_artifact(path)
if (sha256, size) != (row.get("sha256"), row.get("size")):
raise ValueError(f"registry artifact bytes differ from locked identity: {filename}")
if group == "python":
expected_url = _tool("resolve-package-artifacts")["_python_artifact_url"](
package_set["registries"]["python"], package=selected, filename=filename,
)
if row.get("url") != expected_url:
raise ValueError("Python artifact URL differs from the selected registry")
identity = inspect_python_wheel(path)
if (identity.package_name, identity.package_version, identity.archive_sha256, identity.archive_size) != (row["name"], row["version"], sha256, size):
raise ValueError("wheel metadata or bytes differ from the registry lock")
identities.append(identity.catalog_payload())
else:
expected_url = (
package_set["registries"]["npm"].rstrip("/") + "/"
+ quote(row["name"], safe="") + "/-/"
+ quote(row["version"], safe="") + "/"
+ quote(row["name"].split("/", 1)[1] + "-" + row["version"] + ".tgz", safe="")
)
if row.get("url") != expected_url:
raise ValueError("WebUI artifact URL differs from the selected registry")
if row.get("integrity") != "sha512-" + base64.b64encode(sha512).decode("ascii"):
raise ValueError("WebUI registry integrity differs from downloaded bytes")
_inspect_npm_metadata(
path, name=row["name"], version=row["version"],
expected_identity=(sha256, sha512, size),
)
actual = set()
for index, path in enumerate(root.iterdir()):
if index >= 1000:
raise ValueError("registry artifact directory exceeds its inspection bound")
actual.add(path.name)
if actual != filenames:
raise ValueError(f"registry directory contains unexpected or missing {group} files")
return sorted(identities, key=lambda row: str(row["package_name"]))
def _hash_artifact(path: Path) -> tuple[str, bytes, int]:
descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0))
try:
initial = os.fstat(descriptor)
if not stat.S_ISREG(initial.st_mode) or not 0 < initial.st_size <= MAX_ARTIFACT_BYTES:
raise ValueError("registry artifact must be a bounded regular file")
digest, sri, total = hashlib.sha256(), hashlib.sha512(), 0
while chunk := os.read(descriptor, 1024 * 1024):
total += len(chunk)
if total > MAX_ARTIFACT_BYTES:
raise ValueError("registry artifact exceeds its byte bound")
digest.update(chunk)
sri.update(chunk)
final = os.fstat(descriptor)
if (initial.st_ino, initial.st_size, initial.st_mtime_ns, initial.st_ctime_ns) != (final.st_ino, total, final.st_mtime_ns, final.st_ctime_ns):
raise ValueError("registry artifact changed while being inspected")
return digest.hexdigest(), sri.digest(), total
finally:
os.close(descriptor)
def _inspect_npm_metadata(
path: Path, *, name: str, version: str,
expected_identity: tuple[str, bytes, int] | None = None,
) -> None:
descriptor = os.open(path, os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0) | getattr(os, "O_NONBLOCK", 0))
try:
initial = os.fstat(descriptor)
if not stat.S_ISREG(initial.st_mode) or not 0 < initial.st_size <= MAX_ARTIFACT_BYTES:
raise ValueError("WebUI archive must be a bounded regular file")
digest, sri, total = hashlib.sha256(), hashlib.sha512(), 0
while chunk := os.read(descriptor, 1024 * 1024):
total += len(chunk)
if total > MAX_ARTIFACT_BYTES:
raise ValueError("WebUI archive exceeds its byte bound")
digest.update(chunk)
sri.update(chunk)
if expected_identity is not None and (digest.hexdigest(), sri.digest(), total) != expected_identity:
raise ValueError("WebUI archive changed before metadata inspection")
os.lseek(descriptor, 0, os.SEEK_SET)
with os.fdopen(os.dup(descriptor), "rb") as stream:
_inspect_npm_stream(stream, name=name, version=version)
final = os.fstat(descriptor)
if (initial.st_ino, initial.st_size, initial.st_mtime_ns, initial.st_ctime_ns) != (final.st_ino, total, final.st_mtime_ns, final.st_ctime_ns):
raise ValueError("WebUI archive changed during metadata inspection")
finally:
os.close(descriptor)
def _inspect_npm_stream(stream: Any, *, name: str, version: str) -> None:
found = False
total = 0
with tarfile.open(fileobj=stream, mode="r|gz") as archive:
for index, member in enumerate(archive):
total += member.size
parts = PurePosixPath(member.name).parts
if index >= 10000 or total > 1024 * 1024 * 1024 or member.size > 64 * 1024 * 1024:
raise ValueError("WebUI archive exceeds its inspection bound")
if not parts or parts[0] != "package" or ".." in parts or not (member.isfile() or member.isdir()):
raise ValueError("WebUI archive contains an unsafe member")
if member.name == "package/package.json":
if found or not member.isfile() or member.size > 1024 * 1024:
raise ValueError("WebUI archive has duplicate or oversized metadata")
stream = archive.extractfile(member)
if stream is None:
raise ValueError("WebUI archive metadata cannot be read")
metadata = json.loads(stream.read(1024 * 1024 + 1))
if not isinstance(metadata, dict) or (metadata.get("name"), metadata.get("version")) != (name, version):
raise ValueError("WebUI metadata differs from the registry lock")
found = True
if not found:
raise ValueError("WebUI archive metadata is missing")
def _write_private_json(path: Path, payload: object) -> None:
encoded = (json.dumps(payload, indent=2, sort_keys=True) + "\n").encode()
if len(encoded) > 16 * 1024 * 1024:
raise ValueError("candidate JSON exceeds its byte bound")
descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0), 0o600)
with os.fdopen(descriptor, "wb") as handle:
handle.write(encoded)
+20 -1
View File
@@ -27,7 +27,10 @@ from .candidate_artifact import validate_release_channel
from .model import CatalogPublishResult, CatalogPublishStep
from .module_directory import module_directory_payloads
from .selective_catalog import read_bounded_json_source, trusted_keys_from_keyring
from .source_provenance import catalog_source_selection, source_tag_provenance_issues
from .source_provenance import (
catalog_source_selection, registered_source_origin_issues,
source_tag_provenance_issues,
)
from .version_alignment import candidate_catalog_version_issues
from .workspace import (
DEFAULT_WORKSPACE_ROOT,
@@ -176,6 +179,22 @@ def publish_catalog_candidate(
for issue in version_issues
)
source_selection = catalog_source_selection(candidate_payload)
entries = [candidate_payload.get("core_release")]
if isinstance(candidate_payload.get("modules"), list):
entries.extend(candidate_payload["modules"])
if any(
isinstance(entry, dict)
and isinstance(entry.get("python_ref"), str)
and " @ https://" in entry["python_ref"]
for entry in entries
):
blockers.extend(
f"registered source origin: {issue.describe()}"
for issue in registered_source_origin_issues(
repo_versions=source_selection.all_versions,
workspace=workspace, remote=source_remote,
)
)
blockers.extend(
f"source provenance: {issue.describe()}"
for issue in source_selection.issues
@@ -0,0 +1,178 @@
"""Strict source identities for immutable, registry-backed catalog entries."""
from __future__ import annotations
import base64
from dataclasses import dataclass
import json
import re
from urllib.parse import unquote, urlsplit
_SHA256 = re.compile(r"[0-9a-f]{64}\Z")
_COMMIT = re.compile(r"[0-9a-f]{40}(?:[0-9a-f]{24})?\Z")
_REPO = re.compile(r"govoplan-[a-z0-9-]+\Z")
_VERSION = re.compile(r"\d+\.\d+\.\d+(?:[-+][0-9A-Za-z.-]+)?\Z")
_PYTHON = re.compile(
r"(?P<name>govoplan-[a-z0-9-]+)(?:\[[a-z0-9_,.-]+\])? @ "
r"(?P<url>https://\S+)#sha256=(?P<digest>[0-9a-f]{64})\Z"
)
@dataclass(frozen=True, slots=True)
class RegistrySource:
repository: str
version: str
commit: str
tag_object: str
def registry_artifact_conflicts(entries: list[object]) -> tuple[str, ...]:
"""Allow repeated module projections only when package artifacts agree."""
observed: dict[tuple[str, str], str] = {}
issues = []
for entry in entries:
if not isinstance(entry, dict):
continue
try:
source = registry_entry_source(entry)
except ValueError as exc:
issues.append(str(exc))
continue
if source is None:
continue
for kind, identity in entry["artifact_integrity"].items():
if kind not in {"python", "webui"}:
continue
key = (source.repository, kind)
encoded = json.dumps(identity, sort_keys=True, separators=(",", ":"))
if observed.setdefault(key, encoded) != encoded:
issues.append(f"conflicting {kind} registry artifacts for {source.repository}")
return tuple(issues)
def registry_entry_source(entry: dict[str, object]) -> RegistrySource | None:
"""Admit registry refs only with a complete matching artifact/source binding.
Git-backed catalogs keep their existing validation path. An HTTPS Python
requirement cannot masquerade as a source-only/non-Python entry when its
registry provenance is missing or inconsistent.
"""
ref = entry.get("python_ref")
if not isinstance(ref, str) or " @ https://" not in ref:
return None
match = _PYTHON.fullmatch(ref)
source = entry.get("source")
version = entry.get("version")
package = entry.get("python_package")
integrity = entry.get("artifact_integrity")
if (
match is None
or not isinstance(source, dict)
or not isinstance(integrity, dict)
or not isinstance(version, str)
or _VERSION.fullmatch(version) is None
or package != match.group("name")
):
raise ValueError("registry entry has no complete package/source identity")
repo = source.get("repository")
commit = source.get("commit")
tag_object = source.get("tag_object_sha")
if (
not isinstance(repo, str)
or _REPO.fullmatch(repo) is None
or repo != package
or source.get("tag") != f"v{version}"
or not isinstance(commit, str)
or _COMMIT.fullmatch(commit) is None
or not isinstance(tag_object, str)
or _COMMIT.fullmatch(tag_object) is None
):
raise ValueError("registry entry has invalid immutable tag provenance")
python = _artifact(
integrity.get("python"), ref=ref, package=package, version=version,
commit=commit,
)
if python["url"] != match.group("url") or python["sha256"] != match.group("digest"):
raise ValueError("registry Python ref differs from its artifact identity")
webui_package = entry.get("webui_package")
webui_ref = entry.get("webui_ref")
if bool(webui_package) != bool(webui_ref):
raise ValueError("registry WebUI package and ref must be declared together")
if webui_package:
if not isinstance(webui_package, str) or re.fullmatch(
r"@govoplan/[a-z0-9-]+-webui", webui_package
) is None or not isinstance(webui_ref, str):
raise ValueError("registry WebUI package identity is malformed")
if webui_package != f"@govoplan/{repo.removeprefix('govoplan-')}-webui":
raise ValueError("registry WebUI package belongs to another source repository")
webui = _artifact(
integrity.get("webui"), ref=webui_ref, package=webui_package,
version=version, commit=commit,
)
if webui_ref != webui["url"]:
raise ValueError("registry WebUI ref differs from its artifact identity")
sri = webui.get("integrity")
try:
valid_sri = isinstance(sri, str) and sri.startswith("sha512-") and len(
base64.b64decode(sri[7:], validate=True)
) == 64
except ValueError:
valid_sri = False
if not valid_sri:
raise ValueError("registry WebUI artifact needs a SHA-512 integrity identity")
elif "webui" in integrity:
raise ValueError("registry entry carries an unexpected WebUI artifact")
return RegistrySource(repo, version, commit, tag_object)
def _artifact(
value: object, *, ref: str, package: str, version: str, commit: str,
) -> dict[str, object]:
if not isinstance(value, dict):
raise ValueError("registry entry is missing artifact integrity")
url = value.get("url")
filename = value.get("filename")
digest = value.get("sha256")
size = value.get("size")
parsed = urlsplit(url) if isinstance(url, str) else None
url_filename = unquote(parsed.path.rsplit("/", 1)[-1]) if parsed else ""
decoded_parts = unquote(parsed.path).split("/") if parsed else []
expected_filenames = {url_filename}
expected_path = [package, version, url_filename]
if package.startswith("@govoplan/"):
# npm pack includes the scope in its local filename; the registry's
# immutable download URL uses the unscoped package basename.
expected_filenames = {
f"govoplan-{package.split('/', 1)[1]}-{version}.tgz",
} if url_filename == f"{package.split('/', 1)[1]}-{version}.tgz" else set()
expected_path = [*package.split("/"), "-", version, url_filename]
if (
parsed is None
or parsed.scheme != "https"
or not parsed.netloc
or parsed.username is not None
or parsed.password is not None
or parsed.fragment
or parsed.query
or any(part in {".", ".."} for part in unquote(parsed.path).split("/"))
or "%" in unquote(parsed.path)
or "\\" in unquote(parsed.path)
or any(ord(character) < 32 for character in url)
or decoded_parts[-len(expected_path):] != expected_path
or not isinstance(filename, str)
or re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,254}", filename) is None
or filename not in expected_filenames
or not isinstance(digest, str)
or _SHA256.fullmatch(digest) is None
or not isinstance(size, int)
or isinstance(size, bool)
or not 0 < size <= 512 * 1024 * 1024
or value.get("ref") != ref
or value.get("registry_identity") != f"{package}@{version}"
or value.get("git_ref") != f"v{version}"
or value.get("source_commit") != commit
):
raise ValueError("registry artifact ref, version, bytes, or source binding is inconsistent")
return value
@@ -19,10 +19,33 @@ from .git_state import (
scoped_git_command,
)
from .repository_tag import RemoteTagResult, ref_commit, remote_tag_commit
from .registry_reference import registry_entry_source
from .version_alignment import repository_version_issues
from .workspace import load_repository_specs, resolve_repo_path
def registered_source_origin_issues(
*, repo_versions: dict[str, str], workspace: Path, remote: str,
) -> tuple[SourceTagProvenanceIssue, ...]:
"""Bind registry candidate attestations to registered source endpoints."""
specs = {spec.name: spec for spec in load_repository_specs(include_website=False)}
issues = []
for repo, version in sorted(repo_versions.items()):
spec = specs.get(repo)
if spec is None:
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "source repository is not registered"))
continue
path = resolve_repo_path(spec, workspace)
if path.absolute() != path.resolve() or not path.resolve().is_relative_to(workspace.resolve()):
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "source checkout leaves the private workspace or traverses a symlink"))
continue
fetch = git_text(path, "remote", "get-url", "--all", remote).splitlines()
push = git_text(path, "remote", "get-url", "--push", "--all", remote).splitlines()
if fetch != [spec.remote] or push != [spec.remote]:
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "source remote does not exactly match the registered origin"))
return tuple(issues)
_CATALOG_PYTHON_REF = re.compile(
r"/(?P<repo>govoplan-[a-z0-9-]+)\.git@v(?P<version>[^\s;]+)$"
)
@@ -61,6 +84,8 @@ def catalog_source_selection(payload: object) -> CatalogSourceSelection:
)
versions: dict[str, str] = {}
registry_commits: dict[str, str] = {}
registry_tag_objects: dict[str, str] = {}
issues: list[SourceTagProvenanceIssue] = []
entries: list[tuple[str, object]] = [("core_release", payload.get("core_release"))]
modules = payload.get("modules")
@@ -70,6 +95,21 @@ def catalog_source_selection(payload: object) -> CatalogSourceSelection:
for source, raw_entry in entries:
if not isinstance(raw_entry, dict):
continue
try:
registry_source = registry_entry_source(raw_entry)
except ValueError as exc:
issues.append(SourceTagProvenanceIssue(source, "", str(exc)))
continue
if registry_source is not None:
repo, version = registry_source.repository, registry_source.version
previous = versions.setdefault(repo, version)
previous_commit = registry_commits.setdefault(repo, registry_source.commit)
previous_object = registry_tag_objects.setdefault(repo, registry_source.tag_object)
if (previous, previous_commit, previous_object) != (
version, registry_source.commit, registry_source.tag_object,
):
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "registry entries have conflicting immutable source identities"))
continue
python_ref = raw_entry.get("python_ref")
match = _CATALOG_PYTHON_REF.search(python_ref) if isinstance(python_ref, str) else None
if match is None:
@@ -89,8 +129,8 @@ def catalog_source_selection(payload: object) -> CatalogSourceSelection:
release = payload.get("release")
selected_units = release.get("selected_units") if isinstance(release, dict) else None
selected: dict[str, str] = {}
selected_commits: dict[str, str] = {}
selected_tag_objects: dict[str, str] = {}
selected_commits: dict[str, str] = dict(registry_commits)
selected_tag_objects: dict[str, str] = dict(registry_tag_objects)
if not isinstance(selected_units, list) or not selected_units:
issues.append(
SourceTagProvenanceIssue(
@@ -106,16 +146,22 @@ def catalog_source_selection(payload: object) -> CatalogSourceSelection:
repo = unit.get("repo")
version = unit.get("version")
if isinstance(repo, str) and isinstance(version, str):
if repo in selected:
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "duplicate selected repository"))
selected[repo] = version.removeprefix("v")
commit = unit.get("commit_sha")
tag_object = unit.get("tag_object_sha")
if isinstance(commit, str) and re.fullmatch(r"[0-9a-fA-F]{40}|[0-9a-fA-F]{64}", commit):
if repo in registry_commits and registry_commits[repo] != commit.lower():
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "selected commit differs from registry artifact source"))
selected_commits[repo] = commit.lower()
else:
issues.append(
SourceTagProvenanceIssue(repo, f"v{version.removeprefix('v')}", "selected unit has no valid commit_sha provenance")
)
if isinstance(tag_object, str) and re.fullmatch(r"[0-9a-fA-F]{40}|[0-9a-fA-F]{64}", tag_object):
if repo in registry_tag_objects and registry_tag_objects[repo] != tag_object.lower():
issues.append(SourceTagProvenanceIssue(repo, f"v{version}", "selected tag object differs from registry artifact source"))
selected_tag_objects[repo] = tag_object.lower()
else:
issues.append(
@@ -10,6 +10,7 @@ import subprocess
import tomllib
from .git_state import collect_versions, sanitized_git_environment
from .registry_reference import registry_artifact_conflicts, registry_entry_source
from .workspace import load_repository_specs, resolve_repo_path
@@ -407,6 +408,11 @@ def candidate_catalog_version_issues(payload: object) -> tuple[VersionAlignmentI
)
release = payload.get("release")
registry_entries = [core_release, *(modules if isinstance(modules, list) else [])]
issues.extend(
_catalog_shape_issue("artifact_integrity", issue)
for issue in registry_artifact_conflicts(registry_entries)
)
if isinstance(release, dict):
issues.extend(_catalog_release_issues(release, represented=represented))
return tuple(issues)
@@ -419,6 +425,15 @@ def _catalog_entry_issues(
represented: dict[str, str],
) -> list[VersionAlignmentIssue]:
issues: list[VersionAlignmentIssue] = []
try:
registry_source = registry_entry_source(entry)
except ValueError as exc:
return [_catalog_shape_issue(source, str(exc))]
if registry_source is not None:
previous = represented.setdefault(registry_source.repository, registry_source.version)
if previous != registry_source.version:
issues.append(_catalog_shape_issue(source, "repository appears with conflicting catalog versions"))
return issues
version = entry.get("version")
normalized_version = version.removeprefix("v") if isinstance(version, str) and version else None
if normalized_version is None:
+31
View File
@@ -21,6 +21,22 @@ def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
subparsers = parser.add_subparsers(dest="command", required=True)
full = subparsers.add_parser("full-registry", help="Build a strict full-profile candidate from verified registry artifacts.")
full.add_argument("--workspace-root", type=Path, default=DEFAULT_WORKSPACE_ROOT)
full.add_argument("--package-set", type=Path, required=True)
full.add_argument("--package-lock", type=Path, required=True)
full.add_argument("--wheelhouse", type=Path, required=True)
full.add_argument("--webui-packages", type=Path, required=True)
full.add_argument("--output-dir", type=Path, required=True)
full.add_argument("--selected-repository", action="append", required=True)
full.add_argument("--catalog-signing-key", action="append", required=True)
full.add_argument("--channel", default="stable")
full.add_argument("--source-remote", default="origin")
full.add_argument("--public-base-url", default="https://govoplan.add-ideas.de")
full.add_argument("--expires-days", type=int, default=90)
full.add_argument("--sequence", type=int)
full.add_argument("--json", action="store_true")
selective = subparsers.add_parser(
"selective", help="Build a signed selective channel catalog candidate."
)
@@ -143,6 +159,21 @@ def main() -> int:
)
args = parser.parse_args()
if args.command == "full-registry":
require_release_runtime_trust()
from govoplan_release.full_catalog import build_full_registry_candidate
result = build_full_registry_candidate(
package_set_path=args.package_set, package_lock_path=args.package_lock,
wheelhouse=args.wheelhouse, webui_packages=args.webui_packages,
output_dir=args.output_dir, selected_repositories=tuple(args.selected_repository),
signing_keys=tuple(args.catalog_signing_key), workspace_root=args.workspace_root,
channel=args.channel, source_remote=args.source_remote,
public_base_url=args.public_base_url, expires_days=args.expires_days,
sequence=args.sequence,
)
print(json.dumps(result, indent=2, sort_keys=True))
return 0
if args.command == "selective":
require_release_runtime_trust()
result = build_selective_catalog_candidate(