Compare commits

..
42 Commits
Author SHA1 Message Date
zemion ba82a85547 Repair Gitea Actions bootstrap and module matrix
Dependency Audit / dependency-audit (push) Successful in 2m28s
Deployment Installer / deployment-installer (push) Successful in 7s
Security Audit / security-audit (push) In progress
2026-07-31 05:57:33 +02:00
zemion f1fd143ef5 Complete guided release console workflow
Dependency Audit / dependency-audit (push) Successful in 2m35s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Failing after 15s
2026-07-31 05:46:50 +02:00
zemion b4248a849e chore: register tenancy WebUI in release catalog
Dependency Audit / dependency-audit (push) Successful in 2m27s
Deployment Installer / deployment-installer (push) Successful in 6s
Security Audit / security-audit (push) Failing after 14s
2026-07-31 04:21:34 +02:00
zemion ff47659899 Provision Python for security audit workflow
Dependency Audit / dependency-audit (push) Failing after 7s
Security Audit / security-audit (push) Failing after 6s
Deployment Installer / deployment-installer (push) Failing after 4s
2026-07-31 02:53:37 +02:00
zemion 908090dd0f Add scalable deployment planning and guided releases
Security Audit / security-audit (push) Failing after 4s
Dependency Audit / dependency-audit (push) Failing after 7s
Deployment Installer / deployment-installer (push) Failing after 4s
2026-07-31 02:49:03 +02:00
zemion 3864ce28b1 Delete .gitea/workflows/runner_test.yml
Dependency Audit / dependency-audit (push) Failing after 6s
Deployment Installer / deployment-installer (push) Failing after 4s
Security Audit / security-audit (push) Failing after 4s
2026-07-31 02:24:30 +02:00
zemion 857dbe55f7 Add .gitea/workflows/runner_test.yml
Dependency Audit / dependency-audit (push) Failing after 6s
Deployment Installer / deployment-installer (push) Failing after 4s
Security Audit / security-audit (push) Failing after 5s
2026-07-31 02:15:18 +02:00
zemion c9fcdc90c1 feat: add platform interface inventory and encryption registry
Dependency Audit / dependency-audit (push) Failing after 24s
Deployment Installer / deployment-installer (push) Failing after 4s
Security Audit / security-audit (push) Failing after 5s
2026-07-30 17:42:11 +02:00
zemion 82e836b720 feat(deploy): add declarative installation workflow
Deployment Installer / deployment-installer (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
Dependency Audit / dependency-audit (push) Has been cancelled
2026-07-30 15:36:37 +02:00
zemion fcb8296812 test: dispatch queued dataflow composition runs
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-30 02:33:16 +02:00
zemion f2e2eb5517 ci: enforce complete security audit coverage
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-30 01:40:36 +02:00
zemion 1aea3e7c4f fix: repair dev modules and scoped release git trust
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 20:54:58 +02:00
zemion 3f9567af18 Gate Search on PostgreSQL in module CI
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 20:37:08 +02:00
zemion de16f11ce8 test(datasources): include immutable payload schema
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 20:13:49 +02:00
zemion 9d6cdff4b8 CSS changes for release console
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 19:35:49 +02:00
zemion aa4050c0ca feat: expose focused backend reload mode
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 18:52:55 +02:00
zemion d3cdbd8c7a feat: include risk compliance in dev composition
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 18:46:53 +02:00
zemion 7115c4711d chore: register search wiki projects and tickets
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 15:50:35 +02:00
zemion a3beca6fc5 chore: harden audit checks and issue taxonomy
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-29 14:16:27 +02:00
zemion 11d45bce25 Register governed Views module
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 21:04:55 +02:00
zemion 7b6135b89b Document governed Views architecture
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 19:05:37 +02:00
zemion 5b79e7d377 test: verify pinned dataflow publication composition
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 13:48:11 +02:00
zemion 6afb8fea76 fix: exclude generated artifacts from wiki sync
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 12:52:46 +02:00
zemion 163b35c0af feat: integrate datasource and workflow modules
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 12:45:05 +02:00
zemion 603e07cec5 Integrate Connectors and Dataflow validation
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 11:14:38 +02:00
zemion 97dfd333c6 Register Dataflow module and work taxonomy
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-28 01:34:06 +02:00
zemion ba88c574b9 chore: move repositories to GovOPlaN organization
Dependency Audit / dependency-audit (push) Has been cancelled
Security Audit / security-audit (push) Has been cancelled
2026-07-27 15:46:51 +02:00
zemion 8d292184d4 ci: execute tagged module pytest suites
Security Audit / security-audit (push) Successful in 4m3s
Security Audit Toolbox Update / toolbox-update (push) Successful in 4s
Dependency Audit / dependency-audit (push) Successful in 1m51s
2026-07-23 01:18:47 +02:00
zemion 52bd3527cd ci: install release test harness dependencies
Dependency Audit / dependency-audit (push) Successful in 1m37s
Security Audit / security-audit (push) Has been cancelled
2026-07-23 01:15:07 +02:00
zemion ff49dabf8f ci: retain installed artifact checks in matrix
Dependency Audit / dependency-audit (push) Successful in 1m42s
Security Audit / security-audit (push) Successful in 4m7s
2026-07-23 00:57:24 +02:00
zemion cd15aa514e ci: run artifact-aware module matrix tests
Dependency Audit / dependency-audit (push) Successful in 1m36s
Security Audit / security-audit (push) Has been cancelled
2026-07-23 00:53:16 +02:00
zemion a042baa1d3 fix(release): verify tags in mounted workspaces 2026-07-23 00:53:16 +02:00
zemion e80de00f29 chore(release): align Mail with Campaign contract
Dependency Audit / dependency-audit (push) Successful in 1m37s
Security Audit / security-audit (push) Has been cancelled
2026-07-23 00:47:46 +02:00
zemion c69acf0dee ci: use Gitea-compatible artifact upload
Dependency Audit / dependency-audit (push) Successful in 1m39s
Security Audit / security-audit (push) Successful in 4m11s
2026-07-23 00:36:59 +02:00
zemion 8b93bbc6b6 ci: mount audit workspace safely
Dependency Audit / dependency-audit (push) Successful in 1m43s
Security Audit / security-audit (push) Failing after 4m10s
2026-07-23 00:30:28 +02:00
zemion 3fc17701df ci: fix action repository access
Dependency Audit / dependency-audit (push) Successful in 1m45s
Security Audit / security-audit (push) Failing after 3m33s
2026-07-23 00:10:29 +02:00
zemion 9788bdde0c ci: harden repository bootstrap transport 2026-07-23 00:09:43 +02:00
zemion a10a01c903 docs: surface system action status
Dependency Audit / dependency-audit (push) Failing after 14s
Security Audit / security-audit (push) Failing after 13s
2026-07-22 23:45:42 +02:00
zemion e005e54333 Seal catalog publication Git metadata
Dependency Audit / dependency-audit (push) Failing after 15s
Security Audit / security-audit (push) Failing after 13s
2026-07-22 22:31:58 +02:00
zemion 76e4baa76e Define the catalog publication identity
Dependency Audit / dependency-audit (push) Failing after 15s
Security Audit / security-audit (push) Failing after 13s
2026-07-22 22:23:01 +02:00
zemion b6e9a9bd81 Pin the website publication toolchain
Dependency Audit / dependency-audit (push) Failing after 15s
Security Audit / security-audit (push) Failing after 13s
2026-07-22 22:20:09 +02:00
zemion dc46bda224 Validate complete Core release bundle sources
Dependency Audit / dependency-audit (push) Failing after 14s
Security Audit / security-audit (push) Failing after 13s
2026-07-22 22:10:35 +02:00
100 changed files with 12102 additions and 617 deletions
+13 -2
View File
@@ -2,17 +2,24 @@
# Copy to a deployment-local .env or secret store. Do not commit populated secrets.
APP_ENV=production
# Live graph changes are useful in development. Production should apply saved
# module state through a coordinated restart of all API and worker processes.
GOVOPLAN_MODULE_LIVE_APPLY_ENABLED=
GOVOPLAN_INSTALL_PROFILE=self-hosted
MASTER_KEY_B64=<generate-with-govoplan-config-env-template-generate-secrets>
DATABASE_URL=postgresql+psycopg://govoplan:change-me@127.0.0.1:5432/govoplan
GOVOPLAN_DATABASE_URL_PGTOOLS=postgresql://govoplan:change-me@127.0.0.1:5432/govoplan
GOVOPLAN_DB_POOL_SIZE=5
GOVOPLAN_DB_MAX_OVERFLOW=10
GOVOPLAN_DB_POOL_TIMEOUT_SECONDS=30
GOVOPLAN_DB_POOL_RECYCLE_SECONDS=1800
ENABLED_MODULES=tenancy,organizations,identity,access,admin,dashboard,policy,audit,files,mail,campaigns,calendar,docs,ops
ENABLED_MODULES=tenancy,organizations,identity,idm,access,admin,dashboard,policy,audit,files,mail,campaigns,calendar,poll,scheduling,connectors,datasources,dataflow,workflow,views,search,risk_compliance,postbox,notifications,docs,ops
CELERY_ENABLED=true
REDIS_URL=redis://127.0.0.1:6379/0
CELERY_QUEUES=send_email,append_sent,notifications,calendar,default
CELERY_QUEUES=send_email,append_sent,notifications,calendar,dataflow,events,default
CALENDAR_OUTBOX_TERMINAL_RETENTION_DAYS=90
GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS=false
@@ -24,10 +31,14 @@ GOVOPLAN_HTTP_MAX_REQUEST_BODY_BYTES=536870912
GOVOPLAN_HTTP_HSTS_SECONDS=31536000
AUTH_LOGIN_THROTTLE_ENABLED=true
AUTH_ACTIVITY_TOUCH_INTERVAL_SECONDS=300
AUTH_LOGIN_THROTTLE_IDENTITY_LIMIT=10
AUTH_LOGIN_THROTTLE_CLIENT_LIMIT=100
AUTH_LOGIN_THROTTLE_WINDOW_SECONDS=900
AUTH_LOGIN_THROTTLE_REDIS_RETRY_SECONDS=30
# Production startup fails without Redis unless this explicit single-process
# risk acknowledgement is enabled.
GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE=false
CORS_ORIGINS=https://govoplan.example.org
GOVOPLAN_TRUSTED_HOSTS=govoplan.example.org
+4 -14
View File
@@ -21,23 +21,13 @@ jobs:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: "22"
- name: Configure SSH for release dependencies
env:
GOVOPLAN_RELEASE_SSH_KEY_B64: ${{ secrets.GOVOPLAN_RELEASE_SSH_KEY_B64 }}
- name: Use anonymous HTTPS for public GovOPlaN repositories
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
if [ -z "${GOVOPLAN_RELEASE_SSH_KEY_B64:-}" ]; then
echo "GOVOPLAN_RELEASE_SSH_KEY_B64 secret is required for git+ssh release dependencies."
exit 1
fi
printf '%s' "$GOVOPLAN_RELEASE_SSH_KEY_B64" | base64 -d > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo 'git.add-ideas.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDe48IOof2fJS1dTbJtLWQnWnr+JorZXKIFdOAM9ct8G' > ~/.ssh/known_hosts
chmod 600 ~/.ssh/known_hosts
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "git@git.add-ideas.de:GovOPlaN/govoplan"
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "ssh://git@git.add-ideas.de/GovOPlaN/govoplan"
- name: Bootstrap GovOPlaN repositories
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent ..
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --exclude-repo addideas-govoplan-website
- name: Install backend dev audit dependencies
working-directory: govoplan
run: |
+30
View File
@@ -0,0 +1,30 @@
name: Deployment Installer
on:
pull_request:
push:
branches:
- main
workflow_dispatch:
jobs:
deployment-installer:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
path: govoplan
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- name: Compile deployment tooling
working-directory: govoplan
run: python -m py_compile tools/deployment/govoplan-deploy.py tools/deployment/govoplan_deploy/*.py
- name: Test declarative deployment bundle
working-directory: govoplan
run: python -m unittest -v tests.test_deployment_installer
- name: Build single-file deployer artifact
working-directory: govoplan
run: |
python tools/deployment/build-deployer-zipapp.py --output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help
+32 -14
View File
@@ -7,6 +7,18 @@ on:
jobs:
module-matrix:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_DB: govoplan_test
POSTGRES_USER: govoplan
POSTGRES_PASSWORD: govoplan_test
options: >-
--health-cmd "pg_isready -U govoplan -d govoplan_test"
--health-interval 5s
--health-timeout 5s
--health-retries 20
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
@@ -17,32 +29,38 @@ jobs:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: "22"
- name: Configure SSH for release dependencies
env:
GOVOPLAN_RELEASE_SSH_KEY_B64: ${{ secrets.GOVOPLAN_RELEASE_SSH_KEY_B64 }}
- name: Use anonymous HTTPS for public GovOPlaN repositories
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
if [ -z "${GOVOPLAN_RELEASE_SSH_KEY_B64:-}" ]; then
echo "GOVOPLAN_RELEASE_SSH_KEY_B64 secret is required for git+ssh release dependencies."
exit 1
fi
printf '%s' "$GOVOPLAN_RELEASE_SSH_KEY_B64" | base64 -d > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo 'git.add-ideas.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDe48IOof2fJS1dTbJtLWQnWnr+JorZXKIFdOAM9ct8G' > ~/.ssh/known_hosts
chmod 600 ~/.ssh/known_hosts
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "git@git.add-ideas.de:GovOPlaN/govoplan"
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "ssh://git@git.add-ideas.de/GovOPlaN/govoplan"
- name: Bootstrap GovOPlaN repositories
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent ..
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --exclude-repo addideas-govoplan-website
- name: Install backend release dependencies
working-directory: govoplan
run: |
python -m venv .venv
.venv/bin/python tools/repo/sync-python-environment.py --requirements requirements-release.txt --python .venv/bin/python --upgrade-pip
.venv/bin/python -m pip install '../govoplan-core[dev]'
.venv/bin/python -m pip install --no-deps ../govoplan-search
- name: Install WebUI release dependencies with test scripts
working-directory: govoplan
run: bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui
- name: Validate Search against PostgreSQL
working-directory: govoplan
env:
GOVOPLAN_SEARCH_POSTGRES_URL: postgresql+psycopg://govoplan:govoplan_test@postgres:5432/govoplan_test
run: |
GOVOPLAN_CORE_ROOT="$PWD/../govoplan-core" .venv/bin/python tools/checks/postgres-integration-check.py \
--database-url "$GOVOPLAN_SEARCH_POSTGRES_URL" \
--module-set search=tenancy,access,search \
--reset-schema \
--skip-retirement-atomicity
PYTHONPATH="$PWD/../govoplan-search/src:$PWD/../govoplan-core/src" \
.venv/bin/python -m unittest discover \
-s ../govoplan-search/tests \
-p test_postgres_search.py \
-v
- name: Run module matrix and contract tests
working-directory: govoplan
run: GOVOPLAN_CORE_ROOT="$PWD/../govoplan-core" PYTHON="$PWD/.venv/bin/python" bash tools/checks/check-module-matrix.sh
+5 -15
View File
@@ -16,29 +16,19 @@ jobs:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020
with:
node-version: "22"
- name: Configure SSH for release dependencies
env:
GOVOPLAN_RELEASE_SSH_KEY_B64: ${{ secrets.GOVOPLAN_RELEASE_SSH_KEY_B64 }}
- name: Use anonymous HTTPS for public GovOPlaN repositories
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
if [ -z "${GOVOPLAN_RELEASE_SSH_KEY_B64:-}" ]; then
echo "GOVOPLAN_RELEASE_SSH_KEY_B64 secret is required for git+ssh release dependencies."
exit 1
fi
printf '%s' "$GOVOPLAN_RELEASE_SSH_KEY_B64" | base64 -d > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo 'git.add-ideas.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDe48IOof2fJS1dTbJtLWQnWnr+JorZXKIFdOAM9ct8G' > ~/.ssh/known_hosts
chmod 600 ~/.ssh/known_hosts
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "git@git.add-ideas.de:GovOPlaN/govoplan"
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "ssh://git@git.add-ideas.de/GovOPlaN/govoplan"
- name: Bootstrap GovOPlaN repositories
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent ..
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --exclude-repo addideas-govoplan-website
- name: Install backend release integration dependencies
working-directory: govoplan
run: |
python -m venv .venv
.venv/bin/python tools/repo/sync-python-environment.py --requirements requirements-release.txt --python .venv/bin/python --upgrade-pip
.venv/bin/python -m pip install '../govoplan-core[dev]'
.venv/bin/python -m pip install -r requirements-release-tests.txt '../govoplan-core[dev]'
- name: Install WebUI release dependencies
working-directory: govoplan
run: bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui
+10 -18
View File
@@ -1,7 +1,6 @@
name: Security Audit
on:
pull_request:
push:
branches:
- main
@@ -13,7 +12,7 @@ jobs:
security-audit:
runs-on: ubuntu-latest
env:
SECURITY_AUDIT_MODE: ci
SECURITY_AUDIT_MODE: full
SECURITY_AUDIT_SCOPE: govoplan
SECURITY_AUDIT_FAIL_ON_FINDINGS: "0"
SECURITY_AUDIT_REQUIRE_TOOLS: "1"
@@ -21,29 +20,22 @@ jobs:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5
with:
path: govoplan
- name: Configure SSH for repository bootstrap
env:
GOVOPLAN_RELEASE_SSH_KEY_B64: ${{ secrets.GOVOPLAN_RELEASE_SSH_KEY_B64 }}
- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065
with:
python-version: "3.12"
- name: Use anonymous HTTPS for public GovOPlaN repositories
run: |
mkdir -p ~/.ssh
chmod 700 ~/.ssh
if [ -z "${GOVOPLAN_RELEASE_SSH_KEY_B64:-}" ]; then
echo "GOVOPLAN_RELEASE_SSH_KEY_B64 secret is required for git+ssh repository bootstrap."
exit 1
fi
printf '%s' "$GOVOPLAN_RELEASE_SSH_KEY_B64" | base64 -d > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
echo 'git.add-ideas.de ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDe48IOof2fJS1dTbJtLWQnWnr+JorZXKIFdOAM9ct8G' > ~/.ssh/known_hosts
chmod 600 ~/.ssh/known_hosts
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "git@git.add-ideas.de:GovOPlaN/govoplan"
git config --global --add url."https://git.add-ideas.de/GovOPlaN/govoplan".insteadOf "ssh://git@git.add-ideas.de/GovOPlaN/govoplan"
- name: Bootstrap GovOPlaN repositories
working-directory: govoplan
run: python tools/repo/bootstrap-repositories.py --parent ..
- name: Run security audit
run: python tools/repo/bootstrap-repositories.py --parent .. --transport public-https --exclude-repo addideas-govoplan-website
- name: Run whole-system security audit
working-directory: govoplan
run: tools/checks/security-audit/run.sh --mode "$SECURITY_AUDIT_MODE" --scope "$SECURITY_AUDIT_SCOPE" --reports-dir audit-reports
- name: Upload audit reports
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32
with:
name: security-audit-reports
path: govoplan/audit-reports
+38
View File
@@ -4,6 +4,12 @@
**Repository type:** system (meta).
<!-- govoplan-repository-type:end -->
[![Module Matrix](https://git.add-ideas.de/GovOPlaN/govoplan/actions/workflows/module-matrix.yml/badge.svg?branch=main)](https://git.add-ideas.de/GovOPlaN/govoplan/actions?workflow=module-matrix.yml&actor=0&status=0)
[![Release Integration](https://git.add-ideas.de/GovOPlaN/govoplan/actions/workflows/release-integration.yml/badge.svg?branch=main)](https://git.add-ideas.de/GovOPlaN/govoplan/actions?workflow=release-integration.yml&actor=0&status=0)
[![Deployment Installer](https://git.add-ideas.de/GovOPlaN/govoplan/actions/workflows/deployment-installer.yml/badge.svg?branch=main)](https://git.add-ideas.de/GovOPlaN/govoplan/actions?workflow=deployment-installer.yml&actor=0&status=0)
[![Dependency Audit](https://git.add-ideas.de/GovOPlaN/govoplan/actions/workflows/dependency-audit.yml/badge.svg?branch=main)](https://git.add-ideas.de/GovOPlaN/govoplan/actions?workflow=dependency-audit.yml&actor=0&status=0)
[![Security Audit](https://git.add-ideas.de/GovOPlaN/govoplan/actions/workflows/security-audit.yml/badge.svg?branch=main)](https://git.add-ideas.de/GovOPlaN/govoplan/actions?workflow=security-audit.yml&actor=0&status=0)
This is the GovOPlaN meta repository. It is the operator entry point for
whole-product development, release orchestration, repository bootstrap, and
system-level Docker composition.
@@ -36,6 +42,16 @@ Open the WebUI in a browser after launch only when explicitly requested:
GOVOPLAN_OPEN_BROWSER=1 ./tools/launch/launch-dev.sh
```
Limit backend reload triggers during focused module work without changing the
enabled module graph:
```sh
GOVOPLAN_BACKEND_RELOAD_MODULES=calendar,campaign ./tools/launch/launch-dev.sh
```
Set `GOVOPLAN_BACKEND_RELOAD_MODULES=none` to watch only core/config sources.
Leaving it unset keeps the broad default and watches all enabled modules.
Start the shared development PostgreSQL service:
```sh
@@ -126,6 +142,20 @@ Start the local release console:
./.venv/bin/python tools/release/release-console.py
```
Create and validate a private, declarative installation bundle:
```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py init \
--directory ~/.local/share/govoplan/installations/default
./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
--directory ~/.local/share/govoplan/installations/default
```
The current executable slice and remaining production gates are documented in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
Same-host replica balancing and the multi-host promotion boundary are documented
in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
## Configuration
The repository root `.env.example` is the self-hosted operator template for a
@@ -146,6 +176,9 @@ Meta ownership and module install/contract boundaries are documented in
`docs/META_REPO_SCAN.md` and `docs/MODULE_CONTRACTS_AND_INSTALLS.md`.
Frontend layout principles for module pages are documented in
`docs/FRONTEND_LAYOUT_PRINCIPLES.md`.
The provider-neutral datasource boundary and reusable Dataflow/Workflow graph
contract are documented in
`docs/DATASOURCE_AND_DEFINITION_GRAPH_ARCHITECTURE.md`.
The cross-product destination, stakeholder visions, configuration archetypes,
connected outcome stories, and capability horizons are documented in
the [Connected Governance Platform Roadmap](docs/CONNECTED_GOVERNANCE_PLATFORM_ROADMAP.md).
@@ -155,6 +188,11 @@ including stage gates and shared documentation expectations, is in the
The administrator journey from Core-only bootstrap through online module
installation, scale-out, and reversible environment promotion is defined in
[System Administrator Lifecycle User Story](docs/SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md).
The corresponding host deployment compiler, managed/external component choices,
reconfiguration semantics, and safe Web update boundary are defined in
[Installation and Deployment Architecture](docs/INSTALLATION_AND_DEPLOYMENT_ARCHITECTURE.md).
The concrete replica, worker-node, load-balancer, and shared-state topology is
defined in [Scaling and Multi-Host Deployment](docs/SCALING_AND_MULTI_HOST_DEPLOYMENT.md).
The first Campaign-centric capability and infrastructure fit assessment is in
`docs/CAPABILITY_AND_INFRASTRUCTURE_FIT.md`. Its rerun tooling can collect and
verify a bounded installed composition; target, provider and production claims
+5
View File
@@ -7,6 +7,11 @@ Current shared profiles:
- `govoplan/dev/postgres`
- `govoplan/dev/production-like`
The generated whole-product Compose profile is owned by
`tools/deployment/govoplan-deploy.py`. It renders a deployment-specific
`compose.json` from a versioned installation specification; generated files and
secrets remain outside the repository.
Module-specific Docker test beds remain in their owning repositories:
- `govoplan-campaign/dev/mail-testbed`
+10 -10
View File
@@ -57,7 +57,7 @@ profile intentionally runs only PostgreSQL and Redis in containers; API,
WebUI, worker, and scheduler processes still run from editable source trees. A
target deployment must supply TLS termination, process supervision, secret
injection, monitoring, backup storage, and recovery procedures. The open
[Core backup/restore issue #29](https://git.add-ideas.de/add-ideas/govoplan-core/issues/29)
[Core backup/restore issue #29](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/29)
is a production gate. Identity federation, external audit export, and a tested
disaster-recovery plan are also not complete.
@@ -134,7 +134,7 @@ persistence, and durable shared or object storage. Run one scheduler only when a
selected module needs scheduled recovery. Multiple scheduler replicas require
leader election or an external lock, which is not established by this report.
This topology is a recommendation from the [Ops scalability profiles](https://git.add-ideas.de/add-ideas/govoplan-ops/src/branch/main/docs/SCALABILITY_PROFILES.md),
This topology is a recommendation from the [Ops scalability profiles](https://git.add-ideas.de/GovOPlaN/govoplan-ops/src/branch/main/docs/SCALABILITY_PROFILES.md),
not a currently shipped production Compose/Kubernetes/systemd package.
## Functional capability matrix
@@ -154,7 +154,7 @@ not a currently shipped production Compose/Kubernetes/systemd package.
| Configured-system documentation and Ops status pages | `verified` | Docs/Ops manifests contribute protected routes and WebUI packages; Ops code checks database, Redis, workers, storage and deployment-security settings. | This does not replace external monitoring or a target runbook. |
| Calendar/CalDAV integration | `partial` | Calendar `v0.1.8` supplies the catalogued storage/sync foundation. The durable external-write outbox, worker recovery, reconciliation, and retention work is committed, tested, and pushed on Calendar `main` after that tag. | The post-tag outbox work is remote-integrated source, not local-only WIP, but it is not in the signed stable package baseline and has not passed a target CalDAV drill. Bulk synchronized-calendar migration semantics remain separate work. |
| External LDAP/AD, OIDC/SAML or SCIM identity integration | `scaffold` | IDM owns normalized assignment APIs and documents connector boundaries. | Provider connectors, login callback flow and target directory reconciliation are not an implemented end-to-end capability. Use local accounts for this pilot. |
| Export-control/embargo-list screening | `planned` | Product-level [GovOPlaN #12](https://git.add-ideas.de/add-ideas/govoplan/issues/12) defines the consumer-independent user story. | No screening provider, list provenance, matching policy, review flow or legal evidence exists in this composition. |
| Export-control/embargo-list screening | `planned` | Product-level [GovOPlaN #12](https://git.add-ideas.de/GovOPlaN/govoplan/issues/12) defines the consumer-independent user story. | No screening provider, list provenance, matching policy, review flow or legal evidence exists in this composition. |
| Workflow-driven journeys and views | `planned` | Workflow contracts/concepts exist outside this assessment. | Explicitly postponed. Do not include Workflow in pilot or production claims from this report. |
## Infrastructure matrix
@@ -177,13 +177,13 @@ not a currently shipped production Compose/Kubernetes/systemd package.
| Health/readiness | `verified` | `/health`, protected `/health/details`, and Ops checks for DB, Redis, workers, storage, maintenance and cookie/CORS posture. | Add external probes and distinguish liveness from dependency readiness for the chosen orchestrator. |
| Metrics, logs and alerting | `partial` | Correlation IDs, slow-request/query metrics in logs, worker inspection and operator status are implemented. | No bundled metrics exporter, log collector, dashboards, queue-depth alerts, pager route or SLO is verified. |
| Audit | `partial` | Local audit tables and retry outbox are verified. | Retention, tamper-evident export, privileged access review and SIEM integration are unproved. |
| Backup and restore | `partial` | Operator guide and installer hooks describe `pg_dump`/`pg_restore`; SQLite and simulated installer rollback drills exist. | [Core #29](https://git.add-ideas.de/add-ideas/govoplan-core/issues/29) remains open. No target PostgreSQL + files + secrets restore drill or measured RTO/RPO exists. |
| Backup and restore | `partial` | Operator guide and installer hooks describe `pg_dump`/`pg_restore`; SQLite and simulated installer rollback drills exist. | [Core #29](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/29) remains open. No target PostgreSQL + files + secrets restore drill or measured RTO/RPO exists. |
| Disaster recovery | `not_assessed` | The Ops guide asks for RPO/RTO and restore drills. | No agreed RPO/RTO, off-site copy, failover topology, dependency recovery order, communications plan or exercise evidence was supplied. |
The scalability and sizing documentation delivered the documentation portions
of [Core #217](https://git.add-ideas.de/add-ideas/govoplan-core/issues/217) and
[Core #219](https://git.add-ideas.de/add-ideas/govoplan-core/issues/219).
[Core #28](https://git.add-ideas.de/add-ideas/govoplan-core/issues/28) records the
of [Core #217](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/217) and
[Core #219](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/219).
[Core #28](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/28) records the
operator-documentation slice. These closed tickets are evidence of documented
models, not evidence that an organization's production environment has passed
them.
@@ -584,13 +584,13 @@ source tests or signed release metadata.
- [Production-like profile](../dev/production-like/README.md) and
[Compose dependencies](../dev/production-like/docker-compose.yml)
- [Module contracts and install boundaries](MODULE_CONTRACTS_AND_INSTALLS.md)
- [Core deployment operator guide](https://git.add-ideas.de/add-ideas/govoplan-core/src/branch/main/docs/DEPLOYMENT_OPERATOR_GUIDE.md)
- [Ops scalability profiles](https://git.add-ideas.de/add-ideas/govoplan-ops/src/branch/main/docs/SCALABILITY_PROFILES.md)
- [Core deployment operator guide](https://git.add-ideas.de/GovOPlaN/govoplan-core/src/branch/main/docs/DEPLOYMENT_OPERATOR_GUIDE.md)
- [Ops scalability profiles](https://git.add-ideas.de/GovOPlaN/govoplan-ops/src/branch/main/docs/SCALABILITY_PROFILES.md)
- Actual module manifests in the pinned repositories and the static contract
checker in this meta repository
- Core module-system/API smoke/auth/install-config tests, plus focused Campaign,
Files, Mail, Audit and Addresses tests
- [Campaign delivery runbook](https://git.add-ideas.de/add-ideas/govoplan-campaign/src/branch/main/docs/CAMPAIGN_DELIVERY_RUNBOOK.md)
- [Campaign delivery runbook](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/src/branch/main/docs/CAMPAIGN_DELIVERY_RUNBOOK.md)
- [Live signed stable catalog](https://govoplan.add-ideas.de/catalogs/v1/channels/stable.json)
and [published keyring](https://govoplan.add-ideas.de/catalogs/v1/keyring.json),
verified against a separately provisioned local trust keyring
+20 -19
View File
@@ -10,7 +10,7 @@ baseline to that destination.
It is a durable direction, not a release promise or a substitute for issue
tracking. Live work state belongs in Gitea issues. The
[Core master roadmap](https://git.add-ideas.de/add-ideas/govoplan-core/src/branch/main/docs/GOVOPLAN_MASTER_ROADMAP.md)
[Core master roadmap](https://git.add-ideas.de/GovOPlaN/govoplan-core/src/branch/main/docs/GOVOPLAN_MASTER_ROADMAP.md)
remains the technical module and wave sequence; this document supplies the
cross-product vision that sequence serves.
@@ -448,7 +448,7 @@ without creating disconnected ticket systems.
- As a manager, I can distinguish demand, backlog, SLA risk, recurring cause,
and verified resolution.
**Composition.** Issue reporting, helpdesk, cases, tasks, facilities, assets,
**Composition.** Tickets, helpdesk profiles, cases, tasks, facilities, assets,
inspections, booking, resources, calendar, files, notifications, connectors,
reporting, policy, and audit.
@@ -530,7 +530,7 @@ access, no hidden cross-module coupling, correction rather than fictional undo,
and a durable action/effect trail.
**Roadmap role.** The canonical product story is
[meta issue #12](https://git.add-ideas.de/add-ideas/govoplan/issues/12). It is a
[meta issue #12](https://git.add-ideas.de/GovOPlaN/govoplan/issues/12). It is a
future cross-product configuration and contract program, not Campaign work and
not a claim that screening is implemented today.
@@ -1279,8 +1279,9 @@ them; they do not block the product vision today.
- Which exact HIS/CampusOnline interfaces, student-statistics fields and
official keys, accepted calculation, freeze/correction policy, privacy
profile, and drill-down level should define the first analytical data product?
- After the first source-to-report proof, do repeated source/dataflow contracts
justify separate `govoplan-datasources` and `govoplan-dataflow` modules?
- Which database, REST, directory, and managed-file providers should follow the
implemented separation of `govoplan-connectors`, `govoplan-datasources`, and
`govoplan-dataflow`, and which quality/promotion policy should prove it first?
- Which collaborative editor should be the first target, and should its first
accepted experience emphasize concurrent editing, controlled check-out, or
both?
@@ -1381,7 +1382,7 @@ integration. Conversely, 30 repositories had no open issue; for many
later-wave modules this meant no implementation program had been opened, not
that the capability was complete.
[Poll #2](https://git.add-ideas.de/add-ideas/govoplan-poll/issues/2) was a clear
[Poll #2](https://git.add-ideas.de/GovOPlaN/govoplan-poll/issues/2) was a clear
tracker-drift example: its configurable transition engine, agreed transition
matrix/history, idempotent keyed retries, re-decision audit, archive/unarchive,
and preservation behavior were implemented and pushed while the issue still
@@ -1389,27 +1390,27 @@ reported `needs-info`.
Issue anchors that informed the bridge from the baseline into this roadmap:
- [Meta #10](https://git.add-ideas.de/add-ideas/govoplan/issues/10) for the
- [Meta #10](https://git.add-ideas.de/GovOPlaN/govoplan/issues/10) for the
capability/infrastructure assessment and its target proof;
- [Meta #11](https://git.add-ideas.de/add-ideas/govoplan/issues/11) for the
- [Meta #11](https://git.add-ideas.de/GovOPlaN/govoplan/issues/11) for the
universal interface and focused-view direction;
- [Core #225](https://git.add-ideas.de/add-ideas/govoplan-core/issues/225) for
- [Core #225](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/225) for
guided, safe configuration;
- [Core #29](https://git.add-ideas.de/add-ideas/govoplan-core/issues/29) for the
- [Core #29](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/29) for the
backup/restore production gate;
- [Core #263](https://git.add-ideas.de/add-ideas/govoplan-core/issues/263) and
[Campaign #63](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/63),
[#62](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/62),
[#65](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/65), and
[#69](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/69) for the
- [Core #263](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/263) and
[Campaign #63](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/63),
[#62](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/62),
[#65](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/65), and
[#69](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/69) for the
reference interface/delivery vocabulary and behavior;
- [Poll #1](https://git.add-ideas.de/add-ideas/govoplan-poll/issues/1) for the
- [Poll #1](https://git.add-ideas.de/GovOPlaN/govoplan-poll/issues/1) for the
database-enforced respondent invariant exposed by Scheduling;
- [Connectors #6](https://git.add-ideas.de/add-ideas/govoplan-connectors/issues/6)
- [Connectors #6](https://git.add-ideas.de/GovOPlaN/govoplan-connectors/issues/6)
for the governed connector configuration/simulation foundation;
- [Meta #9](https://git.add-ideas.de/add-ideas/govoplan/issues/9) for the first
- [Meta #9](https://git.add-ideas.de/GovOPlaN/govoplan/issues/9) for the first
permit-to-payment reference process; and
- [Meta #12](https://git.add-ideas.de/add-ideas/govoplan/issues/12) for the
- [Meta #12](https://git.add-ideas.de/GovOPlaN/govoplan/issues/12) for the
deliberately deferred, consumer-independent export-control story.
Live Gitea issue state remains canonical. These dated facts explain the roadmap
@@ -0,0 +1,82 @@
# Datasource And Definition Graph Architecture
## Two-Layer Data Boundary
GovOPlaN separates governed data identity from external acquisition:
| Layer | Owner | Responsibilities |
| --- | --- | --- |
| Datasource layer | `govoplan-datasources` | Catalogue, tenant visibility, live/cached/static mode, staging, immutable materializations, frozen states, schema, fingerprints, provenance, and bounded reads |
| Connector layer | `govoplan-connectors` and protocol/provider modules | External protocols, endpoints, connection profiles, credentials, discovery, provider health, source-side filtering, and query pushdown |
Connectors publish versioned datasource origins. Datasources registers those
origins and presents one stable capability to Dataflow, Workflow, Reporting,
Risk Compliance, and other consumers. Consumers must not import connector
implementations or retain credentials.
The initial provider path is:
1. Connectors imports a bounded JSON/CSV snapshot and exposes it as an origin.
2. Datasources registers it as live or cached, or accepts a direct static upload
through staging.
3. Cached refreshes and static promotions append immutable materializations.
4. Any datasource may expose a frozen state for reproducible execution evidence.
5. Dataflow stores an opaque datasource reference, state policy, and expected
fingerprint.
6. A pinned Dataflow run may publish a complete bounded result as a new
immutable materialization through an idempotent Datasources capability.
Database, REST/HTTP, LDAP/directory, managed file, watched-directory, feed, and
stream providers fit behind the same origin contract. Provider-specific
configuration remains in Connectors.
## Shared Definition Graph
Core owns domain-neutral graph primitives:
- nodes, typed ports, edges, and configuration field descriptors;
- node libraries and category labels;
- graph size, connectivity, cycle, and node-count constraints;
- shared backend validation and frontend connection checks.
Domain modules own their semantics:
- Dataflow provides load, combine, filter, transform, and output nodes. Its
graph is acyclic and has one output.
- Workflow provides trigger, activity, review, decision, wait, module-action,
Dataflow, and outcome nodes. It permits governed loops and has exactly one
trigger plus one or more outcomes.
This division permits a shared editor shell without making Workflow a special
kind of Dataflow or leaking either module into Core.
## Current Implementation
- Core graph and datasource contracts are versioned at `0.1.0`.
- Workflow exposes a reusable graph editor, node-library discovery, validation,
tenant-isolated definitions, immutable revisions, and activation pinning.
- Datasources exposes catalogue, origins, staging, promotion, preview,
materialization history, refresh, freeze, retirement, and producer
publication APIs.
- Datasources WebUI exposes all current lifecycle views.
- Connectors adapts existing tabular snapshots to datasource origins.
- Dataflow consumes only Datasources catalogue/lifecycle capabilities and can
request current, live, or latest-frozen state.
- Dataflow exposes a pinned run-lifecycle capability and a Run/Publish surface.
Its first synchronous runner records lineage and terminal state, publishes
only complete bounded results, and retains output datasource/materialization
references.
- The focused composition check proves Connector origin -> Datasource ->
pinned Dataflow run -> frozen published materialization, including replay.
## Next Slices
1. Add persisted Workflow instances, resumable transitions, human activities,
retry policy, and event subscriptions against pinned definition revisions.
2. Add SQL database and governed REST origin providers with credential-envelope
references and bounded pushdown.
3. Add managed-file and directory origins.
4. Add datasource quality rules, schema compatibility policy, retention, and
promotion approvals.
5. Add asynchronous Dataflow workers and durable artifact-backed outputs for
runs that exceed the synchronous row/time/byte limits.
+3 -3
View File
@@ -8,11 +8,11 @@ The same pattern is reusable outside GovOPlaN for any project where Codex works
The repository contains Gitea issue templates in `.gitea/ISSUE_TEMPLATE`, a pull request template in `.gitea/PULL_REQUEST_TEMPLATE.md`, and the label taxonomy in `docs/gitea-labels.json`.
The scripts infer this repository from `origin` (`git@git.add-ideas.de:add-ideas/govoplan.git`). Override inference when needed:
The scripts infer this repository from `origin` (`git@git.add-ideas.de:GovOPlaN/govoplan.git`). Override inference when needed:
```bash
export GITEA_URL=https://git.add-ideas.de
export GITEA_OWNER=add-ideas
export GITEA_OWNER=GovOPlaN
export GITEA_REPO=govoplan
export GITEA_TOKEN=...
```
@@ -23,7 +23,7 @@ The API scripts also read `GITEA_*` values from the target repository's `.env` f
GITEA_TOKEN=...
# Optional if origin inference is not enough:
GITEA_URL=https://git.add-ideas.de
GITEA_OWNER=add-ideas
GITEA_OWNER=GovOPlaN
GITEA_REPO=govoplan
```
@@ -0,0 +1,321 @@
# Installation And Deployment Architecture
## Goal
A supported GovOPlaN installation starts with one downloaded, verified
bootstrap artifact. The administrator answers a bounded set of questions and
receives a working base system. Re-running the same tool repairs or
reconfigures that installation instead of creating unrelated state.
The canonical product journey remains
[System Administrator Lifecycle User Story](SYSTEM_ADMINISTRATOR_LIFECYCLE_USER_STORY.md).
This document defines the deployer boundary and the first executable slice.
## First Executable Slice
`tools/deployment/govoplan-deploy.py` is a standard-library-only deployment
compiler and reconciler. It can be tested without installing GovOPlaN itself.
It currently supports:
- evaluation and self-hosted profiles;
- managed or external PostgreSQL;
- managed, external, or evaluation-only disabled Redis;
- disabled mail, an external relay declaration, or an evaluation-only
GreenMail service;
- durable local file storage, managed single-node Garage S3, or external
S3-compatible storage;
- an explicit HAProxy service that load-balances configured WebUI and API
replicas without access to the Docker socket;
- declarative API, WebUI, and worker replica counts while keeping migrations
and the scheduler singleton;
- Core, base, or full initial module selections;
- deterministic Compose JSON accepted by Compose v2;
- generated secrets stored in a private `0600` file;
- service-specific environment allowlists so infrastructure containers do not
receive unrelated application credentials;
- plan, render, doctor, status, and apply commands;
- an installation lock, migration-before-start ordering, readiness polling,
and an applied-state receipt;
- idempotent reconfiguration that preserves generated secrets;
- a keyed environment fingerprint that detects private binding changes without
writing secret values to plans or receipts;
- host CPU, memory, disk, entropy, architecture, Docker daemon, Compose,
listen-port, and external endpoint preflight checks;
- service removal without implicit data-volume deletion.
Create a local evaluation bundle:
```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py init \
--directory /tmp/govoplan-evaluation \
--profile evaluation \
--postgres managed \
--redis managed \
--storage garage \
--mail test-mail \
--api-replicas 2 \
--web-replicas 2 \
--worker-replicas 2 \
--module-set base
```
Inspect the generated intent and host requirements:
```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py doctor \
--directory /tmp/govoplan-evaluation
```
Change a component without rotating existing generated secrets:
```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
--directory /tmp/govoplan-evaluation \
--redis external \
--redis-url 'rediss://:password@redis.example.org:6379/0'
```
The private installation directory contains:
| File | Purpose |
| --- | --- |
| `installation.json` | Versioned, non-secret desired state |
| `secrets.env` | Deployment-local secrets and external service bindings |
| `compose.json` | Deterministic generated Compose definition |
| `garage.toml` | Non-secret managed Garage server configuration |
| `load-balancer.cfg` | Non-secret HAProxy WebUI/API discovery configuration |
| `plan.json` | Latest desired-state diff and readiness findings |
| `receipt.json` | Last successfully applied immutable identities |
| `.deployment.lock` | Same-host operation exclusion |
The specification contract is
[`installation-spec.schema.json`](installation-spec.schema.json).
Build the same dependency-free tool as one downloadable artifact:
```sh
./.venv/bin/python tools/deployment/build-deployer-zipapp.py \
--output /tmp/govoplan-deploy.pyz
python /tmp/govoplan-deploy.pyz --help
```
To exercise reconciliation with locally available evaluation images:
```sh
python /tmp/govoplan-deploy.pyz init \
--non-interactive \
--directory /tmp/govoplan-evaluation \
--profile evaluation \
--api-image local/govoplan-api:test \
--web-image local/govoplan-web:test
python /tmp/govoplan-deploy.pyz apply \
--directory /tmp/govoplan-evaluation \
--allow-unverified-images \
--skip-pull
```
Those images must already contain the selected module set. The override exists
only to exercise local orchestration before release artifacts exist; it is
rejected for `self-hosted`.
## Current Production Gates
The tool deliberately reports blockers instead of pretending the source tree is
a production distribution:
1. **OCI release artifacts.** The release pipeline does not yet publish pinned
multi-architecture API and WebUI images.
2. **Signed distribution manifest.** A channel manifest must bind exact image
digests, Compose compatibility, SBOM/provenance references, and revocation
state. Recording a URL and checksum is not signature verification.
3. **First administrator.** Production needs a one-time, restricted enrollment
identity. The development bootstrap must not be enabled in production.
4. **Image/module composition.** The selected module set must be proven present
in the exact image or installed from verified offline artifacts before it is
enabled.
5. **Deployment agent.** Web updates need a separate privileged reconciler with
a typed command allowlist. The API and browser must never receive the Docker
socket or arbitrary shell access.
6. **Ingress and certificates.** The managed HAProxy service provides HTTP
load balancing inside the deployment boundary; it does not issue or renew
certificates. A self-hosted profile still needs an explicit choice
between an existing reverse proxy and a supported managed ingress, including
trusted-proxy boundaries, TLS certificate issuance, renewal, and health
probing through the public route.
`apply --allow-unverified-images` is therefore restricted to the evaluation
profile. It explicitly acknowledges both mutable image identities and
unverified image/module composition. It is a local test escape hatch, not a
production setting.
## Component Choices
### PostgreSQL
`managed` creates a persistent PostgreSQL container and private generated
credentials. `external` requires an explicit `DATABASE_URL`; switching from
managed to external cannot reuse the old `postgres` Docker hostname
accidentally.
Interactive entry hides external URLs because they commonly contain
credentials. For unattended automation, provide them through a protected
operator mechanism and avoid storing secret-bearing flags in shell history.
Production policy should support external managed databases and local managed
PostgreSQL equally at the application boundary. Backup, point-in-time recovery,
high availability, and major-version upgrades remain deployment properties.
### Redis
`managed` creates an authenticated, append-only Redis container. `external`
requires an explicit `REDIS_URL`. `disabled` is evaluation-only and disables
workers while recording the single-process login-throttle risk acknowledgement.
`doctor` performs a bounded TCP connection check for external PostgreSQL,
Redis, and S3 endpoints. This verifies DNS, routing, and that the port accepts a
connection; it is not an authentication or semantic health check.
Production base installations include Redis because durable queues, distributed
throttling, notifications, scheduled work, and transactional event delivery
must survive API restarts.
### Mail
The first slice distinguishes:
- `disabled`;
- `external-relay`, which records the infrastructure decision but leaves Mail
server/credential creation as a visible post-install task;
- `test-mail`, an evaluation-only GreenMail service.
A bundled production mail server is intentionally not a default. Operating one
requires DNS, reverse DNS, TLS, DKIM, SPF, DMARC, reputation, abuse handling,
queue monitoring, and upgrade policy. A later profile may support an
operator-selected MTA/relay, but it must expose these requirements rather than
presenting a container as a complete mail service.
### File Storage
`local` uses a durable Compose volume and is appropriate for one-host
installations. `garage` provisions Garage 2.3 in its supported single-node
bootstrap mode, generates a private application key and bucket, and connects
the Files S3 backend to the exact installer-owned internal endpoint. The
deployment-only trust marker cannot authorize another S3 host; arbitrary
external SDK endpoints remain fail-closed until peer pinning is implemented.
Garage metadata and object data use separate persistent volumes. `s3` requires
an external endpoint, region, access key, secret key, and bucket values.
Self-hosted external S3 endpoints must use HTTPS.
Local storage must be included in backup and restore drills. Horizontal API or
worker scale-out requires shared/object storage. The managed Garage profile is
persistent but has no data redundancy; availability-sensitive installations
must use a tested multi-node Garage cluster or another external S3 service.
### Load Balancing And Replicas
The generated Compose topology publishes only `load-balancer`. HAProxy uses
Docker DNS service discovery to distribute public traffic across WebUI replicas
and WebUI API proxy traffic across API replicas. The WebUI and API services do
not publish host ports. HAProxy has no Docker socket and discovers only the
bounded replica slots rendered into `load-balancer.cfg`.
Replica counts are desired state:
```sh
./.venv/bin/python tools/deployment/govoplan-deploy.py configure \
--directory /tmp/govoplan-evaluation \
--api-replicas 3 \
--web-replicas 2 \
--worker-replicas 4
./.venv/bin/python tools/deployment/govoplan-deploy.py apply \
--directory /tmp/govoplan-evaluation
```
Workers are queue consumers, so they are scaled through Redis rather than put
behind an HTTP load balancer. The migration runner and Celery scheduler remain
singletons. Multiple API replicas are rejected when Redis is disabled because
distributed throttling and queued work cannot then be shared correctly.
This is same-host scaling. Docker Compose uses a bridge network and does not
place containers on another machine. See
[Scaling And Multi-Host Deployment](SCALING_AND_MULTI_HOST_DEPLOYMENT.md) for
the supported topology and promotion path.
## Reconfiguration Semantics
`installation.json` is desired state. `receipt.json` is the last successfully
applied state. `plan` compares their canonical hashes and service sets.
- Adding a managed component creates its service and persistent volume.
- Removing a component removes its service container on apply.
- Volumes are retained by default; deleting data requires a separate,
deliberately destructive workflow.
- Existing generated credentials are retained unless an explicit future rotate
operation is requested.
- Private configuration changes are represented by a keyed fingerprint in the
plan and receipt; plaintext values are never copied there.
- Managed-to-external transitions require the new endpoint in the same
operation.
- Migrations run as a one-shot service before API/worker replacement.
- The first upgrade from a direct WebUI host port stops that legacy WebUI
container immediately before HAProxy claims the same endpoint.
- Health must recover before a new receipt is committed.
This is sufficient for one-host reconciliation. Production updates additionally
need backup/restore gates, maintenance/drain state, database compatibility
windows, image signature verification, and rollback/forward-recovery policy.
## Web Update Boundary
The intended update path is:
1. Ops reads the non-secret installation receipt and reports management mode,
current release, component health, and update availability.
2. An authorized administrator asks Core to create a typed deployment request,
for example `reconcile_release` or `rollback_release`.
3. Core persists the reviewed immutable plan, actor, expected current receipt,
and idempotency key.
4. A separately deployed, narrow deployment agent claims the request.
5. The agent verifies signatures/digests, acquires a fenced deployment lock,
backs up, pulls, migrates, reconciles, probes health, and writes evidence.
6. Ops presents durable progress and the resulting receipt.
The agent owns container-runtime access. It accepts no command strings from the
browser and has no domain-data permissions. Installations managed by Kubernetes,
systemd, or another external orchestrator expose read-only status and an export
of the reviewed update recipe instead of a non-functional update button.
## Distribution Workflow
The downloadable entry point should eventually be:
```sh
curl --proto '=https' --tlsv1.2 --fail --location \
https://govoplan.add-ideas.de/install/v1/bootstrap.pyz \
--output govoplan-bootstrap.pyz
python3 govoplan-bootstrap.pyz init
```
The published documentation must include an independent checksum/signature
verification command before execution. The zipapp then downloads only a signed
distribution manifest, verifies it against an embedded or explicitly installed
keyring, and renders the same installation contract implemented here.
The source-tree script is the test harness for that future zipapp. It is not yet
the internet bootstrap artifact.
## Verification
Run the focused tests:
```sh
./.venv/bin/python -m unittest -v tests.test_deployment_installer
```
The tests cover profile restrictions, secret persistence, external endpoint
requirements, managed Garage bootstrap, S3 policy, replica validation, HAProxy
discovery configuration, Compose service selection, secret non-disclosure,
service-specific environment isolation, private file modes, external endpoint
preflight, first-plan generation, apply ordering, and receipt-based
idempotency.
+8 -8
View File
@@ -444,25 +444,25 @@ reason to infer that a pattern is satisfied.
and does not duplicate a central component.
- Behavioral/accessibility evidence is linked from the rollout matrix and issue.
- Configured-system help can reach the applicable pattern or reference topic
when [Docs #15](https://git.add-ideas.de/add-ideas/govoplan-docs/issues/15)
when [Docs #15](https://git.add-ideas.de/GovOPlaN/govoplan-docs/issues/15)
supplies that experience.
## First Pilot: Campaign
[Campaign #74](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/74)
[Campaign #74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74)
is the first full-domain audit and migration. It should prove patterns before
generic extraction:
- [#59](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/59) and
[#73](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/73): stable,
- [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) and
[#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73): stable,
accessible preview and attachment-detail overlays
- [#63](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/63): review
- [#63](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/63): review
stages, outcomes, blockers, and intervention vocabulary
- [#62](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/62): explicit
- [#62](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/62): explicit
synchronous/asynchronous send mode and durable delivery progress
- [#65](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/65): one
- [#65](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/65): one
coherent report filtering and count-affordance model
- [#35](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/35): guided
- [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35): guided
first-campaign entry
These slices do not depend on the Workflow runtime. Campaign's current
+28 -21
View File
@@ -9,6 +9,12 @@ The applicable design contract is
## Snapshot And Method
The source-derived inventory command is documented in
[`PLATFORM_CONTROL_PLANE.md`](PLATFORM_CONTROL_PLANE.md). It produces
machine-readable field, label, translation, route, API-reference, and module
manifest evidence. This hand-maintained document remains the reviewed product
interpretation and rollout ledger; generated evidence does not replace it.
Snapshot refreshed: 2026-07-22.
Evidence was read from tracked Git `HEAD` in the local GovOPlaN checkouts:
@@ -59,16 +65,16 @@ The access guard column reports only the route-level declaration in
| Route | Owner / render evidence | Route-level access evidence | Primary task | Target archetype | Status / priority |
| --- | --- | --- | --- | --- | --- |
| `/admin` | `govoplan-access` `AdminPage` | Any core `adminReadScopes` | Administer system and tenant concerns assembled from module sections | Administration/configuration | Contributed; unreviewed; P1 under [Core #225](https://git.add-ideas.de/add-ideas/govoplan-core/issues/225) |
| `/admin` | `govoplan-access` `AdminPage` | Any core `adminReadScopes` | Administer system and tenant concerns assembled from module sections | Administration/configuration | Contributed; unreviewed; P1 under [Core #225](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/225) |
| `/address-book` | `govoplan-addresses` `AddressBookPage` | `addresses:contact:read` | Browse and manage contacts, address books, and lists | Directory/list-detail | Contributed; unreviewed; P2 after Campaign |
| `/calendar` | `govoplan-calendar` `CalendarPage` | `calendar:event:read` | Browse calendars/events and act on calendar data | Directory/list-detail | Contributed; metadata gap; unreviewed; P2 after Campaign |
| `/campaigns` | `govoplan-campaign` `CampaignListPage` | `campaigns:campaign:read` | Find, compare, create, and open campaigns | List-detail entry | Pilot; P1 [Campaign #74](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/74) |
| `/campaigns` | `govoplan-campaign` `CampaignListPage` | `campaigns:campaign:read` | Find, compare, create, and open campaigns | List-detail entry | Pilot; P1 [Campaign #74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74) |
| `/campaigns/:campaignId/*` | `govoplan-campaign` `CampaignResourceRoute` and `CampaignWorkspace` | `campaigns:campaign:read`, plus resource probe | Configure, review, send, and inspect one campaign/version | List-detail workspace containing edit, review, monitoring, and evidence surfaces | Pilot; P1 Campaign #74 |
| `/operator` | `govoplan-campaign` `OperatorQueuePage` | `campaigns:campaign:read` and any of queue, control, retry, or reconcile | Monitor and intervene in campaign jobs through authority-specific controls | Monitoring/work queue | Pilot; durable queue controls delivered in [Campaign #78](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/78); #74 audit remains |
| `/reports` | `govoplan-campaign` `AggregateReportsPage` | `campaigns:report:read` | Compare privacy-protected cross-campaign outcome totals without recipient detail, diagnostics, export, or drill-down | Aggregate reporting | Pilot; aggregate-reader surface delivered in [Campaign #80](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/80); #74 audit remains |
| `/operator` | `govoplan-campaign` `OperatorQueuePage` | `campaigns:campaign:read` and any of queue, control, retry, or reconcile | Monitor and intervene in campaign jobs through authority-specific controls | Monitoring/work queue | Pilot; durable queue controls delivered in [Campaign #78](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/78); #74 audit remains |
| `/reports` | `govoplan-campaign` `AggregateReportsPage` | `campaigns:report:read` | Compare privacy-protected cross-campaign outcome totals without recipient detail, diagnostics, export, or drill-down | Aggregate reporting | Pilot; aggregate-reader surface delivered in [Campaign #80](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/80); #74 audit remains |
| `/templates` | `govoplan-campaign` `TemplatesPage` | No route guard declared in `module.ts` | Browse/manage campaign templates | Directory/list-detail | Pilot audit; permission intent must be verified; P2 |
| `/dashboard` | `govoplan-dashboard` `DashboardPage` | No route-specific scope | Assemble module-provided actionable widgets | Dashboard | Contributed; unreviewed; P2 |
| `/docs` | `govoplan-docs` `DocsPage` | Docs read or system/tenant settings read scopes | Read configured, available, and evidence-aware documentation | Documentation directory/reference | Contributed; unreviewed; P1 [Docs #15](https://git.add-ideas.de/add-ideas/govoplan-docs/issues/15) after initial pattern content |
| `/docs` | `govoplan-docs` `DocsPage` | Docs read or system/tenant settings read scopes | Read configured, available, and evidence-aware documentation | Documentation directory/reference | Contributed; unreviewed; P1 [Docs #15](https://git.add-ideas.de/GovOPlaN/govoplan-docs/issues/15) after initial pattern content |
| `/files` | `govoplan-files` `FilesPage` | `files:file:read` | Browse folders/files and perform managed-file work | Directory/explorer | Contributed; metadata gap; unreviewed; P2 after Campaign |
| `/idm` | `govoplan-idm` `IdmPage` | Any IDM assignment/write or organization function-assign scope | Inspect and govern identity/function assignments | List-detail/configuration | Contributed; unreviewed; P2 |
| `/mail` | `govoplan-mail` `MailboxPage` | `mail:mailbox:read` | Browse mailboxes and messages | Directory/list-detail | Contributed; metadata gap; unreviewed; P2 after Campaign |
@@ -138,7 +144,7 @@ above; they are not independent routes.
Campaign is detailed first because it exercises almost every archetype. The
recipient-data editor is now consolidated into the `recipients` section on
remote `main`; [Campaign #67](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/67)
remote `main`; [Campaign #67](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/67)
records the accepted and verified integration boundary.
Campaign already consumes core primitives including `ModuleSubnav`, `Card`,
@@ -150,25 +156,25 @@ prove that the composition or states satisfy the pattern.
| Surface / code evidence | Primary task | Target pattern | Material consequence/state | Known issue / rollout |
| --- | --- | --- | --- | --- |
| Campaign list (`CampaignListPage`) | Find, compare, create, open | List-detail entry | Campaign lifecycle/status and creation | Audit in [#74](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/74); guided entry [#35](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/35) |
| Campaign list (`CampaignListPage`) | Find, compare, create, open | List-detail entry | Campaign lifecycle/status and creation | Audit in [#74](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/74); guided entry [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35) |
| Overview (`CampaignOverviewPage`) | Understand/edit campaign identity, version, access, lifecycle | Object overview plus adaptive edit | Lock/archive/delete/access changes need real consequence and reversibility wording | #74 remaining audit |
| Fields (`CampaignFieldsPage`) | Define recipient/template field schema | Structured editor | Schema changes can invalidate recipient/template data | #74 audit |
| Attachments/files (`AttachmentsDataPage`, `AttachmentRulesOverlay`) | Select sources and attachment/ZIP rules | Directory chooser plus adaptive rule editor | Missing or mismatched files affect built messages | #74; attachment-detail [#59](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/59) |
| Attachments/files (`AttachmentsDataPage`, `AttachmentRulesOverlay`) | Select sources and attachment/ZIP rules | Directory chooser plus adaptive rule editor | Missing or mismatched files affect built messages | #74; attachment-detail [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) |
| Recipients (`RecipientDataPage`) | Select/import/map/edit recipients, address fields and per-recipient values/files | Import/mapping plus list-detail editor | Personal data, validation, bulk activation, file links | Consolidated editor delivered in #67; #74 remaining audit and guided entry #35 |
| Template (`TemplateDataPage`, placeholder/expression dialogs) | Author subject/body and preview substitutions | Adaptive editor plus stable preview | Generated communication content and unresolved expressions | #74; stable overlay [#73](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/73) |
| Template (`TemplateDataPage`, placeholder/expression dialogs) | Author subject/body and preview substitutions | Adaptive editor plus stable preview | Generated communication content and unresolved expressions | #74; stable overlay [#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73) |
| Mail settings (`MailSettingsPage` settings view) | Select/configure campaign mail transport | Adaptive configuration | Credentials, SMTP/IMAP destinations, test outcomes | #74; align with Core #225 mail pattern |
| Campaign settings (`GlobalSettingsPage` settings view) | Configure campaign behavior | Adaptive configuration | Can alter validation/build/send behavior | #74 audit |
| Mail policy (`MailSettingsPage` policy view) | Inspect/override effective mail policy | Effective policy/provenance editor | Inheritance and locks affect allowed delivery | #74; Core #225 policy pattern |
| Campaign policy (`GlobalSettingsPage` policy view) | Inspect/override campaign policy | Effective policy/provenance editor | Inheritance, actor authority, and blocked edits | #74; Core #225 policy pattern |
| Review/send (`ReviewSendPage`) | Validate, build, mock-test, confirm/send, inspect results | Guided review/decision plus durable progress | External communication, bounded synchronous execution, persisted queue mode, partial effects, retries, evidence | Bounded synchronous and explicit/persisted queued modes delivered in [#62](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/62) and [#79](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/79); [#63](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/63) wording and #74 audit remain |
| Message and attachment detail overlays | Inspect one built/mock message and its attachment links | Stable detail/review dialog | Personal data, exact outbound content, reviewed state | Delivered and verified in [#59](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/59) and [#73](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/73) |
| Campaign report (`CampaignReportPage`) | Filter and inspect delivery outcomes | Reporting/list-detail | Partial, failed, explicitly excluded/skipped, SMTP/IMAP outcomes and retries | Server-owned filtering and counts delivered in [#65](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/65) with the full-result DataGrid contract from [Core #263](https://git.add-ideas.de/add-ideas/govoplan-core/issues/263); excluded semantics in [#66](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/66) |
| Review/send (`ReviewSendPage`) | Validate, build, mock-test, confirm/send, inspect results | Guided review/decision plus durable progress | External communication, bounded synchronous execution, persisted queue mode, partial effects, retries, evidence | Bounded synchronous and explicit/persisted queued modes delivered in [#62](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/62) and [#79](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/79); [#63](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/63) wording and #74 audit remain |
| Message and attachment detail overlays | Inspect one built/mock message and its attachment links | Stable detail/review dialog | Personal data, exact outbound content, reviewed state | Delivered and verified in [#59](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/59) and [#73](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/73) |
| Campaign report (`CampaignReportPage`) | Filter and inspect delivery outcomes | Reporting/list-detail | Partial, failed, explicitly excluded/skipped, SMTP/IMAP outcomes and retries | Server-owned filtering and counts delivered in [#65](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/65) with the full-result DataGrid contract from [Core #263](https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/263); excluded semantics in [#66](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/66) |
| Audit (`CampaignAuditPage`) | Inspect campaign evidence/history | Provenance timeline/report | Actor/action/effect trace | #74 audit |
| JSON (`CampaignJsonView`) | Inspect expert representation | Advanced diagnostics/reference | Raw data may contain personal/configuration values; not a primary editor | #74 privacy/redaction audit |
| Create wizard (`CreateWizard`) | Seed a campaign through basics, sender, fields, recipients, template, attachments, review, send | Guided setup | Current steps mix creation and later consequential delivery; completion semantics need audit | Guided first campaign [#35](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/35) |
| Create wizard (`CreateWizard`) | Seed a campaign through basics, sender, fields, recipients, template, attachments, review, send | Guided setup | Current steps mix creation and later consequential delivery; completion semantics need audit | Guided first campaign [#35](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/35) |
| Review/send wizard routes | Alternate guided review/send shells | Guided review | Tracked routes exist; implementation relationship to `ReviewSendPage` must be established, not guessed | #74 inventory decision |
| Operator queue (`OperatorQueuePage`) | Monitor jobs and intervene | Monitoring/work queue | Campaign/version/job identity, historical active-version discovery, fixed action positions, authority-aware disabled states, exact non-overlapping queue counts, server-paged jobs, bounded refresh, retry/queue/reconcile per version, campaign-wide pause/resume/cancel, and leave/return progress | Durable operator controls delivered in [#78](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/78); #74 wording/accessibility audit remains |
| Aggregate reports (`AggregateReportsPage`) | Compare cross-campaign delivery outcomes | Privacy-preserving aggregate reporting | Tenant/campaign ACL, deployment/tenant small-cell policy, complementary and overlapping-cell suppression, explicit denominator, and no recipient detail/diagnostics/export/drill-down | Separate aggregate-reader surface delivered in [#80](https://git.add-ideas.de/add-ideas/govoplan-campaign/issues/80); not parity with the permission-gated per-campaign detail report |
| Operator queue (`OperatorQueuePage`) | Monitor jobs and intervene | Monitoring/work queue | Campaign/version/job identity, historical active-version discovery, fixed action positions, authority-aware disabled states, exact non-overlapping queue counts, server-paged jobs, bounded refresh, retry/queue/reconcile per version, campaign-wide pause/resume/cancel, and leave/return progress | Durable operator controls delivered in [#78](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/78); #74 wording/accessibility audit remains |
| Aggregate reports (`AggregateReportsPage`) | Compare cross-campaign delivery outcomes | Privacy-preserving aggregate reporting | Tenant/campaign ACL, deployment/tenant small-cell policy, complementary and overlapping-cell suppression, explicit denominator, and no recipient detail/diagnostics/export/drill-down | Separate aggregate-reader surface delivered in [#80](https://git.add-ideas.de/GovOPlaN/govoplan-campaign/issues/80); not parity with the permission-gated per-campaign detail report |
| Templates route (`TemplatesPage`) | Browse template records | Directory/list-detail | Template availability and later generated outputs | #74 audit; verify missing route guard intent |
The five review stages currently named in code are `Validate and inspect`,
@@ -186,7 +192,7 @@ The following local repositories contain a backend manifest but no
`govoplan-contracts`, `govoplan-dist-lists`, `govoplan-evaluation`,
`govoplan-facilities`, `govoplan-forms-runtime`, `govoplan-grants`,
`govoplan-helpdesk`, `govoplan-identity`, `govoplan-inspections`,
`govoplan-issue-reporting`, `govoplan-learning`, `govoplan-permits`,
`govoplan-tickets`, `govoplan-learning`, `govoplan-permits`,
`govoplan-poll`, `govoplan-procurement`, `govoplan-records`,
`govoplan-resources`, `govoplan-rest`, `govoplan-risk-compliance`,
`govoplan-soap`, `govoplan-tenancy`, and `govoplan-transparency`.
@@ -199,7 +205,7 @@ backend-only modules may remain intentionally headless.
| Order | Scope | Current evidence | Target | Owner / issue | Verification gate | Status |
| --- | --- | --- | --- | --- | --- | --- |
| 0 | Product grammar and route inventory | Doctrine, ledger, layout rules, module contract, current route sources | One reconciled pattern language and evidence inventory | Meta [#11](https://git.add-ideas.de/add-ideas/govoplan/issues/11) | Docs links/diff checks; issue/wiki sync after integration | Initial slice in this document |
| 0 | Product grammar and route inventory | Doctrine, ledger, layout rules, module contract, current route sources | One reconciled pattern language and evidence inventory | Meta [#11](https://git.add-ideas.de/GovOPlaN/govoplan/issues/11) | Docs links/diff checks; issue/wiki sync after integration | Initial slice in this document |
| 1 | Campaign baseline integration | Recipient-editor WIP and tracker state have been reconciled with remote `main` | Integrated, testable baseline before migration claims | Campaign #67 and tracker cleanup | Backend and focused WebUI suites; issue evidence | Complete 2026-07-22 |
| 2 | Campaign previews/details | Stable shared dialog with bounded scrolling and fixed responsive preview workspace | Stable header/body/footer, accessible long-content detail | Campaign #59 and #73 | Review-preview and overlay structure tests | Complete 2026-07-22 |
| 3 | Campaign review/interventions | Five domain-owned stages with unresolved intervention language | Clear stages, outcomes, blockers, next actor/action, reviewed evidence | Campaign #63 | State matrix behavior/accessibility tests and agreed vocabulary | P1 needs product wording decision |
@@ -215,10 +221,11 @@ backend-only modules may remain intentionally headless.
| 13 | Remaining direct routes | Routes are contributed; most are unreviewed | Per-module bounded audit and migration plan | New module issues derived from this inventory | Applicable definition-of-done gates | P2 after Campaign, not a bulk rewrite |
| 14 | Manifest/runtime alignment | Several executable routes are absent from manifest metadata | Declared alignment or explicit validated exception | Core contract issue to create | Automated manifest/module route check and configured Docs verification | Discovery follow-up |
Workflow/user-story implementation is postponed. It is not on the critical path
for this rollout matrix. Focused views can be specified, manually selected, and
tested through core composition contracts; a later workflow step may become one
activation source without changing the proven surface patterns.
Workflow remains outside this rollout matrix because it has its own runtime and
editor workstream, not because it is postponed. Focused views can be specified,
manually selected, and tested through core composition contracts today.
Workflow steps may activate those views through the same contract without
changing the proven surface patterns.
## Inventory Maintenance
+4 -1
View File
@@ -114,7 +114,10 @@ For release validation:
```sh
./.venv/bin/python tools/repo/sync-python-environment.py --requirements requirements-release.txt --python ./.venv/bin/python
./.venv/bin/python -m pip install -r requirements-release-tests.txt
```
That install is necessary because the release environment intentionally resolves
tagged package refs, not local editable source trees.
tagged package refs, not local editable source trees. The second requirements
file contains only the harness needed to execute tests from those immutable
source tags; it is not part of the deployable release dependency set.
+126
View File
@@ -0,0 +1,126 @@
# Platform Control Plane And Self-Description
## Objective
GovOPlaN should be able to describe its installed structure without becoming a
self-modifying application. The platform model is a declarative control plane:
module manifests, UI contributions, schemas, policy provenance, runtime
capabilities, and generated source evidence describe what can be configured.
Ordinary administrators edit validated data through those contracts; they do
not edit Python, TypeScript, routes, or database code from the product UI.
This distinction provides the requested overview while preserving reviewable
releases, module boundaries, migrations, and security controls.
## Canonical Sources
| Concern | Canonical source |
| --- | --- |
| Installed modules and dependency graph | Runtime `ModuleManifest` registry |
| Backend routes | Registered FastAPI application; Python AST is build-time evidence |
| Frontend routes and navigation | `PlatformWebModule` contributions |
| View-filterable regions | Versioned `viewSurfaces` declarations |
| Admin sections and module settings | `admin.sections`, including `moduleId`, `kind`, scope group, permission guards, and surface ID |
| User settings | `settings.sections` and core settings schemas |
| Labels and translations | Generated translation catalogs plus source usage |
| Fields and help coverage | Shared form components plus generated TypeScript AST inventory |
| API use by the WebUI | Typed API clients plus generated static reference inventory |
| Effective configuration | Owning module data plus Policy provenance |
Runtime introspection is authoritative for an installed system. Static source
inventory is authoritative evidence for a checkout or release candidate. The
two should be compared in CI and by Ops, not conflated.
## Generated Inventory
Run:
```bash
cd /mnt/DATA/git/govoplan
./.venv/bin/python tools/inventory/platform-interface-inventory.py
```
The command writes:
- `audit-reports/platform-inventory/platform-interface-inventory.json`
- `audit-reports/platform-inventory/platform-interface-inventory.md`
It combines:
1. loaded module manifests
2. TypeScript AST extraction of fields, label attributes, visible text,
translations, frontend routes, navigation, capabilities, and API references
3. Python AST extraction of FastAPI route decorators and router prefixes
The JSON includes exact repository, file, and line evidence. A missing-help
entry is a review candidate because dynamic parent components may supply help.
A backend route without a static frontend reference is also a review candidate:
public APIs, workers, callbacks, health checks, connectors, and dynamic URL
assembly are valid explanations.
`--strict` currently enforces only translation-catalog completeness. Endpoint
and help classifications need narrow reviewed baselines before they can become
release gates.
## Admin Information Architecture
The Admin host uses a tree because system, tenant, group, user, and module
settings form a hierarchy rather than one flat list. Every contributed section
can identify:
- its owning `moduleId`
- whether it is `management` or `settings`
- its system/tenant/group/user scope group
- an optional future `parentId`
- permission and View visibility requirements
Existing panels remain their own render owners. The tree only changes discovery
and grouping. A later embedded-settings contract may add named slots inside an
owning page; it must not allow one module to import another module's private
component.
## Navigation And Workflow
The intended maximum visible navigation stack is:
1. global shell context
2. one task/object navigation surface
3. one workflow stage surface when a workflow is active
Workflow instance pages should reuse the Campaign stage language: clear stage
state, optional/skipped/blocked semantics, partial progress, and a stable current
step. Workflow definition pages remain graph editors. Views may activate a
focused workflow view that suppresses unrelated shell and module surfaces while
retaining an explicit way out.
Nested module submenus should not be added merely because a data hierarchy
exists. Prefer a tree inside configuration/directory surfaces, tabs for sibling
views, and the workflow stage rail for ordered work.
## Safe Meta-Configuration
The platform can eventually render many configuration editors from versioned
JSON Schema and UI Schema supplied by modules. Generated editors remain bounded
by:
- explicit typed schemas and migrations
- module-owned validation and preview
- Policy locks and provenance
- permission and View filtering
- preflight, consequence, and rollback information
- auditable apply operations
Custom code, new routes, arbitrary SQL, and executable workflow nodes remain
release artifacts. Modeling them as ordinary configuration would create an
unreviewed code-execution and migration channel.
## Next Enforcement Slices
1. Require every WebUI module route and admin/settings contribution to have
matching manifest metadata or a reviewed exception.
2. Add stable field IDs and optional help-topic IDs to shared field components.
3. Classify each statically unreferenced backend endpoint by consumer type.
4. Compare a running installation's OpenAPI and module registry against the
release inventory.
5. Publish the sanitized installed-system structure through Ops/Docs for
authorized administrators.
+3 -2
View File
@@ -445,8 +445,9 @@ that first needs them:
- exact HIS/CampusOnline source endpoints, student-statistics keys and accepted
calculation, freeze/correction policy, privacy profile, and permitted
drill-down level;
- whether repeated data-source and dataflow contracts justify separate
`govoplan-datasources` and `govoplan-dataflow` modules after the Stage 3 proof;
- datasource provider selection and quality/promotion policy; the architecture
now separates `govoplan-datasources` lifecycle from `govoplan-connectors`
acquisition and `govoplan-dataflow` transformation;
- first collaborative editor/provider and whether the first UX is concurrent
editing, controlled check-out, or both; and
- first Records/archive target and approval/signature assurance level.
+62 -25
View File
@@ -22,10 +22,17 @@ effects can be verified safely:
- freeze a selective plan as a durable, resumable local release run
- durably preflight a creation-time-bound repository, create its annotated tag,
and publish its branch/tag pair atomically
- deterministically update recognized package/manifest version declarations and
commit only the receipt-bound metadata paths
- order selected module providers before consumers, create module tags before
Core, regenerate Core's selected WebUI release lock, and re-run alignment
before any remote push
- build selected Python wheels and generate a private, signed, receipt-bound
catalog candidate
- publish that exact candidate through a verified website commit and immutable
tag after explicit confirmation
- install selected candidate wheels into a private no-network/no-dependency
target and verify their installed metadata against the frozen plan
Start it from the meta repository:
@@ -67,15 +74,35 @@ shows the dry-run commands for the selected rows, and `Generate Candidate`
creates a signed catalog candidate that advances only selected repositories that
already have a catalog entry.
The full-width **Release Workflow** guide projects the server state into seven
operator phases: Inspect, Targets, Validate, Source, Package, Publish, and
Verify. It does not maintain a second workflow state. Completed, current,
blocked, locked, and unavailable phases are derived from the dashboard,
selective plan, and durable run record. The next-action panel opens the exact
section or durable step that needs attention. Changing a channel, target
version, repository selection, or release gate detaches the browser from the
current run and invalidates the draft plan; the persisted run remains available
from the saved-run selector. Problems in unselected repositories remain visible
as workspace notices but do not lock an unrelated release; the selective plan
is the authority for blockers in the selected repository set.
Installation verification is an explicit durable step after catalog
publication. It verifies the exact candidate receipt, installs every selected
Python wheel into a temporary target with network and dependency resolution
disabled, and compares installed names and versions with the frozen plan.
Deployment startup, database upgrades, and module-combination smoke tests remain
release-integration CI gates; the console does not represent its local wheel
check as a production deployment.
`Build Plan` also returns structured release-gate findings for each selected
repository. The plan names the recommended next action and gives an explicit
remediation for source-version, lockfile, Core WebUI composition, Git state, and
worktree findings. A target version that has not yet been applied consistently
to the selected source tree is therefore explained before the source-tag
preflight rather than appearing only as an error after the operator tries to
tag. `source_preflight_ready` means that the plan-visible source gates pass; the
non-mutating `Preview Tag + Publish` remains mandatory for remote, manifest, and
immutable-tag checks.
worktree findings. A target version that differs from internally consistent
source metadata becomes a bounded `UPDATE` step. Unsupported, missing, or
internally inconsistent declarations remain a blocker with exact remediation.
`source_preflight_ready` means that plan-visible source gates pass or have a
bounded deterministic mutation; the non-mutating `Preview Tag + Publish`
remains mandatory for remote, manifest, and immutable-tag checks.
## Durable release runs
@@ -85,11 +112,12 @@ requires the plan to resolve exactly the requested repositories and target
versions; the browser cannot submit or replace the plan snapshot. The input and
plan are then immutable and covered by a canonical SHA-256 integrity digest.
Every executable repository step also carries its creation-time full HEAD,
branch, worktree state, target tag, and a SHA-256 over both the fetch and push
URLs of `origin`. Both URLs must exactly equal the remote registered in
`repositories.json`; a changed HEAD, branch, worktree, remote, or push URL
requires a new run rather than silently retargeting the frozen compatibility
decision.
branch, target tag, a SHA-256 over both the fetch and push URLs of `origin`, and
a SHA-256 over bounded dirty-path names and bytes. Both URLs must exactly equal
the remote registered in `repositories.json`; a changed HEAD, branch, worktree,
remote, push URL, or metadata byte requires a new run or explicit
interrupted-step reconciliation rather than silently retargeting the frozen
compatibility decision.
The complete record also has a checksum so a valid-looking manual edit to its
mutable state fails closed. File permissions remain the authority boundary;
these digests detect accidental or manual corruption, not an attacker who can
@@ -177,24 +205,32 @@ retry. The UI keeps unavailable controls visible and disabled.
Supported executors durably claim the step before invoking an effect. Exact
attempt replays return the recorded outcome and never invoke the executor a
second time. Successful repository preflight, tag, and push steps persist a
bounded repository-state receipt. Tag reconciliation independently requires an
second time. Successful repository preflight, version, commit, Core-bundle,
tag, and push steps persist a bounded repository-state receipt. Version
reconciliation requires aligned declarations in the same commit; commit
reconciliation requires the expected single-parent release commit and only
recognized metadata paths. Tag reconciliation independently requires an
annotated local tag at the frozen HEAD; push reconciliation additionally
requires both the remote annotated tag object and remote branch to match.
Catalog generation persists
only its server-issued opaque candidate ID and canonical catalog SHA-256, then
re-resolves and re-hashes that private candidate before publication.
The safe baseline is intentionally narrower than the complete dry-run plan.
Dirty worktrees and version changes still show commit/version steps, but those
steps have no durable executor because the frozen run does not bind the exact
proposed file content. A run selecting Core together with one or more modules
begins with a disabled dependency-ordering barrier: local module tags, Core
release-lock regeneration/commit, Core tagging, alignment verification, and
pushes need an explicit DAG executor before that composition can mutate.
Operators must prepare and review those changes outside the console and create
a new run from the resulting clean HEAD. The console never skips these steps or
claims an end-to-end release succeeded.
Repository capabilities are frozen into each plan unit (`python-package`,
`webui-package`, `module-manifest`, `database-migrations`, `documentation`,
`core-release-bundle`, and the universal `git-source`) and determine which
steps appear. Internally aligned version changes are rendered deterministically
from recognized TOML, JSON, lockfile, manifest, and package declarations.
Pre-existing dirty worktrees remain visible but have no commit executor; the
console never absorbs unrelated operator changes.
For mixed releases, module interface providers are ordered before consumers
and Core is tagged last. The durable sequence creates and commits module
metadata, creates local module tags, updates Core's selected WebUI references,
regenerates the release lock against those local tags, commits/tags Core, and
runs a receipt-bound alignment gate before exposing any atomic branch/tag push.
A failed step stops later steps while preserving prior receipts for explicit
retry or reconciliation.
The browser likewise retains the request identifier for an uncertain
resume/retry/reconciliation response and replays it after reload. A successful
@@ -226,8 +262,9 @@ other `/api/` route:
- `POST /api/release-runs/{run_id}/steps/{step_id}/preview` provides the
non-mutating preview for receipt-bound catalog publication.
Run-storage errors are confined to the Durable Run State card; dashboard and
release-preview collection continue and show the bounded storage remediation.
Run-storage errors are confined to the Durable Release Run section; dashboard
and release-preview collection continue and the workflow guide points to the
bounded storage remediation.
The run record is execution evidence only for a supported step whose durable
claim and bounded result receipt were persisted. The console never infers
+71 -65
View File
@@ -6,81 +6,87 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s
| Repository | Subtype | Local path | Gitea |
| --- | --- | --- | --- |
| `govoplan` | `meta` | `../govoplan` | [govoplan](https://git.add-ideas.de/add-ideas/govoplan) |
| `govoplan-core` | `kernel` | `../govoplan-core` | [govoplan-core](https://git.add-ideas.de/add-ideas/govoplan-core) |
| `govoplan` | `meta` | `../govoplan` | [govoplan](https://git.add-ideas.de/GovOPlaN/govoplan) |
| `govoplan-core` | `kernel` | `../govoplan-core` | [govoplan-core](https://git.add-ideas.de/GovOPlaN/govoplan-core) |
## Module
| Repository | Subtype | Local path | Gitea |
| --- | --- | --- | --- |
| `govoplan-access` | `platform` | `../govoplan-access` | [govoplan-access](https://git.add-ideas.de/add-ideas/govoplan-access) |
| `govoplan-addresses` | `domain` | `../govoplan-addresses` | [govoplan-addresses](https://git.add-ideas.de/add-ideas/govoplan-addresses) |
| `govoplan-admin` | `platform` | `../govoplan-admin` | [govoplan-admin](https://git.add-ideas.de/add-ideas/govoplan-admin) |
| `govoplan-appointments` | `domain` | `../govoplan-appointments` | [govoplan-appointments](https://git.add-ideas.de/add-ideas/govoplan-appointments) |
| `govoplan-approvals` | `domain` | `../govoplan-approvals` | [govoplan-approvals](https://git.add-ideas.de/add-ideas/govoplan-approvals) |
| `govoplan-assets` | `domain` | `../govoplan-assets` | [govoplan-assets](https://git.add-ideas.de/add-ideas/govoplan-assets) |
| `govoplan-audit` | `platform` | `../govoplan-audit` | [govoplan-audit](https://git.add-ideas.de/add-ideas/govoplan-audit) |
| `govoplan-booking` | `domain` | `../govoplan-booking` | [govoplan-booking](https://git.add-ideas.de/add-ideas/govoplan-booking) |
| `govoplan-calendar` | `domain` | `../govoplan-calendar` | [govoplan-calendar](https://git.add-ideas.de/add-ideas/govoplan-calendar) |
| `govoplan-campaign` | `domain` | `../govoplan-campaign` | [govoplan-campaign](https://git.add-ideas.de/add-ideas/govoplan-campaign) |
| `govoplan-cases` | `domain` | `../govoplan-cases` | [govoplan-cases](https://git.add-ideas.de/add-ideas/govoplan-cases) |
| `govoplan-certificates` | `domain` | `../govoplan-certificates` | [govoplan-certificates](https://git.add-ideas.de/add-ideas/govoplan-certificates) |
| `govoplan-committee` | `domain` | `../govoplan-committee` | [govoplan-committee](https://git.add-ideas.de/add-ideas/govoplan-committee) |
| `govoplan-consultation` | `domain` | `../govoplan-consultation` | [govoplan-consultation](https://git.add-ideas.de/add-ideas/govoplan-consultation) |
| `govoplan-contracts` | `domain` | `../govoplan-contracts` | [govoplan-contracts](https://git.add-ideas.de/add-ideas/govoplan-contracts) |
| `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/add-ideas/govoplan-dashboard) |
| `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/add-ideas/govoplan-dms) |
| `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/add-ideas/govoplan-dist-lists) |
| `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/add-ideas/govoplan-docs) |
| `govoplan-erp` | `domain` | `../govoplan-erp` | [govoplan-erp](https://git.add-ideas.de/add-ideas/govoplan-erp) |
| `govoplan-evaluation` | `domain` | `../govoplan-evaluation` | [govoplan-evaluation](https://git.add-ideas.de/add-ideas/govoplan-evaluation) |
| `govoplan-facilities` | `domain` | `../govoplan-facilities` | [govoplan-facilities](https://git.add-ideas.de/add-ideas/govoplan-facilities) |
| `govoplan-files` | `domain` | `../govoplan-files` | [govoplan-files](https://git.add-ideas.de/add-ideas/govoplan-files) |
| `govoplan-forms` | `domain` | `../govoplan-forms` | [govoplan-forms](https://git.add-ideas.de/add-ideas/govoplan-forms) |
| `govoplan-forms-runtime` | `platform` | `../govoplan-forms-runtime` | [govoplan-forms-runtime](https://git.add-ideas.de/add-ideas/govoplan-forms-runtime) |
| `govoplan-grants` | `domain` | `../govoplan-grants` | [govoplan-grants](https://git.add-ideas.de/add-ideas/govoplan-grants) |
| `govoplan-helpdesk` | `domain` | `../govoplan-helpdesk` | [govoplan-helpdesk](https://git.add-ideas.de/add-ideas/govoplan-helpdesk) |
| `govoplan-identity` | `platform` | `../govoplan-identity` | [govoplan-identity](https://git.add-ideas.de/add-ideas/govoplan-identity) |
| `govoplan-identity-trust` | `platform` | `../govoplan-identity-trust` | [govoplan-identity-trust](https://git.add-ideas.de/add-ideas/govoplan-identity-trust) |
| `govoplan-idm` | `platform` | `../govoplan-idm` | [govoplan-idm](https://git.add-ideas.de/add-ideas/govoplan-idm) |
| `govoplan-inspections` | `domain` | `../govoplan-inspections` | [govoplan-inspections](https://git.add-ideas.de/add-ideas/govoplan-inspections) |
| `govoplan-issue-reporting` | `domain` | `../govoplan-issue-reporting` | [govoplan-issue-reporting](https://git.add-ideas.de/add-ideas/govoplan-issue-reporting) |
| `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/add-ideas/govoplan-learning) |
| `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/add-ideas/govoplan-ledger) |
| `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/add-ideas/govoplan-mail) |
| `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/add-ideas/govoplan-notifications) |
| `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/add-ideas/govoplan-ops) |
| `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/add-ideas/govoplan-organizations) |
| `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/add-ideas/govoplan-payments) |
| `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/add-ideas/govoplan-permits) |
| `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/add-ideas/govoplan-policy) |
| `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/add-ideas/govoplan-poll) |
| `govoplan-portal` | `domain` | `../govoplan-portal` | [govoplan-portal](https://git.add-ideas.de/add-ideas/govoplan-portal) |
| `govoplan-postbox` | `domain` | `../govoplan-postbox` | [govoplan-postbox](https://git.add-ideas.de/add-ideas/govoplan-postbox) |
| `govoplan-procurement` | `domain` | `../govoplan-procurement` | [govoplan-procurement](https://git.add-ideas.de/add-ideas/govoplan-procurement) |
| `govoplan-records` | `domain` | `../govoplan-records` | [govoplan-records](https://git.add-ideas.de/add-ideas/govoplan-records) |
| `govoplan-reporting` | `domain` | `../govoplan-reporting` | [govoplan-reporting](https://git.add-ideas.de/add-ideas/govoplan-reporting) |
| `govoplan-resources` | `domain` | `../govoplan-resources` | [govoplan-resources](https://git.add-ideas.de/add-ideas/govoplan-resources) |
| `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/add-ideas/govoplan-risk-compliance) |
| `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/add-ideas/govoplan-scheduling) |
| `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/add-ideas/govoplan-search) |
| `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/add-ideas/govoplan-tasks) |
| `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/add-ideas/govoplan-templates) |
| `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/add-ideas/govoplan-tenancy) |
| `govoplan-transparency` | `domain` | `../govoplan-transparency` | [govoplan-transparency](https://git.add-ideas.de/add-ideas/govoplan-transparency) |
| `govoplan-workflow` | `platform` | `../govoplan-workflow` | [govoplan-workflow](https://git.add-ideas.de/add-ideas/govoplan-workflow) |
| `govoplan-access` | `platform` | `../govoplan-access` | [govoplan-access](https://git.add-ideas.de/GovOPlaN/govoplan-access) |
| `govoplan-addresses` | `domain` | `../govoplan-addresses` | [govoplan-addresses](https://git.add-ideas.de/GovOPlaN/govoplan-addresses) |
| `govoplan-admin` | `platform` | `../govoplan-admin` | [govoplan-admin](https://git.add-ideas.de/GovOPlaN/govoplan-admin) |
| `govoplan-appointments` | `domain` | `../govoplan-appointments` | [govoplan-appointments](https://git.add-ideas.de/GovOPlaN/govoplan-appointments) |
| `govoplan-approvals` | `domain` | `../govoplan-approvals` | [govoplan-approvals](https://git.add-ideas.de/GovOPlaN/govoplan-approvals) |
| `govoplan-assets` | `domain` | `../govoplan-assets` | [govoplan-assets](https://git.add-ideas.de/GovOPlaN/govoplan-assets) |
| `govoplan-audit` | `platform` | `../govoplan-audit` | [govoplan-audit](https://git.add-ideas.de/GovOPlaN/govoplan-audit) |
| `govoplan-booking` | `domain` | `../govoplan-booking` | [govoplan-booking](https://git.add-ideas.de/GovOPlaN/govoplan-booking) |
| `govoplan-calendar` | `domain` | `../govoplan-calendar` | [govoplan-calendar](https://git.add-ideas.de/GovOPlaN/govoplan-calendar) |
| `govoplan-campaign` | `domain` | `../govoplan-campaign` | [govoplan-campaign](https://git.add-ideas.de/GovOPlaN/govoplan-campaign) |
| `govoplan-cases` | `domain` | `../govoplan-cases` | [govoplan-cases](https://git.add-ideas.de/GovOPlaN/govoplan-cases) |
| `govoplan-certificates` | `domain` | `../govoplan-certificates` | [govoplan-certificates](https://git.add-ideas.de/GovOPlaN/govoplan-certificates) |
| `govoplan-committee` | `domain` | `../govoplan-committee` | [govoplan-committee](https://git.add-ideas.de/GovOPlaN/govoplan-committee) |
| `govoplan-consultation` | `domain` | `../govoplan-consultation` | [govoplan-consultation](https://git.add-ideas.de/GovOPlaN/govoplan-consultation) |
| `govoplan-contracts` | `domain` | `../govoplan-contracts` | [govoplan-contracts](https://git.add-ideas.de/GovOPlaN/govoplan-contracts) |
| `govoplan-dashboard` | `platform` | `../govoplan-dashboard` | [govoplan-dashboard](https://git.add-ideas.de/GovOPlaN/govoplan-dashboard) |
| `govoplan-dataflow` | `platform` | `../govoplan-dataflow` | [govoplan-dataflow](https://git.add-ideas.de/GovOPlaN/govoplan-dataflow) |
| `govoplan-datasources` | `platform` | `../govoplan-datasources` | [govoplan-datasources](https://git.add-ideas.de/GovOPlaN/govoplan-datasources) |
| `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) |
| `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) |
| `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) |
| `govoplan-encryption` | `platform` | `../govoplan-encryption` | [govoplan-encryption](https://git.add-ideas.de/GovOPlaN/govoplan-encryption) |
| `govoplan-erp` | `domain` | `../govoplan-erp` | [govoplan-erp](https://git.add-ideas.de/GovOPlaN/govoplan-erp) |
| `govoplan-evaluation` | `domain` | `../govoplan-evaluation` | [govoplan-evaluation](https://git.add-ideas.de/GovOPlaN/govoplan-evaluation) |
| `govoplan-facilities` | `domain` | `../govoplan-facilities` | [govoplan-facilities](https://git.add-ideas.de/GovOPlaN/govoplan-facilities) |
| `govoplan-files` | `domain` | `../govoplan-files` | [govoplan-files](https://git.add-ideas.de/GovOPlaN/govoplan-files) |
| `govoplan-forms` | `domain` | `../govoplan-forms` | [govoplan-forms](https://git.add-ideas.de/GovOPlaN/govoplan-forms) |
| `govoplan-forms-runtime` | `platform` | `../govoplan-forms-runtime` | [govoplan-forms-runtime](https://git.add-ideas.de/GovOPlaN/govoplan-forms-runtime) |
| `govoplan-grants` | `domain` | `../govoplan-grants` | [govoplan-grants](https://git.add-ideas.de/GovOPlaN/govoplan-grants) |
| `govoplan-helpdesk` | `domain` | `../govoplan-helpdesk` | [govoplan-helpdesk](https://git.add-ideas.de/GovOPlaN/govoplan-helpdesk) |
| `govoplan-identity` | `platform` | `../govoplan-identity` | [govoplan-identity](https://git.add-ideas.de/GovOPlaN/govoplan-identity) |
| `govoplan-identity-trust` | `platform` | `../govoplan-identity-trust` | [govoplan-identity-trust](https://git.add-ideas.de/GovOPlaN/govoplan-identity-trust) |
| `govoplan-idm` | `platform` | `../govoplan-idm` | [govoplan-idm](https://git.add-ideas.de/GovOPlaN/govoplan-idm) |
| `govoplan-inspections` | `domain` | `../govoplan-inspections` | [govoplan-inspections](https://git.add-ideas.de/GovOPlaN/govoplan-inspections) |
| `govoplan-learning` | `domain` | `../govoplan-learning` | [govoplan-learning](https://git.add-ideas.de/GovOPlaN/govoplan-learning) |
| `govoplan-ledger` | `domain` | `../govoplan-ledger` | [govoplan-ledger](https://git.add-ideas.de/GovOPlaN/govoplan-ledger) |
| `govoplan-mail` | `domain` | `../govoplan-mail` | [govoplan-mail](https://git.add-ideas.de/GovOPlaN/govoplan-mail) |
| `govoplan-notifications` | `platform` | `../govoplan-notifications` | [govoplan-notifications](https://git.add-ideas.de/GovOPlaN/govoplan-notifications) |
| `govoplan-ops` | `platform` | `../govoplan-ops` | [govoplan-ops](https://git.add-ideas.de/GovOPlaN/govoplan-ops) |
| `govoplan-organizations` | `platform` | `../govoplan-organizations` | [govoplan-organizations](https://git.add-ideas.de/GovOPlaN/govoplan-organizations) |
| `govoplan-payments` | `domain` | `../govoplan-payments` | [govoplan-payments](https://git.add-ideas.de/GovOPlaN/govoplan-payments) |
| `govoplan-permits` | `domain` | `../govoplan-permits` | [govoplan-permits](https://git.add-ideas.de/GovOPlaN/govoplan-permits) |
| `govoplan-policy` | `platform` | `../govoplan-policy` | [govoplan-policy](https://git.add-ideas.de/GovOPlaN/govoplan-policy) |
| `govoplan-poll` | `domain` | `../govoplan-poll` | [govoplan-poll](https://git.add-ideas.de/GovOPlaN/govoplan-poll) |
| `govoplan-portal` | `domain` | `../govoplan-portal` | [govoplan-portal](https://git.add-ideas.de/GovOPlaN/govoplan-portal) |
| `govoplan-postbox` | `domain` | `../govoplan-postbox` | [govoplan-postbox](https://git.add-ideas.de/GovOPlaN/govoplan-postbox) |
| `govoplan-procurement` | `domain` | `../govoplan-procurement` | [govoplan-procurement](https://git.add-ideas.de/GovOPlaN/govoplan-procurement) |
| `govoplan-projects` | `domain` | `../govoplan-projects` | [govoplan-projects](https://git.add-ideas.de/GovOPlaN/govoplan-projects) |
| `govoplan-records` | `domain` | `../govoplan-records` | [govoplan-records](https://git.add-ideas.de/GovOPlaN/govoplan-records) |
| `govoplan-reporting` | `domain` | `../govoplan-reporting` | [govoplan-reporting](https://git.add-ideas.de/GovOPlaN/govoplan-reporting) |
| `govoplan-resources` | `domain` | `../govoplan-resources` | [govoplan-resources](https://git.add-ideas.de/GovOPlaN/govoplan-resources) |
| `govoplan-risk-compliance` | `domain` | `../govoplan-risk-compliance` | [govoplan-risk-compliance](https://git.add-ideas.de/GovOPlaN/govoplan-risk-compliance) |
| `govoplan-scheduling` | `domain` | `../govoplan-scheduling` | [govoplan-scheduling](https://git.add-ideas.de/GovOPlaN/govoplan-scheduling) |
| `govoplan-search` | `platform` | `../govoplan-search` | [govoplan-search](https://git.add-ideas.de/GovOPlaN/govoplan-search) |
| `govoplan-tasks` | `domain` | `../govoplan-tasks` | [govoplan-tasks](https://git.add-ideas.de/GovOPlaN/govoplan-tasks) |
| `govoplan-templates` | `domain` | `../govoplan-templates` | [govoplan-templates](https://git.add-ideas.de/GovOPlaN/govoplan-templates) |
| `govoplan-tenancy` | `platform` | `../govoplan-tenancy` | [govoplan-tenancy](https://git.add-ideas.de/GovOPlaN/govoplan-tenancy) |
| `govoplan-tickets` | `domain` | `../govoplan-tickets` | [govoplan-tickets](https://git.add-ideas.de/GovOPlaN/govoplan-tickets) |
| `govoplan-transparency` | `domain` | `../govoplan-transparency` | [govoplan-transparency](https://git.add-ideas.de/GovOPlaN/govoplan-transparency) |
| `govoplan-views` | `platform` | `../govoplan-views` | [govoplan-views](https://git.add-ideas.de/GovOPlaN/govoplan-views) |
| `govoplan-wiki` | `domain` | `../govoplan-wiki` | [govoplan-wiki](https://git.add-ideas.de/GovOPlaN/govoplan-wiki) |
| `govoplan-workflow` | `platform` | `../govoplan-workflow` | [govoplan-workflow](https://git.add-ideas.de/GovOPlaN/govoplan-workflow) |
## Connector
| Repository | Subtype | Local path | Gitea |
| --- | --- | --- | --- |
| `govoplan-connectors` | `connector-hub` | `../govoplan-connectors` | [govoplan-connectors](https://git.add-ideas.de/add-ideas/govoplan-connectors) |
| `govoplan-fit-connect` | `standard` | `../govoplan-fit-connect` | [govoplan-fit-connect](https://git.add-ideas.de/add-ideas/govoplan-fit-connect) |
| `govoplan-rest` | `protocol` | `../govoplan-rest` | [govoplan-rest](https://git.add-ideas.de/add-ideas/govoplan-rest) |
| `govoplan-soap` | `protocol` | `../govoplan-soap` | [govoplan-soap](https://git.add-ideas.de/add-ideas/govoplan-soap) |
| `govoplan-xoev` | `standard` | `../govoplan-xoev` | [govoplan-xoev](https://git.add-ideas.de/add-ideas/govoplan-xoev) |
| `govoplan-xrechnung` | `standard` | `../govoplan-xrechnung` | [govoplan-xrechnung](https://git.add-ideas.de/add-ideas/govoplan-xrechnung) |
| `govoplan-xta-osci` | `standard` | `../govoplan-xta-osci` | [govoplan-xta-osci](https://git.add-ideas.de/add-ideas/govoplan-xta-osci) |
| `govoplan-connectors` | `connector-hub` | `../govoplan-connectors` | [govoplan-connectors](https://git.add-ideas.de/GovOPlaN/govoplan-connectors) |
| `govoplan-fit-connect` | `standard` | `../govoplan-fit-connect` | [govoplan-fit-connect](https://git.add-ideas.de/GovOPlaN/govoplan-fit-connect) |
| `govoplan-rest` | `protocol` | `../govoplan-rest` | [govoplan-rest](https://git.add-ideas.de/GovOPlaN/govoplan-rest) |
| `govoplan-soap` | `protocol` | `../govoplan-soap` | [govoplan-soap](https://git.add-ideas.de/GovOPlaN/govoplan-soap) |
| `govoplan-xoev` | `standard` | `../govoplan-xoev` | [govoplan-xoev](https://git.add-ideas.de/GovOPlaN/govoplan-xoev) |
| `govoplan-xrechnung` | `standard` | `../govoplan-xrechnung` | [govoplan-xrechnung](https://git.add-ideas.de/GovOPlaN/govoplan-xrechnung) |
| `govoplan-xta-osci` | `standard` | `../govoplan-xta-osci` | [govoplan-xta-osci](https://git.add-ideas.de/GovOPlaN/govoplan-xta-osci) |
## Website
+156
View File
@@ -0,0 +1,156 @@
# Scaling And Multi-Host Deployment
## What Is Implemented
The default installer now supports explicit same-host horizontal scaling:
- one HAProxy container accepts the published HTTP endpoint;
- one or more WebUI containers serve assets and proxy API requests;
- one or more API containers serve normal domain traffic;
- one or more Celery workers consume shared Redis queues;
- one migration job runs before replacement;
- exactly one Celery scheduler runs;
- PostgreSQL, Redis, and file storage are shared by every runtime replica.
HAProxy discovers Compose replicas through Docker's internal DNS and performs
health-checked round-robin WebUI balancing and least-connection API balancing.
It does not mount the Docker socket. Replica counts live in
`installation.json`, so `configure`, `plan`, `apply`, and the receipt agree on
the desired topology.
This is not multi-host scheduling. Docker Compose's bridge network belongs to
one Docker Engine. Adding a second machine requires an orchestrator or
deployment manager that can place equivalent role definitions on multiple
hosts.
## Recommended Topologies
### One Host
Use the generated Compose bundle:
```text
client
-> external TLS proxy, when required
-> managed HAProxy
-> WebUI replica(s)
-> managed HAProxy API listener
-> API replica(s)
API/worker/scheduler
-> PostgreSQL
-> Redis
-> local volume, managed Garage, or external S3
```
This improves concurrency and permits rolling process replacement, but the host
and every managed stateful component remain single failure domains. Two API
containers on one failed host do not provide host-level availability.
### Multiple Hosts
Use Kubernetes, Nomad, Docker Swarm, or another reviewed orchestrator. Do not
extend the Compose installer into a proprietary scheduler. The target topology
is:
```text
public TLS ingress/load balancer
-> WebUI replicas on at least two nodes
-> internal API service/load balancer
-> API replicas on at least two nodes
queue-specific worker pools on worker nodes
one fenced scheduler
one fenced migration/deployment job
shared PostgreSQL
shared Redis
multi-node Garage or external S3
shared secret/config provider
central logs, metrics, and health alerts
```
All API and worker nodes must receive the same immutable software composition,
`MASTER_KEY_B64`, database URL, Redis URL, enabled-module graph, and object
storage binding. Node-local file storage is not valid in this topology.
## Adding Capacity
### API And WebUI
Increase API replicas for measured request concurrency, CPU saturation, or
latency after checking database load. Increase WebUI replicas for static asset
and proxy capacity. No sticky session should be required because durable
sessions and throttling use shared services, but this must remain covered by
multi-replica integration tests.
Every added API process also adds database connections. Set the application
pool size and the total replica ceiling against PostgreSQL's connection budget;
adding replicas can otherwise reduce throughput.
### Workers
Workers are not placed behind a load balancer. They compete for jobs on shared
Redis queues. Add workers by queue and cap each pool according to the external
system it calls. SMTP, IMAP, directory, and connector jobs often reach provider
rate limits before CPU limits.
Use distinct pools when load warrants it:
- mail send and Sent-folder append;
- notifications;
- calendar and connector synchronization;
- dataflow/workflow execution;
- reporting and export;
- platform events and default work.
Before stopping a worker, mark it draining, stop new claims, and let or safely
requeue active jobs. Worker registration and drain control remain part of the
larger platform scale-out story.
### Stateful Services
- PostgreSQL needs backups, restore drills, connection limits, and an HA/failover
design appropriate to the service level.
- Redis needs persistence and, for high availability, a supported failover
topology. Queue loss is not equivalent to a harmless cache loss.
- Managed Garage from the Compose installer is single-node. For redundant
storage, deploy a multi-node Garage cluster separately and select external
`s3`, or use another compatible object store.
- The scheduler stays at one replica until distributed leader election or a
fenced lease is implemented.
- Migrations and module lifecycle mutations remain one-at-a-time operations.
## Promotion Path
1. Measure concurrent users, request latency, database query time, connection
use, queue age, worker saturation, storage latency, and external throttling.
2. Move files to managed Garage or external S3 before introducing independent
runtime hosts.
3. Keep PostgreSQL and Redis external to stateless runtime nodes, or deploy
their reviewed HA operators.
4. Publish immutable API/WebUI images and one versioned configuration/secret
contract.
5. Translate the generated role commands, environment allowlists, health
checks, and singleton constraints into the chosen orchestrator.
6. Put API and WebUI replicas behind health-aware services and a TLS ingress.
7. Add queue-specific workers with fixed upper bounds.
8. Prove replica loss, rolling replacement, job redelivery, session continuity,
migration exclusion, backup restore, and storage-node loss before claiming
high availability.
## Remaining Platform Work
The one-host installer does not yet provide:
- a Kubernetes, Nomad, or Swarm deployment export/profile;
- distributed deployment locks and fenced scheduler leadership;
- worker registration, composition-skew reporting, and drain controls;
- managed PostgreSQL, Redis, or multi-node Garage high availability;
- autoscaling policy;
- managed TLS certificate issuance and renewal;
- measured multi-replica and failover integration evidence.
These are separate production capabilities. The local load balancer and
replica model provide the contract they should implement, but they do not by
themselves make a cluster highly available.
+21 -4
View File
@@ -50,6 +50,10 @@ accept findings, but scanner execution errors and malformed JSON/SARIF reports
always fail the run. The manifest lists the expected, present, and missing
reports for that invocation. Validation and checksums use that explicit set, so
reusing a report directory cannot make stale output look like part of a new run.
It also contains `coverage_status` and structured `scanner_coverage` entries.
Every required scanner is recorded as `no-findings`, `findings`,
`scanner-failure`, or `skipped`; this makes an incomplete local run visible
without treating it as a clean audit.
The wrapper tags the toolbox image by a fingerprint of the Dockerfile,
`requirements-audit.txt`, and the Semgrep smoke-test inputs. If those inputs have
@@ -90,6 +94,12 @@ tools/checks/security-audit/run.sh --mode quick --scope current --update --build
- `ci`: quick plus Semgrep public registry rulesets, Trivy, pip-audit, npm audit.
- `full`: ci plus OSV-Scanner, jscpd, Radon, and Xenon.
The Gitea workflow uses `full` mode so its coverage contract includes every
scanner above. Missing scanners fail strict runs and all Actions runs, even
while actual findings remain report-only. Local report-only runs may finish
with missing tools for diagnostics, but their manifest is marked
`coverage_status: incomplete`.
Semgrep and Trivy are invoked with finding-sensitive exit codes. Their exit 1
is therefore a finding under the wrapper contract; higher exit codes, missing
output, invalid JSON/SARIF, and scanner error payloads are execution failures.
@@ -103,7 +113,7 @@ baseline.
## Gating
The initial Gitea workflow runs in report-only mode:
The Gitea workflow currently runs findings in report-only mode:
```bash
SECURITY_AUDIT_FAIL_ON_FINDINGS=0
@@ -119,13 +129,13 @@ SECURITY_AUDIT_FAIL_ON_FINDINGS=1
or run locally with:
```bash
tools/checks/security-audit/run.sh --mode ci --scope current --strict
tools/checks/security-audit/run.sh --mode full --scope govoplan --strict
```
## Audit Burndown Workflow
Treat Gitea issues as the active audit state. A full GovOPlaN audit should
produce one tracker issue in `add-ideas/govoplan` and child issues in the
produce one tracker issue in `GovOPlaN/govoplan` and child issues in the
repository that owns each fix.
Use the tracker issue for:
@@ -162,7 +172,14 @@ clusters that cross module ownership or make behavior harder to change safely.
The regular `Security Audit` workflow reuses the fingerprinted toolbox image
when the Docker daemon is persistent, which is the normal case for the
self-hosted Gitea runner using the host Docker socket. The separate
self-hosted Gitea runner using the host Docker socket. Trusted push, schedule,
and manual runs scan all registered repositories; authenticated SSH is used
only for the private website repository. The wrapper inspects the Actions job
mount table and forwards only the narrowest writable mount covering the audit
scope; it never inherits the job's Docker socket or unrelated runner mounts.
Pull-request audit runs stay disabled while the audit runner exposes its host
Docker socket: PR-controlled audit code must run on a disposable or rootless
runner without host-socket access. The separate
`Security Audit Toolbox Update` workflow runs weekly with
`SECURITY_AUDIT_UPDATE=1`; it pulls current base images and re-resolves the
allowed tool version ranges into a refreshed local image.
@@ -19,7 +19,7 @@ This is a product-level story owned by the GovOPlaN platform rather than by an
individual domain module. It joins installation, module lifecycle, operations,
configuration packages, and release provenance into one administrator journey.
The canonical backlog item is
[GovOPlaN #13](https://git.add-ideas.de/add-ideas/govoplan/issues/13).
[GovOPlaN #13](https://git.add-ideas.de/GovOPlaN/govoplan/issues/13).
## Terms
+137
View File
@@ -0,0 +1,137 @@
# GovOPlaN Views Architecture
## Purpose
GovOPlaN Views are governed presentation projections for a task,
responsibility, or workflow step. A View can reduce the visible modules,
navigation entries, routes, page sections, and commands to the interface
needed for the current job.
Views are optional. If `govoplan-views` is not installed or enabled, the normal
permission-derived interface remains unchanged.
## Security Boundary
A View is not an authorization mechanism.
- Access, tenant isolation, resource guards, and backend permission checks
remain authoritative.
- A View may hide an interface surface that the actor is otherwise allowed to
use.
- A View can never expose a route, action, tenant, or resource that normal
authorization denies.
- An authorized deep link outside the current View should offer an explicit
temporary escape or View switch. It must not be presented as a permission
denial.
This boundary lets Views improve focus without creating a second, weaker RBAC
system.
## Ownership
Core owns the versioned, module-neutral surface contract and WebUI runtime
hooks. Modules declare stable surfaces and use shared hooks to respect the
effective projection. Modules do not import `govoplan-views`.
`govoplan-views` owns:
- draft and immutable published View revisions
- system, tenant, group, and user assignments
- default, mandatory, and user-selectable Views
- active per-user View state
- effective projection resolution and provenance
- the View editor, preview, validation, and stale-surface diagnostics
Policy optionally owns inherited ceilings and explainable decisions. Workflow
optionally references a pinned View revision for an instance or step and may
narrow it further.
## Surface Contract
Modules announce only useful, semantic surfaces:
- module
- navigation item
- route or workspace
- section or panel
- command or action
Each descriptor has a stable namespaced id, parent id, kind, label, default
visibility, ordering, and dependency metadata where needed. Surface ids are
public module contracts, not CSS selectors, component paths, or arbitrary DOM
fragments.
The first release supports visible or hidden. Read-only states, layout
replacement, visual emphasis, and arbitrary styling are separate concerns and
are deferred.
## Effective Resolution
The effective interface is the intersection of:
1. installed and enabled modules
2. actor permissions and resource access
3. administrator and Policy ceilings
4. an assigned or user-selected View
5. an optional workflow instance or step overlay
Lower scopes and workflow overlays may narrow inherited visibility but cannot
broaden it. Every inherited, locked, hidden, unavailable, or stale choice
should carry provenance that the editor and runtime can explain.
Published View revisions are immutable. Active workflow instances pin the
revision they use. Unknown or retired surface ids produce diagnostics rather
than breaking startup. If no valid effective View can be resolved, the system
uses the last valid projection or the normal authorized interface and reports
the configuration problem to administrators.
## Workflow Behavior
A workflow definition may reference a View for the whole instance or a
particular step. Starting, resuming, or advancing the workflow activates the
appropriate projection. Users can intentionally leave focused mode and return
from an open-work widget or notification without losing workflow state.
Module handoffs carry the workflow and View context through Core contracts.
Workflow does not import the target module or the Views implementation.
## Delivery Order
1. Define the Core surface registry and runtime hooks.
2. Initialize `govoplan-views` and persist versioned definitions.
3. Add assignment, selection, resolution, provenance, and the editor.
4. Add Policy inheritance and administrator ceilings.
5. Add Workflow instance and step activation.
6. Adopt semantic section/action descriptors module by module.
## Implementation Status
Implemented in the initial Views slice:
- Core contract version `1`, stable module/navigation/route identifiers, custom
section/action descriptors, manifest validation, and platform API metadata
- shell navigation, route-boundary, settings, administration, dashboard-widget,
embedded-capability, and organization-action filtering
- `govoplan-views` definitions, immutable revisions, system/tenant/group/user
assignments, user selection, provenance, and stale-surface recovery
- a system and tenant administration editor with unsaved-change protection,
publish/archive controls, assignment management, and server-enforced lockout
prevention
- surface declarations for every currently installed module that contributes a
WebUI, including finer-grained shared administration and settings surfaces
Still intentionally separate:
- Policy-owned inherited ceilings and policy decision provenance
- workflow-instance and workflow-step activation of pinned View revisions
- read-only and layout-replacement projections beyond the version `1`
visible/hidden contract
## Gitea Work Packages
- `govoplan#17`: task-focused Views user story
- `govoplan#16`: initialize and implement `govoplan-views`
- `govoplan-core#271`: versioned surface and runtime contracts
- `govoplan-policy#9`: inheritance, ceilings, and provenance
- `govoplan-workflow#7`: workflow instance and step activation
- `govoplan-workflow#3`: focused workflow mode user story
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/capability-fit-boundary-evidence.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/capability-fit-boundary-evidence.schema.json",
"title": "GovOPlaN externally issued capability-fit boundary evidence",
"type": "object",
"additionalProperties": false,
+2 -2
View File
@@ -507,7 +507,7 @@
{
"kind": "issue",
"scope": "documented_model",
"locator": "https://git.add-ideas.de/add-ideas/govoplan/issues/12"
"locator": "https://git.add-ideas.de/GovOPlaN/govoplan/issues/12"
}
],
"conditions": [],
@@ -793,7 +793,7 @@
{
"kind": "issue",
"scope": "documented_model",
"locator": "https://git.add-ideas.de/add-ideas/govoplan-core/issues/29"
"locator": "https://git.add-ideas.de/GovOPlaN/govoplan-core/issues/29"
}
],
"conditions": [],
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/capability-fit-proof-authority-keyring.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/capability-fit-proof-authority-keyring.schema.json",
"title": "GovOPlaN capability-fit proof authority keyring",
"type": "object",
"additionalProperties": false,
+1 -1
View File
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/capability-fit.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/capability-fit.schema.json",
"title": "GovOPlaN capability and infrastructure fit assessment",
"type": "object",
"additionalProperties": false,
+66
View File
@@ -149,12 +149,36 @@
"description": "GovOPlaN core runner, shared primitives, shell, or extension points.",
"exclusive": false
},
{
"name": "module/dashboard",
"color": "1d76db",
"description": "GovOPlaN Dashboard module behavior or integration.",
"exclusive": false
},
{
"name": "module/dataflow",
"color": "1d76db",
"description": "GovOPlaN Dataflow module behavior or integration.",
"exclusive": false
},
{
"name": "module/datasources",
"color": "006b75",
"description": "GovOPlaN governed datasource contracts, catalogs, and integrations.",
"exclusive": false
},
{
"name": "module/dms",
"color": "c5def5",
"description": "GovOPlaN Dms module behavior or integration.",
"exclusive": false
},
{
"name": "module/docs",
"color": "c5def5",
"description": "GovOPlaN Docs module behavior or integration.",
"exclusive": false
},
{
"name": "module/dist-lists",
"color": "0e8a16",
@@ -167,6 +191,12 @@
"description": "GovOPlaN Erp module behavior or integration.",
"exclusive": false
},
{
"name": "module/encryption",
"color": "b60205",
"description": "GovOPlaN Encryption key custody, cryptographic policy, and E2EE integration.",
"exclusive": false
},
{
"name": "module/evaluation",
"color": "bfdadc",
@@ -191,6 +221,12 @@
"description": "GovOPlaN Forms module behavior or integration.",
"exclusive": false
},
{
"name": "module/helpdesk",
"color": "c2e0c6",
"description": "GovOPlaN Helpdesk module behavior or integration.",
"exclusive": false
},
{
"name": "module/identity-trust",
"color": "d4c5f9",
@@ -275,12 +311,24 @@
"description": "GovOPlaN Postbox module behavior or integration.",
"exclusive": false
},
{
"name": "module/projects",
"color": "5319e7",
"description": "GovOPlaN Projects module behavior or integration.",
"exclusive": false
},
{
"name": "module/reporting",
"color": "c2e0c6",
"description": "GovOPlaN Reporting module behavior or integration.",
"exclusive": false
},
{
"name": "module/risk-compliance",
"color": "b60205",
"description": "GovOPlaN Risk Compliance module behavior or integration.",
"exclusive": false
},
{
"name": "module/search",
"color": "bfdadc",
@@ -311,6 +359,24 @@
"description": "GovOPlaN Tenancy module behavior or integration.",
"exclusive": false
},
{
"name": "module/tickets",
"color": "0e8a16",
"description": "GovOPlaN Tickets module behavior or integration.",
"exclusive": false
},
{
"name": "module/views",
"color": "c5def5",
"description": "GovOPlaN governed task views, interface projections, and workflow view integration.",
"exclusive": false
},
{
"name": "module/wiki",
"color": "006b75",
"description": "GovOPlaN Wiki module behavior or integration.",
"exclusive": false
},
{
"name": "module/workflow",
"color": "f9d0c4",
+320
View File
@@ -0,0 +1,320 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://govoplan.add-ideas.de/schemas/installation-spec-v1.json",
"title": "GovOPlaN installation specification",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"installation_id",
"profile",
"public_url",
"listen",
"network_subnet",
"release",
"components",
"enabled_modules"
],
"properties": {
"schema_version": {
"const": 1
},
"installation_id": {
"type": "string",
"pattern": "^[a-z][a-z0-9-]{1,47}$"
},
"profile": {
"enum": [
"evaluation",
"self-hosted"
]
},
"public_url": {
"type": "string",
"format": "uri",
"pattern": "^https?://"
},
"listen": {
"type": "object",
"additionalProperties": false,
"required": [
"address",
"port"
],
"properties": {
"address": {
"type": "string"
},
"port": {
"type": "integer",
"minimum": 1,
"maximum": 65535
}
}
},
"network_subnet": {
"type": "string"
},
"release": {
"type": "object",
"additionalProperties": false,
"required": [
"channel",
"version",
"manifest_url",
"manifest_sha256",
"api_image",
"web_image"
],
"properties": {
"channel": {
"type": "string",
"pattern": "^[a-z][a-z0-9-]{1,31}$"
},
"version": {
"type": "string",
"minLength": 1,
"maxLength": 80
},
"manifest_url": {
"type": "string"
},
"manifest_sha256": {
"type": "string",
"pattern": "^$|^[0-9a-f]{64}$"
},
"api_image": {
"type": "string",
"minLength": 1,
"maxLength": 300
},
"web_image": {
"type": "string",
"minLength": 1,
"maxLength": 300
}
}
},
"components": {
"type": "object",
"additionalProperties": false,
"required": [
"postgres",
"redis",
"mail",
"storage"
],
"properties": {
"postgres": {
"$ref": "#/$defs/postgres"
},
"redis": {
"$ref": "#/$defs/redis"
},
"mail": {
"$ref": "#/$defs/mail"
},
"storage": {
"type": "object",
"additionalProperties": false,
"required": [
"mode"
],
"properties": {
"mode": {
"enum": [
"local",
"garage",
"s3"
]
},
"image": {
"type": "string",
"maxLength": 300
}
}
},
"load_balancer": {
"$ref": "#/$defs/load_balancer"
}
}
},
"replicas": {
"type": "object",
"additionalProperties": false,
"required": [
"api",
"web",
"worker"
],
"properties": {
"api": {
"type": "integer",
"minimum": 1,
"maximum": 64
},
"web": {
"type": "integer",
"minimum": 1,
"maximum": 64
},
"worker": {
"type": "integer",
"minimum": 0,
"maximum": 128
}
}
},
"enabled_modules": {
"type": "array",
"uniqueItems": true,
"items": {
"type": "string",
"pattern": "^[a-z][a-z0-9_]{1,63}$"
}
}
},
"$defs": {
"service": {
"type": "object",
"additionalProperties": false,
"required": [
"mode",
"image",
"url_env"
],
"properties": {
"mode": {
"type": "string"
},
"image": {
"type": "string"
},
"url_env": {
"type": "string"
}
}
},
"postgres": {
"allOf": [
{
"$ref": "#/$defs/service"
},
{
"properties": {
"mode": {
"enum": [
"managed",
"external"
]
},
"url_env": {
"const": "DATABASE_URL"
}
}
}
]
},
"redis": {
"allOf": [
{
"$ref": "#/$defs/service"
},
{
"properties": {
"mode": {
"enum": [
"managed",
"external",
"disabled"
]
},
"url_env": {
"const": "REDIS_URL"
}
}
}
]
},
"mail": {
"allOf": [
{
"$ref": "#/$defs/service"
},
{
"properties": {
"mode": {
"enum": [
"disabled",
"external-relay",
"test-mail"
]
},
"url_env": {
"const": ""
}
}
}
]
},
"load_balancer": {
"allOf": [
{
"$ref": "#/$defs/service"
},
{
"properties": {
"mode": {
"const": "managed"
},
"url_env": {
"const": ""
}
}
}
]
}
},
"allOf": [
{
"if": {
"properties": {
"profile": {
"const": "self-hosted"
}
}
},
"then": {
"properties": {
"public_url": {
"pattern": "^https://"
},
"components": {
"properties": {
"redis": {
"properties": {
"mode": {
"enum": [
"managed",
"external"
]
}
}
},
"mail": {
"properties": {
"mode": {
"enum": [
"disabled",
"external-relay"
]
}
}
}
}
}
}
}
}
]
}
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/installed-composition-evidence.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/installed-composition-evidence.schema.json",
"title": "GovOPlaN installed composition evidence",
"type": "object",
"additionalProperties": false,
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/installer-receipt-authority-keyring.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/installer-receipt-authority-keyring.schema.json",
"title": "GovOPlaN installer receipt authority keyring",
"type": "object",
"additionalProperties": false,
+1 -1
View File
@@ -1,6 +1,6 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://git.add-ideas.de/add-ideas/govoplan/src/branch/main/docs/installer-receipt.schema.json",
"$id": "https://git.add-ideas.de/GovOPlaN/govoplan/src/branch/main/docs/installer-receipt.schema.json",
"title": "GovOPlaN signed installer receipt",
"type": "object",
"additionalProperties": false,
+73 -67
View File
@@ -1,73 +1,79 @@
{
"version": 1,
"organization": "add-ideas",
"organization": "GovOPlaN",
"default_parent": "/mnt/DATA/git",
"repositories": [
{"name": "govoplan", "category": "system", "subtype": "meta", "remote": "git@git.add-ideas.de:add-ideas/govoplan.git", "path": "govoplan"},
{"name": "govoplan-core", "category": "system", "subtype": "kernel", "remote": "git@git.add-ideas.de:add-ideas/govoplan-core.git", "path": "govoplan-core"},
{"name": "govoplan-access", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-access.git", "path": "govoplan-access"},
{"name": "govoplan-addresses", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-addresses.git", "path": "govoplan-addresses"},
{"name": "govoplan-admin", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-admin.git", "path": "govoplan-admin"},
{"name": "govoplan-appointments", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-appointments.git", "path": "govoplan-appointments"},
{"name": "govoplan-approvals", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-approvals.git", "path": "govoplan-approvals"},
{"name": "govoplan-assets", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-assets.git", "path": "govoplan-assets"},
{"name": "govoplan-audit", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-audit.git", "path": "govoplan-audit"},
{"name": "govoplan-booking", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-booking.git", "path": "govoplan-booking"},
{"name": "govoplan-calendar", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-calendar.git", "path": "govoplan-calendar"},
{"name": "govoplan-campaign", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-campaign.git", "path": "govoplan-campaign"},
{"name": "govoplan-cases", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-cases.git", "path": "govoplan-cases"},
{"name": "govoplan-certificates", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-certificates.git", "path": "govoplan-certificates"},
{"name": "govoplan-committee", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-committee.git", "path": "govoplan-committee"},
{"name": "govoplan-connectors", "category": "connector", "subtype": "connector-hub", "remote": "git@git.add-ideas.de:add-ideas/govoplan-connectors.git", "path": "govoplan-connectors"},
{"name": "govoplan-consultation", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-consultation.git", "path": "govoplan-consultation"},
{"name": "govoplan-contracts", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-contracts.git", "path": "govoplan-contracts"},
{"name": "govoplan-dashboard", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-dashboard.git", "path": "govoplan-dashboard"},
{"name": "govoplan-dms", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-dms.git", "path": "govoplan-dms"},
{"name": "govoplan-dist-lists", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-dist-lists.git", "path": "govoplan-dist-lists"},
{"name": "govoplan-docs", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-docs.git", "path": "govoplan-docs"},
{"name": "govoplan-erp", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-erp.git", "path": "govoplan-erp"},
{"name": "govoplan-evaluation", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-evaluation.git", "path": "govoplan-evaluation"},
{"name": "govoplan-facilities", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-facilities.git", "path": "govoplan-facilities"},
{"name": "govoplan-files", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-files.git", "path": "govoplan-files"},
{"name": "govoplan-fit-connect", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:add-ideas/govoplan-fit-connect.git", "path": "govoplan-fit-connect"},
{"name": "govoplan-forms", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-forms.git", "path": "govoplan-forms"},
{"name": "govoplan-forms-runtime", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-forms-runtime.git", "path": "govoplan-forms-runtime"},
{"name": "govoplan-grants", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-grants.git", "path": "govoplan-grants"},
{"name": "govoplan-helpdesk", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-helpdesk.git", "path": "govoplan-helpdesk"},
{"name": "govoplan-identity", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-identity.git", "path": "govoplan-identity"},
{"name": "govoplan-identity-trust", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-identity-trust.git", "path": "govoplan-identity-trust"},
{"name": "govoplan-idm", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-idm.git", "path": "govoplan-idm"},
{"name": "govoplan-inspections", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-inspections.git", "path": "govoplan-inspections"},
{"name": "govoplan-issue-reporting", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-issue-reporting.git", "path": "govoplan-issue-reporting"},
{"name": "govoplan-learning", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-learning.git", "path": "govoplan-learning"},
{"name": "govoplan-ledger", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-ledger.git", "path": "govoplan-ledger"},
{"name": "govoplan-mail", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-mail.git", "path": "govoplan-mail"},
{"name": "govoplan-notifications", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-notifications.git", "path": "govoplan-notifications"},
{"name": "govoplan-ops", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-ops.git", "path": "govoplan-ops"},
{"name": "govoplan-organizations", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-organizations.git", "path": "govoplan-organizations"},
{"name": "govoplan-payments", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-payments.git", "path": "govoplan-payments"},
{"name": "govoplan-permits", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-permits.git", "path": "govoplan-permits"},
{"name": "govoplan-policy", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-policy.git", "path": "govoplan-policy"},
{"name": "govoplan-poll", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-poll.git", "path": "govoplan-poll"},
{"name": "govoplan-portal", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-portal.git", "path": "govoplan-portal"},
{"name": "govoplan-postbox", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-postbox.git", "path": "govoplan-postbox"},
{"name": "govoplan-procurement", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-procurement.git", "path": "govoplan-procurement"},
{"name": "govoplan-records", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-records.git", "path": "govoplan-records"},
{"name": "govoplan-reporting", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-reporting.git", "path": "govoplan-reporting"},
{"name": "govoplan-resources", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-resources.git", "path": "govoplan-resources"},
{"name": "govoplan-rest", "category": "connector", "subtype": "protocol", "remote": "git@git.add-ideas.de:add-ideas/govoplan-rest.git", "path": "govoplan-rest"},
{"name": "govoplan-risk-compliance", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-risk-compliance.git", "path": "govoplan-risk-compliance"},
{"name": "govoplan-scheduling", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-scheduling.git", "path": "govoplan-scheduling"},
{"name": "govoplan-search", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-search.git", "path": "govoplan-search"},
{"name": "govoplan-soap", "category": "connector", "subtype": "protocol", "remote": "git@git.add-ideas.de:add-ideas/govoplan-soap.git", "path": "govoplan-soap"},
{"name": "govoplan-tasks", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-tasks.git", "path": "govoplan-tasks"},
{"name": "govoplan-templates", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-templates.git", "path": "govoplan-templates"},
{"name": "govoplan-tenancy", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-tenancy.git", "path": "govoplan-tenancy"},
{"name": "govoplan-transparency", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:add-ideas/govoplan-transparency.git", "path": "govoplan-transparency"},
{"name": "addideas-govoplan-website", "category": "website", "subtype": "public-site", "remote": "git@git.add-ideas.de:add-ideas/addideas-govoplan-website.git", "path": "addideas-govoplan-website"},
{"name": "govoplan-workflow", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:add-ideas/govoplan-workflow.git", "path": "govoplan-workflow"},
{"name": "govoplan-xoev", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:add-ideas/govoplan-xoev.git", "path": "govoplan-xoev"},
{"name": "govoplan-xrechnung", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:add-ideas/govoplan-xrechnung.git", "path": "govoplan-xrechnung"},
{"name": "govoplan-xta-osci", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:add-ideas/govoplan-xta-osci.git", "path": "govoplan-xta-osci"}
{"name": "govoplan", "category": "system", "subtype": "meta", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan.git", "path": "govoplan"},
{"name": "govoplan-core", "category": "system", "subtype": "kernel", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-core.git", "path": "govoplan-core"},
{"name": "govoplan-access", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-access.git", "path": "govoplan-access"},
{"name": "govoplan-addresses", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-addresses.git", "path": "govoplan-addresses"},
{"name": "govoplan-admin", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-admin.git", "path": "govoplan-admin"},
{"name": "govoplan-appointments", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-appointments.git", "path": "govoplan-appointments"},
{"name": "govoplan-approvals", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-approvals.git", "path": "govoplan-approvals"},
{"name": "govoplan-assets", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-assets.git", "path": "govoplan-assets"},
{"name": "govoplan-audit", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-audit.git", "path": "govoplan-audit"},
{"name": "govoplan-booking", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-booking.git", "path": "govoplan-booking"},
{"name": "govoplan-calendar", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-calendar.git", "path": "govoplan-calendar"},
{"name": "govoplan-campaign", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-campaign.git", "path": "govoplan-campaign"},
{"name": "govoplan-cases", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-cases.git", "path": "govoplan-cases"},
{"name": "govoplan-certificates", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-certificates.git", "path": "govoplan-certificates"},
{"name": "govoplan-committee", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-committee.git", "path": "govoplan-committee"},
{"name": "govoplan-connectors", "category": "connector", "subtype": "connector-hub", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-connectors.git", "path": "govoplan-connectors"},
{"name": "govoplan-consultation", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-consultation.git", "path": "govoplan-consultation"},
{"name": "govoplan-contracts", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-contracts.git", "path": "govoplan-contracts"},
{"name": "govoplan-dashboard", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dashboard.git", "path": "govoplan-dashboard"},
{"name": "govoplan-dataflow", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dataflow.git", "path": "govoplan-dataflow"},
{"name": "govoplan-datasources", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-datasources.git", "path": "govoplan-datasources"},
{"name": "govoplan-dms", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dms.git", "path": "govoplan-dms"},
{"name": "govoplan-dist-lists", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dist-lists.git", "path": "govoplan-dist-lists"},
{"name": "govoplan-docs", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-docs.git", "path": "govoplan-docs"},
{"name": "govoplan-encryption", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-encryption.git", "path": "govoplan-encryption"},
{"name": "govoplan-erp", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-erp.git", "path": "govoplan-erp"},
{"name": "govoplan-evaluation", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-evaluation.git", "path": "govoplan-evaluation"},
{"name": "govoplan-facilities", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-facilities.git", "path": "govoplan-facilities"},
{"name": "govoplan-files", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-files.git", "path": "govoplan-files"},
{"name": "govoplan-fit-connect", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-fit-connect.git", "path": "govoplan-fit-connect"},
{"name": "govoplan-forms", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-forms.git", "path": "govoplan-forms"},
{"name": "govoplan-forms-runtime", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-forms-runtime.git", "path": "govoplan-forms-runtime"},
{"name": "govoplan-grants", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-grants.git", "path": "govoplan-grants"},
{"name": "govoplan-helpdesk", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-helpdesk.git", "path": "govoplan-helpdesk"},
{"name": "govoplan-identity", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-identity.git", "path": "govoplan-identity"},
{"name": "govoplan-identity-trust", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-identity-trust.git", "path": "govoplan-identity-trust"},
{"name": "govoplan-idm", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-idm.git", "path": "govoplan-idm"},
{"name": "govoplan-inspections", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-inspections.git", "path": "govoplan-inspections"},
{"name": "govoplan-learning", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-learning.git", "path": "govoplan-learning"},
{"name": "govoplan-ledger", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-ledger.git", "path": "govoplan-ledger"},
{"name": "govoplan-mail", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-mail.git", "path": "govoplan-mail"},
{"name": "govoplan-notifications", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-notifications.git", "path": "govoplan-notifications"},
{"name": "govoplan-ops", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-ops.git", "path": "govoplan-ops"},
{"name": "govoplan-organizations", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-organizations.git", "path": "govoplan-organizations"},
{"name": "govoplan-payments", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-payments.git", "path": "govoplan-payments"},
{"name": "govoplan-permits", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-permits.git", "path": "govoplan-permits"},
{"name": "govoplan-policy", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-policy.git", "path": "govoplan-policy"},
{"name": "govoplan-poll", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-poll.git", "path": "govoplan-poll"},
{"name": "govoplan-portal", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-portal.git", "path": "govoplan-portal"},
{"name": "govoplan-postbox", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-postbox.git", "path": "govoplan-postbox"},
{"name": "govoplan-procurement", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-procurement.git", "path": "govoplan-procurement"},
{"name": "govoplan-projects", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-projects.git", "path": "govoplan-projects"},
{"name": "govoplan-records", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-records.git", "path": "govoplan-records"},
{"name": "govoplan-reporting", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-reporting.git", "path": "govoplan-reporting"},
{"name": "govoplan-resources", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-resources.git", "path": "govoplan-resources"},
{"name": "govoplan-rest", "category": "connector", "subtype": "protocol", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-rest.git", "path": "govoplan-rest"},
{"name": "govoplan-risk-compliance", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-risk-compliance.git", "path": "govoplan-risk-compliance"},
{"name": "govoplan-scheduling", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-scheduling.git", "path": "govoplan-scheduling"},
{"name": "govoplan-search", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-search.git", "path": "govoplan-search"},
{"name": "govoplan-soap", "category": "connector", "subtype": "protocol", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-soap.git", "path": "govoplan-soap"},
{"name": "govoplan-tasks", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-tasks.git", "path": "govoplan-tasks"},
{"name": "govoplan-templates", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-templates.git", "path": "govoplan-templates"},
{"name": "govoplan-tenancy", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-tenancy.git", "path": "govoplan-tenancy"},
{"name": "govoplan-tickets", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-tickets.git", "path": "govoplan-tickets"},
{"name": "govoplan-transparency", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-transparency.git", "path": "govoplan-transparency"},
{"name": "govoplan-views", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-views.git", "path": "govoplan-views"},
{"name": "govoplan-wiki", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-wiki.git", "path": "govoplan-wiki"},
{"name": "addideas-govoplan-website", "category": "website", "subtype": "public-site", "remote": "git@git.add-ideas.de:add-ideas/addideas-govoplan-website.git", "path": "addideas-govoplan-website", "bootstrap_transport": "registered"},
{"name": "govoplan-workflow", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-workflow.git", "path": "govoplan-workflow"},
{"name": "govoplan-xoev", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-xoev.git", "path": "govoplan-xoev"},
{"name": "govoplan-xrechnung", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-xrechnung.git", "path": "govoplan-xrechnung"},
{"name": "govoplan-xta-osci", "category": "connector", "subtype": "standard", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-xta-osci.git", "path": "govoplan-xta-osci"}
]
}
+11
View File
@@ -19,11 +19,20 @@
-e ../govoplan-mail
-e ../govoplan-campaign
-e ../govoplan-calendar
-e ../govoplan-connectors
-e ../govoplan-datasources
-e ../govoplan-dataflow
-e ../govoplan-workflow
-e ../govoplan-views
-e ../govoplan-search
-e ../govoplan-risk-compliance
-e ../govoplan-postbox
-e ../govoplan-poll
-e ../govoplan-scheduling
-e ../govoplan-notifications
-e ../govoplan-evaluation
-e ../govoplan-docs
-e ../govoplan-encryption
-e ../govoplan-ops
httpx==0.28.1
httpx2>=2.5,<3
@@ -32,5 +41,7 @@ idna>=3.15
jsonschema>=4,<5
pip>=26.1.2
pip-audit>=2.9,<3
pytest>=9.0.3,<10
pygments>=2.20,<3
python-multipart>=0.0.31
ruff>=0.14,<1
+5
View File
@@ -0,0 +1,5 @@
# Test-harness dependencies used against immutable release source tags.
# Keep these separate from requirements-release.txt so they are not part of the
# deployable product dependency set.
pytest>=9.0.3,<10
pygments>=2.20,<3
+15 -15
View File
@@ -1,18 +1,18 @@
# Whole-product release install from immutable, independently versioned module tags.
# Only add a module after its referenced tag has been published.
../govoplan-core[server]
govoplan-tenancy @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-tenancy.git@v0.1.8
govoplan-organizations @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-organizations.git@v0.1.8
govoplan-identity @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-identity.git@v0.1.8
govoplan-idm @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-idm.git@v0.1.8
govoplan-access @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-access.git@v0.1.8
govoplan-admin @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-admin.git@v0.1.8
govoplan-policy @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-policy.git@v0.1.8
govoplan-audit @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-audit.git@v0.1.8
govoplan-dashboard @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-dashboard.git@v0.1.8
govoplan-files @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-files.git@v0.1.8
govoplan-mail @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-mail.git@v0.1.8
govoplan-campaign @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-campaign.git@v0.1.11
govoplan-calendar @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-calendar.git@v0.1.8
govoplan-docs @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-docs.git@v0.1.8
govoplan-ops @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-ops.git@v0.1.8
govoplan-tenancy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-tenancy.git@v0.1.8
govoplan-organizations @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-organizations.git@v0.1.8
govoplan-identity @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-identity.git@v0.1.8
govoplan-idm @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-idm.git@v0.1.8
govoplan-access @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-access.git@v0.1.8
govoplan-admin @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-admin.git@v0.1.8
govoplan-policy @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-policy.git@v0.1.8
govoplan-audit @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-audit.git@v0.1.8
govoplan-dashboard @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-dashboard.git@v0.1.8
govoplan-files @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-files.git@v0.1.8
govoplan-mail @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-mail.git@v0.1.10
govoplan-campaign @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-campaign.git@v0.1.11
govoplan-calendar @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-calendar.git@v0.1.8
govoplan-docs @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-docs.git@v0.1.8
govoplan-ops @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-ops.git@v0.1.8
+782
View File
@@ -0,0 +1,782 @@
from __future__ import annotations
from contextlib import redirect_stderr, redirect_stdout
import io
import json
from pathlib import Path
import stat
import subprocess
import sys
import tempfile
import unittest
from unittest.mock import MagicMock, patch
META_ROOT = Path(__file__).resolve().parents[1]
DEPLOYMENT_TOOLS = META_ROOT / "tools" / "deployment"
if str(DEPLOYMENT_TOOLS) not in sys.path:
sys.path.insert(0, str(DEPLOYMENT_TOOLS))
from govoplan_deploy.bundle import ( # noqa: E402
atomic_write,
bundle_paths,
canonical_json,
environment_fingerprint,
initial_secrets,
read_env,
reconcile_runtime_environment,
render_compose,
render_load_balancer_config,
write_env,
)
from govoplan_deploy.cli import _receipt_uses_direct_web_port, main # noqa: E402
import govoplan_deploy.cli as deployment_cli # noqa: E402
from govoplan_deploy.model import ( # noqa: E402
SpecError,
default_spec,
parse_spec,
)
from govoplan_deploy.planning import _endpoint_check, build_plan # noqa: E402
import govoplan_deploy.planning as deployment_planning # noqa: E402
def run_cli(arguments: list[str]) -> tuple[int, str, str]:
stdout = io.StringIO()
stderr = io.StringIO()
with redirect_stdout(stdout), redirect_stderr(stderr):
result = main(arguments)
return result, stdout.getvalue(), stderr.getvalue()
class DeploymentInstallerTests(unittest.TestCase):
def test_default_bundle_has_base_modules_and_managed_dependencies(self) -> None:
spec = default_spec()
compose = render_compose(spec)
self.assertEqual("evaluation", spec.profile)
self.assertIn("access", spec.enabled_modules)
self.assertIn("postgres", compose["services"])
self.assertIn("redis", compose["services"])
self.assertIn("worker", compose["services"])
self.assertIn("load-balancer", compose["services"])
self.assertNotIn("test-mail", compose["services"])
self.assertEqual(
["127.0.0.1:8080:8080"],
compose["services"]["load-balancer"]["ports"],
)
self.assertNotIn("ports", compose["services"]["web"])
self.assertEqual(1, compose["services"]["api"]["scale"])
self.assertEqual(1, compose["services"]["web"]["scale"])
def test_disabled_redis_removes_workers_and_sets_single_process_acknowledgement(
self,
) -> None:
spec = default_spec(redis_mode="disabled")
values = initial_secrets(spec)
compose = render_compose(spec)
self.assertNotIn("redis", compose["services"])
self.assertNotIn("worker", compose["services"])
self.assertNotIn("scheduler", compose["services"])
self.assertEqual("false", values["CELERY_ENABLED"])
self.assertEqual("true", values["GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE"])
def test_self_hosted_rejects_insecure_or_test_only_choices(self) -> None:
with self.assertRaisesRegex(SpecError, "must use HTTPS"):
default_spec(profile="self-hosted")
with self.assertRaisesRegex(SpecError, "require managed or external Redis"):
default_spec(
profile="self-hosted",
public_url="https://govoplan.example.test",
redis_mode="disabled",
)
with self.assertRaisesRegex(SpecError, "test-mail"):
default_spec(
profile="self-hosted",
public_url="https://govoplan.example.test",
mail_mode="test-mail",
)
def test_spec_rejects_unknown_fields_and_duplicate_modules(self) -> None:
raw = default_spec().to_dict()
raw["unexpected"] = True
with self.assertRaisesRegex(SpecError, "unknown fields"):
parse_spec(raw)
raw = default_spec().to_dict()
raw["enabled_modules"].append(raw["enabled_modules"][0])
with self.assertRaisesRegex(SpecError, "duplicate module"):
parse_spec(raw)
raw = default_spec().to_dict()
raw["network_subnet"] = "127.0.0.0/24"
with self.assertRaisesRegex(SpecError, "RFC1918"):
parse_spec(raw)
def test_generated_secrets_are_stable_across_reconfiguration(self) -> None:
first = default_spec()
values = initial_secrets(first)
master_key = values["MASTER_KEY_B64"]
postgres_password = values["POSTGRES_PASSWORD"]
redis_password = values["REDIS_PASSWORD"]
second = default_spec(mail_mode="test-mail", module_set="full")
reconciled = reconcile_runtime_environment(second, values)
self.assertEqual(master_key, reconciled["MASTER_KEY_B64"])
self.assertEqual(postgres_password, reconciled["POSTGRES_PASSWORD"])
self.assertEqual(redis_password, reconciled["REDIS_PASSWORD"])
self.assertEqual(
",".join(second.enabled_modules), reconciled["ENABLED_MODULES"]
)
def test_external_services_require_explicit_valid_urls(self) -> None:
spec = default_spec(postgres_mode="external", redis_mode="external")
with self.assertRaisesRegex(ValueError, "external PostgreSQL"):
initial_secrets(spec)
with self.assertRaisesRegex(ValueError, "external Redis"):
initial_secrets(
spec,
supplied={
"DATABASE_URL": (
"postgresql+psycopg://user:secret@db.example.test/govoplan"
)
},
)
values = initial_secrets(
spec,
supplied={
"DATABASE_URL": (
"postgresql+psycopg://user:secret@db.example.test/govoplan"
),
"REDIS_URL": "rediss://:secret@redis.example.test/0",
},
)
self.assertEqual("rediss://:secret@redis.example.test/0", values["REDIS_URL"])
def test_s3_requires_complete_private_configuration_and_https_in_production(
self,
) -> None:
evaluation = default_spec(storage_mode="s3")
with self.assertRaisesRegex(ValueError, "S3 storage requires"):
initial_secrets(evaluation)
supplied = {
"FILE_STORAGE_S3_ENDPOINT_URL": "http://s3.example.test",
"FILE_STORAGE_S3_REGION": "eu-test-1",
"FILE_STORAGE_S3_ACCESS_KEY_ID": "key",
"FILE_STORAGE_S3_SECRET_ACCESS_KEY": "secret",
"FILE_STORAGE_S3_BUCKET": "govoplan",
}
self.assertEqual(
"s3",
initial_secrets(evaluation, supplied=supplied)["FILE_STORAGE_BACKEND"],
)
production = default_spec(
profile="self-hosted",
public_url="https://govoplan.example.test",
storage_mode="s3",
)
with self.assertRaisesRegex(ValueError, "must use HTTPS"):
initial_secrets(production, supplied=supplied)
def test_managed_garage_bootstraps_private_s3_storage(self) -> None:
spec = default_spec(storage_mode="garage")
values = initial_secrets(spec)
compose = render_compose(spec)
self.assertEqual("s3", values["FILE_STORAGE_BACKEND"])
self.assertEqual("true", values["FILE_STORAGE_S3_DEPLOYMENT_MANAGED"])
self.assertEqual(
"http://garage:3900",
values["FILE_STORAGE_S3_ENDPOINT_URL"],
)
self.assertEqual("garage", values["FILE_STORAGE_S3_REGION"])
self.assertEqual(
values["GARAGE_DEFAULT_ACCESS_KEY"],
values["FILE_STORAGE_S3_ACCESS_KEY_ID"],
)
self.assertEqual(
values["GARAGE_DEFAULT_SECRET_KEY"],
values["FILE_STORAGE_S3_SECRET_ACCESS_KEY"],
)
self.assertTrue(values["GARAGE_DEFAULT_ACCESS_KEY"].startswith("GK"))
self.assertEqual(34, len(values["GARAGE_DEFAULT_ACCESS_KEY"]))
self.assertEqual(64, len(values["GARAGE_DEFAULT_SECRET_KEY"]))
self.assertIn("garage", compose["services"])
self.assertIn("garage-meta", compose["volumes"])
self.assertIn("garage-data", compose["volumes"])
self.assertEqual(
["/garage", "server", "--single-node", "--default-bucket"],
compose["services"]["garage"]["command"],
)
self.assertNotIn(
values["GARAGE_DEFAULT_SECRET_KEY"],
json.dumps(compose),
)
reconciled = reconcile_runtime_environment(spec, values)
self.assertEqual(
values["GARAGE_DEFAULT_ACCESS_KEY"],
reconciled["GARAGE_DEFAULT_ACCESS_KEY"],
)
self.assertEqual(
values["GARAGE_RPC_SECRET"],
reconciled["GARAGE_RPC_SECRET"],
)
def test_replica_counts_drive_compose_and_load_balancer_discovery(self) -> None:
spec = default_spec(
storage_mode="garage",
api_replicas=3,
web_replicas=2,
worker_replicas=4,
)
compose = render_compose(spec)
config = render_load_balancer_config(spec)
self.assertEqual(3, compose["services"]["api"]["scale"])
self.assertEqual(2, compose["services"]["web"]["scale"])
self.assertEqual(4, compose["services"]["worker"]["scale"])
self.assertEqual(
"http://load-balancer:8000",
compose["services"]["web"]["environment"]["GOVOPLAN_API_UPSTREAM"],
)
self.assertIn("server-template web- 2 web:8080", config)
self.assertIn("server-template api- 3 api:8000", config)
def test_multi_replica_runtime_requires_shared_redis(self) -> None:
with self.assertRaisesRegex(SpecError, "multiple API replicas"):
default_spec(redis_mode="disabled", api_replicas=2)
with self.assertRaisesRegex(SpecError, "worker must be 0"):
default_spec(redis_mode="disabled", worker_replicas=1)
with self.assertRaisesRegex(SpecError, "at least 1"):
default_spec(redis_mode="managed", worker_replicas=0)
def test_legacy_spec_defaults_new_topology_fields(self) -> None:
raw = default_spec().to_dict()
raw.pop("replicas")
raw["components"].pop("load_balancer")
raw["components"]["storage"].pop("image")
parsed = parse_spec(raw)
self.assertEqual(1, parsed.replicas.api)
self.assertEqual(1, parsed.replicas.web)
self.assertEqual(1, parsed.replicas.worker)
self.assertEqual("managed", parsed.components.load_balancer.mode)
def test_compose_contains_no_secret_values(self) -> None:
spec = default_spec()
values = initial_secrets(spec)
rendered = json.dumps(render_compose(spec), sort_keys=True)
for key in (
"MASTER_KEY_B64",
"POSTGRES_PASSWORD",
"REDIS_PASSWORD",
):
self.assertNotIn(values[key], rendered)
self.assertNotIn("secrets.env", rendered)
self.assertNotIn("env_file", rendered)
self.assertNotIn("./:/etc/govoplan", rendered)
self.assertNotIn("installer", render_compose(spec)["services"])
postgres_environment = render_compose(spec)["services"]["postgres"][
"environment"
]
redis_environment = render_compose(spec)["services"]["redis"]["environment"]
self.assertEqual(
{"POSTGRES_DB", "POSTGRES_USER", "POSTGRES_PASSWORD"},
set(postgres_environment),
)
self.assertEqual({"REDIS_PASSWORD"}, set(redis_environment))
self.assertNotIn("MASTER_KEY_B64", postgres_environment)
self.assertNotIn("MASTER_KEY_B64", redis_environment)
def test_secret_environment_round_trips_literal_special_characters(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
path = Path(directory) / "secrets.env"
values = {
"PASSWORD": r"dollar$ quote' slash\\ hash# space ",
"EMPTY_VALUE": "",
}
write_env(path, values)
self.assertEqual(values, read_env(path))
def test_first_plan_creates_services_and_applied_receipt_becomes_noop(
self,
) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
paths = bundle_paths(Path(directory))
paths.root.chmod(0o700)
spec = default_spec()
write_env(paths.env, initial_secrets(spec))
first = build_plan(spec, paths, include_host_checks=False)
self.assertIn(
("create", "installation"),
{(action.action, action.target) for action in first.actions},
)
self.assertIn(
("start", "api"),
{(action.action, action.target) for action in first.actions},
)
receipt = {
"spec_sha256": first.desired_spec_sha256,
"compose_sha256": first.desired_compose_sha256,
"environment_fingerprint": (first.desired_environment_fingerprint),
"services": list(render_compose(spec)["services"]),
}
atomic_write(paths.receipt, canonical_json(receipt), mode=0o600)
second = build_plan(spec, paths, include_host_checks=False)
self.assertEqual(
[("noop", "installation")],
[(action.action, action.target) for action in second.actions],
)
def test_reconfiguration_plans_removed_component_containers(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
paths = bundle_paths(Path(directory))
paths.root.chmod(0o700)
first_spec = default_spec(mail_mode="test-mail")
first_environment = initial_secrets(first_spec)
write_env(paths.env, first_environment)
first_plan = build_plan(first_spec, paths, include_host_checks=False)
atomic_write(
paths.receipt,
canonical_json(
{
"spec_sha256": first_plan.desired_spec_sha256,
"compose_sha256": first_plan.desired_compose_sha256,
"environment_fingerprint": (
first_plan.desired_environment_fingerprint
),
"services": list(render_compose(first_spec)["services"]),
}
),
mode=0o600,
)
second_spec = default_spec(redis_mode="disabled", mail_mode="disabled")
write_env(
paths.env,
reconcile_runtime_environment(second_spec, first_environment),
)
second_plan = build_plan(second_spec, paths, include_host_checks=False)
removed = {
action.target
for action in second_plan.actions
if action.action == "remove"
}
self.assertEqual(
{"redis", "worker", "scheduler", "test-mail"},
removed,
)
def test_secret_change_is_planned_without_exposing_secret_values(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
paths = bundle_paths(Path(directory))
paths.root.chmod(0o700)
spec = default_spec()
values = initial_secrets(spec)
write_env(paths.env, values)
first = build_plan(spec, paths, include_host_checks=False)
atomic_write(
paths.receipt,
canonical_json(
{
"spec_sha256": first.desired_spec_sha256,
"compose_sha256": first.desired_compose_sha256,
"environment_fingerprint": (
first.desired_environment_fingerprint
),
"services": list(render_compose(spec)["services"]),
}
),
mode=0o600,
)
rotated = dict(values)
rotated["REDIS_PASSWORD"] = "rotated-high-entropy-value"
write_env(paths.env, rotated)
second = build_plan(spec, paths, include_host_checks=False)
plan_json = json.dumps(second.to_dict())
self.assertIn(
("reconfigure", "environment"),
{(action.action, action.target) for action in second.actions},
)
self.assertNotIn(rotated["REDIS_PASSWORD"], plan_json)
self.assertEqual(
environment_fingerprint(rotated),
second.desired_environment_fingerprint,
)
def test_external_endpoint_preflight_reports_reachability(self) -> None:
connection = MagicMock()
connection.__enter__.return_value = connection
with patch.object(
deployment_planning.socket,
"create_connection",
return_value=connection,
) as connect:
check = _endpoint_check(
"external.redis",
"External Redis",
"rediss://redis.example.test/0",
default_ports={"redis": 6379, "rediss": 6380},
)
self.assertEqual("ok", check.level)
connect.assert_called_once_with(
("redis.example.test", 6380),
timeout=1.5,
)
def test_cli_init_writes_private_idempotent_bundle(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
root = Path(directory) / "installation"
result, _stdout, _stderr = run_cli(
[
"init",
"--non-interactive",
"--directory",
str(root),
"--redis",
"disabled",
"--mail",
"test-mail",
]
)
self.assertEqual(0, result)
self.assertEqual(0o700, stat.S_IMODE(root.stat().st_mode))
for name in (
"installation.json",
"secrets.env",
"compose.json",
"plan.json",
):
self.assertEqual(0o600, stat.S_IMODE((root / name).stat().st_mode))
for name in ("garage.toml", "load-balancer.cfg"):
self.assertEqual(
0o644,
stat.S_IMODE((root / name).stat().st_mode),
)
values = read_env(root / "secrets.env")
self.assertTrue(values["MASTER_KEY_B64"])
self.assertNotIn(
values["POSTGRES_PASSWORD"],
(root / "compose.json").read_text(encoding="utf-8"),
)
self.assertEqual(
1,
run_cli(
[
"init",
"--non-interactive",
"--directory",
str(root),
]
)[0],
)
def test_cli_requires_external_url_when_switching_from_managed(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
root = Path(directory) / "installation"
self.assertEqual(
0,
run_cli(
[
"init",
"--non-interactive",
"--directory",
str(root),
]
)[0],
)
self.assertEqual(
1,
run_cli(
[
"configure",
"--directory",
str(root),
"--postgres",
"external",
]
)[0],
)
self.assertEqual(
"managed",
json.loads((root / "installation.json").read_text(encoding="utf-8"))[
"components"
]["postgres"]["mode"],
)
self.assertEqual(
1,
run_cli(
[
"configure",
"--directory",
str(root),
"--installation-id",
"renamed-installation",
]
)[0],
)
def test_cli_requires_explicit_external_s3_values_after_garage(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
root = Path(directory) / "installation"
self.assertEqual(
0,
run_cli(
[
"init",
"--non-interactive",
"--directory",
str(root),
"--storage",
"garage",
]
)[0],
)
result, _stdout, stderr = run_cli(
[
"configure",
"--directory",
str(root),
"--storage",
"s3",
]
)
self.assertEqual(1, result)
self.assertIn("switching to external S3 requires", stderr)
self.assertEqual(
"garage",
json.loads((root / "installation.json").read_text(encoding="utf-8"))[
"components"
]["storage"]["mode"],
)
def test_deployer_builds_and_runs_as_one_zipapp(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
output = Path(directory) / "govoplan-deploy.pyz"
build = subprocess.run(
[
sys.executable,
str(DEPLOYMENT_TOOLS / "build-deployer-zipapp.py"),
"--output",
str(output),
],
cwd=META_ROOT,
check=False,
capture_output=True,
text=True,
)
help_result = subprocess.run(
[sys.executable, str(output), "--help"],
cwd=META_ROOT,
check=False,
capture_output=True,
text=True,
)
install_root = Path(directory) / "install"
init_result = subprocess.run(
[
sys.executable,
str(output),
"init",
"--non-interactive",
"--directory",
str(install_root),
],
cwd=META_ROOT,
check=False,
capture_output=True,
text=True,
)
render_result = subprocess.run(
[
sys.executable,
str(output),
"render",
"--directory",
str(install_root),
"--json",
],
cwd=META_ROOT,
check=False,
capture_output=True,
text=True,
)
self.assertEqual(0, build.returncode, build.stderr)
self.assertTrue(output.exists())
self.assertEqual(0o755, stat.S_IMODE(output.stat().st_mode))
self.assertEqual(0, help_result.returncode, help_result.stderr)
self.assertIn("govoplan-deploy", help_result.stdout)
self.assertEqual(0, init_result.returncode, init_result.stderr)
self.assertEqual(1, render_result.returncode, render_result.stderr)
self.assertTrue(json.loads(render_result.stdout)["blocked"])
def test_generated_environment_passes_core_startup_validation_when_available(
self,
) -> None:
try:
from govoplan_core.core.install_config import (
validate_runtime_configuration,
)
except ModuleNotFoundError:
self.skipTest("govoplan-core is not installed in this test environment")
for profile, public_url in (
("evaluation", "http://127.0.0.1:8080"),
("self-hosted", "https://govoplan.example.test"),
):
with self.subTest(profile=profile):
environment = initial_secrets(
default_spec(profile=profile, public_url=public_url)
)
validation = validate_runtime_configuration(environment)
self.assertEqual(
(),
validation.errors,
validation.to_text(),
)
def test_apply_orders_dependencies_migration_and_runtime_before_receipt(
self,
) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
root = Path(directory) / "installation"
self.assertEqual(
0,
run_cli(
[
"init",
"--non-interactive",
"--directory",
str(root),
"--api-image",
"local/govoplan-api:test",
"--web-image",
"local/govoplan-web:test",
]
)[0],
)
commands: list[list[str]] = []
def record_command(argv, *, cwd):
self.assertEqual(root, cwd)
commands.append(list(argv))
compose_version = subprocess.CompletedProcess(
args=["docker", "compose", "version"],
returncode=0,
stdout="2.30.0\n",
stderr="",
)
with (
patch.object(
deployment_cli.shutil,
"which",
return_value="/usr/bin/docker",
),
patch.object(
deployment_planning.shutil,
"which",
return_value="/usr/bin/docker",
),
patch.object(
deployment_planning,
"_run_command",
return_value=compose_version,
),
patch.object(deployment_cli, "_run", side_effect=record_command),
patch.object(deployment_cli, "_wait_for_health") as wait_for_health,
):
result, _stdout, _stderr = run_cli(
[
"apply",
"--directory",
str(root),
"--skip-pull",
"--allow-unverified-images",
]
)
self.assertEqual(0, result)
migrate_index = next(
index
for index, command in enumerate(commands)
if command[-3:] == ["run", "--rm", "migrate"]
)
runtime_index = next(
index
for index, command in enumerate(commands)
if "--remove-orphans" in command
)
self.assertLess(migrate_index, runtime_index)
self.assertIn("postgres", commands[0])
self.assertIn("redis", commands[0])
wait_for_health.assert_called_once_with(
"http://127.0.0.1:8080/health",
timeout_seconds=120.0,
)
receipt = json.loads((root / "receipt.json").read_text(encoding="utf-8"))
self.assertEqual("govoplan-local", receipt["installation_id"])
self.assertEqual(
{"address": "127.0.0.1", "port": 8080},
receipt["listen"],
)
self.assertNotIn("installer", receipt["services"])
def test_installation_root_symlink_is_rejected(self) -> None:
if not hasattr(Path, "symlink_to"):
self.skipTest("symbolic links are unavailable")
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
base = Path(directory)
target = base / "real"
target.mkdir()
link = base / "linked"
link.symlink_to(target, target_is_directory=True)
with self.assertRaisesRegex(ValueError, "symbolic link"):
bundle_paths(link)
def test_legacy_receipt_requests_direct_web_port_handoff(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-deploy-test-") as directory:
receipt = Path(directory) / "receipt.json"
receipt.write_text(
json.dumps({"services": ["api", "web", "worker"]}),
encoding="utf-8",
)
self.assertTrue(_receipt_uses_direct_web_port(receipt))
receipt.write_text(
json.dumps(
{
"services": [
"api",
"web",
"load-balancer",
"worker",
]
}
),
encoding="utf-8",
)
self.assertFalse(_receipt_uses_direct_web_port(receipt))
if __name__ == "__main__":
unittest.main()
+49
View File
@@ -0,0 +1,49 @@
from __future__ import annotations
import importlib.util
import pathlib
import sys
import tempfile
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
SCRIPT = ROOT / "tools" / "gitea" / "gitea-sync-wiki.py"
TOOLS = SCRIPT.parent
if str(TOOLS) not in sys.path:
sys.path.insert(0, str(TOOLS))
SPEC = importlib.util.spec_from_file_location("gitea_sync_wiki", SCRIPT)
assert SPEC and SPEC.loader
wiki_sync = importlib.util.module_from_spec(SPEC)
sys.modules[SPEC.name] = wiki_sync
SPEC.loader.exec_module(wiki_sync)
class WikiSourceDiscoveryTests(unittest.TestCase):
def test_generated_and_incidental_govoplan_files_are_not_docs(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = pathlib.Path(temporary)
paths = {
"readme": root / "README.md",
"architecture": root / "docs" / "DATASOURCE_ARCHITECTURE.md",
"plan": root / "workflow-plan.md",
"audit": root / "audit-reports" / "full" / "manifest.json",
"runtime": root / "runtime" / "release" / "manifest.json",
"incidental": root / "tools" / "semgrep" / "govoplan.yml",
"manifest": root / "manifest.json",
}
for path in paths.values():
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text("content\n", encoding="utf-8")
self.assertTrue(wiki_sync.is_repo_doc(root, paths["readme"]))
self.assertTrue(wiki_sync.is_repo_doc(root, paths["architecture"]))
self.assertTrue(wiki_sync.is_repo_doc(root, paths["plan"]))
self.assertFalse(wiki_sync.is_repo_doc(root, paths["audit"]))
self.assertFalse(wiki_sync.is_repo_doc(root, paths["runtime"]))
self.assertFalse(wiki_sync.is_repo_doc(root, paths["incidental"]))
self.assertFalse(wiki_sync.is_repo_doc(root, paths["manifest"]))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,66 @@
from __future__ import annotations
import ast
import importlib.util
from pathlib import Path
import unittest
SCRIPT = (
Path(__file__).resolve().parents[1]
/ "tools"
/ "inventory"
/ "platform-interface-inventory.py"
)
SPEC = importlib.util.spec_from_file_location("platform_interface_inventory", SCRIPT)
assert SPEC is not None and SPEC.loader is not None
inventory = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(inventory)
class PlatformInterfaceInventoryTests(unittest.TestCase):
def test_canonical_api_path_normalizes_versions_and_parameters(self) -> None:
self.assertEqual(
inventory.canonical_api_path(
"http://localhost/api/v1/campaigns/${campaignId}?limit=10"
),
"/campaigns/{}",
)
self.assertEqual(
inventory.canonical_api_path("/api/v2/campaigns/{campaign_id}"),
"/campaigns/{}",
)
def test_fastapi_route_scanner_includes_router_prefix(self) -> None:
tree = ast.parse(
"""
from fastapi import APIRouter
router = APIRouter(prefix="/api/v1/items")
@router.get("/{item_id}")
def read_item(item_id: str):
return item_id
"""
)
prefixes = inventory._router_prefixes(tree)
function = next(
node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef)
)
route = inventory._endpoint_from_decorator(
function.decorator_list[0],
prefixes=prefixes,
)
self.assertEqual(
route,
{
"method": "GET",
"path": "/api/v1/items/{item_id}",
"router": "router",
},
)
if __name__ == "__main__":
unittest.main()
+70
View File
@@ -0,0 +1,70 @@
from __future__ import annotations
import importlib.util
import os
from pathlib import Path
import unittest
from unittest.mock import patch
META_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = META_ROOT / "tools" / "checks" / "postgres-integration-check.py"
def _load_script():
spec = importlib.util.spec_from_file_location(
"postgres_integration_check",
SCRIPT,
)
if spec is None or spec.loader is None:
raise RuntimeError(f"Could not load {SCRIPT}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
class PostgresIntegrationCheckTests(unittest.TestCase):
def test_staging_check_has_explicit_runtime_safety_settings(self) -> None:
module = _load_script()
with patch.dict(os.environ, {}, clear=True):
env = module._check_env(
database_url="postgresql+psycopg://user:password@postgres/db",
modules="tenancy,access,search",
app_env="staging",
)
self.assertEqual(
env["CORS_ORIGINS"],
"http://127.0.0.1:5173,http://localhost:5173",
)
self.assertEqual(
env["GOVOPLAN_TRUSTED_HOSTS"],
"127.0.0.1,localhost,testserver",
)
self.assertEqual(
env["GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS"],
"false",
)
def test_existing_runtime_safety_settings_are_preserved(self) -> None:
module = _load_script()
configured = {
"CORS_ORIGINS": "https://govoplan.example.test",
"GOVOPLAN_TRUSTED_HOSTS": "govoplan.example.test",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "true",
}
with patch.dict(os.environ, configured, clear=True):
env = module._check_env(
database_url="postgresql+psycopg://user:password@postgres/db",
modules="tenancy,access,search",
app_env="staging",
)
for key, value in configured.items():
self.assertEqual(env[key], value)
if __name__ == "__main__":
unittest.main()
+70
View File
@@ -77,6 +77,76 @@ class PythonEnvironmentSyncTests(unittest.TestCase):
self.assertIn(str(root / "govoplan-core"), command)
self.assertIn(str(root / "govoplan-access"), command)
def test_missing_editable_distribution_is_not_hidden_by_current_stamp(self) -> None:
sync = load_sync_module()
with tempfile.TemporaryDirectory(prefix="govoplan-python-sync-") as directory:
root = Path(directory)
project_root = root / "govoplan-probe-missing"
project_root.mkdir()
(project_root / "pyproject.toml").write_text(
"\n".join(
(
"[project]",
'name = "govoplan-probe-definitely-not-installed"',
'version = "0.1.0"',
"",
'[project.entry-points."govoplan.modules"]',
'probe_missing = "govoplan_probe.backend.manifest:get_manifest"',
"",
)
),
encoding="utf-8",
)
requirements = root / "requirements-dev.txt"
requirements.write_text("-e ./govoplan-probe-missing\n", encoding="utf-8")
entries = sync.local_requirement_entries(requirements)
validation = sync.validate_local_installations(sys.executable, entries)
plan = sync.build_environment_repair_plan(
python=sys.executable,
stale_requirements=validation.stale_requirements,
)
self.assertFalse(validation.current)
self.assertEqual(validation.stale_requirements, entries)
self.assertIn("distribution is not installed", validation.issues[0])
self.assertEqual(plan.mode, "Selective Python environment repair")
self.assertEqual(plan.commands[0][-2:], ("-e", str(project_root)))
def test_declared_module_entry_points_are_part_of_environment_validation(self) -> None:
sync = load_sync_module()
with tempfile.TemporaryDirectory(prefix="govoplan-python-sync-") as directory:
root = Path(directory)
project_root = root / "govoplan-probe"
project_root.mkdir()
(project_root / "pyproject.toml").write_text(
"\n".join(
(
"[project]",
'name = "govoplan-probe"',
'version = "0.1.0"',
"",
'[project.entry-points."govoplan.modules"]',
'probe = "govoplan_probe.backend.manifest:get_manifest"',
"",
)
),
encoding="utf-8",
)
requirements = root / "requirements-dev.txt"
requirements.write_text("-e ./govoplan-probe\n", encoding="utf-8")
expectations = sync.local_installation_expectations(
sync.local_requirement_entries(requirements)
)
self.assertEqual(len(expectations), 1)
self.assertEqual(expectations[0].distribution, "govoplan-probe")
self.assertEqual(
expectations[0].entry_points,
(("govoplan.modules", "probe", "govoplan_probe.backend.manifest:get_manifest"),),
)
if __name__ == "__main__":
unittest.main()
@@ -24,7 +24,7 @@ class ReleaseCatalogEntrySynthesisTests(unittest.TestCase):
"module_id": "access",
"version": "0.1.11",
"python_package": "govoplan-access",
"python_ref": "govoplan-access @ git+ssh://git@git.add-ideas.de/add-ideas/govoplan-access.git@v0.1.11",
"python_ref": "govoplan-access @ git+ssh://git@git.add-ideas.de/GovOPlaN/govoplan-access.git@v0.1.11",
}
],
}
@@ -37,7 +37,7 @@ class ReleaseCatalogEntrySynthesisTests(unittest.TestCase):
"govoplan-scheduling": "0.1.11",
},
repo_contracts={},
repository_base="git+ssh://git@git.add-ideas.de/add-ideas",
repository_base="git+ssh://git@git.add-ideas.de/GovOPlaN",
workspace=META_ROOT.parent,
)
+136
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import hashlib
import json
import os
from pathlib import Path
@@ -22,13 +23,148 @@ from govoplan_release.publisher import ( # noqa: E402
publication_mutation_trust_issues,
publish_catalog_candidate,
remote_publication_identity,
run_checked,
verify_committed_publication,
verify_remote_branch_head,
verify_remote_publication,
_git_bytes,
_sanitized_git_environment,
_seal_git_metadata_file,
_trusted_npm_command,
)
class ReleaseCatalogPublicationTests(unittest.TestCase):
def test_publication_git_writes_ignore_permissive_operator_umask(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
repository = Path(temp_dir) / "website"
repository.mkdir()
self._git(repository, "init", "--quiet")
payload_number = 0
while True:
payload = f"private publication object {payload_number}\n".encode()
header = f"blob {len(payload)}\0".encode()
expected_object = hashlib.sha1(
header + payload,
usedforsecurity=False,
).hexdigest()
object_parent = repository / ".git" / "objects" / expected_object[:2]
if not object_parent.exists():
break
payload_number += 1
previous_umask = os.umask(0o002)
try:
object_id = (
_git_bytes(
repository,
"hash-object",
"-w",
"--stdin",
input_bytes=payload,
)
.decode("ascii")
.strip()
)
finally:
os.umask(previous_umask)
object_path = object_parent / expected_object[2:]
self.assertEqual(expected_object, object_id)
self.assertEqual(os.geteuid(), object_parent.stat().st_uid)
self.assertEqual(0o700, object_parent.stat().st_mode & 0o777)
self.assertEqual(os.geteuid(), object_path.stat().st_uid)
self.assertEqual(0o400, object_path.stat().st_mode & 0o777)
def test_publication_git_metadata_is_sealed_after_git_writes(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
metadata = Path(temp_dir) / "index"
metadata.write_bytes(b"index")
metadata.chmod(0o664)
_seal_git_metadata_file(metadata, label="test index")
self.assertEqual(0o600, metadata.stat().st_mode & 0o777)
def test_publication_git_identity_is_fixed_and_non_personal(self) -> None:
with patch.dict(
os.environ,
{
"GIT_AUTHOR_NAME": "Attacker",
"GIT_AUTHOR_EMAIL": "attacker@example.test",
"GIT_COMMITTER_NAME": "Attacker",
"GIT_COMMITTER_EMAIL": "attacker@example.test",
},
clear=False,
):
environment = _sanitized_git_environment()
self.assertEqual(
"GovOPlaN Release Automation",
environment["GIT_AUTHOR_NAME"],
)
self.assertEqual(
"release@govoplan.invalid",
environment["GIT_AUTHOR_EMAIL"],
)
self.assertEqual(
environment["GIT_AUTHOR_NAME"],
environment["GIT_COMMITTER_NAME"],
)
self.assertEqual(
environment["GIT_AUTHOR_EMAIL"],
environment["GIT_COMMITTER_EMAIL"],
)
def test_build_command_uses_its_pinned_node_directory(self) -> None:
completed = subprocess.CompletedProcess(
["/trusted/node/bin/npm"],
0,
stdout=b"",
stderr=b"",
)
with patch(
"govoplan_release.publisher.subprocess.run",
return_value=completed,
) as runner:
run_checked(
["/trusted/node/bin/npm", "run", "build"],
cwd=Path("/trusted/website"),
)
environment = runner.call_args.kwargs["env"]
self.assertEqual(
"/trusted/node/bin:/usr/bin:/bin",
environment["PATH"],
)
def test_npm_command_requires_trusted_npm_and_sibling_node(self) -> None:
with (
patch(
"govoplan_release.publisher.shutil.which",
return_value="/trusted/node/bin/npm",
),
patch(
"govoplan_release.publisher._trusted_runtime_executable_issue",
side_effect=(None, None),
) as trust_check,
):
self.assertEqual(
"/trusted/node/bin/npm",
_trusted_npm_command("npm"),
)
self.assertEqual(Path("/trusted/node/bin/npm"), trust_check.call_args_list[0].args[0])
self.assertEqual(Path("/trusted/node/bin/node"), trust_check.call_args_list[1].args[0])
def test_npm_command_rejects_caller_relative_path(self) -> None:
with (
patch("govoplan_release.publisher.shutil.which") as which,
self.assertRaisesRegex(RuntimeError, "absolute path or the bare name"),
):
_trusted_npm_command("./npm")
which.assert_not_called()
def test_committed_publication_must_match_validated_blobs_exactly(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
web_root = Path(tmp) / "website"
+312 -13
View File
@@ -1,5 +1,6 @@
from __future__ import annotations
import json
from pathlib import Path
import os
import subprocess
@@ -8,6 +9,7 @@ import tempfile
from types import SimpleNamespace
import unittest
from unittest.mock import patch
import zipfile
META_ROOT = Path(__file__).resolve().parents[1]
@@ -16,6 +18,7 @@ if str(RELEASE_ROOT) not in sys.path:
sys.path.insert(0, str(RELEASE_ROOT))
from govoplan_release.model import RepositorySpec # noqa: E402
from govoplan_release.artifact_identity import inspect_python_wheel # noqa: E402
from govoplan_release.git_state import sanitized_git_environment # noqa: E402
from govoplan_release.release_execution import ( # noqa: E402
ExecutorSpec,
@@ -29,6 +32,7 @@ from govoplan_release.release_execution import ( # noqa: E402
require_trusted_release_runtime,
verify_frozen_repository_tag_receipt,
verify_repository_preflight_binding,
verify_candidate_installation,
_clone_catalog_sources,
_checkout_frozen_source,
_flatpak_proxy_environment,
@@ -488,20 +492,180 @@ class ReleaseExecutionTests(unittest.TestCase):
expected_receipt={"kind": "repository_state"},
)
def test_commit_step_has_no_durable_executor(self) -> None:
self.assertIsNone(
executor_spec(
{
"id": "govoplan-files:commit",
"status": "planned",
"mutating": True,
"repo": "govoplan-files",
"source_binding": {
"kind": "repository_state",
},
}
)
def test_commit_step_has_narrow_confirmation(self) -> None:
spec = executor_spec(
{
"id": "govoplan-files:commit",
"status": "planned",
"mutating": True,
"repo": "govoplan-files",
"source_binding": {
"kind": "repository_state",
},
}
)
self.assertIsNotNone(spec)
self.assertEqual("commit", spec.kind)
self.assertEqual("COMMIT", spec.confirmation)
def test_version_and_commit_executors_bind_only_release_metadata(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
workspace = Path(temp_dir)
repository = workspace / "govoplan-files"
remote = workspace / "files.git"
_git(workspace, "init", "--bare", str(remote))
_git(workspace, "init", "-b", "main", str(repository))
_git(repository, "config", "user.name", "Release Test")
_git(repository, "config", "user.email", "release@example.test")
(repository / "pyproject.toml").write_text(
'[project]\nname = "govoplan-files"\nversion = "1.2.3"\n',
encoding="utf-8",
)
_git(repository, "add", "pyproject.toml")
_git(repository, "commit", "-m", "Initial")
_git(repository, "remote", "add", "origin", str(remote))
_git(repository, "push", "-u", "origin", "main")
repository_spec = _repository_spec(remote)
plan_step = {
"id": "govoplan-files:version",
"repo": "govoplan-files",
"source_binding": {"kind": "repository_state"},
}
with (
patch(
"govoplan_release.release_execution.load_repository_specs",
return_value=(repository_spec,),
),
patch(
"govoplan_release.release_execution.require_trusted_release_runtime"
),
):
before = repository_state_receipt(
repo="govoplan-files",
target_tag="v1.2.4",
workspace_root=workspace,
)
_result, updated = execute_repository_step(
spec=ExecutorSpec(
"version",
"UPDATE",
"version_metadata_updated",
True,
),
plan_step=plan_step,
repo_versions={"govoplan-files": "1.2.4"},
workspace_root=workspace,
remote="origin",
expected_receipt=before,
)
self.assertFalse(updated["worktree_clean"])
self.assertNotEqual(
before["worktree_sha256"],
updated["worktree_sha256"],
)
plan_step["id"] = "govoplan-files:commit"
_result, committed = execute_repository_step(
spec=ExecutorSpec(
"commit",
"COMMIT",
"release_commit_created",
True,
),
plan_step=plan_step,
repo_versions={"govoplan-files": "1.2.4"},
workspace_root=workspace,
remote="origin",
expected_receipt=updated,
)
self.assertTrue(committed["worktree_clean"])
self.assertNotEqual(before["head"], committed["head"])
self.assertIn(
'version = "1.2.4"',
(repository / "pyproject.toml").read_text(encoding="utf-8"),
)
def test_candidate_installation_verifies_installed_metadata(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
workspace = Path(temp_dir)
repository = workspace / "govoplan-files"
repository.mkdir()
(repository / "pyproject.toml").write_text(
'[project]\nname = "govoplan-files"\nversion = "1.2.4"\n',
encoding="utf-8",
)
artifacts = workspace / "candidate" / "artifacts"
artifacts.mkdir(parents=True)
wheel = artifacts / "govoplan_files-1.2.4-py3-none-any.whl"
members = {
"govoplan_files/__init__.py": '__version__ = "1.2.4"\n',
"govoplan_files-1.2.4.dist-info/METADATA": (
"Metadata-Version: 2.1\n"
"Name: govoplan-files\n"
"Version: 1.2.4\n"
),
"govoplan_files-1.2.4.dist-info/WHEEL": (
"Wheel-Version: 1.0\n"
"Generator: release-test\n"
"Root-Is-Purelib: true\n"
"Tag: py3-none-any\n"
),
}
record = "\n".join(f"{name},," for name in members)
record += "\ngovoplan_files-1.2.4.dist-info/RECORD,,\n"
with zipfile.ZipFile(wheel, "w") as archive:
for name, content in members.items():
archive.writestr(name, content)
archive.writestr(
"govoplan_files-1.2.4.dist-info/RECORD",
record,
)
identity = inspect_python_wheel(wheel)
channels = workspace / "candidate" / "channels"
channels.mkdir()
(channels / "stable.json").write_text(
json.dumps(
{
"release": {
"selected_units": [
{
"repo": "govoplan-files",
"version": "1.2.4",
}
],
"artifacts": [
{
"package_name": "govoplan-files",
"archive_sha256": identity.archive_sha256,
}
],
},
"core_release": {},
"modules": [
{
"python_package": "govoplan-files",
"python_ref": (
"govoplan-files @ "
"git+ssh://git@example.test/GovOPlaN/"
"govoplan-files.git@v1.2.4"
),
}
],
}
)
+ "\n",
encoding="utf-8",
)
result = verify_candidate_installation(
candidate_path=workspace / "candidate",
repo_versions={"govoplan-files": "1.2.4"},
)
self.assertEqual("verified", result["status"])
self.assertEqual("govoplan-files", result["packages"][0]["repo"])
self.assertEqual("1.2.4", result["packages"][0]["version"])
def test_frozen_checkout_uses_receipt_commit_not_live_worktree(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
@@ -762,6 +926,141 @@ class ReleaseExecutionTests(unittest.TestCase):
_git_output(cloned["govoplan-core"], "rev-parse", "--abbrev-ref", "HEAD"),
)
def test_catalog_synthesis_adds_exact_core_bundle_dependency(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
workspace = Path(temp_dir)
idm_repo, idm_remote = _tagged_repository(
workspace,
repo="govoplan-idm",
bare="idm.git",
version="0.1.8",
)
idm_commit = _git_output(idm_repo, "rev-parse", "v0.1.8^{commit}")
(idm_repo / "pyproject.toml").write_text(
'[project]\nname = "govoplan-idm"\nversion = "0.1.9"\n',
encoding="utf-8",
)
_git(idm_repo, "add", "pyproject.toml")
_git(idm_repo, "commit", "-m", "Next release")
_git(idm_repo, "tag", "-a", "v0.1.9", "-m", "Next release")
_git(idm_repo, "push", "origin", "main", "refs/tags/v0.1.9")
catalog_commit = _git_output(idm_repo, "rev-parse", "v0.1.9^{commit}")
core_repo = workspace / "govoplan-core"
core_remote = workspace / "core.git"
_git(workspace, "init", "--bare", str(core_remote))
_git(workspace, "init", "-b", "main", str(core_repo))
_git(core_repo, "config", "user.name", "Release Test")
_git(core_repo, "config", "user.email", "release@example.test")
(core_repo / "webui").mkdir()
(core_repo / "pyproject.toml").write_text(
'[project]\nname = "govoplan-core"\nversion = "1.0.0"\n',
encoding="utf-8",
)
reference = (
"git+ssh://git@example.test/add-ideas/"
"govoplan-idm.git#v0.1.8"
)
(core_repo / "webui/package.release.json").write_text(
json.dumps(
{
"version": "1.0.0",
"dependencies": {"@govoplan/idm-webui": reference},
}
),
encoding="utf-8",
)
(core_repo / "webui/package-lock.release.json").write_text(
json.dumps(
{
"packages": {
"": {
"version": "1.0.0",
"dependencies": {
"@govoplan/idm-webui": reference,
},
},
"node_modules/@govoplan/idm-webui": {
"version": "0.1.8",
"resolved": f"{reference.rsplit('#', 1)[0]}#{idm_commit}",
},
}
}
),
encoding="utf-8",
)
_git(core_repo, "add", ".")
_git(core_repo, "commit", "-m", "Release")
_git(core_repo, "tag", "-a", "v1.0.0", "-m", "Release")
_git(core_repo, "remote", "add", "origin", str(core_remote))
_git(core_repo, "push", "-u", "origin", "main", "refs/tags/v1.0.0")
specs = (
RepositorySpec(
name="govoplan-core",
category="system",
subtype="kernel",
remote=str(core_remote),
path="govoplan-core",
),
RepositorySpec(
name="govoplan-idm",
category="module",
subtype="platform",
remote=str(idm_remote),
path="govoplan-idm",
),
)
with patch(
"govoplan_release.release_execution.load_repository_specs",
return_value=specs,
):
synthesis = workspace / "synthesis"
synthesis.mkdir()
cloned = _clone_catalog_sources(
source_versions={"govoplan-core": "1.0.0"},
repo_versions={},
source_receipts={},
workspace_root=workspace,
destination=synthesis,
)
newer_synthesis = workspace / "newer-synthesis"
newer_synthesis.mkdir()
cloned_with_newer_catalog_source = _clone_catalog_sources(
source_versions={
"govoplan-core": "1.0.0",
"govoplan-idm": "0.1.9",
},
repo_versions={},
source_receipts={},
workspace_root=workspace,
destination=newer_synthesis,
)
self.assertEqual(
{"govoplan-core", "govoplan-idm"},
set(cloned),
)
self.assertEqual(
idm_commit,
_git_output(cloned["govoplan-idm"], "rev-parse", "HEAD"),
)
self.assertEqual(
catalog_commit,
_git_output(
cloned_with_newer_catalog_source["govoplan-idm"],
"rev-parse",
"HEAD",
),
)
self.assertEqual(
idm_commit,
_git_output(
cloned_with_newer_catalog_source["govoplan-idm"],
"rev-parse",
"v0.1.8^{commit}",
),
)
def _repository_spec(remote: Path) -> RepositorySpec:
return RepositorySpec(
+39
View File
@@ -0,0 +1,39 @@
from __future__ import annotations
from pathlib import Path
import subprocess
import sys
import unittest
from unittest.mock import patch
META_ROOT = Path(__file__).resolve().parents[1]
RELEASE_TOOLS_ROOT = META_ROOT / "tools" / "release"
if str(RELEASE_TOOLS_ROOT) not in sys.path:
sys.path.insert(0, str(RELEASE_TOOLS_ROOT))
from govoplan_release import git_state # noqa: E402
class ReleaseGitStateTests(unittest.TestCase):
def test_git_trusts_only_the_resolved_repository_for_each_command(self) -> None:
repository = Path("/workspace/../workspace/govoplan-core")
completed = subprocess.CompletedProcess([], 0, "", "")
with patch.object(git_state.subprocess, "run", return_value=completed) as run:
result = git_state.git(repository, "status", "--porcelain")
self.assertEqual(result.returncode, 0)
command = run.call_args.args[0]
self.assertIn(f"safe.directory={repository.resolve()}", command)
self.assertNotIn("safe.directory=*", command)
self.assertEqual(run.call_args.kwargs["cwd"], repository)
self.assertEqual(
run.call_args.kwargs["env"]["GIT_CONFIG_GLOBAL"],
"/dev/null",
)
if __name__ == "__main__":
unittest.main()
+74 -3
View File
@@ -1,8 +1,10 @@
from __future__ import annotations
import os
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from tools.checks.release_integration import (
SOURCE_COUPLED_CORE_TESTS,
@@ -10,6 +12,7 @@ from tools.checks.release_integration import (
artifact_contract_issues,
filter_test_suite,
release_package_names,
run_module_release_tests,
)
@@ -32,9 +35,9 @@ class ReleaseIntegrationTests(unittest.TestCase):
"\n".join(
(
"../govoplan-core[server]",
"govoplan-idm @ git+ssh://git@example.test/add-ideas/govoplan-idm.git@v0.1.8",
"govoplan-campaign @ git+ssh://git@example.test/add-ideas/govoplan-campaign.git@v0.1.8",
"govoplan-idm @ git+ssh://git@example.test/add-ideas/govoplan-idm.git@v0.1.8",
"govoplan-idm @ git+ssh://git@example.test/GovOPlaN/govoplan-idm.git@v0.1.8",
"govoplan-campaign @ git+ssh://git@example.test/GovOPlaN/govoplan-campaign.git@v0.1.8",
"govoplan-idm @ git+ssh://git@example.test/GovOPlaN/govoplan-idm.git@v0.1.8",
"other-package==1.0",
)
),
@@ -89,8 +92,76 @@ class ReleaseIntegrationTests(unittest.TestCase):
self.assertLess(artifact_check, core_tests)
self.assertLess(core_tests, module_tests)
self.assertNotIn("unittest discover", script)
self.assertNotIn('"$PYTHON" -m unittest \\\n tests.test_module_system', script)
def test_tagged_module_runner_executes_pytest_functions_from_module_root(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-release-module-tests-") as temp_dir:
module_root = Path(temp_dir)
tests_root = module_root / "tests"
tests_root.mkdir()
marker = module_root / "pytest-function-ran"
(tests_root / "test_probe.py").write_text(
"""from pathlib import Path
import os
def test_pytest_style_function():
expected_root = Path(os.environ[\"GOVOPLAN_TEST_MODULE_ROOT\"])
assert Path.cwd() == expected_root
Path(os.environ[\"GOVOPLAN_TEST_MARKER\"]).write_text(\"ran\", encoding=\"utf-8\")
""",
encoding="utf-8",
)
with patch.dict(
os.environ,
{
"GOVOPLAN_TEST_MODULE_ROOT": str(module_root),
"GOVOPLAN_TEST_MARKER": str(marker),
},
):
status = run_module_release_tests(module_root)
self.assertEqual(status, 0)
self.assertEqual(marker.read_text(encoding="utf-8"), "ran")
def test_tagged_module_runner_skips_missing_test_tree(self) -> None:
with tempfile.TemporaryDirectory(prefix="govoplan-release-module-tests-") as temp_dir:
self.assertEqual(run_module_release_tests(Path(temp_dir)), 0)
def test_module_matrix_uses_artifact_aware_core_suite(self) -> None:
meta_root = Path(__file__).resolve().parents[1]
script = (meta_root / "tools" / "checks" / "check-module-matrix.sh").read_text(encoding="utf-8")
artifact_check = script.index('"$META_ROOT/tools/checks/release_integration.py" artifacts')
core_tests = script.index('"$META_ROOT/tools/checks/release_integration.py" core-tests')
self.assertLess(artifact_check, core_tests)
self.assertIn('--requirements "$META_ROOT/requirements-release.txt"', script)
self.assertIn('--core-root "$ROOT"', script)
self.assertNotIn('"$PYTHON" -m unittest tests.test_module_system', script)
def test_module_matrix_installs_search_before_postgres_validation(self) -> None:
meta_root = Path(__file__).resolve().parents[1]
workflow = (meta_root / ".gitea" / "workflows" / "module-matrix.yml").read_text(encoding="utf-8")
install = workflow.index("pip install --no-deps ../govoplan-search")
validation = workflow.index("Validate Search against PostgreSQL")
self.assertLess(install, validation)
def test_release_workflow_installs_tagged_source_test_harness(self) -> None:
meta_root = Path(__file__).resolve().parents[1]
workflow = (meta_root / ".gitea" / "workflows" / "release-integration.yml").read_text(encoding="utf-8")
requirements = (meta_root / "requirements-release-tests.txt").read_text(encoding="utf-8")
install = workflow.index("pip install -r requirements-release-tests.txt")
checks = workflow.index("bash tools/checks/check-release-integration.sh")
self.assertLess(install, checks)
self.assertIn("pytest>=9.0.3,<10", requirements)
self.assertNotIn("requirements-release-tests.txt", (meta_root / "requirements-release.txt").read_text(encoding="utf-8"))
if __name__ == "__main__":
unittest.main()
+32
View File
@@ -47,6 +47,38 @@ branch_labels: Union[str, Sequence[str], None] = None
self.assertEqual(migration.depends_on, ("core",))
self.assertEqual(migration.branch_labels, ())
def test_development_wrapper_uses_release_revision_metadata(self) -> None:
audit = load_audit_module()
with tempfile.TemporaryDirectory(prefix="migration-audit-test-") as directory:
root = Path(directory)
versions = root / "versions"
development = root / "dev_versions"
versions.mkdir()
development.mkdir()
release = versions / "1234_example.py"
release.write_text(
'revision = "1234"\n'
'down_revision = "base"\n'
'depends_on = "core"\n'
"branch_labels = None\n",
encoding="utf-8",
)
wrapper = development / release.name
wrapper.write_text(
"revision = _migration.revision\n"
"down_revision = _migration.down_revision\n"
"depends_on = _migration.depends_on\n"
"branch_labels = _migration.branch_labels\n",
encoding="utf-8",
)
migration = audit.parse_migration_file("govoplan-core", wrapper)
self.assertIsNotNone(migration)
self.assertEqual("1234", migration.revision)
self.assertEqual(("base",), migration.down_revisions)
self.assertEqual(("core",), migration.depends_on)
def test_release_baseline_matches_current_heads_in_strict_report(self) -> None:
audit = load_audit_module()
migrations = [
+128 -15
View File
@@ -23,7 +23,7 @@ from govoplan_release.selective_planner import build_selective_release_plan # n
class ReleasePlanGuidanceTests(unittest.TestCase):
def test_unprepared_target_has_structured_remediation_and_recommendation(
def test_unprepared_target_gets_bounded_version_and_commit_steps(
self,
) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
@@ -40,20 +40,25 @@ class ReleasePlanGuidanceTests(unittest.TestCase):
repo_versions={"govoplan-files": "1.2.4"},
)
self.assertEqual("blocked", plan.status)
self.assertFalse(plan.source_preflight_ready)
finding = next(
item
for item in plan.gate_findings
if item.code == "repository_version_alignment"
self.assertEqual("attention", plan.status)
self.assertTrue(plan.source_preflight_ready)
self.assertFalse(
any(
item.code == "repository_version_alignment"
for item in plan.gate_findings
)
)
self.assertEqual("govoplan-files", finding.repo)
self.assertEqual("pyproject.toml", finding.source)
self.assertEqual("1.2.4", finding.expected)
self.assertEqual("1.2.3", finding.actual)
self.assertIn("Regenerate lockfiles", finding.remediation)
self.assertEqual("resolve_release_gate", plan.recommended_action.id)
self.assertEqual("govoplan-files", plan.recommended_action.repo)
steps = {step.id: step for step in plan.dry_run_steps}
self.assertEqual("planned", steps["govoplan-files:version"].status)
self.assertEqual("planned", steps["govoplan-files:commit"].status)
self.assertIn("pyproject.toml", steps["govoplan-files:version"].detail)
self.assertTrue(
any(
"version metadata will be updated" in warning
for warning in plan.units[0].warnings
)
)
self.assertIn("python-package", plan.units[0].capabilities)
def test_aligned_target_recommends_non_mutating_tag_preview(self) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
@@ -74,6 +79,91 @@ class ReleasePlanGuidanceTests(unittest.TestCase):
self.assertEqual("preview_source_release", plan.recommended_action.id)
self.assertIn("Preview Tag + Publish", plan.recommended_action.remediation)
def test_mixed_plan_tags_modules_before_core_and_aligns_before_push(
self,
) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
workspace = Path(temp_dir)
for repo_name in ("govoplan-core", "govoplan-files"):
repo_path = workspace / repo_name
repo_path.mkdir()
(repo_path / "pyproject.toml").write_text(
f'[project]\nname = "{repo_name}"\nversion = "1.2.3"\n',
encoding="utf-8",
)
repositories = tuple(
RepositorySnapshot(
spec=RepositorySpec(
name=repo_name,
category="core" if repo_name == "govoplan-core" else "module",
subtype="platform" if repo_name == "govoplan-core" else "infrastructure",
remote=f"git@example.test:GovOPlaN/{repo_name}.git",
path=repo_name,
),
absolute_path=str(workspace / repo_name),
exists=True,
is_git=True,
has_head=True,
branch="main",
versions=VersionSnapshot(pyproject="1.2.3"),
local_target_tag_exists=False,
)
for repo_name in ("govoplan-core", "govoplan-files")
)
base = dashboard(workspace=workspace, version="1.2.3")
mixed = ReleaseDashboard(
generated_at=base.generated_at,
meta_root=base.meta_root,
workspace_root=base.workspace_root,
target_version=None,
target_tag=None,
online=False,
include_migrations=False,
summary=DashboardSummary(
repository_count=2,
missing_count=0,
dirty_count=0,
ahead_count=0,
behind_count=0,
no_head_count=0,
error_count=0,
safe_directory_count=0,
local_target_tag_missing_count=0,
status="ready",
),
repositories=repositories,
catalog=base.catalog,
)
plan = build_selective_release_plan(
mixed,
selected_repos=("govoplan-core", "govoplan-files"),
repo_versions={
"govoplan-core": "1.2.3",
"govoplan-files": "1.2.3",
},
)
self.assertEqual(
["govoplan-files", "govoplan-core"],
[unit.repo for unit in plan.units],
)
step_ids = [step.id for step in plan.dry_run_steps]
self.assertLess(
step_ids.index("govoplan-files:tag"),
step_ids.index("govoplan-core:tag"),
)
self.assertLess(
step_ids.index("govoplan-core:tag"),
step_ids.index("release:alignment"),
)
self.assertLess(
step_ids.index("release:alignment"),
step_ids.index("govoplan-files:push"),
)
self.assertEqual("release:install-verify", step_ids[-1])
core = next(unit for unit in plan.units if unit.repo == "govoplan-core")
self.assertIn("core-release-bundle", core.capabilities)
def test_malformed_version_metadata_is_a_structured_blocker(self) -> None:
cases = (
("package.json", "{not-json\n", "JSONDecodeError"),
@@ -149,6 +239,29 @@ class ReleasePlanGuidanceTests(unittest.TestCase):
self.assertIn('pill("recommended next", recommendationKind)', webui)
self.assertIn("<strong>Remediation:</strong>", webui)
def test_webui_projects_release_state_into_a_guided_workflow(self) -> None:
webui = (RELEASE_ROOT / "webui" / "index.html").read_text(encoding="utf-8")
for phase in (
"Inspect",
"Targets",
"Validate",
"Source",
"Package",
"Publish",
"Verify",
):
self.assertIn(f'label: "{phase}"', webui)
self.assertIn("releaseWorkflowPhases", webui)
self.assertIn("workflowPrimaryAction", webui)
self.assertIn("data-run-step-id", webui)
self.assertIn("invalidateReleaseDraft", webui)
self.assertIn("Installation Verification", webui)
self.assertIn("receipt-bound local gate", webui)
self.assertIn('step.id === "release:install-verify"', webui)
self.assertIn('const inspectionNotices = (summary.missing_count || 0)', webui)
self.assertIn('(summary.repository_count || 0) === 0', webui)
def dashboard(*, workspace: Path, version: str) -> ReleaseDashboard:
repo = RepositorySnapshot(
@@ -156,7 +269,7 @@ def dashboard(*, workspace: Path, version: str) -> ReleaseDashboard:
name="govoplan-files",
category="module",
subtype="infrastructure",
remote="git@example.test:add-ideas/govoplan-files.git",
remote="git@example.test:GovOPlaN/govoplan-files.git",
path="govoplan-files",
),
absolute_path=str(workspace / "govoplan-files"),
+105
View File
@@ -0,0 +1,105 @@
from __future__ import annotations
import json
from pathlib import Path
import sys
import tempfile
import unittest
META_ROOT = Path(__file__).resolve().parents[1]
RELEASE_ROOT = META_ROOT / "tools" / "release"
if str(RELEASE_ROOT) not in sys.path:
sys.path.insert(0, str(RELEASE_ROOT))
from govoplan_release.version_metadata import ( # noqa: E402
apply_version_metadata_mutations,
version_metadata_mutations,
)
class ReleaseVersionMetadataTests(unittest.TestCase):
def test_updates_recognized_metadata_without_changing_interface_versions(
self,
) -> None:
with tempfile.TemporaryDirectory() as temp_dir:
root = Path(temp_dir)
package = root / "src" / "govoplan_example"
backend = package / "backend"
webui = root / "webui"
backend.mkdir(parents=True)
webui.mkdir()
(root / "pyproject.toml").write_text(
'[project]\nname = "govoplan-example"\nversion = "1.2.3"\n',
encoding="utf-8",
)
(root / "package.json").write_text(
'{"name":"root","version":"1.2.3"}\n',
encoding="utf-8",
)
(webui / "package.json").write_text(
'{"name":"@govoplan/example-webui","version":"1.2.3"}\n',
encoding="utf-8",
)
(webui / "package-lock.json").write_text(
json.dumps(
{
"name": "@govoplan/example-webui",
"version": "1.2.3",
"packages": {
"": {
"name": "@govoplan/example-webui",
"version": "1.2.3",
}
},
}
)
+ "\n",
encoding="utf-8",
)
(backend / "manifest.py").write_text(
"def get_manifest():\n"
" return ModuleManifest(\n"
' id="example",\n'
' version="1.2.3",\n'
" provides_interfaces=(\n"
' ModuleInterfaceProvider(name="example.api", version="4.0"),\n'
" ),\n"
" )\n",
encoding="utf-8",
)
(package / "__init__.py").write_text(
'__version__ = "1.2.3"\n',
encoding="utf-8",
)
preview = version_metadata_mutations(
root,
target_version="1.2.4",
)
changed = apply_version_metadata_mutations(
root,
target_version="1.2.4",
)
self.assertEqual(
{mutation.path for mutation in preview},
set(changed),
)
self.assertEqual(
"1.2.4",
json.loads((webui / "package-lock.json").read_text())[
"packages"
][""]["version"],
)
manifest = (backend / "manifest.py").read_text(encoding="utf-8")
self.assertIn('version="1.2.4"', manifest)
self.assertIn('version="4.0"', manifest)
self.assertEqual(
'__version__ = "1.2.4"\n',
(package / "__init__.py").read_text(encoding="utf-8"),
)
if __name__ == "__main__":
unittest.main()
+393
View File
@@ -0,0 +1,393 @@
from __future__ import annotations
import importlib.util
import json
from pathlib import Path
import sys
import tempfile
import unittest
from unittest.mock import patch
META_ROOT = Path(__file__).resolve().parents[1]
SCRIPT = META_ROOT / "tools" / "repo" / "bootstrap-repositories.py"
def load_bootstrap_module():
spec = importlib.util.spec_from_file_location("bootstrap_repositories", SCRIPT)
if spec is None or spec.loader is None:
raise RuntimeError(f"Could not load {SCRIPT}")
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
class RepositoryBootstrapTests(unittest.TestCase):
def test_public_https_transport_rewrites_registered_gitea_remotes(self) -> None:
bootstrap = load_bootstrap_module()
self.assertEqual(
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
bootstrap.clone_remote(
"git@git.add-ideas.de:GovOPlaN/govoplan-core.git",
transport=bootstrap.PUBLIC_HTTPS_TRANSPORT,
),
)
self.assertEqual(
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
bootstrap.clone_remote(
"ssh://git@git.add-ideas.de/GovOPlaN/govoplan-core.git",
transport=bootstrap.PUBLIC_HTTPS_TRANSPORT,
),
)
self.assertEqual(
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
bootstrap.clone_remote(
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
transport=bootstrap.PUBLIC_HTTPS_TRANSPORT,
),
)
def test_registered_transport_preserves_the_manifest_remote(self) -> None:
bootstrap = load_bootstrap_module()
remote = "git@example.test:private/repository.git"
self.assertEqual(
remote,
bootstrap.clone_remote(
remote,
transport=bootstrap.REGISTERED_TRANSPORT,
),
)
def test_public_https_transport_fails_closed_for_other_hosts(self) -> None:
bootstrap = load_bootstrap_module()
unsafe_remotes = (
"git@example.test:GovOPlaN/govoplan-core.git",
"https://token@git.add-ideas.de/GovOPlaN/govoplan-core.git",
"https://git.add-ideas.de:443/GovOPlaN/govoplan-core.git",
"https://git.add-ideas.de/add-ideas/../govoplan-core.git",
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git?ref=main",
"ssh://root@git.add-ideas.de/GovOPlaN/govoplan-core.git",
)
for remote in unsafe_remotes:
with self.subTest(remote=remote), self.assertRaises(ValueError):
bootstrap.clone_remote(
remote,
transport=bootstrap.PUBLIC_HTTPS_TRANSPORT,
)
def test_main_clones_missing_repositories_over_public_https(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": "govoplan-core",
"path": "govoplan-core",
"remote": (
"git@git.add-ideas.de:"
"GovOPlaN/govoplan-core.git"
),
}
],
}
),
encoding="utf-8",
)
with (
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
):
status = bootstrap.main(
[
"--parent",
str(parent),
"--transport",
bootstrap.PUBLIC_HTTPS_TRANSPORT,
]
)
self.assertEqual(0, status)
runner.assert_called_once_with(
[
"git",
"-c",
"credential.helper=",
"clone",
"--",
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
str(parent / "govoplan-core"),
],
check=True,
env=runner.call_args.kwargs["env"],
)
environment = runner.call_args.kwargs["env"]
self.assertEqual("/bin/false", environment["GIT_ASKPASS"])
self.assertEqual("0", environment["GIT_TERMINAL_PROMPT"])
def test_main_validates_every_remote_before_cloning(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": "govoplan-core",
"path": "govoplan-core",
"remote": (
"git@git.add-ideas.de:"
"GovOPlaN/govoplan-core.git"
),
},
{
"name": "unsafe",
"path": "unsafe",
"remote": "git@example.test:private/unsafe.git",
},
],
}
),
encoding="utf-8",
)
with (
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
self.assertRaises(ValueError),
):
bootstrap.main(
[
"--parent",
str(parent),
"--transport",
bootstrap.PUBLIC_HTTPS_TRANSPORT,
]
)
runner.assert_not_called()
def test_main_preserves_an_explicit_private_repository_transport(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": "website",
"path": "website",
"remote": (
"git@git.add-ideas.de:"
"add-ideas/addideas-govoplan-website.git"
),
"bootstrap_transport": (
bootstrap.REGISTERED_TRANSPORT
),
}
],
}
),
encoding="utf-8",
)
with (
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
):
status = bootstrap.main(
[
"--parent",
str(parent),
"--transport",
bootstrap.PUBLIC_HTTPS_TRANSPORT,
]
)
self.assertEqual(0, status)
command = runner.call_args.args[0]
self.assertEqual(
"git@git.add-ideas.de:add-ideas/addideas-govoplan-website.git",
command[-2],
)
def test_main_limits_bootstrap_to_selected_repositories(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": name,
"path": name,
"remote": (
"git@git.add-ideas.de:GovOPlaN/"
f"{name}.git"
),
}
for name in ("govoplan-core", "govoplan-poll")
],
}
),
encoding="utf-8",
)
for repository_filter in (
["--repo", "govoplan-core"],
["--exclude-repo", "govoplan-poll"],
):
with (
self.subTest(repository_filter=repository_filter),
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
):
status = bootstrap.main(
[
"--parent",
str(parent),
"--transport",
bootstrap.PUBLIC_HTTPS_TRANSPORT,
*repository_filter,
]
)
self.assertEqual(0, status)
runner.assert_called_once()
self.assertEqual(
"https://git.add-ideas.de/GovOPlaN/govoplan-core.git",
runner.call_args.args[0][-2],
)
def test_main_rejects_unknown_or_conflicting_repository_filters(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": "govoplan-core",
"path": "govoplan-core",
"remote": (
"git@git.add-ideas.de:"
"GovOPlaN/govoplan-core.git"
),
}
],
}
),
encoding="utf-8",
)
with (
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
):
with self.assertRaisesRegex(ValueError, "unknown registered"):
bootstrap.main(["--repo", "govoplan-missing"])
with self.assertRaisesRegex(ValueError, "selected and excluded"):
bootstrap.main(
[
"--repo",
"govoplan-core",
"--exclude-repo",
"govoplan-core",
]
)
runner.assert_not_called()
def test_check_reports_missing_without_cloning(self) -> None:
bootstrap = load_bootstrap_module()
with tempfile.TemporaryDirectory(prefix="govoplan-bootstrap-") as directory:
root = Path(directory)
parent = root / "checkouts"
parent.mkdir()
root.joinpath("repositories.json").write_text(
json.dumps(
{
"default_parent": str(parent),
"repositories": [
{
"name": "govoplan-core",
"path": "govoplan-core",
"remote": (
"git@git.add-ideas.de:"
"GovOPlaN/govoplan-core.git"
),
}
],
}
),
encoding="utf-8",
)
with (
patch.object(bootstrap, "ROOT", root),
patch.object(bootstrap.subprocess, "run") as runner,
):
status = bootstrap.main(
[
"--check",
"--parent",
str(parent),
"--transport",
bootstrap.PUBLIC_HTTPS_TRANSPORT,
]
)
self.assertEqual(1, status)
runner.assert_not_called()
def test_anonymous_ci_bootstrap_excludes_registered_transport_repositories(
self,
) -> None:
manifest = json.loads(
(META_ROOT / "repositories.json").read_text(encoding="utf-8")
)
registered_only = {
entry["name"]
for entry in manifest["repositories"]
if entry.get("bootstrap_transport") == "registered"
}
self.assertTrue(registered_only)
for workflow in sorted(
(META_ROOT / ".gitea" / "workflows").glob("*.yml")
):
contents = workflow.read_text(encoding="utf-8")
if (
"bootstrap-repositories.py" not in contents
or "--transport public-https" not in contents
):
continue
with self.subTest(workflow=workflow.name):
for repository in registered_only:
self.assertIn(
f"--exclude-repo {repository}",
contents,
)
if __name__ == "__main__":
unittest.main()
+248
View File
@@ -0,0 +1,248 @@
from __future__ import annotations
import importlib.util
from pathlib import Path
import sys
import unittest
META_ROOT = Path(__file__).resolve().parents[1]
RESOLVER = (
META_ROOT / "tools" / "checks" / "security-audit" / "resolve_workspace_mount.py"
)
def load_resolver_module():
spec = importlib.util.spec_from_file_location(
"security_audit_mount_resolver",
RESOLVER,
)
if spec is None or spec.loader is None:
raise RuntimeError(f"Could not load {RESOLVER}")
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
class SecurityAuditMountResolverTests(unittest.TestCase):
def setUp(self) -> None:
self.resolver = load_resolver_module()
self.root = "/workspace/GovOPlaN/govoplan/govoplan"
self.workspace = "/workspace/GovOPlaN/govoplan"
def test_resolves_only_the_named_workspace_volume_for_both_scopes(self) -> None:
mounts = [
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Destination": "/var/run/docker.sock",
"RW": True,
},
{
"Type": "volume",
"Name": "actions-workspace",
"Source": "/var/lib/docker/volumes/actions-workspace/_data",
"Destination": self.workspace,
"RW": True,
},
{
"Type": "volume",
"Name": "actions-environment",
"Source": "/var/lib/docker/volumes/actions-environment/_data",
"Destination": "/var/run/act",
"RW": True,
},
]
for scope in ("current", "govoplan"):
with self.subTest(scope=scope):
self.assertEqual(
(f"type=volume,source=actions-workspace,target={self.workspace}"),
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope=scope,
reports_dir="audit-reports",
),
)
def test_resolves_a_workspace_bind_without_other_mounts(self) -> None:
mounts = [
{
"Type": "bind",
"Source": "/srv/gitea/actions/task-123",
"Destination": self.workspace,
"RW": True,
},
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Destination": "/var/run/docker.sock",
"RW": True,
},
]
self.assertEqual(
(f"type=bind,source=/srv/gitea/actions/task-123,target={self.workspace}"),
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope="govoplan",
reports_dir="audit-reports",
),
)
def test_nested_repository_mount_is_valid_only_for_current_scope(self) -> None:
mounts = [
{
"Type": "volume",
"Name": "all-repositories",
"Destination": self.workspace,
"RW": True,
},
{
"Type": "volume",
"Name": "current-repository",
"Destination": self.root,
"RW": True,
},
]
self.assertIn(
"source=current-repository",
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope="current",
reports_dir="audit-reports",
),
)
with self.assertRaisesRegex(
self.resolver.MountResolutionError,
"nested job-container mounts",
):
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope="govoplan",
reports_dir="audit-reports",
)
def test_rejects_unsafe_or_unusable_mount_layouts(self) -> None:
cases = {
"missing": [],
"path-boundary": [
{
"Type": "volume",
"Name": "wrong-workspace",
"Destination": "/workspace/GovOPlaN/govoplan-other",
"RW": True,
}
],
"read-only": [
{
"Type": "volume",
"Name": "actions-workspace",
"Destination": self.workspace,
"RW": False,
}
],
"ambiguous": [
{
"Type": "volume",
"Name": name,
"Destination": self.workspace,
"RW": True,
}
for name in ("workspace-one", "workspace-two")
],
"scope-too-narrow": [
{
"Type": "volume",
"Name": "repository-only",
"Destination": self.root,
"RW": True,
}
],
}
for name, mounts in cases.items():
with (
self.subTest(name=name),
self.assertRaises(self.resolver.MountResolutionError),
):
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope="govoplan",
reports_dir="audit-reports",
)
def test_rejects_reports_outside_the_selected_workspace_mount(self) -> None:
mounts = [
{
"Type": "volume",
"Name": "actions-workspace",
"Destination": self.workspace,
"RW": True,
}
]
with self.assertRaisesRegex(
self.resolver.MountResolutionError,
"reports path .* outside",
):
self.resolver.resolve_workspace_mount(
mounts,
root=self.root,
scope="current",
reports_dir="/tmp/audit-reports",
)
def test_rejects_broad_or_sensitive_workspace_bind_sources(self) -> None:
for source in (
"/",
"/home",
"/var/run/docker.sock",
"/srv/../etc/shadow",
):
with (
self.subTest(source=source),
self.assertRaisesRegex(
self.resolver.MountResolutionError,
"too broad or sensitive",
),
):
self.resolver.resolve_workspace_mount(
[
{
"Type": "bind",
"Source": source,
"Destination": self.workspace,
"RW": True,
}
],
root=self.root,
scope="govoplan",
reports_dir="audit-reports",
)
with self.assertRaises(self.resolver.MountResolutionError):
self.resolver.resolve_workspace_mount(
[
{
"Type": "bind",
"Source": "//var/lib/workspace",
"Destination": self.workspace,
"RW": True,
}
],
root=self.root,
scope="govoplan",
reports_dir="audit-reports",
)
if __name__ == "__main__":
unittest.main()
+245
View File
@@ -12,6 +12,7 @@ import unittest
META_ROOT = Path(__file__).resolve().parents[1]
AUDIT_SCRIPT = META_ROOT / "tools" / "checks" / "check-security-audit.sh"
CONTAINER_RUNNER = META_ROOT / "tools" / "checks" / "security-audit" / "run.sh"
class SecurityAuditWrapperTests(unittest.TestCase):
@@ -252,6 +253,12 @@ class SecurityAuditWrapperTests(unittest.TestCase):
self.assertEqual(0, manifest["overall_status"])
self.assertEqual(1, manifest["finding_status"])
self.assertEqual(0, manifest["execution_error_status"])
self.assertEqual("complete", manifest["coverage_status"])
scanner_results = {
result["id"]: result["status"]
for result in manifest["scanner_coverage"]["results"]
}
self.assertEqual("findings", scanner_results["semgrep"])
self.assertIn(
"Semgrep SAST\t1\tfindings", (reports / "step-status.tsv").read_text()
)
@@ -311,6 +318,86 @@ class SecurityAuditWrapperTests(unittest.TestCase):
)
self.assertEqual(1, self._manifest(invalid_reports)["execution_error_status"])
def test_missing_tool_is_machine_readable_and_strictly_enforced(self) -> None:
gitleaks_stub = self.stub_bin / "gitleaks"
disabled_stub = self.stub_bin / "gitleaks.disabled"
gitleaks_stub.rename(disabled_stub)
try:
result, reports = self._run(
"--report-only",
SECURITY_AUDIT_REQUIRE_TOOLS="0",
CI="false",
GITEA_ACTIONS="false",
)
self.assertEqual(0, result.returncode, result.stderr)
manifest = self._manifest(reports)
self.assertEqual("incomplete", manifest["coverage_status"])
self.assertEqual(["gitleaks"], manifest["scanner_coverage"]["incomplete"])
scanner_results = {
item["id"]: item for item in manifest["scanner_coverage"]["results"]
}
self.assertEqual("skipped", scanner_results["gitleaks"]["status"])
self.assertEqual(127, scanner_results["gitleaks"]["exit_code"])
self.assertIn(
"gitleaks\tmissing",
(reports / "tool-versions.txt").read_text(encoding="utf-8"),
)
strict_result, strict_reports = self._run(
"--strict",
SECURITY_AUDIT_REQUIRE_TOOLS="0",
CI="false",
GITEA_ACTIONS="false",
)
self.assertEqual(1, strict_result.returncode)
self.assertEqual(
"incomplete",
self._manifest(strict_reports)["coverage_status"],
)
ci_result, ci_reports = self._run(
"--report-only",
SECURITY_AUDIT_REQUIRE_TOOLS="0",
CI="true",
GITEA_ACTIONS="false",
)
self.assertEqual(1, ci_result.returncode)
self.assertEqual(
"incomplete",
self._manifest(ci_reports)["coverage_status"],
)
finally:
disabled_stub.rename(gitleaks_stub)
def test_full_mode_records_every_required_scanner(self) -> None:
self._install_full_mode_stubs()
result, reports = self._run("--report-only", mode="full")
self.assertEqual(0, result.returncode, result.stderr)
manifest = self._manifest(reports)
self.assertEqual("complete", manifest["coverage_status"])
required = manifest["scanner_coverage"]["required"]
results = manifest["scanner_coverage"]["results"]
self.assertEqual(
{
"semgrep",
"bandit",
"ruff-security",
"gitleaks",
"trivy",
"pip-audit",
"npm-audit",
"osv-scanner",
"jscpd",
"radon",
"xenon",
},
set(required),
)
self.assertEqual(set(required), {item["id"] for item in results})
self.assertTrue(all(item["status"] == "no-findings" for item in results))
def test_workspace_mutation_invalidates_the_audit(self) -> None:
result, reports = self._run(
"--report-only",
@@ -354,5 +441,163 @@ class SecurityAuditWrapperTests(unittest.TestCase):
self.assertEqual(set(manifest["reports"]), checksummed_paths)
class SecurityAuditContainerRunnerTests(unittest.TestCase):
def setUp(self) -> None:
self._temporary_directory = tempfile.TemporaryDirectory(
prefix="govoplan-audit-runner-"
)
root = Path(self._temporary_directory.name)
self.stub_bin = root / "bin"
self.stub_bin.mkdir()
self.docker_log = root / "docker.jsonl"
docker_stub = self.stub_bin / "docker"
docker_stub.write_text(
textwrap.dedent(
"""\
#!/usr/bin/env python3
import json
import os
from pathlib import Path
import sys
arguments = sys.argv[1:]
with Path(os.environ["DOCKER_STUB_LOG"]).open(
"a", encoding="utf-8"
) as handle:
handle.write(json.dumps(arguments) + "\\n")
if arguments and arguments[0] == "version":
print("26.1.0")
if arguments[:2] == ["container", "inspect"]:
print(os.environ["DOCKER_STUB_MOUNTS"])
"""
),
encoding="utf-8",
)
docker_stub.chmod(0o755)
self.environment = os.environ.copy()
self.environment.update(
{
"DOCKER_STUB_LOG": str(self.docker_log),
"DOCKER_STUB_MOUNTS": "[]",
"PATH": f"{self.stub_bin}:{self.environment['PATH']}",
}
)
def tearDown(self) -> None:
self._temporary_directory.cleanup()
def _run(
self,
*,
scope: str,
actions_mounts: object | None = None,
) -> tuple[subprocess.CompletedProcess[str], list[list[str]]]:
self.docker_log.write_text("", encoding="utf-8")
environment = self.environment.copy()
if actions_mounts is None:
environment.pop("GITEA_ACTIONS", None)
else:
environment["GITEA_ACTIONS"] = "true"
environment["DOCKER_STUB_MOUNTS"] = json.dumps(actions_mounts)
result = subprocess.run(
[
"bash",
str(CONTAINER_RUNNER),
"--mode",
"quick",
"--scope",
scope,
],
cwd=META_ROOT,
env=environment,
check=False,
capture_output=True,
text=True,
)
commands = [
json.loads(line)
for line in self.docker_log.read_text(encoding="utf-8").splitlines()
]
return result, commands
def test_gitea_job_shares_only_its_workspace_mount(self) -> None:
mounts = [
{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Destination": "/var/run/docker.sock",
"RW": True,
},
{
"Type": "volume",
"Name": "govoplan-actions-workspace",
"Destination": str(META_ROOT.parent),
"RW": True,
},
{
"Type": "volume",
"Name": "govoplan-actions-environment",
"Destination": "/var/run/act",
"RW": True,
},
]
container_id = subprocess.run(
["hostname"],
check=True,
capture_output=True,
text=True,
).stdout.strip()
for scope in ("current", "govoplan"):
with self.subTest(scope=scope):
result, commands = self._run(scope=scope, actions_mounts=mounts)
self.assertEqual(0, result.returncode, result.stderr)
run_command = next(
command for command in commands if command[0] == "run"
)
inspect_command = next(
command
for command in commands
if command[:2] == ["container", "inspect"]
)
self.assertEqual(container_id, inspect_command[-1])
mount_index = run_command.index("--mount")
self.assertEqual(
(
"type=volume,source=govoplan-actions-workspace,"
f"target={META_ROOT.parent}"
),
run_command[mount_index + 1],
)
self.assertNotIn("--volumes-from", run_command)
self.assertNotIn("-v", run_command)
self.assertNotIn("/var/run/docker.sock", " ".join(run_command))
self.assertNotIn("/var/run/act", " ".join(run_command))
repository_roots = [
value
for value in run_command
if value.startswith("GOVOPLAN_REPOS_ROOT=")
]
expected_roots = (
[f"GOVOPLAN_REPOS_ROOT={META_ROOT.parent}"]
if scope == "govoplan"
else []
)
self.assertEqual(expected_roots, repository_roots)
self.assertIn("SECURITY_AUDIT_REQUIRE_TOOLS=1", run_command)
def test_non_actions_runner_keeps_the_scoped_bind_mount(self) -> None:
result, commands = self._run(scope="govoplan")
self.assertEqual(0, result.returncode, result.stderr)
run_command = next(command for command in commands if command[0] == "run")
bind_index = run_command.index("-v")
self.assertEqual(
f"{META_ROOT.parent}:/workspace",
run_command[bind_index + 1],
)
self.assertNotIn("--mount", run_command)
if __name__ == "__main__":
unittest.main()
+64
View File
@@ -6,6 +6,7 @@ import subprocess
import sys
import tempfile
import unittest
from unittest.mock import patch
META_ROOT = Path(__file__).resolve().parents[1]
@@ -20,6 +21,7 @@ from govoplan_release.version_alignment import ( # noqa: E402
repository_version_issues,
selected_repository_version_issues,
)
from govoplan_release.git_state import sanitized_git_environment # noqa: E402
from govoplan_release.model import ( # noqa: E402
CatalogSnapshot,
DashboardSummary,
@@ -398,6 +400,68 @@ class VersionAlignmentTests(unittest.TestCase):
)
self.assertTrue(all("must contain" in issue.message for issue in issues))
def test_annotated_release_tags_work_with_sanitized_git_configuration(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
workspace = Path(tmp)
meta = workspace / "govoplan"
core = workspace / "govoplan-core"
module = workspace / "govoplan-example"
meta.mkdir()
(core / "webui").mkdir(parents=True)
(module / "webui").mkdir(parents=True)
backend_ref = "git+ssh://git@example.test/acme/govoplan-example.git@v1.2.3"
webui_ref = "git+ssh://git@example.test/acme/govoplan-example.git#v1.2.3"
(meta / "requirements-release.txt").write_text(f"govoplan-example @ {backend_ref}\n")
(module / "pyproject.toml").write_text('[project]\nname="govoplan-example"\nversion="1.2.3"\n')
(module / "webui" / "package.json").write_text(
'{"name":"@govoplan/example-webui","version":"1.2.3"}\n'
)
subprocess.run(["git", "init", "-q", str(module)], check=True)
subprocess.run(["git", "-C", str(module), "config", "user.email", "test@example.test"], check=True)
subprocess.run(["git", "-C", str(module), "config", "user.name", "Test"], check=True)
subprocess.run(["git", "-C", str(module), "add", "pyproject.toml", "webui/package.json"], check=True)
subprocess.run(["git", "-C", str(module), "commit", "-qm", "release"], check=True)
subprocess.run(
["git", "-C", str(module), "tag", "-a", "v1.2.3", "-m", "Release v1.2.3"],
check=True,
)
tagged_commit = subprocess.run(
["git", "-C", str(module), "rev-parse", "refs/tags/v1.2.3^{commit}"],
check=True,
text=True,
stdout=subprocess.PIPE,
).stdout.strip()
dependencies = {"@govoplan/example-webui": webui_ref}
(core / "pyproject.toml").write_text('[project]\nname="govoplan-core"\nversion="2.0.0"\n')
(core / "webui" / "package.release.json").write_text(
json.dumps({"version": "2.0.0", "dependencies": dependencies})
)
(core / "webui" / "package-lock.release.json").write_text(
json.dumps(
{
"packages": {
"": {"version": "2.0.0", "dependencies": dependencies},
"node_modules/@govoplan/example-webui": {
"version": "1.2.3",
"resolved": f"git+ssh://git@example.test/acme/govoplan-example.git#{tagged_commit}",
},
}
}
)
)
git_environment = sanitized_git_environment()
git_environment["GIT_TEST_ASSUME_DIFFERENT_OWNER"] = "1"
with patch(
"govoplan_release.version_alignment.sanitized_git_environment",
return_value=git_environment,
):
composition_issues = release_composition_issues(meta, core_root=core)
core_issues = repository_version_issues(core)
self.assertEqual((), composition_issues)
self.assertEqual((), core_issues)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,322 @@
#!/usr/bin/env python3
"""Exercise connector -> datasource -> dataflow publication capabilities."""
from __future__ import annotations
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_connectors.backend.db.models import ConnectorTabularSource
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.dataflows import (
DataflowPublicationTarget,
DataflowRunRequest,
dataflow_run_lifecycle,
)
from govoplan_core.core.automation import AutomationPrincipalResolution
from govoplan_core.core.access import (
CAPABILITY_AUTH_AUTOMATION_PRINCIPAL_PROVIDER,
)
from govoplan_core.core.datasources import (
DatasourceReadRequest,
datasource_catalogue,
datasource_lifecycle,
datasource_publication,
)
from govoplan_core.core.modules import ModuleContext
from govoplan_core.core.tabular_sources import (
TabularSnapshotInput,
tabular_snapshot_writer,
)
from govoplan_core.db.base import Base
from govoplan_core.server.registry import build_platform_registry
from govoplan_dataflow.backend.schemas import (
GraphEdge,
GraphNode,
GraphPosition,
PipelineGraph,
PipelineCreateRequest,
PipelinePreviewRequest,
)
from govoplan_dataflow.backend.db.models import (
DataflowPipeline,
DataflowPipelineRevision,
DataflowRun,
)
from govoplan_dataflow.backend.service import create_pipeline, preview_pipeline
from govoplan_dataflow.backend.run_worker import SqlDataflowRunWorker
from govoplan_datasources.backend.db.models import (
DatasourceMaterializationRecord,
DatasourcePayloadRecord,
DatasourcePayloadRowRecord,
DatasourcePublicationRecord,
DatasourceRecord,
DatasourceStageRecord,
)
def main() -> int:
registry = build_platform_registry(
("connectors", "datasources", "dataflow", "workflow")
)
registry.configure_capability_context(
ModuleContext(registry=registry, settings=object())
)
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(
engine,
tables=[
ConnectorTabularSource.__table__,
DatasourceRecord.__table__,
DatasourcePayloadRecord.__table__,
DatasourcePayloadRowRecord.__table__,
DatasourceMaterializationRecord.__table__,
DatasourceStageRecord.__table__,
DatasourcePublicationRecord.__table__,
DataflowPipeline.__table__,
DataflowPipelineRevision.__table__,
DataflowRun.__table__,
],
)
session_factory = sessionmaker(bind=engine)
with session_factory() as session:
principal = _principal()
writer = tabular_snapshot_writer(registry)
lifecycle = datasource_lifecycle(registry)
catalogue = datasource_catalogue(registry)
publisher = datasource_publication(registry)
runner = dataflow_run_lifecycle(registry)
if (
writer is None
or lifecycle is None
or catalogue is None
or publisher is None
or runner is None
):
raise RuntimeError("Datasource composition capabilities are incomplete.")
origin = writer.create_snapshot(
session,
principal,
snapshot=TabularSnapshotInput(
name="Monthly cases",
source_name="connector_monthly_cases",
rows=(
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
),
),
)
datasource = lifecycle.register_origin(
session,
principal,
origin_ref=origin.ref,
name="Monthly cases cache",
source_name="monthly_cases",
mode="cached",
)
result = preview_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelinePreviewRequest(
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
),
row_limit=100,
),
principal=principal,
registry=registry,
)
expected_rows = [
{"id": 1, "amount": 5},
{"id": 2, "amount": 15},
]
if result.status != "succeeded":
raise RuntimeError(f"Dataflow preview failed: {result.diagnostics}")
if result.rows != expected_rows:
raise RuntimeError(f"Unexpected Dataflow rows: {result.rows!r}")
if result.source_fingerprints[0]["source_ref"] != datasource.ref:
raise RuntimeError("Dataflow lineage did not retain the datasource reference.")
pipeline = create_pipeline(
session,
tenant_id="tenant-1",
actor_id="account-1",
payload=PipelineCreateRequest(
name="Monthly case output",
status="active",
graph=_graph(
datasource_ref=datasource.ref,
fingerprint=datasource.fingerprint,
),
editor_mode="graph",
),
)
run_request = DataflowRunRequest(
pipeline_ref=f"pipeline:{pipeline.id}",
revision=1,
idempotency_key="composition-run-1",
publication=DataflowPublicationTarget(
name="Monthly case result",
source_name="monthly_case_result",
freeze=True,
frozen_label="Composition evidence",
),
)
published = runner.start_run(
session,
principal,
request=run_request,
)
replayed = runner.start_run(
session,
principal,
request=run_request,
)
if published.status != "queued":
raise RuntimeError(
f"Dataflow run was not queued: {published.status}"
)
worker = SqlDataflowRunWorker(
registry=_AutomationRegistry(registry, principal)
)
worker_result = worker.dispatch_pending(
session,
worker_id="composition-worker",
)
if worker_result["succeeded"] != 1:
raise RuntimeError(
f"Dataflow worker failed: {worker_result!r}"
)
completed = runner.get_run(
session,
principal,
run_ref=published.ref,
)
if completed is None:
raise RuntimeError("Dataflow run evidence disappeared.")
published = completed
if published.status != "succeeded":
raise RuntimeError(f"Dataflow publication failed: {published.error}")
if replayed.ref != published.ref or not replayed.replayed:
raise RuntimeError("Dataflow run idempotency did not replay the prior run.")
if (
not published.output_datasource_ref
or not published.output_materialization_ref
):
raise RuntimeError("Dataflow publication did not retain output references.")
output = catalogue.read_datasource(
session,
principal,
request=DatasourceReadRequest(
datasource_ref=published.output_datasource_ref,
),
)
if list(output.rows) != expected_rows:
raise RuntimeError(
f"Unexpected published Dataflow rows: {list(output.rows)!r}"
)
if (
output.materialization is None
or output.materialization.ref != published.output_materialization_ref
or output.materialization.frozen_at is None
):
raise RuntimeError(
"Published Datasource materialization is not pinned and frozen."
)
engine.dispose()
print(
"Connector -> Datasources -> pinned Dataflow publication composition passed."
)
return 0
class _AutomationProvider:
def __init__(self, principal: ApiPrincipal) -> None:
self.principal = principal
def resolve_automation_principal(self, _session, *, request):
return AutomationPrincipalResolution(
allowed=True,
principal=self.principal,
granted_scopes=request.grant_scopes,
provenance={"status": "composition_recheck"},
)
class _AutomationRegistry:
def __init__(self, registry, principal: ApiPrincipal) -> None:
self.registry = registry
self.provider = _AutomationProvider(principal)
def has_capability(self, name: str) -> bool:
return (
name == CAPABILITY_AUTH_AUTOMATION_PRINCIPAL_PROVIDER
or self.registry.has_capability(name)
)
def capability(self, name: str):
if name == CAPABILITY_AUTH_AUTOMATION_PRINCIPAL_PROVIDER:
return self.provider
return self.registry.capability(name)
def _principal() -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id="tenant-1",
scopes=frozenset(
{
"connectors:source:read",
"connectors:source:write",
"datasources:catalogue:read",
"datasources:source:write",
"datasources:stage:write",
"dataflow:pipeline:run",
}
),
),
account=object(),
user=object(),
)
def _graph(*, datasource_ref: str, fingerprint: str) -> PipelineGraph:
return PipelineGraph(
nodes=[
GraphNode(
id="source",
type="source.reference",
label="Cases",
position=GraphPosition(x=0, y=0),
config={
"source_ref": datasource_ref,
"source_name": "monthly_cases",
"expected_fingerprint": fingerprint,
"consistency": "current",
},
),
GraphNode(
id="output",
type="output",
label="Output",
position=GraphPosition(x=200, y=0),
config={},
),
],
edges=[
GraphEdge(
id="source-output",
source="source",
target="output",
)
],
)
if __name__ == "__main__":
raise SystemExit(main())
+27
View File
@@ -30,6 +30,10 @@ GOVOPLAN_CORE_ROOT="$ROOT" PYTHON="$PYTHON" CHECK_TESTCLIENT_DEPRECATIONS=1 bash
"$PYTHON" "$META_ROOT/tools/checks/check-contracts.py" --no-impact
PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-manifest-shapes.py"
cd "$META_ROOT"
"$PYTHON" -m unittest tests.test_deployment_installer
cd "$ROOT"
"$PYTHON" - <<'PY'
import ast
import pathlib
@@ -69,6 +73,14 @@ PY
"$PYTHON" -c 'import govoplan_core.db.bootstrap; import govoplan_access.backend.admin.service; import govoplan_addresses.backend.manifest; import govoplan_files.backend.router; import govoplan_mail.backend.sending.imap; print("targeted backend imports passed")'
"$META_ROOT/tools/checks/check_dependency_boundaries.py"
"$PYTHON" -m unittest tests.test_module_system
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-connectors/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-datasources/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-dataflow/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-workflow/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-views/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-dashboard/tests
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-postbox/tests
"$PYTHON" "$META_ROOT/tools/checks/check-datasource-composition.py"
"$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-mail/tests
"$PYTHON" -m unittest tests.test_api_smoke.ApiSmokeTests.test_mailbox_message_listing_reports_total_count
@@ -77,6 +89,21 @@ cd "$ROOT/webui"
"$NPM" run test:module-capabilities
"$NPM" run test:module-permutations
cd /mnt/DATA/git/govoplan-dataflow/webui
"$NPM" run test:structure
cd /mnt/DATA/git/govoplan-datasources/webui
"$NPM" run typecheck
cd /mnt/DATA/git/govoplan-workflow/webui
"$NPM" run typecheck
cd /mnt/DATA/git/govoplan-dashboard/webui
"$NPM" run test:dashboard-layout
cd /mnt/DATA/git/govoplan-postbox/webui
"$NPM" run test:ui-structure
cd /mnt/DATA/git/govoplan-mail/webui
"$NPM" run test:mail-ui
+4 -1
View File
@@ -19,7 +19,10 @@ fi
cd "$ROOT"
"$PYTHON" "$META_ROOT/tools/checks/check-contracts.py" --no-impact
"$PYTHON" -m unittest tests.test_module_system tests.test_access_contracts
"$PYTHON" "$META_ROOT/tools/checks/release_integration.py" artifacts \
--requirements "$META_ROOT/requirements-release.txt"
"$PYTHON" "$META_ROOT/tools/checks/release_integration.py" core-tests \
--core-root "$ROOT"
"$PYTHON" "$META_ROOT/tools/checks/check_dependency_boundaries.py"
cd "$ROOT/webui"
+5 -22
View File
@@ -60,24 +60,6 @@ retry() {
done
}
run_discovered_tests() {
local suite_dir="$1"
local status
if ! find "$suite_dir" -type f -name 'test*.py' -print -quit | grep -q .; then
echo "No unittest test files found under $suite_dir; skipping."
return 0
fi
set +e
"$PYTHON" -m unittest discover -s "$suite_dir"
status=$?
set -e
if [[ "$status" == 5 ]]; then
echo "No unittest tests discovered under $suite_dir; skipping."
return 0
fi
return "$status"
}
install_cloned_webui_dependencies() {
local webui_dir="$1"
if [[ ! -f "$webui_dir/package.json" ]]; then
@@ -244,7 +226,7 @@ PY
run_step "Clone release module test sources"
for repo in govoplan-mail govoplan-calendar govoplan-campaign; do
repo_tag="$(release_tag_for_package "$repo")"
git clone --depth 1 --branch "$repo_tag" "git@git.add-ideas.de:add-ideas/${repo}.git" "$WORK_ROOT/$repo"
git clone --depth 1 --branch "$repo_tag" "git@git.add-ideas.de:GovOPlaN/${repo}.git" "$WORK_ROOT/$repo"
done
run_step "Run core backend release tests"
@@ -253,9 +235,10 @@ run_step "Run core backend release tests"
"$PYTHON" "$META_ROOT/tools/checks/check_dependency_boundaries.py"
run_step "Run cloned module backend tests"
run_discovered_tests "$WORK_ROOT/govoplan-mail/tests"
run_discovered_tests "$WORK_ROOT/govoplan-calendar/tests"
run_discovered_tests "$WORK_ROOT/govoplan-campaign/tests"
for repo in govoplan-mail govoplan-calendar govoplan-campaign; do
"$PYTHON" "$META_ROOT/tools/checks/release_integration.py" module-tests \
--module-root "$WORK_ROOT/$repo"
done
run_step "Run core WebUI release tests and build"
cd "$ROOT/webui"
+264 -42
View File
@@ -87,6 +87,37 @@ for flag_name in FAIL_ON_FINDINGS REQUIRE_TOOLS; do
fi
done
# A strict audit is only meaningful when every scanner in the selected mode is
# available. CI must enforce the same coverage even while findings are
# temporarily report-only.
if [[ "$FAIL_ON_FINDINGS" == "1" \
|| "${CI:-false}" == "true" \
|| "${GITEA_ACTIONS:-false}" == "true" ]]; then
REQUIRE_TOOLS=1
fi
declare -a REQUIRED_SCANNERS=(
semgrep
bandit
ruff-security
gitleaks
)
if [[ "$MODE" != "quick" ]]; then
REQUIRED_SCANNERS+=(
trivy
pip-audit
npm-audit
)
fi
if [[ "$MODE" == "full" ]]; then
REQUIRED_SCANNERS+=(
osv-scanner
jscpd
radon
xenon
)
fi
declare -a REPOS=()
if [[ "$SCOPE" == "govoplan" ]]; then
REPOS_ROOT="${GOVOPLAN_REPOS_ROOT:-$(dirname "$ROOT")}"
@@ -138,12 +169,14 @@ overall_status=0
finding_status=0
missing_status=0
execution_status=0
coverage_complete=true
audit_started_at="$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
audit_completed_at=""
workspace_unchanged="unknown"
audit_toolbox_fingerprint="${SECURITY_AUDIT_TOOLBOX_FINGERPRINT:-direct-host}"
declare -A REPORT_FILES=()
declare -A MACHINE_REPORTS=()
declare -A SCANNER_TERMINAL_STATUSES=()
register_report() {
local path="$1"
@@ -169,7 +202,9 @@ prepare_machine_report() {
fi
}
run_step() {
run_step_internal() {
local scanner_id="$1"
shift
local name="$1"
local exit_contract="$2"
shift 2
@@ -178,10 +213,12 @@ run_step() {
"$@"
local status=$?
local outcome="passed"
local scanner_outcome="no-findings"
if [[ "$status" -eq 0 ]]; then
:
elif [[ "$exit_contract" == "findings-exit-one" && "$status" -eq 1 ]]; then
outcome="findings"
scanner_outcome="findings"
echo "Audit step reported findings: $name (exit $status)" >&2
finding_status=1
if [[ "$FAIL_ON_FINDINGS" == "1" ]]; then
@@ -189,21 +226,77 @@ run_step() {
fi
else
outcome="execution-error"
scanner_outcome="scanner-failure"
echo "Audit step failed to execute successfully: $name (exit $status)" >&2
execution_status=1
overall_status=1
fi
printf '%s\t%s\t%s\n' "$name" "$status" "$outcome" >> "$REPORTS_DIR/step-status.tsv"
if [[ -n "$scanner_id" ]]; then
SCANNER_TERMINAL_STATUSES["$scanner_id"]="$scanner_outcome"
if [[ "$scanner_outcome" == "scanner-failure" ]]; then
coverage_complete=false
fi
printf '%s\t%s\t%s\t%s\n' \
"$scanner_id" \
"$name" \
"$status" \
"$scanner_outcome" \
>> "$REPORTS_DIR/scanner-status.tsv"
fi
}
run_step() {
run_step_internal "" "$@"
}
run_scanner_step() {
local scanner_id="$1"
shift
run_step_internal "$scanner_id" "$@"
}
skip_or_fail_missing() {
local tool="$1"
local scanner_id="$1"
local tool="$2"
echo "Skipping $tool: command not found. Use tools/checks/security-audit/run.sh for the containerized toolbox." >&2
missing_status=1
coverage_complete=false
SCANNER_TERMINAL_STATUSES["$scanner_id"]="skipped"
if [[ "$REQUIRE_TOOLS" == "1" ]]; then
overall_status=1
fi
printf '%s\t127\tmissing\n' "$tool" >> "$REPORTS_DIR/step-status.tsv"
printf '%s\t%s\t127\tskipped\n' \
"$scanner_id" \
"$tool" \
>> "$REPORTS_DIR/scanner-status.tsv"
}
validate_scanner_status_contract() {
local scanner_id
local scanner_status
local invalid_status=0
for scanner_id in "${REQUIRED_SCANNERS[@]}"; do
scanner_status="${SCANNER_TERMINAL_STATUSES[$scanner_id]:-}"
case "$scanner_status" in
no-findings|findings) ;;
scanner-failure|skipped)
coverage_complete=false
;;
"")
echo "Required scanner did not record a terminal status: $scanner_id" >&2
coverage_complete=false
invalid_status=2
;;
*)
echo "Required scanner recorded an invalid status: $scanner_id ($scanner_status)" >&2
coverage_complete=false
invalid_status=2
;;
esac
done
return "$invalid_status"
}
write_manifest() {
@@ -229,7 +322,10 @@ write_manifest() {
"$finding_status" \
"$execution_status" \
"$missing_status" \
"$coverage_complete" \
"$audit_toolbox_fingerprint" \
"${#REQUIRED_SCANNERS[@]}" \
"${REQUIRED_SCANNERS[@]}" \
"${#REPOS[@]}" \
"${REPOS[@]}" \
"${#present_reports[@]}" \
@@ -254,16 +350,51 @@ import sys
finding_status,
execution_status,
missing_status,
coverage_complete,
toolbox_fingerprint,
repository_count,
*remaining,
) = sys.argv[1:]
required_scanner_count = int(remaining[0])
required_scanners = remaining[1 : required_scanner_count + 1]
remaining = remaining[required_scanner_count + 1 :]
repository_count = remaining[0]
remaining = remaining[1:]
repo_count = int(repository_count)
repositories = remaining[:repo_count]
remaining = remaining[repo_count:]
report_count = int(remaining[0])
reports = remaining[1 : report_count + 1]
expected_reports = remaining[report_count + 1 :]
scanner_results = []
scanner_status_path = Path(manifest_path).with_name("scanner-status.tsv")
if scanner_status_path.is_file():
for line_number, raw_line in enumerate(
scanner_status_path.read_text(encoding="utf-8").splitlines(),
start=1,
):
fields = raw_line.split("\t")
if len(fields) != 4:
raise SystemExit(
f"{scanner_status_path}:{line_number}: expected four fields"
)
scanner_id, name, exit_code, status = fields
scanner_results.append(
{
"id": scanner_id,
"name": name,
"exit_code": int(exit_code),
"status": status,
}
)
result_ids = {result["id"] for result in scanner_results}
incomplete_scanners = sorted(
{
result["id"]
for result in scanner_results
if result["status"] in {"scanner-failure", "skipped"}
}
| (set(required_scanners) - result_ids)
)
payload = {
"mode": mode,
"scope": scope,
@@ -276,6 +407,12 @@ payload = {
"finding_status": int(finding_status),
"execution_error_status": int(execution_status),
"missing_tool_status": int(missing_status),
"coverage_status": "complete" if coverage_complete == "true" else "incomplete",
"scanner_coverage": {
"required": required_scanners,
"results": scanner_results,
"incomplete": incomplete_scanners,
},
"tool_versions": "tool-versions.txt",
"workspace_state_start": "workspace-state-start.tsv",
"workspace_state_end": "workspace-state-end.tsv",
@@ -350,7 +487,10 @@ write_tool_versions() {
: > "$destination"
local tool
for tool in semgrep bandit ruff gitleaks trivy pip-audit npm osv-scanner jscpd radon xenon; do
has_tool "$tool" || continue
if ! has_tool "$tool"; then
printf '%s\t%s\n' "$tool" "missing" >> "$destination"
continue
fi
if [[ "$tool" == "gitleaks" ]]; then
version="$("$tool" version 2>&1)"
else
@@ -496,7 +636,12 @@ run_bandit() {
fi
if [[ "${#PY_TEST_ROOTS[@]}" -gt 0 ]]; then
prepare_machine_report "$REPORTS_DIR/bandit-tests.json" || return 2
bandit -r "${PY_TEST_ROOTS[@]}" -f json -o "$REPORTS_DIR/bandit-tests.json"
# Tests intentionally use assertions and synthetic credentials. Keep the
# separate test scan useful by excluding only those test-specific signals.
bandit -r "${PY_TEST_ROOTS[@]}" \
--skip B101,B105,B106,B107 \
-f json \
-o "$REPORTS_DIR/bandit-tests.json"
local test_status=$?
[[ "$test_status" -le 1 ]] || status=2
fi
@@ -512,7 +657,11 @@ run_ruff_security() {
fi
if [[ "${#PY_TEST_ROOTS[@]}" -gt 0 ]]; then
prepare_machine_report "$REPORTS_DIR/ruff-security-tests.json" || return 2
ruff check --select S --output-format json "${PY_TEST_ROOTS[@]}" > "$REPORTS_DIR/ruff-security-tests.json"
ruff check \
--select S \
--ignore S101,S105,S106,S107 \
--output-format json \
"${PY_TEST_ROOTS[@]}" > "$REPORTS_DIR/ruff-security-tests.json"
local test_status=$?
[[ "$test_status" -le 1 ]] || status=2
fi
@@ -582,14 +731,39 @@ run_trivy() {
run_pip_audit_manifests() {
local status=0
for repo in "${REPOS[@]}"; do
[[ -f "$repo/requirements.txt" ]] || continue
local name
name="$(safe_name "$repo")"
prepare_machine_report "$REPORTS_DIR/pip-audit-$name.json" || return 2
# Requirements may contain project-relative entries such as `.[server]`.
# Resolve them from the owning repository instead of the meta-repository.
(cd "$repo" && pip-audit -r requirements.txt --progress-spinner off --format json --output "$REPORTS_DIR/pip-audit-$name.json")
accumulate_exit_status status "$?"
local repo_name
repo_name="$(safe_name "$repo")"
while IFS= read -r -d '' requirements_file; do
local manifest_name report_path
manifest_name="$(safe_name "$requirements_file")"
report_path="$REPORTS_DIR/pip-audit-$repo_name-$manifest_name.json"
prepare_machine_report "$report_path" || return 2
# Requirements may contain project-relative entries such as `.[server]`.
# Resolve them from the directory containing the manifest.
(
cd "$(dirname "$requirements_file")" &&
pip-audit \
-r "$(basename "$requirements_file")" \
--progress-spinner off \
--format json \
--output "$report_path"
)
accumulate_exit_status status "$?"
done < <(
find "$repo" \
-type d \( \
-name .git \
-o -name .venv \
-o -name node_modules \
-o -name dist \
-o -name build \
-o -name runtime \
-o -name audit-reports \
-o -name '.*-test-build' \
\) -prune \
-o -type f -name 'requirements*.txt' -print0 |
sort -z
)
done
return "$status"
}
@@ -597,12 +771,44 @@ run_pip_audit_manifests() {
run_npm_audit_manifests() {
local status=0
for repo in "${REPOS[@]}"; do
[[ -f "$repo/webui/package-lock.json" ]] || continue
local name
name="$(safe_name "$repo")"
prepare_machine_report "$REPORTS_DIR/npm-audit-$name.json" || return 2
(cd "$repo/webui" && npm audit --omit=dev --json > "$REPORTS_DIR/npm-audit-$name.json")
accumulate_exit_status status "$?"
local repo_name
repo_name="$(safe_name "$repo")"
while IFS= read -r -d '' lock_file; do
local lock_dir lock_name runtime_report all_report
lock_dir="$(dirname "$lock_file")"
lock_name="$(
printf '%s' "${lock_file#"$repo"/}" |
tr -c 'A-Za-z0-9_.-' '_'
)"
runtime_report="$REPORTS_DIR/npm-audit-runtime-$repo_name-$lock_name.json"
all_report="$REPORTS_DIR/npm-audit-all-$repo_name-$lock_name.json"
prepare_machine_report "$runtime_report" || return 2
(
cd "$lock_dir" &&
npm audit --omit=dev --json > "$runtime_report"
)
accumulate_exit_status status "$?"
prepare_machine_report "$all_report" || return 2
(
cd "$lock_dir" &&
npm audit --json > "$all_report"
)
accumulate_exit_status status "$?"
done < <(
find "$repo" \
-type d \( \
-name .git \
-o -name .venv \
-o -name node_modules \
-o -name dist \
-o -name build \
-o -name runtime \
-o -name audit-reports \
-o -name '.*-test-build' \
\) -prune \
-o -type f -name package-lock.json -print0 |
sort -z
)
done
return "$status"
}
@@ -665,6 +871,19 @@ run_jscpd() {
--output "$REPORTS_DIR/jscpd" \
--ignore "$ignored_path_pattern" \
"${REPOS[@]}"
local jscpd_status=$?
[[ "$jscpd_status" -eq 0 ]] || return 2
python3 - "$REPORTS_DIR/jscpd/jscpd-report.json" <<'PY'
from __future__ import annotations
import json
from pathlib import Path
import sys
report = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
raise SystemExit(1 if report.get("duplicates") else 0)
PY
}
run_radon() {
@@ -680,10 +899,12 @@ run_xenon() {
}
register_report "$REPORTS_DIR/step-status.tsv"
register_report "$REPORTS_DIR/scanner-status.tsv"
register_report "$REPORTS_DIR/tool-versions.txt"
register_report "$REPORTS_DIR/workspace-state-start.tsv"
register_report "$REPORTS_DIR/workspace-state-end.tsv"
: > "$REPORTS_DIR/step-status.tsv"
: > "$REPORTS_DIR/scanner-status.tsv"
if ! write_workspace_state "$REPORTS_DIR/workspace-state-start.tsv"; then
echo "Could not capture the repository state before the audit." >&2
exit 1
@@ -691,75 +912,76 @@ fi
run_step "Record audit tool versions" execution-only write_tool_versions
if has_tool semgrep; then
run_step "Semgrep SAST" findings-exit-one run_semgrep
run_scanner_step semgrep "Semgrep SAST" findings-exit-one run_semgrep
else
skip_or_fail_missing semgrep
skip_or_fail_missing semgrep semgrep
fi
if has_tool bandit; then
run_step "Bandit Python security scan" findings-exit-one run_bandit
run_scanner_step bandit "Bandit Python security scan" findings-exit-one run_bandit
else
skip_or_fail_missing bandit
skip_or_fail_missing bandit bandit
fi
if has_tool ruff; then
run_step "Ruff flake8-bandit security rules" findings-exit-one run_ruff_security
run_scanner_step ruff-security "Ruff flake8-bandit security rules" findings-exit-one run_ruff_security
else
skip_or_fail_missing ruff
skip_or_fail_missing ruff-security ruff
fi
if has_tool gitleaks; then
run_step "Gitleaks secret scan" findings-exit-one run_gitleaks
run_scanner_step gitleaks "Gitleaks secret scan" findings-exit-one run_gitleaks
else
skip_or_fail_missing gitleaks
skip_or_fail_missing gitleaks gitleaks
fi
if [[ "$MODE" != "quick" ]]; then
if has_tool trivy; then
run_step "Trivy filesystem vulnerability/secret/misconfig scan" findings-exit-one run_trivy
run_scanner_step trivy "Trivy filesystem vulnerability/secret/misconfig scan" findings-exit-one run_trivy
else
skip_or_fail_missing trivy
skip_or_fail_missing trivy trivy
fi
if has_tool pip-audit; then
run_step "pip-audit requirements scan" findings-exit-one run_pip_audit_manifests
run_scanner_step pip-audit "pip-audit requirements scan" findings-exit-one run_pip_audit_manifests
else
skip_or_fail_missing pip-audit
skip_or_fail_missing pip-audit pip-audit
fi
if has_tool npm; then
run_step "npm audit lockfile scan" findings-exit-one run_npm_audit_manifests
run_scanner_step npm-audit "npm audit lockfile scan" findings-exit-one run_npm_audit_manifests
else
skip_or_fail_missing npm
skip_or_fail_missing npm-audit npm
fi
fi
if [[ "$MODE" == "full" ]]; then
if has_tool osv-scanner; then
run_step "OSV-Scanner recursive dependency scan" findings-exit-one run_osv_scanner
run_scanner_step osv-scanner "OSV-Scanner recursive dependency scan" findings-exit-one run_osv_scanner
else
skip_or_fail_missing osv-scanner
skip_or_fail_missing osv-scanner osv-scanner
fi
if has_tool jscpd; then
run_step "jscpd duplicate code scan" execution-only run_jscpd
run_scanner_step jscpd "jscpd duplicate code scan" findings-exit-one run_jscpd
else
skip_or_fail_missing jscpd
skip_or_fail_missing jscpd jscpd
fi
if has_tool radon; then
run_step "Radon complexity report" execution-only run_radon
run_scanner_step radon "Radon complexity report" execution-only run_radon
else
skip_or_fail_missing radon
skip_or_fail_missing radon radon
fi
if has_tool xenon; then
run_step "Xenon complexity threshold scan" findings-exit-one run_xenon
run_scanner_step xenon "Xenon complexity threshold scan" findings-exit-one run_xenon
else
skip_or_fail_missing xenon
skip_or_fail_missing xenon xenon
fi
fi
run_step "Validate required scanner coverage records" execution-only validate_scanner_status_contract
run_step "Validate machine-readable audit reports" execution-only validate_machine_reports
echo
@@ -116,6 +116,9 @@ def _check_env(*, database_url: str, modules: str, app_env: str) -> dict[str, st
env.setdefault("FILE_STORAGE_BACKEND", "local")
env.setdefault("FILE_STORAGE_LOCAL_ROOT", str(ROOT / "runtime" / "postgres-check-files"))
env.setdefault("MOCK_MAILBOX_DIR", str(ROOT / "runtime" / "postgres-check-mock-mailbox"))
env.setdefault("CORS_ORIGINS", "http://127.0.0.1:5173,http://localhost:5173")
env.setdefault("GOVOPLAN_TRUSTED_HOSTS", "127.0.0.1,localhost,testserver")
env.setdefault("GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS", "false")
env["PYTHONPATH"] = os.pathsep.join(part for part in (str(SRC), env.get("PYTHONPATH", "")) if part)
return env
+24 -1
View File
@@ -5,6 +5,7 @@ import argparse
import importlib.metadata as metadata
import os
import re
import subprocess
import sys
import tempfile
import unittest
@@ -354,6 +355,24 @@ def run_core_release_tests(core_root: Path) -> int:
return 0 if result.wasSuccessful() else 1
def run_module_release_tests(module_root: Path) -> int:
resolved_root = module_root.resolve()
tests_root = resolved_root / "tests"
if not tests_root.is_dir() or not any(tests_root.rglob("test*.py")):
print(f"No test files found under {tests_root}; skipping.")
return 0
result = subprocess.run(
(sys.executable, "-m", "pytest", "-q", "tests"),
cwd=resolved_root,
check=False,
)
if result.returncode == 5:
print(f"No tests collected under {tests_root}; skipping.")
return 0
return result.returncode
def _parser() -> argparse.ArgumentParser:
meta_root = Path(__file__).resolve().parents[2]
parser = argparse.ArgumentParser(description="Artifact-aware GovOPlaN release integration checks")
@@ -366,6 +385,8 @@ def _parser() -> argparse.ArgumentParser:
)
core_parser = subparsers.add_parser("core-tests", help="Run Core tests that are valid for tagged module artifacts")
core_parser.add_argument("--core-root", type=Path, required=True)
module_parser = subparsers.add_parser("module-tests", help="Run tests from one tagged module source checkout")
module_parser.add_argument("--module-root", type=Path, required=True)
return parser
@@ -373,7 +394,9 @@ def main(argv: Sequence[str] | None = None) -> int:
args = _parser().parse_args(argv)
if args.command == "artifacts":
return run_installed_artifact_checks(args.requirements)
return run_core_release_tests(args.core_root)
if args.command == "core-tests":
return run_core_release_tests(args.core_root)
return run_module_release_tests(args.module_root)
if __name__ == "__main__":
+4 -1
View File
@@ -48,7 +48,10 @@ RUN python -m pip install --upgrade pip \
&& osv-scanner --version \
&& trivy --version \
&& jscpd --version \
&& pip-audit --progress-spinner off
&& pip-audit --progress-spinner off \
&& npm --version \
&& radon --version \
&& xenon --version
RUN mkdir -p /workspace \
&& chmod 0777 /workspace
@@ -0,0 +1,202 @@
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import json
import posixpath
from pathlib import Path
import re
import sys
from typing import Any
VOLUME_NAME = re.compile(r"[A-Za-z0-9][A-Za-z0-9_.-]*")
UNSAFE_BIND_SOURCES = {
Path("/"),
Path("/bin"),
Path("/boot"),
Path("/dev"),
Path("/etc"),
Path("/home"),
Path("/lib"),
Path("/lib64"),
Path("/mnt"),
Path("/opt"),
Path("/proc"),
Path("/root"),
Path("/run"),
Path("/sbin"),
Path("/srv"),
Path("/sys"),
Path("/tmp"),
Path("/usr"),
Path("/var"),
Path("/var/run"),
}
SENSITIVE_BIND_ROOTS = {
Path("/dev"),
Path("/etc"),
Path("/proc"),
Path("/root"),
Path("/run"),
Path("/sys"),
Path("/var/run"),
}
class MountResolutionError(ValueError):
pass
def normalized_absolute_path(
value: str,
*,
field: str,
resolve_symlinks: bool = True,
) -> Path:
if not value.startswith("/") or value.startswith("//"):
raise MountResolutionError(f"{field} must be an absolute path")
path = Path(posixpath.normpath(value))
return path.resolve(strict=False) if resolve_symlinks else path
def contains_path(container: Path, path: Path) -> bool:
return path == container or container in path.parents
def mount_option_value(value: str, *, field: str) -> str:
if not value or any(character in value for character in (",", "\n", "\r")):
raise MountResolutionError(f"{field} cannot be represented safely")
return value
def resolve_workspace_mount(
mounts: Any,
*,
root: str,
scope: str,
reports_dir: str,
) -> str:
if scope not in {"current", "govoplan"}:
raise MountResolutionError(f"unsupported audit scope: {scope}")
if not isinstance(mounts, list):
raise MountResolutionError("Docker mount metadata must be a list")
root_path = normalized_absolute_path(root, field="audit root")
scan_root = root_path if scope == "current" else root_path.parent
reports_path = Path(reports_dir)
if not reports_path.is_absolute():
reports_path = root_path / reports_path
reports_path = reports_path.resolve(strict=False)
parsed_mounts: list[tuple[Path, dict[str, Any]]] = []
for mount in mounts:
if not isinstance(mount, dict) or not isinstance(mount.get("Destination"), str):
continue
destination = normalized_absolute_path(
mount["Destination"],
field="mount destination",
)
parsed_mounts.append((destination, mount))
candidates = [
(destination, mount)
for destination, mount in parsed_mounts
if destination != Path("/") and contains_path(destination, scan_root)
]
if not candidates:
raise MountResolutionError(
f"no job-container mount covers audit scope root {scan_root}"
)
longest_depth = max(len(destination.parts) for destination, _ in candidates)
selected = [
candidate
for candidate in candidates
if len(candidate[0].parts) == longest_depth
]
if len(selected) != 1:
raise MountResolutionError("job-container workspace mount is ambiguous")
destination, mount = selected[0]
nested_mounts = [
nested_destination
for nested_destination, _ in parsed_mounts
if nested_destination != destination
and contains_path(scan_root, nested_destination)
]
if nested_mounts:
raise MountResolutionError(
"nested job-container mounts inside the audit scope cannot be "
"reproduced safely"
)
if mount.get("RW") is not True:
raise MountResolutionError("job-container workspace mount is not writable")
if not contains_path(destination, reports_path):
raise MountResolutionError(
f"audit reports path {reports_path} is outside the workspace mount"
)
destination_value = mount_option_value(
str(destination),
field="mount destination",
)
mount_type = mount.get("Type")
if mount_type == "volume":
name = mount.get("Name")
if not isinstance(name, str) or VOLUME_NAME.fullmatch(name) is None:
raise MountResolutionError("workspace volume name is missing or malformed")
source_value = name
elif mount_type == "bind":
source = mount.get("Source")
if not isinstance(source, str):
raise MountResolutionError("workspace bind source is missing")
source_path = normalized_absolute_path(
source,
field="mount source",
resolve_symlinks=False,
)
if source_path in UNSAFE_BIND_SOURCES or any(
contains_path(sensitive_root, source_path)
for sensitive_root in SENSITIVE_BIND_ROOTS
):
raise MountResolutionError(
"workspace bind source is too broad or sensitive"
)
source_value = mount_option_value(str(source_path), field="mount source")
else:
raise MountResolutionError(f"unsupported workspace mount type: {mount_type!r}")
return f"type={mount_type},source={source_value},target={destination_value}"
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Resolve one safe Gitea Actions workspace mount for an audit container."
)
parser.add_argument("--root", required=True)
parser.add_argument("--scope", choices=("current", "govoplan"), required=True)
parser.add_argument("--reports-dir", required=True)
return parser.parse_args()
def main() -> int:
args = parse_args()
try:
mounts = json.load(sys.stdin)
print(
resolve_workspace_mount(
mounts,
root=args.root,
scope=args.scope,
reports_dir=args.reports_dir,
)
)
except (json.JSONDecodeError, MountResolutionError) as exc:
print(f"workspace mount error: {exc}", file=sys.stderr)
return 2
return 0
if __name__ == "__main__":
raise SystemExit(main())
+47 -3
View File
@@ -70,6 +70,10 @@ done
REBUILD="${REBUILD:-0}"
UPDATE="${UPDATE:-0}"
BUILD_ONLY="${BUILD_ONLY:-0}"
REQUIRE_TOOLS="${SECURITY_AUDIT_REQUIRE_TOOLS:-0}"
if [[ "${CI:-false}" == "true" || "${GITEA_ACTIONS:-false}" == "true" ]]; then
REQUIRE_TOOLS=1
fi
declare -a DOCKER_CLI=()
DOCKER_LOCATION=""
@@ -139,13 +143,53 @@ if [[ "$BUILD_ONLY" == "1" ]]; then
exit 0
fi
if [[ "$SCOPE" == "govoplan" ]]; then
declare -a WORKSPACE_MOUNT=()
if [[ "${GITEA_ACTIONS:-}" == "true" ]]; then
ACTIONS_CONTAINER_ID="$(hostname)"
if ! ACTIONS_MOUNTS_JSON="$(
"${DOCKER_CLI[@]}" container inspect \
--format '{{json .Mounts}}' \
"$ACTIONS_CONTAINER_ID"
)"; then
echo "Gitea Actions is using a host Docker daemon, but the current job container could not be resolved." >&2
echo "Cannot safely share the checked-out workspace with the audit container." >&2
exit 1
fi
if command -v python3 >/dev/null 2>&1; then
MOUNT_PYTHON=python3
elif command -v python >/dev/null 2>&1; then
MOUNT_PYTHON=python
else
echo "Python is required to validate the Gitea Actions workspace mount." >&2
exit 1
fi
if ! ACTIONS_WORKSPACE_MOUNT="$(
printf '%s' "$ACTIONS_MOUNTS_JSON" \
| "$MOUNT_PYTHON" "$ROOT/tools/checks/security-audit/resolve_workspace_mount.py" \
--root "$ROOT" \
--scope "$SCOPE" \
--reports-dir "$REPORTS_DIR"
)"; then
echo "Cannot safely share the Gitea Actions workspace with the audit container." >&2
exit 1
fi
WORKSPACE_MOUNT=(--mount "$ACTIONS_WORKSPACE_MOUNT")
MOUNT_ROOT="$(dirname "$ROOT")"
WORKDIR="$ROOT"
if [[ "$SCOPE" == "govoplan" ]]; then
EXTRA_ENV=(-e "GOVOPLAN_REPOS_ROOT=$MOUNT_ROOT")
else
EXTRA_ENV=()
fi
elif [[ "$SCOPE" == "govoplan" ]]; then
MOUNT_ROOT="$(dirname "$ROOT")"
WORKDIR="/workspace/$(basename "$ROOT")"
WORKSPACE_MOUNT=(-v "$MOUNT_ROOT:/workspace")
EXTRA_ENV=(-e "GOVOPLAN_REPOS_ROOT=/workspace")
else
MOUNT_ROOT="$ROOT"
WORKDIR="/workspace"
WORKSPACE_MOUNT=(-v "$MOUNT_ROOT:/workspace")
EXTRA_ENV=()
fi
@@ -156,10 +200,10 @@ fi
-e SECURITY_AUDIT_MODE="$MODE" \
-e SECURITY_AUDIT_REPORTS_DIR="$REPORTS_DIR" \
-e SECURITY_AUDIT_FAIL_ON_FINDINGS="${SECURITY_AUDIT_FAIL_ON_FINDINGS:-0}" \
-e SECURITY_AUDIT_REQUIRE_TOOLS="${SECURITY_AUDIT_REQUIRE_TOOLS:-0}" \
-e SECURITY_AUDIT_REQUIRE_TOOLS="$REQUIRE_TOOLS" \
-e SECURITY_AUDIT_TOOLBOX_FINGERPRINT="$IMAGE_FINGERPRINT" \
"${EXTRA_ENV[@]}" \
-v "$MOUNT_ROOT:/workspace" \
"${WORKSPACE_MOUNT[@]}" \
-w "$WORKDIR" \
"$FINGERPRINT_IMAGE" \
bash tools/checks/check-security-audit.sh --mode "$MODE" --scope "$SCOPE" --reports-dir "$REPORTS_DIR" "${SCRIPT_ARGS[@]}"
+6
View File
@@ -0,0 +1,6 @@
"""Executable entry point for the single-file GovOPlaN deployer."""
from govoplan_deploy.cli import main
raise SystemExit(main())
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Build the dependency-free GovOPlaN deployer as one executable zipapp."""
from __future__ import annotations
import argparse
from hashlib import sha256
from pathlib import Path
import zipapp
ROOT = Path(__file__).resolve().parent
DEFAULT_OUTPUT = ROOT.parent.parent / "runtime" / "deployment" / "govoplan-deploy.pyz"
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--output", type=Path, default=DEFAULT_OUTPUT)
args = parser.parse_args(argv)
output = args.output.expanduser().resolve()
if output == ROOT or ROOT in output.parents:
raise SystemExit("output must be outside the deployment source directory")
output.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
temporary = output.with_name(f".{output.name}.tmp")
if temporary.exists():
temporary.unlink()
zipapp.create_archive(
ROOT,
target=temporary,
interpreter="/usr/bin/env python3",
compressed=True,
filter=_include_source,
)
temporary.chmod(0o755)
temporary.replace(output)
digest = sha256(output.read_bytes()).hexdigest()
print(f"{digest} {output}")
return 0
def _include_source(path: Path) -> bool:
return (
"__pycache__" not in path.parts
and path.suffix not in {".pyc", ".pyo"}
and path.name != "build-deployer-zipapp.py"
)
if __name__ == "__main__":
raise SystemExit(main())
+18
View File
@@ -0,0 +1,18 @@
#!/usr/bin/env python3
"""GovOPlaN deployment entry point."""
from __future__ import annotations
from pathlib import Path
import sys
TOOLS_ROOT = Path(__file__).resolve().parent
if str(TOOLS_ROOT) not in sys.path:
sys.path.insert(0, str(TOOLS_ROOT))
from govoplan_deploy.cli import main # noqa: E402
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,19 @@
"""Declarative GovOPlaN host deployment tooling."""
from .model import (
BASE_MODULES,
FULL_MODULES,
InstallationSpec,
SpecError,
default_spec,
load_spec,
)
__all__ = [
"BASE_MODULES",
"FULL_MODULES",
"InstallationSpec",
"SpecError",
"default_spec",
"load_spec",
]
+738
View File
@@ -0,0 +1,738 @@
"""Secret handling and deterministic Compose bundle rendering."""
from __future__ import annotations
import base64
from dataclasses import dataclass
from hashlib import sha256
import hmac
import json
import os
from pathlib import Path
import secrets
import stat
from typing import Mapping
from urllib.parse import quote, urlsplit
from .model import ENV_NAME_PATTERN, InstallationSpec
SPEC_FILENAME = "installation.json"
ENV_FILENAME = "secrets.env"
COMPOSE_FILENAME = "compose.json"
GARAGE_CONFIG_FILENAME = "garage.toml"
LOAD_BALANCER_CONFIG_FILENAME = "load-balancer.cfg"
PLAN_FILENAME = "plan.json"
RECEIPT_FILENAME = "receipt.json"
LOCK_FILENAME = ".deployment.lock"
RUNTIME_ENV_KEYS = (
"APP_ENV",
"GOVOPLAN_INSTALL_PROFILE",
"MASTER_KEY_B64",
"DATABASE_URL",
"GOVOPLAN_DATABASE_URL_PGTOOLS",
"ENABLED_MODULES",
"CELERY_ENABLED",
"CELERY_QUEUES",
"REDIS_URL",
"CORS_ORIGINS",
"GOVOPLAN_TRUSTED_HOSTS",
"FORWARDED_ALLOW_IPS",
"AUTH_COOKIE_SECURE",
"AUTH_COOKIE_SAMESITE",
"GOVOPLAN_HTTP_HSTS_SECONDS",
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS",
"GOVOPLAN_MIGRATION_TRACK",
"DEV_AUTO_MIGRATE_ENABLED",
"DEV_BOOTSTRAP_ENABLED",
"GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE",
"GOVOPLAN_DEPLOYMENT_SPEC_PATH",
"FILE_STORAGE_BACKEND",
"FILE_STORAGE_LOCAL_ROOT",
"FILE_STORAGE_S3_ENDPOINT_URL",
"FILE_STORAGE_S3_REGION",
"FILE_STORAGE_S3_ACCESS_KEY_ID",
"FILE_STORAGE_S3_SECRET_ACCESS_KEY",
"FILE_STORAGE_S3_BUCKET",
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED",
)
@dataclass(frozen=True, slots=True)
class BundlePaths:
root: Path
spec: Path
env: Path
compose: Path
garage_config: Path
load_balancer_config: Path
plan: Path
receipt: Path
lock: Path
def bundle_paths(root: Path) -> BundlePaths:
expanded = root.expanduser()
if expanded.is_symlink():
raise ValueError(f"installation root must not be a symbolic link: {expanded}")
resolved = expanded.resolve()
return BundlePaths(
root=resolved,
spec=resolved / SPEC_FILENAME,
env=resolved / ENV_FILENAME,
compose=resolved / COMPOSE_FILENAME,
garage_config=resolved / GARAGE_CONFIG_FILENAME,
load_balancer_config=resolved / LOAD_BALANCER_CONFIG_FILENAME,
plan=resolved / PLAN_FILENAME,
receipt=resolved / RECEIPT_FILENAME,
lock=resolved / LOCK_FILENAME,
)
def ensure_private_directory(path: Path) -> None:
path.mkdir(mode=0o700, parents=True, exist_ok=True)
if path.is_symlink() or not path.is_dir():
raise ValueError(f"installation root must be a real directory: {path}")
current = stat.S_IMODE(path.stat().st_mode)
if current & 0o077:
path.chmod(0o700)
def initial_secrets(
spec: InstallationSpec,
*,
supplied: Mapping[str, str] | None = None,
) -> dict[str, str]:
values = dict(supplied or {})
values.setdefault(
"MASTER_KEY_B64",
base64.urlsafe_b64encode(os.urandom(32)).decode("ascii"),
)
values.setdefault("POSTGRES_DB", "govoplan")
values.setdefault("POSTGRES_USER", "govoplan")
values.setdefault("POSTGRES_PASSWORD", secrets.token_urlsafe(36))
values.setdefault("REDIS_PASSWORD", secrets.token_urlsafe(36))
return reconcile_runtime_environment(spec, values)
def reconcile_runtime_environment(
spec: InstallationSpec,
current: Mapping[str, str],
) -> dict[str, str]:
values = dict(current)
postgres = spec.components.postgres
if postgres.mode == "managed":
database = values.setdefault("POSTGRES_DB", "govoplan")
username = values.setdefault("POSTGRES_USER", "govoplan")
password = values.setdefault("POSTGRES_PASSWORD", secrets.token_urlsafe(36))
encoded_user = quote(username, safe="")
encoded_password = quote(password, safe="")
encoded_database = quote(database, safe="")
values["DATABASE_URL"] = (
f"postgresql+psycopg://{encoded_user}:{encoded_password}"
f"@postgres:5432/{encoded_database}"
)
values["GOVOPLAN_DATABASE_URL_PGTOOLS"] = (
f"postgresql://{encoded_user}:{encoded_password}"
f"@postgres:5432/{encoded_database}"
)
elif not values.get(postgres.url_env):
raise ValueError(
f"external PostgreSQL requires {postgres.url_env} in {ENV_FILENAME}"
)
else:
_validate_service_url(
values[postgres.url_env],
schemes={"postgresql", "postgresql+psycopg"},
label="external PostgreSQL URL",
)
redis = spec.components.redis
if redis.mode == "managed":
password = values.setdefault("REDIS_PASSWORD", secrets.token_urlsafe(36))
values["REDIS_URL"] = f"redis://:{quote(password, safe='')}@redis:6379/0"
elif redis.mode == "external":
if not values.get(redis.url_env):
raise ValueError(
f"external Redis requires {redis.url_env} in {ENV_FILENAME}"
)
_validate_service_url(
values[redis.url_env],
schemes={"redis", "rediss"},
label="external Redis URL",
)
else:
values["REDIS_URL"] = ""
public = urlsplit(spec.public_url)
values.update(
{
"APP_ENV": "production" if spec.profile == "self-hosted" else "staging",
"GOVOPLAN_INSTALL_PROFILE": spec.profile,
"ENABLED_MODULES": ",".join(spec.enabled_modules),
"CELERY_ENABLED": "true" if redis.mode != "disabled" else "false",
"CELERY_QUEUES": (
"send_email,append_sent,notifications,calendar,dataflow,events,default"
),
"CORS_ORIGINS": spec.public_url,
"GOVOPLAN_TRUSTED_HOSTS": public.hostname or "",
"FORWARDED_ALLOW_IPS": spec.network_subnet,
"AUTH_COOKIE_SECURE": "true" if public.scheme == "https" else "false",
"AUTH_COOKIE_SAMESITE": "lax",
"GOVOPLAN_HTTP_HSTS_SECONDS": (
"31536000" if public.scheme == "https" else "0"
),
"GOVOPLAN_CONNECTOR_ALLOW_PRIVATE_NETWORKS": "false",
"GOVOPLAN_MIGRATION_TRACK": "release",
"DEV_AUTO_MIGRATE_ENABLED": "false",
"DEV_BOOTSTRAP_ENABLED": "false",
"GOVOPLAN_DEPLOYMENT_SPEC_PATH": "/etc/govoplan/deployment/installation.json",
}
)
if redis.mode == "disabled":
values["GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE"] = "true"
else:
values["GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE"] = "false"
storage = spec.components.storage
if storage.mode == "local":
values["FILE_STORAGE_BACKEND"] = "local"
values["FILE_STORAGE_S3_DEPLOYMENT_MANAGED"] = "false"
values["FILE_STORAGE_LOCAL_ROOT"] = "/var/lib/govoplan/files"
elif storage.mode == "garage":
access_key = values.setdefault(
"GARAGE_DEFAULT_ACCESS_KEY",
f"GK{secrets.token_hex(16)}",
)
secret_key = values.setdefault(
"GARAGE_DEFAULT_SECRET_KEY",
secrets.token_hex(32),
)
bucket = values.setdefault("GARAGE_DEFAULT_BUCKET", "govoplan-files")
values.setdefault("GARAGE_RPC_SECRET", secrets.token_hex(32))
values.setdefault("GARAGE_ADMIN_TOKEN", secrets.token_urlsafe(48))
values.setdefault("GARAGE_METRICS_TOKEN", secrets.token_urlsafe(48))
values.update(
{
"FILE_STORAGE_BACKEND": "s3",
"FILE_STORAGE_S3_ENDPOINT_URL": "http://garage:3900",
"FILE_STORAGE_S3_REGION": "garage",
"FILE_STORAGE_S3_ACCESS_KEY_ID": access_key,
"FILE_STORAGE_S3_SECRET_ACCESS_KEY": secret_key,
"FILE_STORAGE_S3_BUCKET": bucket,
"FILE_STORAGE_S3_DEPLOYMENT_MANAGED": "true",
}
)
else:
values["FILE_STORAGE_BACKEND"] = "s3"
values["FILE_STORAGE_S3_DEPLOYMENT_MANAGED"] = "false"
required = (
"FILE_STORAGE_S3_ENDPOINT_URL",
"FILE_STORAGE_S3_REGION",
"FILE_STORAGE_S3_ACCESS_KEY_ID",
"FILE_STORAGE_S3_SECRET_ACCESS_KEY",
"FILE_STORAGE_S3_BUCKET",
)
missing = [name for name in required if not values.get(name)]
if missing:
raise ValueError(
"S3 storage requires values in secrets.env: " + ", ".join(missing)
)
endpoint = _validate_service_url(
values["FILE_STORAGE_S3_ENDPOINT_URL"],
schemes={"http", "https"},
label="S3 endpoint URL",
)
if spec.profile == "self-hosted" and endpoint.scheme != "https":
raise ValueError("self-hosted S3 endpoint URL must use HTTPS")
return dict(sorted(values.items()))
def render_compose(spec: InstallationSpec) -> dict[str, object]:
runtime_env = {
"environment": _environment_references(RUNTIME_ENV_KEYS),
}
deployment_mount = (
f"./{SPEC_FILENAME}:/etc/govoplan/deployment/installation.json:ro"
)
data_mounts = [deployment_mount]
if spec.components.storage.mode == "local":
data_mounts.append("files-data:/var/lib/govoplan/files")
dependency_conditions: dict[str, dict[str, str]] = {}
services: dict[str, object] = {}
if spec.components.postgres.mode == "managed":
services["postgres"] = {
"image": spec.components.postgres.image,
"restart": "unless-stopped",
"environment": _environment_references(
("POSTGRES_DB", "POSTGRES_USER", "POSTGRES_PASSWORD"),
required=True,
),
"healthcheck": {
"test": [
"CMD-SHELL",
'pg_isready -U "$${POSTGRES_USER}" -d "$${POSTGRES_DB}"',
],
"interval": "5s",
"timeout": "3s",
"retries": 30,
},
"volumes": ["postgres-data:/var/lib/postgresql/data"],
"networks": ["internal"],
}
dependency_conditions["postgres"] = {"condition": "service_healthy"}
if spec.components.redis.mode == "managed":
services["redis"] = {
"image": spec.components.redis.image,
"restart": "unless-stopped",
"environment": _environment_references(
("REDIS_PASSWORD",),
required=True,
),
"command": [
"sh",
"-ec",
'exec redis-server --appendonly yes --requirepass "$$REDIS_PASSWORD"',
],
"healthcheck": {
"test": [
"CMD-SHELL",
'redis-cli -a "$${REDIS_PASSWORD}" --no-auth-warning ping',
],
"interval": "5s",
"timeout": "3s",
"retries": 30,
},
"volumes": ["redis-data:/data"],
"networks": ["internal"],
}
dependency_conditions["redis"] = {"condition": "service_healthy"}
if spec.components.storage.mode == "garage":
garage_environment = _environment_references(
(
"GARAGE_DEFAULT_ACCESS_KEY",
"GARAGE_DEFAULT_SECRET_KEY",
"GARAGE_DEFAULT_BUCKET",
"GARAGE_RPC_SECRET",
"GARAGE_ADMIN_TOKEN",
"GARAGE_METRICS_TOKEN",
),
required=True,
)
services["garage"] = {
"image": spec.components.storage.image,
"restart": "unless-stopped",
"command": [
"/garage",
"server",
"--single-node",
"--default-bucket",
],
"environment": garage_environment,
"healthcheck": {
"test": ["CMD", "/garage", "status"],
"interval": "10s",
"timeout": "5s",
"retries": 30,
"start_period": "15s",
},
"labels": {
"org.govoplan.configuration-sha256": sha256(
render_garage_config().encode("utf-8")
).hexdigest()
},
"security_opt": ["no-new-privileges:true"],
"volumes": [
f"./{GARAGE_CONFIG_FILENAME}:/etc/garage.toml:ro",
"garage-meta:/var/lib/garage/meta",
"garage-data:/var/lib/garage/data",
],
"networks": ["internal"],
}
dependency_conditions["garage"] = {"condition": "service_healthy"}
if spec.components.mail.mode == "test-mail":
services["test-mail"] = {
"image": spec.components.mail.image,
"restart": "unless-stopped",
"environment": {
"GREENMAIL_OPTS": (
"-Dgreenmail.setup.test.smtp -Dgreenmail.setup.test.imap "
"-Dgreenmail.hostname=0.0.0.0"
)
},
"networks": ["internal"],
}
common_runtime: dict[str, object] = {
**runtime_env,
"restart": "unless-stopped",
"volumes": data_mounts,
"networks": ["internal"],
}
if dependency_conditions:
common_runtime["depends_on"] = dependency_conditions
services["migrate"] = {
**runtime_env,
"image": spec.release.api_image,
"command": ["python", "-m", "govoplan_core.commands.init_db"],
"restart": "no",
"volumes": data_mounts,
"networks": ["internal"],
**({"depends_on": dependency_conditions} if dependency_conditions else {}),
}
services["api"] = {
**common_runtime,
"image": spec.release.api_image,
"scale": spec.replicas.api,
"command": [
"python",
"-m",
"uvicorn",
"govoplan_core.server.app:app",
"--host",
"0.0.0.0",
"--port",
"8000",
"--proxy-headers",
],
"healthcheck": {
"test": [
"CMD",
"python",
"-c",
(
"import urllib.request;"
"urllib.request.urlopen('http://127.0.0.1:8000/health',timeout=3)"
),
],
"interval": "10s",
"timeout": "5s",
"retries": 30,
"start_period": "20s",
},
}
services["web"] = {
"image": spec.release.web_image,
"restart": "unless-stopped",
"scale": spec.replicas.web,
"environment": {"GOVOPLAN_API_UPSTREAM": "http://load-balancer:8000"},
"networks": ["internal"],
}
services["load-balancer"] = {
"image": spec.components.load_balancer.image,
"restart": "unless-stopped",
"healthcheck": {
"test": [
"CMD",
"haproxy",
"-c",
"-f",
"/usr/local/etc/haproxy/haproxy.cfg",
],
"interval": "10s",
"timeout": "5s",
"retries": 10,
},
"labels": {
"org.govoplan.configuration-sha256": sha256(
render_load_balancer_config(spec).encode("utf-8")
).hexdigest()
},
"ports": [_published_port(spec.listen.address, spec.listen.port, 8080)],
"read_only": True,
"security_opt": ["no-new-privileges:true"],
"volumes": [
(f"./{LOAD_BALANCER_CONFIG_FILENAME}:/usr/local/etc/haproxy/haproxy.cfg:ro")
],
"networks": ["internal"],
}
if spec.components.redis.mode != "disabled":
services["worker"] = {
**common_runtime,
"image": spec.release.api_image,
"scale": spec.replicas.worker,
"command": [
"python",
"-m",
"celery",
"-A",
"govoplan_core.celery_app:celery",
"worker",
"--queues",
(
"send_email,append_sent,notifications,calendar,"
"dataflow,events,default"
),
"--loglevel",
"INFO",
],
}
services["scheduler"] = {
**common_runtime,
"image": spec.release.api_image,
"command": [
"python",
"-m",
"celery",
"-A",
"govoplan_core.celery_app:celery",
"beat",
"--loglevel",
"INFO",
],
}
volumes: dict[str, object] = {}
if spec.components.postgres.mode == "managed":
volumes["postgres-data"] = {}
if spec.components.redis.mode == "managed":
volumes["redis-data"] = {}
if spec.components.storage.mode == "local":
volumes["files-data"] = {}
if spec.components.storage.mode == "garage":
volumes["garage-meta"] = {}
volumes["garage-data"] = {}
return {
"name": spec.installation_id,
"services": services,
"volumes": volumes,
"networks": {
"internal": {
"driver": "bridge",
"ipam": {"config": [{"subnet": spec.network_subnet}]},
}
},
}
def render_garage_config() -> str:
return """metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = ".s3.garage.localhost"
[admin]
api_bind_addr = "[::]:3903"
"""
def render_load_balancer_config(spec: InstallationSpec) -> str:
return f"""global
log stdout format raw local0
maxconn 4096
defaults
log global
mode http
option httplog
option redispatch
timeout connect 5s
timeout client 60s
timeout server 60s
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 10s
hold obsolete 10s
frontend public_web
bind :8080
default_backend web_replicas
backend web_replicas
balance roundrobin
option httpchk GET /health
http-check expect status 200
server-template web- {spec.replicas.web} web:8080 check resolvers docker init-addr libc,none
frontend internal_api
bind :8000
default_backend api_replicas
backend api_replicas
balance leastconn
option httpchk GET /health
http-check expect status 200
server-template api- {spec.replicas.api} api:8000 check resolvers docker init-addr libc,none
"""
def service_names(spec: InstallationSpec) -> tuple[str, ...]:
return tuple(render_compose(spec)["services"].keys())
def canonical_json(value: object) -> bytes:
return (
json.dumps(value, indent=2, sort_keys=True, separators=(",", ": ")) + "\n"
).encode("utf-8")
def digest_json(value: object) -> str:
return sha256(canonical_json(value)).hexdigest()
def environment_fingerprint(values: Mapping[str, str]) -> str:
key = values.get("MASTER_KEY_B64", "").encode("utf-8")
if not key:
return ""
payload = canonical_json(dict(sorted(values.items())))
return hmac.new(key, payload, sha256).hexdigest()
def read_env(path: Path) -> dict[str, str]:
if not path.exists():
return {}
values: dict[str, str] = {}
for line_number, raw_line in enumerate(
path.read_text(encoding="utf-8").splitlines(), start=1
):
line = raw_line.strip()
if not line or line.startswith("#"):
continue
key, separator, raw_value = line.partition("=")
if not separator or not ENV_NAME_PATTERN.fullmatch(key):
raise ValueError(f"invalid environment line {line_number} in {path}")
if key in values:
raise ValueError(f"duplicate environment key {key!r} on line {line_number}")
value = raw_value
if value.startswith('"'):
try:
decoded = json.loads(value)
except json.JSONDecodeError as exc:
raise ValueError(
f"invalid quoted environment value on line {line_number}"
) from exc
if not isinstance(decoded, str):
raise ValueError(
f"environment value on line {line_number} must be a string"
)
value = decoded
elif value.startswith("'"):
value = _single_quoted_env_value(value, line_number=line_number)
values[key] = value
return values
def write_env(path: Path, values: Mapping[str, str]) -> None:
invalid = sorted(key for key in values if not ENV_NAME_PATTERN.fullmatch(key))
if invalid:
raise ValueError("invalid environment variable names: " + ", ".join(invalid))
lines = [
"# Generated by govoplan-deploy. Keep this file private.",
*[f"{key}={_env_value(value)}" for key, value in sorted(values.items())],
"",
]
atomic_write(path, "\n".join(lines).encode("utf-8"), mode=0o600)
def atomic_write(path: Path, payload: bytes, *, mode: int) -> None:
path.parent.mkdir(mode=0o700, parents=True, exist_ok=True)
temporary = path.with_name(f".{path.name}.{os.getpid()}.{secrets.token_hex(8)}.tmp")
descriptor = os.open(
temporary,
os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,
mode,
)
try:
with os.fdopen(descriptor, "wb", closefd=True) as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
path.chmod(mode)
directory_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
try:
os.fsync(directory_fd)
finally:
os.close(directory_fd)
finally:
if temporary.exists():
temporary.unlink()
def _env_value(value: str) -> str:
if "\x00" in value or "\n" in value or "\r" in value:
raise ValueError("environment values must not contain NUL or line breaks")
if value and all(
character.isalnum() or character in "_./:@%+,-" for character in value
):
return value
escaped = value.replace("\\", "\\\\").replace("'", "\\'")
return f"'{escaped}'"
def _validate_service_url(
value: str,
*,
schemes: set[str],
label: str,
):
parsed = urlsplit(value)
try:
parsed.port
except ValueError as exc:
raise ValueError(f"{label} contains an invalid port") from exc
if parsed.scheme not in schemes or not parsed.hostname:
raise ValueError(
f"{label} must use one of {', '.join(sorted(schemes))} and include a host"
)
if parsed.fragment:
raise ValueError(f"{label} must not contain a fragment")
return parsed
def _single_quoted_env_value(value: str, *, line_number: int) -> str:
if len(value) < 2 or not value.endswith("'"):
raise ValueError(
f"unterminated single-quoted environment value on line {line_number}"
)
body = value[1:-1]
output: list[str] = []
index = 0
while index < len(body):
character = body[index]
if character != "\\":
output.append(character)
index += 1
continue
index += 1
if index >= len(body) or body[index] not in {"\\", "'"}:
raise ValueError(f"invalid single-quoted escape on line {line_number}")
output.append(body[index])
index += 1
return "".join(output)
def _published_port(address: str, host_port: int, container_port: int) -> str:
host = f"[{address}]" if ":" in address else address
return f"{host}:{host_port}:{container_port}"
def _environment_references(
keys: tuple[str, ...],
*,
required: bool = False,
) -> dict[str, str]:
suffix = ":?required by GovOPlaN deployment" if required else ":-"
return {key: f"${{{key}{suffix}}}" for key in keys}
+850
View File
@@ -0,0 +1,850 @@
"""Command-line interface for GovOPlaN deployment reconciliation."""
from __future__ import annotations
import argparse
from contextlib import contextmanager
from dataclasses import replace
from datetime import UTC, datetime
import fcntl
import getpass
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import time
from typing import Iterator, Mapping, Sequence
from urllib.error import URLError
from urllib.request import urlopen
from .bundle import (
atomic_write,
bundle_paths,
canonical_json,
digest_json,
environment_fingerprint,
ensure_private_directory,
initial_secrets,
read_env,
reconcile_runtime_environment,
render_compose,
render_garage_config,
render_load_balancer_config,
service_names,
write_env,
)
from .model import (
ComponentConfig,
DEFAULT_GARAGE_IMAGE,
DEFAULT_LOAD_BALANCER_IMAGE,
InstallationSpec,
ListenConfig,
ReplicaConfig,
SpecError,
default_spec,
load_spec,
parse_spec,
)
from .planning import DeploymentPlan, build_plan
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="govoplan-deploy",
description=(
"Create, validate, and reconcile a declarative GovOPlaN Compose deployment."
),
)
subparsers = parser.add_subparsers(dest="command", required=True)
init = subparsers.add_parser(
"init", help="Create a new private installation bundle."
)
_directory_argument(init)
_configuration_arguments(init)
init.add_argument(
"--non-interactive",
action="store_true",
help="Use flags/defaults without prompting.",
)
configure = subparsers.add_parser(
"configure",
help="Change installation choices while preserving generated secrets.",
)
_directory_argument(configure)
_configuration_arguments(configure, defaults=False)
render = subparsers.add_parser(
"render", help="Regenerate Compose and the deployment plan without applying."
)
_directory_argument(render)
render.add_argument("--json", action="store_true", help="Print the plan as JSON.")
doctor = subparsers.add_parser(
"doctor", help="Run specification, secret, host, and provenance checks."
)
_directory_argument(doctor)
doctor.add_argument("--json", action="store_true", help="Print the plan as JSON.")
apply_parser = subparsers.add_parser(
"apply", help="Apply an allowed plan with Docker Compose."
)
_directory_argument(apply_parser)
apply_parser.add_argument(
"--allow-unverified-images",
action="store_true",
help="Allow mutable/unverified images for an evaluation profile only.",
)
apply_parser.add_argument(
"--skip-pull",
action="store_true",
help="Do not pull images before reconciliation.",
)
apply_parser.add_argument(
"--health-timeout-seconds",
type=float,
default=120.0,
help="Maximum time to wait for the public health endpoint.",
)
status = subparsers.add_parser(
"status", help="Show desired state and current Compose process state."
)
_directory_argument(status)
status.add_argument("--json", action="store_true", help="Print JSON.")
return parser
def _directory_argument(parser: argparse.ArgumentParser) -> None:
parser.add_argument(
"--directory",
type=Path,
default=Path.home()
/ ".local"
/ "share"
/ "govoplan"
/ "installations"
/ "default",
help="Private installation state directory.",
)
def _configuration_arguments(
parser: argparse.ArgumentParser, *, defaults: bool = True
) -> None:
default = (lambda value: value) if defaults else (lambda _value: None)
parser.add_argument("--installation-id", default=default("govoplan-local"))
parser.add_argument(
"--profile",
choices=("evaluation", "self-hosted"),
default=default("evaluation"),
)
parser.add_argument("--public-url", default=default("http://127.0.0.1:8080"))
parser.add_argument("--listen-address", default=default("127.0.0.1"))
parser.add_argument("--listen-port", type=int, default=default(8080))
parser.add_argument(
"--postgres",
choices=("managed", "external"),
default=default("managed"),
)
parser.add_argument(
"--database-url", help="External PostgreSQL URL; stored privately."
)
parser.add_argument(
"--redis",
choices=("managed", "external", "disabled"),
default=default("managed"),
)
parser.add_argument("--redis-url", help="External Redis URL; stored privately.")
parser.add_argument(
"--mail",
choices=("disabled", "external-relay", "test-mail"),
default=default("disabled"),
)
parser.add_argument(
"--storage",
choices=("local", "garage", "s3"),
default=default("local"),
)
parser.add_argument(
"--garage-image",
default=default(DEFAULT_GARAGE_IMAGE),
help="Garage image used by managed object storage.",
)
parser.add_argument("--s3-endpoint-url", help="S3-compatible endpoint URL.")
parser.add_argument("--s3-region", help="S3 region.")
parser.add_argument(
"--s3-access-key-id", help="S3 access key id; stored privately."
)
parser.add_argument(
"--s3-secret-access-key",
help="S3 secret access key; stored privately.",
)
parser.add_argument("--s3-bucket", help="S3 bucket for managed files.")
parser.add_argument(
"--load-balancer-image",
default=default(DEFAULT_LOAD_BALANCER_IMAGE),
help="HAProxy image used by the managed local load balancer.",
)
parser.add_argument(
"--api-replicas",
type=int,
default=default(1),
help="API containers on this Compose host (1-64).",
)
parser.add_argument(
"--web-replicas",
type=int,
default=default(1),
help="WebUI containers on this Compose host (1-64).",
)
parser.add_argument(
"--worker-replicas",
type=int,
default=None,
help="Worker containers on this Compose host (1-128, or 0 without Redis).",
)
parser.add_argument(
"--module-set",
choices=("core", "base", "full"),
default=default("base"),
)
parser.add_argument("--api-image", default=default("govoplan-api:unpublished"))
parser.add_argument("--web-image", default=default("govoplan-web:unpublished"))
parser.add_argument("--release-version", default=default("unpublished"))
parser.add_argument("--release-channel", default=default("stable"))
parser.add_argument("--manifest-url", default=default(""))
parser.add_argument("--manifest-sha256", default=default(""))
def main(argv: Sequence[str] | None = None) -> int:
args = build_parser().parse_args(argv)
try:
if args.command == "init":
return _init(args)
if args.command == "configure":
return _configure(args)
if args.command in {"render", "doctor"}:
return _render_or_doctor(args)
if args.command == "apply":
return _apply(args)
if args.command == "status":
return _status(args)
except (SpecError, ValueError, OSError, subprocess.SubprocessError) as exc:
print(f"error: {exc}", file=sys.stderr)
return 1
raise RuntimeError(f"unsupported command: {args.command}")
def _init(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
ensure_private_directory(paths.root)
if paths.spec.exists():
raise ValueError(
f"{paths.spec} already exists; use configure for an existing installation"
)
if not args.non_interactive and sys.stdin.isatty():
_prompt_configuration(args)
spec = default_spec(
installation_id=args.installation_id,
profile=args.profile,
public_url=args.public_url,
listen_address=args.listen_address,
listen_port=args.listen_port,
postgres_mode=args.postgres,
redis_mode=args.redis,
mail_mode=args.mail,
storage_mode=args.storage,
garage_image=args.garage_image,
load_balancer_image=args.load_balancer_image,
api_replicas=args.api_replicas,
web_replicas=args.web_replicas,
worker_replicas=args.worker_replicas,
module_set=args.module_set,
api_image=args.api_image,
web_image=args.web_image,
manifest_url=args.manifest_url,
manifest_sha256=args.manifest_sha256,
version=args.release_version,
channel=args.release_channel,
)
supplied = _supplied_secret_values(args)
secrets = initial_secrets(spec, supplied=supplied)
_write_bundle(spec, paths, secrets)
plan = build_plan(spec, paths, include_host_checks=False)
_write_plan(paths.plan, plan)
print(f"Created GovOPlaN installation bundle at {paths.root}")
print(f"Edit choices with: govoplan-deploy configure --directory {paths.root}")
print(f"Check readiness with: govoplan-deploy doctor --directory {paths.root}")
if any(check.level == "error" for check in plan.checks):
print("The bundle is not apply-ready yet; run doctor for the blocking checks.")
return 0
def _configure(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
ensure_private_directory(paths.root)
current = load_spec(paths.spec)
if args.installation_id and args.installation_id != current.installation_id:
raise ValueError(
"installation_id is immutable; create a separate installation "
"instead of renaming a Compose project"
)
spec = _updated_spec(current, args)
if (
current.components.postgres.mode == "managed"
and spec.components.postgres.mode == "external"
and not args.database_url
):
raise ValueError(
"switching PostgreSQL from managed to external requires --database-url"
)
if (
current.components.redis.mode != "external"
and spec.components.redis.mode == "external"
and not args.redis_url
):
raise ValueError("switching Redis to external requires --redis-url")
if current.components.storage.mode != "s3" and spec.components.storage.mode == "s3":
missing_s3_options = [
flag
for flag, value in (
("--s3-endpoint-url", args.s3_endpoint_url),
("--s3-region", args.s3_region),
("--s3-access-key-id", args.s3_access_key_id),
("--s3-secret-access-key", args.s3_secret_access_key),
("--s3-bucket", args.s3_bucket),
)
if not value
]
if missing_s3_options:
raise ValueError(
"switching to external S3 requires: " + ", ".join(missing_s3_options)
)
secrets = read_env(paths.env)
secrets.update(_supplied_secret_values(args))
secrets = reconcile_runtime_environment(spec, secrets)
_write_bundle(spec, paths, secrets)
plan = build_plan(spec, paths, include_host_checks=False)
_write_plan(paths.plan, plan)
print(f"Updated desired state at {paths.root}")
_print_plan(plan)
return 0
def _render_or_doctor(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
spec = load_spec(paths.spec)
secrets = reconcile_runtime_environment(spec, read_env(paths.env))
_write_bundle(spec, paths, secrets)
plan = build_plan(
spec,
paths,
include_host_checks=args.command == "doctor",
)
_write_plan(paths.plan, plan)
if args.json:
print(json.dumps(plan.to_dict(), indent=2, sort_keys=True))
else:
_print_plan(plan)
return 1 if plan.blocked else 0
def _apply(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
spec = load_spec(paths.spec)
if args.allow_unverified_images and spec.profile != "evaluation":
raise ValueError(
"--allow-unverified-images is restricted to evaluation installations"
)
ensure_private_directory(paths.root)
with _deployment_lock(paths.lock):
secrets = reconcile_runtime_environment(spec, read_env(paths.env))
_write_bundle(spec, paths, secrets)
plan = build_plan(spec, paths, include_host_checks=True)
_write_plan(paths.plan, plan)
effective_errors = [
check
for check in plan.checks
if check.level == "error"
and not (
args.allow_unverified_images
and check.id.startswith(
(
"release.api_image.",
"release.web_image.",
"components.postgres.image.",
"components.redis.image.",
"components.mail.image.",
"components.storage.image.",
"components.load_balancer.image.",
"release.manifest",
"modules.image_composition",
)
)
)
]
if effective_errors:
_print_plan(plan)
raise ValueError("deployment plan is blocked; resolve doctor errors first")
docker = shutil.which("docker")
if docker is None:
raise ValueError("Docker CLI is required for apply")
compose = [
docker,
"compose",
"--env-file",
str(paths.env),
"--project-name",
spec.installation_id,
"--file",
str(paths.compose),
]
if not args.skip_pull:
_run([*compose, "pull"], cwd=paths.root)
dependencies = [
name
for name in ("postgres", "redis", "garage", "test-mail")
if name in service_names(spec)
]
if dependencies:
_run([*compose, "up", "--detach", *dependencies], cwd=paths.root)
_run([*compose, "run", "--rm", "migrate"], cwd=paths.root)
if _receipt_uses_direct_web_port(paths.receipt):
_run([*compose, "stop", "web"], cwd=paths.root)
runtime_services = [
name
for name in ("api", "web", "load-balancer", "worker", "scheduler")
if name in service_names(spec)
]
_run(
[*compose, "up", "--detach", "--remove-orphans", *runtime_services],
cwd=paths.root,
)
_wait_for_health(
f"{spec.public_url}/health",
timeout_seconds=args.health_timeout_seconds,
)
receipt = {
"schema_version": 1,
"installation_id": spec.installation_id,
"applied_at": _now(),
"spec_sha256": digest_json(spec.to_dict()),
"compose_sha256": digest_json(render_compose(spec)),
"environment_fingerprint": environment_fingerprint(secrets),
"release": {
"channel": spec.release.channel,
"version": spec.release.version,
"manifest_sha256": spec.release.manifest_sha256,
"api_image": spec.release.api_image,
"web_image": spec.release.web_image,
},
"services": list(service_names(spec)),
"replicas": {
"api": spec.replicas.api,
"web": spec.replicas.web,
"worker": spec.replicas.worker,
},
"listen": {
"address": spec.listen.address,
"port": spec.listen.port,
},
"management": {
"mode": "govoplan-deploy",
"agent": "cli",
"web_updates": False,
},
}
atomic_write(paths.receipt, canonical_json(receipt), mode=0o600)
print(f"GovOPlaN is ready at {spec.public_url}")
return 0
def _status(args: argparse.Namespace) -> int:
paths = bundle_paths(args.directory)
spec = load_spec(paths.spec)
plan = build_plan(spec, paths, include_host_checks=False)
docker = shutil.which("docker")
processes: object = []
process_error = ""
if docker:
result = subprocess.run(
[
docker,
"compose",
"--env-file",
str(paths.env),
"--project-name",
spec.installation_id,
"--file",
str(paths.compose),
"ps",
"--format",
"json",
],
cwd=paths.root,
check=False,
capture_output=True,
text=True,
timeout=15,
)
if result.returncode == 0:
processes = _parse_compose_ps(result.stdout)
else:
process_error = result.stderr.strip() or result.stdout.strip()
else:
process_error = "Docker CLI is unavailable."
payload = {
"installation_id": spec.installation_id,
"public_url": spec.public_url,
"plan": plan.to_dict(),
"processes": processes,
"process_error": process_error,
}
if args.json:
print(json.dumps(payload, indent=2, sort_keys=True))
else:
_print_plan(plan)
if process_error:
print(f"Runtime: {process_error}")
elif isinstance(processes, list):
print(f"Runtime: {len(processes)} Compose process(es) reported.")
return 0
def _updated_spec(
current: InstallationSpec, args: argparse.Namespace
) -> InstallationSpec:
module_set = getattr(args, "module_set", None)
if module_set:
modules = default_spec(module_set=module_set).enabled_modules
else:
modules = current.enabled_modules
release = replace(
current.release,
channel=args.release_channel or current.release.channel,
version=args.release_version or current.release.version,
manifest_url=(
args.manifest_url
if args.manifest_url is not None
else current.release.manifest_url
),
manifest_sha256=(
args.manifest_sha256
if args.manifest_sha256 is not None
else current.release.manifest_sha256
),
api_image=args.api_image or current.release.api_image,
web_image=args.web_image or current.release.web_image,
)
storage_mode = args.storage or current.components.storage.mode
components = ComponentConfig(
postgres=replace(
current.components.postgres,
mode=args.postgres or current.components.postgres.mode,
),
redis=replace(
current.components.redis,
mode=args.redis or current.components.redis.mode,
),
mail=replace(
current.components.mail,
mode=args.mail or current.components.mail.mode,
),
storage=replace(
current.components.storage,
mode=storage_mode,
image=(
args.garage_image
or current.components.storage.image
or DEFAULT_GARAGE_IMAGE
)
if storage_mode == "garage"
else "",
),
load_balancer=replace(
current.components.load_balancer,
image=(args.load_balancer_image or current.components.load_balancer.image),
),
)
replicas = ReplicaConfig(
api=(
args.api_replicas if args.api_replicas is not None else current.replicas.api
),
web=(
args.web_replicas if args.web_replicas is not None else current.replicas.web
),
worker=(
args.worker_replicas
if args.worker_replicas is not None
else (
0
if components.redis.mode == "disabled"
else max(current.replicas.worker, 1)
)
),
)
value = replace(
current,
installation_id=args.installation_id or current.installation_id,
profile=args.profile or current.profile,
public_url=args.public_url or current.public_url,
listen=ListenConfig(
address=args.listen_address or current.listen.address,
port=args.listen_port or current.listen.port,
),
release=release,
components=components,
replicas=replicas,
enabled_modules=modules,
)
return parse_spec(value.to_dict())
def _supplied_secret_values(args: argparse.Namespace) -> dict[str, str]:
values: dict[str, str] = {}
if getattr(args, "database_url", None):
values["DATABASE_URL"] = args.database_url
values["GOVOPLAN_DATABASE_URL_PGTOOLS"] = _pgtools_url(args.database_url)
if getattr(args, "redis_url", None):
values["REDIS_URL"] = args.redis_url
s3_values = {
"FILE_STORAGE_S3_ENDPOINT_URL": getattr(args, "s3_endpoint_url", None),
"FILE_STORAGE_S3_REGION": getattr(args, "s3_region", None),
"FILE_STORAGE_S3_ACCESS_KEY_ID": getattr(args, "s3_access_key_id", None),
"FILE_STORAGE_S3_SECRET_ACCESS_KEY": getattr(
args, "s3_secret_access_key", None
),
"FILE_STORAGE_S3_BUCKET": getattr(args, "s3_bucket", None),
}
values.update({key: value for key, value in s3_values.items() if value})
return values
def _pgtools_url(database_url: str) -> str:
if database_url.startswith("postgresql+psycopg://"):
return "postgresql://" + database_url.removeprefix("postgresql+psycopg://")
return database_url
def _write_bundle(
spec: InstallationSpec,
paths,
secrets: Mapping[str, str],
) -> None:
ensure_private_directory(paths.root)
atomic_write(paths.spec, canonical_json(spec.to_dict()), mode=0o600)
write_env(paths.env, secrets)
atomic_write(paths.compose, canonical_json(render_compose(spec)), mode=0o600)
atomic_write(
paths.load_balancer_config,
render_load_balancer_config(spec).encode("utf-8"),
mode=0o644,
)
atomic_write(
paths.garage_config,
render_garage_config().encode("utf-8"),
mode=0o644,
)
def _write_plan(path: Path, plan: DeploymentPlan) -> None:
atomic_write(path, canonical_json(plan.to_dict()), mode=0o600)
def _print_plan(plan: DeploymentPlan) -> None:
state = "BLOCKED" if plan.blocked else "READY"
print(f"GovOPlaN deployment plan: {state}")
for action in plan.actions:
print(f" {action.action:12} {action.target}: {action.detail}")
for check in plan.checks:
prefix = {"ok": "OK", "warning": "WARN", "error": "ERROR"}.get(
check.level, check.level.upper()
)
print(f" [{prefix}] {check.message}")
if check.action and check.level != "ok":
print(f" {check.action}")
def _prompt_configuration(args: argparse.Namespace) -> None:
args.profile = _prompt_choice(
"Installation profile", args.profile, ("evaluation", "self-hosted")
)
if args.profile == "self-hosted" and args.public_url.startswith("http://"):
args.public_url = "https://govoplan.example.org"
args.public_url = _prompt("Public URL", args.public_url)
args.postgres = _prompt_choice("PostgreSQL", args.postgres, ("managed", "external"))
if args.postgres == "external" and not args.database_url:
args.database_url = getpass.getpass(
"External PostgreSQL URL (input hidden): "
).strip()
redis_choices = (
("managed", "external")
if args.profile == "self-hosted"
else ("managed", "external", "disabled")
)
args.redis = _prompt_choice("Redis", args.redis, redis_choices)
if args.redis == "external" and not args.redis_url:
args.redis_url = getpass.getpass("External Redis URL (input hidden): ").strip()
mail_choices = (
("disabled", "external-relay")
if args.profile == "self-hosted"
else ("disabled", "external-relay", "test-mail")
)
args.mail = _prompt_choice(
"Mail integration",
args.mail,
mail_choices,
)
args.storage = _prompt_choice(
"File storage",
args.storage,
("local", "garage", "s3"),
)
if args.storage == "s3":
args.s3_endpoint_url = args.s3_endpoint_url or _prompt(
"S3 endpoint URL", "https://s3.example.org"
)
args.s3_region = args.s3_region or _prompt("S3 region", "eu-central-1")
args.s3_access_key_id = args.s3_access_key_id or _prompt("S3 access key id", "")
args.s3_secret_access_key = (
args.s3_secret_access_key
or getpass.getpass("S3 secret access key (input hidden): ").strip()
)
args.s3_bucket = args.s3_bucket or _prompt("S3 bucket", "govoplan-files")
args.api_replicas = _prompt_integer(
"API replicas on this host",
args.api_replicas,
minimum=1,
maximum=64,
)
args.web_replicas = _prompt_integer(
"WebUI replicas on this host",
args.web_replicas,
minimum=1,
maximum=64,
)
if args.redis != "disabled":
args.worker_replicas = _prompt_integer(
"Worker replicas on this host",
args.worker_replicas or 1,
minimum=1,
maximum=128,
)
else:
args.worker_replicas = 0
args.module_set = _prompt_choice(
"Initial module set", args.module_set, ("core", "base", "full")
)
def _prompt(label: str, default: str) -> str:
value = input(f"{label} [{default}]: ").strip()
return value or default
def _prompt_choice(label: str, default: str, choices: tuple[str, ...]) -> str:
while True:
value = _prompt(f"{label} ({'/'.join(choices)})", default)
if value in choices:
return value
print(f"Choose one of: {', '.join(choices)}")
def _prompt_integer(
label: str,
default: int,
*,
minimum: int,
maximum: int,
) -> int:
while True:
raw = _prompt(label, str(default))
try:
value = int(raw)
except ValueError:
value = minimum - 1
if minimum <= value <= maximum:
return value
print(f"Choose a number between {minimum} and {maximum}.")
@contextmanager
def _deployment_lock(path: Path) -> Iterator[None]:
descriptor = os.open(path, os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW, 0o600)
try:
try:
fcntl.flock(descriptor, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError as exc:
raise ValueError(
"another deployment operation owns the installation lock"
) from exc
os.ftruncate(descriptor, 0)
os.write(descriptor, f"{os.getpid()}\n".encode("ascii"))
os.fsync(descriptor)
yield
finally:
fcntl.flock(descriptor, fcntl.LOCK_UN)
os.close(descriptor)
def _run(argv: Sequence[str], *, cwd: Path) -> None:
result = subprocess.run(list(argv), cwd=cwd, check=False)
if result.returncode != 0:
raise subprocess.CalledProcessError(result.returncode, list(argv))
def _wait_for_health(url: str, *, timeout_seconds: float) -> None:
deadline = time.monotonic() + max(timeout_seconds, 1.0)
last_error = "health endpoint did not answer"
while time.monotonic() < deadline:
try:
with urlopen(url, timeout=3) as response: # noqa: S310 - URL originates in the validated installation specification.
if 200 <= response.status < 300:
return
last_error = f"health endpoint returned HTTP {response.status}"
except (OSError, URLError) as exc:
last_error = str(exc)
time.sleep(2)
raise ValueError(f"GovOPlaN did not become healthy: {last_error}")
def _parse_compose_ps(output: str) -> list[object]:
stripped = output.strip()
if not stripped:
return []
try:
value = json.loads(stripped)
except json.JSONDecodeError:
rows = []
for line in stripped.splitlines():
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
return [{"raw": stripped}]
return rows
return value if isinstance(value, list) else [value]
def _receipt_uses_direct_web_port(path: Path) -> bool:
if not path.exists():
return False
try:
receipt = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return False
services = receipt.get("services") if isinstance(receipt, dict) else None
return (
isinstance(services, list)
and "web" in services
and "load-balancer" not in services
)
def _now() -> str:
return (
datetime.now(tz=UTC).replace(microsecond=0).isoformat().replace("+00:00", "Z")
)
+585
View File
@@ -0,0 +1,585 @@
"""Installation intent model and validation."""
from __future__ import annotations
from dataclasses import asdict, dataclass
import ipaddress
import json
from pathlib import Path
import re
from typing import Any, Mapping
from urllib.parse import urlsplit
SCHEMA_VERSION = 1
DEFAULT_GARAGE_IMAGE = "dxflrs/garage:v2.3.0"
DEFAULT_LOAD_BALANCER_IMAGE = "haproxy:3.2.21-alpine"
INSTALLATION_ID_PATTERN = re.compile(r"^[a-z][a-z0-9-]{1,47}$")
ENV_NAME_PATTERN = re.compile(r"^[A-Z][A-Z0-9_]{1,63}$")
SHA256_PATTERN = re.compile(r"^[0-9a-f]{64}$")
IMAGE_DIGEST_PATTERN = re.compile(r"^[^@\s]+@sha256:[0-9a-f]{64}$")
RFC1918_NETWORKS = tuple(
ipaddress.ip_network(value)
for value in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")
)
BASE_MODULES = (
"tenancy",
"organizations",
"identity",
"idm",
"access",
"admin",
"dashboard",
"policy",
"audit",
"docs",
"ops",
)
FULL_MODULES = (
*BASE_MODULES,
"addresses",
"dist_lists",
"files",
"mail",
"campaigns",
"calendar",
"poll",
"scheduling",
"connectors",
"datasources",
"dataflow",
"workflow",
"views",
"search",
"risk_compliance",
"postbox",
"evaluation",
)
class SpecError(ValueError):
"""Raised when an installation specification is malformed."""
@dataclass(frozen=True, slots=True)
class ListenConfig:
address: str
port: int
@dataclass(frozen=True, slots=True)
class ReleaseConfig:
channel: str
version: str
manifest_url: str
manifest_sha256: str
api_image: str
web_image: str
@dataclass(frozen=True, slots=True)
class ServiceConfig:
mode: str
image: str = ""
url_env: str = ""
@dataclass(frozen=True, slots=True)
class StorageConfig:
mode: str
image: str = ""
@dataclass(frozen=True, slots=True)
class ComponentConfig:
postgres: ServiceConfig
redis: ServiceConfig
mail: ServiceConfig
storage: StorageConfig
load_balancer: ServiceConfig
@dataclass(frozen=True, slots=True)
class ReplicaConfig:
api: int
web: int
worker: int
@dataclass(frozen=True, slots=True)
class InstallationSpec:
schema_version: int
installation_id: str
profile: str
public_url: str
listen: ListenConfig
network_subnet: str
release: ReleaseConfig
components: ComponentConfig
replicas: ReplicaConfig
enabled_modules: tuple[str, ...]
def to_dict(self) -> dict[str, Any]:
value = asdict(self)
value["enabled_modules"] = list(self.enabled_modules)
return value
def default_spec(
*,
installation_id: str = "govoplan-local",
profile: str = "evaluation",
public_url: str = "http://127.0.0.1:8080",
listen_address: str = "127.0.0.1",
listen_port: int = 8080,
postgres_mode: str = "managed",
redis_mode: str = "managed",
mail_mode: str = "disabled",
storage_mode: str = "local",
garage_image: str = DEFAULT_GARAGE_IMAGE,
load_balancer_image: str = DEFAULT_LOAD_BALANCER_IMAGE,
api_replicas: int = 1,
web_replicas: int = 1,
worker_replicas: int | None = None,
module_set: str = "base",
api_image: str = "govoplan-api:unpublished",
web_image: str = "govoplan-web:unpublished",
manifest_url: str = "",
manifest_sha256: str = "",
version: str = "unpublished",
channel: str = "stable",
) -> InstallationSpec:
modules = {
"core": (),
"base": BASE_MODULES,
"full": FULL_MODULES,
}.get(module_set)
if modules is None:
raise SpecError("module_set must be core, base, or full")
effective_worker_replicas = (
(0 if redis_mode == "disabled" else 1)
if worker_replicas is None
else worker_replicas
)
subnet_octet = 32 + (sum(installation_id.encode("utf-8")) % 192)
raw = {
"schema_version": SCHEMA_VERSION,
"installation_id": installation_id,
"profile": profile,
"public_url": public_url,
"listen": {"address": listen_address, "port": listen_port},
"network_subnet": f"172.30.{subnet_octet}.0/24",
"release": {
"channel": channel,
"version": version,
"manifest_url": manifest_url,
"manifest_sha256": manifest_sha256,
"api_image": api_image,
"web_image": web_image,
},
"components": {
"postgres": {
"mode": postgres_mode,
"image": "postgres:16-alpine",
"url_env": "DATABASE_URL",
},
"redis": {
"mode": redis_mode,
"image": "redis:7-alpine",
"url_env": "REDIS_URL",
},
"mail": {
"mode": mail_mode,
"image": "greenmail/standalone:2.1.9",
"url_env": "",
},
"storage": {
"mode": storage_mode,
"image": garage_image if storage_mode == "garage" else "",
},
"load_balancer": {
"mode": "managed",
"image": load_balancer_image,
"url_env": "",
},
},
"replicas": {
"api": api_replicas,
"web": web_replicas,
"worker": effective_worker_replicas,
},
"enabled_modules": list(modules),
}
return parse_spec(raw)
def load_spec(path: Path) -> InstallationSpec:
try:
raw = json.loads(path.read_text(encoding="utf-8"))
except FileNotFoundError as exc:
raise SpecError(f"installation specification does not exist: {path}") from exc
except json.JSONDecodeError as exc:
raise SpecError(f"installation specification is not valid JSON: {exc}") from exc
return parse_spec(raw)
def parse_spec(raw: object) -> InstallationSpec:
root = _mapping(raw, "installation")
_only_keys(
root,
{
"schema_version",
"installation_id",
"profile",
"public_url",
"listen",
"network_subnet",
"release",
"components",
"replicas",
"enabled_modules",
},
"installation",
)
schema_version = _integer(root, "schema_version")
if schema_version != SCHEMA_VERSION:
raise SpecError(
f"schema_version must be {SCHEMA_VERSION}; found {schema_version}"
)
installation_id = _string(root, "installation_id")
if not INSTALLATION_ID_PATTERN.fullmatch(installation_id):
raise SpecError(
"installation_id must start with a lowercase letter and contain only "
"lowercase letters, digits, or hyphens (2-48 characters)"
)
profile = _choice(root, "profile", {"evaluation", "self-hosted"})
public_url = _http_url(_string(root, "public_url"), "public_url")
public_parts = urlsplit(public_url)
if profile == "self-hosted" and public_parts.scheme != "https":
raise SpecError("self-hosted public_url must use HTTPS")
listen_raw = _mapping(root.get("listen"), "listen")
_only_keys(listen_raw, {"address", "port"}, "listen")
listen = ListenConfig(
address=_ip_address(_string(listen_raw, "address"), "listen.address"),
port=_port(_integer(listen_raw, "port"), "listen.port"),
)
network_subnet = _network(_string(root, "network_subnet"), "network_subnet")
release = _release(root.get("release"))
components = _components(root.get("components"), profile=profile)
replicas = _replicas(root.get("replicas"), components=components)
enabled_raw = root.get("enabled_modules")
if not isinstance(enabled_raw, list):
raise SpecError("enabled_modules must be an array")
enabled_modules: list[str] = []
seen_modules: set[str] = set()
for index, value in enumerate(enabled_raw):
if not isinstance(value, str) or not re.fullmatch(
r"[a-z][a-z0-9_]{1,63}", value
):
raise SpecError(f"enabled_modules[{index}] must be a canonical module id")
if value in seen_modules:
raise SpecError(f"enabled_modules contains duplicate module {value!r}")
enabled_modules.append(value)
seen_modules.add(value)
return InstallationSpec(
schema_version=schema_version,
installation_id=installation_id,
profile=profile,
public_url=public_url,
listen=listen,
network_subnet=network_subnet,
release=release,
components=components,
replicas=replicas,
enabled_modules=tuple(enabled_modules),
)
def _release(raw: object) -> ReleaseConfig:
value = _mapping(raw, "release")
_only_keys(
value,
{
"channel",
"version",
"manifest_url",
"manifest_sha256",
"api_image",
"web_image",
},
"release",
)
channel = _string(value, "channel")
if not re.fullmatch(r"[a-z][a-z0-9-]{1,31}", channel):
raise SpecError("release.channel must be a canonical channel name")
version = _string(value, "version")
if not version or len(version) > 80 or any(char.isspace() for char in version):
raise SpecError("release.version must be a non-empty version token")
manifest_url = _optional_string(value, "manifest_url")
if manifest_url:
manifest_url = _http_url(manifest_url, "release.manifest_url")
if urlsplit(manifest_url).scheme != "https":
raise SpecError("release.manifest_url must use HTTPS")
manifest_sha256 = _optional_string(value, "manifest_sha256").lower()
if manifest_sha256 and not SHA256_PATTERN.fullmatch(manifest_sha256):
raise SpecError(
"release.manifest_sha256 must be a lowercase SHA-256 hex digest"
)
api_image = _image(_string(value, "api_image"), "release.api_image")
web_image = _image(_string(value, "web_image"), "release.web_image")
return ReleaseConfig(
channel=channel,
version=version,
manifest_url=manifest_url,
manifest_sha256=manifest_sha256,
api_image=api_image,
web_image=web_image,
)
def _components(raw: object, *, profile: str) -> ComponentConfig:
value = _mapping(raw, "components")
_only_keys(
value,
{"postgres", "redis", "mail", "storage", "load_balancer"},
"components",
)
postgres = _service(
value.get("postgres"),
"components.postgres",
modes={"managed", "external"},
image_required_for={"managed"},
url_env_required_for={"external"},
)
redis = _service(
value.get("redis"),
"components.redis",
modes={"managed", "external", "disabled"},
image_required_for={"managed"},
url_env_required_for={"external"},
)
mail = _service(
value.get("mail"),
"components.mail",
modes={"disabled", "external-relay", "test-mail"},
image_required_for={"test-mail"},
url_env_required_for=set(),
)
storage_raw = _mapping(value.get("storage"), "components.storage")
_only_keys(storage_raw, {"mode", "image"}, "components.storage")
storage_mode = _choice(storage_raw, "mode", {"local", "s3", "garage"})
storage_image = _optional_string(storage_raw, "image")
if storage_mode == "garage":
storage_image = _image(
storage_image or DEFAULT_GARAGE_IMAGE,
"components.storage.image",
)
elif storage_image:
raise SpecError(
"components.storage.image is only valid for managed Garage storage"
)
storage = StorageConfig(mode=storage_mode, image=storage_image)
load_balancer = _service(
value.get(
"load_balancer",
{
"mode": "managed",
"image": DEFAULT_LOAD_BALANCER_IMAGE,
"url_env": "",
},
),
"components.load_balancer",
modes={"managed"},
image_required_for={"managed"},
url_env_required_for=set(),
)
if postgres.url_env != "DATABASE_URL":
raise SpecError("components.postgres.url_env must be DATABASE_URL")
if redis.url_env != "REDIS_URL":
raise SpecError("components.redis.url_env must be REDIS_URL")
if mail.url_env:
raise SpecError("components.mail.url_env must be empty")
if load_balancer.url_env:
raise SpecError("components.load_balancer.url_env must be empty")
if profile == "self-hosted" and redis.mode == "disabled":
raise SpecError("self-hosted installations require managed or external Redis")
if profile == "self-hosted" and mail.mode == "test-mail":
raise SpecError("the test-mail component is restricted to evaluation installs")
return ComponentConfig(
postgres=postgres,
redis=redis,
mail=mail,
storage=storage,
load_balancer=load_balancer,
)
def _replicas(raw: object, *, components: ComponentConfig) -> ReplicaConfig:
if raw is None:
return ReplicaConfig(
api=1,
web=1,
worker=0 if components.redis.mode == "disabled" else 1,
)
value = _mapping(raw, "replicas")
_only_keys(value, {"api", "web", "worker"}, "replicas")
replicas = ReplicaConfig(
api=_bounded_integer(value, "api", minimum=1, maximum=64),
web=_bounded_integer(value, "web", minimum=1, maximum=64),
worker=_bounded_integer(value, "worker", minimum=0, maximum=128),
)
if components.redis.mode == "disabled":
if replicas.worker:
raise SpecError("replicas.worker must be 0 when Redis is disabled")
if replicas.api > 1:
raise SpecError(
"multiple API replicas require Redis for shared throttling and queues"
)
elif replicas.worker < 1:
raise SpecError("replicas.worker must be at least 1 when Redis is enabled")
return replicas
def _service(
raw: object,
label: str,
*,
modes: set[str],
image_required_for: set[str],
url_env_required_for: set[str],
) -> ServiceConfig:
value = _mapping(raw, label)
_only_keys(value, {"mode", "image", "url_env"}, label)
mode = _choice(value, "mode", modes)
image = _optional_string(value, "image")
url_env = _optional_string(value, "url_env")
if mode in image_required_for:
image = _image(image, f"{label}.image")
if mode in url_env_required_for:
if not ENV_NAME_PATTERN.fullmatch(url_env):
raise SpecError(f"{label}.url_env must name an environment variable")
return ServiceConfig(mode=mode, image=image, url_env=url_env)
def image_is_digest_pinned(value: str) -> bool:
return bool(IMAGE_DIGEST_PATTERN.fullmatch(value))
def image_is_unpublished(value: str) -> bool:
return value.endswith(":unpublished")
def _mapping(value: object, label: str) -> Mapping[str, Any]:
if not isinstance(value, dict):
raise SpecError(f"{label} must be an object")
if not all(isinstance(key, str) for key in value):
raise SpecError(f"{label} keys must be strings")
return value
def _only_keys(value: Mapping[str, Any], allowed: set[str], label: str) -> None:
unknown = sorted(set(value) - allowed)
if unknown:
raise SpecError(f"{label} contains unknown fields: {', '.join(unknown)}")
def _string(value: Mapping[str, Any], key: str) -> str:
result = value.get(key)
if not isinstance(result, str):
raise SpecError(f"{key} must be a string")
return result.strip()
def _optional_string(value: Mapping[str, Any], key: str) -> str:
result = value.get(key, "")
if not isinstance(result, str):
raise SpecError(f"{key} must be a string")
return result.strip()
def _integer(value: Mapping[str, Any], key: str) -> int:
result = value.get(key)
if isinstance(result, bool) or not isinstance(result, int):
raise SpecError(f"{key} must be an integer")
return result
def _bounded_integer(
value: Mapping[str, Any],
key: str,
*,
minimum: int,
maximum: int,
) -> int:
result = _integer(value, key)
if result < minimum or result > maximum:
raise SpecError(f"{key} must be between {minimum} and {maximum}")
return result
def _choice(value: Mapping[str, Any], key: str, choices: set[str]) -> str:
result = _string(value, key)
if result not in choices:
raise SpecError(f"{key} must be one of: {', '.join(sorted(choices))}")
return result
def _http_url(value: str, label: str) -> str:
try:
parsed = urlsplit(value)
parsed.port
except ValueError as exc:
raise SpecError(f"{label} contains an invalid host or port") from exc
if parsed.scheme not in {"http", "https"} or not parsed.netloc:
raise SpecError(f"{label} must be an absolute HTTP(S) URL")
if parsed.username or parsed.password or parsed.fragment or parsed.query:
raise SpecError(f"{label} must not contain credentials, a query, or a fragment")
if label == "public_url" and parsed.path not in {"", "/"}:
raise SpecError("public_url must address the site root without a path")
return value.rstrip("/")
def _image(value: str, label: str) -> str:
if (
not value
or len(value) > 300
or any(character.isspace() for character in value)
or value.startswith("-")
):
raise SpecError(f"{label} must be a valid non-empty OCI image reference")
return value
def _ip_address(value: str, label: str) -> str:
try:
return str(ipaddress.ip_address(value))
except ValueError as exc:
raise SpecError(f"{label} must be an IPv4 or IPv6 address") from exc
def _network(value: str, label: str) -> str:
try:
network = ipaddress.ip_network(value, strict=True)
except ValueError as exc:
raise SpecError(f"{label} must be a canonical private IP network") from exc
if (
network.version != 4
or not any(network.subnet_of(parent) for parent in RFC1918_NETWORKS)
or not 24 <= network.prefixlen <= 28
):
raise SpecError(f"{label} must be an RFC1918 IPv4 /24 to /28 network")
return str(network)
def _port(value: int, label: str) -> int:
if value < 1 or value > 65535:
raise SpecError(f"{label} must be between 1 and 65535")
return value
@@ -0,0 +1,626 @@
"""Deployment plan and host preflight checks."""
from __future__ import annotations
from dataclasses import asdict, dataclass
import json
import os
from pathlib import Path
import platform
import shutil
import socket
import stat
import subprocess
from typing import Callable, Mapping, Sequence
from urllib.parse import urlsplit
from .bundle import (
BundlePaths,
digest_json,
environment_fingerprint,
read_env,
render_compose,
service_names,
)
from .model import (
InstallationSpec,
image_is_digest_pinned,
image_is_unpublished,
)
@dataclass(frozen=True, slots=True)
class Check:
id: str
level: str
message: str
action: str = ""
def to_dict(self) -> dict[str, str]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class PlanAction:
action: str
target: str
detail: str
def to_dict(self) -> dict[str, str]:
return asdict(self)
@dataclass(frozen=True, slots=True)
class DeploymentPlan:
installation_id: str
desired_spec_sha256: str
desired_compose_sha256: str
desired_environment_fingerprint: str
actions: tuple[PlanAction, ...]
checks: tuple[Check, ...]
@property
def blocked(self) -> bool:
return any(check.level == "error" for check in self.checks)
def to_dict(self) -> dict[str, object]:
return {
"installation_id": self.installation_id,
"desired_spec_sha256": self.desired_spec_sha256,
"desired_compose_sha256": self.desired_compose_sha256,
"desired_environment_fingerprint": (self.desired_environment_fingerprint),
"blocked": self.blocked,
"actions": [action.to_dict() for action in self.actions],
"checks": [check.to_dict() for check in self.checks],
}
CommandRunner = Callable[[Sequence[str], Path], subprocess.CompletedProcess[str]]
def build_plan(
spec: InstallationSpec,
paths: BundlePaths,
*,
include_host_checks: bool = True,
command_runner: CommandRunner | None = None,
) -> DeploymentPlan:
compose = render_compose(spec)
desired_services = set(service_names(spec))
previous = _read_receipt(paths.receipt)
previous_services = {
str(item) for item in previous.get("services", []) if isinstance(item, str)
}
previous_spec_digest = previous.get("spec_sha256")
previous_compose_digest = previous.get("compose_sha256")
previous_environment_fingerprint = previous.get("environment_fingerprint")
spec_digest = digest_json(spec.to_dict())
compose_digest = digest_json(compose)
environment_digest = environment_fingerprint(read_env(paths.env))
actions: list[PlanAction] = []
if not previous:
actions.append(
PlanAction(
"create",
"installation",
"Create the first deployment revision.",
)
)
if previous_spec_digest != spec_digest:
actions.append(
PlanAction(
"reconfigure",
"installation",
"Reconcile runtime configuration with installation.json.",
)
)
if previous_compose_digest != compose_digest:
actions.append(
PlanAction(
"render",
"compose",
"Render a new deterministic Compose definition.",
)
)
if previous_environment_fingerprint != environment_digest:
actions.append(
PlanAction(
"reconfigure",
"environment",
"Reconcile changed secret bindings or runtime environment values.",
)
)
for name in sorted(desired_services - previous_services):
actions.append(PlanAction("start", name, "Start the selected service."))
for name in sorted(previous_services - desired_services):
actions.append(
PlanAction(
"remove",
name,
"Remove the service container; retained volumes are not deleted.",
)
)
if (
previous
and previous_spec_digest == spec_digest
and previous_compose_digest == compose_digest
and previous_environment_fingerprint == environment_digest
and previous_services == desired_services
):
actions.append(
PlanAction(
"noop", "installation", "Desired state matches the last receipt."
)
)
checks = list(static_checks(spec, paths))
if include_host_checks:
checks.extend(host_checks(spec, paths, command_runner=command_runner))
return DeploymentPlan(
installation_id=spec.installation_id,
desired_spec_sha256=spec_digest,
desired_compose_sha256=compose_digest,
desired_environment_fingerprint=environment_digest,
actions=tuple(actions),
checks=tuple(checks),
)
def static_checks(spec: InstallationSpec, paths: BundlePaths) -> tuple[Check, ...]:
checks: list[Check] = []
images = {
"release.api_image": spec.release.api_image,
"release.web_image": spec.release.web_image,
}
if spec.components.postgres.mode == "managed":
images["components.postgres.image"] = spec.components.postgres.image
if spec.components.redis.mode == "managed":
images["components.redis.image"] = spec.components.redis.image
if spec.components.mail.mode == "test-mail":
images["components.mail.image"] = spec.components.mail.image
if spec.components.storage.mode == "garage":
images["components.storage.image"] = spec.components.storage.image
images["components.load_balancer.image"] = spec.components.load_balancer.image
for label, image in images.items():
if image_is_unpublished(image):
checks.append(
Check(
f"{label}.published",
"error",
f"{label} still uses the unpublished placeholder.",
"Set an available image reference before apply.",
)
)
elif not image_is_digest_pinned(image):
level = "error" if spec.profile == "self-hosted" else "warning"
checks.append(
Check(
f"{label}.digest",
level,
f"{label} is not pinned by OCI digest.",
"Use an image@sha256:... reference from a verified distribution.",
)
)
else:
checks.append(
Check(
f"{label}.digest",
"ok",
f"{label} is pinned by OCI digest.",
)
)
if spec.release.manifest_url and spec.release.manifest_sha256:
checks.append(
Check(
"release.manifest",
"ok",
"A distribution manifest URL and expected digest are recorded.",
)
)
else:
checks.append(
Check(
"release.manifest",
"error" if spec.profile == "self-hosted" else "warning",
"No verified distribution manifest is recorded.",
"Use a published signed distribution manifest for self-hosted apply.",
)
)
if spec.profile == "self-hosted":
checks.append(
Check(
"release.signature_verification",
"error",
"Signed distribution-manifest verification is not implemented in the deployer yet.",
"Use the published verifier/bootstrap slice before a production apply.",
)
)
checks.append(
Check(
"modules.image_composition",
"error" if spec.enabled_modules else "warning",
"The selected module set is not yet verified against image package contents.",
"Use the signed distribution composition evidence before production apply.",
)
)
values = read_env(paths.env)
required = {"MASTER_KEY_B64", "DATABASE_URL"}
if spec.components.redis.mode != "disabled":
required.add("REDIS_URL")
if spec.components.storage.mode in {"s3", "garage"}:
required.update(
{
"FILE_STORAGE_S3_ENDPOINT_URL",
"FILE_STORAGE_S3_REGION",
"FILE_STORAGE_S3_ACCESS_KEY_ID",
"FILE_STORAGE_S3_SECRET_ACCESS_KEY",
"FILE_STORAGE_S3_BUCKET",
}
)
if spec.components.storage.mode == "garage":
required.update(
{
"GARAGE_DEFAULT_ACCESS_KEY",
"GARAGE_DEFAULT_SECRET_KEY",
"GARAGE_DEFAULT_BUCKET",
"GARAGE_RPC_SECRET",
"GARAGE_ADMIN_TOKEN",
"GARAGE_METRICS_TOKEN",
}
)
missing = sorted(name for name in required if not values.get(name))
checks.append(
Check(
"secrets.required",
"error" if missing else "ok",
(
"Missing required secret values: " + ", ".join(missing)
if missing
else "Required runtime secret references are populated."
),
"Re-run configure with the required external service values."
if missing
else "",
)
)
if paths.env.exists():
mode = stat.S_IMODE(paths.env.stat().st_mode)
checks.append(
Check(
"secrets.permissions",
"error" if mode & 0o077 else "ok",
(
f"{paths.env.name} has unsafe mode {mode:04o}."
if mode & 0o077
else f"{paths.env.name} is private ({mode:04o})."
),
f"Run chmod 600 {paths.env}" if mode & 0o077 else "",
)
)
if spec.profile == "self-hosted":
checks.append(
Check(
"bootstrap.administrator",
"warning",
"Production first-administrator enrollment is not automated yet.",
"Use the controlled one-time administrator procedure until the enrollment slice lands.",
)
)
if spec.components.mail.mode == "external-relay":
checks.append(
Check(
"mail.provisioning",
"warning",
"The external relay is selected but Mail profile seeding remains an explicit post-install configuration task.",
"Create the reusable server and credential profile in Mail after first login.",
)
)
if spec.components.storage.mode == "local" and spec.profile == "self-hosted":
checks.append(
Check(
"storage.local",
"warning",
"Local file storage is suitable for one host but prevents stateless API scale-out.",
"Include files-data in backup/restore drills or configure S3 storage.",
)
)
if spec.components.storage.mode == "garage":
checks.append(
Check(
"storage.garage.single_node",
"warning",
(
"Managed Garage is persistent S3-compatible storage, but "
"this Compose profile runs one Garage node without data redundancy."
),
(
"Use an external multi-node Garage/S3 service and tested "
"backup/restore for an availability-sensitive installation."
),
)
)
checks.append(
Check(
"topology.load_balancing",
"ok",
(
"Managed HAProxy balances "
f"{spec.replicas.web} WebUI and {spec.replicas.api} API replica(s)."
),
)
)
if spec.replicas.worker > 1:
checks.append(
Check(
"topology.worker_scaling",
"ok",
f"{spec.replicas.worker} workers share the configured Redis queues.",
)
)
return tuple(checks)
def host_checks(
spec: InstallationSpec,
paths: BundlePaths,
*,
command_runner: CommandRunner | None = None,
) -> tuple[Check, ...]:
checks: list[Check] = []
machine = platform.machine().lower()
supported = machine in {"x86_64", "amd64", "aarch64", "arm64"}
checks.append(
Check(
"host.architecture",
"ok" if supported else "error",
f"Host architecture is {machine or 'unknown'}.",
"Use a supported amd64 or arm64 host." if not supported else "",
)
)
memory_bytes = _memory_bytes()
if memory_bytes is not None:
minimum = 4 * 1024**3
checks.append(
Check(
"host.memory",
"ok" if memory_bytes >= minimum else "warning",
f"Host memory is approximately {memory_bytes / 1024**3:.1f} GiB.",
"Use at least 4 GiB for the base container profile."
if memory_bytes < minimum
else "",
)
)
cpu_count = os.cpu_count()
if cpu_count is not None:
checks.append(
Check(
"host.cpu",
"ok" if cpu_count >= 2 else "warning",
f"Host reports {cpu_count} logical CPU(s).",
"Use at least two logical CPUs for the base container profile."
if cpu_count < 2
else "",
)
)
entropy = _entropy_available()
if entropy is not None:
checks.append(
Check(
"host.entropy",
"ok" if entropy >= 256 else "warning",
f"Kernel entropy availability is {entropy}.",
"Wait for or provide sufficient host entropy before generating production keys."
if entropy < 256
else "",
)
)
free_bytes = shutil.disk_usage(paths.root).free
minimum_free = 10 * 1024**3
checks.append(
Check(
"host.disk",
"ok" if free_bytes >= minimum_free else "warning",
f"Free installation filesystem space is approximately {free_bytes / 1024**3:.1f} GiB.",
"Keep at least 10 GiB free before pulling images and creating data volumes."
if free_bytes < minimum_free
else "",
)
)
docker = shutil.which("docker")
if docker is None:
checks.append(
Check(
"host.compose",
"error",
"Docker CLI is not installed or not on PATH.",
"Install Docker Engine with the Compose v2 plugin.",
)
)
else:
runner = command_runner or _run_command
result = runner((docker, "compose", "version", "--short"), paths.root)
checks.append(
Check(
"host.compose",
"ok" if result.returncode == 0 else "error",
(
f"Docker Compose is available ({result.stdout.strip()})."
if result.returncode == 0
else "Docker Compose v2 is not available."
),
"Install or enable the Docker Compose v2 plugin."
if result.returncode != 0
else "",
)
)
daemon = runner(
(docker, "info", "--format", "{{.ServerVersion}}"),
paths.root,
)
checks.append(
Check(
"host.container_runtime",
"ok" if daemon.returncode == 0 else "error",
(
f"Docker daemon is reachable ({daemon.stdout.strip()})."
if daemon.returncode == 0
else "Docker CLI cannot reach the Docker daemon."
),
"Start Docker and grant this operator access to the daemon."
if daemon.returncode != 0
else "",
)
)
values = read_env(paths.env)
if spec.components.postgres.mode == "external":
checks.append(
_endpoint_check(
"external.postgres",
"External PostgreSQL",
values.get("DATABASE_URL", ""),
default_ports={"postgresql": 5432, "postgresql+psycopg": 5432},
)
)
if spec.components.redis.mode == "external":
checks.append(
_endpoint_check(
"external.redis",
"External Redis",
values.get("REDIS_URL", ""),
default_ports={"redis": 6379, "rediss": 6379},
)
)
if spec.components.storage.mode == "s3":
checks.append(
_endpoint_check(
"external.s3",
"S3-compatible storage",
values.get("FILE_STORAGE_S3_ENDPOINT_URL", ""),
default_ports={"http": 80, "https": 443},
)
)
receipt = _read_receipt(paths.receipt)
previous_listen = receipt.get("listen")
desired_listen = {
"address": spec.listen.address,
"port": spec.listen.port,
}
if not receipt or previous_listen != desired_listen:
available = _port_available(spec.listen.address, spec.listen.port)
checks.append(
Check(
"host.listen_port",
"ok" if available else "error",
(
f"Listen endpoint {spec.listen.address}:{spec.listen.port} is available."
if available
else f"Listen endpoint {spec.listen.address}:{spec.listen.port} is already in use."
),
"Choose another listen port or stop the conflicting service."
if not available
else "",
)
)
return tuple(checks)
def _read_receipt(path: Path) -> Mapping[str, object]:
if not path.exists():
return {}
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return {}
return value if isinstance(value, dict) else {}
def _memory_bytes() -> int | None:
try:
for line in Path("/proc/meminfo").read_text(encoding="utf-8").splitlines():
if line.startswith("MemTotal:"):
return int(line.split()[1]) * 1024
except (OSError, ValueError, IndexError):
return None
return None
def _entropy_available() -> int | None:
try:
return int(
Path("/proc/sys/kernel/random/entropy_avail")
.read_text(encoding="utf-8")
.strip()
)
except (OSError, ValueError):
return None
def _port_available(address: str, port: int) -> bool:
family = socket.AF_INET6 if ":" in address else socket.AF_INET
with socket.socket(family, socket.SOCK_STREAM) as handle:
try:
handle.bind((address, port))
except OSError:
return False
return True
def _endpoint_check(
check_id: str,
label: str,
url: str,
*,
default_ports: Mapping[str, int],
) -> Check:
try:
parsed = urlsplit(url)
host = parsed.hostname
port = parsed.port or default_ports.get(parsed.scheme)
except ValueError as exc:
return Check(
check_id,
"error",
f"{label} endpoint is invalid: {exc}",
"Correct the private endpoint binding and rerun doctor.",
)
if not host or port is None:
return Check(
check_id,
"error",
f"{label} endpoint has no usable host and port.",
"Correct the private endpoint binding and rerun doctor.",
)
try:
with socket.create_connection((host, port), timeout=1.5):
pass
except OSError as exc:
return Check(
check_id,
"error",
f"{label} is not reachable at {host}:{port}: {exc}",
"Check DNS, routing, firewall, service health, and the selected endpoint.",
)
return Check(
check_id,
"ok",
f"{label} is reachable at {host}:{port}.",
)
def _run_command(argv: Sequence[str], cwd: Path) -> subprocess.CompletedProcess[str]:
return subprocess.run(
list(argv),
cwd=cwd,
check=False,
capture_output=True,
text=True,
timeout=10,
)
+15 -1
View File
@@ -27,12 +27,17 @@ EXCLUDED_PARTS = {
".git",
".gitea",
".venv",
".mypy_cache",
".pytest_cache",
".ruff_cache",
"node_modules",
"__pycache__",
"audit-reports",
"dist",
"build",
".cache",
".module-test-build",
"runtime",
}
EXCLUDED_NAMES = {
"package-lock.json",
@@ -40,6 +45,14 @@ EXCLUDED_NAMES = {
"yarn.lock",
"uv.lock",
}
REPO_DOC_NAME_TOKENS = {
"architecture",
"backlog",
"plan",
"roadmap",
"todo",
"workflow",
}
@dataclasses.dataclass(frozen=True)
@@ -216,7 +229,8 @@ def is_repo_doc(repo_root_path: pathlib.Path, path: pathlib.Path) -> bool:
return False
if rel.parts[0] in {"docs", "doc", "codex"} and path.suffix.lower() in {".md", ".txt", ".csv"}:
return True
if re.search(r"(backlog|todo|roadmap|plan|architecture|workflow|manifest)", path.name, re.IGNORECASE):
name_tokens = set(re.split(r"[^a-z0-9]+", path.stem.lower()))
if name_tokens & REPO_DOC_NAME_TOKENS:
return True
return False
+366
View File
@@ -0,0 +1,366 @@
#!/usr/bin/env node
import fs from "node:fs";
import path from "node:path";
import { pathToFileURL } from "node:url";
const [metaRootArgument] = process.argv.slice(2);
if (!metaRootArgument) {
throw new Error("Usage: extract-webui-structure.mjs META_ROOT");
}
const metaRoot = path.resolve(metaRootArgument);
const repositoryCatalog = JSON.parse(
fs.readFileSync(path.join(metaRoot, "repositories.json"), "utf8")
);
const workspaceRoot = path.resolve(repositoryCatalog.default_parent);
const typescriptPath = path.join(
workspaceRoot,
"govoplan-core",
"webui",
"node_modules",
"typescript",
"lib",
"typescript.js"
);
const ts = await import(pathToFileURL(typescriptPath).href);
const fieldComponents = new Set([
"input",
"select",
"textarea",
"Checkbox",
"DateTimeField",
"EmailAddressInput",
"ReferenceSelect",
"SearchableSelect",
"ToggleSwitch"
]);
const fieldComponentPattern =
/(?:Input|Field|Select|Picker|Toggle|Checkbox|Radio|Editor)$/;
const labelAttributes = new Set([
"aria-label",
"description",
"help",
"helperText",
"helpText",
"label",
"placeholder",
"title"
]);
const helpAttributes = new Set([
"description",
"help",
"helperText",
"helpText"
]);
const result = {
fields: [],
labels: [],
visibleText: [],
translationCatalog: {},
translationUsages: [],
dynamicTranslationUsages: [],
routes: [],
navigation: [],
frontendApiReferences: [],
uiCapabilities: []
};
for (const repository of repositoryCatalog.repositories) {
const sourceRoot = path.join(workspaceRoot, repository.path, "webui", "src");
if (!fs.existsSync(sourceRoot)) continue;
for (const sourcePath of sourceFiles(sourceRoot)) {
inspectSource(repository.name, sourceRoot, sourcePath);
}
}
process.stdout.write(`${JSON.stringify(result)}\n`);
function sourceFiles(root) {
const files = [];
const pending = [root];
while (pending.length > 0) {
const current = pending.pop();
for (const entry of fs.readdirSync(current, { withFileTypes: true })) {
if (
entry.name === "node_modules" ||
entry.name.startsWith(".") ||
entry.name === "dist"
) {
continue;
}
const candidate = path.join(current, entry.name);
if (entry.isDirectory()) pending.push(candidate);
else if (/\.(?:ts|tsx)$/.test(entry.name)) files.push(candidate);
}
}
return files.sort();
}
function inspectSource(repository, sourceRoot, sourcePath) {
const sourceText = fs.readFileSync(sourcePath, "utf8");
const sourceFile = ts.createSourceFile(
sourcePath,
sourceText,
ts.ScriptTarget.Latest,
true,
sourcePath.endsWith(".tsx") ? ts.ScriptKind.TSX : ts.ScriptKind.TS
);
const relativeFile = path.relative(path.join(workspaceRoot, repository), sourcePath);
function location(node) {
const position = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile));
return {
repository,
file: relativeFile,
line: position.line + 1,
column: position.character + 1
};
}
function visit(node) {
if (ts.isJsxOpeningElement(node) || ts.isJsxSelfClosingElement(node)) {
inspectJsxOpening(node, location);
}
if (ts.isJsxText(node)) {
const value = node.getText(sourceFile).replace(/\s+/g, " ").trim();
if (value) {
result.visibleText.push({
...location(node),
value,
translationKey: value.startsWith("i18n:") ? value : null
});
}
}
if (ts.isStringLiteralLike(node)) {
inspectString(node.text, node, location);
} else if (ts.isTemplateExpression(node)) {
inspectString(templateText(node), node, location);
}
if (ts.isPropertyAssignment(node)) {
inspectProperty(node, location);
inspectTranslationProperty(node, location);
}
ts.forEachChild(node, visit);
}
visit(sourceFile);
function inspectJsxOpening(node, locate) {
const component = node.tagName.getText(sourceFile);
const attributes = new Map();
for (const attribute of node.attributes.properties) {
if (!ts.isJsxAttribute(attribute)) continue;
attributes.set(
attribute.name.getText(sourceFile),
jsxAttributeValue(attribute)
);
}
for (const [attribute, value] of attributes) {
if (!labelAttributes.has(attribute) || value === null) continue;
result.labels.push({
...locate(node),
component,
attribute,
value,
translationKey: value.startsWith("i18n:") ? value : null
});
}
const isField =
fieldComponents.has(component) ||
(fieldComponentPattern.test(component) && component !== "FormField");
if (!isField) return;
const parentFormField = nearestFormField(node);
const parentAttributes = parentFormField
? jsxAttributes(parentFormField)
: new Map();
const label =
attributes.get("label") ??
attributes.get("aria-label") ??
parentAttributes.get("label") ??
null;
const help = firstAttribute(attributes, helpAttributes) ??
firstAttribute(parentAttributes, helpAttributes);
result.fields.push({
...locate(node),
component,
name:
attributes.get("name") ??
attributes.get("id") ??
attributes.get("field") ??
null,
label,
placeholder: attributes.get("placeholder") ?? null,
help: help ?? null,
helpCandidate: help === null
});
}
function nearestFormField(node) {
let current = node.parent;
while (current) {
if (
ts.isJsxElement(current) &&
current.openingElement.tagName.getText(sourceFile) === "FormField"
) {
return current.openingElement;
}
if (
ts.isJsxOpeningElement(current) ||
ts.isJsxSelfClosingElement(current)
) {
const name = current.tagName.getText(sourceFile);
if (name !== "FormField") return null;
}
current = current.parent;
}
return null;
}
function jsxAttributes(node) {
const mapped = new Map();
for (const attribute of node.attributes.properties) {
if (!ts.isJsxAttribute(attribute)) continue;
mapped.set(
attribute.name.getText(sourceFile),
jsxAttributeValue(attribute)
);
}
return mapped;
}
function jsxAttributeValue(attribute) {
if (!attribute.initializer) return "true";
if (ts.isStringLiteral(attribute.initializer)) return attribute.initializer.text;
if (!ts.isJsxExpression(attribute.initializer)) return null;
const expression = attribute.initializer.expression;
if (!expression) return null;
if (ts.isStringLiteralLike(expression)) return expression.text;
if (ts.isNoSubstitutionTemplateLiteral(expression)) return expression.text;
if (ts.isTemplateExpression(expression)) return templateText(expression);
return null;
}
function inspectString(value, node, locate) {
if (value.startsWith("i18n:") && value.length > "i18n:".length) {
const target = value.includes("${}") || isStringPrefixCheck(node)
? result.dynamicTranslationUsages
: result.translationUsages;
target.push({ ...locate(node), key: value });
}
if (value.includes("/api/")) {
result.frontendApiReferences.push({ ...locate(node), path: value });
}
}
function isStringPrefixCheck(node) {
const call = node.parent;
if (!ts.isCallExpression(call) || !ts.isPropertyAccessExpression(call.expression)) {
return false;
}
return ["endsWith", "includes", "startsWith"].includes(
call.expression.name.text
);
}
function inspectProperty(node, locate) {
const propertyName = propertyNameText(node.name);
const value = staticExpressionText(node.initializer);
if (value === null) return;
if (propertyName === "path" && value.startsWith("/") && !value.includes("/api/")) {
result.routes.push({ ...locate(node), path: value });
}
if (propertyName === "to" && value.startsWith("/")) {
result.navigation.push({ ...locate(node), path: value });
}
if (
ancestorPropertyName(node, "uiCapabilities") &&
typeof propertyName === "string"
) {
result.uiCapabilities.push({ ...locate(node), name: propertyName });
}
}
function inspectTranslationProperty(node, locate) {
const key = propertyNameText(node.name);
if (!key?.startsWith("i18n:")) return;
const value = staticExpressionText(node.initializer);
if (value === null) return;
const locale = nearestLocaleProperty(node);
if (!locale) return;
result.translationCatalog[locale] ??= {};
result.translationCatalog[locale][key] = {
value,
...locate(node)
};
}
function nearestLocaleProperty(node) {
let current = node.parent;
while (current) {
if (
ts.isPropertyAssignment(current) &&
(propertyNameText(current.name) === "en" ||
propertyNameText(current.name) === "de")
) {
return propertyNameText(current.name);
}
current = current.parent;
}
return null;
}
function ancestorPropertyName(node, expected) {
let current = node.parent;
while (current) {
if (
ts.isPropertyAssignment(current) &&
propertyNameText(current.name) === expected
) {
return true;
}
current = current.parent;
}
return false;
}
}
function firstAttribute(attributes, names) {
for (const name of names) {
const value = attributes.get(name);
if (value !== undefined && value !== null) return value;
}
return null;
}
function propertyNameText(name) {
if (
ts.isIdentifier(name) ||
ts.isStringLiteral(name) ||
ts.isNumericLiteral(name)
) {
return name.text;
}
return null;
}
function staticExpressionText(node) {
if (ts.isStringLiteralLike(node) || ts.isNoSubstitutionTemplateLiteral(node)) {
return node.text;
}
if (ts.isTemplateExpression(node)) return templateText(node);
return null;
}
function templateText(node) {
return (
node.head.text +
node.templateSpans
.map((span) => "${}" + span.literal.text)
.join("")
);
}
@@ -0,0 +1,462 @@
#!/usr/bin/env python3
"""Extract a source-derived GovOPlaN UI, translation, module, and API inventory."""
from __future__ import annotations
import argparse
import ast
from collections import Counter
from dataclasses import asdict, is_dataclass
import importlib
import json
from pathlib import Path
import re
import subprocess
import sys
from typing import Any
META_ROOT = Path(__file__).resolve().parents[2]
HTTP_METHODS = {"delete", "get", "head", "options", "patch", "post", "put"}
PATH_PARAMETER = re.compile(r"\$\{[^{}]*\}|\{[^{}]+\}")
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--output-dir",
type=Path,
default=META_ROOT / "audit-reports" / "platform-inventory",
)
parser.add_argument(
"--strict",
action="store_true",
help="Fail when a used translation key is absent from a generated locale catalog.",
)
args = parser.parse_args()
catalog = json.loads(
(META_ROOT / "repositories.json").read_text(encoding="utf-8")
)
workspace_root = Path(catalog["default_parent"]).resolve()
webui = _extract_webui()
backend_endpoints = _extract_backend_endpoints(catalog, workspace_root)
manifests = _extract_manifests(catalog, workspace_root)
inventory = _assemble_inventory(
webui=webui,
backend_endpoints=backend_endpoints,
manifests=manifests,
)
output_dir = args.output_dir.resolve()
output_dir.mkdir(parents=True, exist_ok=True)
json_path = output_dir / "platform-interface-inventory.json"
markdown_path = output_dir / "platform-interface-inventory.md"
json_path.write_text(
json.dumps(inventory, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
markdown_path.write_text(_render_markdown(inventory), encoding="utf-8")
print(f"Platform inventory JSON: {json_path}")
print(f"Platform inventory summary: {markdown_path}")
if args.strict and inventory["translation_health"]["missing_catalog_entries"]:
print(
"Used translation keys are missing from generated catalogs.",
file=sys.stderr,
)
return 1
return 0
def _extract_webui() -> dict[str, Any]:
helper = META_ROOT / "tools" / "inventory" / "extract-webui-structure.mjs"
completed = subprocess.run(
["node", str(helper), str(META_ROOT)],
check=True,
capture_output=True,
text=True,
)
return json.loads(completed.stdout)
def _extract_backend_endpoints(
catalog: dict[str, Any],
workspace_root: Path,
) -> list[dict[str, Any]]:
endpoints: list[dict[str, Any]] = []
for repository in catalog["repositories"]:
repository_root = workspace_root / repository["path"]
source_root = repository_root / "src"
if not source_root.is_dir():
continue
for source_path in sorted(source_root.rglob("*.py")):
try:
tree = ast.parse(
source_path.read_text(encoding="utf-8"),
filename=str(source_path),
)
except (OSError, SyntaxError, UnicodeDecodeError):
continue
prefixes = _router_prefixes(tree)
for node in ast.walk(tree):
if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)):
continue
for decorator in node.decorator_list:
endpoint = _endpoint_from_decorator(
decorator,
prefixes=prefixes,
)
if endpoint is None:
continue
endpoints.append(
{
"repository": repository["name"],
"file": str(source_path.relative_to(repository_root)),
"line": node.lineno,
"handler": node.name,
**endpoint,
}
)
return sorted(
endpoints,
key=lambda item: (
item["repository"],
item["path"],
item["method"],
item["file"],
item["line"],
),
)
def _router_prefixes(tree: ast.AST) -> dict[str, str]:
prefixes: dict[str, str] = {}
for node in ast.walk(tree):
if not isinstance(node, (ast.Assign, ast.AnnAssign)):
continue
value = node.value
if not isinstance(value, ast.Call):
continue
function_name = _call_name(value.func)
if function_name not in {"APIRouter", "fastapi.APIRouter"}:
continue
prefix = ""
for keyword in value.keywords:
if keyword.arg == "prefix":
prefix = _static_string(keyword.value) or ""
targets = node.targets if isinstance(node, ast.Assign) else [node.target]
for target in targets:
if isinstance(target, ast.Name):
prefixes[target.id] = prefix
return prefixes
def _endpoint_from_decorator(
decorator: ast.expr,
*,
prefixes: dict[str, str],
) -> dict[str, str] | None:
if not isinstance(decorator, ast.Call) or not isinstance(
decorator.func, ast.Attribute
):
return None
method = decorator.func.attr.lower()
if method not in HTTP_METHODS or not decorator.args:
return None
route = _static_string(decorator.args[0])
if route is None:
return None
owner = decorator.func.value.id if isinstance(decorator.func.value, ast.Name) else ""
prefix = prefixes.get(owner, "")
return {
"method": method.upper(),
"path": _join_route(prefix, route),
"router": owner,
}
def _extract_manifests(
catalog: dict[str, Any],
workspace_root: Path,
) -> list[dict[str, Any]]:
source_roots = [
workspace_root / repository["path"] / "src"
for repository in catalog["repositories"]
if (workspace_root / repository["path"] / "src").is_dir()
]
sys.path[:0] = [str(path) for path in source_roots]
manifests: list[dict[str, Any]] = []
for repository in catalog["repositories"]:
source_root = workspace_root / repository["path"] / "src"
if not source_root.is_dir():
continue
for manifest_path in sorted(source_root.glob("*/backend/manifest.py")):
module_name = ".".join(
manifest_path.relative_to(source_root).with_suffix("").parts
)
loaded = importlib.import_module(module_name)
manifest = loaded.get_manifest()
frontend = manifest.frontend
manifests.append(
{
"repository": repository["name"],
"id": manifest.id,
"name": manifest.name,
"version": manifest.version,
"dependencies": list(manifest.dependencies),
"optional_dependencies": list(manifest.optional_dependencies),
"required_capabilities": list(manifest.required_capabilities),
"provided_interfaces": [
{"name": item.name, "version": item.version}
for item in manifest.provides_interfaces
],
"runtime_capabilities": sorted(manifest.capability_factories),
"permissions": [
{
"scope": permission.scope,
"label": permission.label,
"level": permission.level,
}
for permission in manifest.permissions
],
"frontend": (
{
"package": frontend.package_name,
"routes": [
_plain_value(route) for route in frontend.routes
],
"nav_items": [
_plain_value(item) for item in frontend.nav_items
],
"view_surfaces": [
_plain_value(surface)
for surface in frontend.view_surfaces
],
}
if frontend is not None
else None
),
}
)
return sorted(manifests, key=lambda item: item["id"])
def _assemble_inventory(
*,
webui: dict[str, Any],
backend_endpoints: list[dict[str, Any]],
manifests: list[dict[str, Any]],
) -> dict[str, Any]:
frontend_refs = webui["frontendApiReferences"]
frontend_paths = {
canonical_api_path(reference["path"])
for reference in frontend_refs
if canonical_api_path(reference["path"])
}
unreferenced = [
endpoint
for endpoint in backend_endpoints
if canonical_api_path(endpoint["path"]) not in frontend_paths
]
usages = {item["key"] for item in webui["translationUsages"]}
catalogs = webui["translationCatalog"]
catalog_keys = {
locale: set(entries)
for locale, entries in catalogs.items()
}
expected_locales = sorted(catalog_keys)
missing_catalog_entries = [
{
"key": key,
"missing_locales": [
locale
for locale in expected_locales
if key not in catalog_keys[locale]
],
}
for key in sorted(usages)
if any(key not in catalog_keys[locale] for locale in expected_locales)
]
fields = webui["fields"]
help_candidates = [field for field in fields if field["helpCandidate"]]
return {
"schema_version": 1,
"scope": {
"source": "local GovOPlaN repository catalog",
"limitations": [
"Static extraction cannot resolve runtime-computed labels, routes, or API paths.",
"A backend endpoint without a static WebUI reference may intentionally serve public clients, workers, connectors, or external integrations.",
"A field marked as a help candidate may receive contextual help from a surrounding dynamic component.",
],
},
"modules": manifests,
"ui": {
"fields": fields,
"labels": webui["labels"],
"visible_text": webui["visibleText"],
"routes": webui["routes"],
"navigation": webui["navigation"],
"ui_capabilities": webui["uiCapabilities"],
"help_candidates": help_candidates,
},
"translations": {
"catalog": catalogs,
"usages": webui["translationUsages"],
"dynamic_usages": webui["dynamicTranslationUsages"],
},
"translation_health": {
"locales": expected_locales,
"used_keys": len(usages),
"missing_catalog_entries": missing_catalog_entries,
},
"api": {
"backend_endpoints": backend_endpoints,
"frontend_references": frontend_refs,
"unreferenced_by_static_webui_scan": unreferenced,
},
"summary": {
"modules": len(manifests),
"ui_fields": len(fields),
"ui_fields_with_static_help": len(fields) - len(help_candidates),
"help_review_candidates": len(help_candidates),
"label_attributes": len(webui["labels"]),
"visible_text_nodes": len(webui["visibleText"]),
"frontend_routes": len(webui["routes"]),
"backend_endpoints": len(backend_endpoints),
"frontend_api_references": len(frontend_refs),
"backend_endpoints_without_static_webui_reference": len(unreferenced),
},
}
def _render_markdown(inventory: dict[str, Any]) -> str:
summary = inventory["summary"]
missing = inventory["translation_health"]["missing_catalog_entries"]
help_by_repository = Counter(
item["repository"] for item in inventory["ui"]["help_candidates"]
)
endpoint_by_repository = Counter(
item["repository"]
for item in inventory["api"]["unreferenced_by_static_webui_scan"]
)
lines = [
"# GovOPlaN Platform Interface Inventory",
"",
"Generated from runtime module manifests plus TypeScript and Python ASTs.",
"Counts are source evidence, not a claim that every surface is enabled or reachable.",
"",
"## Summary",
"",
f"- Modules: {summary['modules']}",
f"- UI fields: {summary['ui_fields']}",
f"- Fields with statically associated help: {summary['ui_fields_with_static_help']}",
f"- Help review candidates: {summary['help_review_candidates']}",
f"- Label attributes: {summary['label_attributes']}",
f"- Frontend routes: {summary['frontend_routes']}",
f"- Backend endpoints: {summary['backend_endpoints']}",
f"- Frontend API references: {summary['frontend_api_references']}",
(
"- Backend endpoints without a static WebUI reference: "
f"{summary['backend_endpoints_without_static_webui_reference']}"
),
f"- Used translation keys missing from a locale catalog: {len(missing)}",
"",
"## Help Review Candidates",
"",
"| Repository | Fields |",
"| --- | ---: |",
]
lines.extend(
f"| `{repository}` | {count} |"
for repository, count in sorted(help_by_repository.items())
)
lines.extend(
[
"",
"## Endpoints Without Static WebUI References",
"",
"These are review candidates. Public APIs, worker callbacks, connector",
"endpoints, health checks, and dynamically assembled paths are legitimate",
"reasons for appearing here.",
"",
"| Repository | Endpoints |",
"| --- | ---: |",
]
)
lines.extend(
f"| `{repository}` | {count} |"
for repository, count in sorted(endpoint_by_repository.items())
)
lines.extend(
[
"",
"## Interpretation",
"",
"Use the JSON artifact for exact file and line evidence. Missing help is",
"a triage list, not an automatic defect. Endpoint coverage requires an",
"owner classification before enforcement. Runtime-computed structures",
"need explicit manifest metadata to become canonically visible.",
"",
]
)
return "\n".join(lines)
def canonical_api_path(value: str) -> str:
path = value.split("?", 1)[0].strip()
if "/api/" in path:
path = path[path.index("/api/") :]
path = re.sub(r"^/api/v\d+", "", path)
path = PATH_PARAMETER.sub("{}", path)
path = re.sub(r"/+", "/", path)
return path.rstrip("/") or "/"
def _join_route(prefix: str, route: str) -> str:
return f"/{prefix.strip('/')}/{route.strip('/')}".replace("//", "/")
def _static_string(node: ast.AST) -> str | None:
if isinstance(node, ast.Constant) and isinstance(node.value, str):
return node.value
if isinstance(node, ast.JoinedStr):
pieces: list[str] = []
for value in node.values:
if isinstance(value, ast.Constant) and isinstance(value.value, str):
pieces.append(value.value)
elif isinstance(value, ast.FormattedValue):
pieces.append("${}")
else:
return None
return "".join(pieces)
return None
def _call_name(node: ast.AST) -> str:
if isinstance(node, ast.Name):
return node.id
if isinstance(node, ast.Attribute):
parent = _call_name(node.value)
return f"{parent}.{node.attr}" if parent else node.attr
return ""
def _plain_value(value: Any) -> Any:
if is_dataclass(value):
return {
key: _plain_value(item)
for key, item in asdict(value).items()
}
if isinstance(value, tuple):
return [_plain_value(item) for item in value]
if isinstance(value, dict):
return {
str(key): _plain_value(item)
for key, item in value.items()
}
return value
if __name__ == "__main__":
raise SystemExit(main())
+11 -1
View File
@@ -22,6 +22,7 @@ FRONTEND_USE_POLLING="${GOVOPLAN_FRONTEND_USE_POLLING:-1}"
FRONTEND_POLLING_INTERVAL="${GOVOPLAN_FRONTEND_POLLING_INTERVAL:-500}"
AUTO_SYNC_PYTHON="${GOVOPLAN_AUTO_SYNC_PYTHON:-1}"
DEV_DATABASE_BACKEND="${GOVOPLAN_DEV_DATABASE_BACKEND:-postgres}"
BACKEND_RELOAD_MODULES="${GOVOPLAN_BACKEND_RELOAD_MODULES:-}"
LOG_DIR="${GOVOPLAN_DEV_LOG_DIR:-$META_ROOT/runtime/dev-launcher}"
BACKEND_LOG="$LOG_DIR/backend.log"
@@ -167,7 +168,16 @@ port_is_free "$BACKEND_HOST" "$BACKEND_PORT" || fail "$BACKEND_URL is already in
port_is_free "$FRONTEND_HOST" "$FRONTEND_PORT" || fail "$FRONTEND_URL is already in use"
printf 'Starting GovOPlaN backend at %s\n' "$BACKEND_URL"
run_grouped "$ROOT" "$BACKEND_LOG" "$PYTHON" -m govoplan_core.devserver --host "$BACKEND_HOST" --port "$BACKEND_PORT"
backend_args=(-m govoplan_core.devserver --host "$BACKEND_HOST" --port "$BACKEND_PORT")
if [ "$BACKEND_RELOAD_MODULES" = "none" ]; then
backend_args+=(--reload-core-only)
elif [ -n "$BACKEND_RELOAD_MODULES" ]; then
IFS=',' read -r -a reload_modules <<< "$BACKEND_RELOAD_MODULES"
for reload_module in "${reload_modules[@]}"; do
[ -n "$reload_module" ] && backend_args+=(--reload-module "$reload_module")
done
fi
run_grouped "$ROOT" "$BACKEND_LOG" "$PYTHON" "${backend_args[@]}"
backend_pid="$run_grouped_pid"
printf 'Waiting for %s/health\n' "$BACKEND_URL"
+1 -1
View File
@@ -51,7 +51,7 @@ set +a
export APP_ENV="${APP_ENV:-staging}"
export GOVOPLAN_INSTALL_PROFILE="${GOVOPLAN_INSTALL_PROFILE:-production-like}"
export ENABLED_MODULES="${ENABLED_MODULES:-tenancy,organizations,identity,access,admin,dashboard,policy,audit,campaigns,files,mail,calendar,poll,scheduling,notifications,docs,ops}"
export ENABLED_MODULES="${ENABLED_MODULES:-tenancy,organizations,identity,access,admin,dashboard,policy,audit,campaigns,files,mail,calendar,poll,scheduling,connectors,datasources,dataflow,workflow,views,search,risk_compliance,notifications,docs,ops}"
export DATABASE_URL="${DATABASE_URL:-${GOVOPLAN_PRODUCTION_LIKE_DATABASE_URL:-postgresql+psycopg://govoplan:govoplan-dev@127.0.0.1:55433/govoplan}}"
export GOVOPLAN_DATABASE_URL_PGTOOLS="${GOVOPLAN_DATABASE_URL_PGTOOLS:-${GOVOPLAN_PRODUCTION_LIKE_DATABASE_URL_PGTOOLS:-postgresql://govoplan:govoplan-dev@127.0.0.1:55433/govoplan}}"
export REDIS_URL="${REDIS_URL:-${GOVOPLAN_PRODUCTION_LIKE_REDIS_URL:-redis://127.0.0.1:56379/0}}"
+1 -1
View File
@@ -66,7 +66,7 @@ export DATABASE_URL="${DATABASE_URL:-${GOVOPLAN_PRODUCTION_LIKE_DATABASE_URL:-po
export GOVOPLAN_DATABASE_URL_PGTOOLS="${GOVOPLAN_DATABASE_URL_PGTOOLS:-${GOVOPLAN_PRODUCTION_LIKE_DATABASE_URL_PGTOOLS:-postgresql://govoplan:govoplan-dev@127.0.0.1:55433/govoplan}}"
export REDIS_URL="${REDIS_URL:-${GOVOPLAN_PRODUCTION_LIKE_REDIS_URL:-redis://127.0.0.1:56379/0}}"
export CELERY_ENABLED="${CELERY_ENABLED:-true}"
export ENABLED_MODULES="${ENABLED_MODULES:-tenancy,organizations,identity,access,admin,dashboard,policy,audit,campaigns,files,mail,calendar,poll,scheduling,notifications,docs,ops}"
export ENABLED_MODULES="${ENABLED_MODULES:-tenancy,organizations,identity,access,admin,dashboard,policy,audit,campaigns,files,mail,calendar,poll,scheduling,connectors,datasources,dataflow,workflow,views,search,risk_compliance,notifications,docs,ops}"
export FILE_STORAGE_BACKEND="${FILE_STORAGE_BACKEND:-local}"
export FILE_STORAGE_LOCAL_ROOT="${FILE_STORAGE_LOCAL_ROOT:-$META_ROOT/runtime/production-like/files}"
export DEV_AUTO_MIGRATE_ENABLED="${DEV_AUTO_MIGRATE_ENABLED:-false}"
+2 -1
View File
@@ -26,7 +26,7 @@ from govoplan_core.server.registry import available_module_manifests # noqa: E4
from govoplan_release.version_alignment import selected_repository_version_issues # noqa: E402
GITEA_BASE = "git+ssh://git@git.add-ideas.de/add-ideas"
GITEA_BASE = "git+ssh://git@git.add-ideas.de/GovOPlaN"
@dataclass(frozen=True, slots=True)
@@ -50,6 +50,7 @@ CATALOG_MODULES = (
name="Tenancy",
description="Tenant registry, tenant settings, and tenant resolution platform module.",
tags=("official", "platform-module"),
webui_package="@govoplan/tenancy-webui",
),
CatalogModule(
module_id="organizations",
+16 -2
View File
@@ -164,9 +164,10 @@ def git_success(path: Path, *args: str, timeout: int = 10) -> bool:
def git(path: Path, *args: str, timeout: int = 10) -> subprocess.CompletedProcess[str]:
command = scoped_git_command(path, *args)
try:
return subprocess.run(
["/usr/bin/git", "-c", "core.hooksPath=/dev/null", *args],
command,
cwd=path,
check=False,
text=True,
@@ -176,7 +177,20 @@ def git(path: Path, *args: str, timeout: int = 10) -> subprocess.CompletedProces
env=sanitized_git_environment(),
)
except subprocess.TimeoutExpired as exc:
return subprocess.CompletedProcess(["/usr/bin/git", *args], 124, exc.stdout or "", exc.stderr or "git command timed out")
return subprocess.CompletedProcess(command, 124, exc.stdout or "", exc.stderr or "git command timed out")
def scoped_git_command(path: Path, *args: str) -> tuple[str, ...]:
"""Trust exactly one resolved catalog checkout for one Git invocation."""
return (
"/usr/bin/git",
"-c",
"core.hooksPath=/dev/null",
"-c",
f"safe.directory={path.resolve()}",
*args,
)
def sanitized_git_environment(
+1
View File
@@ -222,6 +222,7 @@ class ReleasePlanUnit:
status: str
blockers: tuple[str, ...] = ()
warnings: tuple[str, ...] = ()
capabilities: tuple[str, ...] = ()
provides_interfaces: tuple[InterfaceProviderSnapshot, ...] = ()
requires_interfaces: tuple[InterfaceRequirementSnapshot, ...] = ()
gate_findings: tuple[ReleaseGateFinding, ...] = ()
+102 -4
View File
@@ -11,6 +11,7 @@ from pathlib import Path
import re
import secrets
import shlex
import shutil
import stat
import subprocess
import sys
@@ -44,6 +45,8 @@ _MAX_GIT_OUTPUT_BYTES = 16 * 1024 * 1024
_MAX_REMOTE_METADATA_BYTES = 16 * 1024
_MAX_REMOVED_TREE_ENTRIES = 100_000
_MAX_REMOVED_TREE_DEPTH = 64
_RELEASE_GIT_NAME = "GovOPlaN Release Automation"
_RELEASE_GIT_EMAIL = "release@govoplan.invalid"
class CatalogPublicationAmbiguousError(RuntimeError):
@@ -107,6 +110,12 @@ def publish_catalog_candidate(
steps: list[CatalogPublishStep] = []
notes: list[str] = []
blockers: list[str] = []
resolved_npm = npm
if build_web:
try:
resolved_npm = _trusted_npm_command(npm)
except (OSError, RuntimeError) as exc:
blockers.append(f"website build command is not trusted: {exc}")
if not candidate_catalog.exists():
blockers.append(f"candidate catalog is missing: {candidate_catalog}")
@@ -399,9 +408,18 @@ def publish_catalog_candidate(
CatalogPublishStep(
id="build-web",
title="Build website",
detail="Run the website build after copying catalog files.",
detail=(
"Run the website build after copying catalog files with "
f"{resolved_npm}."
),
command=shlex.join(
[npm, "--prefix", str(resolved_web_root), "run", "build"]
[
resolved_npm,
"--prefix",
str(resolved_web_root),
"run",
"build",
]
),
mutating=True,
status="planned" if not apply else "blocked" if blockers else "pending",
@@ -559,7 +577,7 @@ def publish_catalog_candidate(
]
if build_web:
run_checked(
[npm, "--prefix", str(resolved_web_root), "run", "build"],
[resolved_npm, "--prefix", str(resolved_web_root), "run", "build"],
cwd=resolved_web_root,
)
completed_steps = [
@@ -596,6 +614,14 @@ def publish_catalog_candidate(
effective_tag_name,
publication_commit,
)
_seal_git_metadata_file(
resolved_web_root
/ ".git"
/ "refs"
/ "tags"
/ effective_tag_name,
label="website publication tag reference",
)
publication_tag_object = git_text(
resolved_web_root,
"rev-parse",
@@ -1427,9 +1453,38 @@ def commit_publication_tree(
frozen_head,
)
_git_checked(web_root, "read-tree", "--reset", commit_sha)
_seal_git_metadata_file(
web_root / ".git" / "refs" / "heads" / branch,
label="website branch reference",
)
_seal_git_metadata_file(
web_root / ".git" / "index",
label="website Git index",
)
return commit_sha
def _seal_git_metadata_file(path: Path, *, label: str) -> None:
try:
metadata = path.lstat()
if (
stat.S_ISLNK(metadata.st_mode)
or not stat.S_ISREG(metadata.st_mode)
or metadata.st_uid != os.geteuid()
):
raise RuntimeError(f"{label} is not an operator-owned regular file")
os.chmod(path, 0o600, follow_symlinks=False)
sealed = path.lstat()
except OSError as exc:
raise RuntimeError(f"{label} cannot be sealed") from exc
if (
not stat.S_ISREG(sealed.st_mode)
or sealed.st_uid != os.geteuid()
or stat.S_IMODE(sealed.st_mode) != 0o600
):
raise RuntimeError(f"{label} was not sealed to operator-only access")
def verify_committed_publication(
*,
web_root: Path,
@@ -1882,6 +1937,10 @@ def _sanitized_git_environment(
"GIT_CONFIG_COUNT": "0",
"GIT_CONFIG_GLOBAL": os.devnull,
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_AUTHOR_EMAIL": _RELEASE_GIT_EMAIL,
"GIT_AUTHOR_NAME": _RELEASE_GIT_NAME,
"GIT_COMMITTER_EMAIL": _RELEASE_GIT_EMAIL,
"GIT_COMMITTER_NAME": _RELEASE_GIT_NAME,
"GIT_NO_REPLACE_OBJECTS": "1",
"GIT_PAGER": "cat",
"GIT_SSH_COMMAND": "/usr/bin/ssh -o BatchMode=yes -o ConnectTimeout=8",
@@ -1926,6 +1985,10 @@ def _git_bytes(
"-C",
str(path),
"-c",
f"safe.directory={path.resolve()}",
"-c",
"core.sharedRepository=0600",
"-c",
"core.fsmonitor=false",
"-c",
"core.hooksPath=/dev/null",
@@ -2009,10 +2072,14 @@ def git_success(path: Path, *args: str) -> bool:
def run_checked(args: list[str], *, cwd: Path) -> None:
if not args or not Path(args[0]).is_absolute():
raise RuntimeError("publication build command must be an absolute path")
environment = _sanitized_git_environment()
environment["PATH"] = f"{Path(args[0]).parent}:/usr/bin:/bin"
result = subprocess.run(
args,
cwd=cwd,
env=_sanitized_git_environment(),
env=environment,
check=False,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
@@ -2030,3 +2097,34 @@ def run_checked(args: list[str], *, cwd: Path) -> None:
output=result.stdout,
stderr=result.stderr,
)
def _trusted_npm_command(value: str) -> str:
candidate = Path(value)
if not candidate.is_absolute():
if candidate.name != value:
raise RuntimeError(
"npm executable must be an absolute path or the bare name npm"
)
located = shutil.which(value, path="/usr/bin:/bin")
if not located:
raise RuntimeError("npm executable is unavailable on the fixed system path")
candidate = Path(located)
candidate = Path(os.path.abspath(candidate))
permitted_owners = {0, os.geteuid()}
issue = _trusted_runtime_executable_issue(
candidate,
permitted_owners=permitted_owners,
label="npm executable",
)
if issue:
raise RuntimeError(issue)
node = candidate.parent / "node"
issue = _trusted_runtime_executable_issue(
node,
permitted_owners=permitted_owners,
label="Node.js executable",
)
if issue:
raise RuntimeError(issue)
return str(candidate)
File diff suppressed because it is too large Load Diff
@@ -2218,7 +2218,7 @@ def _validated_result_receipt(
"Catalog publication receipt identity is invalid."
)
return dict(value)
if set(value) != {
repository_fields = {
"kind",
"repo",
"head",
@@ -2228,7 +2228,8 @@ def _validated_result_receipt(
"worktree_clean",
"target_tag",
"tag_object",
}:
}
if set(value) not in (repository_fields, repository_fields | {"worktree_sha256"}):
raise ReleaseRunConflict("Repository result receipt is invalid.")
if (
not isinstance(value.get("repo"), str)
@@ -2241,6 +2242,10 @@ def _validated_result_receipt(
or value.get("remote") != "origin"
or not _is_sha256(value.get("remote_sha256"))
or type(value.get("worktree_clean")) is not bool
or (
"worktree_sha256" in value
and not _is_sha256(value.get("worktree_sha256"))
)
or not isinstance(value.get("target_tag"), str)
or len(value["target_tag"]) > 160
or (
@@ -5,7 +5,7 @@ from __future__ import annotations
from pathlib import Path
import subprocess
from .git_state import collect_repository_snapshot
from .git_state import collect_repository_snapshot, scoped_git_command
from .repository_push import command_text, compact_output
from .workspace import load_repository_specs, resolve_repo_path, resolve_workspace_root
@@ -131,7 +131,12 @@ def resolve_message(*, repo: str, version: str | None, message: str | None) -> s
def run(command: tuple[str, ...], *, cwd: Path) -> subprocess.CompletedProcess[str]:
effective_command = (
scoped_git_command(cwd, *command[1:])
if command and Path(command[0]).name == "git"
else command
)
try:
return subprocess.run(command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
return subprocess.run(effective_command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
except subprocess.TimeoutExpired as exc:
return subprocess.CompletedProcess(command, 124, exc.stdout or "", exc.stderr or "git command timed out")
return subprocess.CompletedProcess(effective_command, 124, exc.stdout or "", exc.stderr or "git command timed out")
@@ -6,7 +6,7 @@ from pathlib import Path
import shlex
import subprocess
from .git_state import collect_repository_snapshot
from .git_state import collect_repository_snapshot, scoped_git_command
from .workspace import load_repository_specs, resolve_repo_path, resolve_workspace_root
@@ -138,10 +138,15 @@ def command_text(command: tuple[str, ...]) -> str:
def run(command: tuple[str, ...], *, cwd: Path) -> subprocess.CompletedProcess[str]:
effective_command = (
scoped_git_command(cwd, *command[1:])
if command and Path(command[0]).name == "git"
else command
)
try:
return subprocess.run(command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
return subprocess.run(effective_command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
except subprocess.TimeoutExpired as exc:
return subprocess.CompletedProcess(command, 124, exc.stdout or "", exc.stderr or "git push timed out")
return subprocess.CompletedProcess(effective_command, 124, exc.stdout or "", exc.stderr or "git push timed out")
def compact_output(value: str) -> str:
@@ -5,7 +5,7 @@ from __future__ import annotations
from pathlib import Path
import subprocess
from .git_state import collect_repository_snapshot
from .git_state import collect_repository_snapshot, scoped_git_command
from .repository_push import command_text, compact_output
from .workspace import load_repository_specs, resolve_repo_path, resolve_workspace_root
@@ -93,7 +93,12 @@ def sync_repositories(
def run(command: tuple[str, ...], *, cwd: Path) -> subprocess.CompletedProcess[str]:
effective_command = (
scoped_git_command(cwd, *command[1:])
if command and Path(command[0]).name == "git"
else command
)
try:
return subprocess.run(command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
return subprocess.run(effective_command, cwd=cwd, check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=120)
except subprocess.TimeoutExpired as exc:
return subprocess.CompletedProcess(command, 124, exc.stdout or "", exc.stderr or "git fetch timed out")
return subprocess.CompletedProcess(effective_command, 124, exc.stdout or "", exc.stderr or "git fetch timed out")
@@ -13,6 +13,7 @@ from .git_state import (
collect_repository_snapshot,
git_text,
sanitized_git_environment,
scoped_git_command,
)
from .model import RepositorySnapshot
from .repository_push import command_text, compact_output
@@ -479,12 +480,7 @@ def run(
effective_command = command
effective_environment = env
if command and Path(command[0]).name == "git":
effective_command = (
"/usr/bin/git",
"-c",
"core.hooksPath=/dev/null",
*command[1:],
)
effective_command = scoped_git_command(cwd, *command[1:])
effective_environment = sanitized_git_environment(env)
try:
return subprocess.run(
@@ -53,7 +53,7 @@ from .workspace import (
website_root,
)
GITEA_BASE = "git+ssh://git@git.add-ideas.de/add-ideas"
GITEA_BASE = "git+ssh://git@git.add-ideas.de/GovOPlaN"
MAX_BASE_JSON_BYTES = 16 * 1024 * 1024
_SIGNING_KEY_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,159}$")
@@ -25,6 +25,10 @@ from .version_alignment import (
selected_release_webui_bundle_issues,
selected_repository_version_issues,
)
from .version_metadata import (
VersionMetadataError,
version_metadata_mutations,
)
def build_selective_release_plan(
@@ -46,6 +50,7 @@ def build_selective_release_plan(
)
for repo in repositories
)
units = dependency_ordered_units(units)
units = apply_repository_version_gate(
units, workspace=Path(dashboard.workspace_root)
)
@@ -103,7 +108,35 @@ def apply_repository_version_gate(
workspace: Path,
) -> tuple[ReleasePlanUnit, ...]:
issues_by_repo: dict[str, list[ReleaseGateFinding]] = {}
version_update_supported_by_repo: dict[str, bool] = {}
deferred_core_lock_repos: set[str] = set()
for unit in units:
version_update_supported = unit.current_version == unit.target_version
if unit.current_version and unit.current_version != unit.target_version:
try:
version_update_supported = bool(
version_metadata_mutations(
workspace / unit.repo,
target_version=unit.target_version,
)
)
except VersionMetadataError as exc:
issues_by_repo.setdefault(unit.repo, []).append(
ReleaseGateFinding(
code="repository_version_mutation_unsupported",
severity="blocker",
message=str(exc),
remediation=(
"Align the unsupported version declarations manually, "
"commit the reviewed result, then rebuild this plan."
),
repo=unit.repo,
source="repository version metadata",
expected=unit.target_version,
actual=unit.current_version,
)
)
version_update_supported_by_repo[unit.repo] = version_update_supported
try:
issues = selected_repository_version_issues(
repo_versions={unit.repo: unit.target_version},
@@ -130,6 +163,18 @@ def apply_repository_version_gate(
)
continue
for issue in issues:
if (
version_update_supported
and issue.message
== "source version must match the requested release version"
):
continue
if (
unit.repo == "govoplan-core"
and issue.source.startswith("package-lock.release.json")
):
deferred_core_lock_repos.add(unit.repo)
continue
issues_by_repo.setdefault(issue.repo, []).append(
ReleaseGateFinding(
code="repository_version_alignment",
@@ -161,17 +206,68 @@ def apply_repository_version_gate(
source_preflight_ready=False,
)
if unit.repo in issues_by_repo
else unit
else replace(
unit,
warnings=planned_version_warnings(
unit,
defer_core_lock=unit.repo in deferred_core_lock_repos,
),
status=(
"attention"
if (
(
unit.current_version
and unit.current_version != unit.target_version
)
or unit.repo in deferred_core_lock_repos
)
and unit.status == "ready"
else unit.status
),
source_preflight_ready=(
unit.source_preflight_ready
or (
bool(unit.current_version)
and unit.current_version != unit.target_version
and version_update_supported_by_repo.get(unit.repo, False)
and not any(
finding.code == "worktree_dirty"
for finding in unit.gate_findings
)
)
),
)
for unit in units
)
def planned_version_warnings(
unit: ReleasePlanUnit,
*,
defer_core_lock: bool,
) -> tuple[str, ...]:
warnings = list(unit.warnings)
if unit.current_version and unit.current_version != unit.target_version:
warnings.append(
"version metadata will be updated by the bounded release executor"
)
if defer_core_lock:
warnings.append(
"Core release lock metadata will be regenerated by the bounded "
"bundle executor"
)
return tuple(warnings)
def apply_release_webui_bundle_gate(
units: tuple[ReleasePlanUnit, ...],
*,
workspace: Path,
) -> tuple[ReleasePlanUnit, ...]:
issues_by_repo: dict[str, list[ReleaseGateFinding]] = {}
selected = {unit.repo for unit in units}
core_selected = "govoplan-core" in selected
planned_bundle_repos: set[str] = set()
try:
issues = selected_release_webui_bundle_issues(
repo_versions={unit.repo: unit.target_version for unit in units},
@@ -200,6 +296,9 @@ def apply_release_webui_bundle_gate(
)
issues = ()
for issue in issues:
if core_selected and issue.repo in selected and issue.repo != "govoplan-core":
planned_bundle_repos.add(issue.repo)
continue
issues_by_repo.setdefault(issue.repo, []).append(
ReleaseGateFinding(
code="release_webui_composition_alignment",
@@ -230,7 +329,25 @@ def apply_release_webui_bundle_gate(
source_preflight_ready=False,
)
if unit.repo in issues_by_repo
else unit
else replace(
unit,
warnings=unit.warnings
+ (
(
"Core release WebUI references will be regenerated after "
"the selected module tags are created"
),
)
if unit.repo == "govoplan-core" and planned_bundle_repos
else unit.warnings,
status=(
"attention"
if unit.repo == "govoplan-core"
and planned_bundle_repos
and unit.status == "ready"
else unit.status
),
)
for unit in units
)
@@ -408,6 +525,7 @@ def build_unit(
status="blocked" if blockers else "attention" if warnings else "ready",
blockers=tuple(blockers),
warnings=tuple(warnings),
capabilities=repository_capabilities(repo, contracts=contracts),
provides_interfaces=provides,
requires_interfaces=requires,
gate_findings=tuple(findings),
@@ -416,11 +534,83 @@ def build_unit(
and not repo.dirty
and repo.has_head
and bool(repo.branch)
and current_version == resolved_target
and current_version is not None
),
)
def repository_capabilities(
repo: RepositorySnapshot,
*,
contracts: ModuleContractSnapshot | None,
) -> tuple[str, ...]:
"""Describe the release operations applicable to one repository shape."""
root = Path(repo.absolute_path)
capabilities = {"git-source"}
if (root / "pyproject.toml").is_file():
capabilities.add("python-package")
if (root / "package.json").is_file() or (root / "webui/package.json").is_file():
capabilities.add("webui-package")
if contracts is not None:
capabilities.add("module-manifest")
if (
(root / "alembic/versions").is_dir()
or (root / "src").is_dir()
and any((root / "src").glob("**/migrations/versions"))
):
capabilities.add("database-migrations")
if (root / "docs").is_dir() or repo.spec.subtype in {"docs", "documentation"}:
capabilities.add("documentation")
if repo.spec.name == "govoplan-core":
capabilities.add("core-release-bundle")
if repo.spec.name == "govoplan":
capabilities.add("release-control-plane")
return tuple(sorted(capabilities))
def dependency_ordered_units(
units: tuple[ReleasePlanUnit, ...],
) -> tuple[ReleasePlanUnit, ...]:
"""Order module providers before consumers while keeping Core last."""
by_repo = {unit.repo: unit for unit in units}
providers: dict[str, set[str]] = {}
for unit in units:
if unit.repo == "govoplan-core":
continue
for provided in unit.provides_interfaces:
providers.setdefault(provided.name, set()).add(unit.repo)
dependencies: dict[str, set[str]] = {unit.repo: set() for unit in units}
for unit in units:
for requirement in unit.requires_interfaces:
dependencies[unit.repo].update(
provider
for provider in providers.get(requirement.name, set())
if provider != unit.repo
)
if "govoplan-core" in dependencies:
dependencies["govoplan-core"].update(
repo for repo in by_repo if repo != "govoplan-core"
)
ordered: list[ReleasePlanUnit] = []
remaining = set(by_repo)
while remaining:
ready = sorted(
repo
for repo in remaining
if not (dependencies[repo] & remaining)
)
if not ready:
ready = [sorted(remaining)[0]]
for repo in ready:
ordered.append(by_repo[repo])
remaining.remove(repo)
return tuple(ordered)
def gate_finding(
*,
repo: str,
@@ -501,24 +691,38 @@ def dry_run_steps(
) -> tuple[ReleasePlanStep, ...]:
steps: list[ReleasePlanStep] = []
snapshots = {repo.spec.name: repo for repo in dashboard.repositories}
selected_names = {unit.repo for unit in units}
if "govoplan-core" in selected_names and len(selected_names) > 1:
steps.append(
ReleasePlanStep(
id="release:cross-repository-ordering",
title="Resolve module/Core release ordering",
detail=(
"A combined Core/module release needs a dependency-aware DAG: "
"local module tags, Core release-lock regeneration and commit, "
"Core tag, full alignment, then atomic pushes. The current linear "
"executor must not publish a module before that lock is committed."
),
command=None,
cwd=dashboard.meta_root,
mutating=True,
status="needs-executor",
core_unit = next((unit for unit in units if unit.repo == "govoplan-core"), None)
non_core_units = tuple(unit for unit in units if unit.repo != "govoplan-core")
bundle_repos: tuple[str, ...] = ()
core_lock_regeneration = False
if core_unit is not None:
try:
bundle_repos = tuple(
sorted(
{
issue.repo
for issue in selected_release_webui_bundle_issues(
repo_versions={
unit.repo: unit.target_version for unit in units
},
workspace=Path(dashboard.workspace_root),
)
if issue.repo != "govoplan-core"
}
)
)
)
core_lock_regeneration = bool(bundle_repos) or any(
issue.source.startswith("package-lock.release.json")
for issue in selected_repository_version_issues(
repo_versions={
"govoplan-core": core_unit.target_version,
},
workspace=Path(dashboard.workspace_root),
)
)
except Exception: # noqa: BLE001 - a structured gate already reports it.
bundle_repos = ()
for unit in units:
steps.append(
ReleasePlanStep(
@@ -531,32 +735,74 @@ def dry_run_steps(
repo=unit.repo,
)
)
for unit in units:
if unit.current_version != unit.target_version:
try:
paths = tuple(
mutation.path
for mutation in version_metadata_mutations(
Path(dashboard.workspace_root) / unit.repo,
target_version=unit.target_version,
)
)
except VersionMetadataError:
paths = ()
snapshot = snapshots.get(unit.repo)
version_supported = bool(
paths and snapshot is not None and not snapshot.dirty
)
steps.append(
ReleasePlanStep(
id=f"{unit.repo}:version",
title=f"Update {unit.repo} version metadata",
detail=f"Prepare version files and manifest metadata for {unit.target_version}.",
command=None,
detail=(
f"Update only the recognized version declarations for "
f"{unit.target_version}: {', '.join(paths) or 'unsupported metadata'}."
),
command=(
"bounded version metadata update "
+ " ".join(shlex.quote(path) for path in paths)
)
if paths
else None,
cwd=f"{dashboard.workspace_root}/{unit.repo}",
mutating=True,
repo=unit.repo,
status="needs-executor",
status="planned" if version_supported else "needs-executor",
)
)
for unit in non_core_units:
snapshot = snapshots.get(unit.repo)
if unit.current_version != unit.target_version or (
needs_commit = unit.current_version != unit.target_version or (
snapshot is not None and snapshot.dirty
):
)
if needs_commit:
commit_supported = bool(
unit.current_version != unit.target_version
and snapshot is not None
and not snapshot.dirty
)
steps.append(
ReleasePlanStep(
id=f"{unit.repo}:commit",
title=f"Commit {unit.repo} release changes",
detail="Commit reviewed changes for this release unit only.",
command=f"git add -A && git commit -m {shlex.quote(f'Release {unit.repo} {unit.target_tag}')}",
detail=(
"Commit only the receipt-bound files written by the version "
"executor."
if commit_supported
else "Pre-existing worktree changes require review outside the release executor."
),
command=(
f"git commit -m {shlex.quote(f'Release {unit.repo} {unit.target_tag}')}"
if commit_supported
else None
),
cwd=f"{dashboard.workspace_root}/{unit.repo}",
mutating=True,
repo=unit.repo,
status="planned" if commit_supported else "needs-executor",
)
)
steps.append(
@@ -570,6 +816,97 @@ def dry_run_steps(
repo=unit.repo,
)
)
if core_unit is not None:
if core_lock_regeneration:
steps.append(
ReleasePlanStep(
id="govoplan-core:bundle",
title="Regenerate the Core release WebUI bundle",
detail=(
"After local module tags exist, update and lock the exact "
"selected WebUI references"
+ (
" for: " + ", ".join(bundle_repos)
if bundle_repos
else ""
)
+ ", and reconcile package/lock metadata."
),
command=(
"tools/release/generate-release-lock.sh "
+ " ".join(
f"--local-git-repo {shlex.quote(str(Path(dashboard.workspace_root) / repo))}"
for repo in bundle_repos
)
),
cwd=dashboard.meta_root,
mutating=True,
repo="govoplan-core",
)
)
core_snapshot = snapshots.get(core_unit.repo)
core_needs_commit = (
core_unit.current_version != core_unit.target_version
or core_lock_regeneration
or (core_snapshot is not None and core_snapshot.dirty)
)
if core_needs_commit:
core_commit_supported = bool(
core_snapshot is not None
and not core_snapshot.dirty
and (
core_unit.current_version != core_unit.target_version
or core_lock_regeneration
)
)
steps.append(
ReleasePlanStep(
id="govoplan-core:commit",
title="Commit govoplan-core release changes",
detail=(
"Commit only the receipt-bound Core version and release-bundle files."
if core_commit_supported
else "Pre-existing Core worktree changes require review outside the release executor."
),
command=(
f"git commit -m {shlex.quote(f'Release govoplan-core {core_unit.target_tag}')}"
if core_commit_supported
else None
),
cwd=f"{dashboard.workspace_root}/govoplan-core",
mutating=True,
repo="govoplan-core",
status="planned" if core_commit_supported else "needs-executor",
)
)
steps.append(
ReleasePlanStep(
id="govoplan-core:tag",
title=f"Create govoplan-core tag {core_unit.target_tag}",
detail="Create the Core tag only after selected module tags and the release lock are verified.",
command=f"git tag -a {shlex.quote(core_unit.target_tag)} -m {shlex.quote(f'Release govoplan-core {core_unit.target_tag}')}",
cwd=f"{dashboard.workspace_root}/govoplan-core",
mutating=True,
repo="govoplan-core",
)
)
if units:
steps.append(
ReleasePlanStep(
id="release:alignment",
title="Verify frozen source and release composition",
detail=(
"Re-run version, contract, tag, and Core WebUI composition gates "
"after local mutations and before any remote publication."
),
command="receipt-bound release alignment",
cwd=dashboard.meta_root,
)
)
for unit in units:
steps.append(
ReleasePlanStep(
id=f"{unit.repo}:push",
@@ -640,6 +977,20 @@ def dry_run_steps(
status="planned",
)
)
steps.append(
ReleasePlanStep(
id="release:install-verify",
title="Verify candidate package installation",
detail=(
"Install every selected candidate wheel into a private "
"temporary target without network access or dependencies, "
"then verify installed package metadata against the frozen plan."
),
command="python -m pip install --no-index --no-deps <candidate wheels>",
cwd=dashboard.meta_root,
status="planned",
)
)
return tuple(steps)
@@ -16,6 +16,7 @@ from .git_state import (
collect_repository_snapshot,
git_text,
sanitized_git_environment,
scoped_git_command,
)
from .repository_tag import RemoteTagResult, ref_commit, remote_tag_commit
from .version_alignment import repository_version_issues
@@ -417,9 +418,10 @@ def _git_file(path: Path, tag: str, name: str) -> str | None:
def _git(path: Path, *args: str) -> subprocess.CompletedProcess[str]:
command = scoped_git_command(path, *args)
try:
return subprocess.run(
("/usr/bin/git", "-c", "core.hooksPath=/dev/null", *args),
command,
cwd=path,
check=False,
text=True,
@@ -429,7 +431,7 @@ def _git(path: Path, *args: str) -> subprocess.CompletedProcess[str]:
env=sanitized_git_environment(),
)
except subprocess.TimeoutExpired as exc:
return subprocess.CompletedProcess(("/usr/bin/git", *args), 124, exc.stdout or "", exc.stderr or "Git command timed out")
return subprocess.CompletedProcess(command, 124, exc.stdout or "", exc.stderr or "Git command timed out")
def _deduplicate(issues: list[SourceTagProvenanceIssue]) -> list[SourceTagProvenanceIssue]:
@@ -681,10 +681,22 @@ def _git_lock_resolution_issues(
def _git_tag_commit(repo_path: Path, tag: str) -> str | None:
repo_path = repo_path.resolve()
if not (repo_path / ".git").exists():
return None
result = subprocess.run(
["/usr/bin/git", "-C", str(repo_path), "rev-list", "-n", "1", tag],
[
"/usr/bin/git",
"-c",
"core.hooksPath=/dev/null",
"-c",
f"safe.directory={repo_path}",
"-C",
str(repo_path),
"rev-parse",
"--verify",
f"refs/tags/{tag}^{{commit}}",
],
check=False,
text=True,
stdout=subprocess.PIPE,
@@ -697,10 +709,22 @@ def _git_tag_commit(repo_path: Path, tag: str) -> str | None:
def _git_tag_file(repo_path: Path, tag: str, relative_path: str) -> str | None:
repo_path = repo_path.resolve()
if not (repo_path / ".git").exists():
return None
result = subprocess.run(
["/usr/bin/git", "-C", str(repo_path), "show", f"{tag}:{relative_path}"],
[
"/usr/bin/git",
"-c",
"core.hooksPath=/dev/null",
"-c",
f"safe.directory={repo_path}",
"-C",
str(repo_path),
"cat-file",
"blob",
f"refs/tags/{tag}^{{commit}}:{relative_path}",
],
check=False,
text=True,
stdout=subprocess.PIPE,
@@ -0,0 +1,387 @@
"""Deterministic, bounded updates for repository-owned version metadata."""
from __future__ import annotations
import ast
from dataclasses import dataclass
import json
import os
from pathlib import Path
import re
import tempfile
import tomllib
MAX_VERSION_FILES = 64
MAX_VERSION_FILE_BYTES = 2 * 1024 * 1024
class VersionMetadataError(RuntimeError):
"""Raised when release version metadata cannot be changed safely."""
@dataclass(frozen=True, slots=True)
class VersionFileMutation:
path: str
before: bytes
after: bytes
def version_metadata_mutations(
repo_path: Path,
*,
target_version: str,
) -> tuple[VersionFileMutation, ...]:
"""Render all recognized repository version files without writing them."""
version = target_version.removeprefix("v")
candidates: list[tuple[Path, str]] = []
if (repo_path / "pyproject.toml").is_file():
candidates.append((repo_path / "pyproject.toml", "pyproject"))
for package_name, lock_name in (
("package.json", "package-lock.json"),
("webui/package.json", "webui/package-lock.json"),
("webui/package.release.json", "webui/package-lock.release.json"),
):
package = repo_path / package_name
if not package.is_file():
continue
candidates.append((package, "package"))
lock = repo_path / lock_name
if lock.is_file():
candidates.append((lock, "lock"))
src = repo_path / "src"
if src.is_dir():
candidates.extend(
(path, "manifest")
for path in sorted(src.glob("**/backend/manifest.py"))
if path.is_file()
)
candidates.extend(
(path, "package_init")
for path in sorted(src.glob("*/__init__.py"))
if path.is_file()
)
if len(candidates) > MAX_VERSION_FILES:
raise VersionMetadataError("Repository version metadata exceeds its file bound.")
mutations: list[VersionFileMutation] = []
declarations = 0
for path, kind in candidates:
before = _read_bounded(path)
if kind == "pyproject":
after, found = _render_pyproject(before, version=version)
elif kind == "package":
after, found = _render_package_json(before, version=version)
elif kind == "lock":
after, found = _render_package_lock(before, version=version)
elif kind == "manifest":
after, found = _render_python_version(
before,
path=path,
target_name="ModuleManifest",
keyword_name="version",
version=version,
)
else:
after, found = _render_python_assignment(
before,
path=path,
assignment_name="__version__",
version=version,
)
if not found:
continue
declarations += 1
if before != after:
mutations.append(
VersionFileMutation(
path=path.relative_to(repo_path).as_posix(),
before=before,
after=after,
)
)
if declarations == 0:
raise VersionMetadataError(
"Repository has no supported version declaration to update."
)
return tuple(mutations)
def apply_version_metadata_mutations(
repo_path: Path,
*,
target_version: str,
) -> tuple[str, ...]:
"""Apply one deterministic version update, rolling back on write failure."""
mutations = version_metadata_mutations(
repo_path,
target_version=target_version,
)
written: list[VersionFileMutation] = []
try:
for mutation in mutations:
destination = repo_path / mutation.path
if _read_bounded(destination) != mutation.before:
raise VersionMetadataError(
f"Version metadata changed before update: {mutation.path}"
)
_atomic_write(destination, mutation.after)
written.append(mutation)
except Exception:
for mutation in reversed(written):
_atomic_write(repo_path / mutation.path, mutation.before)
raise
return tuple(mutation.path for mutation in mutations)
def version_metadata_paths(
repo_path: Path,
*,
target_version: str,
) -> tuple[str, ...]:
"""Return the exact paths a deterministic target-version update owns."""
return tuple(
mutation.path
for mutation in version_metadata_mutations(
repo_path,
target_version=target_version,
)
)
def _read_bounded(path: Path) -> bytes:
try:
metadata = path.lstat()
except OSError as exc:
raise VersionMetadataError(
f"Version metadata is unavailable: {path.name}"
) from exc
if not path.is_file() or path.is_symlink():
raise VersionMetadataError(
f"Version metadata must be a regular file: {path.name}"
)
if metadata.st_size > MAX_VERSION_FILE_BYTES:
raise VersionMetadataError(
f"Version metadata exceeds its size bound: {path.name}"
)
try:
return path.read_bytes()
except OSError as exc:
raise VersionMetadataError(
f"Version metadata could not be read: {path.name}"
) from exc
def _decode(payload: bytes, *, path: Path | None = None) -> str:
try:
return payload.decode("utf-8")
except UnicodeDecodeError as exc:
label = path.name if path is not None else "file"
raise VersionMetadataError(
f"Version metadata is not UTF-8: {label}"
) from exc
def _render_pyproject(payload: bytes, *, version: str) -> tuple[bytes, bool]:
text = _decode(payload)
try:
parsed = tomllib.loads(text)
except tomllib.TOMLDecodeError as exc:
raise VersionMetadataError("pyproject.toml is malformed.") from exc
project = parsed.get("project")
if not isinstance(project, dict) or not isinstance(project.get("version"), str):
return payload, False
section = re.search(
r"(?ms)^\[project\][^\n]*\n(?P<body>.*?)(?=^\[[^\n]+\]|\Z)",
text,
)
if section is None:
raise VersionMetadataError("pyproject.toml has no editable [project] block.")
body = section.group("body")
match = re.search(
r'(?m)^(?P<prefix>\s*version\s*=\s*)(?P<quote>["\'])(?P<value>[^"\']+)(?P=quote)(?P<suffix>\s*(?:#.*)?)$',
body,
)
if match is None:
raise VersionMetadataError(
"pyproject.toml project.version is not a supported literal."
)
start = section.start("body") + match.start("value")
end = section.start("body") + match.end("value")
rendered = f"{text[:start]}{version}{text[end:]}"
tomllib.loads(rendered)
return rendered.encode("utf-8"), True
def _render_package_json(payload: bytes, *, version: str) -> tuple[bytes, bool]:
data = _json_object(payload, label="package metadata")
if not isinstance(data.get("version"), str):
return payload, False
data["version"] = version
return _json_bytes(data), True
def _render_package_lock(payload: bytes, *, version: str) -> tuple[bytes, bool]:
data = _json_object(payload, label="package lock")
found = False
if isinstance(data.get("version"), str):
data["version"] = version
found = True
packages = data.get("packages")
root = packages.get("") if isinstance(packages, dict) else None
if isinstance(root, dict) and isinstance(root.get("version"), str):
root["version"] = version
found = True
return (_json_bytes(data), True) if found else (payload, False)
def _json_object(payload: bytes, *, label: str) -> dict[str, object]:
try:
value = json.loads(_decode(payload))
except json.JSONDecodeError as exc:
raise VersionMetadataError(f"{label.capitalize()} is malformed.") from exc
if not isinstance(value, dict):
raise VersionMetadataError(f"{label.capitalize()} must be an object.")
return value
def _json_bytes(value: dict[str, object]) -> bytes:
return (
json.dumps(value, indent=2, ensure_ascii=True, separators=(",", ": "))
+ "\n"
).encode("utf-8")
def _render_python_version(
payload: bytes,
*,
path: Path,
target_name: str,
keyword_name: str,
version: str,
) -> tuple[bytes, bool]:
text = _decode(payload, path=path)
try:
tree = ast.parse(text, filename=str(path))
except SyntaxError as exc:
raise VersionMetadataError(f"Python metadata is malformed: {path.name}") from exc
values: list[ast.Constant] = []
for node in ast.walk(tree):
if not isinstance(node, ast.Call) or _call_name(node.func) != target_name:
continue
for keyword in node.keywords:
if (
keyword.arg == keyword_name
and isinstance(keyword.value, ast.Constant)
and isinstance(keyword.value.value, str)
):
values.append(keyword.value)
if not values:
return payload, False
if len(values) != 1:
raise VersionMetadataError(
f"Python metadata has multiple {target_name}.{keyword_name} values: {path.name}"
)
return _replace_python_literal(text, values[0], version=version, path=path), True
def _render_python_assignment(
payload: bytes,
*,
path: Path,
assignment_name: str,
version: str,
) -> tuple[bytes, bool]:
text = _decode(payload, path=path)
try:
tree = ast.parse(text, filename=str(path))
except SyntaxError as exc:
raise VersionMetadataError(f"Python metadata is malformed: {path.name}") from exc
values: list[ast.Constant] = []
for node in tree.body:
if not isinstance(node, (ast.Assign, ast.AnnAssign)):
continue
targets = node.targets if isinstance(node, ast.Assign) else [node.target]
value = node.value
if (
any(
isinstance(target, ast.Name) and target.id == assignment_name
for target in targets
)
and isinstance(value, ast.Constant)
and isinstance(value.value, str)
):
values.append(value)
if not values:
return payload, False
if len(values) != 1:
raise VersionMetadataError(
f"Python metadata has multiple {assignment_name} assignments: {path.name}"
)
return _replace_python_literal(text, values[0], version=version, path=path), True
def _replace_python_literal(
text: str,
node: ast.Constant,
*,
version: str,
path: Path,
) -> bytes:
if (
node.lineno != node.end_lineno
or node.end_col_offset is None
or node.col_offset < 0
):
raise VersionMetadataError(
f"Python version declaration must be a single-line literal: {path.name}"
)
lines = text.splitlines(keepends=True)
line = lines[node.lineno - 1]
lines[node.lineno - 1] = (
line[: node.col_offset]
+ json.dumps(version)
+ line[node.end_col_offset :]
)
rendered = "".join(lines)
ast.parse(rendered, filename=str(path))
return rendered.encode("utf-8")
def _call_name(node: ast.expr) -> str | None:
if isinstance(node, ast.Name):
return node.id
if isinstance(node, ast.Attribute):
return node.attr
return None
def _atomic_write(path: Path, payload: bytes) -> None:
metadata = path.stat()
descriptor, temporary_name = tempfile.mkstemp(
prefix=f".{path.name}.",
dir=path.parent,
)
temporary = Path(temporary_name)
try:
os.fchmod(descriptor, metadata.st_mode & 0o777)
with os.fdopen(descriptor, "wb") as handle:
handle.write(payload)
handle.flush()
os.fsync(handle.fileno())
temporary.replace(path)
directory = os.open(path.parent, os.O_RDONLY | getattr(os, "O_DIRECTORY", 0))
try:
os.fsync(directory)
finally:
os.close(directory)
except Exception:
try:
os.close(descriptor)
except OSError:
pass
temporary.unlink(missing_ok=True)
raise
+9 -3
View File
@@ -85,9 +85,13 @@ PACKAGE_MODULE_REPOS=(
"$PARENT/govoplan-certificates"
"$PARENT/govoplan-committee"
"$PARENT/govoplan-consultation"
"$PARENT/govoplan-connectors"
"$PARENT/govoplan-contracts"
"$PARENT/govoplan-dashboard"
"$PARENT/govoplan-dataflow"
"$PARENT/govoplan-datasources"
"$PARENT/govoplan-docs"
"$PARENT/govoplan-encryption"
"$PARENT/govoplan-facilities"
"$PARENT/govoplan-files"
"$PARENT/govoplan-forms-runtime"
@@ -96,7 +100,7 @@ PACKAGE_MODULE_REPOS=(
"$PARENT/govoplan-identity"
"$PARENT/govoplan-idm"
"$PARENT/govoplan-inspections"
"$PARENT/govoplan-issue-reporting"
"$PARENT/govoplan-tickets"
"$PARENT/govoplan-learning"
"$PARENT/govoplan-mail"
"$PARENT/govoplan-ops"
@@ -111,12 +115,14 @@ PACKAGE_MODULE_REPOS=(
"$PARENT/govoplan-tenancy"
"$PARENT/govoplan-transparency"
"$PARENT/govoplan-evaluation"
"$PARENT/govoplan-views"
"$PARENT/govoplan-workflow"
)
TAG_ONLY_MODULE_REPOS=(
"$PARENT/govoplan-addresses"
"$PARENT/govoplan-appointments"
"$PARENT/govoplan-cases"
"$PARENT/govoplan-connectors"
"$PARENT/govoplan-projects"
"$PARENT/govoplan-dms"
"$PARENT/govoplan-dist-lists"
"$PARENT/govoplan-erp"
@@ -132,10 +138,10 @@ TAG_ONLY_MODULE_REPOS=(
"$PARENT/govoplan-rest"
"$PARENT/govoplan-scheduling"
"$PARENT/govoplan-search"
"$PARENT/govoplan-wiki"
"$PARENT/govoplan-soap"
"$PARENT/govoplan-tasks"
"$PARENT/govoplan-templates"
"$PARENT/govoplan-workflow"
"$PARENT/govoplan-xoev"
"$PARENT/govoplan-xrechnung"
"$PARENT/govoplan-xta-osci"
+1 -1
View File
@@ -76,7 +76,7 @@ def main() -> int:
)
selective.add_argument("--public-base-url", default="https://govoplan.add-ideas.de")
selective.add_argument(
"--repository-base", default="git+ssh://git@git.add-ideas.de/add-ideas"
"--repository-base", default="git+ssh://git@git.add-ideas.de/GovOPlaN"
)
selective.add_argument(
"--source-remote",
+39 -2
View File
@@ -170,18 +170,30 @@ def owner_for_versions_dir(versions_dir: Path) -> str:
def parse_migration_file(owner: str, path: Path) -> Migration | None:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
values: dict[str, Any] = {}
wrapped: Migration | None = None
release_peer = path.parent.parent / "versions" / path.name
if path.parent.name == "dev_versions" and release_peer.is_file():
wrapped = parse_migration_file(owner, release_peer)
for statement in tree.body:
if isinstance(statement, ast.Assign):
for target in statement.targets:
if isinstance(target, ast.Name) and target.id in {"revision", "down_revision", "depends_on", "branch_labels"}:
values[target.id] = ast.literal_eval(statement.value)
values[target.id] = _migration_assignment_value(
target.id,
statement.value,
wrapped=wrapped,
)
elif (
isinstance(statement, ast.AnnAssign)
and isinstance(statement.target, ast.Name)
and statement.target.id in {"revision", "down_revision", "depends_on", "branch_labels"}
and statement.value is not None
):
values[statement.target.id] = ast.literal_eval(statement.value)
values[statement.target.id] = _migration_assignment_value(
statement.target.id,
statement.value,
wrapped=wrapped,
)
revision = values.get("revision")
if not isinstance(revision, str):
return None
@@ -195,6 +207,31 @@ def parse_migration_file(owner: str, path: Path) -> Migration | None:
)
def _migration_assignment_value(
name: str,
value: ast.expr,
*,
wrapped: Migration | None,
) -> Any:
try:
return ast.literal_eval(value)
except (ValueError, TypeError):
if (
wrapped is None
or not isinstance(value, ast.Attribute)
or not isinstance(value.value, ast.Name)
or value.value.id != "_migration"
or value.attr != name
):
return None
return {
"revision": wrapped.revision,
"down_revision": wrapped.down_revisions,
"depends_on": wrapped.depends_on,
"branch_labels": wrapped.branch_labels,
}[name]
def _normalize_revision_tuple(value: Any) -> tuple[str, ...]:
if value is None:
return ()
+52 -16
View File
@@ -45,7 +45,9 @@ from govoplan_release.release_execution import (
verify_catalog_publication_precondition,
verify_repository_preflight_binding,
verify_repository_step_precondition,
verify_release_alignment,
verify_release_runtime_binding,
verify_candidate_installation,
)
from govoplan_release.release_run import (
ReleaseRunConflict,
@@ -68,7 +70,7 @@ class CatalogCandidateRequest(BaseModel):
expires_days: int = 90
sequence: int | None = None
public_base_url: str = "https://govoplan.add-ideas.de"
repository_base: str = "git+ssh://git@git.add-ideas.de/add-ideas"
repository_base: str = "git+ssh://git@git.add-ideas.de/GovOPlaN"
source_remote: str = "origin"
check_public: bool = True
@@ -532,7 +534,13 @@ def create_app(
) == "interrupted":
plan_step = release_run_plan_step(run, step_id)
spec = executor_spec(plan_step)
if spec is not None and spec.kind in {"tag", "push"}:
if spec is not None and spec.kind in {
"version",
"commit",
"core_bundle",
"tag",
"push",
}:
repo = str(plan_step["repo"])
receipt = reconciled_repository_receipt(
spec=spec,
@@ -542,6 +550,7 @@ def create_app(
expected_receipt=preceding_repository_receipt(
run, step_id=step_id, repo=repo
),
repo_versions=run["immutable"]["input"]["repo_versions"],
)
return release_run_view(
app.state.release_runs.reconcile_step(
@@ -637,7 +646,15 @@ def create_app(
plan_step=plan_step,
workspace_root=app.state.workspace_root,
)
if spec.kind in {"preflight", "tag", "push"}:
repository_kinds = {
"preflight",
"version",
"commit",
"core_bundle",
"tag",
"push",
}
if spec.kind in repository_kinds:
repo = str(plan_step["repo"])
version = run["immutable"]["input"]["repo_versions"][repo]
expected_repository_receipt = preceding_repository_receipt(
@@ -659,7 +676,21 @@ def create_app(
)
receipt = None
if spec.kind in {"preflight", "tag", "push"}:
source_receipts = {
repo: source_receipt
for repo in run["immutable"]["input"]["repo_versions"]
if isinstance(
(
source_receipt := preceding_repository_receipt(
run,
step_id=step_id,
repo=repo,
)
),
dict,
)
}
if spec.kind in repository_kinds:
result, receipt = execute_repository_step(
spec=spec,
plan_step=plan_step,
@@ -667,6 +698,13 @@ def create_app(
workspace_root=app.state.workspace_root,
remote=DURABLE_REMOTE,
expected_receipt=expected_repository_receipt,
source_receipts=source_receipts,
)
elif spec.kind == "alignment":
result = verify_release_alignment(
repo_versions=run["immutable"]["input"]["repo_versions"],
workspace_root=app.state.workspace_root,
source_receipts=source_receipts,
)
elif spec.kind == "catalog_generate":
attempt_fingerprint = (
@@ -690,18 +728,7 @@ def create_app(
signing_keys=signing_keys,
remote=DURABLE_REMOTE,
check_public=run["immutable"]["input"]["public_catalog"],
source_receipts={
repo: receipt
for repo in run["immutable"]["input"]["repo_versions"]
if isinstance(
(
receipt := preceding_repository_receipt(
run, step_id=step_id, repo=repo
)
),
dict,
)
},
source_receipts=source_receipts,
base_catalog=base_catalog,
base_keyring=base_keyring,
)
@@ -738,6 +765,15 @@ def create_app(
"Published candidate changed while its effect was in flight; "
"reconcile the immutable website commit and remote tag."
) from exc
elif spec.kind == "install_verify":
candidate_path = verified_run_candidate(
run,
candidate_root=app.state.release_candidate_root,
)
result = verify_candidate_installation(
candidate_path=candidate_path,
repo_versions=run["immutable"]["input"]["repo_versions"],
)
else:
raise ReleaseRunConflict("Release executor mapping is incomplete.")
+483 -98
View File
@@ -39,6 +39,7 @@
padding: 18px 28px;
position: sticky;
top: 0;
z-index: 10;
}
header h1 {
@@ -59,6 +60,7 @@
.toolbar,
.summary,
.release-guide,
.layout {
width: min(1600px, 100%);
margin: 0 auto 18px;
@@ -281,7 +283,7 @@
table {
width: 100%;
border-collapse: collapse;
min-width: 1220px;
min-width: 960px;
}
th,
@@ -400,8 +402,146 @@
.workflow-stages {
display: grid;
gap: 10px;
grid-template-columns: repeat(7, minmax(112px, 1fr));
gap: 0;
padding: 18px 14px 14px;
overflow-x: auto;
}
.workflow-stage {
position: relative;
min-width: 112px;
min-height: 86px;
border: 0;
border-radius: 0;
background: transparent;
color: var(--muted);
padding: 0 8px;
cursor: pointer;
}
.workflow-stage::after {
content: "";
position: absolute;
z-index: 0;
top: 17px;
left: calc(50% + 21px);
width: calc(100% - 42px);
height: 2px;
background: var(--line);
}
.workflow-stage:last-child::after {
display: none;
}
.workflow-stage.is-complete::after {
background: var(--ok);
}
.workflow-stage:focus-visible {
outline: 2px solid var(--accent);
outline-offset: 2px;
}
.workflow-stage-marker {
position: relative;
z-index: 1;
display: grid;
place-items: center;
width: 36px;
height: 36px;
margin: 0 auto 8px;
border: 2px solid var(--line);
border-radius: 50%;
background: var(--panel);
color: var(--muted);
font-size: 12px;
font-weight: 800;
}
.workflow-stage.is-complete .workflow-stage-marker {
border-color: var(--ok);
background: var(--ok);
color: #fff;
}
.workflow-stage.is-current .workflow-stage-marker {
border-color: var(--accent);
color: var(--accent);
box-shadow: 0 0 0 3px rgba(18, 97, 166, 0.12);
}
.workflow-stage.is-blocked .workflow-stage-marker {
border-color: var(--danger);
color: var(--danger);
}
.workflow-stage.is-unavailable {
cursor: not-allowed;
opacity: 0.72;
}
.workflow-stage-label,
.workflow-stage-status {
display: block;
text-align: center;
}
.workflow-stage-label {
color: var(--text);
font-size: 12px;
font-weight: 800;
}
.workflow-stage-status {
margin-top: 3px;
color: var(--muted);
font-size: 11px;
font-weight: 600;
}
.workflow-guidance {
display: grid;
grid-template-columns: minmax(0, 1fr) auto;
gap: 18px;
align-items: center;
padding: 14px;
border-top: 1px solid var(--line);
background: var(--panel-alt);
}
.workflow-guidance h3 {
margin: 0 0 5px;
font-size: 14px;
}
.workflow-guidance p {
margin: 0;
color: var(--muted);
font-size: 12px;
line-height: 1.45;
}
.workflow-guidance .action-meta {
margin-top: 5px;
}
.workflow-guidance button {
min-width: 160px;
}
.release-run-toolbar {
display: grid;
grid-template-columns: minmax(240px, 1fr) auto;
gap: 12px;
align-items: end;
padding: 14px;
border-bottom: 1px solid var(--line);
}
.release-run-toolbar .button-row {
margin: 0;
}
.action {
@@ -519,6 +659,15 @@
.insight-grid {
grid-template-columns: 1fr;
}
.workflow-stages {
grid-template-columns: repeat(7, minmax(128px, 1fr));
}
.workflow-guidance,
.release-run-toolbar {
grid-template-columns: 1fr;
}
}
</style>
</head>
@@ -546,9 +695,24 @@
<div class="summary" id="summary"></div>
<section class="release-guide" id="releaseGuide">
<div class="section-head">
<h2>Release Workflow</h2>
<small id="workflowStageStatus">Loading...</small>
</div>
<div class="workflow-stages" id="workflowStages"></div>
<div class="workflow-guidance" id="workflowGuidance">
<div>
<h3>Inspect the release workspace</h3>
<p>The console is collecting repository, contract, migration, and publication state.</p>
</div>
<button id="workflowPrimaryAction" disabled>Loading...</button>
</div>
</section>
<div class="layout">
<div class="main-stack">
<section>
<section id="releaseControlSection">
<div class="section-head">
<h2>Release Control</h2>
<small id="releaseControlStatus"></small>
@@ -557,7 +721,7 @@
<div class="issue-list" id="releaseIssues"></div>
</section>
<section>
<section id="publishedModulesSection">
<div class="section-head">
<h2>Published Modules</h2>
<small id="publishedModuleCount"></small>
@@ -580,7 +744,7 @@
</div>
</section>
<section>
<section id="compatibilitySection">
<div class="section-head">
<h2>Compatibility</h2>
<small id="compatibilityStatus"></small>
@@ -601,7 +765,7 @@
</div>
</section>
<section>
<section id="releaseTargetsSection">
<div class="section-head">
<h2>Repositories</h2>
<small id="unitCount"></small>
@@ -633,46 +797,33 @@
</table>
</div>
</section>
</div>
<div class="side">
<section>
<section id="releaseRunSection">
<div class="section-head">
<h2>Durable Run State</h2>
<h2>Durable Release Run</h2>
<small id="runStatus">no run selected</small>
</div>
<div class="details">
<p class="hint"><strong>Durable execution.</strong> A run freezes the selected inputs and plan, claims each supported executor before its effect, and keeps explicit confirmation and reconciliation boundaries. Unsupported dependency-ordering or version-metadata steps stay visible and disabled.</p>
<div class="form-grid">
<div>
<label for="releaseRun">Saved run</label>
<select id="releaseRun">
<option value="">No saved run selected</option>
</select>
</div>
<div class="button-row">
<button class="secondary" id="loadOlderReleaseRuns" disabled>Load older</button>
</div>
<div class="release-run-toolbar">
<div>
<label for="releaseRun">Saved run</label>
<select id="releaseRun">
<option value="">No saved run selected</option>
</select>
</div>
<div class="button-row">
<button id="createReleaseRun">Create Run from Selection</button>
<button class="secondary" id="resumeReleaseRun" disabled>Resume Run</button>
</div>
<div class="actions" id="runOutput">
<div class="muted">Create a run after selecting exact repository versions, or choose a saved run.</div>
<button class="secondary" id="loadOlderReleaseRuns" disabled>Load older</button>
</div>
</div>
</section>
<section>
<div class="section-head">
<h2>Release Workflow</h2>
<small id="workflowStageStatus"></small>
<div class="actions" id="runOutput">
<div class="muted"><strong>Durable execution.</strong> Durable Run State preserves the frozen plan and mutation receipts across refreshes. Select exact repository versions, validate the plan, then create or resume a run.</div>
</div>
<div class="workflow-stages" id="workflowStages"></div>
</section>
</div>
<section>
<div class="side">
<section id="gitSyncSection">
<div class="section-head">
<h2>1. Git Sync</h2>
<small id="pushStatus"></small>
@@ -699,7 +850,7 @@
</div>
</section>
<section>
<section id="prepareSection">
<div class="section-head">
<h2>2. Prepare Changes</h2>
<small id="prepareStatus"></small>
@@ -724,7 +875,7 @@
</div>
</section>
<section>
<section id="sourceReleaseSection">
<div class="section-head">
<h2>3. Source Release Tag</h2>
<small><span id="tagStatus">idle</span> · plan <span id="planStatus">idle</span></small>
@@ -751,7 +902,7 @@
<div class="actions" id="actions"></div>
</section>
<section>
<section id="catalogPublishSection">
<div class="section-head">
<h2>4. Signed Website Catalog</h2>
<small id="workflowStatus"></small>
@@ -781,7 +932,7 @@
</div>
</section>
<section>
<section id="catalogStateSection">
<div class="section-head">
<h2>Catalog State</h2>
<small id="catalogStatus"></small>
@@ -789,13 +940,13 @@
<div class="details" id="catalog"></div>
</section>
<section>
<section id="installVerificationSection">
<div class="section-head">
<h2>5. Install Catalog</h2>
<small>planned</small>
<h2>5. Installation Verification</h2>
<small>receipt-bound local gate</small>
</div>
<div class="details">
<p class="hint">Future work: signed module archives and an installer/update catalog for deployments.</p>
<p class="hint">The durable verification step installs every selected candidate wheel into a private temporary target with network and dependency resolution disabled, then compares installed package metadata with the frozen release plan. Deployment-specific integration checks remain part of CI.</p>
</div>
</section>
</div>
@@ -850,6 +1001,8 @@
unitCount: document.getElementById("unitCount"),
workflowStageStatus: document.getElementById("workflowStageStatus"),
workflowStages: document.getElementById("workflowStages"),
workflowGuidance: document.getElementById("workflowGuidance"),
workflowPrimaryAction: document.getElementById("workflowPrimaryAction"),
catalog: document.getElementById("catalog"),
catalogStatus: document.getElementById("catalogStatus"),
actions: document.getElementById("actions"),
@@ -908,8 +1061,10 @@
const releaseState = {
rows: {},
dashboard: null,
currentPlan: null,
manualRepo: "",
currentRun: null,
workflowAction: null,
runs: [],
runNextCursor: null,
runStoreAvailable: true,
@@ -944,9 +1099,16 @@
elements.cancelManualTarget.addEventListener("click", closeManualTarget);
elements.applyManualTarget.addEventListener("click", applyManualTarget);
elements.manualTargetInput.addEventListener("input", validateManualTarget);
elements.workflowPrimaryAction.addEventListener("click", runWorkflowPrimaryAction);
elements.candidateDir.addEventListener("input", () => {
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
});
for (const control of [elements.channel, elements.publicCatalog, elements.online, elements.migrations]) {
control.addEventListener("change", () => {
invalidateReleaseDraft();
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
});
}
function api(path) {
const separator = path.indexOf("?");
@@ -1067,6 +1229,7 @@
if (selectedId && releaseState.runs.some((run) => run.run_id === selectedId)) {
elements.releaseRun.value = selectedId;
}
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
function renderReleaseRunStoreUnavailable(error) {
@@ -1081,6 +1244,7 @@
elements.resumeReleaseRun.disabled = true;
elements.runStatus.textContent = "unavailable";
elements.runOutput.innerHTML = `<div class="action"><h3>${pill("unavailable", "block")} Durable run storage cannot be read</h3><p>${escapeHtml(error.message)}</p><p class="action-meta">Dashboard and release previews remain available. Correct the private state-directory problem, then refresh.</p></div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
async function createReleaseRun() {
@@ -1203,6 +1367,7 @@
elements.runStatus.textContent = "no run selected";
elements.resumeReleaseRun.disabled = true;
elements.runOutput.innerHTML = `<div class="muted">Create a run after selecting exact repository versions, or choose a saved run.</div>`;
renderIdlePlan();
return;
}
elements.runStatus.textContent = "loading...";
@@ -1217,6 +1382,7 @@
elements.runStatus.textContent = "unavailable";
elements.resumeReleaseRun.disabled = true;
elements.runOutput.innerHTML = `<div class="action"><h3>${pill("unavailable", "block")} Run cannot be read</h3><p>${escapeHtml(error.message)}</p></div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
}
@@ -1264,7 +1430,7 @@
<div><label>Confirmation</label><input type="text" data-run-reconcile-confirm placeholder="Type RECONCILE" autocomplete="off" /></div>
<div class="button-row"><button class="secondary" data-run-reconcile-submit disabled title="Verify local and remote state independently, choose the observed outcome, and type RECONCILE.">Record Reconciliation</button></div>
</div>` : "";
return `<div class="action">
return `<div class="action" data-run-step-id="${escapeAttr(step.id)}">
<div class="stage-title"><h3>${escapeHtml(`${step.order}. ${step.title || step.id}`)}</h3>${pill(step.state, kind)}</div>
<p>${escapeHtml(step.detail || "")}</p>
${step.result_code ? `<p class="action-meta">Result code: ${escapeHtml(step.result_code)}</p>` : ""}
@@ -1304,6 +1470,7 @@
submit.addEventListener("click", () => executeReleaseRunStep(controls.dataset.runExecuteStep, required, submit));
if (preview) preview.addEventListener("click", () => previewReleaseRunStep(controls.dataset.runExecuteStep, preview));
}
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
async function executeReleaseRunStep(stepId, confirmation, button) {
@@ -1571,62 +1738,247 @@
function renderWorkflowStages(data) {
const s = data.summary || {};
const selected = elements.releaseUnits ? elements.releaseUnits.querySelectorAll("[data-release-check]:checked").length : 0;
const gitBlockers = (s.missing_count || 0) + (s.safe_directory_count || 0);
const branchDrift = (s.ahead_count || 0) + (s.behind_count || 0);
const releaseBlockers = gitBlockers + (s.behind_count || 0) + (s.no_head_count || 0) + (s.dirty_count || 0) + (s.error_count || 0);
const catalog = data.catalog || {};
const hasCandidate = Boolean(elements.candidateDir?.value.trim());
const catalogPublished = catalog.catalog_matches_public === true && catalog.keyring_matches_public === true;
const gitStatus = gitBlockers ? "blocked" : branchDrift ? "sync" : "current";
const tagStatus = releaseBlockers ? "blocked" : selected ? "plan ready" : "select repos";
const signedStatus = hasCandidate ? "candidate" : "waiting";
const catalogStatus = catalogPublished ? "published" : catalog.public_checked ? "review" : "local";
const prepareStatus = s.dirty_count ? "dirty" : "clean";
elements.workflowStageStatus.textContent = data.summary?.status || "unknown";
elements.workflowStages.innerHTML = [
workflowStage(
"1. Sync local/remote Git",
gitStatus,
gitBlockers ? "block" : branchDrift ? "warn" : "ok",
`${s.ahead_count || 0} ahead / ${s.behind_count || 0} behind / ${s.dirty_count || 0} dirty repositories`
),
workflowStage(
"2. Prepare changes",
prepareStatus,
s.dirty_count ? "warn" : "ok",
s.dirty_count
? `${s.dirty_count} dirty repositories need explicit prepare commits.`
: "No dirty worktrees in the current dashboard."
),
workflowStage(
"3. Publish source release tags",
tagStatus,
releaseBlockers ? "block" : selected ? "ok" : "warn",
selected
? `${selected} selected; preview here, then create a durable run and execute its enabled tag/push steps.`
: "Select repositories, set target versions, then preview the source release tags."
),
workflowStage(
"4. Sign and publish website catalog",
signedStatus,
hasCandidate ? "ok" : "warn",
hasCandidate ? `Preview candidate: ${elements.candidateDir.value.trim()}` : "After every selected source tag is remotely published, use a durable run to generate and publish its receipt-bound candidate."
),
workflowStage(
"5. Maintain install catalog",
catalogStatus,
catalogPublished ? "ok" : "warn",
"Future: signed module archives for installer and module update workflows."
),
].join("");
const plan = releaseState.currentRun?.immutable?.plan || releaseState.currentPlan;
const run = releaseState.currentRun;
const phases = releaseWorkflowPhases({ summary: s, selected, plan, run });
const completed = phases.filter((phase) => phase.state === "complete").length;
const blocked = phases.find((phase) => phase.state === "blocked" || phase.state === "unavailable");
const active = phases.find((phase) => phase.state === "current") || blocked;
elements.workflowStageStatus.textContent = `${completed}/${phases.length} complete${run ? ` · ${run.state?.status || "unknown"}` : ""}`;
elements.workflowStages.innerHTML = phases.map(workflowStage).join("");
for (const stage of elements.workflowStages.querySelectorAll("[data-workflow-target]")) {
stage.addEventListener("click", () => scrollToReleaseTarget(stage.dataset.workflowTarget));
}
renderWorkflowGuidance({ summary: s, selected, plan, run, active });
}
function workflowStage(title, status, kind, detail) {
return `<div class="action">
<div class="stage-title"><h3>${escapeHtml(title)}</h3>${pill(status, kind)}</div>
<p>${escapeHtml(detail)}</p>
</div>`;
function releaseWorkflowPhases({ summary, selected, plan, run }) {
const inspectionNotices = (summary.missing_count || 0)
+ (summary.safe_directory_count || 0)
+ (summary.error_count || 0);
const inspectionBlockers = (summary.repository_count || 0) === 0 ? 1 : 0;
const phases = [
{
id: "inspect",
label: "Inspect",
target: "releaseControlSection",
state: inspectionBlockers ? "blocked" : "complete",
status: inspectionBlockers
? "no repositories"
: inspectionNotices
? `${inspectionNotices} notices`
: "ready",
},
{
id: "select",
label: "Targets",
target: "releaseTargetsSection",
state: selected || run ? "complete" : inspectionBlockers ? "locked" : "current",
status: selected || run ? `${selected || Object.keys(run?.immutable?.input?.repo_versions || {}).length} selected` : "select",
},
validationWorkflowPhase({ selected, plan, run, inspectionBlockers }),
runWorkflowPhase({
id: "source",
label: "Source",
target: "releaseRunSection",
run,
plan,
match: (step) => /:(preflight|version|commit|bundle|tag|push)$/.test(step.id || "") || step.id === "release:alignment",
waitingStatus: "create run",
}),
runWorkflowPhase({
id: "package",
label: "Package",
target: "releaseRunSection",
run,
plan,
match: (step) => ["catalog:initial-version-metadata", "catalog:selective-generator"].includes(step.id),
waitingStatus: "waiting",
}),
runWorkflowPhase({
id: "publish",
label: "Publish",
target: "releaseRunSection",
run,
plan,
match: (step) => step.id === "catalog:validate-sign-publish",
waitingStatus: "waiting",
}),
runWorkflowPhase({
id: "verify",
label: "Verify",
target: "installVerificationSection",
run,
plan,
match: (step) => step.id === "release:install-verify",
waitingStatus: "waiting",
}),
];
const firstOpen = phases.findIndex((phase) => ["current", "blocked", "unavailable"].includes(phase.state));
if (firstOpen >= 0) {
for (let index = firstOpen + 1; index < phases.length; index += 1) {
if (phases[index].state !== "complete") phases[index] = { ...phases[index], state: "locked" };
}
}
return phases;
}
function validationWorkflowPhase({ selected, plan, run, inspectionBlockers }) {
if (inspectionBlockers) return { id: "validate", label: "Validate", target: "sourceReleaseSection", state: "locked", status: "waiting" };
if (!selected && !run) return { id: "validate", label: "Validate", target: "sourceReleaseSection", state: "locked", status: "waiting" };
if (!plan) return { id: "validate", label: "Validate", target: "sourceReleaseSection", state: "current", status: "build plan" };
if (plan.status === "blocked") return { id: "validate", label: "Validate", target: "sourceReleaseSection", state: "blocked", status: "blocked" };
return { id: "validate", label: "Validate", target: "sourceReleaseSection", state: "complete", status: "gates pass" };
}
function runWorkflowPhase({ id, label, target, run, plan, match, waitingStatus }) {
if (!plan || plan.status === "blocked") return { id, label, target, state: "locked", status: waitingStatus };
if (!run) return {
id,
label,
target,
state: id === "source" ? "current" : "locked",
status: id === "source" ? "create run" : waitingStatus,
};
const steps = (run.state?.steps || []).filter(match);
if (!steps.length) return { id, label, target, state: "unavailable", status: "not available" };
const succeeded = steps.filter((step) => step.state === "succeeded").length;
if (succeeded === steps.length) return { id, label, target, state: "complete", status: `${succeeded}/${steps.length}` };
if (steps.some((step) => ["failed", "interrupted"].includes(step.state))) {
return { id, label, target, state: "blocked", status: `${succeeded}/${steps.length}` };
}
if (steps.some((step) => step.state === "running" || step.available)) {
return { id, label, target, state: "current", status: `${succeeded}/${steps.length}` };
}
const next = steps.find((step) => step.state === "pending");
if (next && next.executor?.available === false && next.status !== "planned") {
return { id, label, target, state: "unavailable", status: next.status || "unavailable" };
}
return { id, label, target, state: "locked", status: `${succeeded}/${steps.length}` };
}
function workflowStage(phase, index) {
const unavailable = phase.state === "unavailable";
const marker = phase.state === "complete" ? "✓" : String(index + 1);
return `<button type="button" class="workflow-stage is-${escapeAttr(phase.state)}" data-workflow-target="${escapeAttr(phase.target)}" ${unavailable ? "disabled" : ""} title="${escapeAttr(phase.status)}">
<span class="workflow-stage-marker">${escapeHtml(marker)}</span>
<span class="workflow-stage-label">${escapeHtml(phase.label)}</span>
<span class="workflow-stage-status">${escapeHtml(phase.status)}</span>
</button>`;
}
function renderWorkflowGuidance({ summary, selected, plan, run, active }) {
const recommendation = run?.recommended_next || plan?.recommended_action || null;
let guidance;
if ((summary.repository_count || 0) === 0) {
guidance = {
title: "Restore the release workspace",
detail: "No registered repositories are available for release planning.",
remediation: "Review Release Control, restore the workspace or repository registry, then refresh.",
button: "Review blockers",
action: { type: "scroll", target: "releaseControlSection" },
};
} else if (!selected && !run) {
guidance = {
title: "Select release targets",
detail: "Choose one or more repositories and confirm each exact target version.",
remediation: "The release plan will include only the selected independently versioned repositories.",
button: "Choose targets",
action: { type: "scroll", target: "releaseTargetsSection" },
};
} else if (!plan) {
guidance = {
title: "Validate the selected release",
detail: "Build a dry-run plan to check versions, contracts, migrations, Git state, and release composition.",
remediation: "Planning is read-only and does not create tags or artifacts.",
button: "Build plan",
action: { type: "build-plan" },
};
} else if (plan.status === "blocked") {
guidance = {
title: recommendation?.title || "Resolve release gates",
detail: recommendation?.detail || "The current plan contains blockers.",
remediation: recommendation?.remediation || "Review the plan findings, correct the source state, then build a fresh plan.",
button: "Review plan",
action: { type: "scroll", target: "sourceReleaseSection" },
};
} else if (!run) {
guidance = releaseState.runStoreAvailable ? {
title: "Freeze a durable release run",
detail: `${selected} repository target${selected === 1 ? "" : "s"} passed the plan-visible gates.`,
remediation: "Creating the run binds the exact plan, source commits, registered remotes, and ordered mutation boundaries.",
button: "Create durable run",
action: { type: "create-run" },
} : {
title: "Restore durable run storage",
detail: "The release plan is valid, but the console cannot persist a safe execution record.",
remediation: "Review the run-storage error and restore the private state directory before executing release mutations.",
button: "Review storage error",
action: { type: "scroll", target: "releaseRunSection" },
};
} else if (recommendation?.step_id) {
guidance = {
title: recommendation.title || "Continue the release run",
detail: recommendation.detail || "The next durable step is ready for review.",
remediation: recommendation.remediation || "Open the step, review its prerequisites, and use its narrowly scoped confirmation.",
button: ["retry_step", "reconcile_step"].includes(recommendation.id) ? "Resolve step" : "Open next step",
action: { type: "run-step", target: recommendation.step_id },
};
} else if (run.state?.status === "completed") {
guidance = {
title: "Release run completed",
detail: "Every durable source, package, signing, publication, and local installation-verification step has a recorded success.",
remediation: "Review the run receipts and continue with deployment-specific integration checks where required.",
button: "Review completed run",
action: { type: "scroll", target: "installVerificationSection" },
};
} else {
guidance = {
title: recommendation?.title || active?.label || "Review release state",
detail: recommendation?.detail || "The next release action is not currently executable.",
remediation: recommendation?.remediation || "Review the blocked or unavailable step and its prerequisites.",
button: "Review run",
action: { type: "scroll", target: "releaseRunSection" },
};
}
releaseState.workflowAction = guidance.action;
elements.workflowGuidance.innerHTML = `<div>
<h3>${escapeHtml(guidance.title)}</h3>
<p>${escapeHtml(guidance.detail)}</p>
<p class="action-meta"><strong>Next:</strong> ${escapeHtml(guidance.remediation)}</p>
</div>
<button id="workflowPrimaryAction">${escapeHtml(guidance.button)}</button>`;
elements.workflowPrimaryAction = document.getElementById("workflowPrimaryAction");
elements.workflowPrimaryAction.addEventListener("click", runWorkflowPrimaryAction);
}
function runWorkflowPrimaryAction() {
const action = releaseState.workflowAction;
if (!action) return;
if (action.type === "build-plan") {
void buildSelectedPlan();
return;
}
if (action.type === "create-run") {
void createReleaseRun();
return;
}
if (action.type === "run-step") {
const step = elements.runOutput.querySelector(`[data-run-step-id="${cssEscape(action.target)}"]`);
if (step) {
step.scrollIntoView({ behavior: "smooth", block: "center" });
step.querySelector("input:not(:disabled), select:not(:disabled), button:not(:disabled)")?.focus();
return;
}
scrollToReleaseTarget("releaseRunSection");
return;
}
scrollToReleaseTarget(action.target);
}
function scrollToReleaseTarget(targetId) {
const target = document.getElementById(targetId);
if (target) target.scrollIntoView({ behavior: "smooth", block: "start" });
}
function renderReleaseIntelligence(data) {
@@ -1722,6 +2074,7 @@
input.addEventListener("change", () => {
const repo = input.dataset.repo;
releaseState.rows[repo] = { ...releaseState.rows[repo], selected: input.checked };
invalidateReleaseDraft();
updateUnitCount();
});
}
@@ -1867,6 +2220,7 @@
}
function resetTarget(repoName) {
invalidateReleaseDraft();
const row = { ...releaseState.rows[repoName] };
delete row.target;
releaseState.rows[repoName] = row;
@@ -1879,6 +2233,7 @@
}
setRowTargetTag(repoName, target);
elements.statusLine.textContent = `${repoName} target reset.`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
function currentRowTarget(repoName) {
@@ -1886,6 +2241,7 @@
}
function setRowTarget(repoName, value) {
invalidateReleaseDraft();
releaseState.rows[repoName] = { ...releaseState.rows[repoName], target: value };
const element = rowTargetElement(repoName);
if (element) {
@@ -1893,6 +2249,7 @@
element.textContent = value;
}
setRowTargetTag(repoName, value);
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
function rowTargetElement(repoName) {
@@ -2002,6 +2359,7 @@
}
function renderPlan(plan) {
releaseState.currentPlan = plan;
elements.planStatus.textContent = plan.status;
const units = plan.units || [];
const compatibility = plan.compatibility || [];
@@ -2010,6 +2368,7 @@
const recommendation = plan.recommended_action || null;
if (!units.length && !compatibility.length && !steps.length && !recommendation) {
elements.actions.innerHTML = `<div class="muted">No selective release candidates. Select repositories above to plan a release.</div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
return;
}
const recommendationKind = plan.status === "blocked" ? "block" : plan.source_preflight_ready ? "ok" : "warn";
@@ -2032,9 +2391,12 @@
const blockers = unit.blockers.length ? `<p>${escapeHtml(unit.blockers.join("; "))}</p>` : "";
const warnings = unit.warnings.length ? `<p>${escapeHtml(unit.warnings.join("; "))}</p>` : "";
const kind = unit.status === "blocked" ? "block" : unit.status === "attention" ? "warn" : "ok";
const capabilities = Array.isArray(unit.capabilities) && unit.capabilities.length
? `<p class="action-meta">Capabilities: ${escapeHtml(unit.capabilities.join(", "))}</p>`
: "";
return `<div class="action">
<h3>${pill(unit.status, kind)} ${escapeHtml(unit.repo)} ${escapeHtml(unit.current_version || "-")} -> ${escapeHtml(unit.target_version)}</h3>
${blockers}${warnings}
${capabilities}${blockers}${warnings}
</div>`;
}).join("");
const compatibilityHtml = compatibility.map((issue) => {
@@ -2060,11 +2422,27 @@
${compatibilityHtml}
${stepHtml}
`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
function renderIdlePlan() {
releaseState.currentPlan = null;
elements.planStatus.textContent = "idle";
elements.actions.innerHTML = `<div class="muted">Select repositories above, adjust target versions, then build a plan.</div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
}
function invalidateReleaseDraft() {
releaseState.currentPlan = null;
if (releaseState.currentRun) {
releaseState.currentRun = null;
elements.releaseRun.value = "";
elements.runStatus.textContent = "no run selected";
elements.resumeReleaseRun.disabled = true;
elements.runOutput.innerHTML = `<div class="muted">The repository selection changed. Build a fresh plan before creating a durable run; saved runs remain available from the selector.</div>`;
}
elements.planStatus.textContent = "idle";
elements.actions.innerHTML = `<div class="muted">The release inputs changed. Build a fresh plan before continuing.</div>`;
}
async function generateCandidate() {
@@ -2099,16 +2477,20 @@
setBusy([elements.buildSelectedPlan], true);
try {
if (!Object.keys(selectedRepoVersions()).length) {
releaseState.currentPlan = null;
elements.planStatus.textContent = "idle";
elements.actions.innerHTML = `<div class="action"><h3>${pill("idle", "warn")} No repositories selected</h3><p>Select one or more rows in the repository table first.</p></div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
return;
}
const plan = await api("/api/selective-plan");
renderPlan(plan);
elements.planStatus.textContent = plan.status;
} catch (error) {
releaseState.currentPlan = null;
elements.planStatus.textContent = "error";
elements.actions.innerHTML = `<div class="action"><h3>${pill("error", "block")} Plan failed</h3><p>${escapeHtml(error.message)}</p></div>`;
if (releaseState.dashboard) renderWorkflowStages(releaseState.dashboard);
} finally {
setBusy([elements.buildSelectedPlan], false);
}
@@ -2116,6 +2498,7 @@
function selectChangedUnits() {
if (!releaseState.dashboard) return;
invalidateReleaseDraft();
let count = 0;
for (const repo of releaseState.dashboard.repositories) {
const current = primaryVersion(repo);
@@ -2133,6 +2516,7 @@
function selectUnpushedUnits() {
if (!releaseState.dashboard) return;
invalidateReleaseDraft();
let count = 0;
for (const repo of releaseState.dashboard.repositories) {
const current = primaryVersion(repo);
@@ -2150,6 +2534,7 @@
function selectUnreleasedUnits() {
if (!releaseState.dashboard) return;
invalidateReleaseDraft();
for (const repo of releaseState.dashboard.repositories) {
if (repo.spec.name === "govoplan") continue;
const unreleased = !primaryVersion(repo);
@@ -2163,11 +2548,11 @@
}
function clearSelectedUnits() {
invalidateReleaseDraft();
for (const repo of Object.keys(releaseState.rows)) {
releaseState.rows[repo] = { ...releaseState.rows[repo], selected: false };
}
if (releaseState.dashboard) renderReleaseUnits(releaseState.dashboard);
renderIdlePlan();
}
function selectedRepoNames() {
+112 -9
View File
@@ -3,35 +3,138 @@ from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
from pathlib import Path
from urllib.parse import urlsplit
ROOT = Path(__file__).resolve().parents[2]
REGISTERED_TRANSPORT = "registered"
PUBLIC_HTTPS_TRANSPORT = "public-https"
GITEA_SSH_PREFIX = "git@git.add-ideas.de:"
GITEA_HTTPS_PREFIX = "https://git.add-ideas.de/"
GITEA_REPOSITORY_PATH = re.compile(
r"(?:GovOPlaN|add-ideas)/[a-z0-9][a-z0-9-]*[.]git"
)
def parse_args() -> argparse.Namespace:
def parse_args(argv: list[str] | None = None) -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Clone GovOPlaN repositories listed in repositories.json.")
parser.add_argument("--check", action="store_true", help="Only report missing repositories.")
parser.add_argument("--parent", type=Path, help="Override checkout parent directory.")
return parser.parse_args()
parser.add_argument(
"--repo",
action="append",
default=[],
metavar="NAME",
help="Clone only this registered repository; repeat for more than one.",
)
parser.add_argument(
"--exclude-repo",
action="append",
default=[],
metavar="NAME",
help="Exclude this registered repository; repeat for more than one.",
)
parser.add_argument(
"--transport",
choices=(REGISTERED_TRANSPORT, PUBLIC_HTTPS_TRANSPORT),
default=REGISTERED_TRANSPORT,
help=(
"Clone registered remotes unchanged, or convert public "
"git.add-ideas.de SSH remotes to anonymous HTTPS."
),
)
return parser.parse_args(argv)
def main() -> int:
args = parse_args()
manifest = json.loads((ROOT / "repositories.json").read_text())
def clone_remote(remote: str, *, transport: str) -> str:
if transport == REGISTERED_TRANSPORT:
return remote
if transport != PUBLIC_HTTPS_TRANSPORT:
raise ValueError(f"unsupported repository transport: {transport}")
if remote.startswith(GITEA_SSH_PREFIX):
repository_path = remote.removeprefix(GITEA_SSH_PREFIX)
else:
parsed = urlsplit(remote)
if (
parsed.scheme not in {"https", "ssh"}
or parsed.hostname != "git.add-ideas.de"
or parsed.port is not None
or parsed.password is not None
or parsed.query
or parsed.fragment
):
raise ValueError(
"public HTTPS bootstrap only accepts registered "
"git.add-ideas.de remotes"
)
if parsed.scheme == "https" and parsed.username is not None:
raise ValueError("public HTTPS bootstrap does not accept credentials")
if parsed.scheme == "ssh" and parsed.username != "git":
raise ValueError("registered Gitea SSH remotes must use the git account")
repository_path = parsed.path.removeprefix("/")
if GITEA_REPOSITORY_PATH.fullmatch(repository_path) is None:
raise ValueError("registered Gitea repository path is malformed")
return GITEA_HTTPS_PREFIX + repository_path
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
manifest = json.loads((ROOT / "repositories.json").read_text(encoding="utf-8"))
parent = args.parent or Path(manifest["default_parent"])
missing: list[dict[str, str]] = []
missing: list[tuple[dict[str, str], Path, str]] = []
repositories = manifest["repositories"]
names = [entry["name"] for entry in repositories]
if len(names) != len(set(names)):
raise ValueError("repository registry contains duplicate names")
requested = set(args.repo)
excluded = set(args.exclude_repo)
unknown = (requested | excluded) - set(names)
if unknown:
raise ValueError(f"unknown registered repositories: {', '.join(sorted(unknown))}")
overlap = requested & excluded
if overlap:
raise ValueError(
f"repositories cannot be selected and excluded: {', '.join(sorted(overlap))}"
)
for entry in manifest["repositories"]:
for entry in repositories:
if (requested and entry["name"] not in requested) or entry["name"] in excluded:
continue
repo = parent / entry["path"]
if repo.exists():
continue
missing.append(entry)
transport = entry.get("bootstrap_transport", args.transport)
remote = clone_remote(entry["remote"], transport=transport)
missing.append((entry, repo, remote))
environment = os.environ.copy()
environment.update(
{
"GIT_ASKPASS": "/bin/false",
"GIT_TERMINAL_PROMPT": "0",
}
)
for entry, repo, remote in missing:
print(f"missing: {entry['name']} -> {repo}")
if not args.check:
subprocess.run(["git", "clone", entry["remote"], str(repo)], check=True)
subprocess.run(
[
"git",
"-c",
"credential.helper=",
"clone",
"--",
remote,
str(repo),
],
check=True,
env=environment,
)
if args.check:
return 1 if missing else 0
+198 -10
View File
@@ -12,6 +12,7 @@ from pathlib import Path
import shlex
import subprocess
import sys
import tomllib
from typing import Any
META_ROOT = Path(__file__).resolve().parents[2]
@@ -49,6 +50,23 @@ class InstallPlan:
warnings: tuple[str, ...] = ()
@dataclass(frozen=True, slots=True)
class LocalInstallationExpectation:
requirement: RequirementEntry
distribution: str
entry_points: tuple[tuple[str, str, str], ...]
@dataclass(frozen=True, slots=True)
class EnvironmentValidation:
stale_requirements: tuple[RequirementEntry, ...] = ()
issues: tuple[str, ...] = ()
@property
def current(self) -> bool:
return not self.stale_requirements
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
@@ -85,8 +103,9 @@ def main() -> int:
fingerprint = build_fingerprint(requirements=requirements, python=python, local_requirements=local_requirements)
previous = load_stamp(stamp)
current = previous.get("digest") == fingerprint["digest"]
environment = validate_local_installations(python, local_requirements)
if current and not args.force:
if current and environment.current and not args.force:
payload = stamp_payload(
requirements=requirements,
python=python,
@@ -107,20 +126,31 @@ def main() -> int:
print(f"Python environment is current for {requirements.name}.")
return 0
print(f"Python environment metadata is stale for {requirements.name}.")
if current and not environment.current:
print(f"Python environment metadata is current for {requirements.name}, but installed packages need repair.")
else:
print(f"Python environment metadata is stale for {requirements.name}.")
print(f"Tracked inputs: {len(fingerprint['inputs'])}")
for issue in environment.issues:
print(f"warning: {issue}", file=sys.stderr)
if args.check:
return 1
plan = build_install_plan(
previous=previous,
fingerprint=fingerprint,
requirements=requirements,
python=python,
local_requirements=local_requirements,
force=args.force,
)
if current and environment.stale_requirements and not args.force:
plan = build_environment_repair_plan(
python=python,
stale_requirements=environment.stale_requirements,
)
else:
plan = build_install_plan(
previous=previous,
fingerprint=fingerprint,
requirements=requirements,
python=python,
local_requirements=local_requirements,
force=args.force,
)
print(f"{plan.mode}: {plan.reason}")
for warning in plan.warnings:
@@ -141,6 +171,13 @@ def main() -> int:
for command in commands:
subprocess.run(command, check=True)
repaired_environment = validate_local_installations(python, local_requirements)
if not repaired_environment.current:
for issue in repaired_environment.issues:
print(f"error: {issue}", file=sys.stderr)
print("Python environment synchronization did not install all local package metadata.", file=sys.stderr)
return 2
write_stamp(
stamp,
stamp_payload(
@@ -297,6 +334,127 @@ def build_install_plan(
)
def build_environment_repair_plan(
*,
python: str,
stale_requirements: tuple[RequirementEntry, ...],
) -> InstallPlan:
command = (
python,
"-m",
"pip",
"install",
*(
argument
for requirement in stale_requirements
for argument in requirement.install_args
),
)
return InstallPlan(
"Selective Python environment repair",
f"reinstalling {len(stale_requirements)} missing or stale local distribution(s).",
(command,),
)
def validate_local_installations(
python: str,
local_requirements: tuple[RequirementEntry, ...],
) -> EnvironmentValidation:
expectations = local_installation_expectations(local_requirements)
if not expectations:
return EnvironmentValidation()
probe_input = [
{
"key": expectation.requirement.key,
"distribution": expectation.distribution,
"entry_points": [
{"group": group, "name": name, "value": value}
for group, name, value in expectation.entry_points
],
}
for expectation in expectations
]
try:
result = subprocess.run(
(python, "-c", _LOCAL_INSTALLATION_PROBE),
check=False,
text=True,
input=json.dumps(probe_input),
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
timeout=20,
)
except (OSError, subprocess.TimeoutExpired) as exc:
return EnvironmentValidation(
stale_requirements=tuple(item.requirement for item in expectations),
issues=(f"could not inspect installed local distributions ({type(exc).__name__})",),
)
if result.returncode != 0:
detail = result.stderr.strip() or result.stdout.strip() or "installation probe failed"
return EnvironmentValidation(
stale_requirements=tuple(item.requirement for item in expectations),
issues=(f"could not inspect installed local distributions: {detail}",),
)
try:
payload = json.loads(result.stdout)
except json.JSONDecodeError:
return EnvironmentValidation(
stale_requirements=tuple(item.requirement for item in expectations),
issues=("installed local distribution probe returned invalid JSON",),
)
raw_issues = payload.get("issues") if isinstance(payload, dict) else None
issues_by_key = raw_issues if isinstance(raw_issues, dict) else {}
stale = tuple(
expectation.requirement
for expectation in expectations
if expectation.requirement.key in issues_by_key
)
issues = tuple(
f"{expectation.distribution}: {issues_by_key[expectation.requirement.key]}"
for expectation in expectations
if expectation.requirement.key in issues_by_key
)
return EnvironmentValidation(stale_requirements=stale, issues=issues)
def local_installation_expectations(
local_requirements: tuple[RequirementEntry, ...],
) -> tuple[LocalInstallationExpectation, ...]:
expectations: list[LocalInstallationExpectation] = []
for requirement in local_requirements:
if requirement.pyproject is None:
continue
with Path(requirement.pyproject).open("rb") as handle:
payload = tomllib.load(handle)
project = payload.get("project")
if not isinstance(project, dict):
continue
distribution = project.get("name")
if not isinstance(distribution, str) or not distribution.strip():
continue
declared_entry_points: list[tuple[str, str, str]] = []
entry_point_groups = project.get("entry-points")
if isinstance(entry_point_groups, dict):
for group, entries in entry_point_groups.items():
if not isinstance(group, str) or not isinstance(entries, dict):
continue
for name, value in entries.items():
if isinstance(name, str) and isinstance(value, str):
declared_entry_points.append((group, name, value))
expectations.append(
LocalInstallationExpectation(
requirement=requirement,
distribution=distribution,
entry_points=tuple(sorted(declared_entry_points)),
)
)
return tuple(expectations)
def requirement_delta_plan(previous_entries: object, current_entries: tuple[RequirementEntry, ...]) -> InstallPlan | None:
parsed_previous = previous_requirement_entries(previous_entries)
if parsed_previous is None:
@@ -481,5 +639,35 @@ def format_command(command: tuple[str, ...]) -> str:
return " ".join(shlex.quote(item) for item in command)
_LOCAL_INSTALLATION_PROBE = """
import importlib.metadata
import json
import sys
expectations = json.load(sys.stdin)
issues = {}
for expectation in expectations:
key = expectation["key"]
distribution_name = expectation["distribution"]
try:
distribution = importlib.metadata.distribution(distribution_name)
except importlib.metadata.PackageNotFoundError:
issues[key] = "distribution is not installed"
continue
installed = {
(entry_point.group, entry_point.name): entry_point.value
for entry_point in distribution.entry_points
}
mismatches = []
for entry_point in expectation["entry_points"]:
identity = (entry_point["group"], entry_point["name"])
if installed.get(identity) != entry_point["value"]:
mismatches.append(f"{identity[0]}:{identity[1]}")
if mismatches:
issues[key] = "entry-point metadata is missing or stale: " + ", ".join(mismatches)
print(json.dumps({"issues": issues}, sort_keys=True))
"""
if __name__ == "__main__":
raise SystemExit(main())