diff --git a/docs/evidence/snapshots/DSAR_PROVIDER_COVERAGE.generated.md b/docs/evidence/snapshots/DSAR_PROVIDER_COVERAGE.generated.md new file mode 100644 index 0000000..09d1a77 --- /dev/null +++ b/docs/evidence/snapshots/DSAR_PROVIDER_COVERAGE.generated.md @@ -0,0 +1,87 @@ +# DSAR Provider Coverage + +This generated matrix is enforced by `tools/checks/check-dsar-coverage.py`. +A migration-owning module must register and document its canonical DSAR provider. +Every other active module requires a reviewed explanation of why it owns no +persistent subject-data store. Adding a migration invalidates that explanation. + +- Active modules: 68 +- Registered and documented DSAR providers: 44 +- Reviewed no-store rationales: 24 +- Unexplained coverage gaps: 0 + +| Module | Repository | Persistence | Coverage | Rationale | +| --- | --- | --- | --- | --- | +| `access` | `govoplan-access` | Migration-owned | Provider | Provider `privacy.dsar.access` is registered and documented. | +| `addresses` | `govoplan-addresses` | Migration-owned | Provider | Provider `privacy.dsar.addresses` is registered and documented. | +| `admin` | `govoplan-admin` | Migration-owned | Provider | Provider `privacy.dsar.admin` is registered and documented. | +| `approvals` | `govoplan-approvals` | Migration-owned | Provider | Provider `privacy.dsar.approvals` is registered and documented. | +| `assets` | `govoplan-assets` | No module migration | Reviewed no-store rationale | Contract-only module: asset persistence and lifecycle APIs are not implemented; reassess before adding a migration-owned store. | +| `audit` | `govoplan-audit` | Migration-owned | Provider | Provider `privacy.dsar.audit` is registered and documented. | +| `booking` | `govoplan-booking` | No module migration | Reviewed no-store rationale | Contract-only module: booking persistence and reservation workflows are not implemented; reassess before adding a migration-owned store. | +| `calendar` | `govoplan-calendar` | Migration-owned | Provider | Provider `privacy.dsar.calendar` is registered and documented. | +| `campaigns` | `govoplan-campaign` | Migration-owned | Provider | Provider `privacy.dsar.campaigns` is registered and documented. | +| `cases` | `govoplan-cases` | Migration-owned | Provider | Provider `privacy.dsar.cases` is registered and documented. | +| `certificates` | `govoplan-certificates` | No module migration | Reviewed no-store rationale | Contract-only module: certificate issuance and revocation persistence are not implemented; reassess before adding a migration-owned store. | +| `committee` | `govoplan-committee` | Migration-owned | Provider | Provider `privacy.dsar.committee` is registered and documented. | +| `connectors` | `govoplan-connectors` | Migration-owned | Provider | Provider `privacy.dsar.connectors` is registered and documented. | +| `consultation` | `govoplan-consultation` | No module migration | Reviewed no-store rationale | Contract-only module: consultation submissions and evaluation persistence are not implemented; reassess before adding a migration-owned store. | +| `contracts` | `govoplan-contracts` | No module migration | Reviewed no-store rationale | Contract-only module: contract, amendment, and obligation persistence are not implemented; reassess before adding a migration-owned store. | +| `dashboard` | `govoplan-dashboard` | Migration-owned | Provider | Provider `privacy.dsar.dashboard` is registered and documented. | +| `dataflow` | `govoplan-dataflow` | Migration-owned | Provider | Provider `privacy.dsar.dataflow` is registered and documented. | +| `datasources` | `govoplan-datasources` | Migration-owned | Provider | Provider `privacy.dsar.datasources` is registered and documented. | +| `decisions` | `govoplan-decisions` | Migration-owned | Provider | Provider `privacy.dsar.decisions` is registered and documented. | +| `dist_lists` | `govoplan-dist-lists` | Migration-owned | Provider | Provider `privacy.dsar.dist_lists` is registered and documented. | +| `docs` | `govoplan-docs` | No module migration | Reviewed no-store rationale | Projection-only module: documentation is contributed by owning manifests and providers; Docs has no migration-owned subject store. | +| `encryption` | `govoplan-encryption` | Migration-owned | Provider | Provider `privacy.dsar.encryption` is registered and documented. | +| `evaluation` | `govoplan-evaluation` | No module migration | Reviewed no-store rationale | Contract-only module: evaluation runs, responses, and scores are not persisted; reassess before adding a migration-owned store. | +| `facilities` | `govoplan-facilities` | No module migration | Reviewed no-store rationale | Contract-only module: facility and maintenance persistence are not implemented; reassess before adding a migration-owned store. | +| `files` | `govoplan-files` | Migration-owned | Provider | Provider `privacy.dsar.files` is registered and documented. | +| `forms` | `govoplan-forms` | Migration-owned | Provider | Provider `privacy.dsar.forms` is registered and documented. | +| `forms_runtime` | `govoplan-forms-runtime` | Migration-owned | Provider | Provider `privacy.dsar.forms_runtime` is registered and documented. | +| `grants` | `govoplan-grants` | No module migration | Reviewed no-store rationale | Contract-only module: grant applications, awards, and monitoring are not persisted; reassess before adding a migration-owned store. | +| `helpdesk` | `govoplan-helpdesk` | No module migration | Reviewed no-store rationale | Contract-only module: helpdesk requests, queues, and resolutions are not persisted; reassess before adding a migration-owned store. | +| `identity` | `govoplan-identity` | Migration-owned | Provider | Provider `privacy.dsar.identity` is registered and documented. | +| `identity_trust` | `govoplan-identity-trust` | Migration-owned | Provider | Provider `privacy.dsar.identity_trust` is registered and documented. | +| `idm` | `govoplan-idm` | Migration-owned | Provider | Provider `privacy.dsar.idm` is registered and documented. | +| `inspections` | `govoplan-inspections` | No module migration | Reviewed no-store rationale | Contract-only module: inspections, findings, and measures are not persisted; reassess before adding a migration-owned store. | +| `learning` | `govoplan-learning` | No module migration | Reviewed no-store rationale | Contract-only module: learning offers, enrollment, and completion are not persisted; reassess before adding a migration-owned store. | +| `mail` | `govoplan-mail` | Migration-owned | Provider | Provider `privacy.dsar.mail` is registered and documented. | +| `mandates` | `govoplan-mandates` | Migration-owned | Provider | Provider `privacy.dsar.mandates` is registered and documented. | +| `notifications` | `govoplan-notifications` | Migration-owned | Provider | Provider `privacy.dsar.notifications` is registered and documented. | +| `ops` | `govoplan-ops` | No module migration | Reviewed no-store rationale | Projection-only module: Ops reads bounded platform and provider status; durable recovery evidence remains owned by Core and domain modules. | +| `organizations` | `govoplan-organizations` | Migration-owned | Provider | Provider `privacy.dsar.organizations` is registered and documented. | +| `parties` | `govoplan-parties` | Migration-owned | Provider | Provider `privacy.dsar.parties` is registered and documented. | +| `payments` | `govoplan-payments` | Migration-owned | Provider | Provider `privacy.dsar.payments` is registered and documented. | +| `permits` | `govoplan-permits` | No module migration | Reviewed no-store rationale | Contract-only module: permit applications, assessments, and decisions are not persisted; reassess before adding a migration-owned store. | +| `policy` | `govoplan-policy` | Migration-owned | Provider | Provider `privacy.dsar.policy` is registered and documented. | +| `poll` | `govoplan-poll` | Migration-owned | Provider | Provider `privacy.dsar.poll` is registered and documented. | +| `portal` | `govoplan-portal` | No module migration | Reviewed no-store rationale | Projection-only module: Portal stores no applicant records; Services, Forms Runtime, Cases, and Postbox own and export authoritative subject data. | +| `postbox` | `govoplan-postbox` | Migration-owned | Provider | Provider `privacy.dsar.postbox` is registered and documented. | +| `procurement` | `govoplan-procurement` | No module migration | Reviewed no-store rationale | Contract-only module: procurement procedures, tenders, and awards are not persisted; reassess before adding a migration-owned store. | +| `projects` | `govoplan-projects` | Migration-owned | Provider | Provider `privacy.dsar.projects` is registered and documented. | +| `quick_access` | `govoplan-quick-access` | Migration-owned | Provider | Provider `privacy.dsar.quick_access` is registered and documented. | +| `records` | `govoplan-records` | Migration-owned | Provider | Provider `privacy.dsar.records` is registered and documented. | +| `reporting` | `govoplan-reporting` | Migration-owned | Provider | Provider `privacy.dsar.reporting` is registered and documented. | +| `resources` | `govoplan-resources` | No module migration | Reviewed no-store rationale | Contract-only module: resource catalog and allocation persistence are not implemented; reassess before adding a migration-owned store. | +| `rest` | `govoplan-rest` | No module migration | Reviewed no-store rationale | Transport-only module: REST binds explicitly published functions and owns no domain or subject-data store. | +| `risk_compliance` | `govoplan-risk-compliance` | Migration-owned | Provider | Provider `privacy.dsar.risk_compliance` is registered and documented. | +| `scheduling` | `govoplan-scheduling` | Migration-owned | Provider | Provider `privacy.dsar.scheduling` is registered and documented. | +| `search` | `govoplan-search` | Migration-owned | Provider | Provider `privacy.dsar.search` is registered and documented. | +| `services` | `govoplan-services` | Migration-owned | Provider | Provider `privacy.dsar.services` is registered and documented. | +| `soap` | `govoplan-soap` | No module migration | Reviewed no-store rationale | Transport-only module: SOAP binds explicitly published operations and owns no domain or subject-data store. | +| `tasks` | `govoplan-tasks` | Migration-owned | Provider | Provider `privacy.dsar.tasks` is registered and documented. | +| `templates` | `govoplan-templates` | Migration-owned | Provider | Provider `privacy.dsar.templates` is registered and documented. | +| `tenancy` | `govoplan-tenancy` | No module migration | Reviewed no-store rationale | Orchestration module: tenant lifecycle and settings use Core-owned storage; Access covers account and membership subject data. | +| `tickets` | `govoplan-tickets` | No module migration | Reviewed no-store rationale | Contract-only module: ticket, queue, SLA, and adapter persistence are not implemented; reassess before adding a migration-owned store. | +| `transparency` | `govoplan-transparency` | No module migration | Reviewed no-store rationale | Contract-only module: requests, disclosure reviews, and publications are not persisted; reassess before adding a migration-owned store. | +| `views` | `govoplan-views` | Migration-owned | Provider | Provider `privacy.dsar.views` is registered and documented. | +| `voting` | `govoplan-voting` | Migration-owned | Provider | Provider `privacy.dsar.voting` is registered and documented. | +| `wiki` | `govoplan-wiki` | No module migration | Reviewed no-store rationale | Contract-only module: wiki spaces, pages, and revisions are not persisted; reassess before adding a migration-owned store. | +| `workflow` | `govoplan-workflow` | No module migration | Reviewed no-store rationale | Presentation-only module: Workflow edits and projects Workflow Engine state; Workflow Engine owns persistence and DSAR coverage. | +| `workflow_engine` | `govoplan-workflow-engine` | Migration-owned | Provider | Provider `privacy.dsar.workflow_engine` is registered and documented. | + +Provider search, export minimization, retention, and erasure behavior remains +documented and tested by each owning module. This matrix verifies adoption and +ownership coverage; Core continues to test disabled providers, partial failure, +retry, authorization evidence, and horizontally coordinated execution. diff --git a/tools/checks/check-dsar-coverage.py b/tools/checks/check-dsar-coverage.py new file mode 100644 index 0000000..6e250c7 --- /dev/null +++ b/tools/checks/check-dsar-coverage.py @@ -0,0 +1,265 @@ +#!/usr/bin/env python3 +"""Require DSAR coverage or a reviewed no-store rationale for every module.""" + +from __future__ import annotations + +import argparse +import importlib +import json +import re +import sys +from dataclasses import dataclass +from pathlib import Path + + +META_ROOT = Path(__file__).resolve().parents[2] +EXEMPTIONS_PATH = Path(__file__).with_name("dsar-coverage-exemptions.json") +REPORT_PATH = ( + META_ROOT + / "docs" + / "evidence" + / "snapshots" + / "DSAR_PROVIDER_COVERAGE.generated.md" +) +MODULE_NAME_PATTERN = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*") +REQUIRED_DOCUMENTATION_TYPES = frozenset({"admin"}) + + +@dataclass(frozen=True, slots=True) +class CoverageRow: + module_id: str + repository: str + migration_owned: bool + capability: str | None + rationale: str + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--workspace-root", + type=Path, + default=None, + help="Directory containing GovOPlaN repositories.", + ) + parser.add_argument( + "--render", + action="store_true", + help="Print the current matrix instead of comparing the checked-in report.", + ) + args = parser.parse_args() + + catalog = json.loads((META_ROOT / "repositories.json").read_text(encoding="utf-8")) + workspace_root = (args.workspace_root or Path(catalog["default_parent"])).resolve() + exemptions = _exemptions() + manifests, load_errors = _load_manifests( + workspace_root=workspace_root, + repositories=tuple(catalog["repositories"]), + ) + + errors = list(load_errors) + rows: list[CoverageRow] = [] + manifest_ids = {manifest.id for _, manifest in manifests} + stale_exemptions = sorted(set(exemptions) - manifest_ids) + if stale_exemptions: + errors.append( + "DSAR coverage exemptions reference unknown modules: " + + ", ".join(stale_exemptions) + ) + + for repository, manifest in manifests: + expected = f"privacy.dsar.{manifest.id}" + provided = { + item.name + for item in manifest.provides_interfaces + if item.name.startswith("privacy.dsar.") + } + factories = { + name + for name in manifest.capability_factories + if name.startswith("privacy.dsar.") + } + migration_owned = manifest.migration_spec is not None + rationale = exemptions.get(manifest.id) + + if provided != factories: + errors.append( + f"{repository}: DSAR interface/factory mismatch: " + f"interfaces={sorted(provided)!r}, factories={sorted(factories)!r}" + ) + if provided and provided != {expected}: + errors.append( + f"{repository}: expected only {expected!r}, found {sorted(provided)!r}" + ) + + capability = ( + expected if expected in provided and expected in factories else None + ) + if migration_owned and capability is None: + errors.append( + f"{repository}: migration-owning module {manifest.id!r} must provide " + f"and register {expected!r}" + ) + if migration_owned and rationale is not None: + errors.append( + f"{repository}: migration-owning module {manifest.id!r} cannot use a " + "no-store DSAR exemption" + ) + if not migration_owned and capability is None and rationale is None: + errors.append( + f"{repository}: module {manifest.id!r} needs a DSAR provider or an " + "explicit reviewed no-store rationale" + ) + if capability is not None and rationale is not None: + errors.append( + f"{repository}: module {manifest.id!r} has both DSAR coverage and a " + "stale exemption" + ) + if capability is not None: + if capability not in manifest.capability_documentation: + errors.append( + f"{repository}: {capability!r} lacks capability documentation" + ) + matching_topics = tuple( + topic + for topic in manifest.documentation + if "data-subject-request" in topic.id + ) + if not matching_topics or not any( + REQUIRED_DOCUMENTATION_TYPES.issubset(topic.documentation_types) + for topic in matching_topics + ): + errors.append( + f"{repository}: DSAR coverage needs a static administrator " + "data-subject-requests DocumentationTopic" + ) + + rows.append( + CoverageRow( + module_id=manifest.id, + repository=repository, + migration_owned=migration_owned, + capability=capability, + rationale=( + f"Provider `{capability}` is registered and documented." + if capability + else rationale or "MISSING" + ), + ) + ) + + report = _report(rows) + if args.render: + print(report, end="") + elif not REPORT_PATH.is_file(): + errors.append(f"DSAR coverage report is missing: {REPORT_PATH}") + elif REPORT_PATH.read_text(encoding="utf-8") != report: + errors.append( + "DSAR coverage report is stale; review changes and replace it with " + "the output of tools/checks/check-dsar-coverage.py --render" + ) + + if errors: + print("\n".join(errors), file=sys.stderr) + return 1 + + provider_count = sum(row.capability is not None for row in rows) + print( + "DSAR coverage check passed: " + f"{provider_count} providers, {len(rows) - provider_count} reviewed " + f"no-store rationales, {len(rows)} active modules." + ) + return 0 + + +def _exemptions() -> dict[str, str]: + values = json.loads(EXEMPTIONS_PATH.read_text(encoding="utf-8")) + if not isinstance(values, dict) or any( + not isinstance(key, str) or not isinstance(value, str) or not value.strip() + for key, value in values.items() + ): + raise ValueError("DSAR coverage exemptions must be non-empty string mappings.") + return {key: value.strip() for key, value in values.items()} + + +def _load_manifests(*, workspace_root: Path, repositories: tuple[dict, ...]): + sources: list[Path] = [] + candidates: list[tuple[str, Path, Path]] = [] + for repository in repositories: + source = workspace_root / repository["path"] / "src" + if not source.is_dir(): + continue + sources.append(source) + candidates.extend( + (repository["name"], source, path) + for path in sorted(source.glob("*/backend/manifest.py")) + ) + core_source = workspace_root / "govoplan-core" / "src" + sys.path[:0] = [ + str(core_source), + *(str(source) for source in sources if source != core_source), + ] + + manifests = [] + errors = [] + for repository, source, path in candidates: + module_name = ".".join(path.relative_to(source).with_suffix("").parts) + if MODULE_NAME_PATTERN.fullmatch(module_name) is None: + errors.append(f"{repository}: unsafe manifest module name {module_name!r}") + continue + try: + module = importlib.import_module(module_name) + manifests.append((repository, module.get_manifest())) + except Exception as exc: # pragma: no cover - emitted as check evidence + errors.append(f"{repository}: could not load {module_name}: {exc}") + return manifests, errors + + +def _report(rows: list[CoverageRow]) -> str: + ordered = sorted(rows, key=lambda row: row.module_id) + providers = sum(row.capability is not None for row in ordered) + lines = [ + "# DSAR Provider Coverage", + "", + "This generated matrix is enforced by `tools/checks/check-dsar-coverage.py`.", + "A migration-owning module must register and document its canonical DSAR provider.", + "Every other active module requires a reviewed explanation of why it owns no", + "persistent subject-data store. Adding a migration invalidates that explanation.", + "", + f"- Active modules: {len(ordered)}", + f"- Registered and documented DSAR providers: {providers}", + f"- Reviewed no-store rationales: {len(ordered) - providers}", + "- Unexplained coverage gaps: 0", + "", + "| Module | Repository | Persistence | Coverage | Rationale |", + "| --- | --- | --- | --- | --- |", + ] + for row in ordered: + lines.append( + "| " + + " | ".join( + ( + f"`{row.module_id}`", + f"`{row.repository}`", + "Migration-owned" if row.migration_owned else "No module migration", + "Provider" if row.capability else "Reviewed no-store rationale", + row.rationale.replace("|", "\\|"), + ) + ) + + " |" + ) + lines.extend( + ( + "", + "Provider search, export minimization, retention, and erasure behavior remains", + "documented and tested by each owning module. This matrix verifies adoption and", + "ownership coverage; Core continues to test disabled providers, partial failure,", + "retry, authorization evidence, and horizontally coordinated execution.", + "", + ) + ) + return "\n".join(lines) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/checks/check-focused.sh b/tools/checks/check-focused.sh index 70a3281..8d93b83 100644 --- a/tools/checks/check-focused.sh +++ b/tools/checks/check-focused.sh @@ -38,6 +38,7 @@ cd "$ROOT" GOVOPLAN_CORE_ROOT="$ROOT" PYTHON="$PYTHON" CHECK_TESTCLIENT_DEPRECATIONS=1 bash "$META_ROOT/tools/checks/check-dependency-hygiene.sh" "$PYTHON" "$META_ROOT/tools/checks/check-contracts.py" --no-impact PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-manifest-shapes.py" --require-architecture +PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-dsar-coverage.py" cd "$META_ROOT" "$PYTHON" tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints diff --git a/tools/checks/dsar-coverage-exemptions.json b/tools/checks/dsar-coverage-exemptions.json new file mode 100644 index 0000000..44bc1a3 --- /dev/null +++ b/tools/checks/dsar-coverage-exemptions.json @@ -0,0 +1,26 @@ +{ + "assets": "Contract-only module: asset persistence and lifecycle APIs are not implemented; reassess before adding a migration-owned store.", + "booking": "Contract-only module: booking persistence and reservation workflows are not implemented; reassess before adding a migration-owned store.", + "certificates": "Contract-only module: certificate issuance and revocation persistence are not implemented; reassess before adding a migration-owned store.", + "consultation": "Contract-only module: consultation submissions and evaluation persistence are not implemented; reassess before adding a migration-owned store.", + "contracts": "Contract-only module: contract, amendment, and obligation persistence are not implemented; reassess before adding a migration-owned store.", + "docs": "Projection-only module: documentation is contributed by owning manifests and providers; Docs has no migration-owned subject store.", + "evaluation": "Contract-only module: evaluation runs, responses, and scores are not persisted; reassess before adding a migration-owned store.", + "facilities": "Contract-only module: facility and maintenance persistence are not implemented; reassess before adding a migration-owned store.", + "grants": "Contract-only module: grant applications, awards, and monitoring are not persisted; reassess before adding a migration-owned store.", + "helpdesk": "Contract-only module: helpdesk requests, queues, and resolutions are not persisted; reassess before adding a migration-owned store.", + "inspections": "Contract-only module: inspections, findings, and measures are not persisted; reassess before adding a migration-owned store.", + "learning": "Contract-only module: learning offers, enrollment, and completion are not persisted; reassess before adding a migration-owned store.", + "ops": "Projection-only module: Ops reads bounded platform and provider status; durable recovery evidence remains owned by Core and domain modules.", + "permits": "Contract-only module: permit applications, assessments, and decisions are not persisted; reassess before adding a migration-owned store.", + "portal": "Projection-only module: Portal stores no applicant records; Services, Forms Runtime, Cases, and Postbox own and export authoritative subject data.", + "procurement": "Contract-only module: procurement procedures, tenders, and awards are not persisted; reassess before adding a migration-owned store.", + "resources": "Contract-only module: resource catalog and allocation persistence are not implemented; reassess before adding a migration-owned store.", + "rest": "Transport-only module: REST binds explicitly published functions and owns no domain or subject-data store.", + "soap": "Transport-only module: SOAP binds explicitly published operations and owns no domain or subject-data store.", + "tenancy": "Orchestration module: tenant lifecycle and settings use Core-owned storage; Access covers account and membership subject data.", + "tickets": "Contract-only module: ticket, queue, SLA, and adapter persistence are not implemented; reassess before adding a migration-owned store.", + "transparency": "Contract-only module: requests, disclosure reviews, and publications are not persisted; reassess before adding a migration-owned store.", + "wiki": "Contract-only module: wiki spaces, pages, and revisions are not persisted; reassess before adding a migration-owned store.", + "workflow": "Presentation-only module: Workflow edits and projects Workflow Engine state; Workflow Engine owns persistence and DSAR coverage." +} diff --git a/tools/release/push-release-tag.sh b/tools/release/push-release-tag.sh index 4454aee..1f7afd7 100644 --- a/tools/release/push-release-tag.sh +++ b/tools/release/push-release-tag.sh @@ -571,6 +571,10 @@ run_manifest_shape_gate() { "$META_ROOT/tools/checks/check-manifest-shapes.py" \ --workspace-root "$PARENT" \ --require-architecture + run env PYTHONDONTWRITEBYTECODE=1 \ + "$PYTHON" \ + "$META_ROOT/tools/checks/check-dsar-coverage.py" \ + --workspace-root "$PARENT" } run_migration_release_audit() {