test(privacy): enforce workspace DSAR coverage

docs/evidence/snapshots/DSAR_PROVIDER_COVERAGE.generated.md

@@ -0,0 +1,87 @@
+# DSAR Provider Coverage
+
+This generated matrix is enforced by `tools/checks/check-dsar-coverage.py`.
+A migration-owning module must register and document its canonical DSAR provider.
+Every other active module requires a reviewed explanation of why it owns no
+persistent subject-data store. Adding a migration invalidates that explanation.
+
+- Active modules: 68
+- Registered and documented DSAR providers: 44
+- Reviewed no-store rationales: 24
+- Unexplained coverage gaps: 0
+
+| Module | Repository | Persistence | Coverage | Rationale |
+| --- | --- | --- | --- | --- |
+| `access` | `govoplan-access` | Migration-owned | Provider | Provider `privacy.dsar.access` is registered and documented. |
+| `addresses` | `govoplan-addresses` | Migration-owned | Provider | Provider `privacy.dsar.addresses` is registered and documented. |
+| `admin` | `govoplan-admin` | Migration-owned | Provider | Provider `privacy.dsar.admin` is registered and documented. |
+| `approvals` | `govoplan-approvals` | Migration-owned | Provider | Provider `privacy.dsar.approvals` is registered and documented. |
+| `assets` | `govoplan-assets` | No module migration | Reviewed no-store rationale | Contract-only module: asset persistence and lifecycle APIs are not implemented; reassess before adding a migration-owned store. |
+| `audit` | `govoplan-audit` | Migration-owned | Provider | Provider `privacy.dsar.audit` is registered and documented. |
+| `booking` | `govoplan-booking` | No module migration | Reviewed no-store rationale | Contract-only module: booking persistence and reservation workflows are not implemented; reassess before adding a migration-owned store. |
+| `calendar` | `govoplan-calendar` | Migration-owned | Provider | Provider `privacy.dsar.calendar` is registered and documented. |
+| `campaigns` | `govoplan-campaign` | Migration-owned | Provider | Provider `privacy.dsar.campaigns` is registered and documented. |
+| `cases` | `govoplan-cases` | Migration-owned | Provider | Provider `privacy.dsar.cases` is registered and documented. |
+| `certificates` | `govoplan-certificates` | No module migration | Reviewed no-store rationale | Contract-only module: certificate issuance and revocation persistence are not implemented; reassess before adding a migration-owned store. |
+| `committee` | `govoplan-committee` | Migration-owned | Provider | Provider `privacy.dsar.committee` is registered and documented. |
+| `connectors` | `govoplan-connectors` | Migration-owned | Provider | Provider `privacy.dsar.connectors` is registered and documented. |
+| `consultation` | `govoplan-consultation` | No module migration | Reviewed no-store rationale | Contract-only module: consultation submissions and evaluation persistence are not implemented; reassess before adding a migration-owned store. |
+| `contracts` | `govoplan-contracts` | No module migration | Reviewed no-store rationale | Contract-only module: contract, amendment, and obligation persistence are not implemented; reassess before adding a migration-owned store. |
+| `dashboard` | `govoplan-dashboard` | Migration-owned | Provider | Provider `privacy.dsar.dashboard` is registered and documented. |
+| `dataflow` | `govoplan-dataflow` | Migration-owned | Provider | Provider `privacy.dsar.dataflow` is registered and documented. |
+| `datasources` | `govoplan-datasources` | Migration-owned | Provider | Provider `privacy.dsar.datasources` is registered and documented. |
+| `decisions` | `govoplan-decisions` | Migration-owned | Provider | Provider `privacy.dsar.decisions` is registered and documented. |
+| `dist_lists` | `govoplan-dist-lists` | Migration-owned | Provider | Provider `privacy.dsar.dist_lists` is registered and documented. |
+| `docs` | `govoplan-docs` | No module migration | Reviewed no-store rationale | Projection-only module: documentation is contributed by owning manifests and providers; Docs has no migration-owned subject store. |
+| `encryption` | `govoplan-encryption` | Migration-owned | Provider | Provider `privacy.dsar.encryption` is registered and documented. |
+| `evaluation` | `govoplan-evaluation` | No module migration | Reviewed no-store rationale | Contract-only module: evaluation runs, responses, and scores are not persisted; reassess before adding a migration-owned store. |
+| `facilities` | `govoplan-facilities` | No module migration | Reviewed no-store rationale | Contract-only module: facility and maintenance persistence are not implemented; reassess before adding a migration-owned store. |
+| `files` | `govoplan-files` | Migration-owned | Provider | Provider `privacy.dsar.files` is registered and documented. |
+| `forms` | `govoplan-forms` | Migration-owned | Provider | Provider `privacy.dsar.forms` is registered and documented. |
+| `forms_runtime` | `govoplan-forms-runtime` | Migration-owned | Provider | Provider `privacy.dsar.forms_runtime` is registered and documented. |
+| `grants` | `govoplan-grants` | No module migration | Reviewed no-store rationale | Contract-only module: grant applications, awards, and monitoring are not persisted; reassess before adding a migration-owned store. |
+| `helpdesk` | `govoplan-helpdesk` | No module migration | Reviewed no-store rationale | Contract-only module: helpdesk requests, queues, and resolutions are not persisted; reassess before adding a migration-owned store. |
+| `identity` | `govoplan-identity` | Migration-owned | Provider | Provider `privacy.dsar.identity` is registered and documented. |
+| `identity_trust` | `govoplan-identity-trust` | Migration-owned | Provider | Provider `privacy.dsar.identity_trust` is registered and documented. |
+| `idm` | `govoplan-idm` | Migration-owned | Provider | Provider `privacy.dsar.idm` is registered and documented. |
+| `inspections` | `govoplan-inspections` | No module migration | Reviewed no-store rationale | Contract-only module: inspections, findings, and measures are not persisted; reassess before adding a migration-owned store. |
+| `learning` | `govoplan-learning` | No module migration | Reviewed no-store rationale | Contract-only module: learning offers, enrollment, and completion are not persisted; reassess before adding a migration-owned store. |
+| `mail` | `govoplan-mail` | Migration-owned | Provider | Provider `privacy.dsar.mail` is registered and documented. |
+| `mandates` | `govoplan-mandates` | Migration-owned | Provider | Provider `privacy.dsar.mandates` is registered and documented. |
+| `notifications` | `govoplan-notifications` | Migration-owned | Provider | Provider `privacy.dsar.notifications` is registered and documented. |
+| `ops` | `govoplan-ops` | No module migration | Reviewed no-store rationale | Projection-only module: Ops reads bounded platform and provider status; durable recovery evidence remains owned by Core and domain modules. |
+| `organizations` | `govoplan-organizations` | Migration-owned | Provider | Provider `privacy.dsar.organizations` is registered and documented. |
+| `parties` | `govoplan-parties` | Migration-owned | Provider | Provider `privacy.dsar.parties` is registered and documented. |
+| `payments` | `govoplan-payments` | Migration-owned | Provider | Provider `privacy.dsar.payments` is registered and documented. |
+| `permits` | `govoplan-permits` | No module migration | Reviewed no-store rationale | Contract-only module: permit applications, assessments, and decisions are not persisted; reassess before adding a migration-owned store. |
+| `policy` | `govoplan-policy` | Migration-owned | Provider | Provider `privacy.dsar.policy` is registered and documented. |
+| `poll` | `govoplan-poll` | Migration-owned | Provider | Provider `privacy.dsar.poll` is registered and documented. |
+| `portal` | `govoplan-portal` | No module migration | Reviewed no-store rationale | Projection-only module: Portal stores no applicant records; Services, Forms Runtime, Cases, and Postbox own and export authoritative subject data. |
+| `postbox` | `govoplan-postbox` | Migration-owned | Provider | Provider `privacy.dsar.postbox` is registered and documented. |
+| `procurement` | `govoplan-procurement` | No module migration | Reviewed no-store rationale | Contract-only module: procurement procedures, tenders, and awards are not persisted; reassess before adding a migration-owned store. |
+| `projects` | `govoplan-projects` | Migration-owned | Provider | Provider `privacy.dsar.projects` is registered and documented. |
+| `quick_access` | `govoplan-quick-access` | Migration-owned | Provider | Provider `privacy.dsar.quick_access` is registered and documented. |
+| `records` | `govoplan-records` | Migration-owned | Provider | Provider `privacy.dsar.records` is registered and documented. |
+| `reporting` | `govoplan-reporting` | Migration-owned | Provider | Provider `privacy.dsar.reporting` is registered and documented. |
+| `resources` | `govoplan-resources` | No module migration | Reviewed no-store rationale | Contract-only module: resource catalog and allocation persistence are not implemented; reassess before adding a migration-owned store. |
+| `rest` | `govoplan-rest` | No module migration | Reviewed no-store rationale | Transport-only module: REST binds explicitly published functions and owns no domain or subject-data store. |
+| `risk_compliance` | `govoplan-risk-compliance` | Migration-owned | Provider | Provider `privacy.dsar.risk_compliance` is registered and documented. |
+| `scheduling` | `govoplan-scheduling` | Migration-owned | Provider | Provider `privacy.dsar.scheduling` is registered and documented. |
+| `search` | `govoplan-search` | Migration-owned | Provider | Provider `privacy.dsar.search` is registered and documented. |
+| `services` | `govoplan-services` | Migration-owned | Provider | Provider `privacy.dsar.services` is registered and documented. |
+| `soap` | `govoplan-soap` | No module migration | Reviewed no-store rationale | Transport-only module: SOAP binds explicitly published operations and owns no domain or subject-data store. |
+| `tasks` | `govoplan-tasks` | Migration-owned | Provider | Provider `privacy.dsar.tasks` is registered and documented. |
+| `templates` | `govoplan-templates` | Migration-owned | Provider | Provider `privacy.dsar.templates` is registered and documented. |
+| `tenancy` | `govoplan-tenancy` | No module migration | Reviewed no-store rationale | Orchestration module: tenant lifecycle and settings use Core-owned storage; Access covers account and membership subject data. |
+| `tickets` | `govoplan-tickets` | No module migration | Reviewed no-store rationale | Contract-only module: ticket, queue, SLA, and adapter persistence are not implemented; reassess before adding a migration-owned store. |
+| `transparency` | `govoplan-transparency` | No module migration | Reviewed no-store rationale | Contract-only module: requests, disclosure reviews, and publications are not persisted; reassess before adding a migration-owned store. |
+| `views` | `govoplan-views` | Migration-owned | Provider | Provider `privacy.dsar.views` is registered and documented. |
+| `voting` | `govoplan-voting` | Migration-owned | Provider | Provider `privacy.dsar.voting` is registered and documented. |
+| `wiki` | `govoplan-wiki` | No module migration | Reviewed no-store rationale | Contract-only module: wiki spaces, pages, and revisions are not persisted; reassess before adding a migration-owned store. |
+| `workflow` | `govoplan-workflow` | No module migration | Reviewed no-store rationale | Presentation-only module: Workflow edits and projects Workflow Engine state; Workflow Engine owns persistence and DSAR coverage. |
+| `workflow_engine` | `govoplan-workflow-engine` | Migration-owned | Provider | Provider `privacy.dsar.workflow_engine` is registered and documented. |
+
+Provider search, export minimization, retention, and erasure behavior remains
+documented and tested by each owning module. This matrix verifies adoption and
+ownership coverage; Core continues to test disabled providers, partial failure,
+retry, authorization evidence, and horizontally coordinated execution.

tools/checks/check-dsar-coverage.py

@@ -0,0 +1,265 @@
+#!/usr/bin/env python3
+"""Require DSAR coverage or a reviewed no-store rationale for every module."""
+
+from __future__ import annotations
+
+import argparse
+import importlib
+import json
+import re
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+
+META_ROOT = Path(__file__).resolve().parents[2]
+EXEMPTIONS_PATH = Path(__file__).with_name("dsar-coverage-exemptions.json")
+REPORT_PATH = (
+    META_ROOT
+    / "docs"
+    / "evidence"
+    / "snapshots"
+    / "DSAR_PROVIDER_COVERAGE.generated.md"
+)
+MODULE_NAME_PATTERN = re.compile(r"[A-Za-z_][A-Za-z0-9_]*(?:\.[A-Za-z_][A-Za-z0-9_]*)*")
+REQUIRED_DOCUMENTATION_TYPES = frozenset({"admin"})
+
+
+@dataclass(frozen=True, slots=True)
+class CoverageRow:
+    module_id: str
+    repository: str
+    migration_owned: bool
+    capability: str | None
+    rationale: str
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--workspace-root",
+        type=Path,
+        default=None,
+        help="Directory containing GovOPlaN repositories.",
+    )
+    parser.add_argument(
+        "--render",
+        action="store_true",
+        help="Print the current matrix instead of comparing the checked-in report.",
+    )
+    args = parser.parse_args()
+
+    catalog = json.loads((META_ROOT / "repositories.json").read_text(encoding="utf-8"))
+    workspace_root = (args.workspace_root or Path(catalog["default_parent"])).resolve()
+    exemptions = _exemptions()
+    manifests, load_errors = _load_manifests(
+        workspace_root=workspace_root,
+        repositories=tuple(catalog["repositories"]),
+    )
+
+    errors = list(load_errors)
+    rows: list[CoverageRow] = []
+    manifest_ids = {manifest.id for _, manifest in manifests}
+    stale_exemptions = sorted(set(exemptions) - manifest_ids)
+    if stale_exemptions:
+        errors.append(
+            "DSAR coverage exemptions reference unknown modules: "
+            + ", ".join(stale_exemptions)
+        )
+
+    for repository, manifest in manifests:
+        expected = f"privacy.dsar.{manifest.id}"
+        provided = {
+            item.name
+            for item in manifest.provides_interfaces
+            if item.name.startswith("privacy.dsar.")
+        }
+        factories = {
+            name
+            for name in manifest.capability_factories
+            if name.startswith("privacy.dsar.")
+        }
+        migration_owned = manifest.migration_spec is not None
+        rationale = exemptions.get(manifest.id)
+
+        if provided != factories:
+            errors.append(
+                f"{repository}: DSAR interface/factory mismatch: "
+                f"interfaces={sorted(provided)!r}, factories={sorted(factories)!r}"
+            )
+        if provided and provided != {expected}:
+            errors.append(
+                f"{repository}: expected only {expected!r}, found {sorted(provided)!r}"
+            )
+
+        capability = (
+            expected if expected in provided and expected in factories else None
+        )
+        if migration_owned and capability is None:
+            errors.append(
+                f"{repository}: migration-owning module {manifest.id!r} must provide "
+                f"and register {expected!r}"
+            )
+        if migration_owned and rationale is not None:
+            errors.append(
+                f"{repository}: migration-owning module {manifest.id!r} cannot use a "
+                "no-store DSAR exemption"
+            )
+        if not migration_owned and capability is None and rationale is None:
+            errors.append(
+                f"{repository}: module {manifest.id!r} needs a DSAR provider or an "
+                "explicit reviewed no-store rationale"
+            )
+        if capability is not None and rationale is not None:
+            errors.append(
+                f"{repository}: module {manifest.id!r} has both DSAR coverage and a "
+                "stale exemption"
+            )
+        if capability is not None:
+            if capability not in manifest.capability_documentation:
+                errors.append(
+                    f"{repository}: {capability!r} lacks capability documentation"
+                )
+            matching_topics = tuple(
+                topic
+                for topic in manifest.documentation
+                if "data-subject-request" in topic.id
+            )
+            if not matching_topics or not any(
+                REQUIRED_DOCUMENTATION_TYPES.issubset(topic.documentation_types)
+                for topic in matching_topics
+            ):
+                errors.append(
+                    f"{repository}: DSAR coverage needs a static administrator "
+                    "data-subject-requests DocumentationTopic"
+                )
+
+        rows.append(
+            CoverageRow(
+                module_id=manifest.id,
+                repository=repository,
+                migration_owned=migration_owned,
+                capability=capability,
+                rationale=(
+                    f"Provider `{capability}` is registered and documented."
+                    if capability
+                    else rationale or "MISSING"
+                ),
+            )
+        )
+
+    report = _report(rows)
+    if args.render:
+        print(report, end="")
+    elif not REPORT_PATH.is_file():
+        errors.append(f"DSAR coverage report is missing: {REPORT_PATH}")
+    elif REPORT_PATH.read_text(encoding="utf-8") != report:
+        errors.append(
+            "DSAR coverage report is stale; review changes and replace it with "
+            "the output of tools/checks/check-dsar-coverage.py --render"
+        )
+
+    if errors:
+        print("\n".join(errors), file=sys.stderr)
+        return 1
+
+    provider_count = sum(row.capability is not None for row in rows)
+    print(
+        "DSAR coverage check passed: "
+        f"{provider_count} providers, {len(rows) - provider_count} reviewed "
+        f"no-store rationales, {len(rows)} active modules."
+    )
+    return 0
+
+
+def _exemptions() -> dict[str, str]:
+    values = json.loads(EXEMPTIONS_PATH.read_text(encoding="utf-8"))
+    if not isinstance(values, dict) or any(
+        not isinstance(key, str) or not isinstance(value, str) or not value.strip()
+        for key, value in values.items()
+    ):
+        raise ValueError("DSAR coverage exemptions must be non-empty string mappings.")
+    return {key: value.strip() for key, value in values.items()}
+
+
+def _load_manifests(*, workspace_root: Path, repositories: tuple[dict, ...]):
+    sources: list[Path] = []
+    candidates: list[tuple[str, Path, Path]] = []
+    for repository in repositories:
+        source = workspace_root / repository["path"] / "src"
+        if not source.is_dir():
+            continue
+        sources.append(source)
+        candidates.extend(
+            (repository["name"], source, path)
+            for path in sorted(source.glob("*/backend/manifest.py"))
+        )
+    core_source = workspace_root / "govoplan-core" / "src"
+    sys.path[:0] = [
+        str(core_source),
+        *(str(source) for source in sources if source != core_source),
+    ]
+
+    manifests = []
+    errors = []
+    for repository, source, path in candidates:
+        module_name = ".".join(path.relative_to(source).with_suffix("").parts)
+        if MODULE_NAME_PATTERN.fullmatch(module_name) is None:
+            errors.append(f"{repository}: unsafe manifest module name {module_name!r}")
+            continue
+        try:
+            module = importlib.import_module(module_name)
+            manifests.append((repository, module.get_manifest()))
+        except Exception as exc:  # pragma: no cover - emitted as check evidence
+            errors.append(f"{repository}: could not load {module_name}: {exc}")
+    return manifests, errors
+
+
+def _report(rows: list[CoverageRow]) -> str:
+    ordered = sorted(rows, key=lambda row: row.module_id)
+    providers = sum(row.capability is not None for row in ordered)
+    lines = [
+        "# DSAR Provider Coverage",
+        "",
+        "This generated matrix is enforced by `tools/checks/check-dsar-coverage.py`.",
+        "A migration-owning module must register and document its canonical DSAR provider.",
+        "Every other active module requires a reviewed explanation of why it owns no",
+        "persistent subject-data store. Adding a migration invalidates that explanation.",
+        "",
+        f"- Active modules: {len(ordered)}",
+        f"- Registered and documented DSAR providers: {providers}",
+        f"- Reviewed no-store rationales: {len(ordered) - providers}",
+        "- Unexplained coverage gaps: 0",
+        "",
+        "| Module | Repository | Persistence | Coverage | Rationale |",
+        "| --- | --- | --- | --- | --- |",
+    ]
+    for row in ordered:
+        lines.append(
+            "| "
+            + " | ".join(
+                (
+                    f"`{row.module_id}`",
+                    f"`{row.repository}`",
+                    "Migration-owned" if row.migration_owned else "No module migration",
+                    "Provider" if row.capability else "Reviewed no-store rationale",
+                    row.rationale.replace("|", "\\|"),
+                )
+            )
+            + " |"
+        )
+    lines.extend(
+        (
+            "",
+            "Provider search, export minimization, retention, and erasure behavior remains",
+            "documented and tested by each owning module. This matrix verifies adoption and",
+            "ownership coverage; Core continues to test disabled providers, partial failure,",
+            "retry, authorization evidence, and horizontally coordinated execution.",
+            "",
+        )
+    )
+    return "\n".join(lines)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

tools/checks/check-focused.sh

@@ -38,6 +38,7 @@ cd "$ROOT"
 GOVOPLAN_CORE_ROOT="$ROOT" PYTHON="$PYTHON" CHECK_TESTCLIENT_DEPRECATIONS=1 bash "$META_ROOT/tools/checks/check-dependency-hygiene.sh"
 "$PYTHON" "$META_ROOT/tools/checks/check-contracts.py" --no-impact
 PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-manifest-shapes.py" --require-architecture
+PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-dsar-coverage.py"
 
 cd "$META_ROOT"
 "$PYTHON" tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints

tools/checks/dsar-coverage-exemptions.json

@@ -0,0 +1,26 @@
+{
+  "assets": "Contract-only module: asset persistence and lifecycle APIs are not implemented; reassess before adding a migration-owned store.",
+  "booking": "Contract-only module: booking persistence and reservation workflows are not implemented; reassess before adding a migration-owned store.",
+  "certificates": "Contract-only module: certificate issuance and revocation persistence are not implemented; reassess before adding a migration-owned store.",
+  "consultation": "Contract-only module: consultation submissions and evaluation persistence are not implemented; reassess before adding a migration-owned store.",
+  "contracts": "Contract-only module: contract, amendment, and obligation persistence are not implemented; reassess before adding a migration-owned store.",
+  "docs": "Projection-only module: documentation is contributed by owning manifests and providers; Docs has no migration-owned subject store.",
+  "evaluation": "Contract-only module: evaluation runs, responses, and scores are not persisted; reassess before adding a migration-owned store.",
+  "facilities": "Contract-only module: facility and maintenance persistence are not implemented; reassess before adding a migration-owned store.",
+  "grants": "Contract-only module: grant applications, awards, and monitoring are not persisted; reassess before adding a migration-owned store.",
+  "helpdesk": "Contract-only module: helpdesk requests, queues, and resolutions are not persisted; reassess before adding a migration-owned store.",
+  "inspections": "Contract-only module: inspections, findings, and measures are not persisted; reassess before adding a migration-owned store.",
+  "learning": "Contract-only module: learning offers, enrollment, and completion are not persisted; reassess before adding a migration-owned store.",
+  "ops": "Projection-only module: Ops reads bounded platform and provider status; durable recovery evidence remains owned by Core and domain modules.",
+  "permits": "Contract-only module: permit applications, assessments, and decisions are not persisted; reassess before adding a migration-owned store.",
+  "portal": "Projection-only module: Portal stores no applicant records; Services, Forms Runtime, Cases, and Postbox own and export authoritative subject data.",
+  "procurement": "Contract-only module: procurement procedures, tenders, and awards are not persisted; reassess before adding a migration-owned store.",
+  "resources": "Contract-only module: resource catalog and allocation persistence are not implemented; reassess before adding a migration-owned store.",
+  "rest": "Transport-only module: REST binds explicitly published functions and owns no domain or subject-data store.",
+  "soap": "Transport-only module: SOAP binds explicitly published operations and owns no domain or subject-data store.",
+  "tenancy": "Orchestration module: tenant lifecycle and settings use Core-owned storage; Access covers account and membership subject data.",
+  "tickets": "Contract-only module: ticket, queue, SLA, and adapter persistence are not implemented; reassess before adding a migration-owned store.",
+  "transparency": "Contract-only module: requests, disclosure reviews, and publications are not persisted; reassess before adding a migration-owned store.",
+  "wiki": "Contract-only module: wiki spaces, pages, and revisions are not persisted; reassess before adding a migration-owned store.",
+  "workflow": "Presentation-only module: Workflow edits and projects Workflow Engine state; Workflow Engine owns persistence and DSAR coverage."
+}

tools/release/push-release-tag.sh

@@ -571,6 +571,10 @@ run_manifest_shape_gate() {
     "$META_ROOT/tools/checks/check-manifest-shapes.py" \
     --workspace-root "$PARENT" \
     --require-architecture
+  run env PYTHONDONTWRITEBYTECODE=1 \
+    "$PYTHON" \
+    "$META_ROOT/tools/checks/check-dsar-coverage.py" \
+    --workspace-root "$PARENT"
 }
 
 run_migration_release_audit() {