diff --git a/docs/REFERENCE_JOURNEY_PROGRAM.md b/docs/REFERENCE_JOURNEY_PROGRAM.md index bb3aa52..5f686dd 100644 --- a/docs/REFERENCE_JOURNEY_PROGRAM.md +++ b/docs/REFERENCE_JOURNEY_PROGRAM.md @@ -42,11 +42,13 @@ does neither stays in the backlog until a concrete consumer exists. The Records vertical now supplies the journey's native file plan, immutable record and item revisions, chronology, close/reopen, retention calculation, holds, appraisal, independent disposition approval, recovery-ledger evidence, -and archive-neutral package simulation. The executable Records fixture proves -those native transitions without claiming archival custody. The journey still -needs Forms Runtime/Decisions filing, resumable user handoffs, search -reconstruction in the pinned composition, and one target-tested archive -profile. +and archive-neutral package simulation. Forms Runtime, Cases, and Decisions +expose exact, permission-rechecked source revisions for explicit filing, and +all three contribute metadata-only native Search projections that can be +rebuilt from authoritative state. The executable fixtures prove those native +transitions without claiming archival custody. The journey still needs +resumable user handoffs, pinned-composition reconstruction evidence, and one +target-tested archive profile. ## Why this sequence diff --git a/docs/STRATEGY_STATUS.md b/docs/STRATEGY_STATUS.md index c139b3a..c7d0fa4 100644 --- a/docs/STRATEGY_STATUS.md +++ b/docs/STRATEGY_STATUS.md @@ -4,7 +4,7 @@ | Field | Value | | --- | --- | -| Reconciled on | 2026-08-05 | +| Reconciled on | 2026-08-06 | | Source scope | Local workspace manifests, source inventory, focused journey checks, signed release evidence, and live Gitea issue state | | Stable direction | [Platform Core Ideas](PLATFORM_CORE_IDEAS.md) and [Connected Governance Platform Roadmap](CONNECTED_GOVERNANCE_PLATFORM_ROADMAP.md) | | Delivery source | Gitea issues | @@ -67,8 +67,12 @@ prevent a reference-ready claim. Services, Forms, Forms Runtime, Cases, Parties, Mandates, Approvals, Committee, Voting, Decisions, Portal, Postbox, and Audit have an executable service-to- -decision fixture. Browser-complete assisted intake, production identity, -records, delivery, and target evidence remain. +decision fixture. Public and invitation intake can retain Files-backed +evidence; Forms submissions, Cases, and formal Decisions can be explicitly +filed as exact eAkte source revisions and reconstructed through permission- +rechecked native Search projections. Browser-complete assisted intake, +resumable work guidance, production identity and delivery, a named archive +profile, and target evidence remain. ### Governed data and assurance @@ -97,9 +101,11 @@ reconciliation, publication/export, and guided handoff evidence. 2. Complete governed communication and Postbox against a named target. 3. Complete the monthly-data flow and use it as the data foundation for sanctions screening. -4. Complete one digital and assisted service-to-decision journey with an eAkte. -5. Add native PostgreSQL search coverage for the objects used by those - journeys; keep OpenSearch optional. +4. Complete the browser and resumable-work proof for the digital and assisted + service-to-decision journey with its existing exact eAkte filing contracts. +5. Extend native PostgreSQL search coverage from the implemented service-to- + decision objects to the communication and monthly-data journeys; keep + OpenSearch optional. 6. Prove one external product connector and one GovOPlaN federation exchange. 7. Finish multi-host, restore, provider, accessibility, and independent signed target evidence before increasing maturity claims. diff --git a/packages/product/governed-data-assurance/README.md b/packages/product/governed-data-assurance/README.md index 0f39894..d122ce4 100644 --- a/packages/product/governed-data-assurance/README.md +++ b/packages/product/governed-data-assurance/README.md @@ -12,7 +12,7 @@ review. 1. Register a typed datasource with source authority, purpose, classification, owner, freshness, and correction policy. 2. Acquire or upload an immutable source state. -3. execute a versioned flow and retain intermediate materializations and +3. Execute a versioned flow and retain intermediate materializations and provenance; 4. publish a report or decision input against exact source and flow revisions; 5. link obligation, governed object, risk, control, evidence, finding, @@ -39,3 +39,24 @@ The artifact remains a `product` package. Promotion to `reference` requires: Optional Connectors, Files, Notifications, and Workflow Engine integrations must remain capability-based and absence-safe. + +## Executable evidence + +- `tools/checks/check-datasource-composition.py` composes Connector snapshots, + governed Datasources, queued Dataflow execution, frozen publication, + idempotent replay, and recovery evidence. +- `govoplan-dataflow/fixtures/golden/monthly-reconciliation` pins synthetic + monthly inputs, stable reconciliation hashes, reviewed decisions, expected + output, source fingerprints, and output hashes. +- `tools/checks/check-sanctions-screening-composition.py` composes an immutable + Connector acquisition, idempotent Risk Compliance import and screening, + independent disposition, a cleared gate, changed-source invalidation, and + the rescreening queue through the registered versioned capabilities. +- `govoplan-dataflow/fixtures/golden/sanctions-screening` independently proves + the deterministic normalization and matching graph with exact expected + output. + +These checks use synthetic data and run without network access. They prove the +module contracts and durable state transitions; they do not replace the +deployment, security, privacy, accessibility, and operator evidence still +listed above. diff --git a/packages/product/governed-data-assurance/package.json b/packages/product/governed-data-assurance/package.json index 9e7a57f..e81d095 100644 --- a/packages/product/governed-data-assurance/package.json +++ b/packages/product/governed-data-assurance/package.json @@ -28,7 +28,17 @@ "kind": "documentation", "reference": "packages/product/governed-data-assurance/README.md", "summary": "Defines the package boundary, provenance chain, and reference-readiness gates." + }, + { + "kind": "target_test", + "reference": "tools/checks/check-datasource-composition.py", + "summary": "Proves governed Connector acquisition, Datasource registration, queued Dataflow execution, frozen publication, idempotency, and recovery evidence." + }, + { + "kind": "target_test", + "reference": "tools/checks/check-sanctions-screening-composition.py", + "summary": "Proves immutable sanctions acquisition, import, screening replay, independent review, freshness gates, and rescreening across module capabilities." } ], - "tags": ["datasources", "dataflow", "reporting", "assurance"] + "tags": ["datasources", "dataflow", "reporting", "sanctions", "assurance"] } diff --git a/packages/product/service-to-decision/README.md b/packages/product/service-to-decision/README.md index c8563e6..2956aec 100644 --- a/packages/product/service-to-decision/README.md +++ b/packages/product/service-to-decision/README.md @@ -24,19 +24,31 @@ grant cross-module table access and can omit optional presentation, work, deliberation, delivery, or records modules while retaining explicit references to externally performed steps. +When Records is present, Forms Runtime, Cases, and Decisions expose exact, +digest-bound source snapshots for explicit filing. The source module rechecks +current access, Records chooses the destination and preserves chronology, and +the filed reference never becomes an editable copy. When Search is present, +the same three owners contribute rebuildable metadata-only projections. Form +values, evidence payloads, Decision reasoning, operative results, and +conditions are excluded; every candidate is authorized again before it is +shown. + ## Security And Recovery Every provider is tenant-bound. Missing or conflicting authority fails closed. Protected Decision content has a separate permission. Writes are replay-safe and OCC-guarded. Database restore is the semantic-state recovery unit; file and communication effects remain governed by their owning providers and are linked -through requested/observed effect, evidence, and audit references. +through requested/observed effect, evidence, and audit references. Search is a +derived recovery unit and can be rebuilt from authoritative module state. The executable fixture in `tests/test_institutional_governance_journey.py` proves SQL-backed Service, Case, Party, Mandate, Committee meeting/agendum/vote/minute, and Decision state. `tests/test_institutional_service_journey.py` separately proves exact Portal Form launch, persisted submission provenance, and idempotent replay. -Target-environment accessibility, security, operator, privacy, -delivery-provider, and recovery evidence are still required before this product -package may claim `reference_ready` maturity. +Module-level Records source tests prove exact Form submission, Case revision, +and Decision revision filing. Target-environment browser accessibility, +resumable work guidance, production identity and delivery, a named archive +profile, and recovery evidence are still required before this product package +may claim `reference_ready` maturity. diff --git a/packages/product/service-to-decision/package.json b/packages/product/service-to-decision/package.json index 35ccc9b..44ba308 100644 --- a/packages/product/service-to-decision/package.json +++ b/packages/product/service-to-decision/package.json @@ -35,6 +35,7 @@ {"module_id": "forms_runtime"}, {"module_id": "postbox"}, {"module_id": "records"}, + {"module_id": "search"}, {"module_id": "tasks"}, {"module_id": "workflow_engine"} ], diff --git a/tools/checks/check-focused.sh b/tools/checks/check-focused.sh index 381add2..f21c09e 100644 --- a/tools/checks/check-focused.sh +++ b/tools/checks/check-focused.sh @@ -109,6 +109,7 @@ PY "$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-encryption/tests "$PYTHON" -m pytest -q /mnt/DATA/git/govoplan-campaign/tests/test_approval_gate.py "$PYTHON" "$META_ROOT/tools/checks/check-datasource-composition.py" +"$PYTHON" "$META_ROOT/tools/checks/check-sanctions-screening-composition.py" "$PYTHON" -m unittest discover -s /mnt/DATA/git/govoplan-mail/tests "$PYTHON" -m unittest tests.test_api_smoke.ApiSmokeTests.test_mailbox_message_listing_reports_total_count diff --git a/tools/checks/check-sanctions-screening-composition.py b/tools/checks/check-sanctions-screening-composition.py new file mode 100644 index 0000000..93d6a69 --- /dev/null +++ b/tools/checks/check-sanctions-screening-composition.py @@ -0,0 +1,306 @@ +#!/usr/bin/env python3 +"""Prove the governed Connectors -> Risk Compliance sanctions journey.""" + +from __future__ import annotations + +from types import SimpleNamespace + +from sqlalchemy import create_engine +from sqlalchemy.orm import Session + +from govoplan_connectors.backend.db.models import ( + ConnectorSanctionsAcquisitionRun, + ConnectorSanctionsSnapshot, +) +from govoplan_connectors.backend.sanctions_sources import ( + SANCTIONS_READ_SCOPE as CONNECTOR_SANCTIONS_READ_SCOPE, + SANCTIONS_REFRESH_SCOPE, + SYNTHETIC_PROVIDER_ID, +) +from govoplan_core.auth import ApiPrincipal +from govoplan_core.core.access import PrincipalRef +from govoplan_core.core.modules import ModuleContext +from govoplan_core.core.recovery import RecoveryCheckpoint, RecoveryOperation +from govoplan_core.core.runtime_coordination import ( + DistributedLease, + RuntimeIdentity, + bind_process_runtime_identity, +) +from govoplan_core.core.sanctions import ( + SanctionsScreeningFreshnessRequest, + SanctionsScreeningPolicy, + SanctionsScreeningRequest, + SanctionsScreeningSubject, + sanctions_screening_provider, + sanctions_snapshot_provider, +) +from govoplan_core.db.base import Base +from govoplan_core.server.registry import build_platform_registry +from govoplan_risk_compliance.backend.db.models import ( + RiskAssuranceEdge, + RiskAssuranceNode, + RiskSanctionsAddress, + RiskSanctionsAlias, + RiskSanctionsDate, + RiskSanctionsEntry, + RiskSanctionsIdentifier, + RiskSanctionsListSnapshot, + RiskScreeningCandidate, + RiskScreeningDisposition, + RiskScreeningException, + RiskScreeningRun, + RiskScreeningSubjectSnapshot, +) +from govoplan_risk_compliance.backend.permissions import ( + SANCTIONS_ADMIN_SCOPE, + SANCTIONS_READ_SCOPE, + SANCTIONS_REVIEW_SCOPE, + SANCTIONS_SCREEN_SCOPE, +) +from govoplan_risk_compliance.backend.review import ( + DispositionInput, + record_disposition, +) +from govoplan_risk_compliance.backend.sanctions_catalog import ( + import_connector_snapshot, +) +from govoplan_risk_compliance.backend.screening import ( + get_screening_run, + list_rescreening_requirements, +) + + +TABLES = ( + DistributedLease.__table__, + RecoveryOperation.__table__, + RecoveryCheckpoint.__table__, + ConnectorSanctionsAcquisitionRun.__table__, + ConnectorSanctionsSnapshot.__table__, + RiskAssuranceNode.__table__, + RiskAssuranceEdge.__table__, + RiskSanctionsListSnapshot.__table__, + RiskSanctionsEntry.__table__, + RiskSanctionsAlias.__table__, + RiskSanctionsIdentifier.__table__, + RiskSanctionsDate.__table__, + RiskSanctionsAddress.__table__, + RiskScreeningSubjectSnapshot.__table__, + RiskScreeningRun.__table__, + RiskScreeningCandidate.__table__, + RiskScreeningDisposition.__table__, + RiskScreeningException.__table__, +) + + +def main() -> int: + registry = build_platform_registry(("connectors", "risk_compliance")) + registry.configure_capability_context( + ModuleContext(registry=registry, settings=object()) + ) + snapshot_provider = sanctions_snapshot_provider(registry) + screening_provider = sanctions_screening_provider(registry) + refresh_source = getattr(snapshot_provider, "refresh_source", None) + if snapshot_provider is None or not callable(refresh_source): + raise RuntimeError("Connectors sanctions acquisition is unavailable.") + if screening_provider is None: + raise RuntimeError("Risk Compliance sanctions screening is unavailable.") + + engine = create_engine("sqlite:///:memory:") + Base.metadata.create_all(engine, tables=TABLES) + bind_process_runtime_identity( + RuntimeIdentity( + installation_id="sanctions-composition-check", + node_id="sanctions-worker", + incarnation="sanctions-worker-incarnation", + role="worker", + software_version="test", + composition_hash="d" * 64, + ) + ) + try: + with Session(engine) as session: + operator = _principal("operator-1", operational=True) + reviewer = _principal("reviewer-1", operational=False) + + acquired = refresh_source( + session, + operator, + provider_id=SYNTHETIC_PROVIDER_ID, + idempotency_key="synthetic-sanctions-2026-08-01", + ) + replay = refresh_source( + session, + operator, + provider_id=SYNTHETIC_PROVIDER_ID, + idempotency_key="synthetic-sanctions-2026-08-01", + ) + if acquired.status != "succeeded" or acquired.snapshot is None: + raise RuntimeError(f"Synthetic acquisition failed: {acquired!r}") + if ( + replay.run_id != acquired.run_id + or replay.snapshot is None + or replay.snapshot.ref != acquired.snapshot.ref + or replay.snapshot.sha256 != acquired.snapshot.sha256 + ): + raise RuntimeError("Acquisition idempotency did not replay exact evidence.") + + imported, created = import_connector_snapshot( + session, + operator, + registry=registry, + connector_snapshot_ref=acquired.snapshot.ref, + ) + imported_replay, replay_created = import_connector_snapshot( + session, + operator, + registry=registry, + connector_snapshot_ref=acquired.snapshot.ref, + ) + if not created or replay_created or imported_replay.id != imported.id: + raise RuntimeError("Risk Compliance snapshot import is not idempotent.") + + subject = SanctionsScreeningSubject( + subject_type="person", + primary_name="Alex Example", + subject_ref="party:fixture-person-1", + ) + policy = SanctionsScreeningPolicy(failure_policy="block") + request = SanctionsScreeningRequest( + list_snapshot_id=imported.id, + idempotency_key="fixture-party-screening-1", + subject=subject, + policy=policy, + ) + screened = screening_provider.request_screening( + session, + operator, + request, + ) + screened_replay = screening_provider.request_screening( + session, + operator, + request, + ) + if not screened.created or screened_replay.created: + raise RuntimeError("Screening request idempotency is not stable.") + if screened.evidence.ref != screened_replay.evidence.ref: + raise RuntimeError("Screening replay returned different evidence.") + if screened.evidence.outcome != "potential" or screened.evidence.candidate_count != 1: + raise RuntimeError(f"Synthetic match was not reviewable: {screened.evidence!r}") + + run = get_screening_run( + session, + operator, + run_id=screened.evidence.run_id, + ) + candidate, disposition = record_disposition( + session, + reviewer, + candidate_id=run.candidates[0].id, + disposition=DispositionInput( + decision="false_positive", + reason="Independent fixture evidence excludes the screened party.", + evidence_refs=(acquired.snapshot.raw_evidence_ref,), + ), + ) + if candidate.review_status != "false_positive": + raise RuntimeError("Independent review did not resolve the candidate.") + if disposition.separation_status != "independent": + raise RuntimeError("Reviewer separation evidence was not retained.") + + cleared = screening_provider.check_freshness( + session, + operator, + SanctionsScreeningFreshnessRequest( + evidence_ref=screened.evidence.ref, + current_subject=subject, + expected_list_snapshot_id=imported.id, + policy=policy, + ), + ) + if not cleared.fresh or cleared.gate_decision != "allow": + raise RuntimeError(f"Reviewed evidence did not clear the gate: {cleared!r}") + + # Acquisition and review are separate durable commands in production. + session.commit() + session.expire_all() + + refreshed = refresh_source( + session, + operator, + provider_id=SYNTHETIC_PROVIDER_ID, + idempotency_key="synthetic-sanctions-2026-08-02", + ) + if refreshed.snapshot is None or refreshed.snapshot.ref == acquired.snapshot.ref: + raise RuntimeError("A new acquisition did not create new immutable evidence.") + current, current_created = import_connector_snapshot( + session, + operator, + registry=registry, + connector_snapshot_ref=refreshed.snapshot.ref, + ) + if not current_created: + raise RuntimeError("The refreshed list state was not imported separately.") + + stale = screening_provider.check_freshness( + session, + operator, + SanctionsScreeningFreshnessRequest( + evidence_ref=screened.evidence.ref, + current_subject=subject, + expected_list_snapshot_id=current.id, + policy=policy, + ), + ) + if stale.fresh or stale.gate_decision != "block": + raise RuntimeError(f"Changed source evidence did not close the gate: {stale!r}") + if "source_snapshot_changed" not in stale.reasons: + raise RuntimeError("Source change provenance was not reported.") + requirements = list_rescreening_requirements(session, operator) + if screened.evidence.run_id not in {item.run.id for item in requirements}: + raise RuntimeError("The stale screening is absent from the rescreening queue.") + + session.commit() + if session.query(RecoveryOperation).count() != 2: + raise RuntimeError("Connector acquisition recovery evidence is incomplete.") + if session.query(RiskScreeningDisposition).count() != 1: + raise RuntimeError("Disposition evidence was duplicated or lost.") + finally: + bind_process_runtime_identity(None) + engine.dispose() + + print( + "Connectors -> immutable sanctions snapshot -> Risk Compliance review " + "and rescreening composition passed." + ) + return 0 + + +def _principal(account_id: str, *, operational: bool) -> ApiPrincipal: + scopes = { + SANCTIONS_READ_SCOPE, + SANCTIONS_REVIEW_SCOPE, + } + if operational: + scopes.update( + { + CONNECTOR_SANCTIONS_READ_SCOPE, + SANCTIONS_REFRESH_SCOPE, + SANCTIONS_ADMIN_SCOPE, + SANCTIONS_SCREEN_SCOPE, + } + ) + return ApiPrincipal( + principal=PrincipalRef( + account_id=account_id, + membership_id=f"membership-{account_id}", + tenant_id="tenant-1", + scopes=frozenset(scopes), + ), + account=SimpleNamespace(id=account_id), + user=SimpleNamespace(id=f"membership-{account_id}"), + ) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/inventory/endpoint-surface-declarations.json b/tools/inventory/endpoint-surface-declarations.json index 1934ce9..d922b95 100644 --- a/tools/inventory/endpoint-surface-declarations.json +++ b/tools/inventory/endpoint-surface-declarations.json @@ -798,6 +798,13 @@ "rationale": "The vault action group rotates the current key with policy, assurance, reason, and revision evidence.", "repository": "govoplan-encryption" }, + { + "category": "public_integration", + "method": "POST", + "path": "/files/form-evidence/upload", + "rationale": "A short-lived purpose-bound bearer grant lets the public Forms Runtime surface stream one attachment directly to Files without granting general Files access.", + "repository": "govoplan-files" + }, { "category": "ui_reachable", "method": "POST",