feat: declare governed external provider state
This commit is contained in:
@@ -0,0 +1,16 @@
|
|||||||
|
# GovOPlaN Connectors Codex Guide
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This repository owns reusable external connection profiles, protocol adapters, governed snapshots, and connector capability contracts.
|
||||||
|
|
||||||
|
## Documentation Contract
|
||||||
|
|
||||||
|
- Treat documentation as part of every behavior change. Update this module's manifest-driven `DocumentationTopic` contributions for affected user and administrator behavior.
|
||||||
|
- Keep feature content here; `govoplan-docs` projects it without importing Connectors internals.
|
||||||
|
- Maintain a static user/admin baseline and run `/mnt/DATA/git/govoplan/tools/checks/check-manifest-shapes.py` after behavior or manifest changes.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Domain modules own business semantics; connectors own transport, credentials, synchronization, and diagnostics.
|
||||||
|
- Keep optional adapters behind capabilities and enforce egress and peer-validation policy.
|
||||||
@@ -11,6 +11,13 @@ results, credential references, and generic integration events. Protocol-heavy
|
|||||||
or domain-heavy integrations may live in dedicated modules once their scope is
|
or domain-heavy integrations may live in dedicated modules once their scope is
|
||||||
clear.
|
clear.
|
||||||
|
|
||||||
|
Connector capability is not source ownership. Each configured binding also
|
||||||
|
declares whether GovOPlaN is authoritative, the external system is
|
||||||
|
authoritative, GovOPlaN keeps a mirror, both sides use governed synchronization,
|
||||||
|
GovOPlaN supplies only a governance overlay, or the object is link-only. The
|
||||||
|
same connector may be configured differently by tenant, service, object type,
|
||||||
|
or field group.
|
||||||
|
|
||||||
Detailed follow-up documents:
|
Detailed follow-up documents:
|
||||||
|
|
||||||
- [Public-sector integration catalogue](PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md)
|
- [Public-sector integration catalogue](PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md)
|
||||||
@@ -29,6 +36,8 @@ The module owns:
|
|||||||
- connector health status and last-test evidence
|
- connector health status and last-test evidence
|
||||||
- operator-visible integration inventory
|
- operator-visible integration inventory
|
||||||
- cross-module discovery of available external capabilities
|
- cross-module discovery of available external capabilities
|
||||||
|
- supported integration maturity, source-authority modes, operation limits,
|
||||||
|
and effect/reconciliation behavior for each connector type
|
||||||
|
|
||||||
The module does not own:
|
The module does not own:
|
||||||
|
|
||||||
@@ -61,6 +70,9 @@ The module should integrate through:
|
|||||||
- module manifest metadata, route factories, permissions, and migrations
|
- module manifest metadata, route factories, permissions, and migrations
|
||||||
- a connector catalogue API for listing available connector types
|
- a connector catalogue API for listing available connector types
|
||||||
- a connection profile API with secret references, not plaintext secrets
|
- a connection profile API with secret references, not plaintext secrets
|
||||||
|
- a provider declaration that composes authority mode, maturity, supported
|
||||||
|
operations, revisions/freshness, health, limits, idempotency, conflicts,
|
||||||
|
evidence, and reconciliation behavior
|
||||||
- capability declarations such as `connectors.catalog`,
|
- capability declarations such as `connectors.catalog`,
|
||||||
`connectors.profileTester`, and `connectors.health`
|
`connectors.profileTester`, and `connectors.health`
|
||||||
- events such as `connector.profile_created`, `connector.test_succeeded`,
|
- events such as `connector.profile_created`, `connector.test_succeeded`,
|
||||||
@@ -113,6 +125,7 @@ The first implementation should provide:
|
|||||||
- WebUI catalogue and profile pages
|
- WebUI catalogue and profile pages
|
||||||
- configuration-package fragment support
|
- configuration-package fragment support
|
||||||
- generic external-reference DTOs
|
- generic external-reference DTOs
|
||||||
|
- source-authority binding and provider-operation metadata
|
||||||
- health summary provider
|
- health summary provider
|
||||||
|
|
||||||
## Permissions
|
## Permissions
|
||||||
|
|||||||
@@ -14,6 +14,22 @@ predictable and avoids hidden module imports.
|
|||||||
- `bidirectional`: GovOPlaN supports both directions with conflict detection and
|
- `bidirectional`: GovOPlaN supports both directions with conflict detection and
|
||||||
reconciliation rules.
|
reconciliation rules.
|
||||||
|
|
||||||
|
Direction describes transport. Every binding also needs a source-authority
|
||||||
|
mode:
|
||||||
|
|
||||||
|
- `native_authoritative`
|
||||||
|
- `external_authoritative`
|
||||||
|
- `external_mirror`
|
||||||
|
- `governed_sync`
|
||||||
|
- `governance_overlay`
|
||||||
|
- `linked_reference`
|
||||||
|
|
||||||
|
The authority mode and the connector's integration maturity are orthogonal. A
|
||||||
|
bidirectional connector may be configured as an external mirror, and a native
|
||||||
|
GovOPlaN object may publish to an external target without transferring
|
||||||
|
authority. The effective binding must identify its scope and provenance rather
|
||||||
|
than relying on a profile-wide `sync` boolean.
|
||||||
|
|
||||||
## Source Data Lifecycle
|
## Source Data Lifecycle
|
||||||
|
|
||||||
Connector profiles have operational states, while individual external records
|
Connector profiles have operational states, while individual external records
|
||||||
@@ -103,8 +119,11 @@ this lifecycle when a connector publishes status.
|
|||||||
shape.
|
shape.
|
||||||
3. Connector sends the remote request.
|
3. Connector sends the remote request.
|
||||||
4. Connector stores the remote id, version/ETag, and response diagnostics.
|
4. Connector stores the remote id, version/ETag, and response diagnostics.
|
||||||
5. Connector emits `connector.record_published` or `connector.publish_failed`.
|
5. A timeout or lost acknowledgement after dispatch becomes outcome-unknown,
|
||||||
6. Domain module stores only the external-reference DTO and any domain result.
|
not an ordinary failure or permission to duplicate the command.
|
||||||
|
6. Connector emits a confirmed, retryable, outcome-unknown, reconciled, or
|
||||||
|
corrected result event.
|
||||||
|
7. Domain module stores only the external-reference DTO and any domain result.
|
||||||
|
|
||||||
## Reconciliation
|
## Reconciliation
|
||||||
|
|
||||||
@@ -116,6 +135,25 @@ Every connector that writes to an external system needs a reconciliation story:
|
|||||||
- retry policy for temporary failures
|
- retry policy for temporary failures
|
||||||
- explicit operator action for destructive overwrite or deletion
|
- explicit operator action for destructive overwrite or deletion
|
||||||
- audit trace from GovOPlaN record to external request and response summary
|
- audit trace from GovOPlaN record to external request and response summary
|
||||||
|
- explicit requested, approved, dispatched, possibly-executed, confirmed, and
|
||||||
|
reconciled/corrected effect states
|
||||||
|
|
||||||
|
## Provider Declaration
|
||||||
|
|
||||||
|
An executable connector type should publish machine-readable metadata for:
|
||||||
|
|
||||||
|
- owned object and field groups, plus supported authority modes;
|
||||||
|
- supported discovery, link, search, read, publish, synchronize, migrate, and
|
||||||
|
replacement maturity;
|
||||||
|
- read/write/delete/preview/dry-run operations and bounded response limits;
|
||||||
|
- revision/concurrency token, freshness, health, timeout, retry, and conflict
|
||||||
|
semantics;
|
||||||
|
- idempotency and outcome-unknown handling;
|
||||||
|
- evidence, rollback/compensation, correction, and reconciliation paths;
|
||||||
|
- classification, purpose, retention, secret, degraded, and outage behavior.
|
||||||
|
|
||||||
|
This declaration composes Core contracts. It does not move protocol behavior
|
||||||
|
or domain semantics into Core or Connectors.
|
||||||
|
|
||||||
## Capability Boundary
|
## Capability Boundary
|
||||||
|
|
||||||
@@ -158,5 +196,6 @@ Before shipping an executable connector type:
|
|||||||
- Add unavailable-optional-module tests for every consuming domain module.
|
- Add unavailable-optional-module tests for every consuming domain module.
|
||||||
- Add profile test and health status fixtures.
|
- Add profile test and health status fixtures.
|
||||||
- Add external-reference DTO tests.
|
- Add external-reference DTO tests.
|
||||||
|
- Add source-authority and provider-declaration validation tests.
|
||||||
- Add lifecycle transition tests for pause, retry, retirement, and uninstall
|
- Add lifecycle transition tests for pause, retry, retirement, and uninstall
|
||||||
guard behavior.
|
guard behavior.
|
||||||
|
|||||||
@@ -20,6 +20,15 @@ from govoplan_core.core.modules import (
|
|||||||
PermissionDefinition,
|
PermissionDefinition,
|
||||||
RoleTemplate,
|
RoleTemplate,
|
||||||
)
|
)
|
||||||
|
from govoplan_core.core.provider_governance import (
|
||||||
|
ExternalProviderDeclaration,
|
||||||
|
ExternalProviderStateProviderRegistration,
|
||||||
|
ModuleArchitectureDeclaration,
|
||||||
|
ModuleArchitectureDocumentation,
|
||||||
|
ModuleMaturityEvidence,
|
||||||
|
ProviderBehaviorDeclaration,
|
||||||
|
ProviderObjectDeclaration,
|
||||||
|
)
|
||||||
from govoplan_core.core.tabular_sources import (
|
from govoplan_core.core.tabular_sources import (
|
||||||
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER,
|
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER,
|
||||||
CAPABILITY_CONNECTORS_TABULAR_SOURCES,
|
CAPABILITY_CONNECTORS_TABULAR_SOURCES,
|
||||||
@@ -48,6 +57,12 @@ from govoplan_connectors.backend.datasource_origins import (
|
|||||||
ConnectorDatasourceOriginProvider,
|
ConnectorDatasourceOriginProvider,
|
||||||
)
|
)
|
||||||
from govoplan_connectors.backend.feeds import ConnectorFeedProvider
|
from govoplan_connectors.backend.feeds import ConnectorFeedProvider
|
||||||
|
from govoplan_connectors.backend.provider_state import (
|
||||||
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
TABULAR_PROVIDER_ID,
|
||||||
|
sanctions_provider_states,
|
||||||
|
tabular_provider_states,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
MODULE_ID = "connectors"
|
MODULE_ID = "connectors"
|
||||||
@@ -57,6 +72,142 @@ DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0"
|
|||||||
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
|
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
|
||||||
FEED_INTERFACE_VERSION = "0.1.0"
|
FEED_INTERFACE_VERSION = "0.1.0"
|
||||||
|
|
||||||
|
ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||||
|
layer="data_reporting_integration",
|
||||||
|
kind="integration",
|
||||||
|
maturity="vertical_slice",
|
||||||
|
evidence=(
|
||||||
|
ModuleMaturityEvidence(
|
||||||
|
kind="test",
|
||||||
|
reference="tests/test_tabular_sources.py",
|
||||||
|
summary="Exercises tenant-safe immutable tabular snapshots and bounded reads.",
|
||||||
|
),
|
||||||
|
ModuleMaturityEvidence(
|
||||||
|
kind="test",
|
||||||
|
reference="tests/test_sanctions_sources.py",
|
||||||
|
summary="Exercises source acquisition health, checksums, retries, and immutable evidence.",
|
||||||
|
),
|
||||||
|
ModuleMaturityEvidence(
|
||||||
|
kind="documentation",
|
||||||
|
reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||||
|
summary="Defines source lifecycle, authority, evidence, and outage boundaries.",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
known_limits=(
|
||||||
|
"The executable generic datasource origin is an immutable tabular snapshot; database and arbitrary REST profiles remain future providers.",
|
||||||
|
"Feed publication renders a governed document but does not yet push it to an external publishing endpoint.",
|
||||||
|
),
|
||||||
|
supported_authority_modes=(
|
||||||
|
"external_authoritative",
|
||||||
|
"external_mirror",
|
||||||
|
"linked_reference",
|
||||||
|
),
|
||||||
|
owned_concepts=(
|
||||||
|
"external transport profiles",
|
||||||
|
"protocol interaction",
|
||||||
|
"immutable connector snapshots",
|
||||||
|
"connector acquisition health",
|
||||||
|
),
|
||||||
|
non_owned_concepts=(
|
||||||
|
"datasource catalogue identity and lifecycle",
|
||||||
|
"domain records and business semantics",
|
||||||
|
"data transformations",
|
||||||
|
"screening dispositions",
|
||||||
|
),
|
||||||
|
target_tested_providers=(
|
||||||
|
TABULAR_PROVIDER_ID,
|
||||||
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
),
|
||||||
|
documentation=ModuleArchitectureDocumentation(
|
||||||
|
migration=("src/govoplan_connectors/backend/migrations/versions",),
|
||||||
|
upgrade=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
||||||
|
recovery=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
||||||
|
security=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
||||||
|
operations=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
EXTERNAL_PROVIDERS = (
|
||||||
|
ExternalProviderDeclaration(
|
||||||
|
id=TABULAR_PROVIDER_ID,
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
label="Immutable tabular snapshot provider",
|
||||||
|
maturity="read",
|
||||||
|
operations=("discover", "search", "read", "preview", "dry_run"),
|
||||||
|
objects=(
|
||||||
|
ProviderObjectDeclaration(
|
||||||
|
object_type="tabular_source_snapshot",
|
||||||
|
field_groups=("identity", "schema", "rows", "source_provenance"),
|
||||||
|
authority_modes=("external_authoritative", "external_mirror"),
|
||||||
|
default_authority_mode="external_mirror",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
behavior=ProviderBehaviorDeclaration(
|
||||||
|
revision_tokens="Source fingerprints and immutable snapshot ids are retained.",
|
||||||
|
concurrency="Reads may require the expected fingerprint; snapshots never mutate in place.",
|
||||||
|
freshness="Snapshot acquisition time and source timestamp are exposed.",
|
||||||
|
health="Import validation and source-read failures are explicit.",
|
||||||
|
max_read_items=1000,
|
||||||
|
evidence="Rows, schema, fingerprint, source metadata, and acquisition provenance remain linked.",
|
||||||
|
correction="Import a replacement snapshot; retain the prior snapshot as evidence.",
|
||||||
|
reconciliation="Compare source and snapshot fingerprints before selecting a new current state.",
|
||||||
|
outage="Existing snapshots remain available and visibly stale; no live-source claim is made.",
|
||||||
|
classifications=("internal", "confidential", "restricted"),
|
||||||
|
purposes=("governed import", "dataflow input", "evidence reconstruction"),
|
||||||
|
retention="Datasources or the consuming domain supplies retention and hold policy.",
|
||||||
|
secret_handling="Generic snapshots contain no connector credential; transport credentials stay in credential envelopes.",
|
||||||
|
),
|
||||||
|
capability_names=(
|
||||||
|
CAPABILITY_CONNECTORS_TABULAR_SOURCES,
|
||||||
|
CAPABILITY_DATASOURCE_ORIGINS,
|
||||||
|
),
|
||||||
|
interface_names=(
|
||||||
|
"connectors.tabular_sources",
|
||||||
|
"connectors.datasource_origins",
|
||||||
|
),
|
||||||
|
documentation_topic_ids=(
|
||||||
|
"connectors.authority-and-effects",
|
||||||
|
"connectors.tabular-sources",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
ExternalProviderDeclaration(
|
||||||
|
id=SANCTIONS_PROVIDER_ID,
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
label="Sanctions source snapshot provider",
|
||||||
|
maturity="read",
|
||||||
|
operations=("discover", "search", "read", "preview"),
|
||||||
|
objects=(
|
||||||
|
ProviderObjectDeclaration(
|
||||||
|
object_type="sanctions_source_snapshot",
|
||||||
|
field_groups=("source_identity", "raw_evidence", "entries", "acquisition_health"),
|
||||||
|
authority_modes=("external_authoritative", "external_mirror"),
|
||||||
|
default_authority_mode="external_mirror",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
behavior=ProviderBehaviorDeclaration(
|
||||||
|
revision_tokens="Provider source version, ETag, Last-Modified, and SHA-256 digest are retained when available.",
|
||||||
|
concurrency="Refreshes use conditional source requests and create immutable snapshots.",
|
||||||
|
freshness="Latest successful acquisition, source timestamp, and stale health are reported.",
|
||||||
|
health="Transport, parsing, source-change, and malformed-source states are explicit.",
|
||||||
|
max_read_items=5000,
|
||||||
|
evidence="Raw source bytes, checksum, acquisition run, parser result, and normalized entry count are linked.",
|
||||||
|
correction="A corrected source creates a new immutable snapshot and acquisition run.",
|
||||||
|
reconciliation="Compare source version and digest, then preserve both prior and corrected evidence.",
|
||||||
|
outage="The latest accepted snapshot stays usable with stale/unavailable source health.",
|
||||||
|
classifications=("public", "internal"),
|
||||||
|
purposes=("sanctions source acquisition", "compliance screening evidence"),
|
||||||
|
retention="Risk and Records policies determine accepted snapshot retention and legal holds.",
|
||||||
|
secret_handling="Public sources require no subject data or source credential; configured proxy secrets remain external to snapshots.",
|
||||||
|
),
|
||||||
|
capability_names=(CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,),
|
||||||
|
interface_names=("connectors.sanctions_snapshots",),
|
||||||
|
documentation_topic_ids=(
|
||||||
|
"connectors.authority-and-effects",
|
||||||
|
"connectors.sanctions-snapshots",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
||||||
module_id, resource, action = scope.split(":", 2)
|
module_id, resource, action = scope.split(":", 2)
|
||||||
@@ -157,17 +308,12 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
|||||||
),
|
),
|
||||||
"connector_sanctions_snapshots": (
|
"connector_sanctions_snapshots": (
|
||||||
session.query(ConnectorSanctionsSnapshot)
|
session.query(ConnectorSanctionsSnapshot)
|
||||||
.filter(
|
.filter(ConnectorSanctionsSnapshot.tenant_id == tenant_id)
|
||||||
ConnectorSanctionsSnapshot.tenant_id == tenant_id
|
|
||||||
)
|
|
||||||
.count()
|
.count()
|
||||||
),
|
),
|
||||||
"connector_sanctions_runs": (
|
"connector_sanctions_runs": (
|
||||||
session.query(ConnectorSanctionsAcquisitionRun)
|
session.query(ConnectorSanctionsAcquisitionRun)
|
||||||
.filter(
|
.filter(ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id)
|
||||||
ConnectorSanctionsAcquisitionRun.tenant_id
|
|
||||||
== tenant_id
|
|
||||||
)
|
|
||||||
.count()
|
.count()
|
||||||
),
|
),
|
||||||
}
|
}
|
||||||
@@ -220,12 +366,24 @@ manifest = ModuleManifest(
|
|||||||
CAPABILITY_CONNECTORS_TABULAR_SOURCES: _provider,
|
CAPABILITY_CONNECTORS_TABULAR_SOURCES: _provider,
|
||||||
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER: _provider,
|
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER: _provider,
|
||||||
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
|
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
|
||||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (
|
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider),
|
||||||
_sanctions_snapshot_provider
|
|
||||||
),
|
|
||||||
CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
|
CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
|
||||||
},
|
},
|
||||||
tenant_summary_providers=(_tenant_summary,),
|
tenant_summary_providers=(_tenant_summary,),
|
||||||
|
architecture=ARCHITECTURE,
|
||||||
|
external_providers=EXTERNAL_PROVIDERS,
|
||||||
|
external_provider_state_providers=(
|
||||||
|
ExternalProviderStateProviderRegistration(
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
provider_id=TABULAR_PROVIDER_ID,
|
||||||
|
provider=tabular_provider_states,
|
||||||
|
),
|
||||||
|
ExternalProviderStateProviderRegistration(
|
||||||
|
module_id=MODULE_ID,
|
||||||
|
provider_id=SANCTIONS_PROVIDER_ID,
|
||||||
|
provider=sanctions_provider_states,
|
||||||
|
),
|
||||||
|
),
|
||||||
migration_spec=MigrationSpec(
|
migration_spec=MigrationSpec(
|
||||||
module_id=MODULE_ID,
|
module_id=MODULE_ID,
|
||||||
metadata=Base.metadata,
|
metadata=Base.metadata,
|
||||||
@@ -251,6 +409,21 @@ manifest = ModuleManifest(
|
|||||||
),
|
),
|
||||||
),
|
),
|
||||||
documentation=(
|
documentation=(
|
||||||
|
DocumentationTopic(
|
||||||
|
id="connectors.authority-and-effects",
|
||||||
|
title="Connector authority and effect behavior",
|
||||||
|
summary="Connector direction, technical maturity, and configured source authority are separate and must remain visible.",
|
||||||
|
body=(
|
||||||
|
"A connector can consume, publish, or work bidirectionally and can mature from discovery through replacement. "
|
||||||
|
"Each binding separately states whether GovOPlaN is authoritative, follows an external authority, keeps a mirror, synchronizes under conflict rules, adds a governance overlay, or retains only a link. "
|
||||||
|
"Writable providers must explain revisions, limits, idempotency, outcome-unknown handling, evidence, reconciliation, correction, outage behavior, and secret requirements."
|
||||||
|
),
|
||||||
|
layer="available",
|
||||||
|
documentation_types=("admin", "user"),
|
||||||
|
audience=("operator", "module_admin", "power_user", "product_owner"),
|
||||||
|
related_modules=("datasources", "dataflow", "ops", "policy", "audit"),
|
||||||
|
order=39,
|
||||||
|
),
|
||||||
DocumentationTopic(
|
DocumentationTopic(
|
||||||
id="connectors.tabular-sources",
|
id="connectors.tabular-sources",
|
||||||
title="Governed tabular sources",
|
title="Governed tabular sources",
|
||||||
|
|||||||
@@ -0,0 +1,207 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import defaultdict
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from hashlib import sha256
|
||||||
|
|
||||||
|
from sqlalchemy import func, select
|
||||||
|
from sqlalchemy.orm import Session
|
||||||
|
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorSanctionsAcquisitionRun,
|
||||||
|
ConnectorSanctionsSnapshot,
|
||||||
|
ConnectorTabularSource,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.provider_governance import (
|
||||||
|
ExternalProviderRuntimeState,
|
||||||
|
ExternalProviderStateContext,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
TABULAR_PROVIDER_ID = "connectors.tabular_snapshot"
|
||||||
|
SANCTIONS_PROVIDER_ID = "connectors.sanctions_snapshot"
|
||||||
|
|
||||||
|
|
||||||
|
def tabular_provider_states(
|
||||||
|
context: ExternalProviderStateContext,
|
||||||
|
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||||
|
session = _session(context)
|
||||||
|
statement = select(ConnectorTabularSource).where(
|
||||||
|
ConnectorTabularSource.deleted_at.is_(None)
|
||||||
|
)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
statement = statement.where(
|
||||||
|
ConnectorTabularSource.tenant_id == context.tenant_id
|
||||||
|
)
|
||||||
|
sources = tuple(
|
||||||
|
session.scalars(
|
||||||
|
statement.order_by(
|
||||||
|
ConnectorTabularSource.tenant_id,
|
||||||
|
ConnectorTabularSource.id,
|
||||||
|
).limit(context.max_items + 1)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
observed_at = datetime.now(UTC)
|
||||||
|
return tuple(_tabular_state(item, observed_at=observed_at) for item in sources)
|
||||||
|
|
||||||
|
|
||||||
|
def sanctions_provider_states(
|
||||||
|
context: ExternalProviderStateContext,
|
||||||
|
) -> tuple[ExternalProviderRuntimeState, ...]:
|
||||||
|
session = _session(context)
|
||||||
|
statement = select(ConnectorSanctionsAcquisitionRun)
|
||||||
|
if context.tenant_id is not None:
|
||||||
|
statement = statement.where(
|
||||||
|
ConnectorSanctionsAcquisitionRun.tenant_id == context.tenant_id
|
||||||
|
)
|
||||||
|
runs = tuple(
|
||||||
|
session.scalars(
|
||||||
|
statement.order_by(
|
||||||
|
ConnectorSanctionsAcquisitionRun.tenant_id,
|
||||||
|
ConnectorSanctionsAcquisitionRun.provider_id,
|
||||||
|
ConnectorSanctionsAcquisitionRun.source_id,
|
||||||
|
ConnectorSanctionsAcquisitionRun.started_at.desc(),
|
||||||
|
).limit(max(context.max_items * 10, context.max_items + 1))
|
||||||
|
)
|
||||||
|
)
|
||||||
|
latest_by_binding: dict[tuple[str, str, str], ConnectorSanctionsAcquisitionRun] = {}
|
||||||
|
for run in runs:
|
||||||
|
key = (run.tenant_id, run.provider_id, run.source_id)
|
||||||
|
latest_by_binding.setdefault(key, run)
|
||||||
|
if len(latest_by_binding) >= context.max_items + 1:
|
||||||
|
break
|
||||||
|
|
||||||
|
snapshot_counts = _snapshot_counts(
|
||||||
|
session,
|
||||||
|
binding_keys=tuple(latest_by_binding),
|
||||||
|
)
|
||||||
|
observed_at = datetime.now(UTC)
|
||||||
|
return tuple(
|
||||||
|
_sanctions_state(
|
||||||
|
run,
|
||||||
|
observed_at=observed_at,
|
||||||
|
snapshot_count=snapshot_counts.get(key, 0),
|
||||||
|
)
|
||||||
|
for key, run in latest_by_binding.items()
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _session(context: ExternalProviderStateContext) -> Session:
|
||||||
|
if not isinstance(context.session, Session):
|
||||||
|
raise RuntimeError("Connectors provider state requires a database session.")
|
||||||
|
return context.session
|
||||||
|
|
||||||
|
|
||||||
|
def _tabular_state(
|
||||||
|
source: ConnectorTabularSource,
|
||||||
|
*,
|
||||||
|
observed_at: datetime,
|
||||||
|
) -> ExternalProviderRuntimeState:
|
||||||
|
active = source.status == "active"
|
||||||
|
return ExternalProviderRuntimeState(
|
||||||
|
provider_id=TABULAR_PROVIDER_ID,
|
||||||
|
binding_ref=f"connectors:tabular-source:{source.id}",
|
||||||
|
authority_mode="external_mirror",
|
||||||
|
observed_at=observed_at,
|
||||||
|
configured=True,
|
||||||
|
active=active,
|
||||||
|
health="healthy" if active else "inactive",
|
||||||
|
freshness="not_applicable",
|
||||||
|
conflict="not_applicable",
|
||||||
|
recovery="ready" if active else "not_applicable",
|
||||||
|
last_success_at=_aware(source.updated_at or source.created_at),
|
||||||
|
detail=(
|
||||||
|
"Immutable tabular snapshot is available."
|
||||||
|
if active
|
||||||
|
else "Immutable tabular snapshot is inactive."
|
||||||
|
),
|
||||||
|
metrics={
|
||||||
|
"row_count": int(source.row_count),
|
||||||
|
"byte_count": int(source.byte_count),
|
||||||
|
"schema_version": int(source.schema_version),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _snapshot_counts(
|
||||||
|
session: Session,
|
||||||
|
*,
|
||||||
|
binding_keys: tuple[tuple[str, str, str], ...],
|
||||||
|
) -> dict[tuple[str, str, str], int]:
|
||||||
|
if not binding_keys:
|
||||||
|
return {}
|
||||||
|
tenant_ids = {item[0] for item in binding_keys}
|
||||||
|
rows = session.execute(
|
||||||
|
select(
|
||||||
|
ConnectorSanctionsSnapshot.tenant_id,
|
||||||
|
ConnectorSanctionsSnapshot.provider_id,
|
||||||
|
ConnectorSanctionsSnapshot.source_id,
|
||||||
|
func.count(ConnectorSanctionsSnapshot.id),
|
||||||
|
)
|
||||||
|
.where(ConnectorSanctionsSnapshot.tenant_id.in_(tenant_ids))
|
||||||
|
.group_by(
|
||||||
|
ConnectorSanctionsSnapshot.tenant_id,
|
||||||
|
ConnectorSanctionsSnapshot.provider_id,
|
||||||
|
ConnectorSanctionsSnapshot.source_id,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
(str(tenant_id), str(provider_id), str(source_id)): int(count)
|
||||||
|
for tenant_id, provider_id, source_id, count in rows
|
||||||
|
if (str(tenant_id), str(provider_id), str(source_id)) in binding_keys
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _sanctions_state(
|
||||||
|
run: ConnectorSanctionsAcquisitionRun,
|
||||||
|
*,
|
||||||
|
observed_at: datetime,
|
||||||
|
snapshot_count: int,
|
||||||
|
) -> ExternalProviderRuntimeState:
|
||||||
|
status = str(run.status)
|
||||||
|
success = status in {"succeeded", "success", "not_modified"}
|
||||||
|
running = status in {"running", "pending", "retry"}
|
||||||
|
has_snapshot = bool(run.snapshot_id) or snapshot_count > 0
|
||||||
|
health = "healthy" if success else "warning" if running else "error"
|
||||||
|
binding_digest = sha256(
|
||||||
|
f"{run.tenant_id}\0{run.provider_id}\0{run.source_id}".encode("utf-8")
|
||||||
|
).hexdigest()[:24]
|
||||||
|
return ExternalProviderRuntimeState(
|
||||||
|
provider_id=SANCTIONS_PROVIDER_ID,
|
||||||
|
binding_ref=f"connectors:sanctions-source:{binding_digest}",
|
||||||
|
authority_mode="external_mirror",
|
||||||
|
observed_at=observed_at,
|
||||||
|
configured=True,
|
||||||
|
active=True,
|
||||||
|
health=health,
|
||||||
|
freshness="unknown",
|
||||||
|
conflict="not_applicable",
|
||||||
|
recovery="ready" if success and has_snapshot else "attention",
|
||||||
|
last_success_at=_aware(run.finished_at) if success else None,
|
||||||
|
detail=(
|
||||||
|
"Latest sanctions acquisition completed."
|
||||||
|
if success
|
||||||
|
else "Sanctions acquisition is in progress."
|
||||||
|
if running
|
||||||
|
else "Latest sanctions acquisition failed; prior accepted snapshots remain separate evidence."
|
||||||
|
),
|
||||||
|
metrics={
|
||||||
|
"latest_status": status,
|
||||||
|
"attempt_count": int(run.attempt_count),
|
||||||
|
"accepted_snapshots": int(snapshot_count),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _aware(value: datetime | None) -> datetime | None:
|
||||||
|
if value is None:
|
||||||
|
return None
|
||||||
|
return value.replace(tzinfo=UTC) if value.tzinfo is None else value.astimezone(UTC)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"SANCTIONS_PROVIDER_ID",
|
||||||
|
"TABULAR_PROVIDER_ID",
|
||||||
|
"sanctions_provider_states",
|
||||||
|
"tabular_provider_states",
|
||||||
|
]
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
from sqlalchemy import create_engine
|
||||||
|
from sqlalchemy.orm import sessionmaker
|
||||||
|
|
||||||
|
from govoplan_connectors.backend.db.models import (
|
||||||
|
ConnectorSanctionsAcquisitionRun,
|
||||||
|
ConnectorSanctionsSnapshot,
|
||||||
|
ConnectorTabularSource,
|
||||||
|
)
|
||||||
|
from govoplan_connectors.backend.manifest import manifest
|
||||||
|
from govoplan_connectors.backend.provider_state import (
|
||||||
|
SANCTIONS_PROVIDER_ID,
|
||||||
|
TABULAR_PROVIDER_ID,
|
||||||
|
sanctions_provider_states,
|
||||||
|
tabular_provider_states,
|
||||||
|
)
|
||||||
|
from govoplan_core.core.provider_governance import ExternalProviderStateContext
|
||||||
|
from govoplan_core.db.base import Base
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectorsProviderStateTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True)
|
||||||
|
Base.metadata.create_all(
|
||||||
|
self.engine,
|
||||||
|
tables=(
|
||||||
|
ConnectorTabularSource.__table__,
|
||||||
|
ConnectorSanctionsAcquisitionRun.__table__,
|
||||||
|
ConnectorSanctionsSnapshot.__table__,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
self.session = sessionmaker(bind=self.engine, expire_on_commit=False)()
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.session.close()
|
||||||
|
self.engine.dispose()
|
||||||
|
|
||||||
|
def test_immutable_tabular_snapshot_reports_ready_state(self) -> None:
|
||||||
|
self.session.add(
|
||||||
|
ConnectorTabularSource(
|
||||||
|
id="tabular-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
source_name="monthly",
|
||||||
|
name="Monthly input",
|
||||||
|
status="active",
|
||||||
|
schema_version=1,
|
||||||
|
schema_=[{"name": "id", "type": "string"}],
|
||||||
|
rows=[{"id": "1"}],
|
||||||
|
fingerprint="a" * 64,
|
||||||
|
row_count=1,
|
||||||
|
byte_count=10,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
state = tabular_provider_states(
|
||||||
|
ExternalProviderStateContext(session=self.session, tenant_id="tenant-1")
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
self.assertEqual("healthy", state.health)
|
||||||
|
self.assertEqual("ready", state.recovery)
|
||||||
|
self.assertEqual("not_applicable", state.freshness)
|
||||||
|
|
||||||
|
def test_sanctions_state_hashes_binding_and_manifest_registers_state(self) -> None:
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
run = ConnectorSanctionsAcquisitionRun(
|
||||||
|
id="run-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id="eu",
|
||||||
|
source_id="secret-source-name",
|
||||||
|
status="succeeded",
|
||||||
|
attempt_count=1,
|
||||||
|
started_at=now,
|
||||||
|
finished_at=now,
|
||||||
|
)
|
||||||
|
snapshot = ConnectorSanctionsSnapshot(
|
||||||
|
id="snapshot-1",
|
||||||
|
tenant_id="tenant-1",
|
||||||
|
provider_id="eu",
|
||||||
|
publisher="European Union",
|
||||||
|
jurisdiction="EU",
|
||||||
|
list_type="sanctions",
|
||||||
|
source_id="secret-source-name",
|
||||||
|
source_version="2026-08-01",
|
||||||
|
acquired_at=now,
|
||||||
|
source_url="https://source.example.test/list.xml",
|
||||||
|
content_type="application/xml",
|
||||||
|
byte_count=8,
|
||||||
|
sha256="b" * 64,
|
||||||
|
parser_version="1",
|
||||||
|
connector_run_id=run.id,
|
||||||
|
raw_content=b"<list/>",
|
||||||
|
)
|
||||||
|
run.snapshot_id = snapshot.id
|
||||||
|
self.session.add_all((run, snapshot))
|
||||||
|
self.session.commit()
|
||||||
|
|
||||||
|
state = sanctions_provider_states(
|
||||||
|
ExternalProviderStateContext(session=self.session, tenant_id="tenant-1")
|
||||||
|
)[0]
|
||||||
|
|
||||||
|
self.assertEqual("healthy", state.health)
|
||||||
|
self.assertEqual("ready", state.recovery)
|
||||||
|
rendered = str(state.to_dict())
|
||||||
|
self.assertNotIn("secret-source-name", rendered)
|
||||||
|
self.assertNotIn("source.example.test", rendered)
|
||||||
|
self.assertEqual(
|
||||||
|
{TABULAR_PROVIDER_ID, SANCTIONS_PROVIDER_ID},
|
||||||
|
{item.provider_id for item in manifest.external_provider_state_providers},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
Reference in New Issue
Block a user