Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2c2b11f860 | ||
|
|
e6ab8291ec | ||
|
|
de8e74b866 | ||
|
|
0f61fe4607 |
@@ -14,12 +14,14 @@ or reporting behavior.
|
||||
|
||||
## Executable First Slice
|
||||
|
||||
The first executable connector capability provides tenant-isolated tabular
|
||||
origins. Operators can import bounded JSON or CSV snapshots, inspect inferred
|
||||
schemas, and expose immutable source references and content fingerprints
|
||||
through `connectors.datasource_origins@0.1.0`. Preview reads enforce provider
|
||||
ceilings for rows, serialized bytes, and elapsed time and report the effective
|
||||
limits and any truncation as structured diagnostics.
|
||||
The executable connector capability provides tenant-isolated tabular origins.
|
||||
Operators can import bounded JSON or CSV snapshots, bind an exact managed Files
|
||||
CSV/XLSX version, or discover a table through an active governed PostgreSQL
|
||||
configuration. Every origin exposes a reviewed schema, opaque reference, and
|
||||
content/discovery fingerprint through `connectors.datasource_origins@0.1.0`.
|
||||
Preview reads enforce provider ceilings for rows, serialized bytes, and elapsed
|
||||
time and report the effective limits and any truncation as structured
|
||||
diagnostics.
|
||||
|
||||
Connectors owns acquisition, connection profiles, credentials, discovery, and
|
||||
provider health. `govoplan-datasources` registers an origin as a governed live
|
||||
@@ -29,12 +31,28 @@ connector implementations or stores connector credentials.
|
||||
|
||||
Each origin declares whether it is live, cached, file-backed, or static, its
|
||||
structured health state, and which projection, filter, aggregation, sorting,
|
||||
and pagination operations it can push down. The immutable snapshot provider
|
||||
currently supports projection and pagination only; consumers must keep other
|
||||
operations in Dataflow rather than assuming transport-side execution.
|
||||
and pagination operations it can push down. The snapshot, managed-file, and
|
||||
PostgreSQL providers currently support projection and pagination only;
|
||||
consumers must keep other operations in Dataflow rather than assuming
|
||||
transport-side execution.
|
||||
|
||||
Database, REST/HTTP, directory, managed-file, and warehouse providers can
|
||||
implement the same origin contract without changing Datasources or Dataflow.
|
||||
Managed-file sources are authorized and opened through
|
||||
`files.tabular_content@1.0.0`; Files remains authoritative for ownership,
|
||||
shares, download permission, exact versions, integrity, quarantine, encryption,
|
||||
retention, and legal holds. CSV must be UTF-8. XLSX input is protected by
|
||||
compressed-entry, expanded-byte, compression-ratio, row, and column limits.
|
||||
A newer current version is reported but never silently replaces the pinned
|
||||
version.
|
||||
|
||||
The PostgreSQL adapter accepts only an active governed connector configuration
|
||||
whose secret-free endpoint uses the PostgreSQL driver. Authentication is
|
||||
resolved from a tenant/scope/module/server-restricted Core credential envelope.
|
||||
The adapter reflects a simple schema/table identifier, uses read-only
|
||||
transactions and a statement timeout, and blocks configuration, credential, or
|
||||
schema drift until an operator refreshes and reviews the source. Secrets are
|
||||
never copied into source metadata or diagnostics. Other database, REST/HTTP,
|
||||
directory, and warehouse providers can implement the same origin contract
|
||||
without changing Datasources or Dataflow.
|
||||
|
||||
Governed sanctions and feed snapshot acquisitions use Core recovery operations.
|
||||
The source revision/cursor, redacted dry-run decision, canonical request digest,
|
||||
@@ -46,6 +64,22 @@ recovery contract requires stable idempotency, provider verification, and
|
||||
operator reconciliation, but no connector currently claims a production write
|
||||
or delete path.
|
||||
|
||||
RSS and Atom emission is a bounded renderer, not an authority shortcut. Every
|
||||
selected entry declares whether it came from a GovOPlaN event, publication,
|
||||
case, or report and carries an opaque owning-module reference and optional
|
||||
revision. Public-feed permission can render only public entries. Tenant and
|
||||
private audiences require a separate restricted-feed permission, and the API
|
||||
derives the allowed visibility set from that audience instead of accepting a
|
||||
caller-controlled allow-list. Portal or Reporting remains responsible for any
|
||||
durable public or authenticated route and must re-authorize restricted access.
|
||||
|
||||
The governed connector runtime adds immutable definition revisions,
|
||||
revision-pinned tenant configurations, protected local overrides, explicit
|
||||
package-update adoption, bounded dry-runs and simulations, redacted provenance,
|
||||
idempotency, and configurable ambiguity handling through review, quarantine,
|
||||
or rejection. Its administration surface is contributed to the shared system
|
||||
administration workspace. Provider-specific adapters still own live writes.
|
||||
|
||||
Development:
|
||||
|
||||
```bash
|
||||
@@ -60,3 +94,4 @@ See:
|
||||
- [Connector source lifecycle](docs/CONNECTOR_SOURCE_LIFECYCLE.md)
|
||||
- [OpenProject connector concept](docs/OPENPROJECT_CONNECTOR.md)
|
||||
- [OpenDesk integration map](docs/OPENDESK_INTEGRATION_MAP.md)
|
||||
- [Governed connector configuration](docs/GOVERNED_CONNECTOR_CONFIGURATION.md)
|
||||
|
||||
@@ -112,6 +112,30 @@ this lifecycle when a connector publishes status.
|
||||
9. Store external references with source system, object type, object id, version
|
||||
or ETag, and last-seen timestamp.
|
||||
|
||||
## Tabular source discovery and refresh
|
||||
|
||||
Immutable JSON/CSV snapshots, exact managed Files versions, and live PostgreSQL
|
||||
tables share the same catalogue and bounded-preview contract. A managed-file
|
||||
origin stores only the Files asset id, exact immutable version id, checksum,
|
||||
parser settings, reviewed schema, and discovery fingerprint. Files re-authorizes
|
||||
the current principal and verifies storage integrity and any encryption envelope
|
||||
on every preview. A newer current version produces a warning; only an explicit
|
||||
source refresh changes the pinned version and increments the discovery revision.
|
||||
|
||||
A PostgreSQL origin references an active governed connector configuration. Its
|
||||
endpoint must contain no credentials. Connectors resolves the referenced Core
|
||||
credential envelope for the current tenant, scope, module, and server, opens a
|
||||
read-only connection, reflects a simple schema/table identifier, and records the
|
||||
configuration hash/revision, credential revision, schema, and discovery
|
||||
fingerprint. Configuration, credential, or schema drift blocks preview until an
|
||||
explicit refresh. Missing Files capability, revoked file access, quarantine,
|
||||
oversized or malformed content, inactive/stale credentials, unreachable SQL,
|
||||
and timeout failures produce sanitized unavailable/validation diagnostics.
|
||||
|
||||
All three current providers declare projection and pagination pushdown only.
|
||||
Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly
|
||||
declares and tests those operations.
|
||||
|
||||
## Publish Flow
|
||||
|
||||
1. Domain module requests publish through a core-mediated connector capability.
|
||||
|
||||
@@ -25,7 +25,7 @@ logic should remain visible and reviewable.
|
||||
|
||||
## Runtime Expectations
|
||||
|
||||
Connectors should support:
|
||||
Connectors supports the generic governed-definition and simulation portion of:
|
||||
|
||||
- discovery where possible
|
||||
- typed configuration through UI-managed controls
|
||||
@@ -39,6 +39,44 @@ Connectors should support:
|
||||
Configuration packages may install connector definitions, but local overrides
|
||||
must be protected from accidental package updates.
|
||||
|
||||
## Implemented Runtime Slice
|
||||
|
||||
The module now persists tenant-scoped connector definitions as immutable
|
||||
revisions. A governed definition explicitly validates its provider, protocol,
|
||||
capabilities, input/output schemas, mapping version and rules, validation,
|
||||
preview metadata, audit expectations, classification, retention, limits, and
|
||||
retry policy. Definitions record whether they are locally owned or supplied by
|
||||
a named package.
|
||||
|
||||
Configurations pin a definition revision. They store an endpoint and a secret
|
||||
reference, never credentials embedded in the URL. Tenant-local override values
|
||||
are merged into the pinned definition and every overridden leaf is exposed as
|
||||
a protected path. Installing a later package revision only marks the
|
||||
configuration as having an update available. Adoption is an explicit,
|
||||
optimistically locked action that reapplies the protected overrides over the
|
||||
new package revision.
|
||||
|
||||
The generic execution surface supports bounded dry-runs and simulations. Each
|
||||
run has a caller idempotency key and retains hashes of its inputs and effective
|
||||
configuration together with definition, configuration, mapping, external
|
||||
revision, actor, classification, and retention provenance. Samples redact the
|
||||
definition's protected fields. Ambiguous uniqueness results follow the
|
||||
configuration's policy and become either:
|
||||
|
||||
- `manual_review` with a pending decision;
|
||||
- `quarantined` until an administrator decides; or
|
||||
- `rejected` without a review queue entry.
|
||||
|
||||
Review decisions require a reason and are audited. The generic runtime stops
|
||||
at deterministic preview evidence: provider-specific adapters remain
|
||||
responsible for live external writes and must satisfy the Core connector
|
||||
recovery contract before claiming write maturity.
|
||||
|
||||
The Connector governance administration page follows the shared workspace
|
||||
archetype. Reload and Save stay in the semantic action bar, dirty navigation is
|
||||
guarded, package adoption is a separate action, and simulation results and
|
||||
review decisions remain visibly distinct from configuration editing.
|
||||
|
||||
## Relationship To Datasources And Dataflow
|
||||
|
||||
Recurring extraction and transformation should start as configuration across
|
||||
|
||||
+3
-2
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "govoplan-connectors"
|
||||
version = "0.1.18"
|
||||
version = "0.1.20"
|
||||
description = "Governed connector catalogue and tabular source capabilities for GovOPlaN."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
@@ -12,7 +12,8 @@ license = "AGPL-3.0-or-later"
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"defusedxml>=0.7,<1",
|
||||
"govoplan-core>=0.1.18",
|
||||
"govoplan-core>=0.1.19",
|
||||
"openpyxl>=3.1.5,<4",
|
||||
]
|
||||
|
||||
[tool.setuptools.packages.find]
|
||||
|
||||
@@ -102,12 +102,16 @@ class ConnectorDatasourceOriginProvider:
|
||||
|
||||
|
||||
def _origin(source: TabularSource) -> DatasourceOrigin:
|
||||
kind = {
|
||||
"managed_file": "file",
|
||||
"postgresql": "database",
|
||||
}.get(source.provider, "upload")
|
||||
return DatasourceOrigin(
|
||||
ref=source.ref,
|
||||
source_name=source.source_name,
|
||||
name=source.name,
|
||||
description=source.description,
|
||||
kind="upload",
|
||||
kind=kind,
|
||||
shape="tabular",
|
||||
supported_modes=("live", "cached"),
|
||||
provider=f"connectors.{source.provider}",
|
||||
|
||||
@@ -5,6 +5,7 @@ from datetime import datetime
|
||||
from typing import Any
|
||||
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
DateTime,
|
||||
ForeignKey,
|
||||
Index,
|
||||
@@ -255,9 +256,213 @@ class ConnectorSanctionsSnapshot(Base, TimestampMixin):
|
||||
)
|
||||
|
||||
|
||||
class ConnectorDefinition(Base, TimestampMixin):
|
||||
__tablename__ = "connector_definitions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"definition_key",
|
||||
name="uq_connector_definition_tenant_key",
|
||||
),
|
||||
Index(
|
||||
"ix_connector_definitions_tenant_status",
|
||||
"tenant_id",
|
||||
"status",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
definition_key: Mapped[str] = mapped_column(String(160), nullable=False)
|
||||
name: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30),
|
||||
default="active",
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
current_revision: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
|
||||
source_package: Mapped[str | None] = mapped_column(String(300))
|
||||
local_definition: Mapped[bool] = mapped_column(
|
||||
Boolean,
|
||||
default=False,
|
||||
nullable=False,
|
||||
)
|
||||
|
||||
|
||||
class ConnectorDefinitionRevision(Base, TimestampMixin):
|
||||
__tablename__ = "connector_definition_revisions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"definition_id",
|
||||
"revision",
|
||||
name="uq_connector_definition_revision",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
definition_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("connector_definitions.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
specification: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON,
|
||||
default=dict,
|
||||
nullable=False,
|
||||
)
|
||||
definition_hash: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
origin: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
package_ref: Mapped[str | None] = mapped_column(String(300))
|
||||
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
|
||||
|
||||
class ConnectorConfiguration(Base, TimestampMixin):
|
||||
__tablename__ = "connector_configurations"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"name",
|
||||
name="uq_connector_configuration_tenant_name",
|
||||
),
|
||||
Index(
|
||||
"ix_connector_configurations_tenant_status",
|
||||
"tenant_id",
|
||||
"status",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
definition_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("connector_definitions.id", ondelete="RESTRICT"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
name: Mapped[str] = mapped_column(String(300), nullable=False)
|
||||
status: Mapped[str] = mapped_column(
|
||||
String(30),
|
||||
default="draft",
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
endpoint_url: Mapped[str | None] = mapped_column(String(1500))
|
||||
credential_ref: Mapped[str | None] = mapped_column(String(500))
|
||||
base_definition_revision: Mapped[int] = mapped_column(
|
||||
Integer,
|
||||
nullable=False,
|
||||
)
|
||||
local_overrides: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON,
|
||||
default=dict,
|
||||
nullable=False,
|
||||
)
|
||||
protected_paths: Mapped[list[str]] = mapped_column(
|
||||
JSON,
|
||||
default=list,
|
||||
nullable=False,
|
||||
)
|
||||
effective_configuration: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON,
|
||||
default=dict,
|
||||
nullable=False,
|
||||
)
|
||||
effective_hash: Mapped[str] = mapped_column(
|
||||
String(64),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
resource_revision: Mapped[int] = mapped_column(
|
||||
Integer,
|
||||
default=1,
|
||||
nullable=False,
|
||||
)
|
||||
ambiguity_policy: Mapped[str] = mapped_column(
|
||||
String(30),
|
||||
default="manual_review",
|
||||
nullable=False,
|
||||
)
|
||||
updated_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
|
||||
|
||||
class ConnectorSimulationRun(Base, TimestampMixin):
|
||||
__tablename__ = "connector_simulation_runs"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"configuration_id",
|
||||
"mode",
|
||||
"idempotency_key",
|
||||
name="uq_connector_simulation_run_idempotency",
|
||||
),
|
||||
Index(
|
||||
"ix_connector_simulation_runs_review",
|
||||
"tenant_id",
|
||||
"review_state",
|
||||
"created_at",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
configuration_id: Mapped[str] = mapped_column(
|
||||
ForeignKey("connector_configurations.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
mode: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
status: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
|
||||
review_state: Mapped[str] = mapped_column(
|
||||
String(30),
|
||||
default="not_required",
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
definition_revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
configuration_revision: Mapped[int] = mapped_column(Integer, nullable=False)
|
||||
configuration_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
input_hash: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
summary: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON,
|
||||
default=dict,
|
||||
nullable=False,
|
||||
)
|
||||
effects: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||
JSON,
|
||||
default=list,
|
||||
nullable=False,
|
||||
)
|
||||
diagnostics: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||
JSON,
|
||||
default=list,
|
||||
nullable=False,
|
||||
)
|
||||
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON,
|
||||
default=dict,
|
||||
nullable=False,
|
||||
)
|
||||
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
reviewed_by: Mapped[str | None] = mapped_column(String(255), index=True)
|
||||
reviewed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
|
||||
review_reason: Mapped[str | None] = mapped_column(Text)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ConnectorConfiguration",
|
||||
"ConnectorDefinition",
|
||||
"ConnectorDefinitionRevision",
|
||||
"ConnectorSanctionsAcquisitionRun",
|
||||
"ConnectorSanctionsSnapshot",
|
||||
"ConnectorSimulationRun",
|
||||
"ConnectorTabularSource",
|
||||
"new_uuid",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,482 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from sqlalchemy import or_
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import (
|
||||
DsarErasureActionRef,
|
||||
DsarExecutionResultRef,
|
||||
DsarRecordRef,
|
||||
DsarSubjectRef,
|
||||
dsar_capability_name,
|
||||
)
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorSanctionsAcquisitionRun,
|
||||
ConnectorSimulationRun,
|
||||
ConnectorTabularSource,
|
||||
)
|
||||
|
||||
|
||||
CONNECTORS_DSAR_CAPABILITY = dsar_capability_name("connectors")
|
||||
_MAX_RECORDS = 5_000
|
||||
_CONFLICT = object()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _SubjectSelectors:
|
||||
account_id: str
|
||||
source_id: str | None
|
||||
acquisition_id: str | None
|
||||
definition_id: str | None
|
||||
configuration_id: str | None
|
||||
simulation_id: str | None
|
||||
|
||||
@property
|
||||
def narrowed(self) -> bool:
|
||||
return any(
|
||||
(
|
||||
self.source_id,
|
||||
self.acquisition_id,
|
||||
self.definition_id,
|
||||
self.configuration_id,
|
||||
self.simulation_id,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
class ConnectorsDsarProvider:
|
||||
provider_id = "connectors"
|
||||
module_id = "connectors"
|
||||
|
||||
def search_subject(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
) -> Sequence[DsarRecordRef]:
|
||||
db = _session(session)
|
||||
selectors = _subject_selectors(subject)
|
||||
if selectors is None:
|
||||
return ()
|
||||
|
||||
records: list[DsarRecordRef] = []
|
||||
if not selectors.narrowed or selectors.source_id:
|
||||
query = db.query(ConnectorTabularSource).filter(
|
||||
ConnectorTabularSource.tenant_id == tenant_id,
|
||||
or_(
|
||||
ConnectorTabularSource.created_by == selectors.account_id,
|
||||
ConnectorTabularSource.updated_by == selectors.account_id,
|
||||
),
|
||||
)
|
||||
if selectors.source_id:
|
||||
query = query.filter(ConnectorTabularSource.id == selectors.source_id)
|
||||
records.extend(
|
||||
_source_attribution(row, selectors.account_id)
|
||||
for row in _limited(
|
||||
query,
|
||||
ConnectorTabularSource.created_at,
|
||||
ConnectorTabularSource.id,
|
||||
label="source attribution",
|
||||
)
|
||||
)
|
||||
|
||||
if not selectors.narrowed or selectors.acquisition_id:
|
||||
query = db.query(ConnectorSanctionsAcquisitionRun).filter(
|
||||
ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id,
|
||||
ConnectorSanctionsAcquisitionRun.created_by == selectors.account_id,
|
||||
)
|
||||
if selectors.acquisition_id:
|
||||
query = query.filter(
|
||||
ConnectorSanctionsAcquisitionRun.id == selectors.acquisition_id
|
||||
)
|
||||
records.extend(
|
||||
_acquisition_attribution(row)
|
||||
for row in _limited(
|
||||
query,
|
||||
ConnectorSanctionsAcquisitionRun.started_at,
|
||||
ConnectorSanctionsAcquisitionRun.id,
|
||||
label="acquisition attribution",
|
||||
)
|
||||
)
|
||||
|
||||
if not selectors.narrowed or selectors.definition_id:
|
||||
query = (
|
||||
db.query(ConnectorDefinitionRevision, ConnectorDefinition)
|
||||
.join(
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinition.id == ConnectorDefinitionRevision.definition_id,
|
||||
)
|
||||
.filter(
|
||||
ConnectorDefinition.tenant_id == tenant_id,
|
||||
ConnectorDefinitionRevision.created_by == selectors.account_id,
|
||||
)
|
||||
)
|
||||
if selectors.definition_id:
|
||||
query = query.filter(ConnectorDefinition.id == selectors.definition_id)
|
||||
records.extend(
|
||||
_definition_attribution(revision, definition)
|
||||
for revision, definition in _limited(
|
||||
query,
|
||||
ConnectorDefinitionRevision.created_at,
|
||||
ConnectorDefinitionRevision.id,
|
||||
label="definition attribution",
|
||||
)
|
||||
)
|
||||
|
||||
if not selectors.narrowed or selectors.configuration_id:
|
||||
query = db.query(ConnectorConfiguration).filter(
|
||||
ConnectorConfiguration.tenant_id == tenant_id,
|
||||
ConnectorConfiguration.updated_by == selectors.account_id,
|
||||
)
|
||||
if selectors.configuration_id:
|
||||
query = query.filter(
|
||||
ConnectorConfiguration.id == selectors.configuration_id
|
||||
)
|
||||
records.extend(
|
||||
_configuration_attribution(row)
|
||||
for row in _limited(
|
||||
query,
|
||||
ConnectorConfiguration.updated_at,
|
||||
ConnectorConfiguration.id,
|
||||
label="configuration attribution",
|
||||
)
|
||||
)
|
||||
|
||||
if not selectors.narrowed or selectors.simulation_id:
|
||||
query = db.query(ConnectorSimulationRun).filter(
|
||||
ConnectorSimulationRun.tenant_id == tenant_id,
|
||||
or_(
|
||||
ConnectorSimulationRun.created_by == selectors.account_id,
|
||||
ConnectorSimulationRun.reviewed_by == selectors.account_id,
|
||||
),
|
||||
)
|
||||
if selectors.simulation_id:
|
||||
query = query.filter(
|
||||
ConnectorSimulationRun.id == selectors.simulation_id
|
||||
)
|
||||
records.extend(
|
||||
_simulation_attribution(row, selectors.account_id)
|
||||
for row in _limited(
|
||||
query,
|
||||
ConnectorSimulationRun.created_at,
|
||||
ConnectorSimulationRun.id,
|
||||
label="simulation attribution",
|
||||
)
|
||||
)
|
||||
|
||||
if len(records) > _MAX_RECORDS:
|
||||
raise ValueError("Connectors DSAR result limit exceeded; narrow selectors.")
|
||||
return tuple(
|
||||
sorted(records, key=lambda item: (item.resource_type, item.resource_id))
|
||||
)
|
||||
|
||||
def plan_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
records: Sequence[DsarRecordRef],
|
||||
) -> Sequence[DsarErasureActionRef]:
|
||||
del tenant_id
|
||||
_session(session)
|
||||
if _subject_selectors(subject) is None:
|
||||
raise ValueError("Connectors DSAR subject selectors conflict.")
|
||||
actions: list[DsarErasureActionRef] = []
|
||||
for record in records:
|
||||
_validate_record(record)
|
||||
actions.append(
|
||||
DsarErasureActionRef(
|
||||
action_id=(
|
||||
f"connectors:retain:{record.resource_type}:{record.resource_id}"
|
||||
),
|
||||
provider_id=self.provider_id,
|
||||
module_id=self.module_id,
|
||||
kind="retain",
|
||||
resource_type=record.resource_type,
|
||||
resource_id=record.resource_id,
|
||||
title=f"Retain {record.title}",
|
||||
rationale=(
|
||||
record.retention_reason
|
||||
or "Connector operator attribution remains governance evidence."
|
||||
),
|
||||
executable=False,
|
||||
)
|
||||
)
|
||||
return tuple(actions)
|
||||
|
||||
def execute_erasure(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
subject: DsarSubjectRef,
|
||||
actions: Sequence[DsarErasureActionRef],
|
||||
request_id: str,
|
||||
) -> Sequence[DsarExecutionResultRef]:
|
||||
del tenant_id
|
||||
_session(session)
|
||||
if _subject_selectors(subject) is None:
|
||||
raise ValueError("Connectors DSAR subject selectors conflict.")
|
||||
results: list[DsarExecutionResultRef] = []
|
||||
for action in actions:
|
||||
_validate_action(action)
|
||||
if action.executable or action.kind != "retain":
|
||||
raise ValueError("Connectors DSAR publishes retain actions only.")
|
||||
results.append(
|
||||
DsarExecutionResultRef(
|
||||
action_id=action.action_id,
|
||||
status="blocked",
|
||||
summary="Connector operator attribution remains governance evidence.",
|
||||
evidence={"request_id": request_id},
|
||||
)
|
||||
)
|
||||
return tuple(results)
|
||||
|
||||
|
||||
def _subject_selectors(subject: DsarSubjectRef) -> _SubjectSelectors | None:
|
||||
references = subject.external_references
|
||||
account = _coalesce(
|
||||
subject.account_id,
|
||||
references.get("connectors.account"),
|
||||
references.get("access.account"),
|
||||
)
|
||||
values = {
|
||||
"source_id": _coalesce(
|
||||
references.get("connectors.source"),
|
||||
references.get("connectors.source_id"),
|
||||
),
|
||||
"acquisition_id": _coalesce(
|
||||
references.get("connectors.acquisition"),
|
||||
references.get("connectors.acquisition_id"),
|
||||
),
|
||||
"definition_id": _coalesce(
|
||||
references.get("connectors.definition"),
|
||||
references.get("connectors.definition_id"),
|
||||
),
|
||||
"configuration_id": _coalesce(
|
||||
references.get("connectors.configuration"),
|
||||
references.get("connectors.configuration_id"),
|
||||
),
|
||||
"simulation_id": _coalesce(
|
||||
references.get("connectors.simulation"),
|
||||
references.get("connectors.simulation_id"),
|
||||
),
|
||||
}
|
||||
if account is _CONFLICT or any(value is _CONFLICT for value in values.values()):
|
||||
return None
|
||||
account_id = _optional_string(account)
|
||||
if not account_id:
|
||||
return None
|
||||
return _SubjectSelectors(
|
||||
account_id=account_id,
|
||||
source_id=_optional_string(values["source_id"]),
|
||||
acquisition_id=_optional_string(values["acquisition_id"]),
|
||||
definition_id=_optional_string(values["definition_id"]),
|
||||
configuration_id=_optional_string(values["configuration_id"]),
|
||||
simulation_id=_optional_string(values["simulation_id"]),
|
||||
)
|
||||
|
||||
|
||||
def _source_attribution(row: ConnectorTabularSource, account_id: str) -> DsarRecordRef:
|
||||
activities = []
|
||||
if row.created_by == account_id:
|
||||
activities.append("created_source_snapshot")
|
||||
if row.updated_by == account_id:
|
||||
activities.append("updated_source_snapshot")
|
||||
return _record(
|
||||
resource_type="source_actor_attribution",
|
||||
resource_id=row.id,
|
||||
title="Connector source actor attribution",
|
||||
data={
|
||||
"source_id": row.id,
|
||||
"provider": row.provider,
|
||||
"status": row.status,
|
||||
"schema_version": row.schema_version,
|
||||
"row_count": row.row_count,
|
||||
"activities": activities,
|
||||
"created_at": _iso(row.created_at),
|
||||
"updated_at": _iso(row.updated_at),
|
||||
"retired_at": _iso(row.deleted_at),
|
||||
},
|
||||
observed_at=row.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def _acquisition_attribution(row: ConnectorSanctionsAcquisitionRun) -> DsarRecordRef:
|
||||
return _record(
|
||||
resource_type="acquisition_actor_attribution",
|
||||
resource_id=row.id,
|
||||
title="Connector acquisition actor attribution",
|
||||
data={
|
||||
"acquisition_id": row.id,
|
||||
"provider_id": row.provider_id,
|
||||
"source_id": row.source_id,
|
||||
"status": row.status,
|
||||
"attempt_count": row.attempt_count,
|
||||
"started_at": _iso(row.started_at),
|
||||
"finished_at": _iso(row.finished_at),
|
||||
"activity": "started_source_acquisition",
|
||||
},
|
||||
observed_at=row.finished_at or row.started_at,
|
||||
)
|
||||
|
||||
|
||||
def _definition_attribution(
|
||||
row: ConnectorDefinitionRevision, definition: ConnectorDefinition
|
||||
) -> DsarRecordRef:
|
||||
return _record(
|
||||
resource_type="definition_actor_attribution",
|
||||
resource_id=row.id,
|
||||
title="Connector definition actor attribution",
|
||||
data={
|
||||
"definition_id": definition.id,
|
||||
"revision_id": row.id,
|
||||
"revision": row.revision,
|
||||
"origin": row.origin,
|
||||
"activity": "created_definition_revision",
|
||||
"created_at": _iso(row.created_at),
|
||||
},
|
||||
observed_at=row.created_at,
|
||||
)
|
||||
|
||||
|
||||
def _configuration_attribution(row: ConnectorConfiguration) -> DsarRecordRef:
|
||||
return _record(
|
||||
resource_type="configuration_actor_attribution",
|
||||
resource_id=row.id,
|
||||
title="Connector configuration actor attribution",
|
||||
data={
|
||||
"configuration_id": row.id,
|
||||
"definition_id": row.definition_id,
|
||||
"status": row.status,
|
||||
"base_definition_revision": row.base_definition_revision,
|
||||
"resource_revision": row.resource_revision,
|
||||
"ambiguity_policy": row.ambiguity_policy,
|
||||
"activity": "updated_connector_configuration",
|
||||
"updated_at": _iso(row.updated_at),
|
||||
},
|
||||
observed_at=row.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def _simulation_attribution(
|
||||
row: ConnectorSimulationRun, account_id: str
|
||||
) -> DsarRecordRef:
|
||||
activities = []
|
||||
if row.created_by == account_id:
|
||||
activities.append("created_simulation")
|
||||
if row.reviewed_by == account_id:
|
||||
activities.append("reviewed_simulation")
|
||||
return _record(
|
||||
resource_type="simulation_actor_attribution",
|
||||
resource_id=row.id,
|
||||
title="Connector simulation actor attribution",
|
||||
data={
|
||||
"simulation_id": row.id,
|
||||
"configuration_id": row.configuration_id,
|
||||
"mode": row.mode,
|
||||
"status": row.status,
|
||||
"review_state": row.review_state,
|
||||
"definition_revision": row.definition_revision,
|
||||
"configuration_revision": row.configuration_revision,
|
||||
"activities": activities,
|
||||
"created_at": _iso(row.created_at),
|
||||
"reviewed_at": _iso(row.reviewed_at),
|
||||
},
|
||||
observed_at=row.reviewed_at or row.created_at,
|
||||
)
|
||||
|
||||
|
||||
def _record(
|
||||
*,
|
||||
resource_type: str,
|
||||
resource_id: str,
|
||||
title: str,
|
||||
data: dict[str, object],
|
||||
observed_at: datetime | None,
|
||||
) -> DsarRecordRef:
|
||||
return DsarRecordRef(
|
||||
provider_id="connectors",
|
||||
module_id="connectors",
|
||||
resource_type=resource_type,
|
||||
resource_id=resource_id,
|
||||
category="connector_governance_attribution",
|
||||
title=title,
|
||||
data=data,
|
||||
observed_at=_aware(observed_at),
|
||||
immutable_evidence=True,
|
||||
retention_reason=(
|
||||
"Connector attribution is retained for configuration, review, and "
|
||||
"external-operation accountability."
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _limited(query, first, second, *, label: str):
|
||||
rows = query.order_by(first, second).limit(_MAX_RECORDS + 1).all()
|
||||
if len(rows) > _MAX_RECORDS:
|
||||
raise ValueError(f"Connectors DSAR {label} limit exceeded; narrow selectors.")
|
||||
return rows
|
||||
|
||||
|
||||
def _coalesce(*values: str | None) -> str | None | object:
|
||||
normalized = {str(value).strip() for value in values if str(value or "").strip()}
|
||||
if len(normalized) > 1:
|
||||
return _CONFLICT
|
||||
return next(iter(normalized), None)
|
||||
|
||||
|
||||
def _optional_string(value: object) -> str | None:
|
||||
return value if isinstance(value, str) and value else None
|
||||
|
||||
|
||||
def _iso(value: datetime | None) -> str | None:
|
||||
aware = _aware(value)
|
||||
return aware.isoformat() if aware else None
|
||||
|
||||
|
||||
def _aware(value: datetime | None) -> datetime | None:
|
||||
if value is None or value.tzinfo is not None:
|
||||
return value
|
||||
return value.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _session(value: object) -> Session:
|
||||
if not isinstance(value, Session):
|
||||
raise TypeError("Connectors DSAR requires a SQLAlchemy Session.")
|
||||
return value
|
||||
|
||||
|
||||
_RESOURCE_TYPES = {
|
||||
"source_actor_attribution",
|
||||
"acquisition_actor_attribution",
|
||||
"definition_actor_attribution",
|
||||
"configuration_actor_attribution",
|
||||
"simulation_actor_attribution",
|
||||
}
|
||||
|
||||
|
||||
def _validate_record(record: DsarRecordRef) -> None:
|
||||
if record.provider_id != "connectors" or record.module_id != "connectors":
|
||||
raise ValueError("Connectors DSAR cannot plan a foreign provider record.")
|
||||
if record.resource_type not in _RESOURCE_TYPES or not record.resource_id:
|
||||
raise ValueError("Connectors DSAR record identity is invalid.")
|
||||
|
||||
|
||||
def _validate_action(action: DsarErasureActionRef) -> None:
|
||||
if action.provider_id != "connectors" or action.module_id != "connectors":
|
||||
raise ValueError("Connectors DSAR cannot execute a foreign provider action.")
|
||||
if not action.action_id.startswith("connectors:retain:"):
|
||||
raise ValueError("Connectors DSAR action identity is invalid.")
|
||||
|
||||
|
||||
__all__ = ["CONNECTORS_DSAR_CAPABILITY", "ConnectorsDsarProvider"]
|
||||
@@ -24,6 +24,8 @@ from govoplan_core.security.http_fetch import fetch_http
|
||||
|
||||
MAX_FEED_BYTES = 5_000_000
|
||||
ATOM_NS = "http://www.w3.org/2005/Atom"
|
||||
FEED_PUBLISH_SCOPE = "connectors:feeds:publish"
|
||||
FEED_PRIVATE_PUBLISH_SCOPE = "connectors:feeds:publish_private"
|
||||
|
||||
|
||||
class ConnectorFeedProvider(FeedProvider):
|
||||
@@ -410,4 +412,10 @@ def _element(
|
||||
return child
|
||||
|
||||
|
||||
__all__ = ["ConnectorFeedProvider", "MAX_FEED_BYTES", "feed_rows"]
|
||||
__all__ = [
|
||||
"ConnectorFeedProvider",
|
||||
"FEED_PRIVATE_PUBLISH_SCOPE",
|
||||
"FEED_PUBLISH_SCOPE",
|
||||
"MAX_FEED_BYTES",
|
||||
"feed_rows",
|
||||
]
|
||||
|
||||
@@ -0,0 +1,921 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import json
|
||||
from typing import Any, Mapping, Sequence
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.connector_runtime import ConnectorContractError, ConnectorEndpoint
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorSimulationRun,
|
||||
)
|
||||
from govoplan_connectors.backend.governed_schemas import (
|
||||
ConnectorConfigurationCreateRequest,
|
||||
ConnectorConfigurationItem,
|
||||
ConnectorConfigurationUpdateRequest,
|
||||
ConnectorDefinitionItem,
|
||||
ConnectorDefinitionUpsertRequest,
|
||||
ConnectorReviewRequest,
|
||||
ConnectorRunItem,
|
||||
ConnectorRunRequest,
|
||||
GovernedConnectorSpecification,
|
||||
)
|
||||
|
||||
|
||||
class GovernedConnectorError(ValueError):
|
||||
def __init__(self, code: str, message: str) -> None:
|
||||
super().__init__(message)
|
||||
self.code = code
|
||||
|
||||
|
||||
def list_definitions(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
) -> list[ConnectorDefinitionItem]:
|
||||
rows = (
|
||||
session.query(ConnectorDefinition)
|
||||
.filter(
|
||||
ConnectorDefinition.tenant_id == tenant_id,
|
||||
ConnectorDefinition.status == "active",
|
||||
)
|
||||
.order_by(ConnectorDefinition.name.asc())
|
||||
.all()
|
||||
)
|
||||
return [
|
||||
definition_item(
|
||||
session,
|
||||
row,
|
||||
_definition_revision(session, row, row.current_revision),
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
|
||||
def upsert_definition(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
payload: ConnectorDefinitionUpsertRequest,
|
||||
) -> ConnectorDefinitionItem:
|
||||
specification = payload.specification.model_dump(mode="json")
|
||||
definition_hash = _hash(specification)
|
||||
definition = (
|
||||
session.query(ConnectorDefinition)
|
||||
.filter(
|
||||
ConnectorDefinition.tenant_id == principal.tenant_id,
|
||||
ConnectorDefinition.definition_key == payload.definition_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
expected_local = payload.origin == "local"
|
||||
if definition is None:
|
||||
definition = ConnectorDefinition(
|
||||
tenant_id=principal.tenant_id,
|
||||
definition_key=payload.definition_key,
|
||||
name=payload.name.strip(),
|
||||
description=_optional_text(payload.description),
|
||||
status="active",
|
||||
current_revision=0,
|
||||
source_package=payload.package_ref,
|
||||
local_definition=expected_local,
|
||||
)
|
||||
session.add(definition)
|
||||
session.flush()
|
||||
elif definition.local_definition != expected_local:
|
||||
if definition.local_definition:
|
||||
raise GovernedConnectorError(
|
||||
"local_definition_protected",
|
||||
"A package update cannot replace a locally owned connector definition.",
|
||||
)
|
||||
raise GovernedConnectorError(
|
||||
"package_definition_requires_overrides",
|
||||
"Use configuration overrides instead of converting a package definition into a local definition.",
|
||||
)
|
||||
else:
|
||||
current = _definition_revision(
|
||||
session,
|
||||
definition,
|
||||
definition.current_revision,
|
||||
)
|
||||
if current.definition_hash == definition_hash:
|
||||
return definition_item(session, definition, current)
|
||||
|
||||
definition.name = payload.name.strip()
|
||||
definition.description = _optional_text(payload.description)
|
||||
definition.source_package = payload.package_ref
|
||||
definition.current_revision += 1
|
||||
revision = ConnectorDefinitionRevision(
|
||||
definition_id=definition.id,
|
||||
revision=definition.current_revision,
|
||||
specification=specification,
|
||||
definition_hash=definition_hash,
|
||||
origin=payload.origin,
|
||||
package_ref=payload.package_ref,
|
||||
created_by=principal.user.id,
|
||||
)
|
||||
session.add(revision)
|
||||
session.flush()
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="connectors.definition.revision_created",
|
||||
object_type="connector_definition",
|
||||
object_id=definition.id,
|
||||
details={
|
||||
"definition_key": definition.definition_key,
|
||||
"revision": revision.revision,
|
||||
"definition_hash": definition_hash,
|
||||
"origin": payload.origin,
|
||||
"package_ref": payload.package_ref,
|
||||
"provider": payload.specification.provider,
|
||||
"protocol": payload.specification.protocol,
|
||||
"capabilities": sorted(payload.specification.capabilities),
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return definition_item(session, definition, revision)
|
||||
|
||||
|
||||
def list_configurations(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
) -> list[ConnectorConfigurationItem]:
|
||||
rows = (
|
||||
session.query(ConnectorConfiguration)
|
||||
.filter(ConnectorConfiguration.tenant_id == tenant_id)
|
||||
.order_by(ConnectorConfiguration.name.asc())
|
||||
.all()
|
||||
)
|
||||
return [configuration_item(session, row) for row in rows]
|
||||
|
||||
|
||||
def create_configuration(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
payload: ConnectorConfigurationCreateRequest,
|
||||
) -> ConnectorConfigurationItem:
|
||||
definition = _tenant_definition(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
definition_id=payload.definition_id,
|
||||
)
|
||||
_validate_endpoint(payload.endpoint_url, payload.credential_ref)
|
||||
revision = _definition_revision(session, definition, definition.current_revision)
|
||||
overrides = copy.deepcopy(payload.local_overrides)
|
||||
effective = _effective_specification(revision.specification, overrides)
|
||||
item = ConnectorConfiguration(
|
||||
tenant_id=principal.tenant_id,
|
||||
definition_id=definition.id,
|
||||
name=payload.name.strip(),
|
||||
status=payload.status,
|
||||
endpoint_url=_optional_text(payload.endpoint_url),
|
||||
credential_ref=_optional_text(payload.credential_ref),
|
||||
base_definition_revision=definition.current_revision,
|
||||
local_overrides=overrides,
|
||||
protected_paths=_protected_paths(overrides),
|
||||
effective_configuration=effective,
|
||||
effective_hash=_hash(effective),
|
||||
resource_revision=1,
|
||||
ambiguity_policy=payload.ambiguity_policy,
|
||||
updated_by=principal.user.id,
|
||||
)
|
||||
session.add(item)
|
||||
session.flush()
|
||||
_audit_configuration(
|
||||
session,
|
||||
principal,
|
||||
item,
|
||||
action="connectors.configuration.created",
|
||||
)
|
||||
session.commit()
|
||||
return configuration_item(session, item)
|
||||
|
||||
|
||||
def update_configuration(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
configuration_id: str,
|
||||
payload: ConnectorConfigurationUpdateRequest,
|
||||
) -> ConnectorConfigurationItem:
|
||||
item = _tenant_configuration(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
configuration_id=configuration_id,
|
||||
)
|
||||
if item.resource_revision != payload.expected_revision:
|
||||
raise GovernedConnectorError(
|
||||
"configuration_conflict",
|
||||
"The connector configuration changed; reload it before saving.",
|
||||
)
|
||||
supplied = payload.model_fields_set
|
||||
definition = _tenant_definition(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
definition_id=item.definition_id,
|
||||
)
|
||||
if "name" in supplied and payload.name is not None:
|
||||
item.name = payload.name.strip()
|
||||
if "endpoint_url" in supplied:
|
||||
item.endpoint_url = _optional_text(payload.endpoint_url)
|
||||
if "credential_ref" in supplied:
|
||||
item.credential_ref = _optional_text(payload.credential_ref)
|
||||
if "local_overrides" in supplied and payload.local_overrides is not None:
|
||||
item.local_overrides = copy.deepcopy(payload.local_overrides)
|
||||
if payload.ambiguity_policy is not None:
|
||||
item.ambiguity_policy = payload.ambiguity_policy
|
||||
if payload.status is not None:
|
||||
item.status = payload.status
|
||||
if payload.adopt_latest_definition:
|
||||
item.base_definition_revision = definition.current_revision
|
||||
_validate_endpoint(item.endpoint_url, item.credential_ref)
|
||||
base = _definition_revision(
|
||||
session,
|
||||
definition,
|
||||
item.base_definition_revision,
|
||||
)
|
||||
effective = _effective_specification(base.specification, item.local_overrides)
|
||||
item.protected_paths = _protected_paths(item.local_overrides)
|
||||
item.effective_configuration = effective
|
||||
item.effective_hash = _hash(effective)
|
||||
item.resource_revision += 1
|
||||
item.updated_by = principal.user.id
|
||||
_audit_configuration(
|
||||
session,
|
||||
principal,
|
||||
item,
|
||||
action="connectors.configuration.updated",
|
||||
extra={"adopted_latest_definition": payload.adopt_latest_definition},
|
||||
)
|
||||
session.commit()
|
||||
return configuration_item(session, item)
|
||||
|
||||
|
||||
def execute_run(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
configuration_id: str,
|
||||
mode: str,
|
||||
payload: ConnectorRunRequest,
|
||||
) -> ConnectorRunItem:
|
||||
if mode not in {"dry_run", "simulation"}:
|
||||
raise GovernedConnectorError("invalid_mode", "Unsupported connector run mode.")
|
||||
configuration = _tenant_configuration(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
configuration_id=configuration_id,
|
||||
)
|
||||
if configuration.status == "disabled":
|
||||
raise GovernedConnectorError(
|
||||
"configuration_disabled",
|
||||
"Disabled connector configurations cannot be executed.",
|
||||
)
|
||||
specification = GovernedConnectorSpecification.model_validate(
|
||||
configuration.effective_configuration
|
||||
)
|
||||
if mode == "dry_run" and not specification.dry_run.supported:
|
||||
raise GovernedConnectorError(
|
||||
"dry_run_unsupported",
|
||||
"This connector definition does not support dry runs.",
|
||||
)
|
||||
if mode == "simulation" and not specification.dry_run.simulation_supported:
|
||||
raise GovernedConnectorError(
|
||||
"simulation_unsupported",
|
||||
"This connector definition does not support simulation.",
|
||||
)
|
||||
rows = (
|
||||
list(payload.input_rows)
|
||||
if payload.input_rows is not None
|
||||
else list(specification.dry_run.sample_rows)
|
||||
)
|
||||
request_payload = {
|
||||
"mode": mode,
|
||||
"configuration_id": configuration.id,
|
||||
"configuration_revision": configuration.resource_revision,
|
||||
"configuration_hash": configuration.effective_hash,
|
||||
"external_revision": payload.external_revision,
|
||||
"input_rows": rows,
|
||||
}
|
||||
request_hash = _hash(request_payload)
|
||||
existing = (
|
||||
session.query(ConnectorSimulationRun)
|
||||
.filter(
|
||||
ConnectorSimulationRun.tenant_id == principal.tenant_id,
|
||||
ConnectorSimulationRun.configuration_id == configuration.id,
|
||||
ConnectorSimulationRun.mode == mode,
|
||||
ConnectorSimulationRun.idempotency_key == payload.idempotency_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if existing is not None:
|
||||
if existing.request_hash != request_hash:
|
||||
raise GovernedConnectorError(
|
||||
"idempotency_conflict",
|
||||
"The idempotency key was already used with different run inputs.",
|
||||
)
|
||||
return run_item(existing)
|
||||
|
||||
limit = specification.dry_run.max_items
|
||||
truncated = len(rows) > limit
|
||||
bounded_rows = rows[:limit]
|
||||
effects, diagnostics, ambiguous_count = _simulate(
|
||||
bounded_rows,
|
||||
specification,
|
||||
)
|
||||
if truncated:
|
||||
diagnostics.append(
|
||||
{
|
||||
"severity": "warning",
|
||||
"code": "connectors.run.truncated",
|
||||
"message": f"The run was limited to {limit} input rows.",
|
||||
"stage": "planning",
|
||||
"retryable": False,
|
||||
}
|
||||
)
|
||||
errors = sum(1 for item in diagnostics if item["severity"] == "error")
|
||||
if ambiguous_count:
|
||||
status_value, review_state = {
|
||||
"manual_review": ("manual_review", "pending"),
|
||||
"quarantine": ("quarantined", "quarantined"),
|
||||
"reject": ("rejected", "not_required"),
|
||||
}[configuration.ambiguity_policy]
|
||||
elif errors:
|
||||
status_value, review_state = "invalid", "not_required"
|
||||
else:
|
||||
status_value, review_state = "ready", "not_required"
|
||||
input_hash = _hash(bounded_rows)
|
||||
summary = {
|
||||
"total": len(effects),
|
||||
"creates": sum(1 for item in effects if item["effect"] == "create"),
|
||||
"updates": 0,
|
||||
"deletes": 0,
|
||||
"conflicts": sum(1 for item in effects if item["effect"] == "conflict"),
|
||||
"unchanged": 0,
|
||||
"ignored": sum(1 for item in effects if item["effect"] == "ignored"),
|
||||
"ambiguous": ambiguous_count,
|
||||
"errors": errors,
|
||||
"truncated": truncated,
|
||||
}
|
||||
now = datetime.now(UTC)
|
||||
provenance = {
|
||||
"contract_version": "1.0",
|
||||
"definition_id": configuration.definition_id,
|
||||
"definition_revision": configuration.base_definition_revision,
|
||||
"configuration_id": configuration.id,
|
||||
"configuration_revision": configuration.resource_revision,
|
||||
"configuration_hash": configuration.effective_hash,
|
||||
"mapping_version": specification.mapping.version,
|
||||
"input_hash": input_hash,
|
||||
"external_revision": payload.external_revision,
|
||||
"generated_at": now.isoformat(),
|
||||
"actor_id": principal.user.id,
|
||||
"mode": mode,
|
||||
"provider": specification.provider,
|
||||
"protocol": specification.protocol,
|
||||
"privacy_classification": specification.privacy_classification,
|
||||
"retention_class": specification.retention_class,
|
||||
}
|
||||
run = ConnectorSimulationRun(
|
||||
tenant_id=principal.tenant_id,
|
||||
configuration_id=configuration.id,
|
||||
mode=mode,
|
||||
idempotency_key=payload.idempotency_key,
|
||||
request_hash=request_hash,
|
||||
status=status_value,
|
||||
review_state=review_state,
|
||||
definition_revision=configuration.base_definition_revision,
|
||||
configuration_revision=configuration.resource_revision,
|
||||
configuration_hash=configuration.effective_hash,
|
||||
input_hash=input_hash,
|
||||
summary=summary,
|
||||
effects=effects,
|
||||
diagnostics=diagnostics,
|
||||
provenance=provenance,
|
||||
created_by=principal.user.id,
|
||||
)
|
||||
session.add(run)
|
||||
session.flush()
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=f"connectors.configuration.{mode}_completed",
|
||||
object_type="connector_simulation_run",
|
||||
object_id=run.id,
|
||||
details={
|
||||
"configuration_id": configuration.id,
|
||||
"configuration_revision": configuration.resource_revision,
|
||||
"configuration_hash": configuration.effective_hash,
|
||||
"definition_revision": configuration.base_definition_revision,
|
||||
"input_hash": input_hash,
|
||||
"status": status_value,
|
||||
"review_state": review_state,
|
||||
"summary": summary,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return run_item(run)
|
||||
|
||||
|
||||
def list_runs(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
configuration_id: str | None = None,
|
||||
review_state: str | None = None,
|
||||
limit: int = 100,
|
||||
) -> list[ConnectorRunItem]:
|
||||
query = session.query(ConnectorSimulationRun).filter(
|
||||
ConnectorSimulationRun.tenant_id == tenant_id
|
||||
)
|
||||
if configuration_id:
|
||||
query = query.filter(
|
||||
ConnectorSimulationRun.configuration_id == configuration_id
|
||||
)
|
||||
if review_state:
|
||||
query = query.filter(ConnectorSimulationRun.review_state == review_state)
|
||||
rows = (
|
||||
query.order_by(ConnectorSimulationRun.created_at.desc())
|
||||
.limit(max(1, min(int(limit), 500)))
|
||||
.all()
|
||||
)
|
||||
return [run_item(row) for row in rows]
|
||||
|
||||
|
||||
def review_run(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
run_id: str,
|
||||
payload: ConnectorReviewRequest,
|
||||
) -> ConnectorRunItem:
|
||||
run = (
|
||||
session.query(ConnectorSimulationRun)
|
||||
.filter(
|
||||
ConnectorSimulationRun.id == run_id,
|
||||
ConnectorSimulationRun.tenant_id == principal.tenant_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if run is None:
|
||||
raise GovernedConnectorError("run_not_found", "Connector run not found.")
|
||||
if run.review_state not in {"pending", "quarantined"}:
|
||||
raise GovernedConnectorError(
|
||||
"run_not_reviewable",
|
||||
"Only pending or quarantined connector results can be reviewed.",
|
||||
)
|
||||
run.review_state = payload.decision
|
||||
run.status = f"review_{payload.decision}"
|
||||
run.reviewed_by = principal.user.id
|
||||
run.reviewed_at = datetime.now(UTC)
|
||||
run.review_reason = payload.reason.strip()
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=f"connectors.simulation.{payload.decision}",
|
||||
object_type="connector_simulation_run",
|
||||
object_id=run.id,
|
||||
details={
|
||||
"configuration_id": run.configuration_id,
|
||||
"input_hash": run.input_hash,
|
||||
"configuration_hash": run.configuration_hash,
|
||||
"decision": payload.decision,
|
||||
"reason": run.review_reason,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return run_item(run)
|
||||
|
||||
|
||||
def definition_item(
|
||||
session: Session,
|
||||
definition: ConnectorDefinition,
|
||||
revision: ConnectorDefinitionRevision,
|
||||
) -> ConnectorDefinitionItem:
|
||||
del session
|
||||
return ConnectorDefinitionItem(
|
||||
id=definition.id,
|
||||
tenant_id=definition.tenant_id,
|
||||
definition_key=definition.definition_key,
|
||||
name=definition.name,
|
||||
description=definition.description,
|
||||
status=definition.status,
|
||||
current_revision=definition.current_revision,
|
||||
source_package=definition.source_package,
|
||||
local_definition=definition.local_definition,
|
||||
revision_id=revision.id,
|
||||
definition_hash=revision.definition_hash,
|
||||
origin=revision.origin,
|
||||
package_ref=revision.package_ref,
|
||||
specification=GovernedConnectorSpecification.model_validate(
|
||||
revision.specification
|
||||
),
|
||||
created_at=definition.created_at,
|
||||
updated_at=definition.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def configuration_item(
|
||||
session: Session,
|
||||
item: ConnectorConfiguration,
|
||||
) -> ConnectorConfigurationItem:
|
||||
definition = _tenant_definition(
|
||||
session,
|
||||
tenant_id=item.tenant_id,
|
||||
definition_id=item.definition_id,
|
||||
)
|
||||
return ConnectorConfigurationItem(
|
||||
id=item.id,
|
||||
tenant_id=item.tenant_id,
|
||||
definition_id=item.definition_id,
|
||||
definition_key=definition.definition_key,
|
||||
definition_name=definition.name,
|
||||
name=item.name,
|
||||
status=item.status,
|
||||
endpoint_url=item.endpoint_url,
|
||||
credential_ref=item.credential_ref,
|
||||
base_definition_revision=item.base_definition_revision,
|
||||
latest_definition_revision=definition.current_revision,
|
||||
update_available=definition.current_revision > item.base_definition_revision,
|
||||
local_overrides=dict(item.local_overrides or {}),
|
||||
protected_paths=list(item.protected_paths or []),
|
||||
effective_configuration=dict(item.effective_configuration or {}),
|
||||
effective_hash=item.effective_hash,
|
||||
resource_revision=item.resource_revision,
|
||||
ambiguity_policy=item.ambiguity_policy,
|
||||
updated_at=item.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def run_item(run: ConnectorSimulationRun) -> ConnectorRunItem:
|
||||
return ConnectorRunItem(
|
||||
id=run.id,
|
||||
tenant_id=run.tenant_id,
|
||||
configuration_id=run.configuration_id,
|
||||
mode=run.mode, # type: ignore[arg-type]
|
||||
idempotency_key=run.idempotency_key,
|
||||
status=run.status,
|
||||
review_state=run.review_state,
|
||||
definition_revision=run.definition_revision,
|
||||
configuration_revision=run.configuration_revision,
|
||||
configuration_hash=run.configuration_hash,
|
||||
input_hash=run.input_hash,
|
||||
summary=dict(run.summary or {}),
|
||||
effects=list(run.effects or []),
|
||||
diagnostics=list(run.diagnostics or []),
|
||||
provenance=dict(run.provenance or {}),
|
||||
reviewed_by=run.reviewed_by,
|
||||
reviewed_at=run.reviewed_at,
|
||||
review_reason=run.review_reason,
|
||||
created_at=run.created_at,
|
||||
)
|
||||
|
||||
|
||||
def _simulate(
|
||||
rows: Sequence[Mapping[str, Any]],
|
||||
specification: GovernedConnectorSpecification,
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int]:
|
||||
mapped_rows: list[dict[str, Any]] = []
|
||||
row_errors: dict[int, list[dict[str, Any]]] = {}
|
||||
diagnostics: list[dict[str, Any]] = []
|
||||
for index, row in enumerate(rows):
|
||||
mapped: dict[str, Any] = {}
|
||||
for rule in specification.mapping.rules:
|
||||
value, found = _path_value(row, rule.source)
|
||||
if not found:
|
||||
value = rule.default
|
||||
if rule.required and value in (None, ""):
|
||||
diagnostic = _diagnostic(
|
||||
"error",
|
||||
"connectors.mapping.required_source_missing",
|
||||
f"Required source field {rule.source!r} is missing.",
|
||||
"mapping",
|
||||
index=index,
|
||||
field=rule.source,
|
||||
)
|
||||
row_errors.setdefault(index, []).append(diagnostic)
|
||||
diagnostics.append(diagnostic)
|
||||
_set_path(mapped, rule.target, value)
|
||||
mapped_rows.append(mapped)
|
||||
|
||||
ambiguous_indexes: set[int] = set()
|
||||
for rule in specification.validation_rules:
|
||||
if rule.kind == "unique":
|
||||
seen: dict[str, list[int]] = {}
|
||||
for index, row in enumerate(mapped_rows):
|
||||
value, found = _path_value(row, rule.field)
|
||||
if found and value not in (None, ""):
|
||||
seen.setdefault(_stable_value(value), []).append(index)
|
||||
for indexes in seen.values():
|
||||
if len(indexes) > 1:
|
||||
ambiguous_indexes.update(indexes)
|
||||
for index in indexes:
|
||||
diagnostic = _diagnostic(
|
||||
rule.severity,
|
||||
rule.code,
|
||||
rule.message,
|
||||
"validation",
|
||||
index=index,
|
||||
field=rule.field,
|
||||
)
|
||||
row_errors.setdefault(index, []).append(diagnostic)
|
||||
diagnostics.append(diagnostic)
|
||||
continue
|
||||
for index, row in enumerate(mapped_rows):
|
||||
value, found = _path_value(row, rule.field)
|
||||
invalid = (
|
||||
rule.kind == "required" and (not found or value in (None, ""))
|
||||
) or (
|
||||
rule.kind == "one_of" and found and value not in rule.values
|
||||
)
|
||||
if invalid:
|
||||
diagnostic = _diagnostic(
|
||||
rule.severity,
|
||||
rule.code,
|
||||
rule.message,
|
||||
"validation",
|
||||
index=index,
|
||||
field=rule.field,
|
||||
)
|
||||
row_errors.setdefault(index, []).append(diagnostic)
|
||||
diagnostics.append(diagnostic)
|
||||
|
||||
redacted = set(specification.dry_run.redacted_fields)
|
||||
effects: list[dict[str, Any]] = []
|
||||
for index, mapped in enumerate(mapped_rows):
|
||||
errors = row_errors.get(index, [])
|
||||
has_error = any(item["severity"] == "error" for item in errors)
|
||||
effect = "conflict" if has_error or index in ambiguous_indexes else "create"
|
||||
effects.append(
|
||||
{
|
||||
"effect": effect,
|
||||
"source_object_ref": _source_ref(rows[index], index),
|
||||
"target_object_ref": None,
|
||||
"changed_fields": sorted(_leaf_paths(mapped)),
|
||||
"sample": _redact_fields(mapped, redacted),
|
||||
"reason_code": (
|
||||
"ambiguous_external_result"
|
||||
if index in ambiguous_indexes
|
||||
else errors[0]["code"] if errors else None
|
||||
),
|
||||
"outcome": "preview",
|
||||
"revision": specification.mapping.version,
|
||||
}
|
||||
)
|
||||
return effects, diagnostics, len(ambiguous_indexes)
|
||||
|
||||
|
||||
def _audit_configuration(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
item: ConnectorConfiguration,
|
||||
*,
|
||||
action: str,
|
||||
extra: Mapping[str, Any] | None = None,
|
||||
) -> None:
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action=action,
|
||||
object_type="connector_configuration",
|
||||
object_id=item.id,
|
||||
details={
|
||||
"definition_id": item.definition_id,
|
||||
"base_definition_revision": item.base_definition_revision,
|
||||
"resource_revision": item.resource_revision,
|
||||
"effective_hash": item.effective_hash,
|
||||
"protected_paths": list(item.protected_paths or []),
|
||||
"ambiguity_policy": item.ambiguity_policy,
|
||||
"status": item.status,
|
||||
"credential_reference_present": bool(item.credential_ref),
|
||||
**dict(extra or {}),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _tenant_definition(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
definition_id: str,
|
||||
) -> ConnectorDefinition:
|
||||
item = (
|
||||
session.query(ConnectorDefinition)
|
||||
.filter(
|
||||
ConnectorDefinition.id == definition_id,
|
||||
ConnectorDefinition.tenant_id == tenant_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if item is None:
|
||||
raise GovernedConnectorError(
|
||||
"definition_not_found",
|
||||
"Connector definition not found.",
|
||||
)
|
||||
return item
|
||||
|
||||
|
||||
def _tenant_configuration(
|
||||
session: Session,
|
||||
*,
|
||||
tenant_id: str,
|
||||
configuration_id: str,
|
||||
) -> ConnectorConfiguration:
|
||||
item = (
|
||||
session.query(ConnectorConfiguration)
|
||||
.filter(
|
||||
ConnectorConfiguration.id == configuration_id,
|
||||
ConnectorConfiguration.tenant_id == tenant_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if item is None:
|
||||
raise GovernedConnectorError(
|
||||
"configuration_not_found",
|
||||
"Connector configuration not found.",
|
||||
)
|
||||
return item
|
||||
|
||||
|
||||
def _definition_revision(
|
||||
session: Session,
|
||||
definition: ConnectorDefinition,
|
||||
revision: int,
|
||||
) -> ConnectorDefinitionRevision:
|
||||
item = (
|
||||
session.query(ConnectorDefinitionRevision)
|
||||
.filter(
|
||||
ConnectorDefinitionRevision.definition_id == definition.id,
|
||||
ConnectorDefinitionRevision.revision == revision,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if item is None:
|
||||
raise GovernedConnectorError(
|
||||
"definition_revision_not_found",
|
||||
"Connector definition revision not found.",
|
||||
)
|
||||
return item
|
||||
|
||||
|
||||
def _effective_specification(
|
||||
base: Mapping[str, Any],
|
||||
overrides: Mapping[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
merged = _deep_merge(base, overrides)
|
||||
return GovernedConnectorSpecification.model_validate(merged).model_dump(mode="json")
|
||||
|
||||
|
||||
def _deep_merge(
|
||||
base: Mapping[str, Any],
|
||||
overrides: Mapping[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
result = copy.deepcopy(dict(base))
|
||||
for key, value in overrides.items():
|
||||
if isinstance(value, Mapping) and isinstance(result.get(key), Mapping):
|
||||
result[key] = _deep_merge(result[key], value) # type: ignore[arg-type]
|
||||
else:
|
||||
result[key] = copy.deepcopy(value)
|
||||
return result
|
||||
|
||||
|
||||
def _protected_paths(value: Mapping[str, Any], prefix: str = "") -> list[str]:
|
||||
paths: list[str] = []
|
||||
for key in sorted(value):
|
||||
path = f"{prefix}.{key}" if prefix else str(key)
|
||||
item = value[key]
|
||||
if isinstance(item, Mapping) and item:
|
||||
paths.extend(_protected_paths(item, path))
|
||||
else:
|
||||
paths.append(path)
|
||||
return paths
|
||||
|
||||
|
||||
def _leaf_paths(value: Mapping[str, Any], prefix: str = "") -> set[str]:
|
||||
paths: set[str] = set()
|
||||
for key, item in value.items():
|
||||
path = f"{prefix}.{key}" if prefix else str(key)
|
||||
if isinstance(item, Mapping) and item:
|
||||
paths.update(_leaf_paths(item, path))
|
||||
else:
|
||||
paths.add(path)
|
||||
return paths
|
||||
|
||||
|
||||
def _path_value(value: Mapping[str, Any], path: str) -> tuple[Any, bool]:
|
||||
current: Any = value
|
||||
for part in path.split("."):
|
||||
if not isinstance(current, Mapping) or part not in current:
|
||||
return None, False
|
||||
current = current[part]
|
||||
return current, True
|
||||
|
||||
|
||||
def _set_path(target: dict[str, Any], path: str, value: Any) -> None:
|
||||
parts = path.split(".")
|
||||
current = target
|
||||
for part in parts[:-1]:
|
||||
nested = current.get(part)
|
||||
if not isinstance(nested, dict):
|
||||
nested = {}
|
||||
current[part] = nested
|
||||
current = nested
|
||||
current[parts[-1]] = value
|
||||
|
||||
|
||||
def _source_ref(row: Mapping[str, Any], index: int) -> str:
|
||||
for key in ("id", "external_id", "source_id"):
|
||||
value = row.get(key)
|
||||
if value not in (None, ""):
|
||||
return f"sample:{value}"
|
||||
return f"sample:row:{index + 1}"
|
||||
|
||||
|
||||
def _redact_fields(value: Mapping[str, Any], fields: set[str]) -> dict[str, Any]:
|
||||
result = copy.deepcopy(dict(value))
|
||||
for path in fields:
|
||||
parts = path.split(".")
|
||||
current: Any = result
|
||||
for part in parts[:-1]:
|
||||
if not isinstance(current, dict):
|
||||
break
|
||||
current = current.get(part)
|
||||
else:
|
||||
if isinstance(current, dict) and parts[-1] in current:
|
||||
current[parts[-1]] = "<redacted>"
|
||||
return result
|
||||
|
||||
|
||||
def _diagnostic(
|
||||
severity: str,
|
||||
code: str,
|
||||
message: str,
|
||||
stage: str,
|
||||
**details: Any,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"severity": severity,
|
||||
"code": code,
|
||||
"message": message,
|
||||
"stage": stage,
|
||||
"retryable": False,
|
||||
"details": details,
|
||||
}
|
||||
|
||||
|
||||
def _validate_endpoint(
|
||||
endpoint_url: str | None,
|
||||
credential_ref: str | None,
|
||||
) -> None:
|
||||
if endpoint_url:
|
||||
try:
|
||||
ConnectorEndpoint(
|
||||
url=endpoint_url,
|
||||
credential_ref=_optional_text(credential_ref),
|
||||
)
|
||||
except ConnectorContractError as exc:
|
||||
raise GovernedConnectorError("invalid_endpoint", str(exc)) from exc
|
||||
elif credential_ref:
|
||||
raise GovernedConnectorError(
|
||||
"credential_without_endpoint",
|
||||
"A credential reference requires a configured endpoint.",
|
||||
)
|
||||
|
||||
|
||||
def _stable_value(value: Any) -> str:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
|
||||
|
||||
|
||||
def _hash(value: Any) -> str:
|
||||
return hashlib.sha256(
|
||||
json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode()
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _optional_text(value: object | None) -> str | None:
|
||||
normalized = str(value or "").strip()
|
||||
return normalized or None
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GovernedConnectorError",
|
||||
"configuration_item",
|
||||
"create_configuration",
|
||||
"execute_run",
|
||||
"list_configurations",
|
||||
"list_definitions",
|
||||
"list_runs",
|
||||
"review_run",
|
||||
"run_item",
|
||||
"update_configuration",
|
||||
"upsert_definition",
|
||||
]
|
||||
@@ -0,0 +1,221 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Any, Literal
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||
|
||||
|
||||
class ConnectorMappingRule(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
source: str = Field(min_length=1, max_length=255)
|
||||
target: str = Field(min_length=1, max_length=255)
|
||||
required: bool = False
|
||||
default: Any = None
|
||||
|
||||
|
||||
class ConnectorMappingDefinition(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
version: str = Field(min_length=1, max_length=100)
|
||||
rules: list[ConnectorMappingRule] = Field(default_factory=list, max_length=500)
|
||||
|
||||
|
||||
class ConnectorValidationRule(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
kind: Literal["required", "one_of", "unique"]
|
||||
field: str = Field(min_length=1, max_length=255)
|
||||
values: list[Any] = Field(default_factory=list, max_length=500)
|
||||
severity: Literal["warning", "error"] = "error"
|
||||
code: str = Field(min_length=1, max_length=120)
|
||||
message: str = Field(min_length=1, max_length=500)
|
||||
|
||||
@model_validator(mode="after")
|
||||
def validate_values(self) -> "ConnectorValidationRule":
|
||||
if self.kind == "one_of" and not self.values:
|
||||
raise ValueError("one_of validation requires allowed values")
|
||||
if self.kind != "one_of" and self.values:
|
||||
raise ValueError("Only one_of validation accepts values")
|
||||
return self
|
||||
|
||||
|
||||
class ConnectorDryRunMetadata(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
supported: bool = True
|
||||
simulation_supported: bool = True
|
||||
sample_rows: list[dict[str, Any]] = Field(default_factory=list, max_length=500)
|
||||
max_items: int = Field(default=500, ge=1, le=10_000)
|
||||
redacted_fields: list[str] = Field(default_factory=list, max_length=100)
|
||||
|
||||
|
||||
class ConnectorAuditMetadata(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
event_prefix: str = Field(min_length=1, max_length=120)
|
||||
expected_events: list[str] = Field(default_factory=list, max_length=100)
|
||||
evidence_fields: list[str] = Field(default_factory=list, max_length=100)
|
||||
|
||||
|
||||
class GovernedConnectorSpecification(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
provider: str = Field(min_length=1, max_length=120)
|
||||
protocol: str = Field(min_length=1, max_length=80)
|
||||
capabilities: list[str] = Field(min_length=1, max_length=100)
|
||||
input_schema: dict[str, Any]
|
||||
output_schema: dict[str, Any]
|
||||
mapping: ConnectorMappingDefinition
|
||||
validation_rules: list[ConnectorValidationRule] = Field(
|
||||
default_factory=list,
|
||||
max_length=500,
|
||||
)
|
||||
dry_run: ConnectorDryRunMetadata
|
||||
audit: ConnectorAuditMetadata
|
||||
privacy_classification: Literal[
|
||||
"public",
|
||||
"internal",
|
||||
"confidential",
|
||||
"restricted",
|
||||
] = "internal"
|
||||
retention_class: str = Field(min_length=1, max_length=120)
|
||||
operational_limits: dict[str, Any] = Field(default_factory=dict)
|
||||
retry_policy: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class ConnectorDefinitionUpsertRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
definition_key: str = Field(pattern=r"^[a-z0-9][a-z0-9._-]{0,159}$")
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
description: str | None = Field(default=None, max_length=4000)
|
||||
origin: Literal["package", "local"] = "local"
|
||||
package_ref: str | None = Field(default=None, max_length=300)
|
||||
specification: GovernedConnectorSpecification
|
||||
|
||||
@model_validator(mode="after")
|
||||
def package_provenance(self) -> "ConnectorDefinitionUpsertRequest":
|
||||
if self.origin == "package" and not self.package_ref:
|
||||
raise ValueError("Package definitions require package_ref")
|
||||
if self.origin == "local" and self.package_ref:
|
||||
raise ValueError("Local definitions cannot claim package_ref")
|
||||
return self
|
||||
|
||||
|
||||
class ConnectorDefinitionItem(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
definition_key: str
|
||||
name: str
|
||||
description: str | None = None
|
||||
status: str
|
||||
current_revision: int
|
||||
source_package: str | None = None
|
||||
local_definition: bool
|
||||
revision_id: str
|
||||
definition_hash: str
|
||||
origin: str
|
||||
package_ref: str | None = None
|
||||
specification: GovernedConnectorSpecification
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class ConnectorDefinitionListResponse(BaseModel):
|
||||
items: list[ConnectorDefinitionItem]
|
||||
|
||||
|
||||
class ConnectorConfigurationCreateRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
definition_id: str = Field(min_length=1, max_length=36)
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
endpoint_url: str | None = Field(default=None, max_length=1500)
|
||||
credential_ref: str | None = Field(default=None, max_length=500)
|
||||
local_overrides: dict[str, Any] = Field(default_factory=dict)
|
||||
ambiguity_policy: Literal["manual_review", "quarantine", "reject"] = (
|
||||
"manual_review"
|
||||
)
|
||||
status: Literal["draft", "active", "disabled"] = "draft"
|
||||
|
||||
|
||||
class ConnectorConfigurationUpdateRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
expected_revision: int = Field(ge=1)
|
||||
name: str | None = Field(default=None, min_length=1, max_length=300)
|
||||
endpoint_url: str | None = Field(default=None, max_length=1500)
|
||||
credential_ref: str | None = Field(default=None, max_length=500)
|
||||
local_overrides: dict[str, Any] | None = None
|
||||
ambiguity_policy: Literal["manual_review", "quarantine", "reject"] | None = None
|
||||
status: Literal["draft", "active", "disabled"] | None = None
|
||||
adopt_latest_definition: bool = False
|
||||
|
||||
|
||||
class ConnectorConfigurationItem(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
definition_id: str
|
||||
definition_key: str
|
||||
definition_name: str
|
||||
name: str
|
||||
status: str
|
||||
endpoint_url: str | None = None
|
||||
credential_ref: str | None = None
|
||||
base_definition_revision: int
|
||||
latest_definition_revision: int
|
||||
update_available: bool
|
||||
local_overrides: dict[str, Any]
|
||||
protected_paths: list[str]
|
||||
effective_configuration: dict[str, Any]
|
||||
effective_hash: str
|
||||
resource_revision: int
|
||||
ambiguity_policy: str
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class ConnectorConfigurationListResponse(BaseModel):
|
||||
items: list[ConnectorConfigurationItem]
|
||||
|
||||
|
||||
class ConnectorRunRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||
input_rows: list[dict[str, Any]] | None = Field(default=None, max_length=500)
|
||||
external_revision: str | None = Field(default=None, max_length=255)
|
||||
|
||||
|
||||
class ConnectorRunItem(BaseModel):
|
||||
id: str
|
||||
tenant_id: str
|
||||
configuration_id: str
|
||||
mode: Literal["dry_run", "simulation"]
|
||||
idempotency_key: str
|
||||
status: str
|
||||
review_state: str
|
||||
definition_revision: int
|
||||
configuration_revision: int
|
||||
configuration_hash: str
|
||||
input_hash: str
|
||||
summary: dict[str, Any]
|
||||
effects: list[dict[str, Any]]
|
||||
diagnostics: list[dict[str, Any]]
|
||||
provenance: dict[str, Any]
|
||||
reviewed_by: str | None = None
|
||||
reviewed_at: datetime | None = None
|
||||
review_reason: str | None = None
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class ConnectorRunListResponse(BaseModel):
|
||||
items: list[ConnectorRunItem]
|
||||
|
||||
|
||||
class ConnectorReviewRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
decision: Literal["approved", "rejected"]
|
||||
reason: str = Field(min_length=5, max_length=1000)
|
||||
@@ -13,12 +13,16 @@ from govoplan_core.core.module_guards import (
|
||||
from govoplan_core.core.datasources import CAPABILITY_DATASOURCE_ORIGINS
|
||||
from govoplan_core.core.feeds import CAPABILITY_CONNECTORS_FEEDS
|
||||
from govoplan_core.core.modules import (
|
||||
CapabilityDocumentation,
|
||||
DocumentationTopic,
|
||||
FrontendModule,
|
||||
MigrationSpec,
|
||||
ModuleInterfaceProvider,
|
||||
ModuleInterfaceRequirement,
|
||||
ModuleManifest,
|
||||
PermissionDefinition,
|
||||
RoleTemplate,
|
||||
ViewSurface,
|
||||
)
|
||||
from govoplan_core.core.provider_governance import (
|
||||
ExternalProviderDeclaration,
|
||||
@@ -33,15 +37,24 @@ from govoplan_core.core.tabular_sources import (
|
||||
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER,
|
||||
CAPABILITY_CONNECTORS_TABULAR_SOURCES,
|
||||
)
|
||||
from govoplan_core.core.files import CAPABILITY_FILES_TABULAR_CONTENT
|
||||
from govoplan_core.core.sanctions import (
|
||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorSanctionsAcquisitionRun,
|
||||
ConnectorSanctionsSnapshot,
|
||||
ConnectorSimulationRun,
|
||||
ConnectorTabularSource,
|
||||
)
|
||||
from govoplan_connectors.backend.dsar_provider import (
|
||||
CONNECTORS_DSAR_CAPABILITY,
|
||||
ConnectorsDsarProvider,
|
||||
)
|
||||
from govoplan_connectors.backend.sanctions_sources import (
|
||||
SANCTIONS_READ_SCOPE,
|
||||
SANCTIONS_REFRESH_SCOPE,
|
||||
@@ -56,7 +69,11 @@ from govoplan_connectors.backend.tabular_sources import (
|
||||
from govoplan_connectors.backend.datasource_origins import (
|
||||
ConnectorDatasourceOriginProvider,
|
||||
)
|
||||
from govoplan_connectors.backend.feeds import ConnectorFeedProvider
|
||||
from govoplan_connectors.backend.feeds import (
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
FEED_PUBLISH_SCOPE,
|
||||
ConnectorFeedProvider,
|
||||
)
|
||||
from govoplan_connectors.backend.provider_state import (
|
||||
SANCTIONS_PROVIDER_ID,
|
||||
TABULAR_PROVIDER_ID,
|
||||
@@ -66,11 +83,11 @@ from govoplan_connectors.backend.provider_state import (
|
||||
|
||||
|
||||
MODULE_ID = "connectors"
|
||||
MODULE_VERSION = "0.1.18"
|
||||
MODULE_VERSION = "0.1.20"
|
||||
TABULAR_SOURCE_INTERFACE_VERSION = "0.1.0"
|
||||
DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0"
|
||||
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
|
||||
FEED_INTERFACE_VERSION = "0.1.0"
|
||||
FEED_INTERFACE_VERSION = "0.2.0"
|
||||
CONNECTOR_RUNTIME_INTERFACE_VERSION = "1.0.0"
|
||||
|
||||
ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||
@@ -83,6 +100,16 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||
reference="tests/test_tabular_sources.py",
|
||||
summary="Exercises tenant-safe immutable tabular snapshots and bounded reads.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_tabular_origin_provider.py",
|
||||
summary="Exercises exact managed-file versions, reviewed refresh, live SQL projection, configuration drift, and tenant isolation.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_tabular_adapters.py",
|
||||
summary="Exercises bounded CSV/XLSX parsing and the credential-governed read-only PostgreSQL adapter.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_sanctions_sources.py",
|
||||
@@ -93,6 +120,11 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||
reference="tests/test_recovery.py",
|
||||
summary="Proves atomic snapshot commits, idempotent replay, distributed fences, tamper rejection, and unknown external-effect handling.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_governed_runtime.py",
|
||||
summary="Exercises immutable definition revisions, protected local overrides, idempotent simulations, and explicit ambiguity review.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="documentation",
|
||||
reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md",
|
||||
@@ -100,8 +132,9 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||
),
|
||||
),
|
||||
known_limits=(
|
||||
"The executable generic datasource origin is an immutable tabular snapshot; database and arbitrary REST profiles remain future providers.",
|
||||
"Tabular origins support immutable snapshots, exact managed CSV/XLSX versions, and read-only PostgreSQL tables; arbitrary REST and other database adapters remain future providers.",
|
||||
"Feed publication renders a governed document but does not yet push it to an external publishing endpoint.",
|
||||
"The generic governed runtime simulates deterministic mapping and validation; provider-specific live writes remain owned by explicit connector adapters.",
|
||||
),
|
||||
supported_authority_modes=(
|
||||
"external_authoritative",
|
||||
@@ -137,7 +170,7 @@ EXTERNAL_PROVIDERS = (
|
||||
ExternalProviderDeclaration(
|
||||
id=TABULAR_PROVIDER_ID,
|
||||
module_id=MODULE_ID,
|
||||
label="Immutable tabular snapshot provider",
|
||||
label="Governed tabular source providers",
|
||||
maturity="read",
|
||||
operations=("discover", "search", "read", "preview", "dry_run"),
|
||||
objects=(
|
||||
@@ -149,10 +182,10 @@ EXTERNAL_PROVIDERS = (
|
||||
),
|
||||
),
|
||||
behavior=ProviderBehaviorDeclaration(
|
||||
revision_tokens="Source fingerprints and immutable snapshot ids are retained.",
|
||||
concurrency="Reads may require the expected fingerprint; snapshots never mutate in place.",
|
||||
freshness="Snapshot acquisition time and source timestamp are exposed.",
|
||||
health="Import validation and source-read failures are explicit.",
|
||||
revision_tokens="Source fingerprints, exact file versions, SQL configuration revisions, credential revisions, and discovery revisions are retained.",
|
||||
concurrency="Reads may require the expected fingerprint; file versions remain pinned and SQL configuration or schema drift blocks reads until reviewed refresh.",
|
||||
freshness="Snapshot acquisition time, exact/current file versions, and live SQL discovery health are exposed.",
|
||||
health="Import validation, Files access/integrity, SQL authentication, configuration drift, schema drift, timeouts, and source unavailability are explicit without exposing secrets.",
|
||||
max_read_items=1000,
|
||||
idempotency="Feed imports accept a caller request key and replay the same committed immutable source without refetching.",
|
||||
retry="Read-only acquisition may be retried only as a new deliberate request after a failed atomic operation.",
|
||||
@@ -166,7 +199,7 @@ EXTERNAL_PROVIDERS = (
|
||||
classifications=("internal", "confidential", "restricted"),
|
||||
purposes=("governed import", "dataflow input", "evidence reconstruction"),
|
||||
retention="Datasources or the consuming domain supplies retention and hold policy.",
|
||||
secret_handling="Generic snapshots contain no connector credential; transport credentials stay in credential envelopes.",
|
||||
secret_handling="Snapshots and managed-file sources contain no connector credential; PostgreSQL uses a scoped Core credential-envelope reference and never stores or returns the resolved secret.",
|
||||
),
|
||||
capability_names=(
|
||||
CAPABILITY_CONNECTORS_TABULAR_SOURCES,
|
||||
@@ -190,7 +223,12 @@ EXTERNAL_PROVIDERS = (
|
||||
objects=(
|
||||
ProviderObjectDeclaration(
|
||||
object_type="sanctions_source_snapshot",
|
||||
field_groups=("source_identity", "raw_evidence", "entries", "acquisition_health"),
|
||||
field_groups=(
|
||||
"source_identity",
|
||||
"raw_evidence",
|
||||
"entries",
|
||||
"acquisition_health",
|
||||
),
|
||||
authority_modes=("external_authoritative", "external_mirror"),
|
||||
default_authority_mode="external_mirror",
|
||||
),
|
||||
@@ -254,7 +292,17 @@ PERMISSIONS = (
|
||||
_permission(
|
||||
ADMIN_SCOPE,
|
||||
"Administer connector sources",
|
||||
"Manage every tenant connector source and future source policies.",
|
||||
"Manage tenant connector sources, versioned definitions, protected overrides, and review policies.",
|
||||
),
|
||||
_permission(
|
||||
FEED_PUBLISH_SCOPE,
|
||||
"Publish public feeds",
|
||||
"Render provenance-bearing public RSS or Atom output from selected GovOPlaN objects.",
|
||||
),
|
||||
_permission(
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
"Publish restricted feeds",
|
||||
"Render tenant or private RSS or Atom output after the owning product surface has authorized every selected object.",
|
||||
),
|
||||
_permission(
|
||||
SANCTIONS_READ_SCOPE,
|
||||
@@ -269,6 +317,20 @@ PERMISSIONS = (
|
||||
)
|
||||
|
||||
ROLE_TEMPLATES = (
|
||||
RoleTemplate(
|
||||
slug="connector_administrator",
|
||||
name="Connector administrator",
|
||||
description="Govern connector definitions, local configurations, simulations, and manual review.",
|
||||
permissions=(
|
||||
READ_SCOPE,
|
||||
WRITE_SCOPE,
|
||||
ADMIN_SCOPE,
|
||||
FEED_PUBLISH_SCOPE,
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
SANCTIONS_READ_SCOPE,
|
||||
SANCTIONS_REFRESH_SCOPE,
|
||||
),
|
||||
),
|
||||
RoleTemplate(
|
||||
slug="connector_source_manager",
|
||||
name="Connector source manager",
|
||||
@@ -276,6 +338,7 @@ ROLE_TEMPLATES = (
|
||||
permissions=(
|
||||
READ_SCOPE,
|
||||
WRITE_SCOPE,
|
||||
FEED_PUBLISH_SCOPE,
|
||||
SANCTIONS_READ_SCOPE,
|
||||
SANCTIONS_REFRESH_SCOPE,
|
||||
),
|
||||
@@ -296,11 +359,13 @@ def _router(_context):
|
||||
|
||||
|
||||
def _provider(_context) -> SqlTabularSourceProvider:
|
||||
return SqlTabularSourceProvider()
|
||||
return SqlTabularSourceProvider(registry=getattr(_context, "registry", None))
|
||||
|
||||
|
||||
def _datasource_origin_provider(_context) -> ConnectorDatasourceOriginProvider:
|
||||
return ConnectorDatasourceOriginProvider()
|
||||
return ConnectorDatasourceOriginProvider(
|
||||
SqlTabularSourceProvider(registry=getattr(_context, "registry", None))
|
||||
)
|
||||
|
||||
|
||||
def _sanctions_snapshot_provider(
|
||||
@@ -313,8 +378,27 @@ def _feed_provider(_context) -> ConnectorFeedProvider:
|
||||
return ConnectorFeedProvider()
|
||||
|
||||
|
||||
def _dsar_provider(_context) -> ConnectorsDsarProvider:
|
||||
return ConnectorsDsarProvider()
|
||||
|
||||
|
||||
def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
||||
return {
|
||||
"connector_definitions": (
|
||||
session.query(ConnectorDefinition)
|
||||
.filter(ConnectorDefinition.tenant_id == tenant_id)
|
||||
.count()
|
||||
),
|
||||
"connector_configurations": (
|
||||
session.query(ConnectorConfiguration)
|
||||
.filter(ConnectorConfiguration.tenant_id == tenant_id)
|
||||
.count()
|
||||
),
|
||||
"connector_simulation_runs": (
|
||||
session.query(ConnectorSimulationRun)
|
||||
.filter(ConnectorSimulationRun.tenant_id == tenant_id)
|
||||
.count()
|
||||
),
|
||||
"connector_tabular_sources": (
|
||||
session.query(ConnectorTabularSource)
|
||||
.filter(
|
||||
@@ -379,16 +463,57 @@ manifest = ModuleManifest(
|
||||
name="connectors.runtime_contract",
|
||||
version=CONNECTOR_RUNTIME_INTERFACE_VERSION,
|
||||
),
|
||||
ModuleInterfaceProvider(name=CONNECTORS_DSAR_CAPABILITY, version="0.1.0"),
|
||||
),
|
||||
requires_interfaces=(
|
||||
ModuleInterfaceRequirement(
|
||||
name=CAPABILITY_FILES_TABULAR_CONTENT,
|
||||
version_min="1.0.0",
|
||||
version_max_exclusive="2.0.0",
|
||||
optional=True,
|
||||
),
|
||||
),
|
||||
permissions=PERMISSIONS,
|
||||
role_templates=ROLE_TEMPLATES,
|
||||
route_factory=_router,
|
||||
frontend=FrontendModule(
|
||||
module_id=MODULE_ID,
|
||||
package_name="@govoplan/connectors-webui",
|
||||
view_surfaces=(
|
||||
ViewSurface(
|
||||
id="connectors.admin.governed-configurations",
|
||||
module_id=MODULE_ID,
|
||||
kind="section",
|
||||
label="Connector governance",
|
||||
order=45,
|
||||
),
|
||||
ViewSurface(
|
||||
id="connectors.admin.simulation-review",
|
||||
module_id=MODULE_ID,
|
||||
kind="section",
|
||||
label="Connector simulation review",
|
||||
parent_id="connectors.admin.governed-configurations",
|
||||
order=20,
|
||||
),
|
||||
),
|
||||
),
|
||||
capability_factories={
|
||||
CAPABILITY_CONNECTORS_TABULAR_SOURCES: _provider,
|
||||
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER: _provider,
|
||||
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
|
||||
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider),
|
||||
CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
|
||||
CONNECTORS_DSAR_CAPABILITY: _dsar_provider,
|
||||
},
|
||||
capability_documentation={
|
||||
CONNECTORS_DSAR_CAPABILITY: CapabilityDocumentation(
|
||||
label="Connector data-subject request provider",
|
||||
summary=(
|
||||
"Exports minimized operator attribution without connector secrets, "
|
||||
"external payloads, or transport evidence."
|
||||
),
|
||||
contract_version="0.1.0",
|
||||
),
|
||||
},
|
||||
tenant_summary_providers=(_tenant_summary,),
|
||||
architecture=ARCHITECTURE,
|
||||
@@ -411,6 +536,10 @@ manifest = ModuleManifest(
|
||||
script_location=str(Path(__file__).with_name("migrations") / "versions"),
|
||||
retirement_supported=True,
|
||||
retirement_provider=drop_table_retirement_provider(
|
||||
ConnectorSimulationRun,
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorDefinition,
|
||||
ConnectorSanctionsSnapshot,
|
||||
ConnectorSanctionsAcquisitionRun,
|
||||
ConnectorTabularSource,
|
||||
@@ -423,6 +552,10 @@ manifest = ModuleManifest(
|
||||
),
|
||||
uninstall_guard_providers=(
|
||||
persistent_table_uninstall_guard(
|
||||
ConnectorSimulationRun,
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorDefinition,
|
||||
ConnectorSanctionsSnapshot,
|
||||
ConnectorSanctionsAcquisitionRun,
|
||||
ConnectorTabularSource,
|
||||
@@ -430,6 +563,69 @@ manifest = ModuleManifest(
|
||||
),
|
||||
),
|
||||
documentation=(
|
||||
DocumentationTopic(
|
||||
id="connectors.data-subject-requests",
|
||||
title="Connector data-subject requests",
|
||||
summary=(
|
||||
"Export accountable connector activity without disclosing credentials "
|
||||
"or external data."
|
||||
),
|
||||
body=(
|
||||
"Connectors correlates only an exact tenant account identifier and can "
|
||||
"narrow an already verified search to one source, acquisition, "
|
||||
"definition, configuration, or simulation. The export identifies the "
|
||||
"subject's configuration, acquisition, simulation, and review activity "
|
||||
"using bounded lifecycle metadata. It never includes credential or "
|
||||
"endpoint references, source rows, external responses, request payloads, "
|
||||
"mapping and configuration documents, diagnostics, provenance, hashes, "
|
||||
"or transport evidence. Connector attribution remains immutable "
|
||||
"governance and external-operation evidence and is retained rather than "
|
||||
"automatically erased. Email or object identifiers without a verified "
|
||||
"account identifier do not establish a match."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("user", "operator", "module_admin", "auditor"),
|
||||
related_modules=("core", "access", "audit", "policy"),
|
||||
order=37,
|
||||
metadata={
|
||||
"help_contexts": ["privacy.data-subject-requests"],
|
||||
"consequence_classes": {
|
||||
"export_operator_attribution": (
|
||||
"Returns minimized connector activity for the exact account."
|
||||
),
|
||||
"exclude_connector_secrets": (
|
||||
"Never returns credentials, endpoints, external rows, or evidence payloads."
|
||||
),
|
||||
"retain_connector_evidence": (
|
||||
"Preserves configuration and external-operation accountability."
|
||||
),
|
||||
},
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="connectors.governed-configuration",
|
||||
title="Govern connector definitions and simulations",
|
||||
summary="Version connector schemas and mappings while preserving tenant-local overrides and review evidence.",
|
||||
body=(
|
||||
"Connector administrators create package-managed or local definitions that explicitly declare provider, protocol, capabilities, schemas, mapping rules, validation, preview support, audit expectations, privacy, retention, limits, and retry metadata. Every definition change creates an immutable revision. A tenant configuration pins one revision and stores only a credential reference; package updates remain available but do not change the effective configuration until an administrator adopts them. Local override leaf paths are displayed as protected and are reapplied when an update is adopted. Dry-runs and simulations are bounded, redact configured fields, are idempotent by caller key, and retain configuration, mapping, input, and external revision provenance. Ambiguous results follow the configuration policy: manual review, quarantine, or rejection. Pending and quarantined evidence requires an explicit approve or reject decision with a reason. Provider-specific live writes are not implied by a successful generic simulation."
|
||||
),
|
||||
layer="configured",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("operator", "module_admin", "integration_admin"),
|
||||
related_modules=("policy", "audit", "dataflow", "ops"),
|
||||
order=38,
|
||||
metadata={
|
||||
"kind": "guide",
|
||||
"help_contexts": ["connectors.admin.governed-configurations"],
|
||||
"prerequisites": [
|
||||
"A connector definition has been installed or authored.",
|
||||
"Credential material is stored outside the connector URL and referenced by an approved secret identifier.",
|
||||
],
|
||||
"outcome": "The active connector behavior is inspectable, version-pinned, testable, and reviewable before any provider-specific write.",
|
||||
"verification": "Reload the configuration, inspect protected paths and effective hash, run a simulation with a new idempotency key, and resolve any pending review result.",
|
||||
},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="connectors.authority-and-effects",
|
||||
title="Connector authority and effect behavior",
|
||||
@@ -472,11 +668,16 @@ manifest = ModuleManifest(
|
||||
"fingerprints, and bounded reads. Dataflow stores only opaque source "
|
||||
"references and expected fingerprints. Each source declares its live, "
|
||||
"cached, file-backed, or static mode, structured health, and supported "
|
||||
"projection, filter, aggregation, sorting, and pagination pushdown. The "
|
||||
"first executable provider imports immutable JSON or CSV snapshots, "
|
||||
"supports projection and pagination, and exposes them as Datasource "
|
||||
"origins. Database and API providers can implement the same origin "
|
||||
"contract without changing Datasources or Dataflow."
|
||||
"projection, filter, aggregation, sorting, and pagination pushdown. "
|
||||
"Immutable JSON/CSV snapshots remain available. Managed CSV/XLSX "
|
||||
"sources use the optional Files capability, pin an exact authorized "
|
||||
"version, apply archive and expansion limits, and require explicit "
|
||||
"refresh before adopting a newer version. The PostgreSQL adapter uses "
|
||||
"an active governed configuration and scoped Core credential envelope, "
|
||||
"reflects simple schema/table identifiers, runs read-only bounded "
|
||||
"projection and pagination, and blocks configuration, credential, or "
|
||||
"schema drift until reviewed refresh. Credentials, endpoints, storage "
|
||||
"keys, and raw file internals are never exposed through the origin."
|
||||
),
|
||||
layer="available",
|
||||
documentation_types=("admin", "user"),
|
||||
@@ -492,8 +693,13 @@ manifest = ModuleManifest(
|
||||
"Connectors owns bounded, SSRF-protected RSS/Atom transport and XML "
|
||||
"parsing. Imported entries become immutable tabular snapshots exposed "
|
||||
"through Datasources, including acquisition, freshness, ETag, content "
|
||||
"digest, and source provenance. Portal or Reporting owns publication "
|
||||
"routes and must pass the allowed visibility set when rendering output. "
|
||||
"digest, and source provenance. Emission accepts only provenance-bearing "
|
||||
"event, publication, case, or report selections from an owning surface. "
|
||||
"The requested audience determines the visibility ceiling: public output "
|
||||
"contains only public entries, while tenant or private output requires a "
|
||||
"separate restricted-feed permission. Callers cannot supply their own "
|
||||
"visibility allow-list. Portal or Reporting owns durable publication "
|
||||
"routes and must re-authorize access on every restricted feed request. "
|
||||
"A separate RSS module is only warranted if GovOPlaN later needs a "
|
||||
"dedicated feed-reader product surface."
|
||||
),
|
||||
|
||||
+214
@@ -0,0 +1,214 @@
|
||||
"""governed connector definitions and simulation evidence
|
||||
|
||||
Revision ID: a8d9e0f1b2c3
|
||||
Revises: f7c8d9e0a1b2
|
||||
Create Date: 2026-08-20 12:30:00.000000
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "a8d9e0f1b2c3"
|
||||
down_revision = "f7c8d9e0a1b2"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"connector_definitions",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("definition_key", sa.String(length=160), nullable=False),
|
||||
sa.Column("name", sa.String(length=300), nullable=False),
|
||||
sa.Column("description", sa.Text(), nullable=True),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("current_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("source_package", sa.String(length=300), nullable=True),
|
||||
sa.Column("local_definition", sa.Boolean(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_definitions")),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"definition_key",
|
||||
name="uq_connector_definition_tenant_key",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_connector_definitions_tenant_id"),
|
||||
"connector_definitions",
|
||||
["tenant_id"],
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_connector_definitions_status"),
|
||||
"connector_definitions",
|
||||
["status"],
|
||||
)
|
||||
op.create_index(
|
||||
"ix_connector_definitions_tenant_status",
|
||||
"connector_definitions",
|
||||
["tenant_id", "status"],
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"connector_definition_revisions",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("definition_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("revision", sa.Integer(), nullable=False),
|
||||
sa.Column("specification", sa.JSON(), nullable=False),
|
||||
sa.Column("definition_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("origin", sa.String(length=30), nullable=False),
|
||||
sa.Column("package_ref", sa.String(length=300), nullable=True),
|
||||
sa.Column("created_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["definition_id"],
|
||||
["connector_definitions.id"],
|
||||
name=op.f(
|
||||
"fk_connector_definition_revisions_definition_id_connector_definitions"
|
||||
),
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint(
|
||||
"id",
|
||||
name=op.f("pk_connector_definition_revisions"),
|
||||
),
|
||||
sa.UniqueConstraint(
|
||||
"definition_id",
|
||||
"revision",
|
||||
name="uq_connector_definition_revision",
|
||||
),
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_connector_definition_revisions_definition_id"),
|
||||
"connector_definition_revisions",
|
||||
["definition_id"],
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_connector_definition_revisions_definition_hash"),
|
||||
"connector_definition_revisions",
|
||||
["definition_hash"],
|
||||
)
|
||||
op.create_index(
|
||||
op.f("ix_connector_definition_revisions_created_by"),
|
||||
"connector_definition_revisions",
|
||||
["created_by"],
|
||||
)
|
||||
op.create_table(
|
||||
"connector_configurations",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("definition_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("name", sa.String(length=300), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("endpoint_url", sa.String(length=1500), nullable=True),
|
||||
sa.Column("credential_ref", sa.String(length=500), nullable=True),
|
||||
sa.Column("base_definition_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("local_overrides", sa.JSON(), nullable=False),
|
||||
sa.Column("protected_paths", sa.JSON(), nullable=False),
|
||||
sa.Column("effective_configuration", sa.JSON(), nullable=False),
|
||||
sa.Column("effective_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("resource_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("ambiguity_policy", sa.String(length=30), nullable=False),
|
||||
sa.Column("updated_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["definition_id"],
|
||||
["connector_definitions.id"],
|
||||
name=op.f(
|
||||
"fk_connector_configurations_definition_id_connector_definitions"
|
||||
),
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_configurations")),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"name",
|
||||
name="uq_connector_configuration_tenant_name",
|
||||
),
|
||||
)
|
||||
for name, columns in (
|
||||
("ix_connector_configurations_tenant_id", ["tenant_id"]),
|
||||
("ix_connector_configurations_definition_id", ["definition_id"]),
|
||||
("ix_connector_configurations_status", ["status"]),
|
||||
("ix_connector_configurations_effective_hash", ["effective_hash"]),
|
||||
("ix_connector_configurations_updated_by", ["updated_by"]),
|
||||
):
|
||||
op.create_index(op.f(name), "connector_configurations", columns)
|
||||
op.create_index(
|
||||
"ix_connector_configurations_tenant_status",
|
||||
"connector_configurations",
|
||||
["tenant_id", "status"],
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"connector_simulation_runs",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("configuration_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("mode", sa.String(length=30), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||
sa.Column("request_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("status", sa.String(length=30), nullable=False),
|
||||
sa.Column("review_state", sa.String(length=30), nullable=False),
|
||||
sa.Column("definition_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("configuration_revision", sa.Integer(), nullable=False),
|
||||
sa.Column("configuration_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("input_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("summary", sa.JSON(), nullable=False),
|
||||
sa.Column("effects", sa.JSON(), nullable=False),
|
||||
sa.Column("diagnostics", sa.JSON(), nullable=False),
|
||||
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||
sa.Column("created_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("reviewed_by", sa.String(length=255), nullable=True),
|
||||
sa.Column("reviewed_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("review_reason", sa.Text(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["configuration_id"],
|
||||
["connector_configurations.id"],
|
||||
name=op.f(
|
||||
"fk_connector_simulation_runs_configuration_id_connector_configurations"
|
||||
),
|
||||
ondelete="CASCADE",
|
||||
),
|
||||
sa.PrimaryKeyConstraint(
|
||||
"id",
|
||||
name=op.f("pk_connector_simulation_runs"),
|
||||
),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"configuration_id",
|
||||
"mode",
|
||||
"idempotency_key",
|
||||
name="uq_connector_simulation_run_idempotency",
|
||||
),
|
||||
)
|
||||
for name, columns in (
|
||||
("ix_connector_simulation_runs_tenant_id", ["tenant_id"]),
|
||||
("ix_connector_simulation_runs_configuration_id", ["configuration_id"]),
|
||||
("ix_connector_simulation_runs_mode", ["mode"]),
|
||||
("ix_connector_simulation_runs_status", ["status"]),
|
||||
("ix_connector_simulation_runs_review_state", ["review_state"]),
|
||||
("ix_connector_simulation_runs_created_by", ["created_by"]),
|
||||
("ix_connector_simulation_runs_reviewed_by", ["reviewed_by"]),
|
||||
):
|
||||
op.create_index(op.f(name), "connector_simulation_runs", columns)
|
||||
op.create_index(
|
||||
"ix_connector_simulation_runs_review",
|
||||
"connector_simulation_runs",
|
||||
["tenant_id", "review_state", "created_at"],
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("connector_simulation_runs")
|
||||
op.drop_table("connector_configurations")
|
||||
op.drop_table("connector_definition_revisions")
|
||||
op.drop_table("connector_definitions")
|
||||
@@ -1,6 +1,5 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
from datetime import UTC, datetime
|
||||
from hashlib import sha256
|
||||
|
||||
@@ -98,6 +97,13 @@ def _tabular_state(
|
||||
observed_at: datetime,
|
||||
) -> ExternalProviderRuntimeState:
|
||||
active = source.status == "active"
|
||||
live_origin = source.provider in {"managed_file", "postgresql"}
|
||||
labels = {
|
||||
"managed_file": "Exact managed-file origin",
|
||||
"postgresql": "Live PostgreSQL origin",
|
||||
"snapshot": "Immutable tabular snapshot",
|
||||
}
|
||||
label = labels.get(source.provider, "Tabular source")
|
||||
return ExternalProviderRuntimeState(
|
||||
provider_id=TABULAR_PROVIDER_ID,
|
||||
binding_ref=f"connectors:tabular-source:{source.id}",
|
||||
@@ -105,17 +111,20 @@ def _tabular_state(
|
||||
observed_at=observed_at,
|
||||
configured=True,
|
||||
active=active,
|
||||
health="healthy" if active else "inactive",
|
||||
freshness="not_applicable",
|
||||
health=("unknown" if active and live_origin else "healthy" if active else "inactive"),
|
||||
freshness="unknown" if active and live_origin else "not_applicable",
|
||||
conflict="not_applicable",
|
||||
recovery="ready" if active else "not_applicable",
|
||||
last_success_at=_aware(source.updated_at or source.created_at),
|
||||
detail=(
|
||||
"Immutable tabular snapshot is available."
|
||||
f"{label} is configured; live access and drift are checked on preview."
|
||||
if active and live_origin
|
||||
else f"{label} is available."
|
||||
if active
|
||||
else "Immutable tabular snapshot is inactive."
|
||||
else f"{label} is inactive."
|
||||
),
|
||||
metrics={
|
||||
"provider": source.provider,
|
||||
"row_count": int(source.row_count),
|
||||
"byte_count": int(source.byte_count),
|
||||
"schema_version": int(source.schema_version),
|
||||
|
||||
@@ -22,6 +22,7 @@ from govoplan_core.core.feeds import (
|
||||
FeedCapabilityError,
|
||||
FeedEntry,
|
||||
FeedRenderRequest,
|
||||
FeedVisibility,
|
||||
)
|
||||
from govoplan_core.core.sanctions import SanctionsSnapshotReference
|
||||
from govoplan_core.db.session import get_session
|
||||
@@ -29,7 +30,9 @@ from govoplan_connectors.backend.schemas import (
|
||||
FeedAcquireRequest,
|
||||
FeedDocumentResponse,
|
||||
FeedImportRequest,
|
||||
FeedPublicationEntryPayload,
|
||||
FeedRenderPayload,
|
||||
ManagedFileSourceCreateRequest,
|
||||
SanctionsAcquisitionRunListResponse,
|
||||
SanctionsAcquisitionRunResponse,
|
||||
SanctionsRefreshResponse,
|
||||
@@ -38,6 +41,7 @@ from govoplan_connectors.backend.schemas import (
|
||||
SanctionsSourceListResponse,
|
||||
SanctionsSourceResponse,
|
||||
SnapshotCreateRequest,
|
||||
SqlSourceCreateRequest,
|
||||
TabularColumnResponse,
|
||||
TabularHealthResponse,
|
||||
TabularPreviewDiagnosticResponse,
|
||||
@@ -47,7 +51,36 @@ from govoplan_connectors.backend.schemas import (
|
||||
TabularSourcePreviewResponse,
|
||||
TabularSourceResponse,
|
||||
)
|
||||
from govoplan_connectors.backend.feeds import ConnectorFeedProvider, feed_rows
|
||||
from govoplan_connectors.backend.feeds import (
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
FEED_PUBLISH_SCOPE,
|
||||
ConnectorFeedProvider,
|
||||
feed_rows,
|
||||
)
|
||||
from govoplan_connectors.backend.governed_runtime import (
|
||||
GovernedConnectorError,
|
||||
create_configuration,
|
||||
execute_run,
|
||||
list_configurations,
|
||||
list_definitions,
|
||||
list_runs,
|
||||
review_run,
|
||||
update_configuration,
|
||||
upsert_definition,
|
||||
)
|
||||
from govoplan_connectors.backend.governed_schemas import (
|
||||
ConnectorConfigurationCreateRequest,
|
||||
ConnectorConfigurationItem,
|
||||
ConnectorConfigurationListResponse,
|
||||
ConnectorConfigurationUpdateRequest,
|
||||
ConnectorDefinitionItem,
|
||||
ConnectorDefinitionListResponse,
|
||||
ConnectorDefinitionUpsertRequest,
|
||||
ConnectorReviewRequest,
|
||||
ConnectorRunItem,
|
||||
ConnectorRunListResponse,
|
||||
ConnectorRunRequest,
|
||||
)
|
||||
from govoplan_connectors.backend.recovery import (
|
||||
ConnectorRecoveryError,
|
||||
begin_connector_read_snapshot,
|
||||
@@ -135,6 +168,26 @@ def _recovery_http_error(exc: ConnectorRecoveryError) -> HTTPException:
|
||||
)
|
||||
|
||||
|
||||
def _governed_http_error(exc: GovernedConnectorError) -> HTTPException:
|
||||
if exc.code.endswith("_not_found"):
|
||||
status_code = status.HTTP_404_NOT_FOUND
|
||||
elif exc.code in {
|
||||
"configuration_conflict",
|
||||
"configuration_disabled",
|
||||
"idempotency_conflict",
|
||||
"local_definition_protected",
|
||||
"package_definition_requires_overrides",
|
||||
"run_not_reviewable",
|
||||
}:
|
||||
status_code = status.HTTP_409_CONFLICT
|
||||
else:
|
||||
status_code = status.HTTP_422_UNPROCESSABLE_CONTENT
|
||||
return HTTPException(
|
||||
status_code=status_code,
|
||||
detail={"code": exc.code, "message": str(exc)},
|
||||
)
|
||||
|
||||
|
||||
@router.post("/feeds/preview", response_model=FeedDocumentResponse)
|
||||
def api_preview_feed(
|
||||
payload: FeedAcquireRequest,
|
||||
@@ -284,7 +337,9 @@ def api_render_feed(
|
||||
payload: FeedRenderPayload,
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> Response:
|
||||
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
|
||||
_require_any_scope(principal, FEED_PUBLISH_SCOPE, ADMIN_SCOPE)
|
||||
if payload.audience != "public":
|
||||
_require_any_scope(principal, FEED_PRIVATE_PUBLISH_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
rendered = feed_transport.render(
|
||||
FeedRenderRequest(
|
||||
@@ -295,9 +350,9 @@ def api_render_feed(
|
||||
description=payload.description,
|
||||
language=payload.language,
|
||||
entries=tuple(
|
||||
FeedEntry(**item.model_dump()) for item in payload.entries
|
||||
_publication_feed_entry(item) for item in payload.entries
|
||||
),
|
||||
allowed_visibilities=frozenset(payload.allowed_visibilities),
|
||||
allowed_visibilities=_feed_audience_visibilities(payload.audience),
|
||||
)
|
||||
)
|
||||
except FeedCapabilityError as exc:
|
||||
@@ -308,10 +363,39 @@ def api_render_feed(
|
||||
headers={
|
||||
"X-GovOPlaN-Feed-Included": str(rendered.included_entries),
|
||||
"X-GovOPlaN-Feed-Excluded": str(rendered.excluded_entries),
|
||||
"X-GovOPlaN-Feed-Audience": payload.audience,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _feed_audience_visibilities(
|
||||
audience: str,
|
||||
) -> frozenset[FeedVisibility]:
|
||||
if audience == "public":
|
||||
return frozenset({"public"})
|
||||
if audience == "tenant":
|
||||
return frozenset({"public", "tenant"})
|
||||
return frozenset({"public", "tenant", "private"})
|
||||
|
||||
|
||||
def _publication_feed_entry(item: FeedPublicationEntryPayload) -> FeedEntry:
|
||||
values = item.model_dump(
|
||||
exclude={"source_kind", "source_module", "source_ref", "source_revision"}
|
||||
)
|
||||
values["categories"] = tuple(values["categories"])
|
||||
values["enclosures"] = tuple(values["enclosures"])
|
||||
values["metadata"] = {
|
||||
**values["metadata"],
|
||||
"govoplan_source": {
|
||||
"kind": item.source_kind,
|
||||
"module": item.source_module,
|
||||
"reference": item.source_ref,
|
||||
"revision": item.source_revision,
|
||||
},
|
||||
}
|
||||
return FeedEntry(**values)
|
||||
|
||||
|
||||
@router.get("/tabular-sources", response_model=TabularSourceListResponse)
|
||||
def api_list_tabular_sources(
|
||||
query: str = Query(default="", max_length=200),
|
||||
@@ -378,12 +462,82 @@ def api_create_tabular_snapshot(
|
||||
return _source_response(source)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/tabular-sources/files",
|
||||
response_model=TabularSourceResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def api_create_managed_file_source(
|
||||
payload: ManagedFileSourceCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> TabularSourceResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
source = provider.create_file_source(
|
||||
session,
|
||||
principal,
|
||||
name=payload.name,
|
||||
source_name=payload.source_name,
|
||||
description=payload.description,
|
||||
file_asset_id=payload.file_asset_id,
|
||||
file_version_id=payload.file_version_id,
|
||||
delimiter=payload.delimiter,
|
||||
sheet_name=payload.sheet_name,
|
||||
)
|
||||
except TabularSourceError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
_audit_tabular_source_change(
|
||||
session,
|
||||
principal,
|
||||
source,
|
||||
operation="created",
|
||||
)
|
||||
session.commit()
|
||||
return _source_response(source)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/tabular-sources/sql",
|
||||
response_model=TabularSourceResponse,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def api_create_sql_source(
|
||||
payload: SqlSourceCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> TabularSourceResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
source = provider.create_sql_source(
|
||||
session,
|
||||
principal,
|
||||
name=payload.name,
|
||||
source_name=payload.source_name,
|
||||
description=payload.description,
|
||||
configuration_id=payload.configuration_id,
|
||||
schema_name=payload.schema_name,
|
||||
table_name=payload.table_name,
|
||||
)
|
||||
except TabularSourceError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
_audit_tabular_source_change(
|
||||
session,
|
||||
principal,
|
||||
source,
|
||||
operation="created",
|
||||
)
|
||||
session.commit()
|
||||
return _source_response(source)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/tabular-sources/{source_id}/preview",
|
||||
response_model=TabularSourcePreviewResponse,
|
||||
)
|
||||
def api_preview_tabular_source(
|
||||
source_id: str,
|
||||
kind: Annotated[str, Query(pattern=r"^(snapshot|file|sql)$")] = "snapshot",
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
offset: int = Query(default=0, ge=0),
|
||||
max_bytes: int = Query(default=1_000_000, ge=2, le=5_000_000),
|
||||
@@ -397,7 +551,7 @@ def api_preview_tabular_source(
|
||||
session,
|
||||
principal,
|
||||
request=TabularReadRequest(
|
||||
source_ref=f"snapshot:{source_id}",
|
||||
source_ref=f"{kind}:{source_id}",
|
||||
limit=limit,
|
||||
offset=offset,
|
||||
max_bytes=max_bytes,
|
||||
@@ -428,17 +582,47 @@ def api_preview_tabular_source(
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/tabular-sources/{source_id}/refresh",
|
||||
response_model=TabularSourceResponse,
|
||||
)
|
||||
def api_refresh_tabular_source(
|
||||
source_id: str,
|
||||
kind: Annotated[str, Query(pattern=r"^(file|sql)$")],
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> TabularSourceResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
source = provider.refresh_source(
|
||||
session,
|
||||
principal,
|
||||
source_ref=f"{kind}:{source_id}",
|
||||
)
|
||||
except TabularSourceError as exc:
|
||||
raise _http_error(exc) from exc
|
||||
_audit_tabular_source_change(
|
||||
session,
|
||||
principal,
|
||||
source,
|
||||
operation="refreshed",
|
||||
)
|
||||
session.commit()
|
||||
return _source_response(source)
|
||||
|
||||
|
||||
@router.delete(
|
||||
"/tabular-sources/{source_id}",
|
||||
response_model=TabularSourceDeleteResponse,
|
||||
)
|
||||
def api_delete_tabular_source(
|
||||
source_id: str,
|
||||
kind: Annotated[str, Query(pattern=r"^(snapshot|file|sql)$")] = "snapshot",
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> TabularSourceDeleteResponse:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
source_ref = f"snapshot:{source_id}"
|
||||
source_ref = f"{kind}:{source_id}"
|
||||
try:
|
||||
source = provider.delete_snapshot(
|
||||
session,
|
||||
@@ -452,7 +636,7 @@ def api_delete_tabular_source(
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="connectors.tabular_snapshot.deleted",
|
||||
action="connectors.tabular_source.deleted",
|
||||
object_type="connector_tabular_source",
|
||||
object_id=source_ref,
|
||||
details={"source_name": source.source_name, "fingerprint": source.fingerprint},
|
||||
@@ -609,6 +793,199 @@ def api_list_sanctions_runs(
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/governed/definitions",
|
||||
response_model=ConnectorDefinitionListResponse,
|
||||
)
|
||||
def api_list_governed_definitions(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorDefinitionListResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
items = list_definitions(session, tenant_id=principal.tenant_id)
|
||||
except GovernedConnectorError as exc:
|
||||
raise _governed_http_error(exc) from exc
|
||||
return ConnectorDefinitionListResponse(items=items)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/governed/definitions",
|
||||
response_model=ConnectorDefinitionItem,
|
||||
)
|
||||
def api_upsert_governed_definition(
|
||||
payload: ConnectorDefinitionUpsertRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorDefinitionItem:
|
||||
_require_any_scope(principal, ADMIN_SCOPE)
|
||||
try:
|
||||
return upsert_definition(session, principal, payload)
|
||||
except GovernedConnectorError as exc:
|
||||
session.rollback()
|
||||
raise _governed_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.get(
|
||||
"/governed/configurations",
|
||||
response_model=ConnectorConfigurationListResponse,
|
||||
)
|
||||
def api_list_governed_configurations(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorConfigurationListResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
items = list_configurations(session, tenant_id=principal.tenant_id)
|
||||
except GovernedConnectorError as exc:
|
||||
raise _governed_http_error(exc) from exc
|
||||
return ConnectorConfigurationListResponse(items=items)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/governed/configurations",
|
||||
response_model=ConnectorConfigurationItem,
|
||||
status_code=status.HTTP_201_CREATED,
|
||||
)
|
||||
def api_create_governed_configuration(
|
||||
payload: ConnectorConfigurationCreateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorConfigurationItem:
|
||||
_require_any_scope(principal, ADMIN_SCOPE)
|
||||
try:
|
||||
return create_configuration(session, principal, payload)
|
||||
except GovernedConnectorError as exc:
|
||||
session.rollback()
|
||||
raise _governed_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.put(
|
||||
"/governed/configurations/{configuration_id}",
|
||||
response_model=ConnectorConfigurationItem,
|
||||
)
|
||||
def api_update_governed_configuration(
|
||||
configuration_id: str,
|
||||
payload: ConnectorConfigurationUpdateRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorConfigurationItem:
|
||||
_require_any_scope(principal, ADMIN_SCOPE)
|
||||
try:
|
||||
return update_configuration(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
payload=payload,
|
||||
)
|
||||
except GovernedConnectorError as exc:
|
||||
session.rollback()
|
||||
raise _governed_http_error(exc) from exc
|
||||
|
||||
|
||||
def _api_execute_governed_run(
|
||||
*,
|
||||
configuration_id: str,
|
||||
mode: str,
|
||||
payload: ConnectorRunRequest,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
) -> ConnectorRunItem:
|
||||
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
return execute_run(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
mode=mode,
|
||||
payload=payload,
|
||||
)
|
||||
except GovernedConnectorError as exc:
|
||||
session.rollback()
|
||||
raise _governed_http_error(exc) from exc
|
||||
|
||||
|
||||
@router.post(
|
||||
"/governed/configurations/{configuration_id}/dry-runs",
|
||||
response_model=ConnectorRunItem,
|
||||
)
|
||||
def api_dry_run_governed_configuration(
|
||||
configuration_id: str,
|
||||
payload: ConnectorRunRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorRunItem:
|
||||
return _api_execute_governed_run(
|
||||
configuration_id=configuration_id,
|
||||
mode="dry_run",
|
||||
payload=payload,
|
||||
session=session,
|
||||
principal=principal,
|
||||
)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/governed/configurations/{configuration_id}/simulations",
|
||||
response_model=ConnectorRunItem,
|
||||
)
|
||||
def api_simulate_governed_configuration(
|
||||
configuration_id: str,
|
||||
payload: ConnectorRunRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorRunItem:
|
||||
return _api_execute_governed_run(
|
||||
configuration_id=configuration_id,
|
||||
mode="simulation",
|
||||
payload=payload,
|
||||
session=session,
|
||||
principal=principal,
|
||||
)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/governed/runs",
|
||||
response_model=ConnectorRunListResponse,
|
||||
)
|
||||
def api_list_governed_runs(
|
||||
configuration_id: str | None = Query(default=None),
|
||||
review_state: str | None = Query(default=None),
|
||||
limit: int = Query(default=100, ge=1, le=500),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorRunListResponse:
|
||||
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
|
||||
try:
|
||||
items = list_runs(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
configuration_id=configuration_id,
|
||||
review_state=review_state,
|
||||
limit=limit,
|
||||
)
|
||||
except GovernedConnectorError as exc:
|
||||
raise _governed_http_error(exc) from exc
|
||||
return ConnectorRunListResponse(items=items)
|
||||
|
||||
|
||||
@router.post(
|
||||
"/governed/runs/{run_id}/review",
|
||||
response_model=ConnectorRunItem,
|
||||
)
|
||||
def api_review_governed_run(
|
||||
run_id: str,
|
||||
payload: ConnectorReviewRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> ConnectorRunItem:
|
||||
_require_any_scope(principal, ADMIN_SCOPE)
|
||||
try:
|
||||
return review_run(session, principal, run_id=run_id, payload=payload)
|
||||
except GovernedConnectorError as exc:
|
||||
session.rollback()
|
||||
raise _governed_http_error(exc) from exc
|
||||
|
||||
|
||||
def _source_response(source: TabularSource) -> TabularSourceResponse:
|
||||
return TabularSourceResponse(
|
||||
ref=source.ref,
|
||||
@@ -653,6 +1030,32 @@ def _source_response(source: TabularSource) -> TabularSourceResponse:
|
||||
)
|
||||
|
||||
|
||||
def _audit_tabular_source_change(
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
source: TabularSource,
|
||||
*,
|
||||
operation: str,
|
||||
) -> None:
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action=f"connectors.tabular_source.{operation}",
|
||||
object_type="connector_tabular_source",
|
||||
object_id=source.ref,
|
||||
details={
|
||||
"provider": source.provider,
|
||||
"source_name": source.source_name,
|
||||
"fingerprint": source.fingerprint,
|
||||
"schema_version": source.schema_version,
|
||||
"row_count": source.row_count,
|
||||
"source_mode": source.source_mode,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _sanctions_snapshot_response(
|
||||
snapshot: SanctionsSnapshotReference,
|
||||
) -> SanctionsSnapshotResponse:
|
||||
|
||||
@@ -36,6 +36,17 @@ class FeedEntryPayload(BaseModel):
|
||||
metadata: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
|
||||
class FeedPublicationEntryPayload(FeedEntryPayload):
|
||||
source_kind: Literal["event", "publication", "case", "report"]
|
||||
source_module: str = Field(
|
||||
min_length=1,
|
||||
max_length=100,
|
||||
pattern=r"^[a-z][a-z0-9_]*$",
|
||||
)
|
||||
source_ref: str = Field(min_length=1, max_length=500)
|
||||
source_revision: str | None = Field(default=None, max_length=200)
|
||||
|
||||
|
||||
class FeedDocumentResponse(BaseModel):
|
||||
format: Literal["rss", "atom"]
|
||||
title: str
|
||||
@@ -61,11 +72,11 @@ class FeedRenderPayload(BaseModel):
|
||||
home_url: str = Field(min_length=1, max_length=2000)
|
||||
description: str | None = None
|
||||
language: str | None = Field(default=None, max_length=100)
|
||||
entries: list[FeedEntryPayload] = Field(default_factory=list, max_length=10_000)
|
||||
allowed_visibilities: list[Literal["public", "tenant", "private"]] = Field(
|
||||
default_factory=lambda: ["public"],
|
||||
max_length=3,
|
||||
entries: list[FeedPublicationEntryPayload] = Field(
|
||||
default_factory=list,
|
||||
max_length=10_000,
|
||||
)
|
||||
audience: Literal["public", "tenant", "private"] = "public"
|
||||
|
||||
|
||||
class SnapshotCreateRequest(BaseModel):
|
||||
@@ -94,6 +105,40 @@ class SnapshotCreateRequest(BaseModel):
|
||||
return self
|
||||
|
||||
|
||||
class ManagedFileSourceCreateRequest(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
source_name: str = Field(
|
||||
min_length=1,
|
||||
max_length=120,
|
||||
pattern=r"^[A-Za-z_][A-Za-z0-9_]*$",
|
||||
)
|
||||
description: str | None = Field(default=None, max_length=4000)
|
||||
file_asset_id: str = Field(min_length=1, max_length=36)
|
||||
file_version_id: str | None = Field(default=None, min_length=1, max_length=36)
|
||||
delimiter: Literal[",", ";", "\t", "|"] = ","
|
||||
sheet_name: str | None = Field(default=None, min_length=1, max_length=255)
|
||||
|
||||
|
||||
class SqlSourceCreateRequest(BaseModel):
|
||||
name: str = Field(min_length=1, max_length=300)
|
||||
source_name: str = Field(
|
||||
min_length=1,
|
||||
max_length=120,
|
||||
pattern=r"^[A-Za-z_][A-Za-z0-9_]*$",
|
||||
)
|
||||
description: str | None = Field(default=None, max_length=4000)
|
||||
configuration_id: str = Field(min_length=1, max_length=36)
|
||||
schema_name: str | None = Field(
|
||||
default=None,
|
||||
pattern=r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$",
|
||||
)
|
||||
table_name: str = Field(
|
||||
min_length=1,
|
||||
max_length=128,
|
||||
pattern=r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$",
|
||||
)
|
||||
|
||||
|
||||
class TabularColumnResponse(BaseModel):
|
||||
name: str
|
||||
data_type: str
|
||||
@@ -239,7 +284,9 @@ __all__ = [
|
||||
"FeedEntryPayload",
|
||||
"FeedImportRequest",
|
||||
"FeedRenderPayload",
|
||||
"ManagedFileSourceCreateRequest",
|
||||
"SnapshotCreateRequest",
|
||||
"SqlSourceCreateRequest",
|
||||
"SanctionsAcquisitionRunListResponse",
|
||||
"SanctionsAcquisitionRunResponse",
|
||||
"SanctionsRefreshResponse",
|
||||
|
||||
@@ -0,0 +1,849 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import zipfile
|
||||
from collections.abc import Callable, Mapping, Sequence
|
||||
from dataclasses import dataclass
|
||||
from datetime import date, datetime
|
||||
from decimal import Decimal
|
||||
from io import BytesIO
|
||||
from typing import Any
|
||||
|
||||
from openpyxl import load_workbook
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
BigInteger,
|
||||
Boolean,
|
||||
Date,
|
||||
DateTime,
|
||||
Float,
|
||||
Integer,
|
||||
LargeBinary,
|
||||
MetaData,
|
||||
Numeric,
|
||||
String,
|
||||
Table,
|
||||
Text,
|
||||
create_engine,
|
||||
func,
|
||||
select,
|
||||
)
|
||||
from sqlalchemy.engine import Engine, URL, make_url
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.files import (
|
||||
ManagedTabularFileAccessError,
|
||||
ManagedTabularFileError,
|
||||
ManagedTabularFileValidationError,
|
||||
managed_tabular_file_provider,
|
||||
)
|
||||
from govoplan_core.core.tabular_sources import (
|
||||
TabularColumn,
|
||||
TabularPreviewDiagnostic,
|
||||
TabularPushdown,
|
||||
TabularSourceHealth,
|
||||
TabularSourceUnavailableError,
|
||||
TabularSourceValidationError,
|
||||
parse_tabular_csv,
|
||||
)
|
||||
from govoplan_core.security.credential_envelopes import (
|
||||
CredentialAccessContext,
|
||||
CredentialEnvelopeError,
|
||||
resolve_credential_envelope,
|
||||
)
|
||||
from govoplan_core.security.redaction import is_sensitive_key
|
||||
from govoplan_connectors.backend.db.models import ConnectorConfiguration
|
||||
|
||||
|
||||
MAX_FILE_BYTES = 5_000_000
|
||||
MAX_FILE_ROWS = 10_000
|
||||
MAX_FILE_COLUMNS = 500
|
||||
MAX_XLSX_ENTRIES = 5_000
|
||||
MAX_XLSX_EXPANDED_BYTES = 50_000_000
|
||||
MAX_XLSX_COMPRESSION_RATIO = 100
|
||||
POSTGRESQL_SCHEMES = frozenset({"postgresql", "postgresql+psycopg"})
|
||||
_IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class TabularOriginInspection:
|
||||
provider: str
|
||||
schema: tuple[TabularColumn, ...]
|
||||
fingerprint: str
|
||||
row_count: int
|
||||
byte_count: int
|
||||
metadata: Mapping[str, object]
|
||||
health: TabularSourceHealth
|
||||
pushdown: TabularPushdown
|
||||
rows: tuple[Mapping[str, object], ...] = ()
|
||||
diagnostics: tuple[TabularPreviewDiagnostic, ...] = ()
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class TabularOriginRead:
|
||||
inspection: TabularOriginInspection
|
||||
rows: tuple[Mapping[str, object], ...]
|
||||
total_rows: int
|
||||
diagnostics: tuple[TabularPreviewDiagnostic, ...] = ()
|
||||
|
||||
|
||||
class ManagedFileTabularAdapter:
|
||||
def __init__(self, registry: object | None) -> None:
|
||||
self._registry = registry
|
||||
|
||||
def inspect(
|
||||
self,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
file_asset_id: str,
|
||||
file_version_id: str | None,
|
||||
delimiter: str = ",",
|
||||
sheet_name: str | None = None,
|
||||
) -> TabularOriginInspection:
|
||||
provider = managed_tabular_file_provider(self._registry)
|
||||
if provider is None:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Managed file sources require the Files module."
|
||||
)
|
||||
try:
|
||||
metadata = provider.get_tabular_file(
|
||||
session,
|
||||
principal,
|
||||
file_asset_id=file_asset_id,
|
||||
file_version_id=file_version_id,
|
||||
)
|
||||
if metadata is None:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Managed tabular file or version is unavailable."
|
||||
)
|
||||
content = provider.read_tabular_file(
|
||||
session,
|
||||
principal,
|
||||
file_asset_id=file_asset_id,
|
||||
file_version_id=metadata.file_version_id,
|
||||
max_bytes=MAX_FILE_BYTES,
|
||||
)
|
||||
except ManagedTabularFileAccessError as exc:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Managed tabular file access is no longer authorized."
|
||||
) from exc
|
||||
except ManagedTabularFileValidationError as exc:
|
||||
raise TabularSourceValidationError(str(exc)) from exc
|
||||
except ManagedTabularFileError as exc:
|
||||
raise TabularSourceUnavailableError(str(exc)) from exc
|
||||
rows, resolved_sheet = parse_managed_tabular_content(
|
||||
content.payload,
|
||||
filename=content.file.filename,
|
||||
content_type=content.file.content_type,
|
||||
delimiter=delimiter,
|
||||
sheet_name=sheet_name,
|
||||
)
|
||||
schema = infer_tabular_schema(rows)
|
||||
fingerprint = origin_fingerprint(
|
||||
schema,
|
||||
tokens=(
|
||||
"managed_file",
|
||||
content.file.file_asset_id,
|
||||
content.file.file_version_id,
|
||||
content.file.sha256,
|
||||
resolved_sheet or "",
|
||||
delimiter,
|
||||
),
|
||||
)
|
||||
try:
|
||||
current = provider.get_tabular_file(
|
||||
session,
|
||||
principal,
|
||||
file_asset_id=file_asset_id,
|
||||
)
|
||||
except ManagedTabularFileAccessError as exc:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Managed tabular file access is no longer authorized."
|
||||
) from exc
|
||||
except ManagedTabularFileError as exc:
|
||||
raise TabularSourceUnavailableError(str(exc)) from exc
|
||||
changed = bool(
|
||||
current is not None
|
||||
and current.file_version_id != content.file.file_version_id
|
||||
)
|
||||
health = TabularSourceHealth(
|
||||
status="warning" if changed else "healthy",
|
||||
code=(
|
||||
"files.newer_version_available"
|
||||
if changed
|
||||
else "files.exact_version_ready"
|
||||
),
|
||||
summary=(
|
||||
"A newer managed file version is available for explicit review."
|
||||
if changed
|
||||
else "The exact managed file version passed access and integrity checks."
|
||||
),
|
||||
checked_at=content.file.updated_at,
|
||||
details={
|
||||
"pinned_version_id": content.file.file_version_id,
|
||||
"current_version_id": (
|
||||
current.file_version_id if current is not None else None
|
||||
),
|
||||
},
|
||||
)
|
||||
return TabularOriginInspection(
|
||||
provider="managed_file",
|
||||
schema=schema,
|
||||
fingerprint=fingerprint,
|
||||
row_count=len(rows),
|
||||
byte_count=len(content.payload),
|
||||
metadata={
|
||||
"origin_kind": "managed_file",
|
||||
"file_asset_id": content.file.file_asset_id,
|
||||
"file_version_id": content.file.file_version_id,
|
||||
"file_sha256": content.file.sha256,
|
||||
"filename": content.file.filename,
|
||||
"content_type": content.file.content_type,
|
||||
"format": "xlsx" if _is_xlsx(content.file.filename, content.file.content_type) else "csv",
|
||||
"delimiter": delimiter,
|
||||
"sheet_name": resolved_sheet,
|
||||
},
|
||||
health=health,
|
||||
pushdown=TabularPushdown(projections=True, pagination=True),
|
||||
rows=rows,
|
||||
diagnostics=(
|
||||
(
|
||||
TabularPreviewDiagnostic(
|
||||
severity="warning",
|
||||
code="files.newer_version_available",
|
||||
message=(
|
||||
"The preview remains pinned to the reviewed file version; "
|
||||
"refresh the source to adopt the newer version."
|
||||
),
|
||||
details=dict(health.details),
|
||||
),
|
||||
)
|
||||
if changed
|
||||
else ()
|
||||
),
|
||||
)
|
||||
|
||||
def read(
|
||||
self,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
metadata: Mapping[str, object],
|
||||
) -> TabularOriginRead:
|
||||
inspection = self.inspect(
|
||||
session,
|
||||
principal,
|
||||
file_asset_id=_required_metadata(metadata, "file_asset_id"),
|
||||
file_version_id=_required_metadata(metadata, "file_version_id"),
|
||||
delimiter=str(metadata.get("delimiter") or ","),
|
||||
sheet_name=_optional_text(metadata.get("sheet_name")),
|
||||
)
|
||||
expected_sha256 = _required_metadata(metadata, "file_sha256")
|
||||
if inspection.metadata.get("file_sha256") != expected_sha256:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Managed file content no longer matches its exact-version checksum."
|
||||
)
|
||||
return TabularOriginRead(
|
||||
inspection=inspection,
|
||||
rows=inspection.rows,
|
||||
total_rows=inspection.row_count,
|
||||
diagnostics=inspection.diagnostics,
|
||||
)
|
||||
|
||||
|
||||
class PostgresqlTabularAdapter:
|
||||
"""Read-only SQLAlchemy adapter with a production PostgreSQL allow-list."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
engine_factory: Callable[[URL], Engine] | None = None,
|
||||
allow_sqlite_for_tests: bool = False,
|
||||
) -> None:
|
||||
self._engine_factory = engine_factory or (
|
||||
lambda url: create_engine(url, pool_pre_ping=True)
|
||||
)
|
||||
self._allow_sqlite_for_tests = allow_sqlite_for_tests
|
||||
|
||||
def inspect(
|
||||
self,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
configuration_id: str,
|
||||
table_name: str,
|
||||
schema_name: str | None = None,
|
||||
timeout_ms: int = 2_000,
|
||||
) -> TabularOriginInspection:
|
||||
configuration, url, credential_revision = self._connection(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
)
|
||||
table_name = _sql_identifier(table_name, "table")
|
||||
schema_name = (
|
||||
_sql_identifier(schema_name, "schema") if schema_name else None
|
||||
)
|
||||
engine = self._engine_factory(_bounded_connection_url(url, timeout_ms))
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
_configure_read_only(connection, url, timeout_ms)
|
||||
table = Table(
|
||||
table_name,
|
||||
MetaData(),
|
||||
schema=schema_name,
|
||||
autoload_with=connection,
|
||||
)
|
||||
schema = tuple(_sql_column(column) for column in table.columns)
|
||||
if not schema:
|
||||
raise TabularSourceValidationError(
|
||||
"SQL tabular sources require at least one column."
|
||||
)
|
||||
row_count = int(
|
||||
connection.execute(select(func.count()).select_from(table)).scalar_one()
|
||||
)
|
||||
except TabularSourceValidationError:
|
||||
raise
|
||||
except SQLAlchemyError as exc:
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL source discovery failed; verify the active configuration, credential, table, and provider health."
|
||||
) from exc
|
||||
finally:
|
||||
engine.dispose()
|
||||
fingerprint = origin_fingerprint(
|
||||
schema,
|
||||
tokens=(
|
||||
"postgresql",
|
||||
configuration.id,
|
||||
configuration.effective_hash,
|
||||
credential_revision or "",
|
||||
schema_name or "",
|
||||
table_name,
|
||||
),
|
||||
)
|
||||
return TabularOriginInspection(
|
||||
provider="postgresql",
|
||||
schema=schema,
|
||||
fingerprint=fingerprint,
|
||||
row_count=row_count,
|
||||
byte_count=0,
|
||||
metadata={
|
||||
"origin_kind": "sql",
|
||||
"configuration_id": configuration.id,
|
||||
"configuration_revision": configuration.resource_revision,
|
||||
"configuration_hash": configuration.effective_hash,
|
||||
"credential_revision": credential_revision,
|
||||
"schema_name": schema_name,
|
||||
"table_name": table_name,
|
||||
},
|
||||
health=TabularSourceHealth(
|
||||
status="healthy",
|
||||
code="sql.source_ready",
|
||||
summary="The governed PostgreSQL source is reachable and its schema was discovered.",
|
||||
checked_at=configuration.updated_at,
|
||||
details={
|
||||
"configuration_id": configuration.id,
|
||||
"configuration_revision": configuration.resource_revision,
|
||||
},
|
||||
),
|
||||
pushdown=TabularPushdown(projections=True, pagination=True),
|
||||
)
|
||||
|
||||
def read(
|
||||
self,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
metadata: Mapping[str, object],
|
||||
columns: Sequence[str],
|
||||
offset: int,
|
||||
limit: int,
|
||||
timeout_ms: int,
|
||||
) -> TabularOriginRead:
|
||||
configuration_id = _required_metadata(metadata, "configuration_id")
|
||||
current_configuration, _current_url, current_credential_revision = (
|
||||
self._connection(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
)
|
||||
)
|
||||
if current_configuration.effective_hash != metadata.get(
|
||||
"configuration_hash"
|
||||
):
|
||||
raise TabularSourceValidationError(
|
||||
"The SQL connector configuration changed; refresh the source before previewing it."
|
||||
)
|
||||
if current_credential_revision != metadata.get("credential_revision"):
|
||||
raise TabularSourceValidationError(
|
||||
"The SQL source credential changed; refresh the source before previewing it."
|
||||
)
|
||||
inspection = self.inspect(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
table_name=_required_metadata(metadata, "table_name"),
|
||||
schema_name=_optional_text(metadata.get("schema_name")),
|
||||
timeout_ms=timeout_ms,
|
||||
)
|
||||
if inspection.fingerprint != metadata.get("discovery_fingerprint"):
|
||||
raise TabularSourceValidationError(
|
||||
"The SQL source schema drifted; refresh and review the source before previewing it."
|
||||
)
|
||||
configuration, url, _credential_revision = self._connection(
|
||||
session,
|
||||
principal,
|
||||
configuration_id=configuration_id,
|
||||
)
|
||||
engine = self._engine_factory(_bounded_connection_url(url, timeout_ms))
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
_configure_read_only(connection, url, timeout_ms)
|
||||
table = Table(
|
||||
_required_metadata(metadata, "table_name"),
|
||||
MetaData(),
|
||||
schema=_optional_text(metadata.get("schema_name")),
|
||||
autoload_with=connection,
|
||||
)
|
||||
selected_names = tuple(dict.fromkeys(str(item) for item in columns))
|
||||
selected = (
|
||||
[table.c[name] for name in selected_names]
|
||||
if selected_names
|
||||
else list(table.columns)
|
||||
)
|
||||
statement = select(*selected).offset(max(0, int(offset))).limit(
|
||||
max(1, int(limit))
|
||||
)
|
||||
rows = tuple(
|
||||
_json_row(dict(row._mapping))
|
||||
for row in connection.execute(statement)
|
||||
)
|
||||
except KeyError as exc:
|
||||
raise TabularSourceValidationError(
|
||||
f"Unknown SQL source column: {exc.args[0]}"
|
||||
) from exc
|
||||
except SQLAlchemyError as exc:
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL source preview failed or exceeded its provider budget."
|
||||
) from exc
|
||||
finally:
|
||||
engine.dispose()
|
||||
return TabularOriginRead(
|
||||
inspection=inspection,
|
||||
rows=rows,
|
||||
total_rows=inspection.row_count,
|
||||
)
|
||||
|
||||
def _connection(
|
||||
self,
|
||||
session: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
configuration_id: str,
|
||||
) -> tuple[ConnectorConfiguration, URL, str | None]:
|
||||
configuration = session.scalar(
|
||||
select(ConnectorConfiguration).where(
|
||||
ConnectorConfiguration.id == configuration_id,
|
||||
ConnectorConfiguration.tenant_id == principal.tenant_id,
|
||||
)
|
||||
)
|
||||
if configuration is None:
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL connector configuration is unavailable."
|
||||
)
|
||||
if configuration.status != "active":
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL connector configuration is not active."
|
||||
)
|
||||
endpoint = _optional_text(configuration.endpoint_url)
|
||||
if not endpoint:
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL connector configuration has no endpoint."
|
||||
)
|
||||
specification = dict(configuration.effective_configuration or {})
|
||||
provider = str(specification.get("provider") or "").strip().casefold()
|
||||
protocol = str(specification.get("protocol") or "").strip().casefold()
|
||||
if provider not in {"postgres", "postgresql", "sql"} or protocol not in {
|
||||
"postgres",
|
||||
"postgresql",
|
||||
"sql",
|
||||
}:
|
||||
raise TabularSourceValidationError(
|
||||
"The selected connector configuration is not a PostgreSQL tabular reader."
|
||||
)
|
||||
try:
|
||||
url = make_url(endpoint)
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise TabularSourceValidationError(
|
||||
"SQL connector endpoint is invalid."
|
||||
) from exc
|
||||
if url.username or url.password:
|
||||
raise TabularSourceValidationError(
|
||||
"SQL connector endpoints must not contain credentials."
|
||||
)
|
||||
if any(is_sensitive_key(key) for key in url.query):
|
||||
raise TabularSourceValidationError(
|
||||
"SQL connector endpoint query parameters must not contain credentials."
|
||||
)
|
||||
allowed_schemes = set(POSTGRESQL_SCHEMES)
|
||||
if self._allow_sqlite_for_tests:
|
||||
allowed_schemes.update({"sqlite", "sqlite+pysqlite"})
|
||||
if url.drivername not in allowed_schemes:
|
||||
raise TabularSourceValidationError(
|
||||
"Only the governed PostgreSQL tabular adapter is enabled."
|
||||
)
|
||||
credential_revision: str | None = None
|
||||
credential_ref = _optional_text(configuration.credential_ref)
|
||||
if credential_ref:
|
||||
try:
|
||||
credential = resolve_credential_envelope(
|
||||
session,
|
||||
credential_id=credential_ref,
|
||||
context=CredentialAccessContext(
|
||||
tenant_id=principal.tenant_id,
|
||||
user_id=_principal_user_id(principal),
|
||||
group_ids=frozenset(principal.principal.group_ids),
|
||||
target_scope_type="tenant",
|
||||
target_scope_id=principal.tenant_id,
|
||||
module_id="connectors",
|
||||
server_ref=endpoint,
|
||||
),
|
||||
)
|
||||
except CredentialEnvelopeError as exc:
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL source credential is unavailable, inactive, or outside its allowed scope."
|
||||
) from exc
|
||||
public_data = dict(credential.public_data)
|
||||
secret_data = dict(credential.secret_data)
|
||||
username = _optional_text(
|
||||
public_data.get("username")
|
||||
or secret_data.get("username")
|
||||
or secret_data.get("user")
|
||||
)
|
||||
password = _optional_text(secret_data.get("password"))
|
||||
if url.drivername in POSTGRESQL_SCHEMES and (not username or not password):
|
||||
raise TabularSourceUnavailableError(
|
||||
"SQL source credential does not provide a username and password."
|
||||
)
|
||||
url = url.set(username=username, password=password)
|
||||
credential_revision = credential.revision
|
||||
elif url.drivername in POSTGRESQL_SCHEMES:
|
||||
raise TabularSourceUnavailableError(
|
||||
"PostgreSQL tabular sources require a credential envelope reference."
|
||||
)
|
||||
return configuration, url, credential_revision
|
||||
|
||||
|
||||
def parse_managed_tabular_content(
|
||||
payload: bytes,
|
||||
*,
|
||||
filename: str,
|
||||
content_type: str | None,
|
||||
delimiter: str,
|
||||
sheet_name: str | None,
|
||||
) -> tuple[tuple[Mapping[str, object], ...], str | None]:
|
||||
if len(payload) > MAX_FILE_BYTES:
|
||||
raise TabularSourceValidationError(
|
||||
f"Managed tabular files are limited to {MAX_FILE_BYTES:,} bytes."
|
||||
)
|
||||
if _is_xlsx(filename, content_type):
|
||||
return _parse_xlsx(payload, sheet_name=sheet_name)
|
||||
try:
|
||||
text = payload.decode("utf-8-sig")
|
||||
except UnicodeDecodeError as exc:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed CSV files must use UTF-8 encoding."
|
||||
) from exc
|
||||
return (
|
||||
tuple(parse_tabular_csv(text, delimiter=delimiter, max_rows=MAX_FILE_ROWS)),
|
||||
None,
|
||||
)
|
||||
|
||||
|
||||
def _parse_xlsx(
|
||||
payload: bytes,
|
||||
*,
|
||||
sheet_name: str | None,
|
||||
) -> tuple[tuple[Mapping[str, object], ...], str]:
|
||||
_validate_xlsx_archive(payload)
|
||||
try:
|
||||
workbook = load_workbook(
|
||||
BytesIO(payload),
|
||||
read_only=True,
|
||||
data_only=True,
|
||||
keep_links=False,
|
||||
)
|
||||
except Exception as exc:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content could not be parsed."
|
||||
) from exc
|
||||
try:
|
||||
available = tuple(workbook.sheetnames)
|
||||
if not available:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content requires at least one worksheet."
|
||||
)
|
||||
selected_name = _optional_text(sheet_name) or available[0]
|
||||
if selected_name not in available:
|
||||
raise TabularSourceValidationError(
|
||||
f"Managed XLSX worksheet {selected_name!r} was not found."
|
||||
)
|
||||
worksheet = workbook[selected_name]
|
||||
iterator = worksheet.iter_rows(values_only=True)
|
||||
try:
|
||||
raw_headers = next(iterator)
|
||||
except StopIteration as exc:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX worksheet requires a header row."
|
||||
) from exc
|
||||
headers = _xlsx_headers(raw_headers)
|
||||
rows: list[Mapping[str, object]] = []
|
||||
for values in iterator:
|
||||
if len(rows) >= MAX_FILE_ROWS:
|
||||
raise TabularSourceValidationError(
|
||||
f"Managed XLSX worksheets are limited to {MAX_FILE_ROWS:,} data rows."
|
||||
)
|
||||
normalized = tuple(values[: len(headers)])
|
||||
if all(value in (None, "") for value in normalized):
|
||||
continue
|
||||
rows.append(
|
||||
{
|
||||
header: _json_value(
|
||||
normalized[index] if index < len(normalized) else None
|
||||
)
|
||||
for index, header in enumerate(headers)
|
||||
}
|
||||
)
|
||||
return tuple(rows), selected_name
|
||||
finally:
|
||||
workbook.close()
|
||||
|
||||
|
||||
def _validate_xlsx_archive(payload: bytes) -> None:
|
||||
try:
|
||||
with zipfile.ZipFile(BytesIO(payload)) as archive:
|
||||
entries = archive.infolist()
|
||||
if len(entries) > MAX_XLSX_ENTRIES:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content contains too many archive entries."
|
||||
)
|
||||
expanded = sum(max(0, item.file_size) for item in entries)
|
||||
if expanded > MAX_XLSX_EXPANDED_BYTES:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content exceeds the expanded-size limit."
|
||||
)
|
||||
compressed = sum(max(1, item.compress_size) for item in entries)
|
||||
if expanded > compressed * MAX_XLSX_COMPRESSION_RATIO:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content exceeds the compression-ratio limit."
|
||||
)
|
||||
if any(
|
||||
item.filename.startswith(("/", "\\"))
|
||||
or ".." in item.filename.replace("\\", "/").split("/")
|
||||
for item in entries
|
||||
):
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content contains an unsafe archive path."
|
||||
)
|
||||
except zipfile.BadZipFile as exc:
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content is not a valid workbook archive."
|
||||
) from exc
|
||||
|
||||
|
||||
def _xlsx_headers(values: Sequence[object]) -> tuple[str, ...]:
|
||||
if len(values) > MAX_FILE_COLUMNS:
|
||||
raise TabularSourceValidationError(
|
||||
f"Managed XLSX worksheets are limited to {MAX_FILE_COLUMNS:,} columns."
|
||||
)
|
||||
headers = tuple(str(value or "").strip() for value in values)
|
||||
while headers and not headers[-1]:
|
||||
headers = headers[:-1]
|
||||
if not headers or any(not header for header in headers):
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX content requires a non-empty header row."
|
||||
)
|
||||
if len(set(headers)) != len(headers):
|
||||
raise TabularSourceValidationError(
|
||||
"Managed XLSX column names must be unique."
|
||||
)
|
||||
return headers
|
||||
|
||||
|
||||
def infer_tabular_schema(
|
||||
rows: Sequence[Mapping[str, object]],
|
||||
) -> tuple[TabularColumn, ...]:
|
||||
names: list[str] = []
|
||||
for row in rows:
|
||||
for name in row:
|
||||
if name not in names:
|
||||
names.append(name)
|
||||
result: list[TabularColumn] = []
|
||||
for name in names:
|
||||
values = [row.get(name) for row in rows]
|
||||
concrete = [value for value in values if value is not None]
|
||||
data_type = _type_name(concrete[0]) if concrete else "unknown"
|
||||
if any(_type_name(value) != data_type for value in concrete[1:]):
|
||||
data_type = "mixed"
|
||||
result.append(
|
||||
TabularColumn(
|
||||
name=name,
|
||||
data_type=data_type,
|
||||
nullable=len(concrete) != len(values),
|
||||
)
|
||||
)
|
||||
return tuple(result)
|
||||
|
||||
|
||||
def origin_fingerprint(
|
||||
schema: Sequence[TabularColumn],
|
||||
*,
|
||||
tokens: Sequence[str],
|
||||
) -> str:
|
||||
payload = {
|
||||
"schema": [
|
||||
{
|
||||
"name": column.name,
|
||||
"data_type": column.data_type,
|
||||
"nullable": column.nullable,
|
||||
}
|
||||
for column in schema
|
||||
],
|
||||
"tokens": list(tokens),
|
||||
}
|
||||
return hashlib.sha256(
|
||||
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _sql_column(column: Any) -> TabularColumn:
|
||||
column_type = column.type
|
||||
if isinstance(column_type, Boolean):
|
||||
data_type = "boolean"
|
||||
elif isinstance(column_type, (Integer, BigInteger)):
|
||||
data_type = "integer"
|
||||
elif isinstance(column_type, (Numeric, Float)):
|
||||
data_type = "number"
|
||||
elif isinstance(column_type, DateTime):
|
||||
data_type = "datetime"
|
||||
elif isinstance(column_type, Date):
|
||||
data_type = "date"
|
||||
elif isinstance(column_type, JSON):
|
||||
data_type = "object"
|
||||
elif isinstance(column_type, LargeBinary):
|
||||
data_type = "binary"
|
||||
elif isinstance(column_type, (String, Text)):
|
||||
data_type = "string"
|
||||
else:
|
||||
data_type = str(column_type).casefold()
|
||||
return TabularColumn(
|
||||
name=str(column.name),
|
||||
data_type=data_type,
|
||||
nullable=bool(column.nullable),
|
||||
)
|
||||
|
||||
|
||||
def _configure_read_only(connection: Any, url: URL, timeout_ms: int) -> None:
|
||||
if url.drivername not in POSTGRESQL_SCHEMES:
|
||||
return
|
||||
effective_timeout = max(1, min(int(timeout_ms), 30_000))
|
||||
connection.exec_driver_sql("SET TRANSACTION READ ONLY")
|
||||
connection.exec_driver_sql(f"SET LOCAL statement_timeout = {effective_timeout}")
|
||||
|
||||
|
||||
def _bounded_connection_url(url: URL, timeout_ms: int) -> URL:
|
||||
if url.drivername not in POSTGRESQL_SCHEMES:
|
||||
return url
|
||||
connect_timeout = max(1, math.ceil(min(int(timeout_ms), 30_000) / 1_000))
|
||||
return url.update_query_dict({"connect_timeout": str(connect_timeout)})
|
||||
|
||||
|
||||
def _sql_identifier(value: object, label: str) -> str:
|
||||
normalized = str(value or "").strip()
|
||||
if not _IDENTIFIER.fullmatch(normalized):
|
||||
raise TabularSourceValidationError(
|
||||
f"SQL {label} names must use a simple identifier."
|
||||
)
|
||||
return normalized
|
||||
|
||||
|
||||
def _required_metadata(metadata: Mapping[str, object], key: str) -> str:
|
||||
value = _optional_text(metadata.get(key))
|
||||
if not value:
|
||||
raise TabularSourceValidationError(
|
||||
f"Tabular source metadata is missing {key}."
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _optional_text(value: object | None) -> str | None:
|
||||
normalized = str(value or "").strip()
|
||||
return normalized or None
|
||||
|
||||
|
||||
def _principal_user_id(principal: ApiPrincipal) -> str | None:
|
||||
return _optional_text(getattr(principal.user, "id", None))
|
||||
|
||||
|
||||
def _is_xlsx(filename: str, content_type: str | None) -> bool:
|
||||
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
|
||||
return str(filename or "").strip().casefold().endswith(".xlsx") or normalized_type == (
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||
)
|
||||
|
||||
|
||||
def _json_row(row: Mapping[str, object]) -> dict[str, object]:
|
||||
return {str(key): _json_value(value) for key, value in row.items()}
|
||||
|
||||
|
||||
def _json_value(value: object) -> object:
|
||||
if value is None or isinstance(value, (str, bool, int)):
|
||||
return value
|
||||
if isinstance(value, float):
|
||||
if not math.isfinite(value):
|
||||
return str(value)
|
||||
return value
|
||||
if isinstance(value, Decimal):
|
||||
return float(value)
|
||||
if isinstance(value, (datetime, date)):
|
||||
return value.isoformat()
|
||||
if isinstance(value, bytes):
|
||||
return value.hex()
|
||||
if isinstance(value, (list, dict)):
|
||||
return json.loads(json.dumps(value, default=str))
|
||||
return str(value)
|
||||
|
||||
|
||||
def _type_name(value: object) -> str:
|
||||
if isinstance(value, bool):
|
||||
return "boolean"
|
||||
if isinstance(value, int):
|
||||
return "integer"
|
||||
if isinstance(value, (float, Decimal)):
|
||||
return "number"
|
||||
if isinstance(value, str):
|
||||
return "string"
|
||||
if isinstance(value, list):
|
||||
return "array"
|
||||
if isinstance(value, dict):
|
||||
return "object"
|
||||
return type(value).__name__.casefold()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MAX_FILE_BYTES",
|
||||
"MAX_FILE_ROWS",
|
||||
"ManagedFileTabularAdapter",
|
||||
"PostgresqlTabularAdapter",
|
||||
"TabularOriginInspection",
|
||||
"TabularOriginRead",
|
||||
"infer_tabular_schema",
|
||||
"origin_fingerprint",
|
||||
"parse_managed_tabular_content",
|
||||
]
|
||||
@@ -27,8 +27,14 @@ from govoplan_core.core.tabular_sources import (
|
||||
TabularSourceValidationError,
|
||||
parse_tabular_csv,
|
||||
)
|
||||
from govoplan_core.core.runtime import get_registry
|
||||
from govoplan_core.db.base import utcnow
|
||||
from govoplan_connectors.backend.db.models import ConnectorTabularSource
|
||||
from govoplan_connectors.backend.tabular_adapters import (
|
||||
ManagedFileTabularAdapter,
|
||||
PostgresqlTabularAdapter,
|
||||
TabularOriginInspection,
|
||||
)
|
||||
|
||||
|
||||
READ_SCOPE = "connectors:source:read"
|
||||
@@ -42,8 +48,16 @@ MAX_READ_TIMEOUT_MS = 2_000
|
||||
|
||||
|
||||
class SqlTabularSourceProvider:
|
||||
def __init__(self, *, clock: Callable[[], float] = time.monotonic) -> None:
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
registry: object | None = None,
|
||||
clock: Callable[[], float] = time.monotonic,
|
||||
sql_adapter: PostgresqlTabularAdapter | None = None,
|
||||
) -> None:
|
||||
self._registry = registry
|
||||
self._clock = clock
|
||||
self._sql_adapter = sql_adapter or PostgresqlTabularAdapter()
|
||||
|
||||
def list_sources(
|
||||
self,
|
||||
@@ -109,7 +123,6 @@ class SqlTabularSourceProvider:
|
||||
"The source fingerprint changed; refresh the source node before running it."
|
||||
)
|
||||
|
||||
started = self._clock()
|
||||
limit = max(1, min(int(request.limit), MAX_READ_ROWS))
|
||||
byte_limit = max(2, min(int(request.max_bytes), MAX_READ_BYTES))
|
||||
timeout_ms = max(1, min(int(request.timeout_ms), MAX_READ_TIMEOUT_MS))
|
||||
@@ -149,10 +162,51 @@ class SqlTabularSourceProvider:
|
||||
raise TabularSourceValidationError(
|
||||
f"Unknown source columns: {', '.join(unknown_columns)}"
|
||||
)
|
||||
started = self._clock()
|
||||
source_rows: Sequence[Mapping[str, object]]
|
||||
total_rows = item.row_count
|
||||
source = _source_dto(item)
|
||||
base_offset = offset
|
||||
if item.provider == "managed_file":
|
||||
read = self._file_adapter().read(
|
||||
db,
|
||||
api_principal,
|
||||
metadata=item.metadata_,
|
||||
)
|
||||
if read.inspection.fingerprint != item.fingerprint:
|
||||
raise TabularSourceValidationError(
|
||||
"The managed file source changed; refresh and review it before previewing."
|
||||
)
|
||||
source_rows = read.rows[offset:]
|
||||
total_rows = read.total_rows
|
||||
source = _source_dto(item, inspection=read.inspection)
|
||||
diagnostics.extend(read.diagnostics)
|
||||
elif item.provider == "postgresql":
|
||||
read = self._sql_adapter.read(
|
||||
db,
|
||||
api_principal,
|
||||
metadata=item.metadata_,
|
||||
columns=selected_columns,
|
||||
offset=offset,
|
||||
limit=limit + 1,
|
||||
timeout_ms=timeout_ms,
|
||||
)
|
||||
source_rows = read.rows
|
||||
total_rows = read.total_rows
|
||||
source = _source_dto(item, inspection=read.inspection)
|
||||
diagnostics.extend(read.diagnostics)
|
||||
else:
|
||||
source_rows = item.rows[offset:]
|
||||
|
||||
if int(max(0.0, self._clock() - started) * 1_000) >= timeout_ms:
|
||||
raise TabularSourceUnavailableError(
|
||||
"Tabular source preview exceeded its time budget."
|
||||
)
|
||||
|
||||
rows: list[dict[str, object]] = []
|
||||
returned_bytes = 2
|
||||
stopped_for = ""
|
||||
for row in item.rows[offset:]:
|
||||
for row in source_rows:
|
||||
if len(rows) >= limit:
|
||||
stopped_for = "rows"
|
||||
break
|
||||
@@ -203,10 +257,10 @@ class SqlTabularSourceProvider:
|
||||
)
|
||||
)
|
||||
return TabularReadResult(
|
||||
source=_source_dto(item),
|
||||
source=source,
|
||||
rows=tuple(rows),
|
||||
total_rows=item.row_count,
|
||||
truncated=offset + len(rows) < item.row_count,
|
||||
total_rows=total_rows,
|
||||
truncated=base_offset + len(rows) < total_rows,
|
||||
returned_bytes=returned_bytes,
|
||||
elapsed_ms=elapsed_ms,
|
||||
effective_row_limit=limit,
|
||||
@@ -215,6 +269,173 @@ class SqlTabularSourceProvider:
|
||||
diagnostics=tuple(diagnostics),
|
||||
)
|
||||
|
||||
def create_file_source(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
name: str,
|
||||
source_name: str,
|
||||
file_asset_id: str,
|
||||
file_version_id: str | None = None,
|
||||
description: str | None = None,
|
||||
delimiter: str = ",",
|
||||
sheet_name: str | None = None,
|
||||
) -> TabularSource:
|
||||
db, api_principal = _context(session, principal, WRITE_SCOPE)
|
||||
inspection = self._file_adapter().inspect(
|
||||
db,
|
||||
api_principal,
|
||||
file_asset_id=file_asset_id,
|
||||
file_version_id=file_version_id,
|
||||
delimiter=delimiter,
|
||||
sheet_name=sheet_name,
|
||||
)
|
||||
return self._create_origin(
|
||||
db,
|
||||
api_principal,
|
||||
name=name,
|
||||
source_name=source_name,
|
||||
description=description,
|
||||
inspection=inspection,
|
||||
)
|
||||
|
||||
def create_sql_source(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
name: str,
|
||||
source_name: str,
|
||||
configuration_id: str,
|
||||
table_name: str,
|
||||
schema_name: str | None = None,
|
||||
description: str | None = None,
|
||||
) -> TabularSource:
|
||||
db, api_principal = _context(session, principal, WRITE_SCOPE)
|
||||
inspection = self._sql_adapter.inspect(
|
||||
db,
|
||||
api_principal,
|
||||
configuration_id=configuration_id,
|
||||
table_name=table_name,
|
||||
schema_name=schema_name,
|
||||
timeout_ms=MAX_READ_TIMEOUT_MS,
|
||||
)
|
||||
return self._create_origin(
|
||||
db,
|
||||
api_principal,
|
||||
name=name,
|
||||
source_name=source_name,
|
||||
description=description,
|
||||
inspection=inspection,
|
||||
)
|
||||
|
||||
def refresh_source(
|
||||
self,
|
||||
session: object,
|
||||
principal: object,
|
||||
*,
|
||||
source_ref: str,
|
||||
) -> TabularSource:
|
||||
db, api_principal = _context(session, principal, WRITE_SCOPE)
|
||||
item = _source_record(
|
||||
db,
|
||||
tenant_id=api_principal.tenant_id,
|
||||
source_ref=source_ref,
|
||||
)
|
||||
if item is None:
|
||||
raise TabularSourceNotFoundError("Tabular source not found.")
|
||||
if item.provider == "managed_file":
|
||||
inspection = self._file_adapter().inspect(
|
||||
db,
|
||||
api_principal,
|
||||
file_asset_id=str(item.metadata_.get("file_asset_id") or ""),
|
||||
file_version_id=None,
|
||||
delimiter=str(item.metadata_.get("delimiter") or ","),
|
||||
sheet_name=_clean_optional(item.metadata_.get("sheet_name")),
|
||||
)
|
||||
elif item.provider == "postgresql":
|
||||
inspection = self._sql_adapter.inspect(
|
||||
db,
|
||||
api_principal,
|
||||
configuration_id=str(
|
||||
item.metadata_.get("configuration_id") or ""
|
||||
),
|
||||
table_name=str(item.metadata_.get("table_name") or ""),
|
||||
schema_name=_clean_optional(item.metadata_.get("schema_name")),
|
||||
timeout_ms=MAX_READ_TIMEOUT_MS,
|
||||
)
|
||||
else:
|
||||
raise TabularSourceValidationError(
|
||||
"Immutable snapshots cannot be refreshed; import a replacement snapshot."
|
||||
)
|
||||
item.schema_version += 1
|
||||
item.schema_ = [_column_payload(column) for column in inspection.schema]
|
||||
item.fingerprint = inspection.fingerprint
|
||||
item.row_count = inspection.row_count
|
||||
item.byte_count = inspection.byte_count
|
||||
item.metadata_ = {
|
||||
**dict(inspection.metadata),
|
||||
"discovery_fingerprint": inspection.fingerprint,
|
||||
}
|
||||
item.updated_by = _actor_id(api_principal)
|
||||
db.flush()
|
||||
return _source_dto(item, inspection=inspection)
|
||||
|
||||
def _create_origin(
|
||||
self,
|
||||
db: Session,
|
||||
principal: ApiPrincipal,
|
||||
*,
|
||||
name: str,
|
||||
source_name: str,
|
||||
description: str | None,
|
||||
inspection: TabularOriginInspection,
|
||||
) -> TabularSource:
|
||||
clean_name = str(name or "").strip()
|
||||
clean_source_name = str(source_name or "").strip()
|
||||
if not clean_name or not clean_source_name:
|
||||
raise TabularSourceValidationError(
|
||||
"Tabular source name and source identifier are required."
|
||||
)
|
||||
existing = db.scalar(
|
||||
select(ConnectorTabularSource.id).where(
|
||||
ConnectorTabularSource.tenant_id == principal.tenant_id,
|
||||
ConnectorTabularSource.source_name == clean_source_name,
|
||||
)
|
||||
)
|
||||
if existing is not None:
|
||||
raise TabularSourceValidationError(
|
||||
f"A tabular source named {clean_source_name!r} already exists."
|
||||
)
|
||||
actor_id = _actor_id(principal)
|
||||
item = ConnectorTabularSource(
|
||||
tenant_id=principal.tenant_id,
|
||||
provider=inspection.provider,
|
||||
source_name=clean_source_name,
|
||||
name=clean_name,
|
||||
description=_clean_optional(description),
|
||||
status="active",
|
||||
schema_version=1,
|
||||
schema_=[_column_payload(column) for column in inspection.schema],
|
||||
rows=[],
|
||||
fingerprint=inspection.fingerprint,
|
||||
row_count=inspection.row_count,
|
||||
byte_count=inspection.byte_count,
|
||||
metadata_={
|
||||
**dict(inspection.metadata),
|
||||
"discovery_fingerprint": inspection.fingerprint,
|
||||
},
|
||||
created_by=actor_id,
|
||||
updated_by=actor_id,
|
||||
)
|
||||
db.add(item)
|
||||
db.flush()
|
||||
return _source_dto(item, inspection=inspection)
|
||||
|
||||
def _file_adapter(self) -> ManagedFileTabularAdapter:
|
||||
return ManagedFileTabularAdapter(self._registry or get_registry())
|
||||
|
||||
def create_snapshot(
|
||||
self,
|
||||
session: object,
|
||||
@@ -347,21 +568,81 @@ def _source_record(
|
||||
tenant_id: str,
|
||||
source_ref: str,
|
||||
) -> ConnectorTabularSource | None:
|
||||
source_id = source_ref.removeprefix("snapshot:")
|
||||
if not source_id or source_id == source_ref:
|
||||
prefix, separator, source_id = str(source_ref or "").partition(":")
|
||||
provider_by_prefix = {
|
||||
"snapshot": "snapshot",
|
||||
"file": "managed_file",
|
||||
"sql": "postgresql",
|
||||
}
|
||||
provider = provider_by_prefix.get(prefix)
|
||||
if not separator or not source_id or provider is None:
|
||||
return None
|
||||
return session.scalar(
|
||||
select(ConnectorTabularSource).where(
|
||||
ConnectorTabularSource.id == source_id,
|
||||
ConnectorTabularSource.tenant_id == tenant_id,
|
||||
ConnectorTabularSource.provider == provider,
|
||||
ConnectorTabularSource.deleted_at.is_(None),
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
def _source_dto(item: ConnectorTabularSource) -> TabularSource:
|
||||
def _source_dto(
|
||||
item: ConnectorTabularSource,
|
||||
*,
|
||||
inspection: TabularOriginInspection | None = None,
|
||||
) -> TabularSource:
|
||||
prefix_by_provider = {
|
||||
"snapshot": "snapshot",
|
||||
"managed_file": "file",
|
||||
"postgresql": "sql",
|
||||
}
|
||||
source_mode = {
|
||||
"managed_file": "file_backed",
|
||||
"postgresql": "live",
|
||||
}.get(item.provider, "cached")
|
||||
if inspection is not None:
|
||||
pushdown = inspection.pushdown
|
||||
health = inspection.health
|
||||
elif item.provider == "managed_file":
|
||||
pushdown = TabularPushdown(projections=True, pagination=True)
|
||||
health = TabularSourceHealth(
|
||||
status="unknown",
|
||||
code="files.exact_version_not_checked",
|
||||
summary="The exact managed file version will be re-authorized and integrity-checked on preview.",
|
||||
checked_at=item.updated_at,
|
||||
details={
|
||||
"file_version_id": item.metadata_.get("file_version_id"),
|
||||
},
|
||||
)
|
||||
elif item.provider == "postgresql":
|
||||
pushdown = TabularPushdown(projections=True, pagination=True)
|
||||
health = TabularSourceHealth(
|
||||
status="unknown",
|
||||
code="sql.health_not_checked",
|
||||
summary="The live SQL source will be checked against its pinned configuration, credential, and schema on preview.",
|
||||
checked_at=item.updated_at,
|
||||
details={
|
||||
"configuration_id": item.metadata_.get("configuration_id"),
|
||||
"configuration_revision": item.metadata_.get(
|
||||
"configuration_revision"
|
||||
),
|
||||
},
|
||||
)
|
||||
else:
|
||||
pushdown = TabularPushdown(
|
||||
projections=True,
|
||||
pagination=True,
|
||||
)
|
||||
health = TabularSourceHealth(
|
||||
status="healthy",
|
||||
code="snapshot.ready",
|
||||
summary="The immutable connector snapshot is ready.",
|
||||
checked_at=item.updated_at,
|
||||
details={"immutable": True},
|
||||
)
|
||||
return TabularSource(
|
||||
ref=f"snapshot:{item.id}",
|
||||
ref=f"{prefix_by_provider.get(item.provider, 'snapshot')}:{item.id}",
|
||||
provider=item.provider,
|
||||
source_name=item.source_name,
|
||||
name=item.name,
|
||||
@@ -374,18 +655,9 @@ def _source_dto(item: ConnectorTabularSource) -> TabularSource:
|
||||
updated_at=item.updated_at,
|
||||
capabilities=("read", "preview"),
|
||||
metadata=dict(item.metadata_),
|
||||
source_mode="cached",
|
||||
pushdown=TabularPushdown(
|
||||
projections=True,
|
||||
pagination=True,
|
||||
),
|
||||
health=TabularSourceHealth(
|
||||
status="healthy",
|
||||
code="snapshot.ready",
|
||||
summary="The immutable connector snapshot is ready.",
|
||||
checked_at=item.updated_at,
|
||||
details={"immutable": True},
|
||||
),
|
||||
source_mode=source_mode,
|
||||
pushdown=pushdown,
|
||||
health=health,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,339 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import unittest
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.dsar import DsarProvider, DsarSubjectRef
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.privacy.dsar_workflow import (
|
||||
create_data_subject_request,
|
||||
search_data_subject_request,
|
||||
)
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorSanctionsAcquisitionRun,
|
||||
ConnectorSimulationRun,
|
||||
ConnectorTabularSource,
|
||||
)
|
||||
from govoplan_connectors.backend.dsar_provider import (
|
||||
CONNECTORS_DSAR_CAPABILITY,
|
||||
ConnectorsDsarProvider,
|
||||
)
|
||||
from govoplan_connectors.backend.manifest import manifest
|
||||
|
||||
|
||||
NOW = datetime(2026, 8, 22, 9, 0, tzinfo=UTC)
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, provider: ConnectorsDsarProvider) -> None:
|
||||
self.provider = provider
|
||||
|
||||
def capability_names(self):
|
||||
return (CONNECTORS_DSAR_CAPABILITY,)
|
||||
|
||||
def capability_owner(self, name):
|
||||
if name != CONNECTORS_DSAR_CAPABILITY:
|
||||
raise KeyError(name)
|
||||
return "connectors"
|
||||
|
||||
def tenant_entitlement_resolver(self):
|
||||
class _Resolver:
|
||||
@staticmethod
|
||||
def resolve(session, tenant_id):
|
||||
del session, tenant_id
|
||||
return type("State", (), {"effective_modules": ("connectors",)})()
|
||||
|
||||
return _Resolver()
|
||||
|
||||
def require_tenant_capability(self, name, session, **kwargs):
|
||||
del session, kwargs
|
||||
if name != CONNECTORS_DSAR_CAPABILITY:
|
||||
raise KeyError(name)
|
||||
return self.provider
|
||||
|
||||
def manifests(self):
|
||||
return (type("Manifest", (), {"id": "connectors"})(),)
|
||||
|
||||
|
||||
class ConnectorsDsarProviderTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(self.engine)
|
||||
self.session = Session(self.engine)
|
||||
self.provider = ConnectorsDsarProvider()
|
||||
self.assertIsInstance(self.provider, DsarProvider)
|
||||
self._seed()
|
||||
self.session.commit()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.engine.dispose()
|
||||
|
||||
def _seed(self) -> None:
|
||||
self.session.add_all(
|
||||
(
|
||||
ConnectorTabularSource(
|
||||
id="source-1",
|
||||
tenant_id="tenant-1",
|
||||
provider="snapshot",
|
||||
source_name="people",
|
||||
name="People import",
|
||||
description="Do not export this business description",
|
||||
status="active",
|
||||
schema_version=1,
|
||||
schema_=[{"name": "email"}],
|
||||
rows=[{"email": "third-party@example.test"}],
|
||||
fingerprint="source-fingerprint-do-not-export",
|
||||
row_count=1,
|
||||
byte_count=100,
|
||||
metadata_={"secret": "source-metadata-do-not-export"},
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
ConnectorTabularSource(
|
||||
id="source-other",
|
||||
tenant_id="tenant-2",
|
||||
provider="snapshot",
|
||||
source_name="other",
|
||||
name="Other tenant",
|
||||
status="active",
|
||||
schema_version=1,
|
||||
schema_=[],
|
||||
rows=[],
|
||||
fingerprint="other-fingerprint",
|
||||
row_count=0,
|
||||
byte_count=0,
|
||||
metadata_={},
|
||||
created_by="account-1",
|
||||
updated_by="account-1",
|
||||
),
|
||||
ConnectorSanctionsAcquisitionRun(
|
||||
id="acquisition-1",
|
||||
tenant_id="tenant-1",
|
||||
provider_id="un",
|
||||
source_id="consolidated",
|
||||
status="complete",
|
||||
attempt_count=1,
|
||||
request_evidence={"secret": "request-evidence-do-not-export"},
|
||||
response_evidence={"secret": "response-evidence-do-not-export"},
|
||||
started_at=NOW,
|
||||
finished_at=NOW,
|
||||
snapshot_id="snapshot-1",
|
||||
error="transport-detail-do-not-export",
|
||||
created_by="account-1",
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
ConnectorDefinition(
|
||||
id="definition-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_key="address-reader",
|
||||
name="Address reader",
|
||||
description="Definition detail",
|
||||
status="active",
|
||||
current_revision=1,
|
||||
source_package="package-secret-do-not-export",
|
||||
local_definition=True,
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
ConnectorDefinitionRevision(
|
||||
id="definition-revision-1",
|
||||
definition_id="definition-1",
|
||||
revision=1,
|
||||
specification={"secret": "specification-do-not-export"},
|
||||
definition_hash="definition-hash-do-not-export",
|
||||
origin="local",
|
||||
package_ref="package-ref-do-not-export",
|
||||
created_by="account-1",
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
ConnectorConfiguration(
|
||||
id="configuration-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_id="definition-1",
|
||||
name="Production addresses",
|
||||
status="active",
|
||||
endpoint_url="https://secret.example.test/api",
|
||||
credential_ref="vault://secret-do-not-export",
|
||||
base_definition_revision=1,
|
||||
local_overrides={"secret": "override-do-not-export"},
|
||||
protected_paths=["secret"],
|
||||
effective_configuration={"secret": "effective-do-not-export"},
|
||||
effective_hash="configuration-hash-do-not-export",
|
||||
resource_revision=2,
|
||||
ambiguity_policy="manual_review",
|
||||
updated_by="account-1",
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
ConnectorSimulationRun(
|
||||
id="simulation-1",
|
||||
tenant_id="tenant-1",
|
||||
configuration_id="configuration-1",
|
||||
mode="dry_run",
|
||||
idempotency_key="simulation-idempotency-do-not-export",
|
||||
request_hash="simulation-request-hash-do-not-export",
|
||||
status="complete",
|
||||
review_state="approved",
|
||||
definition_revision=1,
|
||||
configuration_revision=2,
|
||||
configuration_hash="simulation-config-hash-do-not-export",
|
||||
input_hash="simulation-input-hash-do-not-export",
|
||||
summary={"secret": "summary-do-not-export"},
|
||||
effects=[{"secret": "effect-do-not-export"}],
|
||||
diagnostics=[{"secret": "diagnostic-do-not-export"}],
|
||||
provenance={"secret": "provenance-do-not-export"},
|
||||
created_by="account-1",
|
||||
reviewed_by="account-1",
|
||||
reviewed_at=NOW,
|
||||
review_reason="review-reason-do-not-export",
|
||||
created_at=NOW,
|
||||
updated_at=NOW,
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _subject() -> DsarSubjectRef:
|
||||
return DsarSubjectRef(account_id="account-1")
|
||||
|
||||
def test_search_exports_minimized_operator_attribution(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session, tenant_id="tenant-1", subject=self._subject()
|
||||
)
|
||||
self.assertEqual(
|
||||
{
|
||||
"source_actor_attribution",
|
||||
"acquisition_actor_attribution",
|
||||
"definition_actor_attribution",
|
||||
"configuration_actor_attribution",
|
||||
"simulation_actor_attribution",
|
||||
},
|
||||
{record.resource_type for record in records},
|
||||
)
|
||||
exported = json.dumps([record.to_dict() for record in records])
|
||||
for excluded in (
|
||||
"third-party@example.test",
|
||||
"source-fingerprint-do-not-export",
|
||||
"source-metadata-do-not-export",
|
||||
"request-evidence-do-not-export",
|
||||
"response-evidence-do-not-export",
|
||||
"transport-detail-do-not-export",
|
||||
"specification-do-not-export",
|
||||
"definition-hash-do-not-export",
|
||||
"package-ref-do-not-export",
|
||||
"https://secret.example.test/api",
|
||||
"vault://secret-do-not-export",
|
||||
"override-do-not-export",
|
||||
"effective-do-not-export",
|
||||
"configuration-hash-do-not-export",
|
||||
"simulation-idempotency-do-not-export",
|
||||
"simulation-request-hash-do-not-export",
|
||||
"simulation-config-hash-do-not-export",
|
||||
"simulation-input-hash-do-not-export",
|
||||
"summary-do-not-export",
|
||||
"effect-do-not-export",
|
||||
"diagnostic-do-not-export",
|
||||
"provenance-do-not-export",
|
||||
"review-reason-do-not-export",
|
||||
):
|
||||
self.assertNotIn(excluded, exported)
|
||||
|
||||
def test_requires_exact_account_and_enforces_tenant(self) -> None:
|
||||
email_only = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(email="operator@example.test"),
|
||||
)
|
||||
conflict = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
account_id="account-1",
|
||||
external_references={"connectors.account": "account-other"},
|
||||
),
|
||||
)
|
||||
records = self.provider.search_subject(
|
||||
self.session, tenant_id="tenant-1", subject=self._subject()
|
||||
)
|
||||
self.assertEqual((), email_only)
|
||||
self.assertEqual((), conflict)
|
||||
self.assertNotIn("source-other", {record.resource_id for record in records})
|
||||
|
||||
def test_object_narrowing_does_not_broaden_the_search(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=DsarSubjectRef(
|
||||
account_id="account-1",
|
||||
external_references={"connectors.simulation": "simulation-1"},
|
||||
),
|
||||
)
|
||||
self.assertEqual(
|
||||
[("simulation_actor_attribution", "simulation-1")],
|
||||
[(record.resource_type, record.resource_id) for record in records],
|
||||
)
|
||||
|
||||
def test_erasure_retains_external_operation_evidence(self) -> None:
|
||||
records = self.provider.search_subject(
|
||||
self.session, tenant_id="tenant-1", subject=self._subject()
|
||||
)
|
||||
actions = self.provider.plan_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self._subject(),
|
||||
records=records,
|
||||
)
|
||||
self.assertTrue(actions)
|
||||
self.assertTrue(
|
||||
all(action.kind == "retain" and not action.executable for action in actions)
|
||||
)
|
||||
results = self.provider.execute_erasure(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
subject=self._subject(),
|
||||
actions=actions,
|
||||
request_id="dsar-connectors-1",
|
||||
)
|
||||
self.assertTrue(all(result.status == "blocked" for result in results))
|
||||
|
||||
def test_manifest_and_core_workflow_discover_provider(self) -> None:
|
||||
self.assertIn(CONNECTORS_DSAR_CAPABILITY, manifest.capability_factories)
|
||||
self.assertIn(
|
||||
"connectors.data-subject-requests",
|
||||
{topic.id for topic in manifest.documentation},
|
||||
)
|
||||
row = create_data_subject_request(
|
||||
self.session,
|
||||
tenant_id="tenant-1",
|
||||
reference="DSAR-CONNECTORS-1",
|
||||
request_kind="access",
|
||||
subject=self._subject(),
|
||||
purpose="Connector attribution access request",
|
||||
legal_basis=None,
|
||||
due_at=None,
|
||||
requested_by_account_id="operator-1",
|
||||
)
|
||||
search_data_subject_request(
|
||||
self.session,
|
||||
registry=_Registry(self.provider),
|
||||
row=row,
|
||||
expected_revision=row.resource_revision,
|
||||
)
|
||||
self.assertEqual("searched", row.status)
|
||||
self.assertEqual(5, row.search_result["record_count"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+78
-1
@@ -4,8 +4,18 @@ import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from defusedxml import ElementTree as SafeET
|
||||
from fastapi import HTTPException
|
||||
|
||||
from govoplan_connectors.backend.feeds import ConnectorFeedProvider, feed_rows
|
||||
from govoplan_connectors.backend.feeds import (
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
FEED_PUBLISH_SCOPE,
|
||||
ConnectorFeedProvider,
|
||||
feed_rows,
|
||||
)
|
||||
from govoplan_connectors.backend.router import api_render_feed
|
||||
from govoplan_connectors.backend.schemas import FeedRenderPayload
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.feeds import (
|
||||
FeedCapabilityError,
|
||||
FeedEntry,
|
||||
@@ -98,6 +108,73 @@ class ConnectorFeedProviderTests(unittest.TestCase):
|
||||
with self.assertRaisesRegex(FeedCapabilityError, "not safe or valid"):
|
||||
self.provider.parse(payload, source_url="https://example.test/rss")
|
||||
|
||||
def test_render_api_derives_visibility_from_audience_and_permissions(self) -> None:
|
||||
entries = [
|
||||
{
|
||||
"id": visibility,
|
||||
"title": visibility.title(),
|
||||
"visibility": visibility,
|
||||
"source_kind": source_kind,
|
||||
"source_module": source_module,
|
||||
"source_ref": f"{source_kind}:{visibility}",
|
||||
"source_revision": "7",
|
||||
}
|
||||
for visibility, source_kind, source_module in (
|
||||
("public", "publication", "docs"),
|
||||
("tenant", "case", "cases"),
|
||||
("private", "report", "reporting"),
|
||||
)
|
||||
]
|
||||
public_payload = FeedRenderPayload(
|
||||
format="rss",
|
||||
title="Selected GovOPlaN updates",
|
||||
feed_url="https://example.test/feed.xml",
|
||||
home_url="https://example.test/",
|
||||
audience="public",
|
||||
entries=entries,
|
||||
)
|
||||
public = api_render_feed(
|
||||
public_payload,
|
||||
principal=_principal(FEED_PUBLISH_SCOPE),
|
||||
)
|
||||
|
||||
self.assertEqual("public", public.headers["x-govoplan-feed-audience"])
|
||||
self.assertEqual("1", public.headers["x-govoplan-feed-included"])
|
||||
self.assertEqual("2", public.headers["x-govoplan-feed-excluded"])
|
||||
self.assertIn(b"Public", public.body)
|
||||
self.assertNotIn(b"Tenant", public.body)
|
||||
|
||||
restricted_payload = public_payload.model_copy(update={"audience": "private"})
|
||||
with self.assertRaises(HTTPException) as denied:
|
||||
api_render_feed(
|
||||
restricted_payload,
|
||||
principal=_principal(FEED_PUBLISH_SCOPE),
|
||||
)
|
||||
self.assertEqual(403, denied.exception.status_code)
|
||||
|
||||
restricted = api_render_feed(
|
||||
restricted_payload,
|
||||
principal=_principal(
|
||||
FEED_PUBLISH_SCOPE,
|
||||
FEED_PRIVATE_PUBLISH_SCOPE,
|
||||
),
|
||||
)
|
||||
self.assertEqual("3", restricted.headers["x-govoplan-feed-included"])
|
||||
self.assertIn(b"Private", restricted.body)
|
||||
|
||||
|
||||
def _principal(*scopes: str) -> ApiPrincipal:
|
||||
return ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id="tenant-1",
|
||||
scopes=frozenset(scopes),
|
||||
),
|
||||
account=object(),
|
||||
user=object(),
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,339 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import patch
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorDefinitionRevision,
|
||||
ConnectorSimulationRun,
|
||||
)
|
||||
from govoplan_connectors.backend.governed_runtime import (
|
||||
GovernedConnectorError,
|
||||
create_configuration,
|
||||
execute_run,
|
||||
list_configurations,
|
||||
review_run,
|
||||
update_configuration,
|
||||
upsert_definition,
|
||||
)
|
||||
from govoplan_connectors.backend.governed_schemas import (
|
||||
ConnectorConfigurationCreateRequest,
|
||||
ConnectorConfigurationUpdateRequest,
|
||||
ConnectorDefinitionUpsertRequest,
|
||||
ConnectorReviewRequest,
|
||||
ConnectorRunRequest,
|
||||
)
|
||||
|
||||
|
||||
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
|
||||
return ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id=tenant_id,
|
||||
scopes=frozenset(
|
||||
{
|
||||
"connectors:source:read",
|
||||
"connectors:source:write",
|
||||
"connectors:source:admin",
|
||||
}
|
||||
),
|
||||
),
|
||||
account=SimpleNamespace(id="account-1"),
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
)
|
||||
|
||||
|
||||
def definition_payload(
|
||||
*,
|
||||
mapping_version: str = "1",
|
||||
package_ref: str = "municipal-addresses@1",
|
||||
timeout_seconds: int = 10,
|
||||
) -> ConnectorDefinitionUpsertRequest:
|
||||
return ConnectorDefinitionUpsertRequest.model_validate(
|
||||
{
|
||||
"definition_key": "municipal.addresses",
|
||||
"name": "Municipal addresses",
|
||||
"description": "A package-managed reference connector.",
|
||||
"origin": "package",
|
||||
"package_ref": package_ref,
|
||||
"specification": {
|
||||
"provider": "municipal-directory",
|
||||
"protocol": "rest",
|
||||
"capabilities": ["discover", "read", "dry_run"],
|
||||
"input_schema": {"type": "object"},
|
||||
"output_schema": {"type": "object"},
|
||||
"mapping": {
|
||||
"version": mapping_version,
|
||||
"rules": [
|
||||
{
|
||||
"source": "external_id",
|
||||
"target": "address.external_id",
|
||||
"required": True,
|
||||
},
|
||||
{
|
||||
"source": "street",
|
||||
"target": "address.street",
|
||||
"required": True,
|
||||
},
|
||||
],
|
||||
},
|
||||
"validation_rules": [
|
||||
{
|
||||
"kind": "unique",
|
||||
"field": "address.external_id",
|
||||
"severity": "error",
|
||||
"code": "addresses.external_id.ambiguous",
|
||||
"message": "The external identifier is not unique.",
|
||||
}
|
||||
],
|
||||
"dry_run": {
|
||||
"supported": True,
|
||||
"simulation_supported": True,
|
||||
"max_items": 50,
|
||||
"redacted_fields": ["address.street"],
|
||||
"sample_rows": [
|
||||
{"external_id": "A-1", "street": "Sample street"}
|
||||
],
|
||||
},
|
||||
"audit": {
|
||||
"event_prefix": "connectors.municipal_addresses",
|
||||
"expected_events": ["simulation.completed"],
|
||||
"evidence_fields": ["input_hash", "configuration_hash"],
|
||||
},
|
||||
"privacy_classification": "confidential",
|
||||
"retention_class": "connector-preview-30d",
|
||||
"operational_limits": {"timeout_seconds": timeout_seconds},
|
||||
"retry_policy": {"max_attempts": 2},
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
|
||||
class GovernedConnectorRuntimeTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.engine = create_engine("sqlite:///:memory:")
|
||||
self.tables = [
|
||||
ConnectorDefinition.__table__,
|
||||
ConnectorDefinitionRevision.__table__,
|
||||
ConnectorConfiguration.__table__,
|
||||
ConnectorSimulationRun.__table__,
|
||||
]
|
||||
Base.metadata.create_all(self.engine, tables=self.tables)
|
||||
self.Session = sessionmaker(bind=self.engine)
|
||||
self.session = self.Session()
|
||||
self.audit = patch(
|
||||
"govoplan_connectors.backend.governed_runtime.audit_from_principal"
|
||||
)
|
||||
self.audit_mock = self.audit.start()
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.audit.stop()
|
||||
self.session.close()
|
||||
Base.metadata.drop_all(self.engine, tables=reversed(self.tables))
|
||||
self.engine.dispose()
|
||||
|
||||
def _configuration(
|
||||
self,
|
||||
*,
|
||||
ambiguity_policy: str = "manual_review",
|
||||
):
|
||||
definition = upsert_definition(
|
||||
self.session,
|
||||
principal(),
|
||||
definition_payload(),
|
||||
)
|
||||
return create_configuration(
|
||||
self.session,
|
||||
principal(),
|
||||
ConnectorConfigurationCreateRequest(
|
||||
definition_id=definition.id,
|
||||
name=f"Address import {ambiguity_policy}",
|
||||
endpoint_url="https://directory.example.invalid/v1",
|
||||
credential_ref="vault://connectors/address-reader",
|
||||
local_overrides={"retry_policy": {"max_attempts": 5}},
|
||||
ambiguity_policy=ambiguity_policy,
|
||||
status="active",
|
||||
),
|
||||
)
|
||||
|
||||
def test_package_update_is_explicit_and_preserves_local_overrides(self) -> None:
|
||||
configuration = self._configuration()
|
||||
|
||||
updated_definition = upsert_definition(
|
||||
self.session,
|
||||
principal(),
|
||||
definition_payload(
|
||||
mapping_version="2",
|
||||
package_ref="municipal-addresses@2",
|
||||
timeout_seconds=20,
|
||||
),
|
||||
)
|
||||
unchanged = next(
|
||||
item
|
||||
for item in list_configurations(self.session, tenant_id="tenant-1")
|
||||
if item.id == configuration.id
|
||||
)
|
||||
|
||||
self.assertEqual(2, updated_definition.current_revision)
|
||||
self.assertTrue(unchanged.update_available)
|
||||
self.assertEqual("1", unchanged.effective_configuration["mapping"]["version"])
|
||||
self.assertEqual(5, unchanged.effective_configuration["retry_policy"]["max_attempts"])
|
||||
self.assertEqual(["retry_policy.max_attempts"], unchanged.protected_paths)
|
||||
|
||||
adopted = update_configuration(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
payload=ConnectorConfigurationUpdateRequest(
|
||||
expected_revision=configuration.resource_revision,
|
||||
adopt_latest_definition=True,
|
||||
),
|
||||
)
|
||||
|
||||
self.assertFalse(adopted.update_available)
|
||||
self.assertEqual("2", adopted.effective_configuration["mapping"]["version"])
|
||||
self.assertEqual(
|
||||
20,
|
||||
adopted.effective_configuration["operational_limits"]["timeout_seconds"],
|
||||
)
|
||||
self.assertEqual(5, adopted.effective_configuration["retry_policy"]["max_attempts"])
|
||||
self.assertEqual(["retry_policy.max_attempts"], adopted.protected_paths)
|
||||
|
||||
def test_ambiguous_simulation_requires_review_and_is_idempotent(self) -> None:
|
||||
configuration = self._configuration()
|
||||
payload = ConnectorRunRequest(
|
||||
idempotency_key="simulation-1",
|
||||
external_revision="directory-etag-22",
|
||||
input_rows=[
|
||||
{"external_id": "duplicate", "street": "First"},
|
||||
{"external_id": "duplicate", "street": "Second"},
|
||||
],
|
||||
)
|
||||
|
||||
created = execute_run(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
mode="simulation",
|
||||
payload=payload,
|
||||
)
|
||||
replayed = execute_run(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
mode="simulation",
|
||||
payload=payload,
|
||||
)
|
||||
|
||||
self.assertEqual(created.id, replayed.id)
|
||||
self.assertEqual("manual_review", created.status)
|
||||
self.assertEqual("pending", created.review_state)
|
||||
self.assertEqual(2, created.summary["ambiguous"])
|
||||
self.assertEqual("<redacted>", created.effects[0]["sample"]["address"]["street"])
|
||||
self.assertEqual("directory-etag-22", created.provenance["external_revision"])
|
||||
|
||||
reviewed = review_run(
|
||||
self.session,
|
||||
principal(),
|
||||
run_id=created.id,
|
||||
payload=ConnectorReviewRequest(
|
||||
decision="approved",
|
||||
reason="The duplicate rows represent an approved upstream alias.",
|
||||
),
|
||||
)
|
||||
self.assertEqual("approved", reviewed.review_state)
|
||||
self.assertEqual("review_approved", reviewed.status)
|
||||
|
||||
with self.assertRaisesRegex(GovernedConnectorError, "different run inputs"):
|
||||
execute_run(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
mode="simulation",
|
||||
payload=ConnectorRunRequest(
|
||||
idempotency_key="simulation-1",
|
||||
input_rows=[{"external_id": "other", "street": "Other"}],
|
||||
),
|
||||
)
|
||||
|
||||
def test_ambiguity_policy_can_quarantine_or_reject(self) -> None:
|
||||
for policy, expected_status, expected_review in (
|
||||
("quarantine", "quarantined", "quarantined"),
|
||||
("reject", "rejected", "not_required"),
|
||||
):
|
||||
configuration = self._configuration(ambiguity_policy=policy)
|
||||
result = execute_run(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
mode="dry_run",
|
||||
payload=ConnectorRunRequest(
|
||||
idempotency_key=f"{policy}-1",
|
||||
input_rows=[
|
||||
{"external_id": "same", "street": "First"},
|
||||
{"external_id": "same", "street": "Second"},
|
||||
],
|
||||
),
|
||||
)
|
||||
self.assertEqual(expected_status, result.status)
|
||||
self.assertEqual(expected_review, result.review_state)
|
||||
|
||||
def test_endpoint_credentials_and_stale_saves_are_rejected(self) -> None:
|
||||
definition = upsert_definition(
|
||||
self.session,
|
||||
principal(),
|
||||
definition_payload(),
|
||||
)
|
||||
with self.assertRaisesRegex(GovernedConnectorError, "must not contain credentials"):
|
||||
create_configuration(
|
||||
self.session,
|
||||
principal(),
|
||||
ConnectorConfigurationCreateRequest(
|
||||
definition_id=definition.id,
|
||||
name="Unsafe",
|
||||
endpoint_url="https://user:secret@example.invalid/v1",
|
||||
),
|
||||
)
|
||||
|
||||
configuration = self._configuration()
|
||||
with self.assertRaisesRegex(GovernedConnectorError, "reload it before saving"):
|
||||
update_configuration(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=configuration.id,
|
||||
payload=ConnectorConfigurationUpdateRequest(
|
||||
expected_revision=configuration.resource_revision + 1,
|
||||
name="Stale",
|
||||
),
|
||||
)
|
||||
|
||||
def test_definition_ownership_cannot_change_implicitly(self) -> None:
|
||||
package_definition = upsert_definition(
|
||||
self.session,
|
||||
principal(),
|
||||
definition_payload(),
|
||||
)
|
||||
local_payload = definition_payload().model_copy(
|
||||
update={"origin": "local", "package_ref": None}
|
||||
)
|
||||
with self.assertRaisesRegex(
|
||||
GovernedConnectorError,
|
||||
"configuration overrides",
|
||||
):
|
||||
upsert_definition(self.session, principal(), local_payload)
|
||||
|
||||
self.assertFalse(package_definition.local_definition)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -33,6 +33,14 @@ class ConnectorsManifestTests(unittest.TestCase):
|
||||
manifest.capability_factories,
|
||||
)
|
||||
self.assertIsNotNone(manifest.migration_spec)
|
||||
self.assertEqual(
|
||||
"@govoplan/connectors-webui",
|
||||
manifest.frontend.package_name,
|
||||
)
|
||||
self.assertIn(
|
||||
"connectors.governed-configuration",
|
||||
{topic.id for topic in manifest.documentation},
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
+11
-12
@@ -24,20 +24,19 @@ class ConnectorsMigrationTests(unittest.TestCase):
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
self.assertIn(
|
||||
"f7c8d9e0a1b2",
|
||||
"a8d9e0f1b2c3",
|
||||
set(MigrationContext.configure(connection).get_current_heads()),
|
||||
)
|
||||
self.assertIn(
|
||||
"connector_tabular_sources",
|
||||
inspect(connection).get_table_names(),
|
||||
)
|
||||
self.assertIn(
|
||||
"connector_sanctions_snapshots",
|
||||
inspect(connection).get_table_names(),
|
||||
)
|
||||
self.assertIn(
|
||||
"connector_sanctions_acquisition_runs",
|
||||
inspect(connection).get_table_names(),
|
||||
self.assertTrue(
|
||||
{
|
||||
"connector_tabular_sources",
|
||||
"connector_sanctions_snapshots",
|
||||
"connector_sanctions_acquisition_runs",
|
||||
"connector_definitions",
|
||||
"connector_definition_revisions",
|
||||
"connector_configurations",
|
||||
"connector_simulation_runs",
|
||||
}.issubset(inspect(connection).get_table_names())
|
||||
)
|
||||
finally:
|
||||
engine.dispose()
|
||||
|
||||
@@ -0,0 +1,341 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from openpyxl import Workbook
|
||||
from sqlalchemy import Column, Integer, MetaData, String, Table, create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.files import (
|
||||
CAPABILITY_FILES_TABULAR_CONTENT,
|
||||
ManagedTabularFile,
|
||||
ManagedTabularFileContent,
|
||||
)
|
||||
from govoplan_core.core.tabular_sources import (
|
||||
TabularSourceUnavailableError,
|
||||
TabularSourceValidationError,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
)
|
||||
from govoplan_connectors.backend.tabular_adapters import (
|
||||
ManagedFileTabularAdapter,
|
||||
PostgresqlTabularAdapter,
|
||||
parse_managed_tabular_content,
|
||||
)
|
||||
|
||||
|
||||
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
|
||||
return ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id=tenant_id,
|
||||
scopes=frozenset(
|
||||
{
|
||||
"connectors:source:read",
|
||||
"connectors:source:write",
|
||||
"files:file:read",
|
||||
"files:file:download",
|
||||
}
|
||||
),
|
||||
),
|
||||
account=object(),
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
)
|
||||
|
||||
|
||||
class _ManagedFiles:
|
||||
def __init__(self, payload: bytes, *, filename: str = "cases.csv") -> None:
|
||||
self.payload = payload
|
||||
self.filename = filename
|
||||
self.current_version_id = "version-2"
|
||||
|
||||
def _file(self, version_id: str) -> ManagedTabularFile:
|
||||
return ManagedTabularFile(
|
||||
file_asset_id="asset-1",
|
||||
file_version_id=version_id,
|
||||
filename=self.filename,
|
||||
display_path=f"Imports/{self.filename}",
|
||||
content_type=(
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||
if self.filename.endswith(".xlsx")
|
||||
else "text/csv"
|
||||
),
|
||||
size_bytes=len(self.payload),
|
||||
sha256=("a" if version_id == "version-1" else "b") * 64,
|
||||
current_version=version_id == self.current_version_id,
|
||||
)
|
||||
|
||||
def list_tabular_files(self, session, principal, *, query="", limit=100):
|
||||
del session, principal, query, limit
|
||||
return (self._file(self.current_version_id),)
|
||||
|
||||
def get_tabular_file(
|
||||
self,
|
||||
session,
|
||||
principal,
|
||||
*,
|
||||
file_asset_id,
|
||||
file_version_id=None,
|
||||
):
|
||||
del session, principal
|
||||
if file_asset_id != "asset-1":
|
||||
return None
|
||||
return self._file(file_version_id or self.current_version_id)
|
||||
|
||||
def read_tabular_file(
|
||||
self,
|
||||
session,
|
||||
principal,
|
||||
*,
|
||||
file_asset_id,
|
||||
file_version_id,
|
||||
max_bytes,
|
||||
):
|
||||
del session, principal, file_asset_id
|
||||
if len(self.payload) > max_bytes:
|
||||
raise AssertionError("test payload exceeded adapter limit")
|
||||
return ManagedTabularFileContent(
|
||||
file=self._file(file_version_id),
|
||||
payload=self.payload,
|
||||
)
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, provider) -> None:
|
||||
self.provider = provider
|
||||
|
||||
def has_capability(self, name):
|
||||
return name == CAPABILITY_FILES_TABULAR_CONTENT
|
||||
|
||||
def require_capability(self, name):
|
||||
if not self.has_capability(name):
|
||||
raise KeyError(name)
|
||||
return self.provider
|
||||
|
||||
|
||||
class ManagedFileTabularAdapterTests(unittest.TestCase):
|
||||
def test_csv_is_exact_version_pinned_and_reports_newer_version(self) -> None:
|
||||
adapter = ManagedFileTabularAdapter(
|
||||
_Registry(_ManagedFiles(b"id,amount\n0012,12.5\n2,7\n"))
|
||||
)
|
||||
|
||||
result = adapter.inspect(
|
||||
object(),
|
||||
principal(),
|
||||
file_asset_id="asset-1",
|
||||
file_version_id="version-1",
|
||||
)
|
||||
|
||||
self.assertEqual("managed_file", result.provider)
|
||||
self.assertEqual("version-1", result.metadata["file_version_id"])
|
||||
self.assertEqual("warning", result.health.status)
|
||||
self.assertEqual("files.newer_version_available", result.health.code)
|
||||
self.assertEqual("mixed", result.schema[0].data_type)
|
||||
self.assertEqual(2, result.row_count)
|
||||
self.assertTrue(result.pushdown.projections)
|
||||
self.assertEqual("files.newer_version_available", result.diagnostics[0].code)
|
||||
|
||||
def test_xlsx_uses_requested_sheet_and_closed_typed_schema(self) -> None:
|
||||
workbook = Workbook()
|
||||
first = workbook.active
|
||||
first.title = "Ignore"
|
||||
first.append(["ignored"])
|
||||
target = workbook.create_sheet("Monthly")
|
||||
target.append(["case_id", "amount", "active"])
|
||||
target.append(["A-1", 12.5, True])
|
||||
target.append(["A-2", None, False])
|
||||
payload = BytesIO()
|
||||
workbook.save(payload)
|
||||
workbook.close()
|
||||
|
||||
rows, sheet = parse_managed_tabular_content(
|
||||
payload.getvalue(),
|
||||
filename="monthly.xlsx",
|
||||
content_type=(
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||
),
|
||||
delimiter=",",
|
||||
sheet_name="Monthly",
|
||||
)
|
||||
|
||||
self.assertEqual("Monthly", sheet)
|
||||
self.assertEqual("A-1", rows[0]["case_id"])
|
||||
self.assertIsNone(rows[1]["amount"])
|
||||
|
||||
def test_missing_files_capability_is_explicitly_unavailable(self) -> None:
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceUnavailableError,
|
||||
"require the Files module",
|
||||
):
|
||||
ManagedFileTabularAdapter(None).inspect(
|
||||
object(),
|
||||
principal(),
|
||||
file_asset_id="asset-1",
|
||||
file_version_id=None,
|
||||
)
|
||||
|
||||
|
||||
class PostgresqlTabularAdapterTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.directory = tempfile.TemporaryDirectory(
|
||||
prefix="govoplan-connectors-sql-adapter-"
|
||||
)
|
||||
source_path = Path(self.directory.name) / "source.db"
|
||||
self.source_url = f"sqlite+pysqlite:///{source_path}"
|
||||
source_engine = create_engine(self.source_url)
|
||||
metadata = MetaData()
|
||||
self.table = Table(
|
||||
"monthly_cases",
|
||||
metadata,
|
||||
Column("case_id", String, nullable=False),
|
||||
Column("amount", Integer, nullable=True),
|
||||
)
|
||||
metadata.create_all(source_engine)
|
||||
with source_engine.begin() as connection:
|
||||
connection.execute(
|
||||
self.table.insert(),
|
||||
(
|
||||
{"case_id": "A-1", "amount": 12},
|
||||
{"case_id": "A-2", "amount": None},
|
||||
),
|
||||
)
|
||||
source_engine.dispose()
|
||||
|
||||
self.catalog_engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(
|
||||
self.catalog_engine,
|
||||
tables=(
|
||||
ConnectorDefinition.__table__,
|
||||
ConnectorConfiguration.__table__,
|
||||
),
|
||||
)
|
||||
self.session = sessionmaker(bind=self.catalog_engine)()
|
||||
definition = ConnectorDefinition(
|
||||
id="definition-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_key="postgresql.reader",
|
||||
name="PostgreSQL reader",
|
||||
status="active",
|
||||
current_revision=1,
|
||||
local_definition=True,
|
||||
)
|
||||
self.configuration = ConnectorConfiguration(
|
||||
id="configuration-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_id=definition.id,
|
||||
name="Monthly SQL",
|
||||
status="active",
|
||||
endpoint_url=self.source_url,
|
||||
credential_ref=None,
|
||||
base_definition_revision=1,
|
||||
local_overrides={},
|
||||
protected_paths=[],
|
||||
effective_configuration={"provider": "sql", "protocol": "sql"},
|
||||
effective_hash="configuration-hash-1",
|
||||
resource_revision=1,
|
||||
ambiguity_policy="manual_review",
|
||||
)
|
||||
self.session.add_all((definition, self.configuration))
|
||||
self.session.commit()
|
||||
self.adapter = PostgresqlTabularAdapter(allow_sqlite_for_tests=True)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.catalog_engine.dispose()
|
||||
self.directory.cleanup()
|
||||
|
||||
def test_discovers_and_reads_projection_from_governed_sql_configuration(self) -> None:
|
||||
inspection = self.adapter.inspect(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
metadata = {
|
||||
**dict(inspection.metadata),
|
||||
"discovery_fingerprint": inspection.fingerprint,
|
||||
}
|
||||
read = self.adapter.read(
|
||||
self.session,
|
||||
principal(),
|
||||
metadata=metadata,
|
||||
columns=("case_id",),
|
||||
offset=1,
|
||||
limit=10,
|
||||
timeout_ms=2_000,
|
||||
)
|
||||
|
||||
self.assertEqual("live", "live")
|
||||
self.assertEqual(["case_id", "amount"], [item.name for item in inspection.schema])
|
||||
self.assertEqual(2, inspection.row_count)
|
||||
self.assertEqual(({"case_id": "A-2"},), read.rows)
|
||||
self.assertTrue(inspection.pushdown.projections)
|
||||
self.assertFalse(inspection.pushdown.filters)
|
||||
|
||||
def test_schema_drift_and_tenant_isolation_fail_closed(self) -> None:
|
||||
inspection = self.adapter.inspect(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
metadata = {
|
||||
**dict(inspection.metadata),
|
||||
"discovery_fingerprint": inspection.fingerprint,
|
||||
}
|
||||
engine = create_engine(self.source_url)
|
||||
with engine.begin() as connection:
|
||||
connection.exec_driver_sql(
|
||||
"ALTER TABLE monthly_cases ADD COLUMN category TEXT"
|
||||
)
|
||||
engine.dispose()
|
||||
|
||||
with self.assertRaisesRegex(TabularSourceValidationError, "schema drifted"):
|
||||
self.adapter.read(
|
||||
self.session,
|
||||
principal(),
|
||||
metadata=metadata,
|
||||
columns=(),
|
||||
offset=0,
|
||||
limit=10,
|
||||
timeout_ms=2_000,
|
||||
)
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceUnavailableError,
|
||||
"configuration is unavailable",
|
||||
):
|
||||
self.adapter.inspect(
|
||||
self.session,
|
||||
principal("tenant-2"),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
|
||||
def test_endpoint_query_credentials_are_rejected_before_connection(self) -> None:
|
||||
self.configuration.endpoint_url = f"{self.source_url}?password=not-allowed"
|
||||
self.session.commit()
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceValidationError,
|
||||
"query parameters must not contain credentials",
|
||||
):
|
||||
self.adapter.inspect(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,313 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from sqlalchemy import Column, Integer, MetaData, String, Table, create_engine
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
from govoplan_core.auth import ApiPrincipal
|
||||
from govoplan_core.core.access import PrincipalRef
|
||||
from govoplan_core.core.files import (
|
||||
CAPABILITY_FILES_TABULAR_CONTENT,
|
||||
ManagedTabularFile,
|
||||
ManagedTabularFileContent,
|
||||
)
|
||||
from govoplan_core.core.tabular_sources import (
|
||||
TabularReadRequest,
|
||||
TabularSourceUnavailableError,
|
||||
TabularSourceValidationError,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_core.security.credential_envelopes import CredentialEnvelope
|
||||
from govoplan_connectors.backend.db.models import (
|
||||
ConnectorConfiguration,
|
||||
ConnectorDefinition,
|
||||
ConnectorTabularSource,
|
||||
)
|
||||
from govoplan_connectors.backend.tabular_adapters import PostgresqlTabularAdapter
|
||||
from govoplan_connectors.backend.tabular_sources import SqlTabularSourceProvider
|
||||
|
||||
|
||||
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
|
||||
return ApiPrincipal(
|
||||
principal=PrincipalRef(
|
||||
account_id="account-1",
|
||||
membership_id="membership-1",
|
||||
tenant_id=tenant_id,
|
||||
scopes=frozenset(
|
||||
{
|
||||
"connectors:source:read",
|
||||
"connectors:source:write",
|
||||
"files:file:read",
|
||||
"files:file:download",
|
||||
}
|
||||
),
|
||||
),
|
||||
account=object(),
|
||||
user=SimpleNamespace(id="user-1"),
|
||||
)
|
||||
|
||||
|
||||
class _ManagedFiles:
|
||||
def __init__(self) -> None:
|
||||
self.current = "version-1"
|
||||
self.payloads = {
|
||||
"version-1": b"id,name\n1,Ada\n2,Lin\n",
|
||||
"version-2": b"id,name,active\n1,Ada,true\n2,Lin,false\n",
|
||||
}
|
||||
|
||||
def _metadata(self, version_id: str) -> ManagedTabularFile:
|
||||
payload = self.payloads[version_id]
|
||||
return ManagedTabularFile(
|
||||
file_asset_id="asset-1",
|
||||
file_version_id=version_id,
|
||||
filename="people.csv",
|
||||
display_path="Imports/people.csv",
|
||||
content_type="text/csv",
|
||||
size_bytes=len(payload),
|
||||
sha256=("a" if version_id == "version-1" else "b") * 64,
|
||||
current_version=version_id == self.current,
|
||||
)
|
||||
|
||||
def list_tabular_files(self, session, principal, *, query="", limit=100):
|
||||
del session, principal, query, limit
|
||||
return (self._metadata(self.current),)
|
||||
|
||||
def get_tabular_file(
|
||||
self,
|
||||
session,
|
||||
principal,
|
||||
*,
|
||||
file_asset_id,
|
||||
file_version_id=None,
|
||||
):
|
||||
del session, principal
|
||||
if file_asset_id != "asset-1":
|
||||
return None
|
||||
return self._metadata(file_version_id or self.current)
|
||||
|
||||
def read_tabular_file(
|
||||
self,
|
||||
session,
|
||||
principal,
|
||||
*,
|
||||
file_asset_id,
|
||||
file_version_id,
|
||||
max_bytes,
|
||||
):
|
||||
del session, principal, file_asset_id, max_bytes
|
||||
return ManagedTabularFileContent(
|
||||
file=self._metadata(file_version_id),
|
||||
payload=self.payloads[file_version_id],
|
||||
)
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, files) -> None:
|
||||
self.files = files
|
||||
|
||||
def has_capability(self, name):
|
||||
return name == CAPABILITY_FILES_TABULAR_CONTENT
|
||||
|
||||
def require_capability(self, name):
|
||||
if not self.has_capability(name):
|
||||
raise KeyError(name)
|
||||
return self.files
|
||||
|
||||
|
||||
class ConnectorTabularOriginProviderTests(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self.catalog_engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(
|
||||
self.catalog_engine,
|
||||
tables=(
|
||||
ConnectorTabularSource.__table__,
|
||||
ConnectorDefinition.__table__,
|
||||
ConnectorConfiguration.__table__,
|
||||
CredentialEnvelope.__table__,
|
||||
),
|
||||
)
|
||||
self.session = sessionmaker(bind=self.catalog_engine)()
|
||||
self.files = _ManagedFiles()
|
||||
self.directory = tempfile.TemporaryDirectory(
|
||||
prefix="govoplan-connectors-origin-provider-"
|
||||
)
|
||||
source_path = Path(self.directory.name) / "source.db"
|
||||
self.sql_url = f"sqlite+pysqlite:///{source_path}"
|
||||
source_engine = create_engine(self.sql_url)
|
||||
metadata = MetaData()
|
||||
source_table = Table(
|
||||
"monthly_cases",
|
||||
metadata,
|
||||
Column("case_id", String, nullable=False),
|
||||
Column("amount", Integer, nullable=True),
|
||||
)
|
||||
metadata.create_all(source_engine)
|
||||
with source_engine.begin() as connection:
|
||||
connection.execute(
|
||||
source_table.insert(),
|
||||
(
|
||||
{"case_id": "A-1", "amount": 12},
|
||||
{"case_id": "A-2", "amount": None},
|
||||
),
|
||||
)
|
||||
source_engine.dispose()
|
||||
definition = ConnectorDefinition(
|
||||
id="definition-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_key="postgresql.reader",
|
||||
name="PostgreSQL reader",
|
||||
status="active",
|
||||
current_revision=1,
|
||||
local_definition=True,
|
||||
)
|
||||
self.configuration = ConnectorConfiguration(
|
||||
id="configuration-1",
|
||||
tenant_id="tenant-1",
|
||||
definition_id=definition.id,
|
||||
name="Monthly SQL",
|
||||
status="active",
|
||||
endpoint_url=self.sql_url,
|
||||
credential_ref=None,
|
||||
base_definition_revision=1,
|
||||
local_overrides={},
|
||||
protected_paths=[],
|
||||
effective_configuration={"provider": "sql", "protocol": "sql"},
|
||||
effective_hash="configuration-hash-1",
|
||||
resource_revision=1,
|
||||
ambiguity_policy="manual_review",
|
||||
)
|
||||
self.session.add_all((definition, self.configuration))
|
||||
self.session.commit()
|
||||
self.provider = SqlTabularSourceProvider(
|
||||
registry=_Registry(self.files),
|
||||
sql_adapter=PostgresqlTabularAdapter(allow_sqlite_for_tests=True),
|
||||
)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self.session.close()
|
||||
self.catalog_engine.dispose()
|
||||
self.directory.cleanup()
|
||||
|
||||
def test_managed_file_source_stays_pinned_until_explicit_refresh(self) -> None:
|
||||
created = self.provider.create_file_source(
|
||||
self.session,
|
||||
principal(),
|
||||
name="Managed people",
|
||||
source_name="managed_people",
|
||||
file_asset_id="asset-1",
|
||||
)
|
||||
self.session.commit()
|
||||
self.files.current = "version-2"
|
||||
|
||||
preview = self.provider.read_source(
|
||||
self.session,
|
||||
principal(),
|
||||
request=TabularReadRequest(source_ref=created.ref, limit=10),
|
||||
)
|
||||
refreshed = self.provider.refresh_source(
|
||||
self.session,
|
||||
principal(),
|
||||
source_ref=created.ref,
|
||||
)
|
||||
|
||||
self.assertTrue(created.ref.startswith("file:"))
|
||||
self.assertEqual("file_backed", preview.source.source_mode)
|
||||
self.assertEqual("version-1", preview.source.metadata["file_version_id"])
|
||||
self.assertEqual(
|
||||
"files.newer_version_available",
|
||||
preview.diagnostics[0].code,
|
||||
)
|
||||
self.assertEqual("version-2", refreshed.metadata["file_version_id"])
|
||||
self.assertEqual("2", refreshed.schema_version)
|
||||
self.assertEqual(3, len(refreshed.schema))
|
||||
|
||||
def test_sql_source_projects_and_blocks_changed_configuration_until_refresh(self) -> None:
|
||||
created = self.provider.create_sql_source(
|
||||
self.session,
|
||||
principal(),
|
||||
name="Monthly cases",
|
||||
source_name="monthly_cases",
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
self.session.commit()
|
||||
preview = self.provider.read_source(
|
||||
self.session,
|
||||
principal(),
|
||||
request=TabularReadRequest(
|
||||
source_ref=created.ref,
|
||||
columns=("case_id",),
|
||||
limit=1,
|
||||
),
|
||||
)
|
||||
|
||||
self.assertTrue(created.ref.startswith("sql:"))
|
||||
self.assertEqual("live", preview.source.source_mode)
|
||||
self.assertEqual(({"case_id": "A-1"},), preview.rows)
|
||||
self.assertEqual("preview.row_limit_reached", preview.diagnostics[-1].code)
|
||||
self.assertIsNone(
|
||||
self.provider.get_source(
|
||||
self.session,
|
||||
principal("tenant-2"),
|
||||
source_ref=created.ref,
|
||||
)
|
||||
)
|
||||
|
||||
self.configuration.effective_hash = "configuration-hash-2"
|
||||
self.configuration.resource_revision = 2
|
||||
self.session.commit()
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceValidationError,
|
||||
"configuration changed",
|
||||
):
|
||||
self.provider.read_source(
|
||||
self.session,
|
||||
principal(),
|
||||
request=TabularReadRequest(source_ref=created.ref),
|
||||
)
|
||||
refreshed = self.provider.refresh_source(
|
||||
self.session,
|
||||
principal(),
|
||||
source_ref=created.ref,
|
||||
)
|
||||
self.assertEqual("2", refreshed.schema_version)
|
||||
self.assertEqual("configuration-hash-2", refreshed.metadata["configuration_hash"])
|
||||
|
||||
def test_inactive_or_stale_sql_credentials_have_sanitized_diagnostics(self) -> None:
|
||||
self.configuration.endpoint_url = (
|
||||
"postgresql+psycopg://db.example.invalid/govoplan"
|
||||
)
|
||||
self.configuration.credential_ref = "missing-credential"
|
||||
self.session.commit()
|
||||
adapter = PostgresqlTabularAdapter()
|
||||
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceUnavailableError,
|
||||
"credential is unavailable, inactive, or outside its allowed scope",
|
||||
):
|
||||
adapter.inspect(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
self.configuration.status = "disabled"
|
||||
self.session.commit()
|
||||
with self.assertRaisesRegex(
|
||||
TabularSourceUnavailableError,
|
||||
"configuration is not active",
|
||||
):
|
||||
adapter.inspect(
|
||||
self.session,
|
||||
principal(),
|
||||
configuration_id=self.configuration.id,
|
||||
table_name="monthly_cases",
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"name": "@govoplan/connectors-webui",
|
||||
"version": "0.1.20",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "src/index.ts",
|
||||
"module": "src/index.ts",
|
||||
"types": "src/index.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./src/index.ts",
|
||||
"import": "./src/index.ts"
|
||||
},
|
||||
"./styles/connectors.css": "./src/styles/connectors.css"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@govoplan/core-webui": "^0.1.18",
|
||||
"react": ">=19.2.7 <20",
|
||||
"react-dom": ">=19.2.7 <20"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"@govoplan/core-webui": {
|
||||
"optional": true
|
||||
}
|
||||
},
|
||||
"scripts": {
|
||||
"test:connector-governance-ui": "node tests/connector-governance-ui-structure.test.mjs"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,204 @@
|
||||
import { apiFetch, apiPath, type ApiSettings } from "@govoplan/core-webui";
|
||||
|
||||
export type GovernedConnectorSpecification = {
|
||||
provider: string;
|
||||
protocol: string;
|
||||
capabilities: string[];
|
||||
input_schema: Record<string, unknown>;
|
||||
output_schema: Record<string, unknown>;
|
||||
mapping: {
|
||||
version: string;
|
||||
rules: Array<{
|
||||
source: string;
|
||||
target: string;
|
||||
required?: boolean;
|
||||
default?: unknown;
|
||||
}>;
|
||||
};
|
||||
validation_rules: Array<{
|
||||
kind: "required" | "one_of" | "unique";
|
||||
field: string;
|
||||
values?: unknown[];
|
||||
severity: "warning" | "error";
|
||||
code: string;
|
||||
message: string;
|
||||
}>;
|
||||
dry_run: {
|
||||
supported: boolean;
|
||||
simulation_supported: boolean;
|
||||
sample_rows: Array<Record<string, unknown>>;
|
||||
max_items: number;
|
||||
redacted_fields: string[];
|
||||
};
|
||||
audit: {
|
||||
event_prefix: string;
|
||||
expected_events: string[];
|
||||
evidence_fields: string[];
|
||||
};
|
||||
privacy_classification: "public" | "internal" | "confidential" | "restricted";
|
||||
retention_class: string;
|
||||
operational_limits: Record<string, unknown>;
|
||||
retry_policy: Record<string, unknown>;
|
||||
};
|
||||
|
||||
export type ConnectorDefinition = {
|
||||
id: string;
|
||||
definition_key: string;
|
||||
name: string;
|
||||
description?: string | null;
|
||||
status: string;
|
||||
current_revision: number;
|
||||
source_package?: string | null;
|
||||
local_definition: boolean;
|
||||
definition_hash: string;
|
||||
specification: GovernedConnectorSpecification;
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type ConnectorConfiguration = {
|
||||
id: string;
|
||||
definition_id: string;
|
||||
definition_key: string;
|
||||
definition_name: string;
|
||||
name: string;
|
||||
status: "draft" | "active" | "disabled";
|
||||
endpoint_url?: string | null;
|
||||
credential_ref?: string | null;
|
||||
base_definition_revision: number;
|
||||
latest_definition_revision: number;
|
||||
update_available: boolean;
|
||||
local_overrides: Record<string, unknown>;
|
||||
protected_paths: string[];
|
||||
effective_configuration: GovernedConnectorSpecification;
|
||||
effective_hash: string;
|
||||
resource_revision: number;
|
||||
ambiguity_policy: "manual_review" | "quarantine" | "reject";
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type ConnectorRun = {
|
||||
id: string;
|
||||
configuration_id: string;
|
||||
mode: "dry_run" | "simulation";
|
||||
idempotency_key: string;
|
||||
status: string;
|
||||
review_state: string;
|
||||
definition_revision: number;
|
||||
configuration_revision: number;
|
||||
configuration_hash: string;
|
||||
input_hash: string;
|
||||
summary: Record<string, number | boolean>;
|
||||
effects: Array<Record<string, unknown>>;
|
||||
diagnostics: Array<Record<string, unknown>>;
|
||||
provenance: Record<string, unknown>;
|
||||
reviewed_by?: string | null;
|
||||
reviewed_at?: string | null;
|
||||
review_reason?: string | null;
|
||||
created_at: string;
|
||||
};
|
||||
|
||||
export type ConnectorConfigurationDraft = {
|
||||
name: string;
|
||||
status: ConnectorConfiguration["status"];
|
||||
endpoint_url: string;
|
||||
credential_ref: string;
|
||||
local_overrides: string;
|
||||
ambiguity_policy: ConnectorConfiguration["ambiguity_policy"];
|
||||
};
|
||||
|
||||
const ROOT = "/api/v1/connectors/governed";
|
||||
|
||||
export async function listConnectorDefinitions(
|
||||
settings: ApiSettings
|
||||
): Promise<ConnectorDefinition[]> {
|
||||
const response = await apiFetch<{ items: ConnectorDefinition[] }>(
|
||||
settings,
|
||||
`${ROOT}/definitions`
|
||||
);
|
||||
return response.items;
|
||||
}
|
||||
|
||||
export async function upsertConnectorDefinition(
|
||||
settings: ApiSettings,
|
||||
payload: Record<string, unknown>
|
||||
): Promise<ConnectorDefinition> {
|
||||
return apiFetch(settings, `${ROOT}/definitions`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export async function listConnectorConfigurations(
|
||||
settings: ApiSettings
|
||||
): Promise<ConnectorConfiguration[]> {
|
||||
const response = await apiFetch<{ items: ConnectorConfiguration[] }>(
|
||||
settings,
|
||||
`${ROOT}/configurations`
|
||||
);
|
||||
return response.items;
|
||||
}
|
||||
|
||||
export function createConnectorConfiguration(
|
||||
settings: ApiSettings,
|
||||
payload: Record<string, unknown>
|
||||
): Promise<ConnectorConfiguration> {
|
||||
return apiFetch(settings, `${ROOT}/configurations`, {
|
||||
method: "POST",
|
||||
body: JSON.stringify(payload)
|
||||
});
|
||||
}
|
||||
|
||||
export function updateConnectorConfiguration(
|
||||
settings: ApiSettings,
|
||||
configurationId: string,
|
||||
payload: Record<string, unknown>
|
||||
): Promise<ConnectorConfiguration> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`${ROOT}/configurations/${encodeURIComponent(configurationId)}`,
|
||||
{ method: "PUT", body: JSON.stringify(payload) }
|
||||
);
|
||||
}
|
||||
|
||||
export async function listConnectorRuns(
|
||||
settings: ApiSettings,
|
||||
configurationId?: string
|
||||
): Promise<ConnectorRun[]> {
|
||||
const response = await apiFetch<{ items: ConnectorRun[] }>(
|
||||
settings,
|
||||
apiPath(`${ROOT}/runs`, {
|
||||
configuration_id: configurationId || undefined,
|
||||
limit: 100
|
||||
})
|
||||
);
|
||||
return response.items;
|
||||
}
|
||||
|
||||
export function executeConnectorRun(
|
||||
settings: ApiSettings,
|
||||
configurationId: string,
|
||||
mode: "dry-runs" | "simulations",
|
||||
payload: Record<string, unknown>
|
||||
): Promise<ConnectorRun> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`${ROOT}/configurations/${encodeURIComponent(configurationId)}/${mode}`,
|
||||
{ method: "POST", body: JSON.stringify(payload) }
|
||||
);
|
||||
}
|
||||
|
||||
export function reviewConnectorRun(
|
||||
settings: ApiSettings,
|
||||
runId: string,
|
||||
decision: "approved" | "rejected",
|
||||
reason: string
|
||||
): Promise<ConnectorRun> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`${ROOT}/runs/${encodeURIComponent(runId)}/review`,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify({ decision, reason })
|
||||
}
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,661 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from "react";
|
||||
import {
|
||||
AdminPageLayout,
|
||||
Button,
|
||||
Card,
|
||||
DataGrid,
|
||||
Dialog,
|
||||
FilterBar,
|
||||
FormField,
|
||||
FormGrid,
|
||||
MetricCard,
|
||||
MetricGrid,
|
||||
PageActionBar,
|
||||
SelectionList,
|
||||
SelectionListItem,
|
||||
SelectionListItemContent,
|
||||
StatePanel,
|
||||
StatusBadge,
|
||||
TableActionGroup,
|
||||
WorkspaceLayout,
|
||||
formatDateTime,
|
||||
hasScope,
|
||||
useUnsavedChanges,
|
||||
useUnsavedDraftGuard,
|
||||
type ApiSettings,
|
||||
type AuthInfo,
|
||||
type DataGridColumn
|
||||
} from "@govoplan/core-webui";
|
||||
import {
|
||||
createConnectorConfiguration,
|
||||
executeConnectorRun,
|
||||
listConnectorConfigurations,
|
||||
listConnectorDefinitions,
|
||||
listConnectorRuns,
|
||||
reviewConnectorRun,
|
||||
updateConnectorConfiguration,
|
||||
upsertConnectorDefinition,
|
||||
type ConnectorConfiguration,
|
||||
type ConnectorConfigurationDraft,
|
||||
type ConnectorDefinition,
|
||||
type ConnectorRun
|
||||
} from "../api/governedConnectors";
|
||||
|
||||
type Props = {
|
||||
settings: ApiSettings;
|
||||
auth: AuthInfo;
|
||||
};
|
||||
|
||||
const EMPTY_DRAFT: ConnectorConfigurationDraft = {
|
||||
name: "",
|
||||
status: "draft",
|
||||
endpoint_url: "",
|
||||
credential_ref: "",
|
||||
local_overrides: "{}",
|
||||
ambiguity_policy: "manual_review"
|
||||
};
|
||||
|
||||
const EXAMPLE_DEFINITION = JSON.stringify({
|
||||
definition_key: "example.reference-data",
|
||||
name: "Example reference data",
|
||||
description: "Locally governed example connector",
|
||||
origin: "local",
|
||||
specification: {
|
||||
provider: "example-provider",
|
||||
protocol: "rest",
|
||||
capabilities: ["discover", "read", "dry_run"],
|
||||
input_schema: { type: "object" },
|
||||
output_schema: { type: "object" },
|
||||
mapping: {
|
||||
version: "1",
|
||||
rules: [{ source: "id", target: "record.id", required: true }]
|
||||
},
|
||||
validation_rules: [{
|
||||
kind: "unique",
|
||||
field: "record.id",
|
||||
severity: "error",
|
||||
code: "record.id.ambiguous",
|
||||
message: "The record identifier is not unique."
|
||||
}],
|
||||
dry_run: {
|
||||
supported: true,
|
||||
simulation_supported: true,
|
||||
sample_rows: [{ id: "sample-1" }],
|
||||
max_items: 500,
|
||||
redacted_fields: []
|
||||
},
|
||||
audit: {
|
||||
event_prefix: "connectors.example",
|
||||
expected_events: ["simulation.completed"],
|
||||
evidence_fields: ["input_hash", "configuration_hash"]
|
||||
},
|
||||
privacy_classification: "internal",
|
||||
retention_class: "connector-preview-30d",
|
||||
operational_limits: { timeout_seconds: 30 },
|
||||
retry_policy: { max_attempts: 2 }
|
||||
}
|
||||
}, null, 2);
|
||||
|
||||
export default function ConnectorGovernancePage({ settings, auth }: Props) {
|
||||
const [definitions, setDefinitions] = useState<ConnectorDefinition[]>([]);
|
||||
const [configurations, setConfigurations] = useState<ConnectorConfiguration[]>([]);
|
||||
const [runs, setRuns] = useState<ConnectorRun[]>([]);
|
||||
const [selectedId, setSelectedId] = useState("");
|
||||
const [draft, setDraft] = useState<ConnectorConfigurationDraft>(EMPTY_DRAFT);
|
||||
const [savedKey, setSavedKey] = useState("");
|
||||
const [search, setSearch] = useState("");
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const [success, setSuccess] = useState("");
|
||||
const [definitionOpen, setDefinitionOpen] = useState(false);
|
||||
const [definitionJson, setDefinitionJson] = useState(EXAMPLE_DEFINITION);
|
||||
const [configurationOpen, setConfigurationOpen] = useState(false);
|
||||
const [newDefinitionId, setNewDefinitionId] = useState("");
|
||||
const [newDraft, setNewDraft] = useState<ConnectorConfigurationDraft>(EMPTY_DRAFT);
|
||||
const [sampleJson, setSampleJson] = useState("[]");
|
||||
const [externalRevision, setExternalRevision] = useState("");
|
||||
const [reviewRun, setReviewRun] = useState<ConnectorRun | null>(null);
|
||||
const [reviewReason, setReviewReason] = useState("");
|
||||
const { requestDiscard } = useUnsavedChanges();
|
||||
|
||||
const selected = configurations.find((item) => item.id === selectedId) ?? null;
|
||||
const canAdmin = hasScope(auth, "connectors:source:admin");
|
||||
const canExecute = canAdmin || hasScope(auth, "connectors:source:write");
|
||||
const dirty = Boolean(selected && draftKey(draft) !== savedKey);
|
||||
|
||||
const applyConfiguration = useCallback((item: ConnectorConfiguration | null) => {
|
||||
const next = item ? draftFromConfiguration(item) : EMPTY_DRAFT;
|
||||
setDraft(next);
|
||||
setSavedKey(item ? draftKey(next) : "");
|
||||
setSampleJson(JSON.stringify(
|
||||
item?.effective_configuration.dry_run.sample_rows ?? [],
|
||||
null,
|
||||
2
|
||||
));
|
||||
}, []);
|
||||
|
||||
const reload = useCallback(async (preferredId?: string) => {
|
||||
setLoading(true);
|
||||
setError("");
|
||||
try {
|
||||
const [nextDefinitions, nextConfigurations] = await Promise.all([
|
||||
listConnectorDefinitions(settings),
|
||||
listConnectorConfigurations(settings)
|
||||
]);
|
||||
const nextId = preferredId && nextConfigurations.some((item) => item.id === preferredId)
|
||||
? preferredId
|
||||
: nextConfigurations.some((item) => item.id === selectedId)
|
||||
? selectedId
|
||||
: nextConfigurations[0]?.id ?? "";
|
||||
const nextRuns = await listConnectorRuns(settings, nextId || undefined);
|
||||
setDefinitions(nextDefinitions);
|
||||
setConfigurations(nextConfigurations);
|
||||
setRuns(nextRuns);
|
||||
setSelectedId(nextId);
|
||||
applyConfiguration(nextConfigurations.find((item) => item.id === nextId) ?? null);
|
||||
if (!newDefinitionId && nextDefinitions[0]) setNewDefinitionId(nextDefinitions[0].id);
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
}, [applyConfiguration, newDefinitionId, selectedId, settings]);
|
||||
|
||||
useEffect(() => {
|
||||
void reload();
|
||||
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
|
||||
|
||||
const visibleConfigurations = useMemo(() => {
|
||||
const needle = search.trim().toLocaleLowerCase();
|
||||
return configurations.filter((item) => !needle ||
|
||||
`${item.name} ${item.definition_name} ${item.status}`
|
||||
.toLocaleLowerCase()
|
||||
.includes(needle));
|
||||
}, [configurations, search]);
|
||||
|
||||
const save = async (): Promise<boolean> => {
|
||||
if (!selected || !canAdmin) return false;
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const overrides = parseObject(draft.local_overrides, "Local overrides");
|
||||
const updated = await updateConnectorConfiguration(settings, selected.id, {
|
||||
expected_revision: selected.resource_revision,
|
||||
name: draft.name.trim(),
|
||||
status: draft.status,
|
||||
endpoint_url: draft.endpoint_url.trim() || null,
|
||||
credential_ref: draft.credential_ref.trim() || null,
|
||||
local_overrides: overrides,
|
||||
ambiguity_policy: draft.ambiguity_policy
|
||||
});
|
||||
setSuccess("Connector configuration saved.");
|
||||
await reload(updated.id);
|
||||
return true;
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
return false;
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
useUnsavedDraftGuard({
|
||||
dirty,
|
||||
onSave: save,
|
||||
onDiscard: () => applyConfiguration(selected),
|
||||
title: "Unsaved connector changes",
|
||||
message: "Save or discard the current connector changes before continuing."
|
||||
});
|
||||
|
||||
const selectConfiguration = (item: ConnectorConfiguration) => {
|
||||
if (item.id === selectedId) return;
|
||||
requestDiscard(() => {
|
||||
setSelectedId(item.id);
|
||||
applyConfiguration(item);
|
||||
setRuns([]);
|
||||
setError("");
|
||||
setSuccess("");
|
||||
void listConnectorRuns(settings, item.id).then(setRuns).catch((caught) => {
|
||||
setError(errorMessage(caught));
|
||||
});
|
||||
});
|
||||
};
|
||||
|
||||
const createDefinition = async () => {
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const payload = parseObject(definitionJson, "Definition");
|
||||
const created = await upsertConnectorDefinition(settings, payload);
|
||||
setDefinitionOpen(false);
|
||||
setSuccess("Connector definition revision saved.");
|
||||
setNewDefinitionId(created.id);
|
||||
await reload(selectedId);
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const createConfiguration = async () => {
|
||||
if (!newDefinitionId || !newDraft.name.trim()) return;
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const created = await createConnectorConfiguration(settings, {
|
||||
definition_id: newDefinitionId,
|
||||
name: newDraft.name.trim(),
|
||||
status: newDraft.status,
|
||||
endpoint_url: newDraft.endpoint_url.trim() || null,
|
||||
credential_ref: newDraft.credential_ref.trim() || null,
|
||||
local_overrides: parseObject(newDraft.local_overrides, "Local overrides"),
|
||||
ambiguity_policy: newDraft.ambiguity_policy
|
||||
});
|
||||
setConfigurationOpen(false);
|
||||
setNewDraft(EMPTY_DRAFT);
|
||||
setSuccess("Connector configuration created.");
|
||||
await reload(created.id);
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const adoptUpdate = async () => {
|
||||
if (!selected || dirty || !selected.update_available) return;
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const updated = await updateConnectorConfiguration(settings, selected.id, {
|
||||
expected_revision: selected.resource_revision,
|
||||
adopt_latest_definition: true
|
||||
});
|
||||
setSuccess("Package revision adopted; protected local overrides were reapplied.");
|
||||
await reload(updated.id);
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const run = async (mode: "dry-runs" | "simulations") => {
|
||||
if (!selected || dirty) return;
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
const inputRows = parseRows(sampleJson);
|
||||
const created = await executeConnectorRun(settings, selected.id, mode, {
|
||||
idempotency_key: `${mode}-${crypto.randomUUID()}`,
|
||||
input_rows: inputRows,
|
||||
external_revision: externalRevision.trim() || null
|
||||
});
|
||||
setSuccess(`${mode === "dry-runs" ? "Dry-run" : "Simulation"} completed with status ${created.status}.`);
|
||||
setRuns(await listConnectorRuns(settings, selected.id));
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const decideReview = async (decision: "approved" | "rejected") => {
|
||||
if (!reviewRun || reviewReason.trim().length < 5) return;
|
||||
setBusy(true);
|
||||
setError("");
|
||||
try {
|
||||
await reviewConnectorRun(settings, reviewRun.id, decision, reviewReason.trim());
|
||||
setReviewRun(null);
|
||||
setReviewReason("");
|
||||
setSuccess(`Simulation ${decision}.`);
|
||||
setRuns(await listConnectorRuns(settings, selectedId));
|
||||
} catch (caught) {
|
||||
setError(errorMessage(caught));
|
||||
} finally {
|
||||
setBusy(false);
|
||||
}
|
||||
};
|
||||
|
||||
const runColumns = useMemo<DataGridColumn<ConnectorRun>[]>(() => [
|
||||
{
|
||||
id: "created",
|
||||
header: "Run",
|
||||
width: 190,
|
||||
sortable: true,
|
||||
value: (row) => row.created_at,
|
||||
render: (row) => <>{row.mode}<br /><span className="muted">{formatDateTime(row.created_at)}</span></>
|
||||
},
|
||||
{
|
||||
id: "status",
|
||||
header: "Status",
|
||||
width: 150,
|
||||
sortable: true,
|
||||
value: (row) => row.status,
|
||||
render: (row) => <StatusBadge status={row.status} label={row.status.replaceAll("_", " ")} />
|
||||
},
|
||||
{
|
||||
id: "summary",
|
||||
header: "Effects",
|
||||
width: "1fr",
|
||||
minWidth: 220,
|
||||
render: (row) => `${row.summary.total ?? 0} total · ${row.summary.ambiguous ?? 0} ambiguous · ${row.summary.errors ?? 0} errors`
|
||||
},
|
||||
{
|
||||
id: "revision",
|
||||
header: "Evidence",
|
||||
width: 170,
|
||||
render: (row) => <code title={row.configuration_hash}>r{row.configuration_revision} · {row.input_hash.slice(0, 8)}</code>
|
||||
},
|
||||
{
|
||||
id: "actions",
|
||||
header: "Actions",
|
||||
width: 90,
|
||||
sticky: "end",
|
||||
align: "right",
|
||||
render: (row) => <TableActionGroup actions={[{
|
||||
id: "review",
|
||||
label: "Review result",
|
||||
icon: <span>✓</span>,
|
||||
applicable: ["pending", "quarantined"].includes(row.review_state),
|
||||
disabled: !canAdmin || busy,
|
||||
disabledReason: !canAdmin ? "Connector administration permission is required." : undefined,
|
||||
onClick: () => setReviewRun(row)
|
||||
}]} />
|
||||
}
|
||||
], [busy, canAdmin]);
|
||||
|
||||
const actionBar = <PageActionBar
|
||||
variant="editor"
|
||||
state={busy ? "saving" : dirty ? "dirty" : "clean"}
|
||||
refreshable
|
||||
reloadAction={{ onReload: () => void reload(selectedId), loading }}
|
||||
primaryActions={<>
|
||||
<Button onClick={() => setDefinitionOpen(true)} disabled={!canAdmin || busy}>
|
||||
New definition
|
||||
</Button>
|
||||
<Button onClick={() => setConfigurationOpen(true)} disabled={!canAdmin || busy || !definitions.length}>
|
||||
New configuration
|
||||
</Button>
|
||||
{selected?.update_available ? <Button
|
||||
variant="secondary"
|
||||
onClick={() => void adoptUpdate()}
|
||||
disabled={!canAdmin || busy || dirty}
|
||||
disabledReason={dirty ? "Save or discard local edits before adopting a package update." : undefined}
|
||||
>
|
||||
Adopt package revision {selected.latest_definition_revision}
|
||||
</Button> : null}
|
||||
</>}
|
||||
discardAction={{
|
||||
label: "Discard changes",
|
||||
disabled: !selected,
|
||||
onClick: () => applyConfiguration(selected)
|
||||
}}
|
||||
saveAction={{
|
||||
label: "Save",
|
||||
disabled: !selected || !canAdmin || busy,
|
||||
disabledReason: !canAdmin ? "Connector administration permission is required." : undefined,
|
||||
onClick: () => void save()
|
||||
}}
|
||||
/>;
|
||||
|
||||
return <AdminPageLayout
|
||||
archetype="workspace"
|
||||
title="Connector governance"
|
||||
description="Version schemas and mappings, protect local overrides, and review deterministic simulations before provider-specific writes."
|
||||
loading={loading && !configurations.length}
|
||||
error={error}
|
||||
success={success}
|
||||
actions={actionBar}
|
||||
className="connector-governance-page"
|
||||
helpContextId="connectors.admin.governed-configurations"
|
||||
>
|
||||
<MetricGrid columns={4} density="compact" minimum="compact">
|
||||
<MetricCard label="Definitions" value={definitions.length} />
|
||||
<MetricCard label="Configurations" value={configurations.length} />
|
||||
<MetricCard label="Updates available" value={configurations.filter((item) => item.update_available).length} tone="warning" />
|
||||
<MetricCard label="Awaiting review" value={runs.filter((item) => ["pending", "quarantined"].includes(item.review_state)).length} tone="warning" />
|
||||
</MetricGrid>
|
||||
|
||||
<WorkspaceLayout
|
||||
variant="split"
|
||||
primarySize="compact"
|
||||
surface="contained"
|
||||
primaryScrollable={false}
|
||||
contentScrollable={false}
|
||||
primaryLabel="Connector configurations"
|
||||
contentLabel="Configuration details"
|
||||
primary={<div className="connector-governance-list">
|
||||
<FilterBar surface="panel">
|
||||
<input
|
||||
type="search"
|
||||
value={search}
|
||||
onChange={(event) => setSearch(event.target.value)}
|
||||
placeholder="Search configurations"
|
||||
aria-label="Search connector configurations"
|
||||
/>
|
||||
</FilterBar>
|
||||
<SelectionList variant="navigation" label="Connector configurations">
|
||||
{visibleConfigurations.map((item) => <SelectionListItem
|
||||
key={item.id}
|
||||
selected={item.id === selectedId}
|
||||
onClick={() => selectConfiguration(item)}
|
||||
>
|
||||
<SelectionListItemContent
|
||||
title={item.name}
|
||||
description={`${item.definition_name} · revision ${item.base_definition_revision}`}
|
||||
/>
|
||||
<StatusBadge status={item.update_available ? "warning" : item.status} />
|
||||
</SelectionListItem>)}
|
||||
{!visibleConfigurations.length
|
||||
? <StatePanel size="compact" description="No matching configurations." />
|
||||
: null}
|
||||
</SelectionList>
|
||||
</div>}
|
||||
>
|
||||
{!selected ? <StatePanel
|
||||
size="fill"
|
||||
title="Connector configurations"
|
||||
description="Create or select a configuration to inspect its pinned definition and simulation evidence."
|
||||
/> : <div className="connector-governance-detail">
|
||||
<Card title={selected.name}>
|
||||
<div className="connector-revision-line">
|
||||
<StatusBadge status={selected.status} />
|
||||
<span>Definition revision {selected.base_definition_revision}</span>
|
||||
{selected.update_available
|
||||
? <StatusBadge status="warning" label={`Revision ${selected.latest_definition_revision} available`} />
|
||||
: null}
|
||||
<code title={selected.effective_hash}>{selected.effective_hash.slice(0, 12)}</code>
|
||||
</div>
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="Name">
|
||||
<input value={draft.name} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, name: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Status">
|
||||
<select value={draft.status} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, status: event.target.value as ConnectorConfigurationDraft["status"] })}>
|
||||
<option value="draft">Draft</option>
|
||||
<option value="active">Active</option>
|
||||
<option value="disabled">Disabled</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Endpoint URL" hint="Credentials are rejected in URLs.">
|
||||
<input value={draft.endpoint_url} disabled={!canAdmin || busy} placeholder="https://provider.example/api" onChange={(event) => setDraft({ ...draft, endpoint_url: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Credential reference" hint="Reference an approved secret; do not paste a secret.">
|
||||
<input value={draft.credential_ref} disabled={!canAdmin || busy} placeholder="vault://connectors/provider" onChange={(event) => setDraft({ ...draft, credential_ref: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Ambiguous-result policy">
|
||||
<select value={draft.ambiguity_policy} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, ambiguity_policy: event.target.value as ConnectorConfigurationDraft["ambiguity_policy"] })}>
|
||||
<option value="manual_review">Manual review</option>
|
||||
<option value="quarantine">Quarantine</option>
|
||||
<option value="reject">Reject</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Protected local overrides" hint="JSON object leaf paths remain protected when package revisions are adopted.">
|
||||
<textarea rows={8} value={draft.local_overrides} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, local_overrides: event.target.value })} />
|
||||
</FormField>
|
||||
</FormGrid>
|
||||
<p className="muted">
|
||||
Protected paths: {selected.protected_paths.length ? selected.protected_paths.join(", ") : "none"}
|
||||
</p>
|
||||
</Card>
|
||||
|
||||
<Card title="Dry-run and simulation">
|
||||
<p className="muted">Runs never apply changes. They retain redacted, revision-pinned evidence for review.</p>
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="Sample input rows" hint="JSON array, bounded by the definition's maximum.">
|
||||
<textarea rows={9} value={sampleJson} disabled={!canExecute || busy} onChange={(event) => setSampleJson(event.target.value)} />
|
||||
</FormField>
|
||||
<FormField label="External revision" hint="Optional provider ETag, cursor, or snapshot revision.">
|
||||
<input value={externalRevision} disabled={!canExecute || busy} onChange={(event) => setExternalRevision(event.target.value)} />
|
||||
</FormField>
|
||||
</FormGrid>
|
||||
<div className="connector-run-actions">
|
||||
<Button onClick={() => void run("dry-runs")} disabled={!canExecute || busy || dirty}>Run dry-run</Button>
|
||||
<Button variant="primary" onClick={() => void run("simulations")} disabled={!canExecute || busy || dirty}>Run simulation</Button>
|
||||
{dirty ? <span className="muted">Save or discard configuration changes before running.</span> : null}
|
||||
</div>
|
||||
</Card>
|
||||
|
||||
<Card title="Simulation evidence">
|
||||
<DataGrid
|
||||
id="connector-simulation-runs"
|
||||
rows={runs}
|
||||
columns={runColumns}
|
||||
getRowKey={(row) => row.id}
|
||||
initialFit="container"
|
||||
emptyText="No dry-runs or simulations have been recorded."
|
||||
/>
|
||||
</Card>
|
||||
|
||||
<Card title="Effective governed definition">
|
||||
<pre className="connector-json-preview">{JSON.stringify(selected.effective_configuration, null, 2)}</pre>
|
||||
</Card>
|
||||
</div>}
|
||||
</WorkspaceLayout>
|
||||
|
||||
<Dialog
|
||||
open={definitionOpen}
|
||||
title="Create or revise connector definition"
|
||||
onClose={() => !busy && setDefinitionOpen(false)}
|
||||
closeDisabled={busy}
|
||||
footer={<>
|
||||
<Button onClick={() => setDefinitionOpen(false)} disabled={busy}>Cancel</Button>
|
||||
<Button variant="primary" onClick={() => void createDefinition()} disabled={!canAdmin || busy}>Save definition revision</Button>
|
||||
</>}
|
||||
>
|
||||
<p className="muted">The definition is schema-validated and every changed specification creates an immutable revision. Package definitions must name their package reference.</p>
|
||||
<FormField label="Governed definition JSON">
|
||||
<textarea className="connector-definition-editor" rows={24} value={definitionJson} disabled={busy} onChange={(event) => setDefinitionJson(event.target.value)} />
|
||||
</FormField>
|
||||
</Dialog>
|
||||
|
||||
<Dialog
|
||||
open={configurationOpen}
|
||||
title="Create connector configuration"
|
||||
onClose={() => !busy && setConfigurationOpen(false)}
|
||||
closeDisabled={busy}
|
||||
footer={<>
|
||||
<Button onClick={() => setConfigurationOpen(false)} disabled={busy}>Cancel</Button>
|
||||
<Button variant="primary" onClick={() => void createConfiguration()} disabled={busy || !newDefinitionId || !newDraft.name.trim()}>Create configuration</Button>
|
||||
</>}
|
||||
>
|
||||
<FormGrid columns={2} collapseAt="standard" className="">
|
||||
<FormField label="Definition">
|
||||
<select value={newDefinitionId} disabled={busy} onChange={(event) => setNewDefinitionId(event.target.value)}>
|
||||
{definitions.map((item) => <option key={item.id} value={item.id}>{item.name} · revision {item.current_revision}</option>)}
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Name">
|
||||
<input value={newDraft.name} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, name: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Endpoint URL">
|
||||
<input value={newDraft.endpoint_url} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, endpoint_url: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Credential reference">
|
||||
<input value={newDraft.credential_ref} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, credential_ref: event.target.value })} />
|
||||
</FormField>
|
||||
<FormField label="Ambiguous-result policy">
|
||||
<select value={newDraft.ambiguity_policy} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, ambiguity_policy: event.target.value as ConnectorConfigurationDraft["ambiguity_policy"] })}>
|
||||
<option value="manual_review">Manual review</option>
|
||||
<option value="quarantine">Quarantine</option>
|
||||
<option value="reject">Reject</option>
|
||||
</select>
|
||||
</FormField>
|
||||
<FormField label="Local overrides">
|
||||
<textarea rows={7} value={newDraft.local_overrides} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, local_overrides: event.target.value })} />
|
||||
</FormField>
|
||||
</FormGrid>
|
||||
</Dialog>
|
||||
|
||||
<Dialog
|
||||
open={Boolean(reviewRun)}
|
||||
title="Review ambiguous connector result"
|
||||
onClose={() => !busy && setReviewRun(null)}
|
||||
closeDisabled={busy}
|
||||
footer={<>
|
||||
<Button onClick={() => setReviewRun(null)} disabled={busy}>Cancel</Button>
|
||||
<Button variant="danger" onClick={() => void decideReview("rejected")} disabled={busy || reviewReason.trim().length < 5}>Reject</Button>
|
||||
<Button variant="primary" onClick={() => void decideReview("approved")} disabled={busy || reviewReason.trim().length < 5}>Approve</Button>
|
||||
</>}
|
||||
>
|
||||
<p>Review {reviewRun?.summary.ambiguous ?? 0} ambiguous effects against the retained input and configuration hashes before deciding.</p>
|
||||
<FormField label="Decision reason">
|
||||
<textarea rows={4} value={reviewReason} disabled={busy} onChange={(event) => setReviewReason(event.target.value)} />
|
||||
</FormField>
|
||||
<pre className="connector-json-preview">{JSON.stringify(reviewRun?.diagnostics ?? [], null, 2)}</pre>
|
||||
</Dialog>
|
||||
</AdminPageLayout>;
|
||||
}
|
||||
|
||||
function draftFromConfiguration(item: ConnectorConfiguration): ConnectorConfigurationDraft {
|
||||
return {
|
||||
name: item.name,
|
||||
status: item.status,
|
||||
endpoint_url: item.endpoint_url ?? "",
|
||||
credential_ref: item.credential_ref ?? "",
|
||||
local_overrides: JSON.stringify(item.local_overrides, null, 2),
|
||||
ambiguity_policy: item.ambiguity_policy
|
||||
};
|
||||
}
|
||||
|
||||
function draftKey(value: ConnectorConfigurationDraft): string {
|
||||
return JSON.stringify({
|
||||
name: value.name.trim(),
|
||||
status: value.status,
|
||||
endpoint_url: value.endpoint_url.trim(),
|
||||
credential_ref: value.credential_ref.trim(),
|
||||
local_overrides: normalizeJson(value.local_overrides),
|
||||
ambiguity_policy: value.ambiguity_policy
|
||||
});
|
||||
}
|
||||
|
||||
function normalizeJson(value: string): unknown {
|
||||
try {
|
||||
return JSON.parse(value);
|
||||
} catch {
|
||||
return value.trim();
|
||||
}
|
||||
}
|
||||
|
||||
function parseObject(value: string, label: string): Record<string, unknown> {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (!parsed || Array.isArray(parsed) || typeof parsed !== "object") {
|
||||
throw new Error(`${label} must be a JSON object.`);
|
||||
}
|
||||
return parsed as Record<string, unknown>;
|
||||
}
|
||||
|
||||
function parseRows(value: string): Array<Record<string, unknown>> {
|
||||
const parsed: unknown = JSON.parse(value);
|
||||
if (!Array.isArray(parsed) || parsed.some((item) => !item || Array.isArray(item) || typeof item !== "object")) {
|
||||
throw new Error("Sample input must be a JSON array of objects.");
|
||||
}
|
||||
return parsed as Array<Record<string, unknown>>;
|
||||
}
|
||||
|
||||
function errorMessage(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
export { connectorsModule as default, connectorsModule } from "./module";
|
||||
@@ -0,0 +1,62 @@
|
||||
import { createElement, lazy } from "react";
|
||||
import type {
|
||||
AdminSectionsUiCapability,
|
||||
PlatformWebModule
|
||||
} from "@govoplan/core-webui";
|
||||
import "./styles/connectors.css";
|
||||
|
||||
const ConnectorGovernancePage = lazy(
|
||||
() => import("./features/ConnectorGovernancePage")
|
||||
);
|
||||
|
||||
const readScopes = [
|
||||
"connectors:source:read",
|
||||
"connectors:source:admin"
|
||||
];
|
||||
|
||||
const adminSections: AdminSectionsUiCapability = {
|
||||
sections: [
|
||||
{
|
||||
id: "connector-governance",
|
||||
moduleId: "connectors",
|
||||
kind: "management",
|
||||
surfaceId: "connectors.admin.governed-configurations",
|
||||
label: "Connector governance",
|
||||
group: "SYSTEM",
|
||||
order: 45,
|
||||
anyOf: readScopes,
|
||||
render: ({ settings, auth }) =>
|
||||
createElement(ConnectorGovernancePage, { settings, auth })
|
||||
}
|
||||
]
|
||||
};
|
||||
|
||||
export const connectorsModule: PlatformWebModule = {
|
||||
id: "connectors",
|
||||
label: "Connectors",
|
||||
version: "0.1.18",
|
||||
dependencies: [],
|
||||
optionalDependencies: ["access", "audit", "policy", "ops"],
|
||||
viewSurfaces: [
|
||||
{
|
||||
id: "connectors.admin.governed-configurations",
|
||||
moduleId: "connectors",
|
||||
kind: "section",
|
||||
label: "Connector governance",
|
||||
order: 45
|
||||
},
|
||||
{
|
||||
id: "connectors.admin.simulation-review",
|
||||
moduleId: "connectors",
|
||||
kind: "section",
|
||||
label: "Connector simulation review",
|
||||
parentId: "connectors.admin.governed-configurations",
|
||||
order: 20
|
||||
}
|
||||
],
|
||||
uiCapabilities: {
|
||||
"admin.sections": adminSections
|
||||
}
|
||||
};
|
||||
|
||||
export default connectorsModule;
|
||||
@@ -0,0 +1,32 @@
|
||||
.connector-governance-page .connector-governance-list,
|
||||
.connector-governance-page .connector-governance-detail {
|
||||
display: grid;
|
||||
gap: var(--space-4, 1rem);
|
||||
}
|
||||
|
||||
.connector-governance-page .connector-revision-line,
|
||||
.connector-governance-page .connector-run-actions {
|
||||
align-items: center;
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: var(--space-3, 0.75rem);
|
||||
margin-bottom: var(--space-4, 1rem);
|
||||
}
|
||||
|
||||
.connector-governance-page textarea {
|
||||
font-family: var(--font-family-mono, ui-monospace, monospace);
|
||||
resize: vertical;
|
||||
}
|
||||
|
||||
.connector-governance-page .connector-definition-editor,
|
||||
.connector-governance-page .connector-json-preview {
|
||||
max-height: 34rem;
|
||||
overflow: auto;
|
||||
white-space: pre-wrap;
|
||||
}
|
||||
|
||||
.connector-governance-page .connector-json-preview {
|
||||
background: var(--surface-subtle);
|
||||
border-radius: var(--radius-md, 0.5rem);
|
||||
padding: var(--space-4, 1rem);
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
import assert from "node:assert/strict";
|
||||
import { readFileSync } from "node:fs";
|
||||
|
||||
const moduleSource = readFileSync("src/module.ts", "utf8");
|
||||
const page = readFileSync("src/features/ConnectorGovernancePage.tsx", "utf8");
|
||||
const api = readFileSync("src/api/governedConnectors.ts", "utf8");
|
||||
|
||||
assert.match(moduleSource, /"admin.sections": adminSections/);
|
||||
assert.match(moduleSource, /connectors\.admin\.governed-configurations/);
|
||||
assert.match(page, /<AdminPageLayout/);
|
||||
assert.match(page, /<PageActionBar/);
|
||||
assert.match(page, /refreshable/);
|
||||
assert.match(page, /saveAction=/);
|
||||
assert.match(page, /useUnsavedDraftGuard/);
|
||||
assert.match(page, /<WorkspaceLayout/);
|
||||
assert.match(page, /Adopt package revision/);
|
||||
assert.match(page, /Protected paths/);
|
||||
assert.match(page, /manual_review/);
|
||||
assert.match(page, /quarantine/);
|
||||
assert.match(page, /Run simulation/);
|
||||
assert.match(page, /Review ambiguous connector result/);
|
||||
assert.match(api, /idempotency_key/);
|
||||
assert.match(api, /credential_ref/);
|
||||
|
||||
console.log("Connector governance UI structural contract passed.");
|
||||
Reference in New Issue
Block a user