8 Commits
Author SHA1 Message Date
zemion 79e2315e89 Release Connectors v0.1.21 with external knowledge integration
Module Package Release / publish-packages (push) Successful in 12s
2026-08-22 14:34:36 +02:00
zemion 2c2b11f860 feat(connectors): add managed file and PostgreSQL origins
Module Package Release / publish-packages (push) Successful in 12s
2026-08-21 19:29:58 +02:00
zemion e6ab8291ec feat(connectors): govern RSS and Atom publication
Module Package Release / publish-packages (push) Successful in 11s
2026-08-21 17:36:19 +02:00
zemion de8e74b866 feat(connectors): add governed DSAR coverage 2026-08-21 13:05:54 +02:00
zemion 0f61fe4607 feat: govern connector configurations and simulations 2026-08-20 12:58:53 +02:00
zemion 3964a2e4e7 Release v0.1.18
Module Package Release / publish-packages (push) Successful in 11s
2026-08-05 21:07:44 +02:00
zemion b88f1b2c2d Release v0.1.17
Module Package Release / publish-packages (push) Successful in 10s
2026-08-05 20:33:58 +02:00
zemion 32cac8835b Release v0.1.16
Module Package Release / publish-packages (push) Successful in 11s
2026-08-05 19:52:00 +02:00
44 changed files with 12795 additions and 108 deletions
+61 -15
View File
@@ -14,12 +14,14 @@ or reporting behavior.
## Executable First Slice ## Executable First Slice
The first executable connector capability provides tenant-isolated tabular The executable connector capability provides tenant-isolated tabular origins.
origins. Operators can import bounded JSON or CSV snapshots, inspect inferred Operators can import bounded JSON or CSV snapshots, bind an exact managed Files
schemas, and expose immutable source references and content fingerprints CSV/XLSX version, or discover a table through an active governed PostgreSQL
through `connectors.datasource_origins@0.1.0`. Preview reads enforce provider configuration. Every origin exposes a reviewed schema, opaque reference, and
ceilings for rows, serialized bytes, and elapsed time and report the effective content/discovery fingerprint through `connectors.datasource_origins@0.1.0`.
limits and any truncation as structured diagnostics. Preview reads enforce provider ceilings for rows, serialized bytes, and elapsed
time and report the effective limits and any truncation as structured
diagnostics.
Connectors owns acquisition, connection profiles, credentials, discovery, and Connectors owns acquisition, connection profiles, credentials, discovery, and
provider health. `govoplan-datasources` registers an origin as a governed live provider health. `govoplan-datasources` registers an origin as a governed live
@@ -29,22 +31,64 @@ connector implementations or stores connector credentials.
Each origin declares whether it is live, cached, file-backed, or static, its Each origin declares whether it is live, cached, file-backed, or static, its
structured health state, and which projection, filter, aggregation, sorting, structured health state, and which projection, filter, aggregation, sorting,
and pagination operations it can push down. The immutable snapshot provider and pagination operations it can push down. The snapshot, managed-file, and
currently supports projection and pagination only; consumers must keep other PostgreSQL providers currently support projection and pagination only;
operations in Dataflow rather than assuming transport-side execution. consumers must keep other operations in Dataflow rather than assuming
transport-side execution.
Database, REST/HTTP, directory, managed-file, and warehouse providers can Managed-file sources are authorized and opened through
implement the same origin contract without changing Datasources or Dataflow. `files.tabular_content@1.0.0`; Files remains authoritative for ownership,
shares, download permission, exact versions, integrity, quarantine, encryption,
retention, and legal holds. CSV must be UTF-8. XLSX input is protected by
compressed-entry, expanded-byte, compression-ratio, row, and column limits.
A newer current version is reported but never silently replaces the pinned
version.
The PostgreSQL adapter accepts only an active governed connector configuration
whose secret-free endpoint uses the PostgreSQL driver. Authentication is
resolved from a tenant/scope/module/server-restricted Core credential envelope.
The adapter reflects a simple schema/table identifier, uses read-only
transactions and a statement timeout, and blocks configuration, credential, or
schema drift until an operator refreshes and reviews the source. Secrets are
never copied into source metadata or diagnostics. Other database, REST/HTTP,
directory, and warehouse providers can implement the same origin contract
without changing Datasources or Dataflow.
Governed sanctions and feed snapshot acquisitions use Core recovery operations. Governed sanctions and feed snapshot acquisitions use Core recovery operations.
The source revision/cursor, redacted dry-run decision, canonical request digest, The source revision/cursor, redacted dry-run decision, canonical request digest,
and distributed lease are durable before network I/O. Immutable snapshot rows and distributed lease are durable before network I/O. Immutable snapshot rows
and the terminal recovery checkpoint commit atomically, and an and the terminal recovery checkpoint commit atomically, and an
`Idempotency-Key` replays the committed result without contacting the provider. `Idempotency-Key` replays the committed result without contacting the provider.
Current acquisition transports are read-only. The exported external-mutation Most acquisition transports are read-only. The MediaWiki/BlueSpice knowledge
recovery contract requires stable idempotency, provider verification, and adapter is the first governed publication path: it requires an expected remote
operator reconciliation, but no connector currently claims a production write revision, a stable idempotency key, a scoped credential envelope, and durable
or delete path. forward-recovery evidence. A timeout after dispatch becomes outcome-unknown and
blocks replay until the provider revision has been reconciled.
The knowledge adapter discovers MediaWiki or BlueSpice product/version and
capabilities, maps namespaces, pages, revisions, users, categories, links,
files, discussions, redirects, and permissions into identity-stable connector
snapshots, and consumes bounded full or recent-change deltas. Optional Search
integration indexes active pages and rechecks current profile state and ACLs on
every result. Migration into native Wiki is preview-only: conflicts, attachment
collisions, unsupported macros, truncation, and source fingerprints are
reported before a target-side write is considered.
RSS and Atom emission is a bounded renderer, not an authority shortcut. Every
selected entry declares whether it came from a GovOPlaN event, publication,
case, or report and carries an opaque owning-module reference and optional
revision. Public-feed permission can render only public entries. Tenant and
private audiences require a separate restricted-feed permission, and the API
derives the allowed visibility set from that audience instead of accepting a
caller-controlled allow-list. Portal or Reporting remains responsible for any
durable public or authenticated route and must re-authorize restricted access.
The governed connector runtime adds immutable definition revisions,
revision-pinned tenant configurations, protected local overrides, explicit
package-update adoption, bounded dry-runs and simulations, redacted provenance,
idempotency, and configurable ambiguity handling through review, quarantine,
or rejection. Its administration surface is contributed to the shared system
administration workspace. Provider-specific adapters still own live writes.
Development: Development:
@@ -60,3 +104,5 @@ See:
- [Connector source lifecycle](docs/CONNECTOR_SOURCE_LIFECYCLE.md) - [Connector source lifecycle](docs/CONNECTOR_SOURCE_LIFECYCLE.md)
- [OpenProject connector concept](docs/OPENPROJECT_CONNECTOR.md) - [OpenProject connector concept](docs/OPENPROJECT_CONNECTOR.md)
- [OpenDesk integration map](docs/OPENDESK_INTEGRATION_MAP.md) - [OpenDesk integration map](docs/OPENDESK_INTEGRATION_MAP.md)
- [Governed connector configuration](docs/GOVERNED_CONNECTOR_CONFIGURATION.md)
- [MediaWiki and BlueSpice connector](docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md)
+28 -3
View File
@@ -112,6 +112,30 @@ this lifecycle when a connector publishes status.
9. Store external references with source system, object type, object id, version 9. Store external references with source system, object type, object id, version
or ETag, and last-seen timestamp. or ETag, and last-seen timestamp.
## Tabular source discovery and refresh
Immutable JSON/CSV snapshots, exact managed Files versions, and live PostgreSQL
tables share the same catalogue and bounded-preview contract. A managed-file
origin stores only the Files asset id, exact immutable version id, checksum,
parser settings, reviewed schema, and discovery fingerprint. Files re-authorizes
the current principal and verifies storage integrity and any encryption envelope
on every preview. A newer current version produces a warning; only an explicit
source refresh changes the pinned version and increments the discovery revision.
A PostgreSQL origin references an active governed connector configuration. Its
endpoint must contain no credentials. Connectors resolves the referenced Core
credential envelope for the current tenant, scope, module, and server, opens a
read-only connection, reflects a simple schema/table identifier, and records the
configuration hash/revision, credential revision, schema, and discovery
fingerprint. Configuration, credential, or schema drift blocks preview until an
explicit refresh. Missing Files capability, revoked file access, quarantine,
oversized or malformed content, inactive/stale credentials, unreachable SQL,
and timeout failures produce sanitized unavailable/validation diagnostics.
All three current providers declare projection and pagination pushdown only.
Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly
declares and tests those operations.
## Publish Flow ## Publish Flow
1. Domain module requests publish through a core-mediated connector capability. 1. Domain module requests publish through a core-mediated connector capability.
@@ -154,9 +178,10 @@ start through the connector mutation recovery contract with a stable
idempotency key, SHA-256 request digest, source revision/cursor, and dry-run idempotency key, SHA-256 request digest, source revision/cursor, and dry-run
evidence. Definitive rejection is terminal. A timeout or lost acknowledgement evidence. Definitive rejection is terminal. A timeout or lost acknowledgement
after dispatch is `outcome_unknown` and blocks replay until the owning connector after dispatch is `outcome_unknown` and blocks replay until the owning connector
verifies provider state. The contract and conformance tests exist; no current verifies provider state. The MediaWiki/BlueSpice page publisher implements this
connector advertises a production external mutation, so write/delete adoption path for revision-checked edits. Other connector types do not thereby acquire a
remains explicitly planned rather than implied. write capability; each adapter must declare and prove its own recovery and
reconciliation behavior.
## Provider Declaration ## Provider Declaration
+39 -1
View File
@@ -25,7 +25,7 @@ logic should remain visible and reviewable.
## Runtime Expectations ## Runtime Expectations
Connectors should support: Connectors supports the generic governed-definition and simulation portion of:
- discovery where possible - discovery where possible
- typed configuration through UI-managed controls - typed configuration through UI-managed controls
@@ -39,6 +39,44 @@ Connectors should support:
Configuration packages may install connector definitions, but local overrides Configuration packages may install connector definitions, but local overrides
must be protected from accidental package updates. must be protected from accidental package updates.
## Implemented Runtime Slice
The module now persists tenant-scoped connector definitions as immutable
revisions. A governed definition explicitly validates its provider, protocol,
capabilities, input/output schemas, mapping version and rules, validation,
preview metadata, audit expectations, classification, retention, limits, and
retry policy. Definitions record whether they are locally owned or supplied by
a named package.
Configurations pin a definition revision. They store an endpoint and a secret
reference, never credentials embedded in the URL. Tenant-local override values
are merged into the pinned definition and every overridden leaf is exposed as
a protected path. Installing a later package revision only marks the
configuration as having an update available. Adoption is an explicit,
optimistically locked action that reapplies the protected overrides over the
new package revision.
The generic execution surface supports bounded dry-runs and simulations. Each
run has a caller idempotency key and retains hashes of its inputs and effective
configuration together with definition, configuration, mapping, external
revision, actor, classification, and retention provenance. Samples redact the
definition's protected fields. Ambiguous uniqueness results follow the
configuration's policy and become either:
- `manual_review` with a pending decision;
- `quarantined` until an administrator decides; or
- `rejected` without a review queue entry.
Review decisions require a reason and are audited. The generic runtime stops
at deterministic preview evidence: provider-specific adapters remain
responsible for live external writes and must satisfy the Core connector
recovery contract before claiming write maturity.
The Connector governance administration page follows the shared workspace
archetype. Reload and Save stay in the semantic action bar, dirty navigation is
guarded, package adoption is a separate action, and simulation results and
review decisions remain visibly distinct from configuration editing.
## Relationship To Datasources And Dataflow ## Relationship To Datasources And Dataflow
Recurring extraction and transformation should start as configuration across Recurring extraction and transformation should start as configuration across
+134
View File
@@ -0,0 +1,134 @@
# MediaWiki and BlueSpice Knowledge Connector
The knowledge connector integrates an external MediaWiki or BlueSpice instance
without making Connectors the owner of native knowledge semantics. Connectors
owns endpoint and credential governance, Action API transport, discovery,
synchronization, identity-stable snapshots, diagnostics, and publication
recovery. Wiki continues to own GovOPlaN-native spaces, pages, drafts,
revisions, comments, and publishing policy.
## Configure and discover
Create an active governed connector definition and configuration with:
- provider `mediawiki` or `bluespice`;
- protocol `mediawiki`, `action_api`, or `mediawiki_action_api`;
- an absolute HTTP(S) endpoint without embedded credentials; and
- an optional Core credential-envelope reference. Publication requires the
credential; public read-only sources may omit it.
Core validates the endpoint and every redirect against the deployment egress,
DNS/IP-pinning, peer, downgrade, and response-size policies. Credentials can be
bearer/access tokens or username/password values resolved only for the current
tenant, actor, module, target scope, and server. They never enter connector
profiles, snapshots, diagnostics, URLs, or API responses.
A knowledge profile selects desired maturity and authority and maps each
included source namespace to a target Wiki space reference and path prefix.
Every profile also supplies a tenant or restricted fallback visibility. A
restricted fallback needs at least one normalized Search ACL token such as
`group:<id>`, `role:<id>`, `membership:<id>`, `account:<id>`,
`identity:<id>`, `function:<id>`, or `scope:<permission>`.
Discovery queries site and current-user metadata. It records the product,
version, API/PHP version, extensions, namespaces, advertised edit right,
capabilities, a stable discovery digest, and sanitized diagnostics. BlueSpice
permission and discussion extensions are detected where advertised. Standard
MediaWiki does not expose complete page ACL semantics through the base Action
API, so discovery reports that the configured namespace fallback will be used.
## Synchronize and preserve identity
Run one bounded full backfill and then bounded recent-change deltas. Every run
requires an idempotency key; exact replay returns the original evidence without
contacting the source, while reusing the key with different inputs is rejected.
The profile stores only its latest cursor and high-watermark. Runs retain
counts, redacted effects, diagnostics, configuration/discovery provenance, and
transport evidence.
Snapshots use the MediaWiki page id as stable identity and preserve revision
id, canonical URL, content digest, namespace, title, source timestamp, and
observed timestamp. Page content mapping covers categories, links, file
references, discussion references, revision author references, redirect
targets, permission metadata, target space/path, and detected wikitext macros.
Referenced files remain external references; binary transfer and attachment
ownership remain with Files or DMS.
Moves update the existing stable page. Deletions use the stable page id when the
provider supplies it and otherwise match a previously synchronized namespace
and title. An unmatched deletion becomes a separate tombstone with an explicit
diagnostic instead of silently deleting an unrelated page. Permission changes
are part of the content digest and therefore update the current snapshot even
when page text is unchanged.
Desired maturity is an operator ceiling. A read-only profile cannot be used for
synchronization, migration preview, or publication merely because the provider
advertises those capabilities.
## Search and access safety
When Search is installed, Connectors registers `connectors.mediawiki.pages` for
the `external_knowledge_page` resource type. Backfill includes only active,
non-deleted pages from active profiles. Restricted documents carry the current
snapshot ACL; tenant documents carry no narrower ACL.
Search performs a fail-closed authorization recheck for every result. It
requires the external-knowledge read permission, an exact tenant match, an
active profile, a current non-deleted page, and—when restricted—intersection
with the current account, membership, identity, group, role, function, or scope
tokens. Consequently, an ACL change takes effect even before a deferred index
update completes. Immediate index updates remove deleted pages and refresh ACLs;
the next Search rebuild reconciles any transient writer failure. Search also
removes source projections when Connectors is disabled.
## Publish and reconcile
Publication requires the publish permission, an active profile and
configuration, discovered provider publication capability, desired maturity at
least `publish`, an idempotency key, page title/body, and an optional expected
external revision. The Action API edit carries `baserevid` when supplied and
uses a credential-provided or freshly acquired CSRF token.
Before remote I/O, Connectors prepares a durable forward-recovery operation
with request digest, expected revision, cursor, profile/configuration revision,
and resource identity. A confirmed provider response stores the stable page and
new revision and completes the recovery evidence. A definitive provider
rejection is terminal. A timeout, invalid response, or server failure after
dispatch becomes outcome-unknown: do not retry with another key until an
operator compares the provider page/revision and reconciles the recovery
operation. Local rollback cannot undo a confirmed external edit.
## Migration preview into Wiki
Migration is intentionally a dry-run in this slice. The request names a target
space, supported macro set, existing target paths/source identities, and
existing attachment names. The preview is bounded to 500 source pages and
records a source fingerprint and high-watermark. It reports create, update, or
conflict effects plus:
- target path already owned by a different external page;
- attachment filename collisions;
- unsupported macros;
- pages outside the requested target space; and
- truncation at the configured limit.
`can_apply=true` means the bounded preview contains no error or conflict and is
not truncated. It does not write Wiki pages. A future target-side migration
worker must use the optional connector capability and Wiki-owned mutation
contract, revalidate the preview fingerprint, and keep native Wiki permissions
and revision history authoritative.
## Operations and privacy
Provider state exposes profile product/maturity, active-object count, latest run
status, health, recovery attention, and last success time without endpoint,
credential, title, ACL, or content data. Existing snapshots may remain
available during an outage only to principals still authorized by current local
profile and ACL state; their health and freshness remain explicit.
Data-subject exports include only minimized operator attribution for profile
updates and synchronization/migration/publication runs. They exclude endpoints,
credentials, page content, titles, ACLs, namespace mappings, idempotency keys,
request hashes, effects, diagnostics, provenance, and transport evidence.
Attribution and external-operation evidence is retained for governance rather
than automatically erased.
+3 -2
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "govoplan-connectors" name = "govoplan-connectors"
version = "0.1.15" version = "0.1.21"
description = "Governed connector catalogue and tabular source capabilities for GovOPlaN." description = "Governed connector catalogue and tabular source capabilities for GovOPlaN."
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
@@ -12,7 +12,8 @@ license = "AGPL-3.0-or-later"
authors = [{ name = "GovOPlaN" }] authors = [{ name = "GovOPlaN" }]
dependencies = [ dependencies = [
"defusedxml>=0.7,<1", "defusedxml>=0.7,<1",
"govoplan-core>=0.1.15", "govoplan-core>=0.1.32",
"openpyxl>=3.1.5,<4",
] ]
[tool.setuptools.packages.find] [tool.setuptools.packages.find]
@@ -102,12 +102,16 @@ class ConnectorDatasourceOriginProvider:
def _origin(source: TabularSource) -> DatasourceOrigin: def _origin(source: TabularSource) -> DatasourceOrigin:
kind = {
"managed_file": "file",
"postgresql": "database",
}.get(source.provider, "upload")
return DatasourceOrigin( return DatasourceOrigin(
ref=source.ref, ref=source.ref,
source_name=source.source_name, source_name=source.source_name,
name=source.name, name=source.name,
description=source.description, description=source.description,
kind="upload", kind=kind,
shape="tabular", shape="tabular",
supported_modes=("live", "cached"), supported_modes=("live", "cached"),
provider=f"connectors.{source.provider}", provider=f"connectors.{source.provider}",
@@ -5,6 +5,7 @@ from datetime import datetime
from typing import Any from typing import Any
from sqlalchemy import ( from sqlalchemy import (
Boolean,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index, Index,
@@ -255,9 +256,392 @@ class ConnectorSanctionsSnapshot(Base, TimestampMixin):
) )
class ConnectorDefinition(Base, TimestampMixin):
__tablename__ = "connector_definitions"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"definition_key",
name="uq_connector_definition_tenant_key",
),
Index(
"ix_connector_definitions_tenant_status",
"tenant_id",
"status",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
definition_key: Mapped[str] = mapped_column(String(160), nullable=False)
name: Mapped[str] = mapped_column(String(300), nullable=False)
description: Mapped[str | None] = mapped_column(Text)
status: Mapped[str] = mapped_column(
String(30),
default="active",
nullable=False,
index=True,
)
current_revision: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
source_package: Mapped[str | None] = mapped_column(String(300))
local_definition: Mapped[bool] = mapped_column(
Boolean,
default=False,
nullable=False,
)
class ConnectorDefinitionRevision(Base, TimestampMixin):
__tablename__ = "connector_definition_revisions"
__table_args__ = (
UniqueConstraint(
"definition_id",
"revision",
name="uq_connector_definition_revision",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
definition_id: Mapped[str] = mapped_column(
ForeignKey("connector_definitions.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
revision: Mapped[int] = mapped_column(Integer, nullable=False)
specification: Mapped[dict[str, Any]] = mapped_column(
JSON,
default=dict,
nullable=False,
)
definition_hash: Mapped[str] = mapped_column(
String(64),
nullable=False,
index=True,
)
origin: Mapped[str] = mapped_column(String(30), nullable=False)
package_ref: Mapped[str | None] = mapped_column(String(300))
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
class ConnectorConfiguration(Base, TimestampMixin):
__tablename__ = "connector_configurations"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"name",
name="uq_connector_configuration_tenant_name",
),
Index(
"ix_connector_configurations_tenant_status",
"tenant_id",
"status",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
definition_id: Mapped[str] = mapped_column(
ForeignKey("connector_definitions.id", ondelete="RESTRICT"),
nullable=False,
index=True,
)
name: Mapped[str] = mapped_column(String(300), nullable=False)
status: Mapped[str] = mapped_column(
String(30),
default="draft",
nullable=False,
index=True,
)
endpoint_url: Mapped[str | None] = mapped_column(String(1500))
credential_ref: Mapped[str | None] = mapped_column(String(500))
base_definition_revision: Mapped[int] = mapped_column(
Integer,
nullable=False,
)
local_overrides: Mapped[dict[str, Any]] = mapped_column(
JSON,
default=dict,
nullable=False,
)
protected_paths: Mapped[list[str]] = mapped_column(
JSON,
default=list,
nullable=False,
)
effective_configuration: Mapped[dict[str, Any]] = mapped_column(
JSON,
default=dict,
nullable=False,
)
effective_hash: Mapped[str] = mapped_column(
String(64),
nullable=False,
index=True,
)
resource_revision: Mapped[int] = mapped_column(
Integer,
default=1,
nullable=False,
)
ambiguity_policy: Mapped[str] = mapped_column(
String(30),
default="manual_review",
nullable=False,
)
updated_by: Mapped[str | None] = mapped_column(String(255), index=True)
class ConnectorSimulationRun(Base, TimestampMixin):
__tablename__ = "connector_simulation_runs"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"configuration_id",
"mode",
"idempotency_key",
name="uq_connector_simulation_run_idempotency",
),
Index(
"ix_connector_simulation_runs_review",
"tenant_id",
"review_state",
"created_at",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
configuration_id: Mapped[str] = mapped_column(
ForeignKey("connector_configurations.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
mode: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
status: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
review_state: Mapped[str] = mapped_column(
String(30),
default="not_required",
nullable=False,
index=True,
)
definition_revision: Mapped[int] = mapped_column(Integer, nullable=False)
configuration_revision: Mapped[int] = mapped_column(Integer, nullable=False)
configuration_hash: Mapped[str] = mapped_column(String(64), nullable=False)
input_hash: Mapped[str] = mapped_column(String(64), nullable=False)
summary: Mapped[dict[str, Any]] = mapped_column(
JSON,
default=dict,
nullable=False,
)
effects: Mapped[list[dict[str, Any]]] = mapped_column(
JSON,
default=list,
nullable=False,
)
diagnostics: Mapped[list[dict[str, Any]]] = mapped_column(
JSON,
default=list,
nullable=False,
)
provenance: Mapped[dict[str, Any]] = mapped_column(
JSON,
default=dict,
nullable=False,
)
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
reviewed_by: Mapped[str | None] = mapped_column(String(255), index=True)
reviewed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
review_reason: Mapped[str | None] = mapped_column(Text)
class ConnectorKnowledgeProfile(Base, TimestampMixin):
__tablename__ = "connector_knowledge_profiles"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"configuration_id",
name="uq_connector_knowledge_profile_configuration",
),
Index(
"ix_connector_knowledge_profiles_tenant_status",
"tenant_id",
"status",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
configuration_id: Mapped[str] = mapped_column(
ForeignKey("connector_configurations.id", ondelete="RESTRICT"),
nullable=False,
index=True,
)
status: Mapped[str] = mapped_column(
String(30), default="active", nullable=False, index=True
)
product: Mapped[str] = mapped_column(
String(50), default="unknown", nullable=False, index=True
)
product_version: Mapped[str | None] = mapped_column(String(100))
desired_maturity: Mapped[str] = mapped_column(
String(30), default="read", nullable=False
)
discovered_maturity: Mapped[str] = mapped_column(
String(30), default="discover", nullable=False
)
source_authority_mode: Mapped[str] = mapped_column(
String(40), default="external_mirror", nullable=False
)
default_visibility: Mapped[str] = mapped_column(
String(30), default="restricted", nullable=False
)
default_acl_tokens: Mapped[list[str]] = mapped_column(
JSON, default=list, nullable=False
)
namespace_mappings: Mapped[list[dict[str, Any]]] = mapped_column(
JSON, default=list, nullable=False
)
capabilities: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
discovery_revision: Mapped[str | None] = mapped_column(String(255), index=True)
discovery_evidence: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
health_status: Mapped[str] = mapped_column(
String(30), default="unknown", nullable=False, index=True
)
health_details: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
discovered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
last_sync_cursor: Mapped[str | None] = mapped_column(String(500))
last_high_watermark: Mapped[str | None] = mapped_column(String(500))
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
updated_by: Mapped[str | None] = mapped_column(String(255), index=True)
class ConnectorKnowledgeObject(Base, TimestampMixin):
__tablename__ = "connector_knowledge_objects"
__table_args__ = (
UniqueConstraint(
"profile_id",
"object_type",
"external_id",
name="uq_connector_knowledge_object_identity",
),
Index(
"ix_connector_knowledge_objects_tenant_profile_status",
"tenant_id",
"profile_id",
"status",
),
Index(
"ix_connector_knowledge_objects_tenant_updated",
"tenant_id",
"source_updated_at",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
profile_id: Mapped[str] = mapped_column(
ForeignKey("connector_knowledge_profiles.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
object_type: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
external_id: Mapped[str] = mapped_column(String(255), nullable=False)
external_page_id: Mapped[str | None] = mapped_column(String(255), index=True)
external_revision_id: Mapped[str | None] = mapped_column(String(255), index=True)
namespace_id: Mapped[int | None] = mapped_column(Integer, index=True)
title: Mapped[str] = mapped_column(String(500), nullable=False)
canonical_url: Mapped[str | None] = mapped_column(String(1500))
status: Mapped[str] = mapped_column(
String(30), default="active", nullable=False, index=True
)
redirect_target_external_id: Mapped[str | None] = mapped_column(String(255))
source_revision: Mapped[str] = mapped_column(String(255), nullable=False)
content_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
visibility: Mapped[str] = mapped_column(
String(30), default="restricted", nullable=False
)
acl_tokens: Mapped[list[str]] = mapped_column(JSON, default=list, nullable=False)
mapped_data: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
provenance: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
change_cursor: Mapped[str | None] = mapped_column(String(500), index=True)
source_updated_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), index=True
)
observed_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True
)
resource_revision: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
class ConnectorKnowledgeSyncRun(Base, TimestampMixin):
__tablename__ = "connector_knowledge_sync_runs"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"profile_id",
"mode",
"idempotency_key",
name="uq_connector_knowledge_sync_run_idempotency",
),
Index(
"ix_connector_knowledge_sync_runs_profile_started",
"tenant_id",
"profile_id",
"started_at",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
profile_id: Mapped[str] = mapped_column(
ForeignKey("connector_knowledge_profiles.id", ondelete="CASCADE"),
nullable=False,
index=True,
)
mode: Mapped[str] = mapped_column(String(40), nullable=False, index=True)
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
request_hash: Mapped[str] = mapped_column(String(64), nullable=False)
status: Mapped[str] = mapped_column(String(30), nullable=False, index=True)
cursor_before: Mapped[str | None] = mapped_column(String(500))
cursor_after: Mapped[str | None] = mapped_column(String(500))
high_watermark: Mapped[str | None] = mapped_column(String(500))
counts: Mapped[dict[str, Any]] = mapped_column(JSON, default=dict, nullable=False)
effects: Mapped[list[dict[str, Any]]] = mapped_column(
JSON, default=list, nullable=False
)
diagnostics: Mapped[list[dict[str, Any]]] = mapped_column(
JSON, default=list, nullable=False
)
provenance: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
created_by: Mapped[str | None] = mapped_column(String(255), index=True)
started_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True
)
finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))
__all__ = [ __all__ = [
"ConnectorConfiguration",
"ConnectorDefinition",
"ConnectorDefinitionRevision",
"ConnectorKnowledgeObject",
"ConnectorKnowledgeProfile",
"ConnectorKnowledgeSyncRun",
"ConnectorSanctionsAcquisitionRun", "ConnectorSanctionsAcquisitionRun",
"ConnectorSanctionsSnapshot", "ConnectorSanctionsSnapshot",
"ConnectorSimulationRun",
"ConnectorTabularSource", "ConnectorTabularSource",
"new_uuid", "new_uuid",
] ]
@@ -0,0 +1,580 @@
from __future__ import annotations
from collections.abc import Sequence
from dataclasses import dataclass
from datetime import datetime, timezone
from sqlalchemy import or_
from sqlalchemy.orm import Session
from govoplan_core.core.dsar import (
DsarErasureActionRef,
DsarExecutionResultRef,
DsarRecordRef,
DsarSubjectRef,
dsar_capability_name,
)
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
ConnectorSanctionsAcquisitionRun,
ConnectorSimulationRun,
ConnectorTabularSource,
)
CONNECTORS_DSAR_CAPABILITY = dsar_capability_name("connectors")
_MAX_RECORDS = 5_000
_CONFLICT = object()
@dataclass(frozen=True, slots=True)
class _SubjectSelectors:
account_id: str
source_id: str | None
acquisition_id: str | None
definition_id: str | None
configuration_id: str | None
simulation_id: str | None
knowledge_profile_id: str | None
knowledge_run_id: str | None
@property
def narrowed(self) -> bool:
return any(
(
self.source_id,
self.acquisition_id,
self.definition_id,
self.configuration_id,
self.simulation_id,
self.knowledge_profile_id,
self.knowledge_run_id,
)
)
class ConnectorsDsarProvider:
provider_id = "connectors"
module_id = "connectors"
def search_subject(
self,
session: object,
*,
tenant_id: str,
subject: DsarSubjectRef,
) -> Sequence[DsarRecordRef]:
db = _session(session)
selectors = _subject_selectors(subject)
if selectors is None:
return ()
records: list[DsarRecordRef] = []
if not selectors.narrowed or selectors.source_id:
query = db.query(ConnectorTabularSource).filter(
ConnectorTabularSource.tenant_id == tenant_id,
or_(
ConnectorTabularSource.created_by == selectors.account_id,
ConnectorTabularSource.updated_by == selectors.account_id,
),
)
if selectors.source_id:
query = query.filter(ConnectorTabularSource.id == selectors.source_id)
records.extend(
_source_attribution(row, selectors.account_id)
for row in _limited(
query,
ConnectorTabularSource.created_at,
ConnectorTabularSource.id,
label="source attribution",
)
)
if not selectors.narrowed or selectors.acquisition_id:
query = db.query(ConnectorSanctionsAcquisitionRun).filter(
ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id,
ConnectorSanctionsAcquisitionRun.created_by == selectors.account_id,
)
if selectors.acquisition_id:
query = query.filter(
ConnectorSanctionsAcquisitionRun.id == selectors.acquisition_id
)
records.extend(
_acquisition_attribution(row)
for row in _limited(
query,
ConnectorSanctionsAcquisitionRun.started_at,
ConnectorSanctionsAcquisitionRun.id,
label="acquisition attribution",
)
)
if not selectors.narrowed or selectors.definition_id:
query = (
db.query(ConnectorDefinitionRevision, ConnectorDefinition)
.join(
ConnectorDefinition,
ConnectorDefinition.id == ConnectorDefinitionRevision.definition_id,
)
.filter(
ConnectorDefinition.tenant_id == tenant_id,
ConnectorDefinitionRevision.created_by == selectors.account_id,
)
)
if selectors.definition_id:
query = query.filter(ConnectorDefinition.id == selectors.definition_id)
records.extend(
_definition_attribution(revision, definition)
for revision, definition in _limited(
query,
ConnectorDefinitionRevision.created_at,
ConnectorDefinitionRevision.id,
label="definition attribution",
)
)
if not selectors.narrowed or selectors.configuration_id:
query = db.query(ConnectorConfiguration).filter(
ConnectorConfiguration.tenant_id == tenant_id,
ConnectorConfiguration.updated_by == selectors.account_id,
)
if selectors.configuration_id:
query = query.filter(
ConnectorConfiguration.id == selectors.configuration_id
)
records.extend(
_configuration_attribution(row)
for row in _limited(
query,
ConnectorConfiguration.updated_at,
ConnectorConfiguration.id,
label="configuration attribution",
)
)
if not selectors.narrowed or selectors.simulation_id:
query = db.query(ConnectorSimulationRun).filter(
ConnectorSimulationRun.tenant_id == tenant_id,
or_(
ConnectorSimulationRun.created_by == selectors.account_id,
ConnectorSimulationRun.reviewed_by == selectors.account_id,
),
)
if selectors.simulation_id:
query = query.filter(
ConnectorSimulationRun.id == selectors.simulation_id
)
records.extend(
_simulation_attribution(row, selectors.account_id)
for row in _limited(
query,
ConnectorSimulationRun.created_at,
ConnectorSimulationRun.id,
label="simulation attribution",
)
)
if not selectors.narrowed or selectors.knowledge_profile_id:
query = db.query(ConnectorKnowledgeProfile).filter(
ConnectorKnowledgeProfile.tenant_id == tenant_id,
ConnectorKnowledgeProfile.updated_by == selectors.account_id,
)
if selectors.knowledge_profile_id:
query = query.filter(
ConnectorKnowledgeProfile.id == selectors.knowledge_profile_id
)
records.extend(
_knowledge_profile_attribution(row)
for row in _limited(
query,
ConnectorKnowledgeProfile.created_at,
ConnectorKnowledgeProfile.id,
label="knowledge profile attribution",
)
)
if not selectors.narrowed or selectors.knowledge_run_id:
query = db.query(ConnectorKnowledgeSyncRun).filter(
ConnectorKnowledgeSyncRun.tenant_id == tenant_id,
ConnectorKnowledgeSyncRun.created_by == selectors.account_id,
)
if selectors.knowledge_run_id:
query = query.filter(
ConnectorKnowledgeSyncRun.id == selectors.knowledge_run_id
)
records.extend(
_knowledge_run_attribution(row)
for row in _limited(
query,
ConnectorKnowledgeSyncRun.started_at,
ConnectorKnowledgeSyncRun.id,
label="knowledge run attribution",
)
)
if len(records) > _MAX_RECORDS:
raise ValueError("Connectors DSAR result limit exceeded; narrow selectors.")
return tuple(
sorted(records, key=lambda item: (item.resource_type, item.resource_id))
)
def plan_erasure(
self,
session: object,
*,
tenant_id: str,
subject: DsarSubjectRef,
records: Sequence[DsarRecordRef],
) -> Sequence[DsarErasureActionRef]:
del tenant_id
_session(session)
if _subject_selectors(subject) is None:
raise ValueError("Connectors DSAR subject selectors conflict.")
actions: list[DsarErasureActionRef] = []
for record in records:
_validate_record(record)
actions.append(
DsarErasureActionRef(
action_id=(
f"connectors:retain:{record.resource_type}:{record.resource_id}"
),
provider_id=self.provider_id,
module_id=self.module_id,
kind="retain",
resource_type=record.resource_type,
resource_id=record.resource_id,
title=f"Retain {record.title}",
rationale=(
record.retention_reason
or "Connector operator attribution remains governance evidence."
),
executable=False,
)
)
return tuple(actions)
def execute_erasure(
self,
session: object,
*,
tenant_id: str,
subject: DsarSubjectRef,
actions: Sequence[DsarErasureActionRef],
request_id: str,
) -> Sequence[DsarExecutionResultRef]:
del tenant_id
_session(session)
if _subject_selectors(subject) is None:
raise ValueError("Connectors DSAR subject selectors conflict.")
results: list[DsarExecutionResultRef] = []
for action in actions:
_validate_action(action)
if action.executable or action.kind != "retain":
raise ValueError("Connectors DSAR publishes retain actions only.")
results.append(
DsarExecutionResultRef(
action_id=action.action_id,
status="blocked",
summary="Connector operator attribution remains governance evidence.",
evidence={"request_id": request_id},
)
)
return tuple(results)
def _subject_selectors(subject: DsarSubjectRef) -> _SubjectSelectors | None:
references = subject.external_references
account = _coalesce(
subject.account_id,
references.get("connectors.account"),
references.get("access.account"),
)
values = {
"source_id": _coalesce(
references.get("connectors.source"),
references.get("connectors.source_id"),
),
"acquisition_id": _coalesce(
references.get("connectors.acquisition"),
references.get("connectors.acquisition_id"),
),
"definition_id": _coalesce(
references.get("connectors.definition"),
references.get("connectors.definition_id"),
),
"configuration_id": _coalesce(
references.get("connectors.configuration"),
references.get("connectors.configuration_id"),
),
"simulation_id": _coalesce(
references.get("connectors.simulation"),
references.get("connectors.simulation_id"),
),
"knowledge_profile_id": _coalesce(
references.get("connectors.knowledge_profile"),
references.get("connectors.knowledge_profile_id"),
),
"knowledge_run_id": _coalesce(
references.get("connectors.knowledge_run"),
references.get("connectors.knowledge_run_id"),
),
}
if account is _CONFLICT or any(value is _CONFLICT for value in values.values()):
return None
account_id = _optional_string(account)
if not account_id:
return None
return _SubjectSelectors(
account_id=account_id,
source_id=_optional_string(values["source_id"]),
acquisition_id=_optional_string(values["acquisition_id"]),
definition_id=_optional_string(values["definition_id"]),
configuration_id=_optional_string(values["configuration_id"]),
simulation_id=_optional_string(values["simulation_id"]),
knowledge_profile_id=_optional_string(values["knowledge_profile_id"]),
knowledge_run_id=_optional_string(values["knowledge_run_id"]),
)
def _source_attribution(row: ConnectorTabularSource, account_id: str) -> DsarRecordRef:
activities = []
if row.created_by == account_id:
activities.append("created_source_snapshot")
if row.updated_by == account_id:
activities.append("updated_source_snapshot")
return _record(
resource_type="source_actor_attribution",
resource_id=row.id,
title="Connector source actor attribution",
data={
"source_id": row.id,
"provider": row.provider,
"status": row.status,
"schema_version": row.schema_version,
"row_count": row.row_count,
"activities": activities,
"created_at": _iso(row.created_at),
"updated_at": _iso(row.updated_at),
"retired_at": _iso(row.deleted_at),
},
observed_at=row.updated_at,
)
def _acquisition_attribution(row: ConnectorSanctionsAcquisitionRun) -> DsarRecordRef:
return _record(
resource_type="acquisition_actor_attribution",
resource_id=row.id,
title="Connector acquisition actor attribution",
data={
"acquisition_id": row.id,
"provider_id": row.provider_id,
"source_id": row.source_id,
"status": row.status,
"attempt_count": row.attempt_count,
"started_at": _iso(row.started_at),
"finished_at": _iso(row.finished_at),
"activity": "started_source_acquisition",
},
observed_at=row.finished_at or row.started_at,
)
def _definition_attribution(
row: ConnectorDefinitionRevision, definition: ConnectorDefinition
) -> DsarRecordRef:
return _record(
resource_type="definition_actor_attribution",
resource_id=row.id,
title="Connector definition actor attribution",
data={
"definition_id": definition.id,
"revision_id": row.id,
"revision": row.revision,
"origin": row.origin,
"activity": "created_definition_revision",
"created_at": _iso(row.created_at),
},
observed_at=row.created_at,
)
def _configuration_attribution(row: ConnectorConfiguration) -> DsarRecordRef:
return _record(
resource_type="configuration_actor_attribution",
resource_id=row.id,
title="Connector configuration actor attribution",
data={
"configuration_id": row.id,
"definition_id": row.definition_id,
"status": row.status,
"base_definition_revision": row.base_definition_revision,
"resource_revision": row.resource_revision,
"ambiguity_policy": row.ambiguity_policy,
"activity": "updated_connector_configuration",
"updated_at": _iso(row.updated_at),
},
observed_at=row.updated_at,
)
def _simulation_attribution(
row: ConnectorSimulationRun, account_id: str
) -> DsarRecordRef:
activities = []
if row.created_by == account_id:
activities.append("created_simulation")
if row.reviewed_by == account_id:
activities.append("reviewed_simulation")
return _record(
resource_type="simulation_actor_attribution",
resource_id=row.id,
title="Connector simulation actor attribution",
data={
"simulation_id": row.id,
"configuration_id": row.configuration_id,
"mode": row.mode,
"status": row.status,
"review_state": row.review_state,
"definition_revision": row.definition_revision,
"configuration_revision": row.configuration_revision,
"activities": activities,
"created_at": _iso(row.created_at),
"reviewed_at": _iso(row.reviewed_at),
},
observed_at=row.reviewed_at or row.created_at,
)
def _knowledge_profile_attribution(
row: ConnectorKnowledgeProfile,
) -> DsarRecordRef:
return _record(
resource_type="knowledge_profile_actor_attribution",
resource_id=row.id,
title="External knowledge profile actor attribution",
data={
"knowledge_profile_id": row.id,
"configuration_id": row.configuration_id,
"status": row.status,
"desired_maturity": row.desired_maturity,
"source_authority_mode": row.source_authority_mode,
"resource_revision": row.resource_revision,
"activity": "updated_external_knowledge_profile",
"created_at": _iso(row.created_at),
"updated_at": _iso(row.updated_at),
},
observed_at=row.updated_at,
)
def _knowledge_run_attribution(
row: ConnectorKnowledgeSyncRun,
) -> DsarRecordRef:
return _record(
resource_type="knowledge_run_actor_attribution",
resource_id=row.id,
title="External knowledge operation actor attribution",
data={
"knowledge_run_id": row.id,
"knowledge_profile_id": row.profile_id,
"mode": row.mode,
"status": row.status,
"started_at": _iso(row.started_at),
"finished_at": _iso(row.finished_at),
"activity": "started_external_knowledge_operation",
},
observed_at=row.finished_at or row.started_at,
)
def _record(
*,
resource_type: str,
resource_id: str,
title: str,
data: dict[str, object],
observed_at: datetime | None,
) -> DsarRecordRef:
return DsarRecordRef(
provider_id="connectors",
module_id="connectors",
resource_type=resource_type,
resource_id=resource_id,
category="connector_governance_attribution",
title=title,
data=data,
observed_at=_aware(observed_at),
immutable_evidence=True,
retention_reason=(
"Connector attribution is retained for configuration, review, and "
"external-operation accountability."
),
)
def _limited(query, first, second, *, label: str):
rows = query.order_by(first, second).limit(_MAX_RECORDS + 1).all()
if len(rows) > _MAX_RECORDS:
raise ValueError(f"Connectors DSAR {label} limit exceeded; narrow selectors.")
return rows
def _coalesce(*values: str | None) -> str | None | object:
normalized = {str(value).strip() for value in values if str(value or "").strip()}
if len(normalized) > 1:
return _CONFLICT
return next(iter(normalized), None)
def _optional_string(value: object) -> str | None:
return value if isinstance(value, str) and value else None
def _iso(value: datetime | None) -> str | None:
aware = _aware(value)
return aware.isoformat() if aware else None
def _aware(value: datetime | None) -> datetime | None:
if value is None or value.tzinfo is not None:
return value
return value.replace(tzinfo=timezone.utc)
def _session(value: object) -> Session:
if not isinstance(value, Session):
raise TypeError("Connectors DSAR requires a SQLAlchemy Session.")
return value
_RESOURCE_TYPES = {
"source_actor_attribution",
"acquisition_actor_attribution",
"definition_actor_attribution",
"configuration_actor_attribution",
"simulation_actor_attribution",
"knowledge_profile_actor_attribution",
"knowledge_run_actor_attribution",
}
def _validate_record(record: DsarRecordRef) -> None:
if record.provider_id != "connectors" or record.module_id != "connectors":
raise ValueError("Connectors DSAR cannot plan a foreign provider record.")
if record.resource_type not in _RESOURCE_TYPES or not record.resource_id:
raise ValueError("Connectors DSAR record identity is invalid.")
def _validate_action(action: DsarErasureActionRef) -> None:
if action.provider_id != "connectors" or action.module_id != "connectors":
raise ValueError("Connectors DSAR cannot execute a foreign provider action.")
if not action.action_id.startswith("connectors:retain:"):
raise ValueError("Connectors DSAR action identity is invalid.")
__all__ = ["CONNECTORS_DSAR_CAPABILITY", "ConnectorsDsarProvider"]
+9 -1
View File
@@ -24,6 +24,8 @@ from govoplan_core.security.http_fetch import fetch_http
MAX_FEED_BYTES = 5_000_000 MAX_FEED_BYTES = 5_000_000
ATOM_NS = "http://www.w3.org/2005/Atom" ATOM_NS = "http://www.w3.org/2005/Atom"
FEED_PUBLISH_SCOPE = "connectors:feeds:publish"
FEED_PRIVATE_PUBLISH_SCOPE = "connectors:feeds:publish_private"
class ConnectorFeedProvider(FeedProvider): class ConnectorFeedProvider(FeedProvider):
@@ -410,4 +412,10 @@ def _element(
return child return child
__all__ = ["ConnectorFeedProvider", "MAX_FEED_BYTES", "feed_rows"] __all__ = [
"ConnectorFeedProvider",
"FEED_PRIVATE_PUBLISH_SCOPE",
"FEED_PUBLISH_SCOPE",
"MAX_FEED_BYTES",
"feed_rows",
]
@@ -0,0 +1,921 @@
from __future__ import annotations
import copy
from datetime import UTC, datetime
import hashlib
import json
from typing import Any, Mapping, Sequence
from sqlalchemy.orm import Session
from govoplan_core.audit.logging import audit_from_principal
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.connector_runtime import ConnectorContractError, ConnectorEndpoint
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorSimulationRun,
)
from govoplan_connectors.backend.governed_schemas import (
ConnectorConfigurationCreateRequest,
ConnectorConfigurationItem,
ConnectorConfigurationUpdateRequest,
ConnectorDefinitionItem,
ConnectorDefinitionUpsertRequest,
ConnectorReviewRequest,
ConnectorRunItem,
ConnectorRunRequest,
GovernedConnectorSpecification,
)
class GovernedConnectorError(ValueError):
def __init__(self, code: str, message: str) -> None:
super().__init__(message)
self.code = code
def list_definitions(
session: Session,
*,
tenant_id: str,
) -> list[ConnectorDefinitionItem]:
rows = (
session.query(ConnectorDefinition)
.filter(
ConnectorDefinition.tenant_id == tenant_id,
ConnectorDefinition.status == "active",
)
.order_by(ConnectorDefinition.name.asc())
.all()
)
return [
definition_item(
session,
row,
_definition_revision(session, row, row.current_revision),
)
for row in rows
]
def upsert_definition(
session: Session,
principal: ApiPrincipal,
payload: ConnectorDefinitionUpsertRequest,
) -> ConnectorDefinitionItem:
specification = payload.specification.model_dump(mode="json")
definition_hash = _hash(specification)
definition = (
session.query(ConnectorDefinition)
.filter(
ConnectorDefinition.tenant_id == principal.tenant_id,
ConnectorDefinition.definition_key == payload.definition_key,
)
.one_or_none()
)
expected_local = payload.origin == "local"
if definition is None:
definition = ConnectorDefinition(
tenant_id=principal.tenant_id,
definition_key=payload.definition_key,
name=payload.name.strip(),
description=_optional_text(payload.description),
status="active",
current_revision=0,
source_package=payload.package_ref,
local_definition=expected_local,
)
session.add(definition)
session.flush()
elif definition.local_definition != expected_local:
if definition.local_definition:
raise GovernedConnectorError(
"local_definition_protected",
"A package update cannot replace a locally owned connector definition.",
)
raise GovernedConnectorError(
"package_definition_requires_overrides",
"Use configuration overrides instead of converting a package definition into a local definition.",
)
else:
current = _definition_revision(
session,
definition,
definition.current_revision,
)
if current.definition_hash == definition_hash:
return definition_item(session, definition, current)
definition.name = payload.name.strip()
definition.description = _optional_text(payload.description)
definition.source_package = payload.package_ref
definition.current_revision += 1
revision = ConnectorDefinitionRevision(
definition_id=definition.id,
revision=definition.current_revision,
specification=specification,
definition_hash=definition_hash,
origin=payload.origin,
package_ref=payload.package_ref,
created_by=principal.user.id,
)
session.add(revision)
session.flush()
audit_from_principal(
session,
principal,
action="connectors.definition.revision_created",
object_type="connector_definition",
object_id=definition.id,
details={
"definition_key": definition.definition_key,
"revision": revision.revision,
"definition_hash": definition_hash,
"origin": payload.origin,
"package_ref": payload.package_ref,
"provider": payload.specification.provider,
"protocol": payload.specification.protocol,
"capabilities": sorted(payload.specification.capabilities),
},
)
session.commit()
return definition_item(session, definition, revision)
def list_configurations(
session: Session,
*,
tenant_id: str,
) -> list[ConnectorConfigurationItem]:
rows = (
session.query(ConnectorConfiguration)
.filter(ConnectorConfiguration.tenant_id == tenant_id)
.order_by(ConnectorConfiguration.name.asc())
.all()
)
return [configuration_item(session, row) for row in rows]
def create_configuration(
session: Session,
principal: ApiPrincipal,
payload: ConnectorConfigurationCreateRequest,
) -> ConnectorConfigurationItem:
definition = _tenant_definition(
session,
tenant_id=principal.tenant_id,
definition_id=payload.definition_id,
)
_validate_endpoint(payload.endpoint_url, payload.credential_ref)
revision = _definition_revision(session, definition, definition.current_revision)
overrides = copy.deepcopy(payload.local_overrides)
effective = _effective_specification(revision.specification, overrides)
item = ConnectorConfiguration(
tenant_id=principal.tenant_id,
definition_id=definition.id,
name=payload.name.strip(),
status=payload.status,
endpoint_url=_optional_text(payload.endpoint_url),
credential_ref=_optional_text(payload.credential_ref),
base_definition_revision=definition.current_revision,
local_overrides=overrides,
protected_paths=_protected_paths(overrides),
effective_configuration=effective,
effective_hash=_hash(effective),
resource_revision=1,
ambiguity_policy=payload.ambiguity_policy,
updated_by=principal.user.id,
)
session.add(item)
session.flush()
_audit_configuration(
session,
principal,
item,
action="connectors.configuration.created",
)
session.commit()
return configuration_item(session, item)
def update_configuration(
session: Session,
principal: ApiPrincipal,
*,
configuration_id: str,
payload: ConnectorConfigurationUpdateRequest,
) -> ConnectorConfigurationItem:
item = _tenant_configuration(
session,
tenant_id=principal.tenant_id,
configuration_id=configuration_id,
)
if item.resource_revision != payload.expected_revision:
raise GovernedConnectorError(
"configuration_conflict",
"The connector configuration changed; reload it before saving.",
)
supplied = payload.model_fields_set
definition = _tenant_definition(
session,
tenant_id=principal.tenant_id,
definition_id=item.definition_id,
)
if "name" in supplied and payload.name is not None:
item.name = payload.name.strip()
if "endpoint_url" in supplied:
item.endpoint_url = _optional_text(payload.endpoint_url)
if "credential_ref" in supplied:
item.credential_ref = _optional_text(payload.credential_ref)
if "local_overrides" in supplied and payload.local_overrides is not None:
item.local_overrides = copy.deepcopy(payload.local_overrides)
if payload.ambiguity_policy is not None:
item.ambiguity_policy = payload.ambiguity_policy
if payload.status is not None:
item.status = payload.status
if payload.adopt_latest_definition:
item.base_definition_revision = definition.current_revision
_validate_endpoint(item.endpoint_url, item.credential_ref)
base = _definition_revision(
session,
definition,
item.base_definition_revision,
)
effective = _effective_specification(base.specification, item.local_overrides)
item.protected_paths = _protected_paths(item.local_overrides)
item.effective_configuration = effective
item.effective_hash = _hash(effective)
item.resource_revision += 1
item.updated_by = principal.user.id
_audit_configuration(
session,
principal,
item,
action="connectors.configuration.updated",
extra={"adopted_latest_definition": payload.adopt_latest_definition},
)
session.commit()
return configuration_item(session, item)
def execute_run(
session: Session,
principal: ApiPrincipal,
*,
configuration_id: str,
mode: str,
payload: ConnectorRunRequest,
) -> ConnectorRunItem:
if mode not in {"dry_run", "simulation"}:
raise GovernedConnectorError("invalid_mode", "Unsupported connector run mode.")
configuration = _tenant_configuration(
session,
tenant_id=principal.tenant_id,
configuration_id=configuration_id,
)
if configuration.status == "disabled":
raise GovernedConnectorError(
"configuration_disabled",
"Disabled connector configurations cannot be executed.",
)
specification = GovernedConnectorSpecification.model_validate(
configuration.effective_configuration
)
if mode == "dry_run" and not specification.dry_run.supported:
raise GovernedConnectorError(
"dry_run_unsupported",
"This connector definition does not support dry runs.",
)
if mode == "simulation" and not specification.dry_run.simulation_supported:
raise GovernedConnectorError(
"simulation_unsupported",
"This connector definition does not support simulation.",
)
rows = (
list(payload.input_rows)
if payload.input_rows is not None
else list(specification.dry_run.sample_rows)
)
request_payload = {
"mode": mode,
"configuration_id": configuration.id,
"configuration_revision": configuration.resource_revision,
"configuration_hash": configuration.effective_hash,
"external_revision": payload.external_revision,
"input_rows": rows,
}
request_hash = _hash(request_payload)
existing = (
session.query(ConnectorSimulationRun)
.filter(
ConnectorSimulationRun.tenant_id == principal.tenant_id,
ConnectorSimulationRun.configuration_id == configuration.id,
ConnectorSimulationRun.mode == mode,
ConnectorSimulationRun.idempotency_key == payload.idempotency_key,
)
.one_or_none()
)
if existing is not None:
if existing.request_hash != request_hash:
raise GovernedConnectorError(
"idempotency_conflict",
"The idempotency key was already used with different run inputs.",
)
return run_item(existing)
limit = specification.dry_run.max_items
truncated = len(rows) > limit
bounded_rows = rows[:limit]
effects, diagnostics, ambiguous_count = _simulate(
bounded_rows,
specification,
)
if truncated:
diagnostics.append(
{
"severity": "warning",
"code": "connectors.run.truncated",
"message": f"The run was limited to {limit} input rows.",
"stage": "planning",
"retryable": False,
}
)
errors = sum(1 for item in diagnostics if item["severity"] == "error")
if ambiguous_count:
status_value, review_state = {
"manual_review": ("manual_review", "pending"),
"quarantine": ("quarantined", "quarantined"),
"reject": ("rejected", "not_required"),
}[configuration.ambiguity_policy]
elif errors:
status_value, review_state = "invalid", "not_required"
else:
status_value, review_state = "ready", "not_required"
input_hash = _hash(bounded_rows)
summary = {
"total": len(effects),
"creates": sum(1 for item in effects if item["effect"] == "create"),
"updates": 0,
"deletes": 0,
"conflicts": sum(1 for item in effects if item["effect"] == "conflict"),
"unchanged": 0,
"ignored": sum(1 for item in effects if item["effect"] == "ignored"),
"ambiguous": ambiguous_count,
"errors": errors,
"truncated": truncated,
}
now = datetime.now(UTC)
provenance = {
"contract_version": "1.0",
"definition_id": configuration.definition_id,
"definition_revision": configuration.base_definition_revision,
"configuration_id": configuration.id,
"configuration_revision": configuration.resource_revision,
"configuration_hash": configuration.effective_hash,
"mapping_version": specification.mapping.version,
"input_hash": input_hash,
"external_revision": payload.external_revision,
"generated_at": now.isoformat(),
"actor_id": principal.user.id,
"mode": mode,
"provider": specification.provider,
"protocol": specification.protocol,
"privacy_classification": specification.privacy_classification,
"retention_class": specification.retention_class,
}
run = ConnectorSimulationRun(
tenant_id=principal.tenant_id,
configuration_id=configuration.id,
mode=mode,
idempotency_key=payload.idempotency_key,
request_hash=request_hash,
status=status_value,
review_state=review_state,
definition_revision=configuration.base_definition_revision,
configuration_revision=configuration.resource_revision,
configuration_hash=configuration.effective_hash,
input_hash=input_hash,
summary=summary,
effects=effects,
diagnostics=diagnostics,
provenance=provenance,
created_by=principal.user.id,
)
session.add(run)
session.flush()
audit_from_principal(
session,
principal,
action=f"connectors.configuration.{mode}_completed",
object_type="connector_simulation_run",
object_id=run.id,
details={
"configuration_id": configuration.id,
"configuration_revision": configuration.resource_revision,
"configuration_hash": configuration.effective_hash,
"definition_revision": configuration.base_definition_revision,
"input_hash": input_hash,
"status": status_value,
"review_state": review_state,
"summary": summary,
},
)
session.commit()
return run_item(run)
def list_runs(
session: Session,
*,
tenant_id: str,
configuration_id: str | None = None,
review_state: str | None = None,
limit: int = 100,
) -> list[ConnectorRunItem]:
query = session.query(ConnectorSimulationRun).filter(
ConnectorSimulationRun.tenant_id == tenant_id
)
if configuration_id:
query = query.filter(
ConnectorSimulationRun.configuration_id == configuration_id
)
if review_state:
query = query.filter(ConnectorSimulationRun.review_state == review_state)
rows = (
query.order_by(ConnectorSimulationRun.created_at.desc())
.limit(max(1, min(int(limit), 500)))
.all()
)
return [run_item(row) for row in rows]
def review_run(
session: Session,
principal: ApiPrincipal,
*,
run_id: str,
payload: ConnectorReviewRequest,
) -> ConnectorRunItem:
run = (
session.query(ConnectorSimulationRun)
.filter(
ConnectorSimulationRun.id == run_id,
ConnectorSimulationRun.tenant_id == principal.tenant_id,
)
.one_or_none()
)
if run is None:
raise GovernedConnectorError("run_not_found", "Connector run not found.")
if run.review_state not in {"pending", "quarantined"}:
raise GovernedConnectorError(
"run_not_reviewable",
"Only pending or quarantined connector results can be reviewed.",
)
run.review_state = payload.decision
run.status = f"review_{payload.decision}"
run.reviewed_by = principal.user.id
run.reviewed_at = datetime.now(UTC)
run.review_reason = payload.reason.strip()
audit_from_principal(
session,
principal,
action=f"connectors.simulation.{payload.decision}",
object_type="connector_simulation_run",
object_id=run.id,
details={
"configuration_id": run.configuration_id,
"input_hash": run.input_hash,
"configuration_hash": run.configuration_hash,
"decision": payload.decision,
"reason": run.review_reason,
},
)
session.commit()
return run_item(run)
def definition_item(
session: Session,
definition: ConnectorDefinition,
revision: ConnectorDefinitionRevision,
) -> ConnectorDefinitionItem:
del session
return ConnectorDefinitionItem(
id=definition.id,
tenant_id=definition.tenant_id,
definition_key=definition.definition_key,
name=definition.name,
description=definition.description,
status=definition.status,
current_revision=definition.current_revision,
source_package=definition.source_package,
local_definition=definition.local_definition,
revision_id=revision.id,
definition_hash=revision.definition_hash,
origin=revision.origin,
package_ref=revision.package_ref,
specification=GovernedConnectorSpecification.model_validate(
revision.specification
),
created_at=definition.created_at,
updated_at=definition.updated_at,
)
def configuration_item(
session: Session,
item: ConnectorConfiguration,
) -> ConnectorConfigurationItem:
definition = _tenant_definition(
session,
tenant_id=item.tenant_id,
definition_id=item.definition_id,
)
return ConnectorConfigurationItem(
id=item.id,
tenant_id=item.tenant_id,
definition_id=item.definition_id,
definition_key=definition.definition_key,
definition_name=definition.name,
name=item.name,
status=item.status,
endpoint_url=item.endpoint_url,
credential_ref=item.credential_ref,
base_definition_revision=item.base_definition_revision,
latest_definition_revision=definition.current_revision,
update_available=definition.current_revision > item.base_definition_revision,
local_overrides=dict(item.local_overrides or {}),
protected_paths=list(item.protected_paths or []),
effective_configuration=dict(item.effective_configuration or {}),
effective_hash=item.effective_hash,
resource_revision=item.resource_revision,
ambiguity_policy=item.ambiguity_policy,
updated_at=item.updated_at,
)
def run_item(run: ConnectorSimulationRun) -> ConnectorRunItem:
return ConnectorRunItem(
id=run.id,
tenant_id=run.tenant_id,
configuration_id=run.configuration_id,
mode=run.mode, # type: ignore[arg-type]
idempotency_key=run.idempotency_key,
status=run.status,
review_state=run.review_state,
definition_revision=run.definition_revision,
configuration_revision=run.configuration_revision,
configuration_hash=run.configuration_hash,
input_hash=run.input_hash,
summary=dict(run.summary or {}),
effects=list(run.effects or []),
diagnostics=list(run.diagnostics or []),
provenance=dict(run.provenance or {}),
reviewed_by=run.reviewed_by,
reviewed_at=run.reviewed_at,
review_reason=run.review_reason,
created_at=run.created_at,
)
def _simulate(
rows: Sequence[Mapping[str, Any]],
specification: GovernedConnectorSpecification,
) -> tuple[list[dict[str, Any]], list[dict[str, Any]], int]:
mapped_rows: list[dict[str, Any]] = []
row_errors: dict[int, list[dict[str, Any]]] = {}
diagnostics: list[dict[str, Any]] = []
for index, row in enumerate(rows):
mapped: dict[str, Any] = {}
for rule in specification.mapping.rules:
value, found = _path_value(row, rule.source)
if not found:
value = rule.default
if rule.required and value in (None, ""):
diagnostic = _diagnostic(
"error",
"connectors.mapping.required_source_missing",
f"Required source field {rule.source!r} is missing.",
"mapping",
index=index,
field=rule.source,
)
row_errors.setdefault(index, []).append(diagnostic)
diagnostics.append(diagnostic)
_set_path(mapped, rule.target, value)
mapped_rows.append(mapped)
ambiguous_indexes: set[int] = set()
for rule in specification.validation_rules:
if rule.kind == "unique":
seen: dict[str, list[int]] = {}
for index, row in enumerate(mapped_rows):
value, found = _path_value(row, rule.field)
if found and value not in (None, ""):
seen.setdefault(_stable_value(value), []).append(index)
for indexes in seen.values():
if len(indexes) > 1:
ambiguous_indexes.update(indexes)
for index in indexes:
diagnostic = _diagnostic(
rule.severity,
rule.code,
rule.message,
"validation",
index=index,
field=rule.field,
)
row_errors.setdefault(index, []).append(diagnostic)
diagnostics.append(diagnostic)
continue
for index, row in enumerate(mapped_rows):
value, found = _path_value(row, rule.field)
invalid = (
rule.kind == "required" and (not found or value in (None, ""))
) or (
rule.kind == "one_of" and found and value not in rule.values
)
if invalid:
diagnostic = _diagnostic(
rule.severity,
rule.code,
rule.message,
"validation",
index=index,
field=rule.field,
)
row_errors.setdefault(index, []).append(diagnostic)
diagnostics.append(diagnostic)
redacted = set(specification.dry_run.redacted_fields)
effects: list[dict[str, Any]] = []
for index, mapped in enumerate(mapped_rows):
errors = row_errors.get(index, [])
has_error = any(item["severity"] == "error" for item in errors)
effect = "conflict" if has_error or index in ambiguous_indexes else "create"
effects.append(
{
"effect": effect,
"source_object_ref": _source_ref(rows[index], index),
"target_object_ref": None,
"changed_fields": sorted(_leaf_paths(mapped)),
"sample": _redact_fields(mapped, redacted),
"reason_code": (
"ambiguous_external_result"
if index in ambiguous_indexes
else errors[0]["code"] if errors else None
),
"outcome": "preview",
"revision": specification.mapping.version,
}
)
return effects, diagnostics, len(ambiguous_indexes)
def _audit_configuration(
session: Session,
principal: ApiPrincipal,
item: ConnectorConfiguration,
*,
action: str,
extra: Mapping[str, Any] | None = None,
) -> None:
audit_from_principal(
session,
principal,
action=action,
object_type="connector_configuration",
object_id=item.id,
details={
"definition_id": item.definition_id,
"base_definition_revision": item.base_definition_revision,
"resource_revision": item.resource_revision,
"effective_hash": item.effective_hash,
"protected_paths": list(item.protected_paths or []),
"ambiguity_policy": item.ambiguity_policy,
"status": item.status,
"credential_reference_present": bool(item.credential_ref),
**dict(extra or {}),
},
)
def _tenant_definition(
session: Session,
*,
tenant_id: str,
definition_id: str,
) -> ConnectorDefinition:
item = (
session.query(ConnectorDefinition)
.filter(
ConnectorDefinition.id == definition_id,
ConnectorDefinition.tenant_id == tenant_id,
)
.one_or_none()
)
if item is None:
raise GovernedConnectorError(
"definition_not_found",
"Connector definition not found.",
)
return item
def _tenant_configuration(
session: Session,
*,
tenant_id: str,
configuration_id: str,
) -> ConnectorConfiguration:
item = (
session.query(ConnectorConfiguration)
.filter(
ConnectorConfiguration.id == configuration_id,
ConnectorConfiguration.tenant_id == tenant_id,
)
.one_or_none()
)
if item is None:
raise GovernedConnectorError(
"configuration_not_found",
"Connector configuration not found.",
)
return item
def _definition_revision(
session: Session,
definition: ConnectorDefinition,
revision: int,
) -> ConnectorDefinitionRevision:
item = (
session.query(ConnectorDefinitionRevision)
.filter(
ConnectorDefinitionRevision.definition_id == definition.id,
ConnectorDefinitionRevision.revision == revision,
)
.one_or_none()
)
if item is None:
raise GovernedConnectorError(
"definition_revision_not_found",
"Connector definition revision not found.",
)
return item
def _effective_specification(
base: Mapping[str, Any],
overrides: Mapping[str, Any],
) -> dict[str, Any]:
merged = _deep_merge(base, overrides)
return GovernedConnectorSpecification.model_validate(merged).model_dump(mode="json")
def _deep_merge(
base: Mapping[str, Any],
overrides: Mapping[str, Any],
) -> dict[str, Any]:
result = copy.deepcopy(dict(base))
for key, value in overrides.items():
if isinstance(value, Mapping) and isinstance(result.get(key), Mapping):
result[key] = _deep_merge(result[key], value) # type: ignore[arg-type]
else:
result[key] = copy.deepcopy(value)
return result
def _protected_paths(value: Mapping[str, Any], prefix: str = "") -> list[str]:
paths: list[str] = []
for key in sorted(value):
path = f"{prefix}.{key}" if prefix else str(key)
item = value[key]
if isinstance(item, Mapping) and item:
paths.extend(_protected_paths(item, path))
else:
paths.append(path)
return paths
def _leaf_paths(value: Mapping[str, Any], prefix: str = "") -> set[str]:
paths: set[str] = set()
for key, item in value.items():
path = f"{prefix}.{key}" if prefix else str(key)
if isinstance(item, Mapping) and item:
paths.update(_leaf_paths(item, path))
else:
paths.add(path)
return paths
def _path_value(value: Mapping[str, Any], path: str) -> tuple[Any, bool]:
current: Any = value
for part in path.split("."):
if not isinstance(current, Mapping) or part not in current:
return None, False
current = current[part]
return current, True
def _set_path(target: dict[str, Any], path: str, value: Any) -> None:
parts = path.split(".")
current = target
for part in parts[:-1]:
nested = current.get(part)
if not isinstance(nested, dict):
nested = {}
current[part] = nested
current = nested
current[parts[-1]] = value
def _source_ref(row: Mapping[str, Any], index: int) -> str:
for key in ("id", "external_id", "source_id"):
value = row.get(key)
if value not in (None, ""):
return f"sample:{value}"
return f"sample:row:{index + 1}"
def _redact_fields(value: Mapping[str, Any], fields: set[str]) -> dict[str, Any]:
result = copy.deepcopy(dict(value))
for path in fields:
parts = path.split(".")
current: Any = result
for part in parts[:-1]:
if not isinstance(current, dict):
break
current = current.get(part)
else:
if isinstance(current, dict) and parts[-1] in current:
current[parts[-1]] = "<redacted>"
return result
def _diagnostic(
severity: str,
code: str,
message: str,
stage: str,
**details: Any,
) -> dict[str, Any]:
return {
"severity": severity,
"code": code,
"message": message,
"stage": stage,
"retryable": False,
"details": details,
}
def _validate_endpoint(
endpoint_url: str | None,
credential_ref: str | None,
) -> None:
if endpoint_url:
try:
ConnectorEndpoint(
url=endpoint_url,
credential_ref=_optional_text(credential_ref),
)
except ConnectorContractError as exc:
raise GovernedConnectorError("invalid_endpoint", str(exc)) from exc
elif credential_ref:
raise GovernedConnectorError(
"credential_without_endpoint",
"A credential reference requires a configured endpoint.",
)
def _stable_value(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
def _hash(value: Any) -> str:
return hashlib.sha256(
json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode()
).hexdigest()
def _optional_text(value: object | None) -> str | None:
normalized = str(value or "").strip()
return normalized or None
__all__ = [
"GovernedConnectorError",
"configuration_item",
"create_configuration",
"execute_run",
"list_configurations",
"list_definitions",
"list_runs",
"review_run",
"run_item",
"update_configuration",
"upsert_definition",
]
@@ -0,0 +1,221 @@
from __future__ import annotations
from datetime import datetime
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
class ConnectorMappingRule(BaseModel):
model_config = ConfigDict(extra="forbid")
source: str = Field(min_length=1, max_length=255)
target: str = Field(min_length=1, max_length=255)
required: bool = False
default: Any = None
class ConnectorMappingDefinition(BaseModel):
model_config = ConfigDict(extra="forbid")
version: str = Field(min_length=1, max_length=100)
rules: list[ConnectorMappingRule] = Field(default_factory=list, max_length=500)
class ConnectorValidationRule(BaseModel):
model_config = ConfigDict(extra="forbid")
kind: Literal["required", "one_of", "unique"]
field: str = Field(min_length=1, max_length=255)
values: list[Any] = Field(default_factory=list, max_length=500)
severity: Literal["warning", "error"] = "error"
code: str = Field(min_length=1, max_length=120)
message: str = Field(min_length=1, max_length=500)
@model_validator(mode="after")
def validate_values(self) -> "ConnectorValidationRule":
if self.kind == "one_of" and not self.values:
raise ValueError("one_of validation requires allowed values")
if self.kind != "one_of" and self.values:
raise ValueError("Only one_of validation accepts values")
return self
class ConnectorDryRunMetadata(BaseModel):
model_config = ConfigDict(extra="forbid")
supported: bool = True
simulation_supported: bool = True
sample_rows: list[dict[str, Any]] = Field(default_factory=list, max_length=500)
max_items: int = Field(default=500, ge=1, le=10_000)
redacted_fields: list[str] = Field(default_factory=list, max_length=100)
class ConnectorAuditMetadata(BaseModel):
model_config = ConfigDict(extra="forbid")
event_prefix: str = Field(min_length=1, max_length=120)
expected_events: list[str] = Field(default_factory=list, max_length=100)
evidence_fields: list[str] = Field(default_factory=list, max_length=100)
class GovernedConnectorSpecification(BaseModel):
model_config = ConfigDict(extra="forbid")
provider: str = Field(min_length=1, max_length=120)
protocol: str = Field(min_length=1, max_length=80)
capabilities: list[str] = Field(min_length=1, max_length=100)
input_schema: dict[str, Any]
output_schema: dict[str, Any]
mapping: ConnectorMappingDefinition
validation_rules: list[ConnectorValidationRule] = Field(
default_factory=list,
max_length=500,
)
dry_run: ConnectorDryRunMetadata
audit: ConnectorAuditMetadata
privacy_classification: Literal[
"public",
"internal",
"confidential",
"restricted",
] = "internal"
retention_class: str = Field(min_length=1, max_length=120)
operational_limits: dict[str, Any] = Field(default_factory=dict)
retry_policy: dict[str, Any] = Field(default_factory=dict)
class ConnectorDefinitionUpsertRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
definition_key: str = Field(pattern=r"^[a-z0-9][a-z0-9._-]{0,159}$")
name: str = Field(min_length=1, max_length=300)
description: str | None = Field(default=None, max_length=4000)
origin: Literal["package", "local"] = "local"
package_ref: str | None = Field(default=None, max_length=300)
specification: GovernedConnectorSpecification
@model_validator(mode="after")
def package_provenance(self) -> "ConnectorDefinitionUpsertRequest":
if self.origin == "package" and not self.package_ref:
raise ValueError("Package definitions require package_ref")
if self.origin == "local" and self.package_ref:
raise ValueError("Local definitions cannot claim package_ref")
return self
class ConnectorDefinitionItem(BaseModel):
id: str
tenant_id: str
definition_key: str
name: str
description: str | None = None
status: str
current_revision: int
source_package: str | None = None
local_definition: bool
revision_id: str
definition_hash: str
origin: str
package_ref: str | None = None
specification: GovernedConnectorSpecification
created_at: datetime
updated_at: datetime
class ConnectorDefinitionListResponse(BaseModel):
items: list[ConnectorDefinitionItem]
class ConnectorConfigurationCreateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
definition_id: str = Field(min_length=1, max_length=36)
name: str = Field(min_length=1, max_length=300)
endpoint_url: str | None = Field(default=None, max_length=1500)
credential_ref: str | None = Field(default=None, max_length=500)
local_overrides: dict[str, Any] = Field(default_factory=dict)
ambiguity_policy: Literal["manual_review", "quarantine", "reject"] = (
"manual_review"
)
status: Literal["draft", "active", "disabled"] = "draft"
class ConnectorConfigurationUpdateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
expected_revision: int = Field(ge=1)
name: str | None = Field(default=None, min_length=1, max_length=300)
endpoint_url: str | None = Field(default=None, max_length=1500)
credential_ref: str | None = Field(default=None, max_length=500)
local_overrides: dict[str, Any] | None = None
ambiguity_policy: Literal["manual_review", "quarantine", "reject"] | None = None
status: Literal["draft", "active", "disabled"] | None = None
adopt_latest_definition: bool = False
class ConnectorConfigurationItem(BaseModel):
id: str
tenant_id: str
definition_id: str
definition_key: str
definition_name: str
name: str
status: str
endpoint_url: str | None = None
credential_ref: str | None = None
base_definition_revision: int
latest_definition_revision: int
update_available: bool
local_overrides: dict[str, Any]
protected_paths: list[str]
effective_configuration: dict[str, Any]
effective_hash: str
resource_revision: int
ambiguity_policy: str
updated_at: datetime
class ConnectorConfigurationListResponse(BaseModel):
items: list[ConnectorConfigurationItem]
class ConnectorRunRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
idempotency_key: str = Field(min_length=1, max_length=255)
input_rows: list[dict[str, Any]] | None = Field(default=None, max_length=500)
external_revision: str | None = Field(default=None, max_length=255)
class ConnectorRunItem(BaseModel):
id: str
tenant_id: str
configuration_id: str
mode: Literal["dry_run", "simulation"]
idempotency_key: str
status: str
review_state: str
definition_revision: int
configuration_revision: int
configuration_hash: str
input_hash: str
summary: dict[str, Any]
effects: list[dict[str, Any]]
diagnostics: list[dict[str, Any]]
provenance: dict[str, Any]
reviewed_by: str | None = None
reviewed_at: datetime | None = None
review_reason: str | None = None
created_at: datetime
class ConnectorRunListResponse(BaseModel):
items: list[ConnectorRunItem]
class ConnectorReviewRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
decision: Literal["approved", "rejected"]
reason: str = Field(min_length=5, max_length=1000)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,303 @@
from __future__ import annotations
from datetime import datetime
from typing import Any, Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
KnowledgeMaturity = Literal[
"discover",
"link",
"search",
"read",
"publish",
"synchronize",
"migrate",
]
KnowledgeVisibility = Literal["tenant", "restricted"]
class KnowledgeNamespaceMapping(BaseModel):
model_config = ConfigDict(extra="forbid")
source_namespace_id: int
source_name: str = Field(default="", max_length=200)
target_space_ref: str = Field(min_length=1, max_length=255)
target_path_prefix: str = Field(default="", max_length=500)
include: bool = True
visibility: KnowledgeVisibility | None = None
acl_tokens: list[str] = Field(default_factory=list, max_length=500)
@field_validator("acl_tokens")
@classmethod
def normalize_acl_tokens(cls, values: list[str]) -> list[str]:
return _normalized_tokens(values)
@model_validator(mode="after")
def restricted_mapping_requires_acl(self) -> "KnowledgeNamespaceMapping":
if self.visibility == "restricted" and not self.acl_tokens:
raise ValueError("Restricted namespace mappings require ACL tokens")
return self
class KnowledgeProfileCreateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
configuration_id: str = Field(min_length=1, max_length=36)
desired_maturity: KnowledgeMaturity = "read"
source_authority_mode: Literal[
"external_authoritative",
"external_mirror",
"governed_sync",
"linked_reference",
] = "external_mirror"
default_visibility: KnowledgeVisibility = "restricted"
default_acl_tokens: list[str] = Field(default_factory=list, max_length=500)
namespace_mappings: list[KnowledgeNamespaceMapping] = Field(
min_length=1, max_length=500
)
@field_validator("default_acl_tokens")
@classmethod
def normalize_acl_tokens(cls, values: list[str]) -> list[str]:
return _normalized_tokens(values)
@model_validator(mode="after")
def restricted_profile_requires_acl(self) -> "KnowledgeProfileCreateRequest":
if self.default_visibility == "restricted" and not self.default_acl_tokens:
raise ValueError("Restricted knowledge profiles require default ACL tokens")
return self
class KnowledgeProfileUpdateRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
expected_resource_revision: int = Field(ge=1)
status: Literal["active", "paused"] | None = None
desired_maturity: KnowledgeMaturity | None = None
source_authority_mode: Literal[
"external_authoritative",
"external_mirror",
"governed_sync",
"linked_reference",
] | None = None
default_visibility: KnowledgeVisibility | None = None
default_acl_tokens: list[str] | None = Field(default=None, max_length=500)
namespace_mappings: list[KnowledgeNamespaceMapping] | None = Field(
default=None, min_length=1, max_length=500
)
@field_validator("default_acl_tokens")
@classmethod
def normalize_acl_tokens(cls, values: list[str] | None) -> list[str] | None:
return _normalized_tokens(values) if values is not None else None
class KnowledgeProfileItem(BaseModel):
id: str
tenant_id: str
configuration_id: str
status: str
product: str
product_version: str | None = None
desired_maturity: str
discovered_maturity: str
source_authority_mode: str
default_visibility: str
default_acl_tokens: list[str]
namespace_mappings: list[dict[str, Any]]
capabilities: list[str]
discovery_revision: str | None = None
health_status: str
health_details: dict[str, Any]
discovered_at: datetime | None = None
last_sync_cursor: str | None = None
last_high_watermark: str | None = None
resource_revision: int
credential_reference_present: bool = False
endpoint_configured: bool = False
created_at: datetime
updated_at: datetime
class KnowledgeProfileListResponse(BaseModel):
items: list[KnowledgeProfileItem]
class KnowledgeDiagnostic(BaseModel):
severity: Literal["info", "warning", "error"]
code: str
message: str
object_ref: str | None = None
field: str | None = None
retryable: bool = False
details: dict[str, Any] = Field(default_factory=dict)
class KnowledgeDiscoveryResponse(BaseModel):
profile: KnowledgeProfileItem
product: str
product_version: str | None = None
api_version: str | None = None
capabilities: list[str]
namespaces: list[dict[str, Any]]
extensions: list[dict[str, Any]]
maturity: str
health_status: str
diagnostics: list[KnowledgeDiagnostic]
revision: str
class KnowledgeSyncRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
idempotency_key: str = Field(min_length=1, max_length=255)
cursor: str | None = Field(default=None, max_length=500)
force_full: bool = False
limit: int = Field(default=100, ge=1, le=500)
class KnowledgeSyncRunItem(BaseModel):
id: str
tenant_id: str
profile_id: str
mode: str
idempotency_key: str
status: str
cursor_before: str | None = None
cursor_after: str | None = None
high_watermark: str | None = None
counts: dict[str, Any]
effects: list[dict[str, Any]]
diagnostics: list[KnowledgeDiagnostic]
provenance: dict[str, Any]
started_at: datetime
finished_at: datetime | None = None
created_at: datetime
class KnowledgeSyncRunListResponse(BaseModel):
items: list[KnowledgeSyncRunItem]
class KnowledgeExternalReferenceResponse(BaseModel):
system: str
object_type: str
object_id: str
maturity: str
authority_mode: str
connector_id: str | None = None
canonical_url: str | None = None
version: str | None = None
etag: str | None = None
observed_at: str | None = None
metadata: dict[str, Any]
class KnowledgeObjectItem(BaseModel):
id: str
profile_id: str
object_type: str
external_id: str
external_page_id: str | None = None
external_revision_id: str | None = None
namespace_id: int | None = None
title: str
canonical_url: str | None = None
status: str
redirect_target_external_id: str | None = None
source_revision: str
visibility: str
acl_tokens: list[str]
mapped_data: dict[str, Any]
external_reference: KnowledgeExternalReferenceResponse
source_updated_at: datetime | None = None
observed_at: datetime
resource_revision: int
class KnowledgeObjectListResponse(BaseModel):
items: list[KnowledgeObjectItem]
next_cursor: str | None = None
class KnowledgeMigrationTargetState(BaseModel):
model_config = ConfigDict(extra="forbid")
path: str = Field(min_length=1, max_length=500)
source_external_id: str | None = Field(default=None, max_length=255)
attachment_names: list[str] = Field(default_factory=list, max_length=500)
class KnowledgeMigrationDryRunRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
idempotency_key: str = Field(min_length=1, max_length=255)
target_space_ref: str = Field(min_length=1, max_length=255)
max_items: int = Field(default=100, ge=1, le=500)
supported_macros: list[str] = Field(default_factory=list, max_length=200)
existing_targets: list[KnowledgeMigrationTargetState] = Field(
default_factory=list, max_length=5_000
)
class KnowledgeMigrationDryRunResponse(BaseModel):
run: KnowledgeSyncRunItem
target_space_ref: str
source_revision: str
source_fingerprint: str
summary: dict[str, int]
effects: list[dict[str, Any]]
diagnostics: list[KnowledgeDiagnostic]
truncated: bool
can_apply: bool
class KnowledgePublishRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
idempotency_key: str = Field(min_length=1, max_length=255)
title: str = Field(min_length=1, max_length=500)
body: str = Field(max_length=200_000)
summary: str = Field(default="", max_length=500)
expected_external_revision: str | None = Field(default=None, max_length=255)
minor: bool = False
class KnowledgePublishResponse(BaseModel):
run: KnowledgeSyncRunItem
external_reference: KnowledgeExternalReferenceResponse
accepted: bool
outcome_unknown: bool = False
def _normalized_tokens(values: list[str]) -> list[str]:
normalized = [str(item).strip() for item in values]
if any(not item or len(item) > 500 for item in normalized):
raise ValueError("ACL tokens must contain 1 to 500 characters")
if len(normalized) != len(set(normalized)):
raise ValueError("ACL tokens must be unique")
return normalized
__all__ = [
"KnowledgeDiagnostic",
"KnowledgeDiscoveryResponse",
"KnowledgeExternalReferenceResponse",
"KnowledgeMigrationDryRunRequest",
"KnowledgeMigrationDryRunResponse",
"KnowledgeMigrationTargetState",
"KnowledgeNamespaceMapping",
"KnowledgeObjectItem",
"KnowledgeObjectListResponse",
"KnowledgeProfileCreateRequest",
"KnowledgeProfileItem",
"KnowledgeProfileListResponse",
"KnowledgeProfileUpdateRequest",
"KnowledgePublishRequest",
"KnowledgePublishResponse",
"KnowledgeSyncRequest",
"KnowledgeSyncRunItem",
"KnowledgeSyncRunListResponse",
]
@@ -0,0 +1,269 @@
from __future__ import annotations
from collections.abc import Mapping, Sequence
from urllib.parse import quote
from sqlalchemy import func, select
from sqlalchemy.orm import Session
from govoplan_core.core.external_references import ExternalObjectReference
from govoplan_core.core.modules import ModuleContext
from govoplan_core.core.search import (
SearchAuthorizationRequest,
SearchBackfillPage,
SearchBackfillRequest,
SearchDocument,
SearchResourceType,
)
from govoplan_connectors.backend.db.models import (
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
)
from govoplan_connectors.backend.knowledge_connector import (
KNOWLEDGE_PROVIDER_ID,
KNOWLEDGE_READ_SCOPE,
KNOWLEDGE_RESOURCE_TYPE,
)
class ExternalKnowledgeSearchSource:
def resource_types(self) -> Sequence[SearchResourceType]:
return (
SearchResourceType(
provider_id=KNOWLEDGE_PROVIDER_ID,
module_id="connectors",
resource_type=KNOWLEDGE_RESOURCE_TYPE,
label="External knowledge pages",
requires_authorization_recheck=True,
),
)
def backfill(
self, session: object, *, request: SearchBackfillRequest
) -> SearchBackfillPage:
_assert_source(request.provider_id, request.resource_type)
db = _session(session)
query = (
select(ConnectorKnowledgeObject, ConnectorKnowledgeProfile)
.join(
ConnectorKnowledgeProfile,
ConnectorKnowledgeProfile.id == ConnectorKnowledgeObject.profile_id,
)
.where(
ConnectorKnowledgeObject.tenant_id == request.tenant_id,
ConnectorKnowledgeObject.object_type == "page",
ConnectorKnowledgeObject.status != "deleted",
ConnectorKnowledgeProfile.tenant_id == request.tenant_id,
ConnectorKnowledgeProfile.status == "active",
)
)
if request.cursor:
query = query.where(ConnectorKnowledgeObject.id > request.cursor)
rows = tuple(
db.execute(
query.order_by(ConnectorKnowledgeObject.id.asc()).limit(
request.limit + 1
)
)
)
has_more = len(rows) > request.limit
selected = rows[: request.limit]
watermark = db.scalar(
select(func.max(ConnectorKnowledgeObject.updated_at))
.join(
ConnectorKnowledgeProfile,
ConnectorKnowledgeProfile.id == ConnectorKnowledgeObject.profile_id,
)
.where(
ConnectorKnowledgeObject.tenant_id == request.tenant_id,
ConnectorKnowledgeObject.object_type == "page",
ConnectorKnowledgeObject.status != "deleted",
ConnectorKnowledgeProfile.tenant_id == request.tenant_id,
ConnectorKnowledgeProfile.status == "active",
)
)
return SearchBackfillPage(
documents=tuple(search_document(db, row, profile) for row, profile in selected),
next_cursor=selected[-1][0].id if has_more and selected else None,
complete=not has_more,
high_watermark=watermark.isoformat() if watermark else None,
)
def authorize(
self,
session: object,
principal: object,
*,
requests: Sequence[SearchAuthorizationRequest],
) -> Mapping[str, bool]:
db = _session(session)
tenant_id = str(getattr(principal, "tenant_id", "") or "")
can_read = _has_scope(principal, KNOWLEDGE_READ_SCOPE)
tokens = set(_principal_acl_tokens(principal))
decisions = {item.reference.key: False for item in requests}
if not tenant_id or not can_read:
return decisions
for request in requests:
reference = request.reference
if (
reference.tenant_id != tenant_id
or reference.module_id != "connectors"
or reference.resource_type != KNOWLEDGE_RESOURCE_TYPE
):
continue
row = db.execute(
select(ConnectorKnowledgeObject, ConnectorKnowledgeProfile)
.join(
ConnectorKnowledgeProfile,
ConnectorKnowledgeProfile.id
== ConnectorKnowledgeObject.profile_id,
)
.where(
ConnectorKnowledgeObject.tenant_id == tenant_id,
ConnectorKnowledgeObject.id == reference.resource_id,
ConnectorKnowledgeObject.object_type == "page",
ConnectorKnowledgeObject.status != "deleted",
ConnectorKnowledgeProfile.tenant_id == tenant_id,
ConnectorKnowledgeProfile.status == "active",
)
).first()
if row is None:
continue
page, _profile = row
decisions[reference.key] = page.visibility == "tenant" or bool(
tokens.intersection(str(value) for value in page.acl_tokens or ())
)
return decisions
def create_external_knowledge_search_source(
_context: ModuleContext,
) -> ExternalKnowledgeSearchSource:
return ExternalKnowledgeSearchSource()
def search_document(
session: Session,
row: ConnectorKnowledgeObject,
profile: ConnectorKnowledgeProfile | None = None,
) -> SearchDocument:
if profile is None:
profile = session.scalar(
select(ConnectorKnowledgeProfile).where(
ConnectorKnowledgeProfile.tenant_id == row.tenant_id,
ConnectorKnowledgeProfile.id == row.profile_id,
)
)
if profile is None:
raise ValueError("External knowledge profile is unavailable.")
data = dict(row.mapped_data or {})
categories = tuple(str(value)[:200] for value in data.get("categories") or ())
links = tuple(
str(value.get("title") or value.get("external_id") or "")[:200]
for value in data.get("links") or ()
if isinstance(value, Mapping)
)
external_reference = ExternalObjectReference(
system=profile.product if profile.product != "unknown" else "mediawiki",
object_type=row.object_type,
object_id=row.external_id,
maturity=profile.discovered_maturity,
authority_mode=profile.source_authority_mode,
connector_id=profile.id,
canonical_url=row.canonical_url,
version=row.source_revision,
etag=row.content_hash,
observed_at=row.observed_at,
metadata={
"title": row.title,
"namespace_id": row.namespace_id,
"external_revision_id": row.external_revision_id,
},
)
return SearchDocument(
tenant_id=row.tenant_id,
module_id="connectors",
provider_id=KNOWLEDGE_PROVIDER_ID,
resource_type=KNOWLEDGE_RESOURCE_TYPE,
resource_id=row.id,
title=row.title,
url=(
"/connectors/knowledge?profileId="
f"{quote(row.profile_id, safe='')}&objectId={quote(row.id, safe='')}"
),
summary=str(data.get("summary") or data.get("body") or "")[:4_000] or None,
body=str(data.get("body") or "")[:200_000] or None,
keywords=tuple(dict.fromkeys((*categories, *links)))[:100],
visibility=row.visibility,
acl_tokens=(
tuple(str(value) for value in row.acl_tokens or ())
if row.visibility == "restricted"
else ()
),
external_reference=external_reference,
metadata={
"profile_id": row.profile_id,
"external_page_id": row.external_page_id,
"namespace_id": row.namespace_id,
"target_space_ref": data.get("target_space_ref"),
"target_path": data.get("target_path"),
"status": row.status,
"redirect_target_external_id": row.redirect_target_external_id,
},
source_revision=row.source_revision,
change_cursor=row.change_cursor,
source_updated_at=row.source_updated_at or row.observed_at,
requires_authorization_recheck=True,
)
def _principal_acl_tokens(principal: object) -> tuple[str, ...]:
values: list[str] = []
for prefix, attribute in (
("account", "account_id"),
("membership", "membership_id"),
("identity", "identity_id"),
):
value = getattr(principal, attribute, None)
if value:
values.append(f"{prefix}:{value}")
for prefix, attribute in (
("group", "group_ids"),
("role", "role_ids"),
("function", "function_assignment_ids"),
("scope", "scopes"),
):
values.extend(
f"{prefix}:{value}"
for value in getattr(principal, attribute, ())
if value
)
return tuple(dict.fromkeys(values))[:500]
def _has_scope(principal: object, required: str) -> bool:
check = getattr(principal, "has", None)
if callable(check):
return bool(check(required))
return required in getattr(principal, "scopes", ())
def _assert_source(provider_id: str, resource_type: str) -> None:
if (
provider_id != KNOWLEDGE_PROVIDER_ID
or resource_type != KNOWLEDGE_RESOURCE_TYPE
):
raise ValueError("Unsupported external knowledge search source.")
def _session(value: object) -> Session:
if not isinstance(value, Session):
raise TypeError("External knowledge Search requires a SQLAlchemy session.")
return value
__all__ = [
"ExternalKnowledgeSearchSource",
"create_external_knowledge_search_source",
"search_document",
]
+483 -26
View File
@@ -13,12 +13,16 @@ from govoplan_core.core.module_guards import (
from govoplan_core.core.datasources import CAPABILITY_DATASOURCE_ORIGINS from govoplan_core.core.datasources import CAPABILITY_DATASOURCE_ORIGINS
from govoplan_core.core.feeds import CAPABILITY_CONNECTORS_FEEDS from govoplan_core.core.feeds import CAPABILITY_CONNECTORS_FEEDS
from govoplan_core.core.modules import ( from govoplan_core.core.modules import (
CapabilityDocumentation,
DocumentationTopic, DocumentationTopic,
FrontendModule,
MigrationSpec, MigrationSpec,
ModuleInterfaceProvider, ModuleInterfaceProvider,
ModuleInterfaceRequirement,
ModuleManifest, ModuleManifest,
PermissionDefinition, PermissionDefinition,
RoleTemplate, RoleTemplate,
ViewSurface,
) )
from govoplan_core.core.provider_governance import ( from govoplan_core.core.provider_governance import (
ExternalProviderDeclaration, ExternalProviderDeclaration,
@@ -33,15 +37,42 @@ from govoplan_core.core.tabular_sources import (
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER, CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER,
CAPABILITY_CONNECTORS_TABULAR_SOURCES, CAPABILITY_CONNECTORS_TABULAR_SOURCES,
) )
from govoplan_core.core.files import CAPABILITY_FILES_TABULAR_CONTENT
from govoplan_core.core.sanctions import ( from govoplan_core.core.sanctions import (
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS, CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
) )
from govoplan_core.core.search import SearchSourceProviderRegistration
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
from govoplan_connectors.backend.db.models import ( from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
ConnectorSanctionsAcquisitionRun, ConnectorSanctionsAcquisitionRun,
ConnectorSanctionsSnapshot, ConnectorSanctionsSnapshot,
ConnectorSimulationRun,
ConnectorTabularSource, ConnectorTabularSource,
) )
from govoplan_connectors.backend.knowledge_connector import (
KNOWLEDGE_ADMIN_SCOPE,
KNOWLEDGE_CAPABILITY,
KNOWLEDGE_INTERFACE_VERSION,
KNOWLEDGE_MIGRATE_SCOPE,
KNOWLEDGE_PROVIDER_ID,
KNOWLEDGE_PUBLISH_SCOPE,
KNOWLEDGE_READ_SCOPE,
KNOWLEDGE_SYNC_SCOPE,
ExternalKnowledgeCapability,
)
from govoplan_connectors.backend.knowledge_search import (
create_external_knowledge_search_source,
)
from govoplan_connectors.backend.dsar_provider import (
CONNECTORS_DSAR_CAPABILITY,
ConnectorsDsarProvider,
)
from govoplan_connectors.backend.sanctions_sources import ( from govoplan_connectors.backend.sanctions_sources import (
SANCTIONS_READ_SCOPE, SANCTIONS_READ_SCOPE,
SANCTIONS_REFRESH_SCOPE, SANCTIONS_REFRESH_SCOPE,
@@ -56,21 +87,26 @@ from govoplan_connectors.backend.tabular_sources import (
from govoplan_connectors.backend.datasource_origins import ( from govoplan_connectors.backend.datasource_origins import (
ConnectorDatasourceOriginProvider, ConnectorDatasourceOriginProvider,
) )
from govoplan_connectors.backend.feeds import ConnectorFeedProvider from govoplan_connectors.backend.feeds import (
FEED_PRIVATE_PUBLISH_SCOPE,
FEED_PUBLISH_SCOPE,
ConnectorFeedProvider,
)
from govoplan_connectors.backend.provider_state import ( from govoplan_connectors.backend.provider_state import (
SANCTIONS_PROVIDER_ID, SANCTIONS_PROVIDER_ID,
TABULAR_PROVIDER_ID, TABULAR_PROVIDER_ID,
knowledge_provider_states,
sanctions_provider_states, sanctions_provider_states,
tabular_provider_states, tabular_provider_states,
) )
MODULE_ID = "connectors" MODULE_ID = "connectors"
MODULE_VERSION = "0.1.15" MODULE_VERSION = "0.1.21"
TABULAR_SOURCE_INTERFACE_VERSION = "0.1.0" TABULAR_SOURCE_INTERFACE_VERSION = "0.1.0"
DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0" DATASOURCE_ORIGIN_INTERFACE_VERSION = "0.1.0"
SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0" SANCTIONS_SNAPSHOT_INTERFACE_VERSION = "1.0.0"
FEED_INTERFACE_VERSION = "0.1.0" FEED_INTERFACE_VERSION = "0.2.0"
CONNECTOR_RUNTIME_INTERFACE_VERSION = "1.0.0" CONNECTOR_RUNTIME_INTERFACE_VERSION = "1.0.0"
ARCHITECTURE = ModuleArchitectureDeclaration( ARCHITECTURE = ModuleArchitectureDeclaration(
@@ -83,6 +119,16 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
reference="tests/test_tabular_sources.py", reference="tests/test_tabular_sources.py",
summary="Exercises tenant-safe immutable tabular snapshots and bounded reads.", summary="Exercises tenant-safe immutable tabular snapshots and bounded reads.",
), ),
ModuleMaturityEvidence(
kind="test",
reference="tests/test_tabular_origin_provider.py",
summary="Exercises exact managed-file versions, reviewed refresh, live SQL projection, configuration drift, and tenant isolation.",
),
ModuleMaturityEvidence(
kind="test",
reference="tests/test_tabular_adapters.py",
summary="Exercises bounded CSV/XLSX parsing and the credential-governed read-only PostgreSQL adapter.",
),
ModuleMaturityEvidence( ModuleMaturityEvidence(
kind="test", kind="test",
reference="tests/test_sanctions_sources.py", reference="tests/test_sanctions_sources.py",
@@ -93,6 +139,16 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
reference="tests/test_recovery.py", reference="tests/test_recovery.py",
summary="Proves atomic snapshot commits, idempotent replay, distributed fences, tamper rejection, and unknown external-effect handling.", summary="Proves atomic snapshot commits, idempotent replay, distributed fences, tamper rejection, and unknown external-effect handling.",
), ),
ModuleMaturityEvidence(
kind="test",
reference="tests/test_governed_runtime.py",
summary="Exercises immutable definition revisions, protected local overrides, idempotent simulations, and explicit ambiguity review.",
),
ModuleMaturityEvidence(
kind="test",
reference="tests/test_mediawiki_connector.py",
summary="Exercises deterministic MediaWiki/BlueSpice discovery, stable mapping, bounded deltas, ACL-safe Search, migration loss diagnostics, and publication recovery states.",
),
ModuleMaturityEvidence( ModuleMaturityEvidence(
kind="documentation", kind="documentation",
reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md", reference="docs/CONNECTOR_SOURCE_LIFECYCLE.md",
@@ -100,12 +156,15 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
), ),
), ),
known_limits=( known_limits=(
"The executable generic datasource origin is an immutable tabular snapshot; database and arbitrary REST profiles remain future providers.", "Tabular origins support immutable snapshots, exact managed CSV/XLSX versions, and read-only PostgreSQL tables; arbitrary REST and other database adapters remain future providers.",
"Feed publication renders a governed document but does not yet push it to an external publishing endpoint.", "Feed publication renders a governed document but does not yet push it to an external publishing endpoint.",
"The MediaWiki/BlueSpice adapter publishes revision-checked page edits; generic simulations and all other providers do not imply a live write capability.",
"Migration into native Wiki is preview-only; a target-side write worker and Wiki-owned mutation contract remain future work.",
), ),
supported_authority_modes=( supported_authority_modes=(
"external_authoritative", "external_authoritative",
"external_mirror", "external_mirror",
"governed_sync",
"linked_reference", "linked_reference",
), ),
owned_concepts=( owned_concepts=(
@@ -113,23 +172,38 @@ ARCHITECTURE = ModuleArchitectureDeclaration(
"protocol interaction", "protocol interaction",
"immutable connector snapshots", "immutable connector snapshots",
"connector acquisition health", "connector acquisition health",
"external knowledge synchronization evidence",
), ),
non_owned_concepts=( non_owned_concepts=(
"datasource catalogue identity and lifecycle", "datasource catalogue identity and lifecycle",
"domain records and business semantics", "domain records and business semantics",
"data transformations", "data transformations",
"screening dispositions", "screening dispositions",
"native Wiki spaces, pages, and revision semantics",
), ),
target_tested_providers=( target_tested_providers=(
TABULAR_PROVIDER_ID, TABULAR_PROVIDER_ID,
SANCTIONS_PROVIDER_ID, SANCTIONS_PROVIDER_ID,
KNOWLEDGE_PROVIDER_ID,
), ),
documentation=ModuleArchitectureDocumentation( documentation=ModuleArchitectureDocumentation(
migration=("src/govoplan_connectors/backend/migrations/versions",), migration=("src/govoplan_connectors/backend/migrations/versions",),
upgrade=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",), upgrade=(
recovery=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",), "docs/CONNECTOR_SOURCE_LIFECYCLE.md",
security=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",), "docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
operations=("docs/CONNECTOR_SOURCE_LIFECYCLE.md",), ),
recovery=(
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
),
security=(
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
),
operations=(
"docs/CONNECTOR_SOURCE_LIFECYCLE.md",
"docs/MEDIAWIKI_BLUESPICE_CONNECTOR.md",
),
), ),
) )
@@ -137,7 +211,7 @@ EXTERNAL_PROVIDERS = (
ExternalProviderDeclaration( ExternalProviderDeclaration(
id=TABULAR_PROVIDER_ID, id=TABULAR_PROVIDER_ID,
module_id=MODULE_ID, module_id=MODULE_ID,
label="Immutable tabular snapshot provider", label="Governed tabular source providers",
maturity="read", maturity="read",
operations=("discover", "search", "read", "preview", "dry_run"), operations=("discover", "search", "read", "preview", "dry_run"),
objects=( objects=(
@@ -149,10 +223,10 @@ EXTERNAL_PROVIDERS = (
), ),
), ),
behavior=ProviderBehaviorDeclaration( behavior=ProviderBehaviorDeclaration(
revision_tokens="Source fingerprints and immutable snapshot ids are retained.", revision_tokens="Source fingerprints, exact file versions, SQL configuration revisions, credential revisions, and discovery revisions are retained.",
concurrency="Reads may require the expected fingerprint; snapshots never mutate in place.", concurrency="Reads may require the expected fingerprint; file versions remain pinned and SQL configuration or schema drift blocks reads until reviewed refresh.",
freshness="Snapshot acquisition time and source timestamp are exposed.", freshness="Snapshot acquisition time, exact/current file versions, and live SQL discovery health are exposed.",
health="Import validation and source-read failures are explicit.", health="Import validation, Files access/integrity, SQL authentication, configuration drift, schema drift, timeouts, and source unavailability are explicit without exposing secrets.",
max_read_items=1000, max_read_items=1000,
idempotency="Feed imports accept a caller request key and replay the same committed immutable source without refetching.", idempotency="Feed imports accept a caller request key and replay the same committed immutable source without refetching.",
retry="Read-only acquisition may be retried only as a new deliberate request after a failed atomic operation.", retry="Read-only acquisition may be retried only as a new deliberate request after a failed atomic operation.",
@@ -166,7 +240,7 @@ EXTERNAL_PROVIDERS = (
classifications=("internal", "confidential", "restricted"), classifications=("internal", "confidential", "restricted"),
purposes=("governed import", "dataflow input", "evidence reconstruction"), purposes=("governed import", "dataflow input", "evidence reconstruction"),
retention="Datasources or the consuming domain supplies retention and hold policy.", retention="Datasources or the consuming domain supplies retention and hold policy.",
secret_handling="Generic snapshots contain no connector credential; transport credentials stay in credential envelopes.", secret_handling="Snapshots and managed-file sources contain no connector credential; PostgreSQL uses a scoped Core credential-envelope reference and never stores or returns the resolved secret.",
), ),
capability_names=( capability_names=(
CAPABILITY_CONNECTORS_TABULAR_SOURCES, CAPABILITY_CONNECTORS_TABULAR_SOURCES,
@@ -190,7 +264,12 @@ EXTERNAL_PROVIDERS = (
objects=( objects=(
ProviderObjectDeclaration( ProviderObjectDeclaration(
object_type="sanctions_source_snapshot", object_type="sanctions_source_snapshot",
field_groups=("source_identity", "raw_evidence", "entries", "acquisition_health"), field_groups=(
"source_identity",
"raw_evidence",
"entries",
"acquisition_health",
),
authority_modes=("external_authoritative", "external_mirror"), authority_modes=("external_authoritative", "external_mirror"),
default_authority_mode="external_mirror", default_authority_mode="external_mirror",
), ),
@@ -223,6 +302,88 @@ EXTERNAL_PROVIDERS = (
"connectors.sanctions-snapshots", "connectors.sanctions-snapshots",
), ),
), ),
ExternalProviderDeclaration(
id=KNOWLEDGE_PROVIDER_ID,
module_id=MODULE_ID,
label="MediaWiki and BlueSpice knowledge provider",
maturity="migrate",
operations=(
"discover",
"link",
"search",
"read",
"publish",
"synchronize",
"migrate",
"preview",
"dry_run",
),
objects=(
ProviderObjectDeclaration(
object_type="external_knowledge_page",
field_groups=(
"stable_identity",
"revisions",
"namespaces",
"categories",
"links",
"files",
"discussions",
"permissions",
"source_provenance",
),
authority_modes=(
"external_authoritative",
"external_mirror",
"governed_sync",
"linked_reference",
),
default_authority_mode="external_mirror",
),
),
behavior=ProviderBehaviorDeclaration(
revision_tokens="Stable MediaWiki page ids, revision ids, discovery revisions, recent-change cursors, content digests, and provider timestamps are retained.",
concurrency="Profile changes use optimistic revisions; publication supplies an expected provider revision and a durable idempotency fence.",
freshness="The latest discovery time, sync high-watermark, recent-change cursor, source update time, and health state are exposed.",
health="Product discovery, authentication, transport, mapping loss, ACL degradation, Search deferral, and outcome-unknown publication are explicit without exposing credentials.",
max_read_items=500,
idempotency="Backfill, delta synchronization, migration previews, and publication require caller keys; exact replays return the committed evidence and mismatched reuse is rejected.",
retry="Read-only discovery and synchronization may be retried with a new key; a publication with an unknown outcome must be reconciled before retry.",
timeout_seconds=30,
conflicts="Namespace and path mappings are explicit; migration previews block target-path and attachment conflicts, while publication rejects stale expected revisions.",
outcome_unknown="A timed-out publication remains outcome-unknown behind a durable recovery fence until the institutional operator verifies the provider revision.",
outcome_unknown_supported=True,
evidence="Mapped snapshots retain stable external references, revision identity, source hashes, transport provenance, ACLs, and bounded loss diagnostics.",
audit_event_types=(
"connectors.knowledge.profile.created",
"connectors.knowledge.profile.updated",
"connectors.knowledge.profile.discovered",
"connectors.knowledge.profile.synchronized",
"connectors.knowledge.migration.previewed",
"connectors.knowledge.page.published",
),
correction="A later provider revision or explicit publication creates a new connector snapshot revision while synchronization-run evidence remains retained.",
rollback="Local snapshot and terminal recovery evidence commit atomically; remote publication cannot be rolled back by a local transaction.",
reconciliation="Rediscover capabilities, compare stable page and revision ids, run a bounded delta or full backfill, and inspect unresolved publication evidence.",
outage="Existing snapshots remain visible only through current tenant and ACL authorization and are marked stale or unavailable; no provider freshness claim is made.",
classifications=("public", "internal", "confidential", "restricted"),
purposes=(
"external knowledge discovery",
"authorized federated search",
"knowledge synchronization",
"migration planning",
"governed publication",
),
retention="The tenant's connector, Records, and target Wiki policies determine snapshot and operation-evidence retention.",
secret_handling="Credentials are resolved from a scoped Core credential envelope, never placed in endpoint URLs, persisted snapshots, diagnostics, or API responses.",
),
capability_names=(KNOWLEDGE_CAPABILITY,),
interface_names=(KNOWLEDGE_CAPABILITY,),
documentation_topic_ids=(
"connectors.authority-and-effects",
"connectors.mediawiki-bluespice",
),
),
) )
@@ -254,7 +415,17 @@ PERMISSIONS = (
_permission( _permission(
ADMIN_SCOPE, ADMIN_SCOPE,
"Administer connector sources", "Administer connector sources",
"Manage every tenant connector source and future source policies.", "Manage tenant connector sources, versioned definitions, protected overrides, and review policies.",
),
_permission(
FEED_PUBLISH_SCOPE,
"Publish public feeds",
"Render provenance-bearing public RSS or Atom output from selected GovOPlaN objects.",
),
_permission(
FEED_PRIVATE_PUBLISH_SCOPE,
"Publish restricted feeds",
"Render tenant or private RSS or Atom output after the owning product surface has authorized every selected object.",
), ),
_permission( _permission(
SANCTIONS_READ_SCOPE, SANCTIONS_READ_SCOPE,
@@ -266,9 +437,53 @@ PERMISSIONS = (
"Refresh sanctions sources", "Refresh sanctions sources",
"Acquire a new immutable sanctions source snapshot.", "Acquire a new immutable sanctions source snapshot.",
), ),
_permission(
KNOWLEDGE_READ_SCOPE,
"View external knowledge",
"View authorized MediaWiki and BlueSpice profiles, pages, and synchronization evidence.",
),
_permission(
KNOWLEDGE_ADMIN_SCOPE,
"Administer external knowledge",
"Configure knowledge profiles, namespace mappings, authority, visibility, and discovery.",
),
_permission(
KNOWLEDGE_SYNC_SCOPE,
"Synchronize external knowledge",
"Run bounded MediaWiki and BlueSpice backfills and change synchronization.",
),
_permission(
KNOWLEDGE_PUBLISH_SCOPE,
"Publish external knowledge",
"Publish a governed page revision with concurrency and recovery evidence.",
),
_permission(
KNOWLEDGE_MIGRATE_SCOPE,
"Preview knowledge migration",
"Dry-run a bounded migration into Wiki and inspect loss or conflict diagnostics.",
),
) )
ROLE_TEMPLATES = ( ROLE_TEMPLATES = (
RoleTemplate(
slug="connector_administrator",
name="Connector administrator",
description="Govern connector definitions, local configurations, simulations, and manual review.",
permissions=(
READ_SCOPE,
WRITE_SCOPE,
ADMIN_SCOPE,
FEED_PUBLISH_SCOPE,
FEED_PRIVATE_PUBLISH_SCOPE,
SANCTIONS_READ_SCOPE,
SANCTIONS_REFRESH_SCOPE,
KNOWLEDGE_READ_SCOPE,
KNOWLEDGE_ADMIN_SCOPE,
KNOWLEDGE_SYNC_SCOPE,
KNOWLEDGE_PUBLISH_SCOPE,
KNOWLEDGE_MIGRATE_SCOPE,
),
),
RoleTemplate( RoleTemplate(
slug="connector_source_manager", slug="connector_source_manager",
name="Connector source manager", name="Connector source manager",
@@ -276,15 +491,19 @@ ROLE_TEMPLATES = (
permissions=( permissions=(
READ_SCOPE, READ_SCOPE,
WRITE_SCOPE, WRITE_SCOPE,
FEED_PUBLISH_SCOPE,
SANCTIONS_READ_SCOPE, SANCTIONS_READ_SCOPE,
SANCTIONS_REFRESH_SCOPE, SANCTIONS_REFRESH_SCOPE,
KNOWLEDGE_READ_SCOPE,
KNOWLEDGE_SYNC_SCOPE,
KNOWLEDGE_MIGRATE_SCOPE,
), ),
), ),
RoleTemplate( RoleTemplate(
slug="connector_source_reader", slug="connector_source_reader",
name="Connector source reader", name="Connector source reader",
description="Discover and preview tabular connector sources.", description="Discover and preview tabular connector sources.",
permissions=(READ_SCOPE, SANCTIONS_READ_SCOPE), permissions=(READ_SCOPE, SANCTIONS_READ_SCOPE, KNOWLEDGE_READ_SCOPE),
), ),
) )
@@ -296,11 +515,13 @@ def _router(_context):
def _provider(_context) -> SqlTabularSourceProvider: def _provider(_context) -> SqlTabularSourceProvider:
return SqlTabularSourceProvider() return SqlTabularSourceProvider(registry=getattr(_context, "registry", None))
def _datasource_origin_provider(_context) -> ConnectorDatasourceOriginProvider: def _datasource_origin_provider(_context) -> ConnectorDatasourceOriginProvider:
return ConnectorDatasourceOriginProvider() return ConnectorDatasourceOriginProvider(
SqlTabularSourceProvider(registry=getattr(_context, "registry", None))
)
def _sanctions_snapshot_provider( def _sanctions_snapshot_provider(
@@ -313,8 +534,31 @@ def _feed_provider(_context) -> ConnectorFeedProvider:
return ConnectorFeedProvider() return ConnectorFeedProvider()
def _dsar_provider(_context) -> ConnectorsDsarProvider:
return ConnectorsDsarProvider()
def _knowledge_provider(_context) -> ExternalKnowledgeCapability:
return ExternalKnowledgeCapability()
def _tenant_summary(session, tenant_id: str) -> dict[str, int]: def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
return { return {
"connector_definitions": (
session.query(ConnectorDefinition)
.filter(ConnectorDefinition.tenant_id == tenant_id)
.count()
),
"connector_configurations": (
session.query(ConnectorConfiguration)
.filter(ConnectorConfiguration.tenant_id == tenant_id)
.count()
),
"connector_simulation_runs": (
session.query(ConnectorSimulationRun)
.filter(ConnectorSimulationRun.tenant_id == tenant_id)
.count()
),
"connector_tabular_sources": ( "connector_tabular_sources": (
session.query(ConnectorTabularSource) session.query(ConnectorTabularSource)
.filter( .filter(
@@ -333,6 +577,24 @@ def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
.filter(ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id) .filter(ConnectorSanctionsAcquisitionRun.tenant_id == tenant_id)
.count() .count()
), ),
"connector_knowledge_profiles": (
session.query(ConnectorKnowledgeProfile)
.filter(ConnectorKnowledgeProfile.tenant_id == tenant_id)
.count()
),
"connector_knowledge_objects": (
session.query(ConnectorKnowledgeObject)
.filter(
ConnectorKnowledgeObject.tenant_id == tenant_id,
ConnectorKnowledgeObject.status != "deleted",
)
.count()
),
"connector_knowledge_runs": (
session.query(ConnectorKnowledgeSyncRun)
.filter(ConnectorKnowledgeSyncRun.tenant_id == tenant_id)
.count()
),
} }
@@ -349,6 +611,8 @@ manifest = ModuleManifest(
"portal", "portal",
"reporting", "reporting",
"risk_compliance", "risk_compliance",
"search",
"wiki",
), ),
required_capabilities=( required_capabilities=(
CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
@@ -379,18 +643,79 @@ manifest = ModuleManifest(
name="connectors.runtime_contract", name="connectors.runtime_contract",
version=CONNECTOR_RUNTIME_INTERFACE_VERSION, version=CONNECTOR_RUNTIME_INTERFACE_VERSION,
), ),
ModuleInterfaceProvider(
name=KNOWLEDGE_CAPABILITY,
version=KNOWLEDGE_INTERFACE_VERSION,
),
ModuleInterfaceProvider(name=CONNECTORS_DSAR_CAPABILITY, version="0.1.0"),
),
requires_interfaces=(
ModuleInterfaceRequirement(
name=CAPABILITY_FILES_TABULAR_CONTENT,
version_min="1.0.0",
version_max_exclusive="2.0.0",
optional=True,
),
), ),
permissions=PERMISSIONS, permissions=PERMISSIONS,
role_templates=ROLE_TEMPLATES, role_templates=ROLE_TEMPLATES,
route_factory=_router, route_factory=_router,
frontend=FrontendModule(
module_id=MODULE_ID,
package_name="@govoplan/connectors-webui",
view_surfaces=(
ViewSurface(
id="connectors.admin.governed-configurations",
module_id=MODULE_ID,
kind="section",
label="Connector governance",
order=45,
),
ViewSurface(
id="connectors.admin.simulation-review",
module_id=MODULE_ID,
kind="section",
label="Connector simulation review",
parent_id="connectors.admin.governed-configurations",
order=20,
),
ViewSurface(
id="connectors.admin.external-knowledge",
module_id=MODULE_ID,
kind="section",
label="External knowledge",
parent_id="connectors.admin.governed-configurations",
order=30,
),
),
),
capability_factories={ capability_factories={
CAPABILITY_CONNECTORS_TABULAR_SOURCES: _provider, CAPABILITY_CONNECTORS_TABULAR_SOURCES: _provider,
CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER: _provider, CAPABILITY_CONNECTORS_TABULAR_SNAPSHOT_WRITER: _provider,
CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider, CAPABILITY_DATASOURCE_ORIGINS: _datasource_origin_provider,
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider), CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS: (_sanctions_snapshot_provider),
CAPABILITY_CONNECTORS_FEEDS: _feed_provider, CAPABILITY_CONNECTORS_FEEDS: _feed_provider,
KNOWLEDGE_CAPABILITY: _knowledge_provider,
CONNECTORS_DSAR_CAPABILITY: _dsar_provider,
},
capability_documentation={
CONNECTORS_DSAR_CAPABILITY: CapabilityDocumentation(
label="Connector data-subject request provider",
summary=(
"Exports minimized operator attribution without connector secrets, "
"external payloads, or transport evidence."
),
contract_version="0.1.0",
),
}, },
tenant_summary_providers=(_tenant_summary,), tenant_summary_providers=(_tenant_summary,),
search_sources=(
SearchSourceProviderRegistration(
id=KNOWLEDGE_PROVIDER_ID,
factory=create_external_knowledge_search_source,
order=65,
),
),
architecture=ARCHITECTURE, architecture=ARCHITECTURE,
external_providers=EXTERNAL_PROVIDERS, external_providers=EXTERNAL_PROVIDERS,
external_provider_state_providers=( external_provider_state_providers=(
@@ -404,6 +729,11 @@ manifest = ModuleManifest(
provider_id=SANCTIONS_PROVIDER_ID, provider_id=SANCTIONS_PROVIDER_ID,
provider=sanctions_provider_states, provider=sanctions_provider_states,
), ),
ExternalProviderStateProviderRegistration(
module_id=MODULE_ID,
provider_id=KNOWLEDGE_PROVIDER_ID,
provider=knowledge_provider_states,
),
), ),
migration_spec=MigrationSpec( migration_spec=MigrationSpec(
module_id=MODULE_ID, module_id=MODULE_ID,
@@ -411,6 +741,13 @@ manifest = ModuleManifest(
script_location=str(Path(__file__).with_name("migrations") / "versions"), script_location=str(Path(__file__).with_name("migrations") / "versions"),
retirement_supported=True, retirement_supported=True,
retirement_provider=drop_table_retirement_provider( retirement_provider=drop_table_retirement_provider(
ConnectorKnowledgeSyncRun,
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorSimulationRun,
ConnectorConfiguration,
ConnectorDefinitionRevision,
ConnectorDefinition,
ConnectorSanctionsSnapshot, ConnectorSanctionsSnapshot,
ConnectorSanctionsAcquisitionRun, ConnectorSanctionsAcquisitionRun,
ConnectorTabularSource, ConnectorTabularSource,
@@ -423,6 +760,13 @@ manifest = ModuleManifest(
), ),
uninstall_guard_providers=( uninstall_guard_providers=(
persistent_table_uninstall_guard( persistent_table_uninstall_guard(
ConnectorKnowledgeSyncRun,
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorSimulationRun,
ConnectorConfiguration,
ConnectorDefinitionRevision,
ConnectorDefinition,
ConnectorSanctionsSnapshot, ConnectorSanctionsSnapshot,
ConnectorSanctionsAcquisitionRun, ConnectorSanctionsAcquisitionRun,
ConnectorTabularSource, ConnectorTabularSource,
@@ -430,6 +774,70 @@ manifest = ModuleManifest(
), ),
), ),
documentation=( documentation=(
DocumentationTopic(
id="connectors.data-subject-requests",
title="Connector data-subject requests",
summary=(
"Export accountable connector activity without disclosing credentials "
"or external data."
),
body=(
"Connectors correlates only an exact tenant account identifier and can "
"narrow an already verified search to one source, acquisition, "
"definition, configuration, simulation, external-knowledge profile, or "
"knowledge operation. The export identifies the subject's configuration, "
"acquisition, simulation, knowledge-operation, and review activity "
"using bounded lifecycle metadata. It never includes credential or "
"endpoint references, source rows, external responses, request payloads, "
"mapping and configuration documents, diagnostics, provenance, hashes, "
"or transport evidence. Connector attribution remains immutable "
"governance and external-operation evidence and is retained rather than "
"automatically erased. Email or object identifiers without a verified "
"account identifier do not establish a match."
),
layer="configured",
documentation_types=("admin", "user"),
audience=("user", "operator", "module_admin", "auditor"),
related_modules=("core", "access", "audit", "policy"),
order=37,
metadata={
"help_contexts": ["privacy.data-subject-requests"],
"consequence_classes": {
"export_operator_attribution": (
"Returns minimized connector activity for the exact account."
),
"exclude_connector_secrets": (
"Never returns credentials, endpoints, external rows, or evidence payloads."
),
"retain_connector_evidence": (
"Preserves configuration and external-operation accountability."
),
},
},
),
DocumentationTopic(
id="connectors.governed-configuration",
title="Govern connector definitions and simulations",
summary="Version connector schemas and mappings while preserving tenant-local overrides and review evidence.",
body=(
"Connector administrators create package-managed or local definitions that explicitly declare provider, protocol, capabilities, schemas, mapping rules, validation, preview support, audit expectations, privacy, retention, limits, and retry metadata. Every definition change creates an immutable revision. A tenant configuration pins one revision and stores only a credential reference; package updates remain available but do not change the effective configuration until an administrator adopts them. Local override leaf paths are displayed as protected and are reapplied when an update is adopted. Dry-runs and simulations are bounded, redact configured fields, are idempotent by caller key, and retain configuration, mapping, input, and external revision provenance. Ambiguous results follow the configuration policy: manual review, quarantine, or rejection. Pending and quarantined evidence requires an explicit approve or reject decision with a reason. Provider-specific live writes are not implied by a successful generic simulation."
),
layer="configured",
documentation_types=("admin", "user"),
audience=("operator", "module_admin", "integration_admin"),
related_modules=("policy", "audit", "dataflow", "ops"),
order=38,
metadata={
"kind": "guide",
"help_contexts": ["connectors.admin.governed-configurations"],
"prerequisites": [
"A connector definition has been installed or authored.",
"Credential material is stored outside the connector URL and referenced by an approved secret identifier.",
],
"outcome": "The active connector behavior is inspectable, version-pinned, testable, and reviewable before any provider-specific write.",
"verification": "Reload the configuration, inspect protected paths and effective hash, run a simulation with a new idempotency key, and resolve any pending review result.",
},
),
DocumentationTopic( DocumentationTopic(
id="connectors.authority-and-effects", id="connectors.authority-and-effects",
title="Connector authority and effect behavior", title="Connector authority and effect behavior",
@@ -472,11 +880,16 @@ manifest = ModuleManifest(
"fingerprints, and bounded reads. Dataflow stores only opaque source " "fingerprints, and bounded reads. Dataflow stores only opaque source "
"references and expected fingerprints. Each source declares its live, " "references and expected fingerprints. Each source declares its live, "
"cached, file-backed, or static mode, structured health, and supported " "cached, file-backed, or static mode, structured health, and supported "
"projection, filter, aggregation, sorting, and pagination pushdown. The " "projection, filter, aggregation, sorting, and pagination pushdown. "
"first executable provider imports immutable JSON or CSV snapshots, " "Immutable JSON/CSV snapshots remain available. Managed CSV/XLSX "
"supports projection and pagination, and exposes them as Datasource " "sources use the optional Files capability, pin an exact authorized "
"origins. Database and API providers can implement the same origin " "version, apply archive and expansion limits, and require explicit "
"contract without changing Datasources or Dataflow." "refresh before adopting a newer version. The PostgreSQL adapter uses "
"an active governed configuration and scoped Core credential envelope, "
"reflects simple schema/table identifiers, runs read-only bounded "
"projection and pagination, and blocks configuration, credential, or "
"schema drift until reviewed refresh. Credentials, endpoints, storage "
"keys, and raw file internals are never exposed through the origin."
), ),
layer="available", layer="available",
documentation_types=("admin", "user"), documentation_types=("admin", "user"),
@@ -492,8 +905,13 @@ manifest = ModuleManifest(
"Connectors owns bounded, SSRF-protected RSS/Atom transport and XML " "Connectors owns bounded, SSRF-protected RSS/Atom transport and XML "
"parsing. Imported entries become immutable tabular snapshots exposed " "parsing. Imported entries become immutable tabular snapshots exposed "
"through Datasources, including acquisition, freshness, ETag, content " "through Datasources, including acquisition, freshness, ETag, content "
"digest, and source provenance. Portal or Reporting owns publication " "digest, and source provenance. Emission accepts only provenance-bearing "
"routes and must pass the allowed visibility set when rendering output. " "event, publication, case, or report selections from an owning surface. "
"The requested audience determines the visibility ceiling: public output "
"contains only public entries, while tenant or private output requires a "
"separate restricted-feed permission. Callers cannot supply their own "
"visibility allow-list. Portal or Reporting owns durable publication "
"routes and must re-authorize access on every restricted feed request. "
"A separate RSS module is only warranted if GovOPlaN later needs a " "A separate RSS module is only warranted if GovOPlaN later needs a "
"dedicated feed-reader product surface." "dedicated feed-reader product surface."
), ),
@@ -503,6 +921,45 @@ manifest = ModuleManifest(
related_modules=("datasources", "dataflow", "portal", "reporting"), related_modules=("datasources", "dataflow", "portal", "reporting"),
order=42, order=42,
), ),
DocumentationTopic(
id="connectors.mediawiki-bluespice",
title="Connect MediaWiki and BlueSpice knowledge",
summary="Discover, synchronize, search, publish, and preview migration of external knowledge without weakening source permissions.",
body=(
"A connector administrator first creates a governed MediaWiki Action API configuration whose endpoint passes Core egress and peer validation; credentials remain in a scoped credential envelope. A knowledge profile then maps each included source namespace to a target Wiki space and path prefix, selects source authority, and sets a tenant or restricted fallback ACL. Discovery records product, version, extensions, namespaces, authentication state, capabilities, and loss diagnostics. Run a bounded full backfill once, then cursor-based deltas for revisions, moves, deletions, and permission changes. Stable page and revision identifiers, canonical links, redirects, files, categories, links, discussions, provenance, and current ACLs remain attached to the connector snapshot. Search indexes only active, non-deleted pages and rechecks the current profile, tenant, permission, and ACL before returning every result; disabling Connectors removes its Search projection. Publication requires an expected external revision, an idempotency key, and durable outcome evidence. An unknown provider outcome must be reconciled before retry. Migration into native Wiki is deliberately preview-only in this slice: the dry-run reports target-path collisions, attachment-name conflicts, unsupported macros, truncation, and source fingerprints; a successful preview does not write native Wiki pages. Existing authorized snapshots can remain visible during a provider outage, but health and freshness stay explicit and no current-source claim is made."
),
layer="configured",
documentation_types=("admin", "user"),
audience=(
"operator",
"module_admin",
"integration_admin",
"knowledge_manager",
"auditor",
),
related_modules=("core", "search", "wiki", "files", "audit", "policy"),
order=43,
translations={
"de": {
"title": "Wissen aus MediaWiki und BlueSpice anbinden",
"summary": "Externes Wissen erkennen, synchronisieren, durchsuchen, veröffentlichen und eine Migration prüfen, ohne Quellberechtigungen abzuschwächen.",
"body": (
"Eine Connector-Administration erstellt zuerst eine gesteuerte Konfiguration für die MediaWiki Action API. Der Endpunkt unterliegt der zentralen Ausgangs- und Gegenstellenprüfung; Zugangsdaten bleiben in einem zweckgebundenen Zugangsdaten-Umschlag. Das Wissensprofil ordnet jeden eingeschlossenen Quellnamensraum einem Zielbereich und Pfadpräfix in Wiki zu und legt Quellhoheit sowie eine mandantenweite oder eingeschränkte Ersatz-ACL fest. Die Erkennung dokumentiert Produkt, Version, Erweiterungen, Namensräume, Authentifizierungsstatus, Fähigkeiten und Verlustdiagnosen. Nach einem begrenzten Vollabgleich folgen cursorbasierte Änderungen für Revisionen, Verschiebungen, Löschungen und Berechtigungen. Stabile Seiten- und Revisionskennungen, kanonische Links, Weiterleitungen, Dateien, Kategorien, Links, Diskussionen, Herkunft und aktuelle ACLs bleiben am Snapshot. Search indiziert nur aktive, nicht gelöschte Seiten und prüft bei jedem Treffer Profilstatus, Mandant, Berechtigung und ACL erneut; bei Deaktivierung von Connectors wird dessen Suchprojektion entfernt. Veröffentlichungen erfordern die erwartete externe Revision, einen Idempotenzschlüssel und dauerhafte Ergebnisevidenz. Ein unbekanntes Ergebnis muss vor einem erneuten Versuch abgeglichen werden. Die Migration in das native Wiki ist in diesem Ausbauschritt ausschließlich eine Vorschau: Sie meldet Pfad- und Anhangskonflikte, nicht unterstützte Makros, Begrenzungen und Quellfingerabdrücke, schreibt aber keine Wiki-Seiten. Bei einem Ausfall dürfen bestehende Snapshots nur für weiterhin Berechtigte sichtbar bleiben; Zustand und Aktualität bleiben ausdrücklich erkennbar."
),
}
},
metadata={
"kind": "guide",
"help_contexts": ["connectors.admin.external-knowledge"],
"prerequisites": [
"An active governed MediaWiki Action API configuration exists.",
"Namespace targets and fallback ACLs have been reviewed.",
"The Search and Wiki modules are optional and remain capability-separated.",
],
"outcome": "External knowledge remains identity-stable, loss-visible, ACL-safe, and migration-ready.",
"verification": "Rediscover the profile, run a keyed delta, inspect health and diagnostics, verify an allowed and denied Search principal, and run a migration dry-run before any target-side work.",
},
),
DocumentationTopic( DocumentationTopic(
id="connectors.sanctions-snapshots", id="connectors.sanctions-snapshots",
title="Sanctions source snapshots", title="Sanctions source snapshots",
@@ -0,0 +1,503 @@
from __future__ import annotations
import base64
import json
from collections.abc import Mapping, Sequence
from dataclasses import dataclass
from typing import Any, Protocol
from urllib.parse import quote, urlencode, urljoin, urlsplit, urlunsplit
from govoplan_core.security.http_fetch import fetch_http
MAX_MEDIAWIKI_RESPONSE_BYTES = 10_000_000
class MediaWikiTransportError(RuntimeError):
def __init__(
self,
code: str,
message: str,
*,
retryable: bool = False,
outcome_unknown: bool = False,
) -> None:
super().__init__(message)
self.code = code
self.retryable = retryable
self.outcome_unknown = outcome_unknown
@dataclass(frozen=True, slots=True)
class MediaWikiChangeBatch:
changes: tuple[Mapping[str, Any], ...]
next_cursor: str | None
complete: bool
high_watermark: str | None
evidence: Mapping[str, Any]
@dataclass(frozen=True, slots=True)
class MediaWikiPublishResult:
page_id: str
revision_id: str
title: str
canonical_url: str | None
evidence: Mapping[str, Any]
class MediaWikiTransport(Protocol):
def discover(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
) -> Mapping[str, Any]: ...
def changes(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
cursor: str | None,
limit: int,
force_full: bool,
) -> MediaWikiChangeBatch: ...
def publish(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
title: str,
body: str,
summary: str,
expected_revision: str | None,
minor: bool,
) -> MediaWikiPublishResult: ...
class HttpMediaWikiTransport:
"""Bounded MediaWiki Action API transport with Core outbound policy enforcement."""
def discover(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
) -> Mapping[str, Any]:
return self._request(
endpoint_url,
credential=credential,
params={
"action": "query",
"meta": "siteinfo|userinfo",
"siprop": "general|extensions|namespaces|namespacealiases|rightsinfo",
"uiprop": "rights|groups",
"format": "json",
"formatversion": "2",
"curtimestamp": "1",
},
)
def changes(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
cursor: str | None,
limit: int,
force_full: bool,
) -> MediaWikiChangeBatch:
if force_full:
listing = self._request(
endpoint_url,
credential=credential,
params={
"action": "query",
"list": "allpages",
"aplimit": str(limit),
"apcontinue": cursor or "",
"apfilterredir": "all",
"format": "json",
"formatversion": "2",
"curtimestamp": "1",
},
)
rows = _list(listing, "query", "allpages")
page_ids = tuple(
str(item.get("pageid"))
for item in rows
if isinstance(item, Mapping) and item.get("pageid") is not None
)
changes = self._page_details(
endpoint_url,
credential=credential,
page_ids=page_ids,
)
next_cursor = _continue_token(listing, "apcontinue")
return MediaWikiChangeBatch(
changes=changes,
next_cursor=next_cursor,
complete=next_cursor is None,
high_watermark=_optional_text(listing.get("curtimestamp")),
evidence={
"mode": "backfill",
"listed": len(rows),
"resolved": len(changes),
},
)
listing = self._request(
endpoint_url,
credential=credential,
params={
"action": "query",
"list": "recentchanges",
"rclimit": str(limit),
"rccontinue": cursor or "",
"rcdir": "newer",
"rcprop": "title|ids|sizes|flags|user|timestamp|loginfo|tags",
"rctype": "edit|new|log",
"format": "json",
"formatversion": "2",
"curtimestamp": "1",
},
)
recent = tuple(
item
for item in _list(listing, "query", "recentchanges")
if isinstance(item, Mapping)
)
page_ids = tuple(
dict.fromkeys(
str(item.get("pageid"))
for item in recent
if item.get("pageid") is not None
and str(item.get("logtype") or "") != "delete"
)
)
resolved = {
str(item.get("pageid")): item
for item in self._page_details(
endpoint_url,
credential=credential,
page_ids=page_ids,
)
}
changes: list[Mapping[str, Any]] = []
for item in recent:
page_id = _optional_text(item.get("pageid"))
if str(item.get("logtype") or "") == "delete":
changes.append(
{
"change_kind": "delete",
"pageid": page_id or f"log:{item.get('logid')}",
"title": _optional_text(item.get("title")) or "Deleted page",
"ns": item.get("ns"),
"timestamp": item.get("timestamp"),
"logid": item.get("logid"),
"change_cursor": _change_cursor(item),
}
)
continue
page = dict(resolved.get(page_id or "") or item)
page["change_kind"] = "upsert"
page["change_cursor"] = _change_cursor(item)
page["recent_change"] = _safe_recent_change(item)
changes.append(page)
next_cursor = _continue_token(listing, "rccontinue")
return MediaWikiChangeBatch(
changes=tuple(changes),
next_cursor=next_cursor,
complete=next_cursor is None,
high_watermark=_optional_text(listing.get("curtimestamp")),
evidence={
"mode": "delta",
"listed": len(recent),
"resolved": len(resolved),
},
)
def publish(
self,
*,
endpoint_url: str,
credential: Mapping[str, Any] | None,
title: str,
body: str,
summary: str,
expected_revision: str | None,
minor: bool,
) -> MediaWikiPublishResult:
if not credential:
raise MediaWikiTransportError(
"credential_required",
"Publishing requires a governed credential envelope.",
)
csrf_token = _optional_text(credential.get("csrf_token"))
if not csrf_token:
token_payload = self._request(
endpoint_url,
credential=credential,
params={
"action": "query",
"meta": "tokens",
"type": "csrf",
"format": "json",
"formatversion": "2",
},
)
query = token_payload.get("query")
tokens = query.get("tokens") if isinstance(query, Mapping) else None
csrf_token = (
_optional_text(tokens.get("csrftoken"))
if isinstance(tokens, Mapping)
else None
)
if not csrf_token:
raise MediaWikiTransportError(
"csrf_token_unavailable",
"The provider did not issue a CSRF token for the configured credential.",
)
parameters: dict[str, str] = {
"action": "edit",
"title": title,
"text": body,
"summary": summary,
"token": csrf_token,
"format": "json",
"formatversion": "2",
}
if minor:
parameters["minor"] = "1"
if expected_revision:
parameters["baserevid"] = expected_revision
payload = self._request(
endpoint_url,
credential=credential,
params=parameters,
method="POST",
)
edit = payload.get("edit")
if not isinstance(edit, Mapping) or str(edit.get("result")) != "Success":
raise MediaWikiTransportError(
"publish_rejected",
"MediaWiki rejected the page publication.",
)
page_id = _required_text(edit.get("pageid"), "MediaWiki omitted the page id")
revision_id = _required_text(
edit.get("newrevid"), "MediaWiki omitted the accepted revision id"
)
canonical_url = _canonical_page_url(endpoint_url, title)
return MediaWikiPublishResult(
page_id=page_id,
revision_id=revision_id,
title=_optional_text(edit.get("title")) or title,
canonical_url=canonical_url,
evidence={
"result": "Success",
"page_id": page_id,
"revision_id": revision_id,
"old_revision_id": _optional_text(edit.get("oldrevid")),
"new_page": bool(edit.get("new")),
},
)
def _page_details(
self,
endpoint_url: str,
*,
credential: Mapping[str, Any] | None,
page_ids: Sequence[str],
) -> tuple[Mapping[str, Any], ...]:
if not page_ids:
return ()
payload = self._request(
endpoint_url,
credential=credential,
params={
"action": "query",
"pageids": "|".join(page_ids),
"prop": "info|revisions|categories|links|images|pageprops",
"inprop": "url|displaytitle",
"rvlimit": "1",
"rvprop": "ids|timestamp|user|comment|content|contentmodel|sha1|flags",
"rvslots": "main",
"cllimit": "max",
"pllimit": "max",
"imlimit": "max",
"format": "json",
"formatversion": "2",
},
)
return tuple(
item
for item in _list(payload, "query", "pages")
if isinstance(item, Mapping)
)
def _request(
self,
endpoint_url: str,
*,
credential: Mapping[str, Any] | None,
params: Mapping[str, str],
method: str = "GET",
) -> Mapping[str, Any]:
api_url = _api_url(endpoint_url)
headers = {
"Accept": "application/json",
"User-Agent": "GovOPlaN-Connectors/MediaWiki",
**_auth_headers(credential),
}
body = None
target_url = api_url
if method == "POST":
body = urlencode(params).encode("utf-8")
headers["Content-Type"] = "application/x-www-form-urlencoded"
else:
target_url = f"{api_url}?{urlencode({key: value for key, value in params.items() if value != ''})}"
try:
response = fetch_http(
target_url,
method=method,
headers=headers,
body=body,
max_bytes=MAX_MEDIAWIKI_RESPONSE_BYTES,
timeout=30,
label="MediaWiki Action API URL",
)
except Exception as exc:
raise MediaWikiTransportError(
"transport_unavailable",
"MediaWiki transport failed before a valid response was received.",
retryable=True,
outcome_unknown=method == "POST",
) from exc
if response.status < 200 or response.status >= 300:
raise MediaWikiTransportError(
"http_error",
f"MediaWiki returned HTTP {response.status}.",
retryable=response.status >= 500,
outcome_unknown=method == "POST" and response.status >= 500,
)
try:
payload = json.loads(response.body)
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise MediaWikiTransportError(
"invalid_json",
"MediaWiki returned an invalid JSON response.",
outcome_unknown=method == "POST",
) from exc
if not isinstance(payload, Mapping):
raise MediaWikiTransportError(
"invalid_response",
"MediaWiki returned an unsupported response shape.",
outcome_unknown=method == "POST",
)
error = payload.get("error")
if isinstance(error, Mapping):
code = _optional_text(error.get("code")) or "provider_error"
raise MediaWikiTransportError(
code,
_optional_text(error.get("info")) or "MediaWiki rejected the request.",
retryable=code in {"maxlag", "readonly", "ratelimited"},
)
return payload
def _api_url(endpoint_url: str) -> str:
normalized = endpoint_url.strip().rstrip("/")
parsed = urlsplit(normalized)
if parsed.path.endswith("/api.php"):
return normalized
path = f"{parsed.path.rstrip('/')}/api.php"
return urlunsplit((parsed.scheme, parsed.netloc, path, "", ""))
def _canonical_page_url(endpoint_url: str, title: str) -> str:
normalized = endpoint_url.strip().rstrip("/")
parsed = urlsplit(normalized)
root_path = parsed.path
if root_path.endswith("/api.php"):
root_path = root_path[: -len("/api.php")]
base = urlunsplit((parsed.scheme, parsed.netloc, f"{root_path.rstrip('/')}/", "", ""))
return urljoin(base, f"wiki/{quote(title.replace(' ', '_'), safe=':_-./~')}")
def _auth_headers(credential: Mapping[str, Any] | None) -> dict[str, str]:
if not credential:
return {}
token = _optional_text(
credential.get("bearer_token")
or credential.get("access_token")
or credential.get("token")
)
if token:
return {"Authorization": f"Bearer {token}"}
username = _optional_text(credential.get("username") or credential.get("user"))
password = _optional_text(credential.get("password"))
if username and password:
encoded = base64.b64encode(f"{username}:{password}".encode()).decode("ascii")
return {"Authorization": f"Basic {encoded}"}
return {}
def _list(payload: Mapping[str, Any], *path: str) -> list[Any]:
value: Any = payload
for part in path:
if not isinstance(value, Mapping):
return []
value = value.get(part)
return list(value) if isinstance(value, list) else []
def _continue_token(payload: Mapping[str, Any], name: str) -> str | None:
value = payload.get("continue")
return _optional_text(value.get(name)) if isinstance(value, Mapping) else None
def _change_cursor(change: Mapping[str, Any]) -> str:
for name in ("rcid", "logid", "revid", "old_revid"):
if change.get(name) is not None:
return f"{name}:{change[name]}"
return _optional_text(change.get("timestamp")) or "unknown"
def _safe_recent_change(change: Mapping[str, Any]) -> dict[str, Any]:
allowed = (
"type",
"ns",
"title",
"pageid",
"revid",
"old_revid",
"timestamp",
"logtype",
"logaction",
"tags",
)
return {key: change[key] for key in allowed if key in change}
def _required_text(value: object, message: str) -> str:
normalized = _optional_text(value)
if not normalized:
raise MediaWikiTransportError("invalid_response", message)
return normalized
def _optional_text(value: object) -> str | None:
normalized = str(value or "").strip()
return normalized or None
__all__ = [
"HttpMediaWikiTransport",
"MAX_MEDIAWIKI_RESPONSE_BYTES",
"MediaWikiChangeBatch",
"MediaWikiPublishResult",
"MediaWikiTransport",
"MediaWikiTransportError",
]
@@ -0,0 +1,214 @@
"""governed connector definitions and simulation evidence
Revision ID: a8d9e0f1b2c3
Revises: f7c8d9e0a1b2
Create Date: 2026-08-20 12:30:00.000000
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "a8d9e0f1b2c3"
down_revision = "f7c8d9e0a1b2"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"connector_definitions",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("definition_key", sa.String(length=160), nullable=False),
sa.Column("name", sa.String(length=300), nullable=False),
sa.Column("description", sa.Text(), nullable=True),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("current_revision", sa.Integer(), nullable=False),
sa.Column("source_package", sa.String(length=300), nullable=True),
sa.Column("local_definition", sa.Boolean(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_definitions")),
sa.UniqueConstraint(
"tenant_id",
"definition_key",
name="uq_connector_definition_tenant_key",
),
)
op.create_index(
op.f("ix_connector_definitions_tenant_id"),
"connector_definitions",
["tenant_id"],
)
op.create_index(
op.f("ix_connector_definitions_status"),
"connector_definitions",
["status"],
)
op.create_index(
"ix_connector_definitions_tenant_status",
"connector_definitions",
["tenant_id", "status"],
)
op.create_table(
"connector_definition_revisions",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("definition_id", sa.String(length=36), nullable=False),
sa.Column("revision", sa.Integer(), nullable=False),
sa.Column("specification", sa.JSON(), nullable=False),
sa.Column("definition_hash", sa.String(length=64), nullable=False),
sa.Column("origin", sa.String(length=30), nullable=False),
sa.Column("package_ref", sa.String(length=300), nullable=True),
sa.Column("created_by", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["definition_id"],
["connector_definitions.id"],
name=op.f(
"fk_connector_definition_revisions_definition_id_connector_definitions"
),
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"id",
name=op.f("pk_connector_definition_revisions"),
),
sa.UniqueConstraint(
"definition_id",
"revision",
name="uq_connector_definition_revision",
),
)
op.create_index(
op.f("ix_connector_definition_revisions_definition_id"),
"connector_definition_revisions",
["definition_id"],
)
op.create_index(
op.f("ix_connector_definition_revisions_definition_hash"),
"connector_definition_revisions",
["definition_hash"],
)
op.create_index(
op.f("ix_connector_definition_revisions_created_by"),
"connector_definition_revisions",
["created_by"],
)
op.create_table(
"connector_configurations",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("definition_id", sa.String(length=36), nullable=False),
sa.Column("name", sa.String(length=300), nullable=False),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("endpoint_url", sa.String(length=1500), nullable=True),
sa.Column("credential_ref", sa.String(length=500), nullable=True),
sa.Column("base_definition_revision", sa.Integer(), nullable=False),
sa.Column("local_overrides", sa.JSON(), nullable=False),
sa.Column("protected_paths", sa.JSON(), nullable=False),
sa.Column("effective_configuration", sa.JSON(), nullable=False),
sa.Column("effective_hash", sa.String(length=64), nullable=False),
sa.Column("resource_revision", sa.Integer(), nullable=False),
sa.Column("ambiguity_policy", sa.String(length=30), nullable=False),
sa.Column("updated_by", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["definition_id"],
["connector_definitions.id"],
name=op.f(
"fk_connector_configurations_definition_id_connector_definitions"
),
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_connector_configurations")),
sa.UniqueConstraint(
"tenant_id",
"name",
name="uq_connector_configuration_tenant_name",
),
)
for name, columns in (
("ix_connector_configurations_tenant_id", ["tenant_id"]),
("ix_connector_configurations_definition_id", ["definition_id"]),
("ix_connector_configurations_status", ["status"]),
("ix_connector_configurations_effective_hash", ["effective_hash"]),
("ix_connector_configurations_updated_by", ["updated_by"]),
):
op.create_index(op.f(name), "connector_configurations", columns)
op.create_index(
"ix_connector_configurations_tenant_status",
"connector_configurations",
["tenant_id", "status"],
)
op.create_table(
"connector_simulation_runs",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("configuration_id", sa.String(length=36), nullable=False),
sa.Column("mode", sa.String(length=30), nullable=False),
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("review_state", sa.String(length=30), nullable=False),
sa.Column("definition_revision", sa.Integer(), nullable=False),
sa.Column("configuration_revision", sa.Integer(), nullable=False),
sa.Column("configuration_hash", sa.String(length=64), nullable=False),
sa.Column("input_hash", sa.String(length=64), nullable=False),
sa.Column("summary", sa.JSON(), nullable=False),
sa.Column("effects", sa.JSON(), nullable=False),
sa.Column("diagnostics", sa.JSON(), nullable=False),
sa.Column("provenance", sa.JSON(), nullable=False),
sa.Column("created_by", sa.String(length=255), nullable=True),
sa.Column("reviewed_by", sa.String(length=255), nullable=True),
sa.Column("reviewed_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("review_reason", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["configuration_id"],
["connector_configurations.id"],
name=op.f(
"fk_connector_simulation_runs_configuration_id_connector_configurations"
),
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"id",
name=op.f("pk_connector_simulation_runs"),
),
sa.UniqueConstraint(
"tenant_id",
"configuration_id",
"mode",
"idempotency_key",
name="uq_connector_simulation_run_idempotency",
),
)
for name, columns in (
("ix_connector_simulation_runs_tenant_id", ["tenant_id"]),
("ix_connector_simulation_runs_configuration_id", ["configuration_id"]),
("ix_connector_simulation_runs_mode", ["mode"]),
("ix_connector_simulation_runs_status", ["status"]),
("ix_connector_simulation_runs_review_state", ["review_state"]),
("ix_connector_simulation_runs_created_by", ["created_by"]),
("ix_connector_simulation_runs_reviewed_by", ["reviewed_by"]),
):
op.create_index(op.f(name), "connector_simulation_runs", columns)
op.create_index(
"ix_connector_simulation_runs_review",
"connector_simulation_runs",
["tenant_id", "review_state", "created_at"],
)
def downgrade() -> None:
op.drop_table("connector_simulation_runs")
op.drop_table("connector_configurations")
op.drop_table("connector_definition_revisions")
op.drop_table("connector_definitions")
@@ -0,0 +1,207 @@
"""MediaWiki and BlueSpice knowledge connector state
Revision ID: b9e0f1a2c3d4
Revises: a8d9e0f1b2c3
Create Date: 2026-08-22 14:00:00.000000
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "b9e0f1a2c3d4"
down_revision = "a8d9e0f1b2c3"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"connector_knowledge_profiles",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("configuration_id", sa.String(length=36), nullable=False),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("product", sa.String(length=50), nullable=False),
sa.Column("product_version", sa.String(length=100), nullable=True),
sa.Column("desired_maturity", sa.String(length=30), nullable=False),
sa.Column("discovered_maturity", sa.String(length=30), nullable=False),
sa.Column("source_authority_mode", sa.String(length=40), nullable=False),
sa.Column("default_visibility", sa.String(length=30), nullable=False),
sa.Column("default_acl_tokens", sa.JSON(), nullable=False),
sa.Column("namespace_mappings", sa.JSON(), nullable=False),
sa.Column("capabilities", sa.JSON(), nullable=False),
sa.Column("discovery_revision", sa.String(length=255), nullable=True),
sa.Column("discovery_evidence", sa.JSON(), nullable=False),
sa.Column("health_status", sa.String(length=30), nullable=False),
sa.Column("health_details", sa.JSON(), nullable=False),
sa.Column("discovered_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_sync_cursor", sa.String(length=500), nullable=True),
sa.Column("last_high_watermark", sa.String(length=500), nullable=True),
sa.Column("resource_revision", sa.Integer(), nullable=False),
sa.Column("updated_by", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["configuration_id"],
["connector_configurations.id"],
name=op.f(
"fk_connector_knowledge_profiles_configuration_id_connector_configurations"
),
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint(
"id", name=op.f("pk_connector_knowledge_profiles")
),
sa.UniqueConstraint(
"tenant_id",
"configuration_id",
name="uq_connector_knowledge_profile_configuration",
),
)
for name, columns in (
("ix_connector_knowledge_profiles_tenant_id", ["tenant_id"]),
("ix_connector_knowledge_profiles_configuration_id", ["configuration_id"]),
("ix_connector_knowledge_profiles_status", ["status"]),
("ix_connector_knowledge_profiles_product", ["product"]),
("ix_connector_knowledge_profiles_discovery_revision", ["discovery_revision"]),
("ix_connector_knowledge_profiles_health_status", ["health_status"]),
("ix_connector_knowledge_profiles_updated_by", ["updated_by"]),
):
op.create_index(op.f(name), "connector_knowledge_profiles", columns)
op.create_index(
"ix_connector_knowledge_profiles_tenant_status",
"connector_knowledge_profiles",
["tenant_id", "status"],
)
op.create_table(
"connector_knowledge_objects",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("profile_id", sa.String(length=36), nullable=False),
sa.Column("object_type", sa.String(length=40), nullable=False),
sa.Column("external_id", sa.String(length=255), nullable=False),
sa.Column("external_page_id", sa.String(length=255), nullable=True),
sa.Column("external_revision_id", sa.String(length=255), nullable=True),
sa.Column("namespace_id", sa.Integer(), nullable=True),
sa.Column("title", sa.String(length=500), nullable=False),
sa.Column("canonical_url", sa.String(length=1500), nullable=True),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("redirect_target_external_id", sa.String(length=255), nullable=True),
sa.Column("source_revision", sa.String(length=255), nullable=False),
sa.Column("content_hash", sa.String(length=64), nullable=False),
sa.Column("visibility", sa.String(length=30), nullable=False),
sa.Column("acl_tokens", sa.JSON(), nullable=False),
sa.Column("mapped_data", sa.JSON(), nullable=False),
sa.Column("provenance", sa.JSON(), nullable=False),
sa.Column("change_cursor", sa.String(length=500), nullable=True),
sa.Column("source_updated_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("observed_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("resource_revision", sa.Integer(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["profile_id"],
["connector_knowledge_profiles.id"],
name=op.f(
"fk_connector_knowledge_objects_profile_id_connector_knowledge_profiles"
),
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"id", name=op.f("pk_connector_knowledge_objects")
),
sa.UniqueConstraint(
"profile_id",
"object_type",
"external_id",
name="uq_connector_knowledge_object_identity",
),
)
for name, columns in (
("ix_connector_knowledge_objects_tenant_id", ["tenant_id"]),
("ix_connector_knowledge_objects_profile_id", ["profile_id"]),
("ix_connector_knowledge_objects_object_type", ["object_type"]),
("ix_connector_knowledge_objects_external_page_id", ["external_page_id"]),
("ix_connector_knowledge_objects_external_revision_id", ["external_revision_id"]),
("ix_connector_knowledge_objects_namespace_id", ["namespace_id"]),
("ix_connector_knowledge_objects_status", ["status"]),
("ix_connector_knowledge_objects_content_hash", ["content_hash"]),
("ix_connector_knowledge_objects_change_cursor", ["change_cursor"]),
("ix_connector_knowledge_objects_source_updated_at", ["source_updated_at"]),
("ix_connector_knowledge_objects_observed_at", ["observed_at"]),
):
op.create_index(op.f(name), "connector_knowledge_objects", columns)
op.create_index(
"ix_connector_knowledge_objects_tenant_profile_status",
"connector_knowledge_objects",
["tenant_id", "profile_id", "status"],
)
op.create_index(
"ix_connector_knowledge_objects_tenant_updated",
"connector_knowledge_objects",
["tenant_id", "source_updated_at"],
)
op.create_table(
"connector_knowledge_sync_runs",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("profile_id", sa.String(length=36), nullable=False),
sa.Column("mode", sa.String(length=40), nullable=False),
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
sa.Column("request_hash", sa.String(length=64), nullable=False),
sa.Column("status", sa.String(length=30), nullable=False),
sa.Column("cursor_before", sa.String(length=500), nullable=True),
sa.Column("cursor_after", sa.String(length=500), nullable=True),
sa.Column("high_watermark", sa.String(length=500), nullable=True),
sa.Column("counts", sa.JSON(), nullable=False),
sa.Column("effects", sa.JSON(), nullable=False),
sa.Column("diagnostics", sa.JSON(), nullable=False),
sa.Column("provenance", sa.JSON(), nullable=False),
sa.Column("created_by", sa.String(length=255), nullable=True),
sa.Column("started_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("finished_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["profile_id"],
["connector_knowledge_profiles.id"],
name=op.f(
"fk_connector_knowledge_sync_runs_profile_id_connector_knowledge_profiles"
),
ondelete="CASCADE",
),
sa.PrimaryKeyConstraint(
"id", name=op.f("pk_connector_knowledge_sync_runs")
),
sa.UniqueConstraint(
"tenant_id",
"profile_id",
"mode",
"idempotency_key",
name="uq_connector_knowledge_sync_run_idempotency",
),
)
for name, columns in (
("ix_connector_knowledge_sync_runs_tenant_id", ["tenant_id"]),
("ix_connector_knowledge_sync_runs_profile_id", ["profile_id"]),
("ix_connector_knowledge_sync_runs_mode", ["mode"]),
("ix_connector_knowledge_sync_runs_status", ["status"]),
("ix_connector_knowledge_sync_runs_created_by", ["created_by"]),
("ix_connector_knowledge_sync_runs_started_at", ["started_at"]),
):
op.create_index(op.f(name), "connector_knowledge_sync_runs", columns)
op.create_index(
"ix_connector_knowledge_sync_runs_profile_started",
"connector_knowledge_sync_runs",
["tenant_id", "profile_id", "started_at"],
)
def downgrade() -> None:
op.drop_table("connector_knowledge_sync_runs")
op.drop_table("connector_knowledge_objects")
op.drop_table("connector_knowledge_profiles")
@@ -1,6 +1,5 @@
from __future__ import annotations from __future__ import annotations
from collections import defaultdict
from datetime import UTC, datetime from datetime import UTC, datetime
from hashlib import sha256 from hashlib import sha256
@@ -8,6 +7,9 @@ from sqlalchemy import func, select
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from govoplan_connectors.backend.db.models import ( from govoplan_connectors.backend.db.models import (
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
ConnectorSanctionsAcquisitionRun, ConnectorSanctionsAcquisitionRun,
ConnectorSanctionsSnapshot, ConnectorSanctionsSnapshot,
ConnectorTabularSource, ConnectorTabularSource,
@@ -20,6 +22,7 @@ from govoplan_core.core.provider_governance import (
TABULAR_PROVIDER_ID = "connectors.tabular_snapshot" TABULAR_PROVIDER_ID = "connectors.tabular_snapshot"
SANCTIONS_PROVIDER_ID = "connectors.sanctions_snapshot" SANCTIONS_PROVIDER_ID = "connectors.sanctions_snapshot"
KNOWLEDGE_PROVIDER_ID = "connectors.mediawiki.pages"
def tabular_provider_states( def tabular_provider_states(
@@ -86,6 +89,37 @@ def sanctions_provider_states(
) )
def knowledge_provider_states(
context: ExternalProviderStateContext,
) -> tuple[ExternalProviderRuntimeState, ...]:
session = _session(context)
statement = select(ConnectorKnowledgeProfile)
if context.tenant_id is not None:
statement = statement.where(
ConnectorKnowledgeProfile.tenant_id == context.tenant_id
)
profiles = tuple(
session.scalars(
statement.order_by(
ConnectorKnowledgeProfile.tenant_id,
ConnectorKnowledgeProfile.id,
).limit(context.max_items + 1)
)
)[: context.max_items]
counts = _knowledge_counts(session, profiles)
latest_runs = _latest_knowledge_runs(session, profiles)
observed_at = datetime.now(UTC)
return tuple(
_knowledge_state(
profile,
observed_at=observed_at,
object_count=counts.get(profile.id, 0),
latest_run=latest_runs.get(profile.id),
)
for profile in profiles
)
def _session(context: ExternalProviderStateContext) -> Session: def _session(context: ExternalProviderStateContext) -> Session:
if not isinstance(context.session, Session): if not isinstance(context.session, Session):
raise RuntimeError("Connectors provider state requires a database session.") raise RuntimeError("Connectors provider state requires a database session.")
@@ -98,6 +132,13 @@ def _tabular_state(
observed_at: datetime, observed_at: datetime,
) -> ExternalProviderRuntimeState: ) -> ExternalProviderRuntimeState:
active = source.status == "active" active = source.status == "active"
live_origin = source.provider in {"managed_file", "postgresql"}
labels = {
"managed_file": "Exact managed-file origin",
"postgresql": "Live PostgreSQL origin",
"snapshot": "Immutable tabular snapshot",
}
label = labels.get(source.provider, "Tabular source")
return ExternalProviderRuntimeState( return ExternalProviderRuntimeState(
provider_id=TABULAR_PROVIDER_ID, provider_id=TABULAR_PROVIDER_ID,
binding_ref=f"connectors:tabular-source:{source.id}", binding_ref=f"connectors:tabular-source:{source.id}",
@@ -105,17 +146,20 @@ def _tabular_state(
observed_at=observed_at, observed_at=observed_at,
configured=True, configured=True,
active=active, active=active,
health="healthy" if active else "inactive", health=("unknown" if active and live_origin else "healthy" if active else "inactive"),
freshness="not_applicable", freshness="unknown" if active and live_origin else "not_applicable",
conflict="not_applicable", conflict="not_applicable",
recovery="ready" if active else "not_applicable", recovery="ready" if active else "not_applicable",
last_success_at=_aware(source.updated_at or source.created_at), last_success_at=_aware(source.updated_at or source.created_at),
detail=( detail=(
"Immutable tabular snapshot is available." f"{label} is configured; live access and drift are checked on preview."
if active and live_origin
else f"{label} is available."
if active if active
else "Immutable tabular snapshot is inactive." else f"{label} is inactive."
), ),
metrics={ metrics={
"provider": source.provider,
"row_count": int(source.row_count), "row_count": int(source.row_count),
"byte_count": int(source.byte_count), "byte_count": int(source.byte_count),
"schema_version": int(source.schema_version), "schema_version": int(source.schema_version),
@@ -152,6 +196,119 @@ def _snapshot_counts(
} }
def _knowledge_counts(
session: Session,
profiles: tuple[ConnectorKnowledgeProfile, ...],
) -> dict[str, int]:
profile_ids = tuple(item.id for item in profiles)
if not profile_ids:
return {}
return {
str(profile_id): int(count)
for profile_id, count in session.execute(
select(
ConnectorKnowledgeObject.profile_id,
func.count(ConnectorKnowledgeObject.id),
)
.where(
ConnectorKnowledgeObject.profile_id.in_(profile_ids),
ConnectorKnowledgeObject.status != "deleted",
)
.group_by(ConnectorKnowledgeObject.profile_id)
)
}
def _latest_knowledge_runs(
session: Session,
profiles: tuple[ConnectorKnowledgeProfile, ...],
) -> dict[str, ConnectorKnowledgeSyncRun]:
profile_ids = tuple(item.id for item in profiles)
if not profile_ids:
return {}
rows = tuple(
session.scalars(
select(ConnectorKnowledgeSyncRun)
.where(
ConnectorKnowledgeSyncRun.profile_id.in_(profile_ids),
ConnectorKnowledgeSyncRun.mode.in_(("backfill", "delta")),
)
.order_by(
ConnectorKnowledgeSyncRun.profile_id,
ConnectorKnowledgeSyncRun.started_at.desc(),
ConnectorKnowledgeSyncRun.id.desc(),
)
)
)
latest: dict[str, ConnectorKnowledgeSyncRun] = {}
for row in rows:
latest.setdefault(row.profile_id, row)
return latest
def _knowledge_state(
profile: ConnectorKnowledgeProfile,
*,
observed_at: datetime,
object_count: int,
latest_run: ConnectorKnowledgeSyncRun | None,
) -> ExternalProviderRuntimeState:
active = profile.status == "active"
health = (
"inactive"
if not active
else "healthy"
if profile.health_status == "healthy"
else "warning"
if profile.health_status in {"unknown", "degraded"}
else "error"
)
last_success = (
latest_run.finished_at
if latest_run is not None and latest_run.status == "completed"
else profile.discovered_at
)
return ExternalProviderRuntimeState(
provider_id=KNOWLEDGE_PROVIDER_ID,
binding_ref=f"connectors:knowledge-profile:{profile.id}",
authority_mode=profile.source_authority_mode,
observed_at=observed_at,
configured=True,
active=active,
health=health,
freshness="unknown" if active else "not_applicable",
conflict=(
"pending"
if latest_run is not None and latest_run.status == "outcome_unknown"
else "not_applicable"
),
recovery=(
"attention"
if latest_run is not None
and latest_run.status in {"failed", "outcome_unknown"}
else "ready"
if active
else "not_applicable"
),
last_success_at=_aware(last_success),
detail=(
f"{profile.product} knowledge profile is synchronized and ACL-rechecked."
if active and profile.health_status == "healthy"
else "Knowledge profile requires discovery, synchronization, or health review."
if active
else "Knowledge profile is paused."
),
metrics={
"product": profile.product,
"product_version": profile.product_version,
"desired_maturity": profile.desired_maturity,
"discovered_maturity": profile.discovered_maturity,
"active_objects": int(object_count),
"last_run_status": latest_run.status if latest_run is not None else None,
},
)
def _sanctions_state( def _sanctions_state(
run: ConnectorSanctionsAcquisitionRun, run: ConnectorSanctionsAcquisitionRun,
*, *,
@@ -200,8 +357,10 @@ def _aware(value: datetime | None) -> datetime | None:
__all__ = [ __all__ = [
"KNOWLEDGE_PROVIDER_ID",
"SANCTIONS_PROVIDER_ID", "SANCTIONS_PROVIDER_ID",
"TABULAR_PROVIDER_ID", "TABULAR_PROVIDER_ID",
"sanctions_provider_states", "sanctions_provider_states",
"knowledge_provider_states",
"tabular_provider_states", "tabular_provider_states",
] ]
+15 -1
View File
@@ -70,7 +70,7 @@ CONNECTOR_RECOVERY_OPERATIONS = (
"unknown outcomes remain unresolved until provider-backed reconciliation", "unknown outcomes remain unresolved until provider-backed reconciliation",
"never retry the same remote effect solely to reconstruct local state", "never retry the same remote effect solely to reconstruct local state",
), ),
implemented=False, implemented=True,
), ),
) )
@@ -262,6 +262,20 @@ class ConnectorExternalMutationRecovery:
if self.operation is not None: if self.operation is not None:
self.operation.succeed(evidence=provider_evidence) self.operation.succeed(evidence=provider_evidence)
def commit_success(
self,
session: Session,
*,
provider_evidence: dict[str, Any],
) -> None:
if self.operation is not None:
self.operation.commit_verified_success(
session,
evidence=provider_evidence,
)
else:
session.commit()
def reject(self, *, summary: str, provider_code: str) -> None: def reject(self, *, summary: str, provider_code: str) -> None:
if self.operation is not None: if self.operation is not None:
self.operation.reject( self.operation.reject(
+656 -7
View File
@@ -22,6 +22,7 @@ from govoplan_core.core.feeds import (
FeedCapabilityError, FeedCapabilityError,
FeedEntry, FeedEntry,
FeedRenderRequest, FeedRenderRequest,
FeedVisibility,
) )
from govoplan_core.core.sanctions import SanctionsSnapshotReference from govoplan_core.core.sanctions import SanctionsSnapshotReference
from govoplan_core.db.session import get_session from govoplan_core.db.session import get_session
@@ -29,7 +30,9 @@ from govoplan_connectors.backend.schemas import (
FeedAcquireRequest, FeedAcquireRequest,
FeedDocumentResponse, FeedDocumentResponse,
FeedImportRequest, FeedImportRequest,
FeedPublicationEntryPayload,
FeedRenderPayload, FeedRenderPayload,
ManagedFileSourceCreateRequest,
SanctionsAcquisitionRunListResponse, SanctionsAcquisitionRunListResponse,
SanctionsAcquisitionRunResponse, SanctionsAcquisitionRunResponse,
SanctionsRefreshResponse, SanctionsRefreshResponse,
@@ -38,6 +41,7 @@ from govoplan_connectors.backend.schemas import (
SanctionsSourceListResponse, SanctionsSourceListResponse,
SanctionsSourceResponse, SanctionsSourceResponse,
SnapshotCreateRequest, SnapshotCreateRequest,
SqlSourceCreateRequest,
TabularColumnResponse, TabularColumnResponse,
TabularHealthResponse, TabularHealthResponse,
TabularPreviewDiagnosticResponse, TabularPreviewDiagnosticResponse,
@@ -47,7 +51,63 @@ from govoplan_connectors.backend.schemas import (
TabularSourcePreviewResponse, TabularSourcePreviewResponse,
TabularSourceResponse, TabularSourceResponse,
) )
from govoplan_connectors.backend.feeds import ConnectorFeedProvider, feed_rows from govoplan_connectors.backend.feeds import (
FEED_PRIVATE_PUBLISH_SCOPE,
FEED_PUBLISH_SCOPE,
ConnectorFeedProvider,
feed_rows,
)
from govoplan_connectors.backend.governed_runtime import (
GovernedConnectorError,
create_configuration,
execute_run,
list_configurations,
list_definitions,
list_runs,
review_run,
update_configuration,
upsert_definition,
)
from govoplan_connectors.backend.governed_schemas import (
ConnectorConfigurationCreateRequest,
ConnectorConfigurationItem,
ConnectorConfigurationListResponse,
ConnectorConfigurationUpdateRequest,
ConnectorDefinitionItem,
ConnectorDefinitionListResponse,
ConnectorDefinitionUpsertRequest,
ConnectorReviewRequest,
ConnectorRunItem,
ConnectorRunListResponse,
ConnectorRunRequest,
)
from govoplan_connectors.backend.knowledge_connector import (
KnowledgeConnectorError,
create_profile,
discover_profile,
list_objects as list_knowledge_objects,
list_profiles as list_knowledge_profiles,
list_runs as list_knowledge_runs,
migration_dry_run,
publish_page,
synchronize_profile,
update_profile,
)
from govoplan_connectors.backend.knowledge_schemas import (
KnowledgeDiscoveryResponse,
KnowledgeMigrationDryRunRequest,
KnowledgeMigrationDryRunResponse,
KnowledgeObjectListResponse,
KnowledgeProfileCreateRequest,
KnowledgeProfileItem,
KnowledgeProfileListResponse,
KnowledgeProfileUpdateRequest,
KnowledgePublishRequest,
KnowledgePublishResponse,
KnowledgeSyncRequest,
KnowledgeSyncRunItem,
KnowledgeSyncRunListResponse,
)
from govoplan_connectors.backend.recovery import ( from govoplan_connectors.backend.recovery import (
ConnectorRecoveryError, ConnectorRecoveryError,
begin_connector_read_snapshot, begin_connector_read_snapshot,
@@ -135,6 +195,54 @@ def _recovery_http_error(exc: ConnectorRecoveryError) -> HTTPException:
) )
def _governed_http_error(exc: GovernedConnectorError) -> HTTPException:
if exc.code.endswith("_not_found"):
status_code = status.HTTP_404_NOT_FOUND
elif exc.code in {
"configuration_conflict",
"configuration_disabled",
"idempotency_conflict",
"local_definition_protected",
"package_definition_requires_overrides",
"run_not_reviewable",
}:
status_code = status.HTTP_409_CONFLICT
else:
status_code = status.HTTP_422_UNPROCESSABLE_CONTENT
return HTTPException(
status_code=status_code,
detail={"code": exc.code, "message": str(exc)},
)
def _knowledge_http_error(exc: KnowledgeConnectorError) -> HTTPException:
if exc.code.endswith("_not_found"):
status_code = status.HTTP_404_NOT_FOUND
elif exc.code == "access_denied":
status_code = status.HTTP_403_FORBIDDEN
elif exc.code in {
"idempotency_conflict",
"operation_unresolved",
"profile_conflict",
"profile_paused",
"publication_conflict",
"publication_reconciliation_required",
}:
status_code = status.HTTP_409_CONFLICT
elif exc.retryable or exc.code.endswith("_unavailable"):
status_code = status.HTTP_503_SERVICE_UNAVAILABLE
else:
status_code = status.HTTP_422_UNPROCESSABLE_CONTENT
return HTTPException(
status_code=status_code,
detail={
"code": exc.code,
"message": str(exc),
"retryable": exc.retryable,
},
)
@router.post("/feeds/preview", response_model=FeedDocumentResponse) @router.post("/feeds/preview", response_model=FeedDocumentResponse)
def api_preview_feed( def api_preview_feed(
payload: FeedAcquireRequest, payload: FeedAcquireRequest,
@@ -284,7 +392,9 @@ def api_render_feed(
payload: FeedRenderPayload, payload: FeedRenderPayload,
principal: ApiPrincipal = Depends(get_api_principal), principal: ApiPrincipal = Depends(get_api_principal),
) -> Response: ) -> Response:
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE) _require_any_scope(principal, FEED_PUBLISH_SCOPE, ADMIN_SCOPE)
if payload.audience != "public":
_require_any_scope(principal, FEED_PRIVATE_PUBLISH_SCOPE, ADMIN_SCOPE)
try: try:
rendered = feed_transport.render( rendered = feed_transport.render(
FeedRenderRequest( FeedRenderRequest(
@@ -295,9 +405,9 @@ def api_render_feed(
description=payload.description, description=payload.description,
language=payload.language, language=payload.language,
entries=tuple( entries=tuple(
FeedEntry(**item.model_dump()) for item in payload.entries _publication_feed_entry(item) for item in payload.entries
), ),
allowed_visibilities=frozenset(payload.allowed_visibilities), allowed_visibilities=_feed_audience_visibilities(payload.audience),
) )
) )
except FeedCapabilityError as exc: except FeedCapabilityError as exc:
@@ -308,10 +418,39 @@ def api_render_feed(
headers={ headers={
"X-GovOPlaN-Feed-Included": str(rendered.included_entries), "X-GovOPlaN-Feed-Included": str(rendered.included_entries),
"X-GovOPlaN-Feed-Excluded": str(rendered.excluded_entries), "X-GovOPlaN-Feed-Excluded": str(rendered.excluded_entries),
"X-GovOPlaN-Feed-Audience": payload.audience,
}, },
) )
def _feed_audience_visibilities(
audience: str,
) -> frozenset[FeedVisibility]:
if audience == "public":
return frozenset({"public"})
if audience == "tenant":
return frozenset({"public", "tenant"})
return frozenset({"public", "tenant", "private"})
def _publication_feed_entry(item: FeedPublicationEntryPayload) -> FeedEntry:
values = item.model_dump(
exclude={"source_kind", "source_module", "source_ref", "source_revision"}
)
values["categories"] = tuple(values["categories"])
values["enclosures"] = tuple(values["enclosures"])
values["metadata"] = {
**values["metadata"],
"govoplan_source": {
"kind": item.source_kind,
"module": item.source_module,
"reference": item.source_ref,
"revision": item.source_revision,
},
}
return FeedEntry(**values)
@router.get("/tabular-sources", response_model=TabularSourceListResponse) @router.get("/tabular-sources", response_model=TabularSourceListResponse)
def api_list_tabular_sources( def api_list_tabular_sources(
query: str = Query(default="", max_length=200), query: str = Query(default="", max_length=200),
@@ -378,12 +517,82 @@ def api_create_tabular_snapshot(
return _source_response(source) return _source_response(source)
@router.post(
"/tabular-sources/files",
response_model=TabularSourceResponse,
status_code=status.HTTP_201_CREATED,
)
def api_create_managed_file_source(
payload: ManagedFileSourceCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> TabularSourceResponse:
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
try:
source = provider.create_file_source(
session,
principal,
name=payload.name,
source_name=payload.source_name,
description=payload.description,
file_asset_id=payload.file_asset_id,
file_version_id=payload.file_version_id,
delimiter=payload.delimiter,
sheet_name=payload.sheet_name,
)
except TabularSourceError as exc:
raise _http_error(exc) from exc
_audit_tabular_source_change(
session,
principal,
source,
operation="created",
)
session.commit()
return _source_response(source)
@router.post(
"/tabular-sources/sql",
response_model=TabularSourceResponse,
status_code=status.HTTP_201_CREATED,
)
def api_create_sql_source(
payload: SqlSourceCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> TabularSourceResponse:
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
try:
source = provider.create_sql_source(
session,
principal,
name=payload.name,
source_name=payload.source_name,
description=payload.description,
configuration_id=payload.configuration_id,
schema_name=payload.schema_name,
table_name=payload.table_name,
)
except TabularSourceError as exc:
raise _http_error(exc) from exc
_audit_tabular_source_change(
session,
principal,
source,
operation="created",
)
session.commit()
return _source_response(source)
@router.get( @router.get(
"/tabular-sources/{source_id}/preview", "/tabular-sources/{source_id}/preview",
response_model=TabularSourcePreviewResponse, response_model=TabularSourcePreviewResponse,
) )
def api_preview_tabular_source( def api_preview_tabular_source(
source_id: str, source_id: str,
kind: Annotated[str, Query(pattern=r"^(snapshot|file|sql)$")] = "snapshot",
limit: int = Query(default=100, ge=1, le=500), limit: int = Query(default=100, ge=1, le=500),
offset: int = Query(default=0, ge=0), offset: int = Query(default=0, ge=0),
max_bytes: int = Query(default=1_000_000, ge=2, le=5_000_000), max_bytes: int = Query(default=1_000_000, ge=2, le=5_000_000),
@@ -397,7 +606,7 @@ def api_preview_tabular_source(
session, session,
principal, principal,
request=TabularReadRequest( request=TabularReadRequest(
source_ref=f"snapshot:{source_id}", source_ref=f"{kind}:{source_id}",
limit=limit, limit=limit,
offset=offset, offset=offset,
max_bytes=max_bytes, max_bytes=max_bytes,
@@ -428,17 +637,47 @@ def api_preview_tabular_source(
) )
@router.post(
"/tabular-sources/{source_id}/refresh",
response_model=TabularSourceResponse,
)
def api_refresh_tabular_source(
source_id: str,
kind: Annotated[str, Query(pattern=r"^(file|sql)$")],
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> TabularSourceResponse:
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
try:
source = provider.refresh_source(
session,
principal,
source_ref=f"{kind}:{source_id}",
)
except TabularSourceError as exc:
raise _http_error(exc) from exc
_audit_tabular_source_change(
session,
principal,
source,
operation="refreshed",
)
session.commit()
return _source_response(source)
@router.delete( @router.delete(
"/tabular-sources/{source_id}", "/tabular-sources/{source_id}",
response_model=TabularSourceDeleteResponse, response_model=TabularSourceDeleteResponse,
) )
def api_delete_tabular_source( def api_delete_tabular_source(
source_id: str, source_id: str,
kind: Annotated[str, Query(pattern=r"^(snapshot|file|sql)$")] = "snapshot",
session: Session = Depends(get_session), session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal), principal: ApiPrincipal = Depends(get_api_principal),
) -> TabularSourceDeleteResponse: ) -> TabularSourceDeleteResponse:
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE) _require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
source_ref = f"snapshot:{source_id}" source_ref = f"{kind}:{source_id}"
try: try:
source = provider.delete_snapshot( source = provider.delete_snapshot(
session, session,
@@ -452,7 +691,7 @@ def api_delete_tabular_source(
tenant_id=principal.tenant_id, tenant_id=principal.tenant_id,
user_id=getattr(principal.user, "id", None), user_id=getattr(principal.user, "id", None),
api_key_id=principal.api_key_id, api_key_id=principal.api_key_id,
action="connectors.tabular_snapshot.deleted", action="connectors.tabular_source.deleted",
object_type="connector_tabular_source", object_type="connector_tabular_source",
object_id=source_ref, object_id=source_ref,
details={"source_name": source.source_name, "fingerprint": source.fingerprint}, details={"source_name": source.source_name, "fingerprint": source.fingerprint},
@@ -609,6 +848,390 @@ def api_list_sanctions_runs(
) )
@router.get(
"/governed/definitions",
response_model=ConnectorDefinitionListResponse,
)
def api_list_governed_definitions(
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorDefinitionListResponse:
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
try:
items = list_definitions(session, tenant_id=principal.tenant_id)
except GovernedConnectorError as exc:
raise _governed_http_error(exc) from exc
return ConnectorDefinitionListResponse(items=items)
@router.post(
"/governed/definitions",
response_model=ConnectorDefinitionItem,
)
def api_upsert_governed_definition(
payload: ConnectorDefinitionUpsertRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorDefinitionItem:
_require_any_scope(principal, ADMIN_SCOPE)
try:
return upsert_definition(session, principal, payload)
except GovernedConnectorError as exc:
session.rollback()
raise _governed_http_error(exc) from exc
@router.get(
"/governed/configurations",
response_model=ConnectorConfigurationListResponse,
)
def api_list_governed_configurations(
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorConfigurationListResponse:
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
try:
items = list_configurations(session, tenant_id=principal.tenant_id)
except GovernedConnectorError as exc:
raise _governed_http_error(exc) from exc
return ConnectorConfigurationListResponse(items=items)
@router.post(
"/governed/configurations",
response_model=ConnectorConfigurationItem,
status_code=status.HTTP_201_CREATED,
)
def api_create_governed_configuration(
payload: ConnectorConfigurationCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorConfigurationItem:
_require_any_scope(principal, ADMIN_SCOPE)
try:
return create_configuration(session, principal, payload)
except GovernedConnectorError as exc:
session.rollback()
raise _governed_http_error(exc) from exc
@router.put(
"/governed/configurations/{configuration_id}",
response_model=ConnectorConfigurationItem,
)
def api_update_governed_configuration(
configuration_id: str,
payload: ConnectorConfigurationUpdateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorConfigurationItem:
_require_any_scope(principal, ADMIN_SCOPE)
try:
return update_configuration(
session,
principal,
configuration_id=configuration_id,
payload=payload,
)
except GovernedConnectorError as exc:
session.rollback()
raise _governed_http_error(exc) from exc
def _api_execute_governed_run(
*,
configuration_id: str,
mode: str,
payload: ConnectorRunRequest,
session: Session,
principal: ApiPrincipal,
) -> ConnectorRunItem:
_require_any_scope(principal, WRITE_SCOPE, ADMIN_SCOPE)
try:
return execute_run(
session,
principal,
configuration_id=configuration_id,
mode=mode,
payload=payload,
)
except GovernedConnectorError as exc:
session.rollback()
raise _governed_http_error(exc) from exc
@router.post(
"/governed/configurations/{configuration_id}/dry-runs",
response_model=ConnectorRunItem,
)
def api_dry_run_governed_configuration(
configuration_id: str,
payload: ConnectorRunRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorRunItem:
return _api_execute_governed_run(
configuration_id=configuration_id,
mode="dry_run",
payload=payload,
session=session,
principal=principal,
)
@router.post(
"/governed/configurations/{configuration_id}/simulations",
response_model=ConnectorRunItem,
)
def api_simulate_governed_configuration(
configuration_id: str,
payload: ConnectorRunRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorRunItem:
return _api_execute_governed_run(
configuration_id=configuration_id,
mode="simulation",
payload=payload,
session=session,
principal=principal,
)
@router.get(
"/governed/runs",
response_model=ConnectorRunListResponse,
)
def api_list_governed_runs(
configuration_id: str | None = Query(default=None),
review_state: str | None = Query(default=None),
limit: int = Query(default=100, ge=1, le=500),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorRunListResponse:
_require_any_scope(principal, READ_SCOPE, ADMIN_SCOPE)
try:
items = list_runs(
session,
tenant_id=principal.tenant_id,
configuration_id=configuration_id,
review_state=review_state,
limit=limit,
)
except GovernedConnectorError as exc:
raise _governed_http_error(exc) from exc
return ConnectorRunListResponse(items=items)
@router.post(
"/governed/runs/{run_id}/review",
response_model=ConnectorRunItem,
)
def api_review_governed_run(
run_id: str,
payload: ConnectorReviewRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> ConnectorRunItem:
_require_any_scope(principal, ADMIN_SCOPE)
try:
return review_run(session, principal, run_id=run_id, payload=payload)
except GovernedConnectorError as exc:
session.rollback()
raise _governed_http_error(exc) from exc
@router.get(
"/knowledge/profiles",
response_model=KnowledgeProfileListResponse,
)
def api_list_knowledge_profiles(
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeProfileListResponse:
try:
return KnowledgeProfileListResponse(
items=list(list_knowledge_profiles(session, principal))
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.post(
"/knowledge/profiles",
response_model=KnowledgeProfileItem,
status_code=status.HTTP_201_CREATED,
)
def api_create_knowledge_profile(
payload: KnowledgeProfileCreateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeProfileItem:
try:
return create_profile(session, principal, payload)
except KnowledgeConnectorError as exc:
session.rollback()
raise _knowledge_http_error(exc) from exc
@router.put(
"/knowledge/profiles/{profile_id}",
response_model=KnowledgeProfileItem,
)
def api_update_knowledge_profile(
profile_id: str,
payload: KnowledgeProfileUpdateRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeProfileItem:
try:
return update_profile(
session,
principal,
profile_id=profile_id,
payload=payload,
)
except KnowledgeConnectorError as exc:
session.rollback()
raise _knowledge_http_error(exc) from exc
@router.post(
"/knowledge/profiles/{profile_id}/discover",
response_model=KnowledgeDiscoveryResponse,
)
def api_discover_knowledge_profile(
profile_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeDiscoveryResponse:
try:
return discover_profile(
session,
principal,
profile_id=profile_id,
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.post(
"/knowledge/profiles/{profile_id}/sync",
response_model=KnowledgeSyncRunItem,
)
def api_synchronize_knowledge_profile(
profile_id: str,
payload: KnowledgeSyncRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeSyncRunItem:
try:
return synchronize_profile(
session,
principal,
profile_id=profile_id,
payload=payload,
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.get(
"/knowledge/profiles/{profile_id}/objects",
response_model=KnowledgeObjectListResponse,
)
def api_list_knowledge_objects(
profile_id: str,
cursor: str | None = Query(default=None, max_length=500),
limit: int = Query(default=100, ge=1, le=500),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeObjectListResponse:
try:
items, next_cursor = list_knowledge_objects(
session,
principal,
profile_id=profile_id,
cursor=cursor,
limit=limit,
)
return KnowledgeObjectListResponse(
items=list(items),
next_cursor=next_cursor,
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.get(
"/knowledge/runs",
response_model=KnowledgeSyncRunListResponse,
)
def api_list_knowledge_runs(
profile_id: str | None = Query(default=None),
limit: int = Query(default=100, ge=1, le=500),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeSyncRunListResponse:
try:
return KnowledgeSyncRunListResponse(
items=list(
list_knowledge_runs(
session,
principal,
profile_id=profile_id,
limit=limit,
)
)
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.post(
"/knowledge/profiles/{profile_id}/migration-dry-runs",
response_model=KnowledgeMigrationDryRunResponse,
)
def api_preview_knowledge_migration(
profile_id: str,
payload: KnowledgeMigrationDryRunRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgeMigrationDryRunResponse:
try:
return migration_dry_run(
session,
principal,
profile_id=profile_id,
payload=payload,
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
@router.post(
"/knowledge/profiles/{profile_id}/pages/{external_page_id}/publish",
response_model=KnowledgePublishResponse,
)
def api_publish_knowledge_page(
profile_id: str,
external_page_id: str,
payload: KnowledgePublishRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> KnowledgePublishResponse:
try:
return publish_page(
session,
principal,
profile_id=profile_id,
external_page_id=external_page_id,
payload=payload,
)
except KnowledgeConnectorError as exc:
raise _knowledge_http_error(exc) from exc
def _source_response(source: TabularSource) -> TabularSourceResponse: def _source_response(source: TabularSource) -> TabularSourceResponse:
return TabularSourceResponse( return TabularSourceResponse(
ref=source.ref, ref=source.ref,
@@ -653,6 +1276,32 @@ def _source_response(source: TabularSource) -> TabularSourceResponse:
) )
def _audit_tabular_source_change(
session: Session,
principal: ApiPrincipal,
source: TabularSource,
*,
operation: str,
) -> None:
audit_event(
session,
tenant_id=principal.tenant_id,
user_id=getattr(principal.user, "id", None),
api_key_id=principal.api_key_id,
action=f"connectors.tabular_source.{operation}",
object_type="connector_tabular_source",
object_id=source.ref,
details={
"provider": source.provider,
"source_name": source.source_name,
"fingerprint": source.fingerprint,
"schema_version": source.schema_version,
"row_count": source.row_count,
"source_mode": source.source_mode,
},
)
def _sanctions_snapshot_response( def _sanctions_snapshot_response(
snapshot: SanctionsSnapshotReference, snapshot: SanctionsSnapshotReference,
) -> SanctionsSnapshotResponse: ) -> SanctionsSnapshotResponse:
+51 -4
View File
@@ -36,6 +36,17 @@ class FeedEntryPayload(BaseModel):
metadata: dict[str, Any] = Field(default_factory=dict) metadata: dict[str, Any] = Field(default_factory=dict)
class FeedPublicationEntryPayload(FeedEntryPayload):
source_kind: Literal["event", "publication", "case", "report"]
source_module: str = Field(
min_length=1,
max_length=100,
pattern=r"^[a-z][a-z0-9_]*$",
)
source_ref: str = Field(min_length=1, max_length=500)
source_revision: str | None = Field(default=None, max_length=200)
class FeedDocumentResponse(BaseModel): class FeedDocumentResponse(BaseModel):
format: Literal["rss", "atom"] format: Literal["rss", "atom"]
title: str title: str
@@ -61,11 +72,11 @@ class FeedRenderPayload(BaseModel):
home_url: str = Field(min_length=1, max_length=2000) home_url: str = Field(min_length=1, max_length=2000)
description: str | None = None description: str | None = None
language: str | None = Field(default=None, max_length=100) language: str | None = Field(default=None, max_length=100)
entries: list[FeedEntryPayload] = Field(default_factory=list, max_length=10_000) entries: list[FeedPublicationEntryPayload] = Field(
allowed_visibilities: list[Literal["public", "tenant", "private"]] = Field( default_factory=list,
default_factory=lambda: ["public"], max_length=10_000,
max_length=3,
) )
audience: Literal["public", "tenant", "private"] = "public"
class SnapshotCreateRequest(BaseModel): class SnapshotCreateRequest(BaseModel):
@@ -94,6 +105,40 @@ class SnapshotCreateRequest(BaseModel):
return self return self
class ManagedFileSourceCreateRequest(BaseModel):
name: str = Field(min_length=1, max_length=300)
source_name: str = Field(
min_length=1,
max_length=120,
pattern=r"^[A-Za-z_][A-Za-z0-9_]*$",
)
description: str | None = Field(default=None, max_length=4000)
file_asset_id: str = Field(min_length=1, max_length=36)
file_version_id: str | None = Field(default=None, min_length=1, max_length=36)
delimiter: Literal[",", ";", "\t", "|"] = ","
sheet_name: str | None = Field(default=None, min_length=1, max_length=255)
class SqlSourceCreateRequest(BaseModel):
name: str = Field(min_length=1, max_length=300)
source_name: str = Field(
min_length=1,
max_length=120,
pattern=r"^[A-Za-z_][A-Za-z0-9_]*$",
)
description: str | None = Field(default=None, max_length=4000)
configuration_id: str = Field(min_length=1, max_length=36)
schema_name: str | None = Field(
default=None,
pattern=r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$",
)
table_name: str = Field(
min_length=1,
max_length=128,
pattern=r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$",
)
class TabularColumnResponse(BaseModel): class TabularColumnResponse(BaseModel):
name: str name: str
data_type: str data_type: str
@@ -239,7 +284,9 @@ __all__ = [
"FeedEntryPayload", "FeedEntryPayload",
"FeedImportRequest", "FeedImportRequest",
"FeedRenderPayload", "FeedRenderPayload",
"ManagedFileSourceCreateRequest",
"SnapshotCreateRequest", "SnapshotCreateRequest",
"SqlSourceCreateRequest",
"SanctionsAcquisitionRunListResponse", "SanctionsAcquisitionRunListResponse",
"SanctionsAcquisitionRunResponse", "SanctionsAcquisitionRunResponse",
"SanctionsRefreshResponse", "SanctionsRefreshResponse",
@@ -0,0 +1,849 @@
from __future__ import annotations
import hashlib
import json
import math
import re
import zipfile
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from datetime import date, datetime
from decimal import Decimal
from io import BytesIO
from typing import Any
from openpyxl import load_workbook
from sqlalchemy import (
JSON,
BigInteger,
Boolean,
Date,
DateTime,
Float,
Integer,
LargeBinary,
MetaData,
Numeric,
String,
Table,
Text,
create_engine,
func,
select,
)
from sqlalchemy.engine import Engine, URL, make_url
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.files import (
ManagedTabularFileAccessError,
ManagedTabularFileError,
ManagedTabularFileValidationError,
managed_tabular_file_provider,
)
from govoplan_core.core.tabular_sources import (
TabularColumn,
TabularPreviewDiagnostic,
TabularPushdown,
TabularSourceHealth,
TabularSourceUnavailableError,
TabularSourceValidationError,
parse_tabular_csv,
)
from govoplan_core.security.credential_envelopes import (
CredentialAccessContext,
CredentialEnvelopeError,
resolve_credential_envelope,
)
from govoplan_core.security.redaction import is_sensitive_key
from govoplan_connectors.backend.db.models import ConnectorConfiguration
MAX_FILE_BYTES = 5_000_000
MAX_FILE_ROWS = 10_000
MAX_FILE_COLUMNS = 500
MAX_XLSX_ENTRIES = 5_000
MAX_XLSX_EXPANDED_BYTES = 50_000_000
MAX_XLSX_COMPRESSION_RATIO = 100
POSTGRESQL_SCHEMES = frozenset({"postgresql", "postgresql+psycopg"})
_IDENTIFIER = re.compile(r"^[A-Za-z_][A-Za-z0-9_$-]{0,127}$")
@dataclass(frozen=True, slots=True)
class TabularOriginInspection:
provider: str
schema: tuple[TabularColumn, ...]
fingerprint: str
row_count: int
byte_count: int
metadata: Mapping[str, object]
health: TabularSourceHealth
pushdown: TabularPushdown
rows: tuple[Mapping[str, object], ...] = ()
diagnostics: tuple[TabularPreviewDiagnostic, ...] = ()
@dataclass(frozen=True, slots=True)
class TabularOriginRead:
inspection: TabularOriginInspection
rows: tuple[Mapping[str, object], ...]
total_rows: int
diagnostics: tuple[TabularPreviewDiagnostic, ...] = ()
class ManagedFileTabularAdapter:
def __init__(self, registry: object | None) -> None:
self._registry = registry
def inspect(
self,
session: Session,
principal: ApiPrincipal,
*,
file_asset_id: str,
file_version_id: str | None,
delimiter: str = ",",
sheet_name: str | None = None,
) -> TabularOriginInspection:
provider = managed_tabular_file_provider(self._registry)
if provider is None:
raise TabularSourceUnavailableError(
"Managed file sources require the Files module."
)
try:
metadata = provider.get_tabular_file(
session,
principal,
file_asset_id=file_asset_id,
file_version_id=file_version_id,
)
if metadata is None:
raise TabularSourceUnavailableError(
"Managed tabular file or version is unavailable."
)
content = provider.read_tabular_file(
session,
principal,
file_asset_id=file_asset_id,
file_version_id=metadata.file_version_id,
max_bytes=MAX_FILE_BYTES,
)
except ManagedTabularFileAccessError as exc:
raise TabularSourceUnavailableError(
"Managed tabular file access is no longer authorized."
) from exc
except ManagedTabularFileValidationError as exc:
raise TabularSourceValidationError(str(exc)) from exc
except ManagedTabularFileError as exc:
raise TabularSourceUnavailableError(str(exc)) from exc
rows, resolved_sheet = parse_managed_tabular_content(
content.payload,
filename=content.file.filename,
content_type=content.file.content_type,
delimiter=delimiter,
sheet_name=sheet_name,
)
schema = infer_tabular_schema(rows)
fingerprint = origin_fingerprint(
schema,
tokens=(
"managed_file",
content.file.file_asset_id,
content.file.file_version_id,
content.file.sha256,
resolved_sheet or "",
delimiter,
),
)
try:
current = provider.get_tabular_file(
session,
principal,
file_asset_id=file_asset_id,
)
except ManagedTabularFileAccessError as exc:
raise TabularSourceUnavailableError(
"Managed tabular file access is no longer authorized."
) from exc
except ManagedTabularFileError as exc:
raise TabularSourceUnavailableError(str(exc)) from exc
changed = bool(
current is not None
and current.file_version_id != content.file.file_version_id
)
health = TabularSourceHealth(
status="warning" if changed else "healthy",
code=(
"files.newer_version_available"
if changed
else "files.exact_version_ready"
),
summary=(
"A newer managed file version is available for explicit review."
if changed
else "The exact managed file version passed access and integrity checks."
),
checked_at=content.file.updated_at,
details={
"pinned_version_id": content.file.file_version_id,
"current_version_id": (
current.file_version_id if current is not None else None
),
},
)
return TabularOriginInspection(
provider="managed_file",
schema=schema,
fingerprint=fingerprint,
row_count=len(rows),
byte_count=len(content.payload),
metadata={
"origin_kind": "managed_file",
"file_asset_id": content.file.file_asset_id,
"file_version_id": content.file.file_version_id,
"file_sha256": content.file.sha256,
"filename": content.file.filename,
"content_type": content.file.content_type,
"format": "xlsx" if _is_xlsx(content.file.filename, content.file.content_type) else "csv",
"delimiter": delimiter,
"sheet_name": resolved_sheet,
},
health=health,
pushdown=TabularPushdown(projections=True, pagination=True),
rows=rows,
diagnostics=(
(
TabularPreviewDiagnostic(
severity="warning",
code="files.newer_version_available",
message=(
"The preview remains pinned to the reviewed file version; "
"refresh the source to adopt the newer version."
),
details=dict(health.details),
),
)
if changed
else ()
),
)
def read(
self,
session: Session,
principal: ApiPrincipal,
*,
metadata: Mapping[str, object],
) -> TabularOriginRead:
inspection = self.inspect(
session,
principal,
file_asset_id=_required_metadata(metadata, "file_asset_id"),
file_version_id=_required_metadata(metadata, "file_version_id"),
delimiter=str(metadata.get("delimiter") or ","),
sheet_name=_optional_text(metadata.get("sheet_name")),
)
expected_sha256 = _required_metadata(metadata, "file_sha256")
if inspection.metadata.get("file_sha256") != expected_sha256:
raise TabularSourceUnavailableError(
"Managed file content no longer matches its exact-version checksum."
)
return TabularOriginRead(
inspection=inspection,
rows=inspection.rows,
total_rows=inspection.row_count,
diagnostics=inspection.diagnostics,
)
class PostgresqlTabularAdapter:
"""Read-only SQLAlchemy adapter with a production PostgreSQL allow-list."""
def __init__(
self,
*,
engine_factory: Callable[[URL], Engine] | None = None,
allow_sqlite_for_tests: bool = False,
) -> None:
self._engine_factory = engine_factory or (
lambda url: create_engine(url, pool_pre_ping=True)
)
self._allow_sqlite_for_tests = allow_sqlite_for_tests
def inspect(
self,
session: Session,
principal: ApiPrincipal,
*,
configuration_id: str,
table_name: str,
schema_name: str | None = None,
timeout_ms: int = 2_000,
) -> TabularOriginInspection:
configuration, url, credential_revision = self._connection(
session,
principal,
configuration_id=configuration_id,
)
table_name = _sql_identifier(table_name, "table")
schema_name = (
_sql_identifier(schema_name, "schema") if schema_name else None
)
engine = self._engine_factory(_bounded_connection_url(url, timeout_ms))
try:
with engine.connect() as connection:
_configure_read_only(connection, url, timeout_ms)
table = Table(
table_name,
MetaData(),
schema=schema_name,
autoload_with=connection,
)
schema = tuple(_sql_column(column) for column in table.columns)
if not schema:
raise TabularSourceValidationError(
"SQL tabular sources require at least one column."
)
row_count = int(
connection.execute(select(func.count()).select_from(table)).scalar_one()
)
except TabularSourceValidationError:
raise
except SQLAlchemyError as exc:
raise TabularSourceUnavailableError(
"SQL source discovery failed; verify the active configuration, credential, table, and provider health."
) from exc
finally:
engine.dispose()
fingerprint = origin_fingerprint(
schema,
tokens=(
"postgresql",
configuration.id,
configuration.effective_hash,
credential_revision or "",
schema_name or "",
table_name,
),
)
return TabularOriginInspection(
provider="postgresql",
schema=schema,
fingerprint=fingerprint,
row_count=row_count,
byte_count=0,
metadata={
"origin_kind": "sql",
"configuration_id": configuration.id,
"configuration_revision": configuration.resource_revision,
"configuration_hash": configuration.effective_hash,
"credential_revision": credential_revision,
"schema_name": schema_name,
"table_name": table_name,
},
health=TabularSourceHealth(
status="healthy",
code="sql.source_ready",
summary="The governed PostgreSQL source is reachable and its schema was discovered.",
checked_at=configuration.updated_at,
details={
"configuration_id": configuration.id,
"configuration_revision": configuration.resource_revision,
},
),
pushdown=TabularPushdown(projections=True, pagination=True),
)
def read(
self,
session: Session,
principal: ApiPrincipal,
*,
metadata: Mapping[str, object],
columns: Sequence[str],
offset: int,
limit: int,
timeout_ms: int,
) -> TabularOriginRead:
configuration_id = _required_metadata(metadata, "configuration_id")
current_configuration, _current_url, current_credential_revision = (
self._connection(
session,
principal,
configuration_id=configuration_id,
)
)
if current_configuration.effective_hash != metadata.get(
"configuration_hash"
):
raise TabularSourceValidationError(
"The SQL connector configuration changed; refresh the source before previewing it."
)
if current_credential_revision != metadata.get("credential_revision"):
raise TabularSourceValidationError(
"The SQL source credential changed; refresh the source before previewing it."
)
inspection = self.inspect(
session,
principal,
configuration_id=configuration_id,
table_name=_required_metadata(metadata, "table_name"),
schema_name=_optional_text(metadata.get("schema_name")),
timeout_ms=timeout_ms,
)
if inspection.fingerprint != metadata.get("discovery_fingerprint"):
raise TabularSourceValidationError(
"The SQL source schema drifted; refresh and review the source before previewing it."
)
configuration, url, _credential_revision = self._connection(
session,
principal,
configuration_id=configuration_id,
)
engine = self._engine_factory(_bounded_connection_url(url, timeout_ms))
try:
with engine.connect() as connection:
_configure_read_only(connection, url, timeout_ms)
table = Table(
_required_metadata(metadata, "table_name"),
MetaData(),
schema=_optional_text(metadata.get("schema_name")),
autoload_with=connection,
)
selected_names = tuple(dict.fromkeys(str(item) for item in columns))
selected = (
[table.c[name] for name in selected_names]
if selected_names
else list(table.columns)
)
statement = select(*selected).offset(max(0, int(offset))).limit(
max(1, int(limit))
)
rows = tuple(
_json_row(dict(row._mapping))
for row in connection.execute(statement)
)
except KeyError as exc:
raise TabularSourceValidationError(
f"Unknown SQL source column: {exc.args[0]}"
) from exc
except SQLAlchemyError as exc:
raise TabularSourceUnavailableError(
"SQL source preview failed or exceeded its provider budget."
) from exc
finally:
engine.dispose()
return TabularOriginRead(
inspection=inspection,
rows=rows,
total_rows=inspection.row_count,
)
def _connection(
self,
session: Session,
principal: ApiPrincipal,
*,
configuration_id: str,
) -> tuple[ConnectorConfiguration, URL, str | None]:
configuration = session.scalar(
select(ConnectorConfiguration).where(
ConnectorConfiguration.id == configuration_id,
ConnectorConfiguration.tenant_id == principal.tenant_id,
)
)
if configuration is None:
raise TabularSourceUnavailableError(
"SQL connector configuration is unavailable."
)
if configuration.status != "active":
raise TabularSourceUnavailableError(
"SQL connector configuration is not active."
)
endpoint = _optional_text(configuration.endpoint_url)
if not endpoint:
raise TabularSourceUnavailableError(
"SQL connector configuration has no endpoint."
)
specification = dict(configuration.effective_configuration or {})
provider = str(specification.get("provider") or "").strip().casefold()
protocol = str(specification.get("protocol") or "").strip().casefold()
if provider not in {"postgres", "postgresql", "sql"} or protocol not in {
"postgres",
"postgresql",
"sql",
}:
raise TabularSourceValidationError(
"The selected connector configuration is not a PostgreSQL tabular reader."
)
try:
url = make_url(endpoint)
except (TypeError, ValueError) as exc:
raise TabularSourceValidationError(
"SQL connector endpoint is invalid."
) from exc
if url.username or url.password:
raise TabularSourceValidationError(
"SQL connector endpoints must not contain credentials."
)
if any(is_sensitive_key(key) for key in url.query):
raise TabularSourceValidationError(
"SQL connector endpoint query parameters must not contain credentials."
)
allowed_schemes = set(POSTGRESQL_SCHEMES)
if self._allow_sqlite_for_tests:
allowed_schemes.update({"sqlite", "sqlite+pysqlite"})
if url.drivername not in allowed_schemes:
raise TabularSourceValidationError(
"Only the governed PostgreSQL tabular adapter is enabled."
)
credential_revision: str | None = None
credential_ref = _optional_text(configuration.credential_ref)
if credential_ref:
try:
credential = resolve_credential_envelope(
session,
credential_id=credential_ref,
context=CredentialAccessContext(
tenant_id=principal.tenant_id,
user_id=_principal_user_id(principal),
group_ids=frozenset(principal.principal.group_ids),
target_scope_type="tenant",
target_scope_id=principal.tenant_id,
module_id="connectors",
server_ref=endpoint,
),
)
except CredentialEnvelopeError as exc:
raise TabularSourceUnavailableError(
"SQL source credential is unavailable, inactive, or outside its allowed scope."
) from exc
public_data = dict(credential.public_data)
secret_data = dict(credential.secret_data)
username = _optional_text(
public_data.get("username")
or secret_data.get("username")
or secret_data.get("user")
)
password = _optional_text(secret_data.get("password"))
if url.drivername in POSTGRESQL_SCHEMES and (not username or not password):
raise TabularSourceUnavailableError(
"SQL source credential does not provide a username and password."
)
url = url.set(username=username, password=password)
credential_revision = credential.revision
elif url.drivername in POSTGRESQL_SCHEMES:
raise TabularSourceUnavailableError(
"PostgreSQL tabular sources require a credential envelope reference."
)
return configuration, url, credential_revision
def parse_managed_tabular_content(
payload: bytes,
*,
filename: str,
content_type: str | None,
delimiter: str,
sheet_name: str | None,
) -> tuple[tuple[Mapping[str, object], ...], str | None]:
if len(payload) > MAX_FILE_BYTES:
raise TabularSourceValidationError(
f"Managed tabular files are limited to {MAX_FILE_BYTES:,} bytes."
)
if _is_xlsx(filename, content_type):
return _parse_xlsx(payload, sheet_name=sheet_name)
try:
text = payload.decode("utf-8-sig")
except UnicodeDecodeError as exc:
raise TabularSourceValidationError(
"Managed CSV files must use UTF-8 encoding."
) from exc
return (
tuple(parse_tabular_csv(text, delimiter=delimiter, max_rows=MAX_FILE_ROWS)),
None,
)
def _parse_xlsx(
payload: bytes,
*,
sheet_name: str | None,
) -> tuple[tuple[Mapping[str, object], ...], str]:
_validate_xlsx_archive(payload)
try:
workbook = load_workbook(
BytesIO(payload),
read_only=True,
data_only=True,
keep_links=False,
)
except Exception as exc:
raise TabularSourceValidationError(
"Managed XLSX content could not be parsed."
) from exc
try:
available = tuple(workbook.sheetnames)
if not available:
raise TabularSourceValidationError(
"Managed XLSX content requires at least one worksheet."
)
selected_name = _optional_text(sheet_name) or available[0]
if selected_name not in available:
raise TabularSourceValidationError(
f"Managed XLSX worksheet {selected_name!r} was not found."
)
worksheet = workbook[selected_name]
iterator = worksheet.iter_rows(values_only=True)
try:
raw_headers = next(iterator)
except StopIteration as exc:
raise TabularSourceValidationError(
"Managed XLSX worksheet requires a header row."
) from exc
headers = _xlsx_headers(raw_headers)
rows: list[Mapping[str, object]] = []
for values in iterator:
if len(rows) >= MAX_FILE_ROWS:
raise TabularSourceValidationError(
f"Managed XLSX worksheets are limited to {MAX_FILE_ROWS:,} data rows."
)
normalized = tuple(values[: len(headers)])
if all(value in (None, "") for value in normalized):
continue
rows.append(
{
header: _json_value(
normalized[index] if index < len(normalized) else None
)
for index, header in enumerate(headers)
}
)
return tuple(rows), selected_name
finally:
workbook.close()
def _validate_xlsx_archive(payload: bytes) -> None:
try:
with zipfile.ZipFile(BytesIO(payload)) as archive:
entries = archive.infolist()
if len(entries) > MAX_XLSX_ENTRIES:
raise TabularSourceValidationError(
"Managed XLSX content contains too many archive entries."
)
expanded = sum(max(0, item.file_size) for item in entries)
if expanded > MAX_XLSX_EXPANDED_BYTES:
raise TabularSourceValidationError(
"Managed XLSX content exceeds the expanded-size limit."
)
compressed = sum(max(1, item.compress_size) for item in entries)
if expanded > compressed * MAX_XLSX_COMPRESSION_RATIO:
raise TabularSourceValidationError(
"Managed XLSX content exceeds the compression-ratio limit."
)
if any(
item.filename.startswith(("/", "\\"))
or ".." in item.filename.replace("\\", "/").split("/")
for item in entries
):
raise TabularSourceValidationError(
"Managed XLSX content contains an unsafe archive path."
)
except zipfile.BadZipFile as exc:
raise TabularSourceValidationError(
"Managed XLSX content is not a valid workbook archive."
) from exc
def _xlsx_headers(values: Sequence[object]) -> tuple[str, ...]:
if len(values) > MAX_FILE_COLUMNS:
raise TabularSourceValidationError(
f"Managed XLSX worksheets are limited to {MAX_FILE_COLUMNS:,} columns."
)
headers = tuple(str(value or "").strip() for value in values)
while headers and not headers[-1]:
headers = headers[:-1]
if not headers or any(not header for header in headers):
raise TabularSourceValidationError(
"Managed XLSX content requires a non-empty header row."
)
if len(set(headers)) != len(headers):
raise TabularSourceValidationError(
"Managed XLSX column names must be unique."
)
return headers
def infer_tabular_schema(
rows: Sequence[Mapping[str, object]],
) -> tuple[TabularColumn, ...]:
names: list[str] = []
for row in rows:
for name in row:
if name not in names:
names.append(name)
result: list[TabularColumn] = []
for name in names:
values = [row.get(name) for row in rows]
concrete = [value for value in values if value is not None]
data_type = _type_name(concrete[0]) if concrete else "unknown"
if any(_type_name(value) != data_type for value in concrete[1:]):
data_type = "mixed"
result.append(
TabularColumn(
name=name,
data_type=data_type,
nullable=len(concrete) != len(values),
)
)
return tuple(result)
def origin_fingerprint(
schema: Sequence[TabularColumn],
*,
tokens: Sequence[str],
) -> str:
payload = {
"schema": [
{
"name": column.name,
"data_type": column.data_type,
"nullable": column.nullable,
}
for column in schema
],
"tokens": list(tokens),
}
return hashlib.sha256(
json.dumps(payload, sort_keys=True, separators=(",", ":")).encode("utf-8")
).hexdigest()
def _sql_column(column: Any) -> TabularColumn:
column_type = column.type
if isinstance(column_type, Boolean):
data_type = "boolean"
elif isinstance(column_type, (Integer, BigInteger)):
data_type = "integer"
elif isinstance(column_type, (Numeric, Float)):
data_type = "number"
elif isinstance(column_type, DateTime):
data_type = "datetime"
elif isinstance(column_type, Date):
data_type = "date"
elif isinstance(column_type, JSON):
data_type = "object"
elif isinstance(column_type, LargeBinary):
data_type = "binary"
elif isinstance(column_type, (String, Text)):
data_type = "string"
else:
data_type = str(column_type).casefold()
return TabularColumn(
name=str(column.name),
data_type=data_type,
nullable=bool(column.nullable),
)
def _configure_read_only(connection: Any, url: URL, timeout_ms: int) -> None:
if url.drivername not in POSTGRESQL_SCHEMES:
return
effective_timeout = max(1, min(int(timeout_ms), 30_000))
connection.exec_driver_sql("SET TRANSACTION READ ONLY")
connection.exec_driver_sql(f"SET LOCAL statement_timeout = {effective_timeout}")
def _bounded_connection_url(url: URL, timeout_ms: int) -> URL:
if url.drivername not in POSTGRESQL_SCHEMES:
return url
connect_timeout = max(1, math.ceil(min(int(timeout_ms), 30_000) / 1_000))
return url.update_query_dict({"connect_timeout": str(connect_timeout)})
def _sql_identifier(value: object, label: str) -> str:
normalized = str(value or "").strip()
if not _IDENTIFIER.fullmatch(normalized):
raise TabularSourceValidationError(
f"SQL {label} names must use a simple identifier."
)
return normalized
def _required_metadata(metadata: Mapping[str, object], key: str) -> str:
value = _optional_text(metadata.get(key))
if not value:
raise TabularSourceValidationError(
f"Tabular source metadata is missing {key}."
)
return value
def _optional_text(value: object | None) -> str | None:
normalized = str(value or "").strip()
return normalized or None
def _principal_user_id(principal: ApiPrincipal) -> str | None:
return _optional_text(getattr(principal.user, "id", None))
def _is_xlsx(filename: str, content_type: str | None) -> bool:
normalized_type = str(content_type or "").split(";", 1)[0].strip().casefold()
return str(filename or "").strip().casefold().endswith(".xlsx") or normalized_type == (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
)
def _json_row(row: Mapping[str, object]) -> dict[str, object]:
return {str(key): _json_value(value) for key, value in row.items()}
def _json_value(value: object) -> object:
if value is None or isinstance(value, (str, bool, int)):
return value
if isinstance(value, float):
if not math.isfinite(value):
return str(value)
return value
if isinstance(value, Decimal):
return float(value)
if isinstance(value, (datetime, date)):
return value.isoformat()
if isinstance(value, bytes):
return value.hex()
if isinstance(value, (list, dict)):
return json.loads(json.dumps(value, default=str))
return str(value)
def _type_name(value: object) -> str:
if isinstance(value, bool):
return "boolean"
if isinstance(value, int):
return "integer"
if isinstance(value, (float, Decimal)):
return "number"
if isinstance(value, str):
return "string"
if isinstance(value, list):
return "array"
if isinstance(value, dict):
return "object"
return type(value).__name__.casefold()
__all__ = [
"MAX_FILE_BYTES",
"MAX_FILE_ROWS",
"ManagedFileTabularAdapter",
"PostgresqlTabularAdapter",
"TabularOriginInspection",
"TabularOriginRead",
"infer_tabular_schema",
"origin_fingerprint",
"parse_managed_tabular_content",
]
@@ -27,8 +27,14 @@ from govoplan_core.core.tabular_sources import (
TabularSourceValidationError, TabularSourceValidationError,
parse_tabular_csv, parse_tabular_csv,
) )
from govoplan_core.core.runtime import get_registry
from govoplan_core.db.base import utcnow from govoplan_core.db.base import utcnow
from govoplan_connectors.backend.db.models import ConnectorTabularSource from govoplan_connectors.backend.db.models import ConnectorTabularSource
from govoplan_connectors.backend.tabular_adapters import (
ManagedFileTabularAdapter,
PostgresqlTabularAdapter,
TabularOriginInspection,
)
READ_SCOPE = "connectors:source:read" READ_SCOPE = "connectors:source:read"
@@ -42,8 +48,16 @@ MAX_READ_TIMEOUT_MS = 2_000
class SqlTabularSourceProvider: class SqlTabularSourceProvider:
def __init__(self, *, clock: Callable[[], float] = time.monotonic) -> None: def __init__(
self,
*,
registry: object | None = None,
clock: Callable[[], float] = time.monotonic,
sql_adapter: PostgresqlTabularAdapter | None = None,
) -> None:
self._registry = registry
self._clock = clock self._clock = clock
self._sql_adapter = sql_adapter or PostgresqlTabularAdapter()
def list_sources( def list_sources(
self, self,
@@ -109,7 +123,6 @@ class SqlTabularSourceProvider:
"The source fingerprint changed; refresh the source node before running it." "The source fingerprint changed; refresh the source node before running it."
) )
started = self._clock()
limit = max(1, min(int(request.limit), MAX_READ_ROWS)) limit = max(1, min(int(request.limit), MAX_READ_ROWS))
byte_limit = max(2, min(int(request.max_bytes), MAX_READ_BYTES)) byte_limit = max(2, min(int(request.max_bytes), MAX_READ_BYTES))
timeout_ms = max(1, min(int(request.timeout_ms), MAX_READ_TIMEOUT_MS)) timeout_ms = max(1, min(int(request.timeout_ms), MAX_READ_TIMEOUT_MS))
@@ -149,10 +162,51 @@ class SqlTabularSourceProvider:
raise TabularSourceValidationError( raise TabularSourceValidationError(
f"Unknown source columns: {', '.join(unknown_columns)}" f"Unknown source columns: {', '.join(unknown_columns)}"
) )
started = self._clock()
source_rows: Sequence[Mapping[str, object]]
total_rows = item.row_count
source = _source_dto(item)
base_offset = offset
if item.provider == "managed_file":
read = self._file_adapter().read(
db,
api_principal,
metadata=item.metadata_,
)
if read.inspection.fingerprint != item.fingerprint:
raise TabularSourceValidationError(
"The managed file source changed; refresh and review it before previewing."
)
source_rows = read.rows[offset:]
total_rows = read.total_rows
source = _source_dto(item, inspection=read.inspection)
diagnostics.extend(read.diagnostics)
elif item.provider == "postgresql":
read = self._sql_adapter.read(
db,
api_principal,
metadata=item.metadata_,
columns=selected_columns,
offset=offset,
limit=limit + 1,
timeout_ms=timeout_ms,
)
source_rows = read.rows
total_rows = read.total_rows
source = _source_dto(item, inspection=read.inspection)
diagnostics.extend(read.diagnostics)
else:
source_rows = item.rows[offset:]
if int(max(0.0, self._clock() - started) * 1_000) >= timeout_ms:
raise TabularSourceUnavailableError(
"Tabular source preview exceeded its time budget."
)
rows: list[dict[str, object]] = [] rows: list[dict[str, object]] = []
returned_bytes = 2 returned_bytes = 2
stopped_for = "" stopped_for = ""
for row in item.rows[offset:]: for row in source_rows:
if len(rows) >= limit: if len(rows) >= limit:
stopped_for = "rows" stopped_for = "rows"
break break
@@ -203,10 +257,10 @@ class SqlTabularSourceProvider:
) )
) )
return TabularReadResult( return TabularReadResult(
source=_source_dto(item), source=source,
rows=tuple(rows), rows=tuple(rows),
total_rows=item.row_count, total_rows=total_rows,
truncated=offset + len(rows) < item.row_count, truncated=base_offset + len(rows) < total_rows,
returned_bytes=returned_bytes, returned_bytes=returned_bytes,
elapsed_ms=elapsed_ms, elapsed_ms=elapsed_ms,
effective_row_limit=limit, effective_row_limit=limit,
@@ -215,6 +269,173 @@ class SqlTabularSourceProvider:
diagnostics=tuple(diagnostics), diagnostics=tuple(diagnostics),
) )
def create_file_source(
self,
session: object,
principal: object,
*,
name: str,
source_name: str,
file_asset_id: str,
file_version_id: str | None = None,
description: str | None = None,
delimiter: str = ",",
sheet_name: str | None = None,
) -> TabularSource:
db, api_principal = _context(session, principal, WRITE_SCOPE)
inspection = self._file_adapter().inspect(
db,
api_principal,
file_asset_id=file_asset_id,
file_version_id=file_version_id,
delimiter=delimiter,
sheet_name=sheet_name,
)
return self._create_origin(
db,
api_principal,
name=name,
source_name=source_name,
description=description,
inspection=inspection,
)
def create_sql_source(
self,
session: object,
principal: object,
*,
name: str,
source_name: str,
configuration_id: str,
table_name: str,
schema_name: str | None = None,
description: str | None = None,
) -> TabularSource:
db, api_principal = _context(session, principal, WRITE_SCOPE)
inspection = self._sql_adapter.inspect(
db,
api_principal,
configuration_id=configuration_id,
table_name=table_name,
schema_name=schema_name,
timeout_ms=MAX_READ_TIMEOUT_MS,
)
return self._create_origin(
db,
api_principal,
name=name,
source_name=source_name,
description=description,
inspection=inspection,
)
def refresh_source(
self,
session: object,
principal: object,
*,
source_ref: str,
) -> TabularSource:
db, api_principal = _context(session, principal, WRITE_SCOPE)
item = _source_record(
db,
tenant_id=api_principal.tenant_id,
source_ref=source_ref,
)
if item is None:
raise TabularSourceNotFoundError("Tabular source not found.")
if item.provider == "managed_file":
inspection = self._file_adapter().inspect(
db,
api_principal,
file_asset_id=str(item.metadata_.get("file_asset_id") or ""),
file_version_id=None,
delimiter=str(item.metadata_.get("delimiter") or ","),
sheet_name=_clean_optional(item.metadata_.get("sheet_name")),
)
elif item.provider == "postgresql":
inspection = self._sql_adapter.inspect(
db,
api_principal,
configuration_id=str(
item.metadata_.get("configuration_id") or ""
),
table_name=str(item.metadata_.get("table_name") or ""),
schema_name=_clean_optional(item.metadata_.get("schema_name")),
timeout_ms=MAX_READ_TIMEOUT_MS,
)
else:
raise TabularSourceValidationError(
"Immutable snapshots cannot be refreshed; import a replacement snapshot."
)
item.schema_version += 1
item.schema_ = [_column_payload(column) for column in inspection.schema]
item.fingerprint = inspection.fingerprint
item.row_count = inspection.row_count
item.byte_count = inspection.byte_count
item.metadata_ = {
**dict(inspection.metadata),
"discovery_fingerprint": inspection.fingerprint,
}
item.updated_by = _actor_id(api_principal)
db.flush()
return _source_dto(item, inspection=inspection)
def _create_origin(
self,
db: Session,
principal: ApiPrincipal,
*,
name: str,
source_name: str,
description: str | None,
inspection: TabularOriginInspection,
) -> TabularSource:
clean_name = str(name or "").strip()
clean_source_name = str(source_name or "").strip()
if not clean_name or not clean_source_name:
raise TabularSourceValidationError(
"Tabular source name and source identifier are required."
)
existing = db.scalar(
select(ConnectorTabularSource.id).where(
ConnectorTabularSource.tenant_id == principal.tenant_id,
ConnectorTabularSource.source_name == clean_source_name,
)
)
if existing is not None:
raise TabularSourceValidationError(
f"A tabular source named {clean_source_name!r} already exists."
)
actor_id = _actor_id(principal)
item = ConnectorTabularSource(
tenant_id=principal.tenant_id,
provider=inspection.provider,
source_name=clean_source_name,
name=clean_name,
description=_clean_optional(description),
status="active",
schema_version=1,
schema_=[_column_payload(column) for column in inspection.schema],
rows=[],
fingerprint=inspection.fingerprint,
row_count=inspection.row_count,
byte_count=inspection.byte_count,
metadata_={
**dict(inspection.metadata),
"discovery_fingerprint": inspection.fingerprint,
},
created_by=actor_id,
updated_by=actor_id,
)
db.add(item)
db.flush()
return _source_dto(item, inspection=inspection)
def _file_adapter(self) -> ManagedFileTabularAdapter:
return ManagedFileTabularAdapter(self._registry or get_registry())
def create_snapshot( def create_snapshot(
self, self,
session: object, session: object,
@@ -347,21 +568,81 @@ def _source_record(
tenant_id: str, tenant_id: str,
source_ref: str, source_ref: str,
) -> ConnectorTabularSource | None: ) -> ConnectorTabularSource | None:
source_id = source_ref.removeprefix("snapshot:") prefix, separator, source_id = str(source_ref or "").partition(":")
if not source_id or source_id == source_ref: provider_by_prefix = {
"snapshot": "snapshot",
"file": "managed_file",
"sql": "postgresql",
}
provider = provider_by_prefix.get(prefix)
if not separator or not source_id or provider is None:
return None return None
return session.scalar( return session.scalar(
select(ConnectorTabularSource).where( select(ConnectorTabularSource).where(
ConnectorTabularSource.id == source_id, ConnectorTabularSource.id == source_id,
ConnectorTabularSource.tenant_id == tenant_id, ConnectorTabularSource.tenant_id == tenant_id,
ConnectorTabularSource.provider == provider,
ConnectorTabularSource.deleted_at.is_(None), ConnectorTabularSource.deleted_at.is_(None),
) )
) )
def _source_dto(item: ConnectorTabularSource) -> TabularSource: def _source_dto(
item: ConnectorTabularSource,
*,
inspection: TabularOriginInspection | None = None,
) -> TabularSource:
prefix_by_provider = {
"snapshot": "snapshot",
"managed_file": "file",
"postgresql": "sql",
}
source_mode = {
"managed_file": "file_backed",
"postgresql": "live",
}.get(item.provider, "cached")
if inspection is not None:
pushdown = inspection.pushdown
health = inspection.health
elif item.provider == "managed_file":
pushdown = TabularPushdown(projections=True, pagination=True)
health = TabularSourceHealth(
status="unknown",
code="files.exact_version_not_checked",
summary="The exact managed file version will be re-authorized and integrity-checked on preview.",
checked_at=item.updated_at,
details={
"file_version_id": item.metadata_.get("file_version_id"),
},
)
elif item.provider == "postgresql":
pushdown = TabularPushdown(projections=True, pagination=True)
health = TabularSourceHealth(
status="unknown",
code="sql.health_not_checked",
summary="The live SQL source will be checked against its pinned configuration, credential, and schema on preview.",
checked_at=item.updated_at,
details={
"configuration_id": item.metadata_.get("configuration_id"),
"configuration_revision": item.metadata_.get(
"configuration_revision"
),
},
)
else:
pushdown = TabularPushdown(
projections=True,
pagination=True,
)
health = TabularSourceHealth(
status="healthy",
code="snapshot.ready",
summary="The immutable connector snapshot is ready.",
checked_at=item.updated_at,
details={"immutable": True},
)
return TabularSource( return TabularSource(
ref=f"snapshot:{item.id}", ref=f"{prefix_by_provider.get(item.provider, 'snapshot')}:{item.id}",
provider=item.provider, provider=item.provider,
source_name=item.source_name, source_name=item.source_name,
name=item.name, name=item.name,
@@ -374,18 +655,9 @@ def _source_dto(item: ConnectorTabularSource) -> TabularSource:
updated_at=item.updated_at, updated_at=item.updated_at,
capabilities=("read", "preview"), capabilities=("read", "preview"),
metadata=dict(item.metadata_), metadata=dict(item.metadata_),
source_mode="cached", source_mode=source_mode,
pushdown=TabularPushdown( pushdown=pushdown,
projections=True, health=health,
pagination=True,
),
health=TabularSourceHealth(
status="healthy",
code="snapshot.ready",
summary="The immutable connector snapshot is ready.",
checked_at=item.updated_at,
details={"immutable": True},
),
) )
+389
View File
@@ -0,0 +1,389 @@
from __future__ import annotations
import json
import unittest
from datetime import UTC, datetime
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from govoplan_core.core.dsar import DsarProvider, DsarSubjectRef
from govoplan_core.db.base import Base
from govoplan_core.privacy.dsar_workflow import (
create_data_subject_request,
search_data_subject_request,
)
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
ConnectorSanctionsAcquisitionRun,
ConnectorSimulationRun,
ConnectorTabularSource,
)
from govoplan_connectors.backend.dsar_provider import (
CONNECTORS_DSAR_CAPABILITY,
ConnectorsDsarProvider,
)
from govoplan_connectors.backend.manifest import manifest
NOW = datetime(2026, 8, 22, 9, 0, tzinfo=UTC)
class _Registry:
def __init__(self, provider: ConnectorsDsarProvider) -> None:
self.provider = provider
def capability_names(self):
return (CONNECTORS_DSAR_CAPABILITY,)
def capability_owner(self, name):
if name != CONNECTORS_DSAR_CAPABILITY:
raise KeyError(name)
return "connectors"
def tenant_entitlement_resolver(self):
class _Resolver:
@staticmethod
def resolve(session, tenant_id):
del session, tenant_id
return type("State", (), {"effective_modules": ("connectors",)})()
return _Resolver()
def require_tenant_capability(self, name, session, **kwargs):
del session, kwargs
if name != CONNECTORS_DSAR_CAPABILITY:
raise KeyError(name)
return self.provider
def manifests(self):
return (type("Manifest", (), {"id": "connectors"})(),)
class ConnectorsDsarProviderTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(self.engine)
self.session = Session(self.engine)
self.provider = ConnectorsDsarProvider()
self.assertIsInstance(self.provider, DsarProvider)
self._seed()
self.session.commit()
def tearDown(self) -> None:
self.session.close()
self.engine.dispose()
def _seed(self) -> None:
self.session.add_all(
(
ConnectorTabularSource(
id="source-1",
tenant_id="tenant-1",
provider="snapshot",
source_name="people",
name="People import",
description="Do not export this business description",
status="active",
schema_version=1,
schema_=[{"name": "email"}],
rows=[{"email": "third-party@example.test"}],
fingerprint="source-fingerprint-do-not-export",
row_count=1,
byte_count=100,
metadata_={"secret": "source-metadata-do-not-export"},
created_by="account-1",
updated_by="account-1",
created_at=NOW,
updated_at=NOW,
),
ConnectorTabularSource(
id="source-other",
tenant_id="tenant-2",
provider="snapshot",
source_name="other",
name="Other tenant",
status="active",
schema_version=1,
schema_=[],
rows=[],
fingerprint="other-fingerprint",
row_count=0,
byte_count=0,
metadata_={},
created_by="account-1",
updated_by="account-1",
),
ConnectorSanctionsAcquisitionRun(
id="acquisition-1",
tenant_id="tenant-1",
provider_id="un",
source_id="consolidated",
status="complete",
attempt_count=1,
request_evidence={"secret": "request-evidence-do-not-export"},
response_evidence={"secret": "response-evidence-do-not-export"},
started_at=NOW,
finished_at=NOW,
snapshot_id="snapshot-1",
error="transport-detail-do-not-export",
created_by="account-1",
created_at=NOW,
updated_at=NOW,
),
ConnectorDefinition(
id="definition-1",
tenant_id="tenant-1",
definition_key="address-reader",
name="Address reader",
description="Definition detail",
status="active",
current_revision=1,
source_package="package-secret-do-not-export",
local_definition=True,
created_at=NOW,
updated_at=NOW,
),
ConnectorDefinitionRevision(
id="definition-revision-1",
definition_id="definition-1",
revision=1,
specification={"secret": "specification-do-not-export"},
definition_hash="definition-hash-do-not-export",
origin="local",
package_ref="package-ref-do-not-export",
created_by="account-1",
created_at=NOW,
updated_at=NOW,
),
ConnectorConfiguration(
id="configuration-1",
tenant_id="tenant-1",
definition_id="definition-1",
name="Production addresses",
status="active",
endpoint_url="https://secret.example.test/api",
credential_ref="vault://secret-do-not-export",
base_definition_revision=1,
local_overrides={"secret": "override-do-not-export"},
protected_paths=["secret"],
effective_configuration={"secret": "effective-do-not-export"},
effective_hash="configuration-hash-do-not-export",
resource_revision=2,
ambiguity_policy="manual_review",
updated_by="account-1",
created_at=NOW,
updated_at=NOW,
),
ConnectorSimulationRun(
id="simulation-1",
tenant_id="tenant-1",
configuration_id="configuration-1",
mode="dry_run",
idempotency_key="simulation-idempotency-do-not-export",
request_hash="simulation-request-hash-do-not-export",
status="complete",
review_state="approved",
definition_revision=1,
configuration_revision=2,
configuration_hash="simulation-config-hash-do-not-export",
input_hash="simulation-input-hash-do-not-export",
summary={"secret": "summary-do-not-export"},
effects=[{"secret": "effect-do-not-export"}],
diagnostics=[{"secret": "diagnostic-do-not-export"}],
provenance={"secret": "provenance-do-not-export"},
created_by="account-1",
reviewed_by="account-1",
reviewed_at=NOW,
review_reason="review-reason-do-not-export",
created_at=NOW,
updated_at=NOW,
),
ConnectorKnowledgeProfile(
id="knowledge-profile-1",
tenant_id="tenant-1",
configuration_id="configuration-1",
status="active",
product="bluespice",
desired_maturity="migrate",
discovered_maturity="migrate",
source_authority_mode="external_mirror",
default_visibility="restricted",
default_acl_tokens=["group:secret-do-not-export"],
namespace_mappings=[{"secret": "mapping-do-not-export"}],
capabilities=["read", "synchronize", "migrate"],
discovery_evidence={"secret": "discovery-do-not-export"},
health_details={"secret": "health-do-not-export"},
updated_by="account-1",
created_at=NOW,
updated_at=NOW,
),
ConnectorKnowledgeSyncRun(
id="knowledge-run-1",
tenant_id="tenant-1",
profile_id="knowledge-profile-1",
mode="delta",
idempotency_key="knowledge-idempotency-do-not-export",
request_hash="knowledge-request-hash-do-not-export",
status="completed",
counts={"update": 1},
effects=[{"secret": "knowledge-effect-do-not-export"}],
diagnostics=[{"secret": "knowledge-diagnostic-do-not-export"}],
provenance={"secret": "knowledge-provenance-do-not-export"},
created_by="account-1",
started_at=NOW,
finished_at=NOW,
created_at=NOW,
updated_at=NOW,
),
)
)
@staticmethod
def _subject() -> DsarSubjectRef:
return DsarSubjectRef(account_id="account-1")
def test_search_exports_minimized_operator_attribution(self) -> None:
records = self.provider.search_subject(
self.session, tenant_id="tenant-1", subject=self._subject()
)
self.assertEqual(
{
"source_actor_attribution",
"acquisition_actor_attribution",
"definition_actor_attribution",
"configuration_actor_attribution",
"simulation_actor_attribution",
"knowledge_profile_actor_attribution",
"knowledge_run_actor_attribution",
},
{record.resource_type for record in records},
)
exported = json.dumps([record.to_dict() for record in records])
for excluded in (
"third-party@example.test",
"source-fingerprint-do-not-export",
"source-metadata-do-not-export",
"request-evidence-do-not-export",
"response-evidence-do-not-export",
"transport-detail-do-not-export",
"specification-do-not-export",
"definition-hash-do-not-export",
"package-ref-do-not-export",
"https://secret.example.test/api",
"vault://secret-do-not-export",
"override-do-not-export",
"effective-do-not-export",
"configuration-hash-do-not-export",
"simulation-idempotency-do-not-export",
"simulation-request-hash-do-not-export",
"simulation-config-hash-do-not-export",
"simulation-input-hash-do-not-export",
"summary-do-not-export",
"effect-do-not-export",
"diagnostic-do-not-export",
"provenance-do-not-export",
"review-reason-do-not-export",
"secret-do-not-export",
"mapping-do-not-export",
"discovery-do-not-export",
"health-do-not-export",
"knowledge-idempotency-do-not-export",
"knowledge-request-hash-do-not-export",
"knowledge-effect-do-not-export",
"knowledge-diagnostic-do-not-export",
"knowledge-provenance-do-not-export",
):
self.assertNotIn(excluded, exported)
def test_requires_exact_account_and_enforces_tenant(self) -> None:
email_only = self.provider.search_subject(
self.session,
tenant_id="tenant-1",
subject=DsarSubjectRef(email="operator@example.test"),
)
conflict = self.provider.search_subject(
self.session,
tenant_id="tenant-1",
subject=DsarSubjectRef(
account_id="account-1",
external_references={"connectors.account": "account-other"},
),
)
records = self.provider.search_subject(
self.session, tenant_id="tenant-1", subject=self._subject()
)
self.assertEqual((), email_only)
self.assertEqual((), conflict)
self.assertNotIn("source-other", {record.resource_id for record in records})
def test_object_narrowing_does_not_broaden_the_search(self) -> None:
records = self.provider.search_subject(
self.session,
tenant_id="tenant-1",
subject=DsarSubjectRef(
account_id="account-1",
external_references={"connectors.simulation": "simulation-1"},
),
)
self.assertEqual(
[("simulation_actor_attribution", "simulation-1")],
[(record.resource_type, record.resource_id) for record in records],
)
def test_erasure_retains_external_operation_evidence(self) -> None:
records = self.provider.search_subject(
self.session, tenant_id="tenant-1", subject=self._subject()
)
actions = self.provider.plan_erasure(
self.session,
tenant_id="tenant-1",
subject=self._subject(),
records=records,
)
self.assertTrue(actions)
self.assertTrue(
all(action.kind == "retain" and not action.executable for action in actions)
)
results = self.provider.execute_erasure(
self.session,
tenant_id="tenant-1",
subject=self._subject(),
actions=actions,
request_id="dsar-connectors-1",
)
self.assertTrue(all(result.status == "blocked" for result in results))
def test_manifest_and_core_workflow_discover_provider(self) -> None:
self.assertIn(CONNECTORS_DSAR_CAPABILITY, manifest.capability_factories)
self.assertIn(
"connectors.data-subject-requests",
{topic.id for topic in manifest.documentation},
)
row = create_data_subject_request(
self.session,
tenant_id="tenant-1",
reference="DSAR-CONNECTORS-1",
request_kind="access",
subject=self._subject(),
purpose="Connector attribution access request",
legal_basis=None,
due_at=None,
requested_by_account_id="operator-1",
)
search_data_subject_request(
self.session,
registry=_Registry(self.provider),
row=row,
expected_revision=row.resource_revision,
)
self.assertEqual("searched", row.status)
self.assertEqual(7, row.search_result["record_count"])
if __name__ == "__main__":
unittest.main()
+78 -1
View File
@@ -4,8 +4,18 @@ import unittest
from unittest.mock import patch from unittest.mock import patch
from defusedxml import ElementTree as SafeET from defusedxml import ElementTree as SafeET
from fastapi import HTTPException
from govoplan_connectors.backend.feeds import ConnectorFeedProvider, feed_rows from govoplan_connectors.backend.feeds import (
FEED_PRIVATE_PUBLISH_SCOPE,
FEED_PUBLISH_SCOPE,
ConnectorFeedProvider,
feed_rows,
)
from govoplan_connectors.backend.router import api_render_feed
from govoplan_connectors.backend.schemas import FeedRenderPayload
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.feeds import ( from govoplan_core.core.feeds import (
FeedCapabilityError, FeedCapabilityError,
FeedEntry, FeedEntry,
@@ -98,6 +108,73 @@ class ConnectorFeedProviderTests(unittest.TestCase):
with self.assertRaisesRegex(FeedCapabilityError, "not safe or valid"): with self.assertRaisesRegex(FeedCapabilityError, "not safe or valid"):
self.provider.parse(payload, source_url="https://example.test/rss") self.provider.parse(payload, source_url="https://example.test/rss")
def test_render_api_derives_visibility_from_audience_and_permissions(self) -> None:
entries = [
{
"id": visibility,
"title": visibility.title(),
"visibility": visibility,
"source_kind": source_kind,
"source_module": source_module,
"source_ref": f"{source_kind}:{visibility}",
"source_revision": "7",
}
for visibility, source_kind, source_module in (
("public", "publication", "docs"),
("tenant", "case", "cases"),
("private", "report", "reporting"),
)
]
public_payload = FeedRenderPayload(
format="rss",
title="Selected GovOPlaN updates",
feed_url="https://example.test/feed.xml",
home_url="https://example.test/",
audience="public",
entries=entries,
)
public = api_render_feed(
public_payload,
principal=_principal(FEED_PUBLISH_SCOPE),
)
self.assertEqual("public", public.headers["x-govoplan-feed-audience"])
self.assertEqual("1", public.headers["x-govoplan-feed-included"])
self.assertEqual("2", public.headers["x-govoplan-feed-excluded"])
self.assertIn(b"Public", public.body)
self.assertNotIn(b"Tenant", public.body)
restricted_payload = public_payload.model_copy(update={"audience": "private"})
with self.assertRaises(HTTPException) as denied:
api_render_feed(
restricted_payload,
principal=_principal(FEED_PUBLISH_SCOPE),
)
self.assertEqual(403, denied.exception.status_code)
restricted = api_render_feed(
restricted_payload,
principal=_principal(
FEED_PUBLISH_SCOPE,
FEED_PRIVATE_PUBLISH_SCOPE,
),
)
self.assertEqual("3", restricted.headers["x-govoplan-feed-included"])
self.assertIn(b"Private", restricted.body)
def _principal(*scopes: str) -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id="tenant-1",
scopes=frozenset(scopes),
),
account=object(),
user=object(),
)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+339
View File
@@ -0,0 +1,339 @@
from __future__ import annotations
from types import SimpleNamespace
import unittest
from unittest.mock import patch
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.db.base import Base
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorSimulationRun,
)
from govoplan_connectors.backend.governed_runtime import (
GovernedConnectorError,
create_configuration,
execute_run,
list_configurations,
review_run,
update_configuration,
upsert_definition,
)
from govoplan_connectors.backend.governed_schemas import (
ConnectorConfigurationCreateRequest,
ConnectorConfigurationUpdateRequest,
ConnectorDefinitionUpsertRequest,
ConnectorReviewRequest,
ConnectorRunRequest,
)
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id=tenant_id,
scopes=frozenset(
{
"connectors:source:read",
"connectors:source:write",
"connectors:source:admin",
}
),
),
account=SimpleNamespace(id="account-1"),
user=SimpleNamespace(id="user-1"),
)
def definition_payload(
*,
mapping_version: str = "1",
package_ref: str = "municipal-addresses@1",
timeout_seconds: int = 10,
) -> ConnectorDefinitionUpsertRequest:
return ConnectorDefinitionUpsertRequest.model_validate(
{
"definition_key": "municipal.addresses",
"name": "Municipal addresses",
"description": "A package-managed reference connector.",
"origin": "package",
"package_ref": package_ref,
"specification": {
"provider": "municipal-directory",
"protocol": "rest",
"capabilities": ["discover", "read", "dry_run"],
"input_schema": {"type": "object"},
"output_schema": {"type": "object"},
"mapping": {
"version": mapping_version,
"rules": [
{
"source": "external_id",
"target": "address.external_id",
"required": True,
},
{
"source": "street",
"target": "address.street",
"required": True,
},
],
},
"validation_rules": [
{
"kind": "unique",
"field": "address.external_id",
"severity": "error",
"code": "addresses.external_id.ambiguous",
"message": "The external identifier is not unique.",
}
],
"dry_run": {
"supported": True,
"simulation_supported": True,
"max_items": 50,
"redacted_fields": ["address.street"],
"sample_rows": [
{"external_id": "A-1", "street": "Sample street"}
],
},
"audit": {
"event_prefix": "connectors.municipal_addresses",
"expected_events": ["simulation.completed"],
"evidence_fields": ["input_hash", "configuration_hash"],
},
"privacy_classification": "confidential",
"retention_class": "connector-preview-30d",
"operational_limits": {"timeout_seconds": timeout_seconds},
"retry_policy": {"max_attempts": 2},
},
}
)
class GovernedConnectorRuntimeTests(unittest.TestCase):
def setUp(self) -> None:
self.engine = create_engine("sqlite:///:memory:")
self.tables = [
ConnectorDefinition.__table__,
ConnectorDefinitionRevision.__table__,
ConnectorConfiguration.__table__,
ConnectorSimulationRun.__table__,
]
Base.metadata.create_all(self.engine, tables=self.tables)
self.Session = sessionmaker(bind=self.engine)
self.session = self.Session()
self.audit = patch(
"govoplan_connectors.backend.governed_runtime.audit_from_principal"
)
self.audit_mock = self.audit.start()
def tearDown(self) -> None:
self.audit.stop()
self.session.close()
Base.metadata.drop_all(self.engine, tables=reversed(self.tables))
self.engine.dispose()
def _configuration(
self,
*,
ambiguity_policy: str = "manual_review",
):
definition = upsert_definition(
self.session,
principal(),
definition_payload(),
)
return create_configuration(
self.session,
principal(),
ConnectorConfigurationCreateRequest(
definition_id=definition.id,
name=f"Address import {ambiguity_policy}",
endpoint_url="https://directory.example.invalid/v1",
credential_ref="vault://connectors/address-reader",
local_overrides={"retry_policy": {"max_attempts": 5}},
ambiguity_policy=ambiguity_policy,
status="active",
),
)
def test_package_update_is_explicit_and_preserves_local_overrides(self) -> None:
configuration = self._configuration()
updated_definition = upsert_definition(
self.session,
principal(),
definition_payload(
mapping_version="2",
package_ref="municipal-addresses@2",
timeout_seconds=20,
),
)
unchanged = next(
item
for item in list_configurations(self.session, tenant_id="tenant-1")
if item.id == configuration.id
)
self.assertEqual(2, updated_definition.current_revision)
self.assertTrue(unchanged.update_available)
self.assertEqual("1", unchanged.effective_configuration["mapping"]["version"])
self.assertEqual(5, unchanged.effective_configuration["retry_policy"]["max_attempts"])
self.assertEqual(["retry_policy.max_attempts"], unchanged.protected_paths)
adopted = update_configuration(
self.session,
principal(),
configuration_id=configuration.id,
payload=ConnectorConfigurationUpdateRequest(
expected_revision=configuration.resource_revision,
adopt_latest_definition=True,
),
)
self.assertFalse(adopted.update_available)
self.assertEqual("2", adopted.effective_configuration["mapping"]["version"])
self.assertEqual(
20,
adopted.effective_configuration["operational_limits"]["timeout_seconds"],
)
self.assertEqual(5, adopted.effective_configuration["retry_policy"]["max_attempts"])
self.assertEqual(["retry_policy.max_attempts"], adopted.protected_paths)
def test_ambiguous_simulation_requires_review_and_is_idempotent(self) -> None:
configuration = self._configuration()
payload = ConnectorRunRequest(
idempotency_key="simulation-1",
external_revision="directory-etag-22",
input_rows=[
{"external_id": "duplicate", "street": "First"},
{"external_id": "duplicate", "street": "Second"},
],
)
created = execute_run(
self.session,
principal(),
configuration_id=configuration.id,
mode="simulation",
payload=payload,
)
replayed = execute_run(
self.session,
principal(),
configuration_id=configuration.id,
mode="simulation",
payload=payload,
)
self.assertEqual(created.id, replayed.id)
self.assertEqual("manual_review", created.status)
self.assertEqual("pending", created.review_state)
self.assertEqual(2, created.summary["ambiguous"])
self.assertEqual("<redacted>", created.effects[0]["sample"]["address"]["street"])
self.assertEqual("directory-etag-22", created.provenance["external_revision"])
reviewed = review_run(
self.session,
principal(),
run_id=created.id,
payload=ConnectorReviewRequest(
decision="approved",
reason="The duplicate rows represent an approved upstream alias.",
),
)
self.assertEqual("approved", reviewed.review_state)
self.assertEqual("review_approved", reviewed.status)
with self.assertRaisesRegex(GovernedConnectorError, "different run inputs"):
execute_run(
self.session,
principal(),
configuration_id=configuration.id,
mode="simulation",
payload=ConnectorRunRequest(
idempotency_key="simulation-1",
input_rows=[{"external_id": "other", "street": "Other"}],
),
)
def test_ambiguity_policy_can_quarantine_or_reject(self) -> None:
for policy, expected_status, expected_review in (
("quarantine", "quarantined", "quarantined"),
("reject", "rejected", "not_required"),
):
configuration = self._configuration(ambiguity_policy=policy)
result = execute_run(
self.session,
principal(),
configuration_id=configuration.id,
mode="dry_run",
payload=ConnectorRunRequest(
idempotency_key=f"{policy}-1",
input_rows=[
{"external_id": "same", "street": "First"},
{"external_id": "same", "street": "Second"},
],
),
)
self.assertEqual(expected_status, result.status)
self.assertEqual(expected_review, result.review_state)
def test_endpoint_credentials_and_stale_saves_are_rejected(self) -> None:
definition = upsert_definition(
self.session,
principal(),
definition_payload(),
)
with self.assertRaisesRegex(GovernedConnectorError, "must not contain credentials"):
create_configuration(
self.session,
principal(),
ConnectorConfigurationCreateRequest(
definition_id=definition.id,
name="Unsafe",
endpoint_url="https://user:secret@example.invalid/v1",
),
)
configuration = self._configuration()
with self.assertRaisesRegex(GovernedConnectorError, "reload it before saving"):
update_configuration(
self.session,
principal(),
configuration_id=configuration.id,
payload=ConnectorConfigurationUpdateRequest(
expected_revision=configuration.resource_revision + 1,
name="Stale",
),
)
def test_definition_ownership_cannot_change_implicitly(self) -> None:
package_definition = upsert_definition(
self.session,
principal(),
definition_payload(),
)
local_payload = definition_payload().model_copy(
update={"origin": "local", "package_ref": None}
)
with self.assertRaisesRegex(
GovernedConnectorError,
"configuration overrides",
):
upsert_definition(self.session, principal(), local_payload)
self.assertFalse(package_definition.local_definition)
if __name__ == "__main__":
unittest.main()
+33
View File
@@ -10,6 +10,10 @@ from govoplan_core.core.sanctions import (
CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS, CAPABILITY_CONNECTORS_SANCTIONS_SNAPSHOTS,
) )
from govoplan_connectors.backend.manifest import manifest from govoplan_connectors.backend.manifest import manifest
from govoplan_connectors.backend.knowledge_connector import (
KNOWLEDGE_CAPABILITY,
KNOWLEDGE_PROVIDER_ID,
)
class ConnectorsManifestTests(unittest.TestCase): class ConnectorsManifestTests(unittest.TestCase):
@@ -33,6 +37,35 @@ class ConnectorsManifestTests(unittest.TestCase):
manifest.capability_factories, manifest.capability_factories,
) )
self.assertIsNotNone(manifest.migration_spec) self.assertIsNotNone(manifest.migration_spec)
self.assertEqual(
"@govoplan/connectors-webui",
manifest.frontend.package_name,
)
self.assertIn(
"connectors.governed-configuration",
{topic.id for topic in manifest.documentation},
)
self.assertIn(KNOWLEDGE_CAPABILITY, manifest.capability_factories)
self.assertIn(
KNOWLEDGE_CAPABILITY,
{interface.name for interface in manifest.provides_interfaces},
)
self.assertIn("search", manifest.optional_dependencies)
self.assertIn("wiki", manifest.optional_dependencies)
self.assertIn(
KNOWLEDGE_PROVIDER_ID,
{provider.id for provider in manifest.external_providers},
)
self.assertIn(
KNOWLEDGE_PROVIDER_ID,
{registration.id for registration in manifest.search_sources},
)
topic = next(
item
for item in manifest.documentation
if item.id == "connectors.mediawiki-bluespice"
)
self.assertIn("de", topic.translations)
if __name__ == "__main__": if __name__ == "__main__":
+659
View File
@@ -0,0 +1,659 @@
from __future__ import annotations
from types import SimpleNamespace
import unittest
from unittest.mock import patch
from sqlalchemy import create_engine, select
from sqlalchemy.orm import Session
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.search import (
SearchAuthorizationRequest,
SearchBackfillRequest,
)
from govoplan_core.core.recovery import RecoveryOperation, RecoveryStatus
from govoplan_core.core.runtime_coordination import (
RuntimeIdentity,
bind_process_runtime_identity,
)
from govoplan_core.db.base import Base
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorDefinitionRevision,
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
)
from govoplan_connectors.backend.knowledge_connector import (
KNOWLEDGE_PROVIDER_ID,
KNOWLEDGE_RESOURCE_TYPE,
KnowledgeConnectorError,
create_profile,
discover_profile,
list_objects,
migration_dry_run,
publish_page,
synchronize_profile,
update_profile,
)
from govoplan_connectors.backend.knowledge_schemas import (
KnowledgeMigrationDryRunRequest,
KnowledgeMigrationTargetState,
KnowledgeNamespaceMapping,
KnowledgeProfileCreateRequest,
KnowledgeProfileUpdateRequest,
KnowledgePublishRequest,
KnowledgeSyncRequest,
)
from govoplan_connectors.backend.knowledge_search import (
ExternalKnowledgeSearchSource,
)
from govoplan_connectors.backend.mediawiki_transport import (
MediaWikiChangeBatch,
MediaWikiPublishResult,
MediaWikiTransportError,
)
ALL_SCOPES = frozenset(
{
"connectors:knowledge:read",
"connectors:knowledge:admin",
"connectors:knowledge:sync",
"connectors:knowledge:publish",
"connectors:knowledge:migrate",
}
)
def principal(
tenant_id: str = "tenant-1",
*,
scopes: frozenset[str] = ALL_SCOPES,
groups: frozenset[str] = frozenset({"editors"}),
) -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id=tenant_id,
scopes=scopes,
group_ids=groups,
),
account=SimpleNamespace(id="account-1"),
user=SimpleNamespace(id="account-1"),
)
class StaticTransport:
def __init__(self) -> None:
self.batches: list[MediaWikiChangeBatch] = []
self.change_calls = 0
self.publish_calls = 0
self.publish_error: MediaWikiTransportError | None = None
def discover(self, *, endpoint_url, credential):
del endpoint_url, credential
return {
"curtimestamp": "2026-08-22T10:00:00Z",
"query": {
"general": {
"generator": "MediaWiki 1.43.1",
"phpversion": "8.3.8",
},
"extensions": [
{"name": "BlueSpiceFoundation", "version": "4.5.2"},
{"name": "BlueSpicePermissionManager", "version": "4.5.2"},
],
"namespaces": {
"0": {"id": 0, "name": "", "content": True},
"4": {"id": 4, "name": "GovWiki", "content": True},
},
"userinfo": {
"id": 17,
"name": "govoplan",
"rights": ["read", "edit"],
"groups": ["bot"],
},
},
}
def changes(self, **kwargs):
del kwargs
self.change_calls += 1
if not self.batches:
raise AssertionError("No deterministic change batch remains")
return self.batches.pop(0)
def publish(self, **kwargs):
self.publish_calls += 1
if self.publish_error is not None:
raise self.publish_error
return MediaWikiPublishResult(
page_id=str(kwargs.get("expected_page_id") or "99"),
revision_id="901",
title=str(kwargs["title"]),
canonical_url="https://wiki.example.invalid/wiki/Published_Guide",
evidence={"result": "Success", "fixture": True},
)
def page(
*,
page_id: str = "42",
revision_id: str = "501",
title: str = "Citizen Guide",
acl_tokens: list[str] | None = None,
) -> dict[str, object]:
return {
"change_kind": "upsert",
"change_cursor": f"rcid:{revision_id}",
"pageid": page_id,
"ns": 0,
"title": title,
"fullurl": f"https://wiki.example.invalid/wiki/{title.replace(' ', '_')}",
"lastrevid": revision_id,
"revisions": [
{
"revid": revision_id,
"timestamp": "2026-08-22T10:00:00Z",
"user": "Ada Admin",
"comment": "Reviewed guidance",
"contentmodel": "wikitext",
"sha1": f"sha-{revision_id}",
"slots": {
"main": {
"content": "Welcome {{UnsupportedBox|important}} [[Services]]"
}
},
}
],
"categories": [{"title": "Category:Citizen service"}],
"links": [{"ns": 0, "title": "Services"}],
"images": [{"ns": 6, "title": "File:guide.pdf", "pageid": 71}],
"discussions": [
{
"id": "discussion-1",
"author": "Ada Admin",
"body": "Please verify this section.",
"created_at": "2026-08-20T09:00:00Z",
}
],
"permissions": {
"visibility": "restricted",
"acl_tokens": acl_tokens or ["group:editors"],
},
}
class MediaWikiConnectorTests(unittest.TestCase):
def setUp(self) -> None:
bind_process_runtime_identity(
RuntimeIdentity(
installation_id="mediawiki-connector-tests",
node_id="node-1",
incarnation="incarnation-1",
role="worker",
software_version="test",
composition_hash="a" * 64,
)
)
self.engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(self.engine)
self.session = Session(self.engine)
self.audit = patch(
"govoplan_connectors.backend.knowledge_connector.audit_event"
)
self.audit.start()
self.credential = patch(
"govoplan_connectors.backend.knowledge_connector._credential",
return_value={"access_token": "fixture-token"},
)
self.credential.start()
self.transport = StaticTransport()
self.configuration_id = self._seed_configuration()
self.profile_id = self._profile()
def tearDown(self) -> None:
bind_process_runtime_identity(None)
self.credential.stop()
self.audit.stop()
self.session.close()
self.engine.dispose()
def _seed_configuration(self) -> str:
definition = ConnectorDefinition(
id="definition-1",
tenant_id="tenant-1",
definition_key="knowledge.mediawiki",
name="MediaWiki",
description="Knowledge transport",
status="active",
current_revision=1,
local_definition=True,
)
self.session.add(definition)
self.session.add(
ConnectorDefinitionRevision(
id="definition-revision-1",
definition_id=definition.id,
revision=1,
specification={
"provider": "mediawiki",
"protocol": "mediawiki_action_api",
},
definition_hash="definition-hash",
origin="local",
created_by="account-1",
)
)
configuration = ConnectorConfiguration(
id="configuration-1",
tenant_id="tenant-1",
definition_id=definition.id,
name="Institutional knowledge",
status="active",
endpoint_url="https://wiki.example.invalid",
credential_ref="credential-envelope-1",
base_definition_revision=1,
local_overrides={},
protected_paths=[],
effective_configuration={
"provider": "mediawiki",
"protocol": "mediawiki_action_api",
},
effective_hash="configuration-hash",
resource_revision=1,
ambiguity_policy="manual_review",
updated_by="account-1",
)
self.session.add(configuration)
self.session.flush()
return configuration.id
def _profile(self) -> str:
created = create_profile(
self.session,
principal(),
KnowledgeProfileCreateRequest(
configuration_id=self.configuration_id,
desired_maturity="migrate",
source_authority_mode="external_mirror",
default_visibility="restricted",
default_acl_tokens=["group:knowledge-managers"],
namespace_mappings=[
KnowledgeNamespaceMapping(
source_namespace_id=0,
source_name="",
target_space_ref="service-guidance",
target_path_prefix="imported",
)
],
),
)
return created.id
def _discover(self):
return discover_profile(
self.session,
principal(),
profile_id=self.profile_id,
transport=self.transport,
)
def _sync(self, raw_pages, *, key="sync-1", force_full=True):
self.transport.batches.append(
MediaWikiChangeBatch(
changes=tuple(raw_pages),
next_cursor=None,
complete=True,
high_watermark="2026-08-22T10:05:00Z",
evidence={"fixture": True},
)
)
return synchronize_profile(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeSyncRequest(
idempotency_key=key,
force_full=force_full,
),
transport=self.transport,
registry=None,
)
def test_discovery_mapping_idempotency_and_migration_loss_diagnostics(self) -> None:
discovery = self._discover()
self.assertEqual("bluespice", discovery.product)
self.assertEqual("4.5.2", discovery.product_version)
self.assertEqual("migrate", discovery.maturity)
self.assertIn("publish", discovery.capabilities)
self.assertIn("permission_metadata", discovery.capabilities)
run = self._sync([page()])
self.assertEqual({"create": 1}, run.counts)
self.assertIn(
"attachments_reference_only", {item.code for item in run.diagnostics}
)
objects, _cursor = list_objects(
self.session, principal(), profile_id=self.profile_id
)
self.assertEqual(1, len(objects))
item = objects[0]
self.assertEqual("42", item.external_reference.object_id)
self.assertEqual("501", item.external_reference.version)
self.assertEqual("imported/Citizen-Guide", item.mapped_data["target_path"])
self.assertEqual("Ada Admin", item.mapped_data["revision_author"])
self.assertEqual(
"user", item.mapped_data["revision_author_reference"]["object_type"]
)
self.assertEqual("guide.pdf", item.mapped_data["files"][0]["name"])
self.assertEqual("discussion-1", item.mapped_data["discussions"][0]["external_id"])
replay = synchronize_profile(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeSyncRequest(
idempotency_key="sync-1",
force_full=True,
),
transport=self.transport,
registry=None,
)
self.assertEqual(run.id, replay.id)
self.assertEqual(1, self.transport.change_calls)
with self.assertRaisesRegex(KnowledgeConnectorError, "different request"):
synchronize_profile(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeSyncRequest(
idempotency_key="sync-1",
force_full=True,
limit=25,
),
transport=self.transport,
registry=None,
)
preview = migration_dry_run(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeMigrationDryRunRequest(
idempotency_key="migration-1",
target_space_ref="service-guidance",
supported_macros=["SupportedBox"],
existing_targets=[
KnowledgeMigrationTargetState(
path="imported/Citizen-Guide",
source_external_id="different-page",
attachment_names=["guide.pdf"],
)
],
),
)
self.assertFalse(preview.can_apply)
self.assertEqual("2026-08-22T10:05:00Z", preview.source_revision)
self.assertEqual(
{"attachment_name_conflict", "target_path_conflict", "unsupported_macro"},
{item.code for item in preview.diagnostics},
)
def test_acl_changes_moves_deletes_and_search_authorization_are_current(self) -> None:
self._discover()
self._sync([page()])
source = ExternalKnowledgeSearchSource()
backfill = source.backfill(
self.session,
request=SearchBackfillRequest(
tenant_id="tenant-1",
provider_id=KNOWLEDGE_PROVIDER_ID,
resource_type=KNOWLEDGE_RESOURCE_TYPE,
rebuild_id="rebuild-1",
limit=100,
),
)
self.assertEqual(1, len(backfill.documents))
document = backfill.documents[0]
request = SearchAuthorizationRequest(
reference=document.reference,
source_revision=document.source_revision,
)
self.assertTrue(
source.authorize(self.session, principal(), requests=[request])[
document.reference.key
]
)
denied = principal(groups=frozenset({"other"}))
self.assertFalse(
source.authorize(self.session, denied, requests=[request])[
document.reference.key
]
)
self._sync(
[
page(
revision_id="502",
title="Resident Guide",
acl_tokens=["group:reviewers"],
)
],
key="sync-2",
force_full=False,
)
self.assertFalse(
source.authorize(self.session, principal(), requests=[request])[
document.reference.key
]
)
reviewers = principal(groups=frozenset({"reviewers"}))
self.assertTrue(
source.authorize(self.session, reviewers, requests=[request])[
document.reference.key
]
)
stored = self.session.scalar(
select(ConnectorKnowledgeObject).where(
ConnectorKnowledgeObject.profile_id == self.profile_id,
ConnectorKnowledgeObject.external_id == "42",
)
)
self.assertEqual("Resident Guide", stored.title)
self.assertEqual("imported/Resident-Guide", stored.mapped_data["target_path"])
self._sync(
[
{
"change_kind": "delete",
"change_cursor": "logid:700",
"pageid": 0,
"ns": 0,
"title": "Resident Guide",
"timestamp": "2026-08-22T11:00:00Z",
"logid": 700,
}
],
key="sync-3",
force_full=False,
)
self.assertEqual("deleted", stored.status)
self.assertFalse(
source.authorize(self.session, reviewers, requests=[request])[
document.reference.key
]
)
after_delete = source.backfill(
self.session,
request=SearchBackfillRequest(
tenant_id="tenant-1",
provider_id=KNOWLEDGE_PROVIDER_ID,
resource_type=KNOWLEDGE_RESOURCE_TYPE,
rebuild_id="rebuild-2",
limit=100,
),
)
self.assertEqual((), after_delete.documents)
def test_publication_replay_and_outcome_unknown_are_evidenced(self) -> None:
self._discover()
payload = KnowledgePublishRequest(
idempotency_key="publish-1",
title="Published Guide",
body="Reviewed body",
summary="Publish approved guidance",
expected_external_revision="900",
)
result = publish_page(
self.session,
principal(),
profile_id=self.profile_id,
external_page_id="99",
payload=payload,
transport=self.transport,
registry=None,
durable_recovery=True,
)
replay = publish_page(
self.session,
principal(),
profile_id=self.profile_id,
external_page_id="99",
payload=payload,
transport=self.transport,
registry=None,
durable_recovery=False,
)
self.assertTrue(result.accepted)
self.assertEqual(result.run.id, replay.run.id)
self.assertEqual("901", result.external_reference.version)
self.assertEqual(1, self.transport.publish_calls)
recovery = self.session.scalar(
select(RecoveryOperation).where(
RecoveryOperation.resource_type == "external_knowledge_page",
RecoveryOperation.resource_id == "99",
)
)
self.assertEqual(RecoveryStatus.SUCCEEDED.value, recovery.status)
self.transport.publish_error = MediaWikiTransportError(
"transport_timeout",
"Provider response timed out.",
retryable=True,
outcome_unknown=True,
)
with self.assertRaisesRegex(KnowledgeConnectorError, "outcome is unknown"):
publish_page(
self.session,
principal(),
profile_id=self.profile_id,
external_page_id="100",
payload=KnowledgePublishRequest(
idempotency_key="publish-unknown",
title="Uncertain Guide",
body="Body",
),
transport=self.transport,
registry=None,
durable_recovery=False,
)
unresolved = self.session.scalar(
select(ConnectorKnowledgeSyncRun).where(
ConnectorKnowledgeSyncRun.profile_id == self.profile_id,
ConnectorKnowledgeSyncRun.idempotency_key == "publish-unknown",
)
)
self.assertEqual("outcome_unknown", unresolved.status)
def test_fallback_acl_changes_and_profile_pause_fail_closed_immediately(self) -> None:
self._discover()
fallback_page = page(page_id="43", title="Fallback Guide")
fallback_page.pop("permissions")
self._sync([fallback_page])
source = ExternalKnowledgeSearchSource()
document = source.backfill(
self.session,
request=SearchBackfillRequest(
tenant_id="tenant-1",
provider_id=KNOWLEDGE_PROVIDER_ID,
resource_type=KNOWLEDGE_RESOURCE_TYPE,
rebuild_id="fallback-rebuild",
),
).documents[0]
request = SearchAuthorizationRequest(
reference=document.reference,
source_revision=document.source_revision,
)
managers = principal(groups=frozenset({"knowledge-managers"}))
self.assertTrue(
source.authorize(self.session, managers, requests=[request])[
document.reference.key
]
)
profile = self.session.get(ConnectorKnowledgeProfile, self.profile_id)
before_hash = self.session.get(
ConnectorKnowledgeObject, document.resource_id
).content_hash
updated = update_profile(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeProfileUpdateRequest(
expected_resource_revision=profile.resource_revision,
default_acl_tokens=["group:reviewers"],
),
registry=None,
)
self.assertFalse(
source.authorize(self.session, managers, requests=[request])[
document.reference.key
]
)
reviewers = principal(groups=frozenset({"reviewers"}))
self.assertTrue(
source.authorize(self.session, reviewers, requests=[request])[
document.reference.key
]
)
self.assertNotEqual(
before_hash,
self.session.get(ConnectorKnowledgeObject, document.resource_id).content_hash,
)
update_profile(
self.session,
principal(),
profile_id=self.profile_id,
payload=KnowledgeProfileUpdateRequest(
expected_resource_revision=updated.resource_revision,
status="paused",
),
registry=None,
)
self.assertFalse(
source.authorize(self.session, reviewers, requests=[request])[
document.reference.key
]
)
def test_profiles_and_objects_are_tenant_isolated(self) -> None:
self._discover()
self._sync([page()])
with self.assertRaisesRegex(KnowledgeConnectorError, "not found"):
list_objects(
self.session,
principal("tenant-2"),
profile_id=self.profile_id,
)
profile = self.session.get(ConnectorKnowledgeProfile, self.profile_id)
self.assertEqual("tenant-1", profile.tenant_id)
if __name__ == "__main__":
unittest.main()
+158
View File
@@ -0,0 +1,158 @@
from __future__ import annotations
import json
from urllib.parse import parse_qs, urlsplit
import unittest
from unittest.mock import patch
from govoplan_core.security.http_fetch import HttpFetchResponse
from govoplan_connectors.backend.mediawiki_transport import HttpMediaWikiTransport
def response(payload: dict[str, object]) -> HttpFetchResponse:
return HttpFetchResponse(
status=200,
headers={"Content-Type": "application/json"},
body=json.dumps(payload).encode(),
)
class MediaWikiHttpTransportTests(unittest.TestCase):
def setUp(self) -> None:
self.transport = HttpMediaWikiTransport()
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
def test_discovery_uses_the_action_api_and_sanitized_auth_header(self, fetch) -> None:
fetch.return_value = response(
{"query": {"general": {"generator": "MediaWiki 1.43"}}}
)
self.transport.discover(
endpoint_url="https://wiki.example.test",
credential={"access_token": "secret-token"},
)
call = fetch.call_args
self.assertEqual("https://wiki.example.test/api.php", urlsplit(call.args[0])._replace(query="").geturl())
query = parse_qs(urlsplit(call.args[0]).query)
self.assertEqual(["query"], query["action"])
self.assertEqual(["siteinfo|userinfo"], query["meta"])
self.assertEqual("Bearer secret-token", call.kwargs["headers"]["Authorization"])
self.assertNotIn("secret-token", call.args[0])
self.assertEqual(10_000_000, call.kwargs["max_bytes"])
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
def test_full_backfill_resolves_page_details_and_cursor(self, fetch) -> None:
fetch.side_effect = (
response(
{
"curtimestamp": "2026-08-22T10:00:00Z",
"continue": {"apcontinue": "Next_Page"},
"query": {
"allpages": [
{"pageid": 41, "ns": 0, "title": "First"},
{"pageid": 42, "ns": 0, "title": "Second"},
]
},
}
),
response(
{
"query": {
"pages": [
{"pageid": 41, "ns": 0, "title": "First"},
{"pageid": 42, "ns": 0, "title": "Second"},
]
}
}
),
)
batch = self.transport.changes(
endpoint_url="https://wiki.example.test/api.php",
credential=None,
cursor=None,
limit=2,
force_full=True,
)
self.assertEqual(2, len(batch.changes))
self.assertEqual("Next_Page", batch.next_cursor)
self.assertFalse(batch.complete)
details = parse_qs(urlsplit(fetch.call_args_list[1].args[0]).query)
self.assertEqual(["41|42"], details["pageids"])
self.assertIn("revisions", details["prop"][0])
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
def test_delta_preserves_deletion_log_tombstone_and_cursor(self, fetch) -> None:
fetch.return_value = response(
{
"curtimestamp": "2026-08-22T11:00:00Z",
"query": {
"recentchanges": [
{
"type": "log",
"logtype": "delete",
"logid": 700,
"pageid": 0,
"ns": 0,
"title": "Deleted page",
"timestamp": "2026-08-22T10:59:00Z",
}
]
},
}
)
batch = self.transport.changes(
endpoint_url="https://wiki.example.test",
credential=None,
cursor="rccontinue-token",
limit=50,
force_full=False,
)
self.assertEqual("delete", batch.changes[0]["change_kind"])
self.assertEqual("logid:700", batch.changes[0]["change_cursor"])
self.assertTrue(batch.complete)
@patch("govoplan_connectors.backend.mediawiki_transport.fetch_http")
def test_publish_uses_csrf_body_expected_revision_and_canonical_url(self, fetch) -> None:
fetch.return_value = response(
{
"edit": {
"result": "Success",
"pageid": 99,
"oldrevid": 900,
"newrevid": 901,
"title": "Published Guide",
}
}
)
result = self.transport.publish(
endpoint_url="https://wiki.example.test",
credential={"csrf_token": "csrf-secret", "access_token": "token"},
title="Published Guide",
body="Reviewed body",
summary="Approved",
expected_revision="900",
minor=True,
)
call = fetch.call_args
self.assertEqual("POST", call.kwargs["method"])
body = parse_qs(call.kwargs["body"].decode())
self.assertEqual(["edit"], body["action"])
self.assertEqual(["900"], body["baserevid"])
self.assertEqual(["1"], body["minor"])
self.assertEqual(["csrf-secret"], body["token"])
self.assertEqual("99", result.page_id)
self.assertEqual(
"https://wiki.example.test/wiki/Published_Guide",
result.canonical_url,
)
if __name__ == "__main__":
unittest.main()
+14 -12
View File
@@ -24,20 +24,22 @@ class ConnectorsMigrationTests(unittest.TestCase):
try: try:
with engine.connect() as connection: with engine.connect() as connection:
self.assertIn( self.assertIn(
"f7c8d9e0a1b2", "b9e0f1a2c3d4",
set(MigrationContext.configure(connection).get_current_heads()), set(MigrationContext.configure(connection).get_current_heads()),
) )
self.assertIn( self.assertTrue(
"connector_tabular_sources", {
inspect(connection).get_table_names(), "connector_tabular_sources",
) "connector_sanctions_snapshots",
self.assertIn( "connector_sanctions_acquisition_runs",
"connector_sanctions_snapshots", "connector_definitions",
inspect(connection).get_table_names(), "connector_definition_revisions",
) "connector_configurations",
self.assertIn( "connector_simulation_runs",
"connector_sanctions_acquisition_runs", "connector_knowledge_profiles",
inspect(connection).get_table_names(), "connector_knowledge_objects",
"connector_knowledge_sync_runs",
}.issubset(inspect(connection).get_table_names())
) )
finally: finally:
engine.dispose() engine.dispose()
+65 -6
View File
@@ -7,14 +7,19 @@ from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import sessionmaker
from govoplan_connectors.backend.db.models import ( from govoplan_connectors.backend.db.models import (
ConnectorKnowledgeObject,
ConnectorKnowledgeProfile,
ConnectorKnowledgeSyncRun,
ConnectorSanctionsAcquisitionRun, ConnectorSanctionsAcquisitionRun,
ConnectorSanctionsSnapshot, ConnectorSanctionsSnapshot,
ConnectorTabularSource, ConnectorTabularSource,
) )
from govoplan_connectors.backend.manifest import manifest from govoplan_connectors.backend.manifest import manifest
from govoplan_connectors.backend.provider_state import ( from govoplan_connectors.backend.provider_state import (
KNOWLEDGE_PROVIDER_ID,
SANCTIONS_PROVIDER_ID, SANCTIONS_PROVIDER_ID,
TABULAR_PROVIDER_ID, TABULAR_PROVIDER_ID,
knowledge_provider_states,
sanctions_provider_states, sanctions_provider_states,
tabular_provider_states, tabular_provider_states,
) )
@@ -27,11 +32,6 @@ class ConnectorsProviderStateTests(unittest.TestCase):
self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True) self.engine = create_engine("sqlite+pysqlite:///:memory:", future=True)
Base.metadata.create_all( Base.metadata.create_all(
self.engine, self.engine,
tables=(
ConnectorTabularSource.__table__,
ConnectorSanctionsAcquisitionRun.__table__,
ConnectorSanctionsSnapshot.__table__,
),
) )
self.session = sessionmaker(bind=self.engine, expire_on_commit=False)() self.session = sessionmaker(bind=self.engine, expire_on_commit=False)()
@@ -109,10 +109,69 @@ class ConnectorsProviderStateTests(unittest.TestCase):
self.assertNotIn("secret-source-name", rendered) self.assertNotIn("secret-source-name", rendered)
self.assertNotIn("source.example.test", rendered) self.assertNotIn("source.example.test", rendered)
self.assertEqual( self.assertEqual(
{TABULAR_PROVIDER_ID, SANCTIONS_PROVIDER_ID}, {TABULAR_PROVIDER_ID, SANCTIONS_PROVIDER_ID, KNOWLEDGE_PROVIDER_ID},
{item.provider_id for item in manifest.external_provider_state_providers}, {item.provider_id for item in manifest.external_provider_state_providers},
) )
def test_external_knowledge_state_reports_health_without_acl_or_endpoint_data(self) -> None:
now = datetime.now(UTC)
profile = ConnectorKnowledgeProfile(
id="knowledge-profile-1",
tenant_id="tenant-1",
configuration_id="configuration-secret",
status="active",
product="bluespice",
product_version="4.5.2",
desired_maturity="migrate",
discovered_maturity="migrate",
source_authority_mode="external_mirror",
default_visibility="restricted",
default_acl_tokens=["group:secret-acl"],
namespace_mappings=[{"secret": "mapping"}],
capabilities=["read", "synchronize", "migrate"],
health_status="healthy",
discovered_at=now,
)
page = ConnectorKnowledgeObject(
id="knowledge-object-1",
tenant_id="tenant-1",
profile_id=profile.id,
object_type="page",
external_id="42",
title="Secret page title",
status="active",
source_revision="501",
content_hash="c" * 64,
visibility="restricted",
acl_tokens=["group:secret-acl"],
observed_at=now,
)
run = ConnectorKnowledgeSyncRun(
id="knowledge-run-1",
tenant_id="tenant-1",
profile_id=profile.id,
mode="delta",
idempotency_key="secret-key",
request_hash="d" * 64,
status="completed",
started_at=now,
finished_at=now,
)
self.session.add_all((profile, page, run))
self.session.commit()
state = knowledge_provider_states(
ExternalProviderStateContext(session=self.session, tenant_id="tenant-1")
)[0]
self.assertEqual("healthy", state.health)
self.assertEqual("ready", state.recovery)
self.assertEqual(1, state.metrics["active_objects"])
rendered = str(state.to_dict())
self.assertNotIn("Secret page title", rendered)
self.assertNotIn("group:secret-acl", rendered)
self.assertNotIn("configuration-secret", rendered)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+1 -1
View File
@@ -137,7 +137,7 @@ class ConnectorRecoveryTests(unittest.TestCase):
self.assertFalse(declarations["read-snapshot"].provider_mutation) self.assertFalse(declarations["read-snapshot"].provider_mutation)
self.assertTrue(declarations["read-snapshot"].implemented) self.assertTrue(declarations["read-snapshot"].implemented)
self.assertTrue(declarations["external-mutation"].provider_mutation) self.assertTrue(declarations["external-mutation"].provider_mutation)
self.assertFalse(declarations["external-mutation"].implemented) self.assertTrue(declarations["external-mutation"].implemented)
def test_stale_atomic_connector_fence_fails_without_claiming_an_effect(self) -> None: def test_stale_atomic_connector_fence_fails_without_claiming_an_effect(self) -> None:
recovery = begin_connector_read_snapshot( recovery = begin_connector_read_snapshot(
+341
View File
@@ -0,0 +1,341 @@
from __future__ import annotations
import tempfile
import unittest
from io import BytesIO
from pathlib import Path
from types import SimpleNamespace
from openpyxl import Workbook
from sqlalchemy import Column, Integer, MetaData, String, Table, create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.files import (
CAPABILITY_FILES_TABULAR_CONTENT,
ManagedTabularFile,
ManagedTabularFileContent,
)
from govoplan_core.core.tabular_sources import (
TabularSourceUnavailableError,
TabularSourceValidationError,
)
from govoplan_core.db.base import Base
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
)
from govoplan_connectors.backend.tabular_adapters import (
ManagedFileTabularAdapter,
PostgresqlTabularAdapter,
parse_managed_tabular_content,
)
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id=tenant_id,
scopes=frozenset(
{
"connectors:source:read",
"connectors:source:write",
"files:file:read",
"files:file:download",
}
),
),
account=object(),
user=SimpleNamespace(id="user-1"),
)
class _ManagedFiles:
def __init__(self, payload: bytes, *, filename: str = "cases.csv") -> None:
self.payload = payload
self.filename = filename
self.current_version_id = "version-2"
def _file(self, version_id: str) -> ManagedTabularFile:
return ManagedTabularFile(
file_asset_id="asset-1",
file_version_id=version_id,
filename=self.filename,
display_path=f"Imports/{self.filename}",
content_type=(
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
if self.filename.endswith(".xlsx")
else "text/csv"
),
size_bytes=len(self.payload),
sha256=("a" if version_id == "version-1" else "b") * 64,
current_version=version_id == self.current_version_id,
)
def list_tabular_files(self, session, principal, *, query="", limit=100):
del session, principal, query, limit
return (self._file(self.current_version_id),)
def get_tabular_file(
self,
session,
principal,
*,
file_asset_id,
file_version_id=None,
):
del session, principal
if file_asset_id != "asset-1":
return None
return self._file(file_version_id or self.current_version_id)
def read_tabular_file(
self,
session,
principal,
*,
file_asset_id,
file_version_id,
max_bytes,
):
del session, principal, file_asset_id
if len(self.payload) > max_bytes:
raise AssertionError("test payload exceeded adapter limit")
return ManagedTabularFileContent(
file=self._file(file_version_id),
payload=self.payload,
)
class _Registry:
def __init__(self, provider) -> None:
self.provider = provider
def has_capability(self, name):
return name == CAPABILITY_FILES_TABULAR_CONTENT
def require_capability(self, name):
if not self.has_capability(name):
raise KeyError(name)
return self.provider
class ManagedFileTabularAdapterTests(unittest.TestCase):
def test_csv_is_exact_version_pinned_and_reports_newer_version(self) -> None:
adapter = ManagedFileTabularAdapter(
_Registry(_ManagedFiles(b"id,amount\n0012,12.5\n2,7\n"))
)
result = adapter.inspect(
object(),
principal(),
file_asset_id="asset-1",
file_version_id="version-1",
)
self.assertEqual("managed_file", result.provider)
self.assertEqual("version-1", result.metadata["file_version_id"])
self.assertEqual("warning", result.health.status)
self.assertEqual("files.newer_version_available", result.health.code)
self.assertEqual("mixed", result.schema[0].data_type)
self.assertEqual(2, result.row_count)
self.assertTrue(result.pushdown.projections)
self.assertEqual("files.newer_version_available", result.diagnostics[0].code)
def test_xlsx_uses_requested_sheet_and_closed_typed_schema(self) -> None:
workbook = Workbook()
first = workbook.active
first.title = "Ignore"
first.append(["ignored"])
target = workbook.create_sheet("Monthly")
target.append(["case_id", "amount", "active"])
target.append(["A-1", 12.5, True])
target.append(["A-2", None, False])
payload = BytesIO()
workbook.save(payload)
workbook.close()
rows, sheet = parse_managed_tabular_content(
payload.getvalue(),
filename="monthly.xlsx",
content_type=(
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
),
delimiter=",",
sheet_name="Monthly",
)
self.assertEqual("Monthly", sheet)
self.assertEqual("A-1", rows[0]["case_id"])
self.assertIsNone(rows[1]["amount"])
def test_missing_files_capability_is_explicitly_unavailable(self) -> None:
with self.assertRaisesRegex(
TabularSourceUnavailableError,
"require the Files module",
):
ManagedFileTabularAdapter(None).inspect(
object(),
principal(),
file_asset_id="asset-1",
file_version_id=None,
)
class PostgresqlTabularAdapterTests(unittest.TestCase):
def setUp(self) -> None:
self.directory = tempfile.TemporaryDirectory(
prefix="govoplan-connectors-sql-adapter-"
)
source_path = Path(self.directory.name) / "source.db"
self.source_url = f"sqlite+pysqlite:///{source_path}"
source_engine = create_engine(self.source_url)
metadata = MetaData()
self.table = Table(
"monthly_cases",
metadata,
Column("case_id", String, nullable=False),
Column("amount", Integer, nullable=True),
)
metadata.create_all(source_engine)
with source_engine.begin() as connection:
connection.execute(
self.table.insert(),
(
{"case_id": "A-1", "amount": 12},
{"case_id": "A-2", "amount": None},
),
)
source_engine.dispose()
self.catalog_engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(
self.catalog_engine,
tables=(
ConnectorDefinition.__table__,
ConnectorConfiguration.__table__,
),
)
self.session = sessionmaker(bind=self.catalog_engine)()
definition = ConnectorDefinition(
id="definition-1",
tenant_id="tenant-1",
definition_key="postgresql.reader",
name="PostgreSQL reader",
status="active",
current_revision=1,
local_definition=True,
)
self.configuration = ConnectorConfiguration(
id="configuration-1",
tenant_id="tenant-1",
definition_id=definition.id,
name="Monthly SQL",
status="active",
endpoint_url=self.source_url,
credential_ref=None,
base_definition_revision=1,
local_overrides={},
protected_paths=[],
effective_configuration={"provider": "sql", "protocol": "sql"},
effective_hash="configuration-hash-1",
resource_revision=1,
ambiguity_policy="manual_review",
)
self.session.add_all((definition, self.configuration))
self.session.commit()
self.adapter = PostgresqlTabularAdapter(allow_sqlite_for_tests=True)
def tearDown(self) -> None:
self.session.close()
self.catalog_engine.dispose()
self.directory.cleanup()
def test_discovers_and_reads_projection_from_governed_sql_configuration(self) -> None:
inspection = self.adapter.inspect(
self.session,
principal(),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
metadata = {
**dict(inspection.metadata),
"discovery_fingerprint": inspection.fingerprint,
}
read = self.adapter.read(
self.session,
principal(),
metadata=metadata,
columns=("case_id",),
offset=1,
limit=10,
timeout_ms=2_000,
)
self.assertEqual("live", "live")
self.assertEqual(["case_id", "amount"], [item.name for item in inspection.schema])
self.assertEqual(2, inspection.row_count)
self.assertEqual(({"case_id": "A-2"},), read.rows)
self.assertTrue(inspection.pushdown.projections)
self.assertFalse(inspection.pushdown.filters)
def test_schema_drift_and_tenant_isolation_fail_closed(self) -> None:
inspection = self.adapter.inspect(
self.session,
principal(),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
metadata = {
**dict(inspection.metadata),
"discovery_fingerprint": inspection.fingerprint,
}
engine = create_engine(self.source_url)
with engine.begin() as connection:
connection.exec_driver_sql(
"ALTER TABLE monthly_cases ADD COLUMN category TEXT"
)
engine.dispose()
with self.assertRaisesRegex(TabularSourceValidationError, "schema drifted"):
self.adapter.read(
self.session,
principal(),
metadata=metadata,
columns=(),
offset=0,
limit=10,
timeout_ms=2_000,
)
with self.assertRaisesRegex(
TabularSourceUnavailableError,
"configuration is unavailable",
):
self.adapter.inspect(
self.session,
principal("tenant-2"),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
def test_endpoint_query_credentials_are_rejected_before_connection(self) -> None:
self.configuration.endpoint_url = f"{self.source_url}?password=not-allowed"
self.session.commit()
with self.assertRaisesRegex(
TabularSourceValidationError,
"query parameters must not contain credentials",
):
self.adapter.inspect(
self.session,
principal(),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
if __name__ == "__main__":
unittest.main()
+313
View File
@@ -0,0 +1,313 @@
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from types import SimpleNamespace
from sqlalchemy import Column, Integer, MetaData, String, Table, create_engine
from sqlalchemy.orm import sessionmaker
from govoplan_core.auth import ApiPrincipal
from govoplan_core.core.access import PrincipalRef
from govoplan_core.core.files import (
CAPABILITY_FILES_TABULAR_CONTENT,
ManagedTabularFile,
ManagedTabularFileContent,
)
from govoplan_core.core.tabular_sources import (
TabularReadRequest,
TabularSourceUnavailableError,
TabularSourceValidationError,
)
from govoplan_core.db.base import Base
from govoplan_core.security.credential_envelopes import CredentialEnvelope
from govoplan_connectors.backend.db.models import (
ConnectorConfiguration,
ConnectorDefinition,
ConnectorTabularSource,
)
from govoplan_connectors.backend.tabular_adapters import PostgresqlTabularAdapter
from govoplan_connectors.backend.tabular_sources import SqlTabularSourceProvider
def principal(tenant_id: str = "tenant-1") -> ApiPrincipal:
return ApiPrincipal(
principal=PrincipalRef(
account_id="account-1",
membership_id="membership-1",
tenant_id=tenant_id,
scopes=frozenset(
{
"connectors:source:read",
"connectors:source:write",
"files:file:read",
"files:file:download",
}
),
),
account=object(),
user=SimpleNamespace(id="user-1"),
)
class _ManagedFiles:
def __init__(self) -> None:
self.current = "version-1"
self.payloads = {
"version-1": b"id,name\n1,Ada\n2,Lin\n",
"version-2": b"id,name,active\n1,Ada,true\n2,Lin,false\n",
}
def _metadata(self, version_id: str) -> ManagedTabularFile:
payload = self.payloads[version_id]
return ManagedTabularFile(
file_asset_id="asset-1",
file_version_id=version_id,
filename="people.csv",
display_path="Imports/people.csv",
content_type="text/csv",
size_bytes=len(payload),
sha256=("a" if version_id == "version-1" else "b") * 64,
current_version=version_id == self.current,
)
def list_tabular_files(self, session, principal, *, query="", limit=100):
del session, principal, query, limit
return (self._metadata(self.current),)
def get_tabular_file(
self,
session,
principal,
*,
file_asset_id,
file_version_id=None,
):
del session, principal
if file_asset_id != "asset-1":
return None
return self._metadata(file_version_id or self.current)
def read_tabular_file(
self,
session,
principal,
*,
file_asset_id,
file_version_id,
max_bytes,
):
del session, principal, file_asset_id, max_bytes
return ManagedTabularFileContent(
file=self._metadata(file_version_id),
payload=self.payloads[file_version_id],
)
class _Registry:
def __init__(self, files) -> None:
self.files = files
def has_capability(self, name):
return name == CAPABILITY_FILES_TABULAR_CONTENT
def require_capability(self, name):
if not self.has_capability(name):
raise KeyError(name)
return self.files
class ConnectorTabularOriginProviderTests(unittest.TestCase):
def setUp(self) -> None:
self.catalog_engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(
self.catalog_engine,
tables=(
ConnectorTabularSource.__table__,
ConnectorDefinition.__table__,
ConnectorConfiguration.__table__,
CredentialEnvelope.__table__,
),
)
self.session = sessionmaker(bind=self.catalog_engine)()
self.files = _ManagedFiles()
self.directory = tempfile.TemporaryDirectory(
prefix="govoplan-connectors-origin-provider-"
)
source_path = Path(self.directory.name) / "source.db"
self.sql_url = f"sqlite+pysqlite:///{source_path}"
source_engine = create_engine(self.sql_url)
metadata = MetaData()
source_table = Table(
"monthly_cases",
metadata,
Column("case_id", String, nullable=False),
Column("amount", Integer, nullable=True),
)
metadata.create_all(source_engine)
with source_engine.begin() as connection:
connection.execute(
source_table.insert(),
(
{"case_id": "A-1", "amount": 12},
{"case_id": "A-2", "amount": None},
),
)
source_engine.dispose()
definition = ConnectorDefinition(
id="definition-1",
tenant_id="tenant-1",
definition_key="postgresql.reader",
name="PostgreSQL reader",
status="active",
current_revision=1,
local_definition=True,
)
self.configuration = ConnectorConfiguration(
id="configuration-1",
tenant_id="tenant-1",
definition_id=definition.id,
name="Monthly SQL",
status="active",
endpoint_url=self.sql_url,
credential_ref=None,
base_definition_revision=1,
local_overrides={},
protected_paths=[],
effective_configuration={"provider": "sql", "protocol": "sql"},
effective_hash="configuration-hash-1",
resource_revision=1,
ambiguity_policy="manual_review",
)
self.session.add_all((definition, self.configuration))
self.session.commit()
self.provider = SqlTabularSourceProvider(
registry=_Registry(self.files),
sql_adapter=PostgresqlTabularAdapter(allow_sqlite_for_tests=True),
)
def tearDown(self) -> None:
self.session.close()
self.catalog_engine.dispose()
self.directory.cleanup()
def test_managed_file_source_stays_pinned_until_explicit_refresh(self) -> None:
created = self.provider.create_file_source(
self.session,
principal(),
name="Managed people",
source_name="managed_people",
file_asset_id="asset-1",
)
self.session.commit()
self.files.current = "version-2"
preview = self.provider.read_source(
self.session,
principal(),
request=TabularReadRequest(source_ref=created.ref, limit=10),
)
refreshed = self.provider.refresh_source(
self.session,
principal(),
source_ref=created.ref,
)
self.assertTrue(created.ref.startswith("file:"))
self.assertEqual("file_backed", preview.source.source_mode)
self.assertEqual("version-1", preview.source.metadata["file_version_id"])
self.assertEqual(
"files.newer_version_available",
preview.diagnostics[0].code,
)
self.assertEqual("version-2", refreshed.metadata["file_version_id"])
self.assertEqual("2", refreshed.schema_version)
self.assertEqual(3, len(refreshed.schema))
def test_sql_source_projects_and_blocks_changed_configuration_until_refresh(self) -> None:
created = self.provider.create_sql_source(
self.session,
principal(),
name="Monthly cases",
source_name="monthly_cases",
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
self.session.commit()
preview = self.provider.read_source(
self.session,
principal(),
request=TabularReadRequest(
source_ref=created.ref,
columns=("case_id",),
limit=1,
),
)
self.assertTrue(created.ref.startswith("sql:"))
self.assertEqual("live", preview.source.source_mode)
self.assertEqual(({"case_id": "A-1"},), preview.rows)
self.assertEqual("preview.row_limit_reached", preview.diagnostics[-1].code)
self.assertIsNone(
self.provider.get_source(
self.session,
principal("tenant-2"),
source_ref=created.ref,
)
)
self.configuration.effective_hash = "configuration-hash-2"
self.configuration.resource_revision = 2
self.session.commit()
with self.assertRaisesRegex(
TabularSourceValidationError,
"configuration changed",
):
self.provider.read_source(
self.session,
principal(),
request=TabularReadRequest(source_ref=created.ref),
)
refreshed = self.provider.refresh_source(
self.session,
principal(),
source_ref=created.ref,
)
self.assertEqual("2", refreshed.schema_version)
self.assertEqual("configuration-hash-2", refreshed.metadata["configuration_hash"])
def test_inactive_or_stale_sql_credentials_have_sanitized_diagnostics(self) -> None:
self.configuration.endpoint_url = (
"postgresql+psycopg://db.example.invalid/govoplan"
)
self.configuration.credential_ref = "missing-credential"
self.session.commit()
adapter = PostgresqlTabularAdapter()
with self.assertRaisesRegex(
TabularSourceUnavailableError,
"credential is unavailable, inactive, or outside its allowed scope",
):
adapter.inspect(
self.session,
principal(),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
self.configuration.status = "disabled"
self.session.commit()
with self.assertRaisesRegex(
TabularSourceUnavailableError,
"configuration is not active",
):
adapter.inspect(
self.session,
principal(),
configuration_id=self.configuration.id,
table_name="monthly_cases",
)
if __name__ == "__main__":
unittest.main()
+29
View File
@@ -0,0 +1,29 @@
{
"name": "@govoplan/connectors-webui",
"version": "0.1.21",
"private": true,
"type": "module",
"main": "src/index.ts",
"module": "src/index.ts",
"types": "src/index.ts",
"exports": {
".": {
"types": "./src/index.ts",
"import": "./src/index.ts"
},
"./styles/connectors.css": "./src/styles/connectors.css"
},
"peerDependencies": {
"@govoplan/core-webui": "^0.1.18",
"react": ">=19.2.7 <20",
"react-dom": ">=19.2.7 <20"
},
"peerDependenciesMeta": {
"@govoplan/core-webui": {
"optional": true
}
},
"scripts": {
"test:connector-governance-ui": "node tests/connector-governance-ui-structure.test.mjs"
}
}
+212
View File
@@ -0,0 +1,212 @@
import { apiFetch, apiPath, type ApiSettings } from "@govoplan/core-webui";
export type KnowledgeDiagnostic = {
severity: "info" | "warning" | "error";
code: string;
message: string;
object_ref?: string | null;
field?: string | null;
retryable: boolean;
details: Record<string, unknown>;
};
export type KnowledgeNamespaceMapping = {
source_namespace_id: number;
source_name: string;
target_space_ref: string;
target_path_prefix: string;
include: boolean;
visibility?: "tenant" | "restricted" | null;
acl_tokens: string[];
};
export type KnowledgeProfile = {
id: string;
tenant_id: string;
configuration_id: string;
status: "active" | "paused";
product: string;
product_version?: string | null;
desired_maturity: "discover" | "link" | "search" | "read" | "publish" | "synchronize" | "migrate";
discovered_maturity: string;
source_authority_mode: "external_authoritative" | "external_mirror" | "governed_sync" | "linked_reference";
default_visibility: "tenant" | "restricted";
default_acl_tokens: string[];
namespace_mappings: KnowledgeNamespaceMapping[];
capabilities: string[];
discovery_revision?: string | null;
health_status: string;
health_details: Record<string, unknown>;
discovered_at?: string | null;
last_sync_cursor?: string | null;
last_high_watermark?: string | null;
resource_revision: number;
credential_reference_present: boolean;
endpoint_configured: boolean;
created_at: string;
updated_at: string;
};
export type KnowledgeObject = {
id: string;
profile_id: string;
object_type: string;
external_id: string;
external_page_id?: string | null;
external_revision_id?: string | null;
namespace_id?: number | null;
title: string;
canonical_url?: string | null;
status: string;
source_revision: string;
visibility: string;
acl_tokens: string[];
mapped_data: Record<string, unknown>;
observed_at: string;
resource_revision: number;
};
export type KnowledgeRun = {
id: string;
profile_id: string;
mode: string;
idempotency_key: string;
status: string;
cursor_before?: string | null;
cursor_after?: string | null;
high_watermark?: string | null;
counts: Record<string, number>;
effects: Array<Record<string, unknown>>;
diagnostics: KnowledgeDiagnostic[];
provenance: Record<string, unknown>;
started_at: string;
finished_at?: string | null;
created_at: string;
};
export type KnowledgeDiscovery = {
profile: KnowledgeProfile;
product: string;
product_version?: string | null;
capabilities: string[];
namespaces: Array<Record<string, unknown>>;
extensions: Array<Record<string, unknown>>;
maturity: string;
health_status: string;
diagnostics: KnowledgeDiagnostic[];
revision: string;
};
export type KnowledgeMigrationPreview = {
run: KnowledgeRun;
target_space_ref: string;
source_revision: string;
source_fingerprint: string;
summary: Record<string, number>;
effects: Array<Record<string, unknown>>;
diagnostics: KnowledgeDiagnostic[];
truncated: boolean;
can_apply: boolean;
};
export type KnowledgePublishResult = {
run: KnowledgeRun;
external_reference: Record<string, unknown>;
accepted: boolean;
outcome_unknown: boolean;
};
const ROOT = "/api/v1/connectors/knowledge";
export async function listKnowledgeProfiles(settings: ApiSettings): Promise<KnowledgeProfile[]> {
const response = await apiFetch<{ items: KnowledgeProfile[] }>(settings, `${ROOT}/profiles`);
return response.items;
}
export function createKnowledgeProfile(
settings: ApiSettings,
payload: Record<string, unknown>
): Promise<KnowledgeProfile> {
return apiFetch(settings, `${ROOT}/profiles`, {
method: "POST",
body: JSON.stringify(payload)
});
}
export function updateKnowledgeProfile(
settings: ApiSettings,
profileId: string,
payload: Record<string, unknown>
): Promise<KnowledgeProfile> {
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}`, {
method: "PUT",
body: JSON.stringify(payload)
});
}
export function discoverKnowledgeProfile(
settings: ApiSettings,
profileId: string
): Promise<KnowledgeDiscovery> {
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/discover`, {
method: "POST"
});
}
export function synchronizeKnowledgeProfile(
settings: ApiSettings,
profileId: string,
payload: Record<string, unknown>
): Promise<KnowledgeRun> {
return apiFetch(settings, `${ROOT}/profiles/${encodeURIComponent(profileId)}/sync`, {
method: "POST",
body: JSON.stringify(payload)
});
}
export async function listKnowledgeObjects(
settings: ApiSettings,
profileId: string
): Promise<KnowledgeObject[]> {
const response = await apiFetch<{ items: KnowledgeObject[] }>(
settings,
apiPath(`${ROOT}/profiles/${encodeURIComponent(profileId)}/objects`, { limit: 100 })
);
return response.items;
}
export async function listKnowledgeRuns(
settings: ApiSettings,
profileId?: string
): Promise<KnowledgeRun[]> {
const response = await apiFetch<{ items: KnowledgeRun[] }>(
settings,
apiPath(`${ROOT}/runs`, { profile_id: profileId || undefined, limit: 100 })
);
return response.items;
}
export function previewKnowledgeMigration(
settings: ApiSettings,
profileId: string,
payload: Record<string, unknown>
): Promise<KnowledgeMigrationPreview> {
return apiFetch(
settings,
`${ROOT}/profiles/${encodeURIComponent(profileId)}/migration-dry-runs`,
{ method: "POST", body: JSON.stringify(payload) }
);
}
export function publishKnowledgePage(
settings: ApiSettings,
profileId: string,
externalPageId: string,
payload: Record<string, unknown>
): Promise<KnowledgePublishResult> {
return apiFetch(
settings,
`${ROOT}/profiles/${encodeURIComponent(profileId)}/pages/${encodeURIComponent(externalPageId)}/publish`,
{ method: "POST", body: JSON.stringify(payload) }
);
}
+204
View File
@@ -0,0 +1,204 @@
import { apiFetch, apiPath, type ApiSettings } from "@govoplan/core-webui";
export type GovernedConnectorSpecification = {
provider: string;
protocol: string;
capabilities: string[];
input_schema: Record<string, unknown>;
output_schema: Record<string, unknown>;
mapping: {
version: string;
rules: Array<{
source: string;
target: string;
required?: boolean;
default?: unknown;
}>;
};
validation_rules: Array<{
kind: "required" | "one_of" | "unique";
field: string;
values?: unknown[];
severity: "warning" | "error";
code: string;
message: string;
}>;
dry_run: {
supported: boolean;
simulation_supported: boolean;
sample_rows: Array<Record<string, unknown>>;
max_items: number;
redacted_fields: string[];
};
audit: {
event_prefix: string;
expected_events: string[];
evidence_fields: string[];
};
privacy_classification: "public" | "internal" | "confidential" | "restricted";
retention_class: string;
operational_limits: Record<string, unknown>;
retry_policy: Record<string, unknown>;
};
export type ConnectorDefinition = {
id: string;
definition_key: string;
name: string;
description?: string | null;
status: string;
current_revision: number;
source_package?: string | null;
local_definition: boolean;
definition_hash: string;
specification: GovernedConnectorSpecification;
updated_at: string;
};
export type ConnectorConfiguration = {
id: string;
definition_id: string;
definition_key: string;
definition_name: string;
name: string;
status: "draft" | "active" | "disabled";
endpoint_url?: string | null;
credential_ref?: string | null;
base_definition_revision: number;
latest_definition_revision: number;
update_available: boolean;
local_overrides: Record<string, unknown>;
protected_paths: string[];
effective_configuration: GovernedConnectorSpecification;
effective_hash: string;
resource_revision: number;
ambiguity_policy: "manual_review" | "quarantine" | "reject";
updated_at: string;
};
export type ConnectorRun = {
id: string;
configuration_id: string;
mode: "dry_run" | "simulation";
idempotency_key: string;
status: string;
review_state: string;
definition_revision: number;
configuration_revision: number;
configuration_hash: string;
input_hash: string;
summary: Record<string, number | boolean>;
effects: Array<Record<string, unknown>>;
diagnostics: Array<Record<string, unknown>>;
provenance: Record<string, unknown>;
reviewed_by?: string | null;
reviewed_at?: string | null;
review_reason?: string | null;
created_at: string;
};
export type ConnectorConfigurationDraft = {
name: string;
status: ConnectorConfiguration["status"];
endpoint_url: string;
credential_ref: string;
local_overrides: string;
ambiguity_policy: ConnectorConfiguration["ambiguity_policy"];
};
const ROOT = "/api/v1/connectors/governed";
export async function listConnectorDefinitions(
settings: ApiSettings
): Promise<ConnectorDefinition[]> {
const response = await apiFetch<{ items: ConnectorDefinition[] }>(
settings,
`${ROOT}/definitions`
);
return response.items;
}
export async function upsertConnectorDefinition(
settings: ApiSettings,
payload: Record<string, unknown>
): Promise<ConnectorDefinition> {
return apiFetch(settings, `${ROOT}/definitions`, {
method: "POST",
body: JSON.stringify(payload)
});
}
export async function listConnectorConfigurations(
settings: ApiSettings
): Promise<ConnectorConfiguration[]> {
const response = await apiFetch<{ items: ConnectorConfiguration[] }>(
settings,
`${ROOT}/configurations`
);
return response.items;
}
export function createConnectorConfiguration(
settings: ApiSettings,
payload: Record<string, unknown>
): Promise<ConnectorConfiguration> {
return apiFetch(settings, `${ROOT}/configurations`, {
method: "POST",
body: JSON.stringify(payload)
});
}
export function updateConnectorConfiguration(
settings: ApiSettings,
configurationId: string,
payload: Record<string, unknown>
): Promise<ConnectorConfiguration> {
return apiFetch(
settings,
`${ROOT}/configurations/${encodeURIComponent(configurationId)}`,
{ method: "PUT", body: JSON.stringify(payload) }
);
}
export async function listConnectorRuns(
settings: ApiSettings,
configurationId?: string
): Promise<ConnectorRun[]> {
const response = await apiFetch<{ items: ConnectorRun[] }>(
settings,
apiPath(`${ROOT}/runs`, {
configuration_id: configurationId || undefined,
limit: 100
})
);
return response.items;
}
export function executeConnectorRun(
settings: ApiSettings,
configurationId: string,
mode: "dry-runs" | "simulations",
payload: Record<string, unknown>
): Promise<ConnectorRun> {
return apiFetch(
settings,
`${ROOT}/configurations/${encodeURIComponent(configurationId)}/${mode}`,
{ method: "POST", body: JSON.stringify(payload) }
);
}
export function reviewConnectorRun(
settings: ApiSettings,
runId: string,
decision: "approved" | "rejected",
reason: string
): Promise<ConnectorRun> {
return apiFetch(
settings,
`${ROOT}/runs/${encodeURIComponent(runId)}/review`,
{
method: "POST",
body: JSON.stringify({ decision, reason })
}
);
}
@@ -0,0 +1,661 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import {
AdminPageLayout,
Button,
Card,
DataGrid,
Dialog,
FilterBar,
FormField,
FormGrid,
MetricCard,
MetricGrid,
PageActionBar,
SelectionList,
SelectionListItem,
SelectionListItemContent,
StatePanel,
StatusBadge,
TableActionGroup,
WorkspaceLayout,
formatDateTime,
hasScope,
useUnsavedChanges,
useUnsavedDraftGuard,
type ApiSettings,
type AuthInfo,
type DataGridColumn
} from "@govoplan/core-webui";
import {
createConnectorConfiguration,
executeConnectorRun,
listConnectorConfigurations,
listConnectorDefinitions,
listConnectorRuns,
reviewConnectorRun,
updateConnectorConfiguration,
upsertConnectorDefinition,
type ConnectorConfiguration,
type ConnectorConfigurationDraft,
type ConnectorDefinition,
type ConnectorRun
} from "../api/governedConnectors";
type Props = {
settings: ApiSettings;
auth: AuthInfo;
};
const EMPTY_DRAFT: ConnectorConfigurationDraft = {
name: "",
status: "draft",
endpoint_url: "",
credential_ref: "",
local_overrides: "{}",
ambiguity_policy: "manual_review"
};
const EXAMPLE_DEFINITION = JSON.stringify({
definition_key: "example.reference-data",
name: "Example reference data",
description: "Locally governed example connector",
origin: "local",
specification: {
provider: "example-provider",
protocol: "rest",
capabilities: ["discover", "read", "dry_run"],
input_schema: { type: "object" },
output_schema: { type: "object" },
mapping: {
version: "1",
rules: [{ source: "id", target: "record.id", required: true }]
},
validation_rules: [{
kind: "unique",
field: "record.id",
severity: "error",
code: "record.id.ambiguous",
message: "The record identifier is not unique."
}],
dry_run: {
supported: true,
simulation_supported: true,
sample_rows: [{ id: "sample-1" }],
max_items: 500,
redacted_fields: []
},
audit: {
event_prefix: "connectors.example",
expected_events: ["simulation.completed"],
evidence_fields: ["input_hash", "configuration_hash"]
},
privacy_classification: "internal",
retention_class: "connector-preview-30d",
operational_limits: { timeout_seconds: 30 },
retry_policy: { max_attempts: 2 }
}
}, null, 2);
export default function ConnectorGovernancePage({ settings, auth }: Props) {
const [definitions, setDefinitions] = useState<ConnectorDefinition[]>([]);
const [configurations, setConfigurations] = useState<ConnectorConfiguration[]>([]);
const [runs, setRuns] = useState<ConnectorRun[]>([]);
const [selectedId, setSelectedId] = useState("");
const [draft, setDraft] = useState<ConnectorConfigurationDraft>(EMPTY_DRAFT);
const [savedKey, setSavedKey] = useState("");
const [search, setSearch] = useState("");
const [loading, setLoading] = useState(true);
const [busy, setBusy] = useState(false);
const [error, setError] = useState("");
const [success, setSuccess] = useState("");
const [definitionOpen, setDefinitionOpen] = useState(false);
const [definitionJson, setDefinitionJson] = useState(EXAMPLE_DEFINITION);
const [configurationOpen, setConfigurationOpen] = useState(false);
const [newDefinitionId, setNewDefinitionId] = useState("");
const [newDraft, setNewDraft] = useState<ConnectorConfigurationDraft>(EMPTY_DRAFT);
const [sampleJson, setSampleJson] = useState("[]");
const [externalRevision, setExternalRevision] = useState("");
const [reviewRun, setReviewRun] = useState<ConnectorRun | null>(null);
const [reviewReason, setReviewReason] = useState("");
const { requestDiscard } = useUnsavedChanges();
const selected = configurations.find((item) => item.id === selectedId) ?? null;
const canAdmin = hasScope(auth, "connectors:source:admin");
const canExecute = canAdmin || hasScope(auth, "connectors:source:write");
const dirty = Boolean(selected && draftKey(draft) !== savedKey);
const applyConfiguration = useCallback((item: ConnectorConfiguration | null) => {
const next = item ? draftFromConfiguration(item) : EMPTY_DRAFT;
setDraft(next);
setSavedKey(item ? draftKey(next) : "");
setSampleJson(JSON.stringify(
item?.effective_configuration.dry_run.sample_rows ?? [],
null,
2
));
}, []);
const reload = useCallback(async (preferredId?: string) => {
setLoading(true);
setError("");
try {
const [nextDefinitions, nextConfigurations] = await Promise.all([
listConnectorDefinitions(settings),
listConnectorConfigurations(settings)
]);
const nextId = preferredId && nextConfigurations.some((item) => item.id === preferredId)
? preferredId
: nextConfigurations.some((item) => item.id === selectedId)
? selectedId
: nextConfigurations[0]?.id ?? "";
const nextRuns = await listConnectorRuns(settings, nextId || undefined);
setDefinitions(nextDefinitions);
setConfigurations(nextConfigurations);
setRuns(nextRuns);
setSelectedId(nextId);
applyConfiguration(nextConfigurations.find((item) => item.id === nextId) ?? null);
if (!newDefinitionId && nextDefinitions[0]) setNewDefinitionId(nextDefinitions[0].id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setLoading(false);
}
}, [applyConfiguration, newDefinitionId, selectedId, settings]);
useEffect(() => {
void reload();
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
const visibleConfigurations = useMemo(() => {
const needle = search.trim().toLocaleLowerCase();
return configurations.filter((item) => !needle ||
`${item.name} ${item.definition_name} ${item.status}`
.toLocaleLowerCase()
.includes(needle));
}, [configurations, search]);
const save = async (): Promise<boolean> => {
if (!selected || !canAdmin) return false;
setBusy(true);
setError("");
try {
const overrides = parseObject(draft.local_overrides, "Local overrides");
const updated = await updateConnectorConfiguration(settings, selected.id, {
expected_revision: selected.resource_revision,
name: draft.name.trim(),
status: draft.status,
endpoint_url: draft.endpoint_url.trim() || null,
credential_ref: draft.credential_ref.trim() || null,
local_overrides: overrides,
ambiguity_policy: draft.ambiguity_policy
});
setSuccess("Connector configuration saved.");
await reload(updated.id);
return true;
} catch (caught) {
setError(errorMessage(caught));
return false;
} finally {
setBusy(false);
}
};
useUnsavedDraftGuard({
dirty,
onSave: save,
onDiscard: () => applyConfiguration(selected),
title: "Unsaved connector changes",
message: "Save or discard the current connector changes before continuing."
});
const selectConfiguration = (item: ConnectorConfiguration) => {
if (item.id === selectedId) return;
requestDiscard(() => {
setSelectedId(item.id);
applyConfiguration(item);
setRuns([]);
setError("");
setSuccess("");
void listConnectorRuns(settings, item.id).then(setRuns).catch((caught) => {
setError(errorMessage(caught));
});
});
};
const createDefinition = async () => {
setBusy(true);
setError("");
try {
const payload = parseObject(definitionJson, "Definition");
const created = await upsertConnectorDefinition(settings, payload);
setDefinitionOpen(false);
setSuccess("Connector definition revision saved.");
setNewDefinitionId(created.id);
await reload(selectedId);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const createConfiguration = async () => {
if (!newDefinitionId || !newDraft.name.trim()) return;
setBusy(true);
setError("");
try {
const created = await createConnectorConfiguration(settings, {
definition_id: newDefinitionId,
name: newDraft.name.trim(),
status: newDraft.status,
endpoint_url: newDraft.endpoint_url.trim() || null,
credential_ref: newDraft.credential_ref.trim() || null,
local_overrides: parseObject(newDraft.local_overrides, "Local overrides"),
ambiguity_policy: newDraft.ambiguity_policy
});
setConfigurationOpen(false);
setNewDraft(EMPTY_DRAFT);
setSuccess("Connector configuration created.");
await reload(created.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const adoptUpdate = async () => {
if (!selected || dirty || !selected.update_available) return;
setBusy(true);
setError("");
try {
const updated = await updateConnectorConfiguration(settings, selected.id, {
expected_revision: selected.resource_revision,
adopt_latest_definition: true
});
setSuccess("Package revision adopted; protected local overrides were reapplied.");
await reload(updated.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const run = async (mode: "dry-runs" | "simulations") => {
if (!selected || dirty) return;
setBusy(true);
setError("");
try {
const inputRows = parseRows(sampleJson);
const created = await executeConnectorRun(settings, selected.id, mode, {
idempotency_key: `${mode}-${crypto.randomUUID()}`,
input_rows: inputRows,
external_revision: externalRevision.trim() || null
});
setSuccess(`${mode === "dry-runs" ? "Dry-run" : "Simulation"} completed with status ${created.status}.`);
setRuns(await listConnectorRuns(settings, selected.id));
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const decideReview = async (decision: "approved" | "rejected") => {
if (!reviewRun || reviewReason.trim().length < 5) return;
setBusy(true);
setError("");
try {
await reviewConnectorRun(settings, reviewRun.id, decision, reviewReason.trim());
setReviewRun(null);
setReviewReason("");
setSuccess(`Simulation ${decision}.`);
setRuns(await listConnectorRuns(settings, selectedId));
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const runColumns = useMemo<DataGridColumn<ConnectorRun>[]>(() => [
{
id: "created",
header: "Run",
width: 190,
sortable: true,
value: (row) => row.created_at,
render: (row) => <>{row.mode}<br /><span className="muted">{formatDateTime(row.created_at)}</span></>
},
{
id: "status",
header: "Status",
width: 150,
sortable: true,
value: (row) => row.status,
render: (row) => <StatusBadge status={row.status} label={row.status.replaceAll("_", " ")} />
},
{
id: "summary",
header: "Effects",
width: "1fr",
minWidth: 220,
render: (row) => `${row.summary.total ?? 0} total · ${row.summary.ambiguous ?? 0} ambiguous · ${row.summary.errors ?? 0} errors`
},
{
id: "revision",
header: "Evidence",
width: 170,
render: (row) => <code title={row.configuration_hash}>r{row.configuration_revision} · {row.input_hash.slice(0, 8)}</code>
},
{
id: "actions",
header: "Actions",
width: 90,
sticky: "end",
align: "right",
render: (row) => <TableActionGroup actions={[{
id: "review",
label: "Review result",
icon: <span></span>,
applicable: ["pending", "quarantined"].includes(row.review_state),
disabled: !canAdmin || busy,
disabledReason: !canAdmin ? "Connector administration permission is required." : undefined,
onClick: () => setReviewRun(row)
}]} />
}
], [busy, canAdmin]);
const actionBar = <PageActionBar
variant="editor"
state={busy ? "saving" : dirty ? "dirty" : "clean"}
refreshable
reloadAction={{ onReload: () => void reload(selectedId), loading }}
primaryActions={<>
<Button onClick={() => setDefinitionOpen(true)} disabled={!canAdmin || busy}>
New definition
</Button>
<Button onClick={() => setConfigurationOpen(true)} disabled={!canAdmin || busy || !definitions.length}>
New configuration
</Button>
{selected?.update_available ? <Button
variant="secondary"
onClick={() => void adoptUpdate()}
disabled={!canAdmin || busy || dirty}
disabledReason={dirty ? "Save or discard local edits before adopting a package update." : undefined}
>
Adopt package revision {selected.latest_definition_revision}
</Button> : null}
</>}
discardAction={{
label: "Discard changes",
disabled: !selected,
onClick: () => applyConfiguration(selected)
}}
saveAction={{
label: "Save",
disabled: !selected || !canAdmin || busy,
disabledReason: !canAdmin ? "Connector administration permission is required." : undefined,
onClick: () => void save()
}}
/>;
return <AdminPageLayout
archetype="workspace"
title="Connector governance"
description="Version schemas and mappings, protect local overrides, and review deterministic simulations before provider-specific writes."
loading={loading && !configurations.length}
error={error}
success={success}
actions={actionBar}
className="connector-governance-page"
helpContextId="connectors.admin.governed-configurations"
>
<MetricGrid columns={4} density="compact" minimum="compact">
<MetricCard label="Definitions" value={definitions.length} />
<MetricCard label="Configurations" value={configurations.length} />
<MetricCard label="Updates available" value={configurations.filter((item) => item.update_available).length} tone="warning" />
<MetricCard label="Awaiting review" value={runs.filter((item) => ["pending", "quarantined"].includes(item.review_state)).length} tone="warning" />
</MetricGrid>
<WorkspaceLayout
variant="split"
primarySize="compact"
surface="contained"
primaryScrollable={false}
contentScrollable={false}
primaryLabel="Connector configurations"
contentLabel="Configuration details"
primary={<div className="connector-governance-list">
<FilterBar surface="panel">
<input
type="search"
value={search}
onChange={(event) => setSearch(event.target.value)}
placeholder="Search configurations"
aria-label="Search connector configurations"
/>
</FilterBar>
<SelectionList variant="navigation" label="Connector configurations">
{visibleConfigurations.map((item) => <SelectionListItem
key={item.id}
selected={item.id === selectedId}
onClick={() => selectConfiguration(item)}
>
<SelectionListItemContent
title={item.name}
description={`${item.definition_name} · revision ${item.base_definition_revision}`}
/>
<StatusBadge status={item.update_available ? "warning" : item.status} />
</SelectionListItem>)}
{!visibleConfigurations.length
? <StatePanel size="compact" description="No matching configurations." />
: null}
</SelectionList>
</div>}
>
{!selected ? <StatePanel
size="fill"
title="Connector configurations"
description="Create or select a configuration to inspect its pinned definition and simulation evidence."
/> : <div className="connector-governance-detail">
<Card title={selected.name}>
<div className="connector-revision-line">
<StatusBadge status={selected.status} />
<span>Definition revision {selected.base_definition_revision}</span>
{selected.update_available
? <StatusBadge status="warning" label={`Revision ${selected.latest_definition_revision} available`} />
: null}
<code title={selected.effective_hash}>{selected.effective_hash.slice(0, 12)}</code>
</div>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Name">
<input value={draft.name} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, name: event.target.value })} />
</FormField>
<FormField label="Status">
<select value={draft.status} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, status: event.target.value as ConnectorConfigurationDraft["status"] })}>
<option value="draft">Draft</option>
<option value="active">Active</option>
<option value="disabled">Disabled</option>
</select>
</FormField>
<FormField label="Endpoint URL" hint="Credentials are rejected in URLs.">
<input value={draft.endpoint_url} disabled={!canAdmin || busy} placeholder="https://provider.example/api" onChange={(event) => setDraft({ ...draft, endpoint_url: event.target.value })} />
</FormField>
<FormField label="Credential reference" hint="Reference an approved secret; do not paste a secret.">
<input value={draft.credential_ref} disabled={!canAdmin || busy} placeholder="vault://connectors/provider" onChange={(event) => setDraft({ ...draft, credential_ref: event.target.value })} />
</FormField>
<FormField label="Ambiguous-result policy">
<select value={draft.ambiguity_policy} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, ambiguity_policy: event.target.value as ConnectorConfigurationDraft["ambiguity_policy"] })}>
<option value="manual_review">Manual review</option>
<option value="quarantine">Quarantine</option>
<option value="reject">Reject</option>
</select>
</FormField>
<FormField label="Protected local overrides" hint="JSON object leaf paths remain protected when package revisions are adopted.">
<textarea rows={8} value={draft.local_overrides} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, local_overrides: event.target.value })} />
</FormField>
</FormGrid>
<p className="muted">
Protected paths: {selected.protected_paths.length ? selected.protected_paths.join(", ") : "none"}
</p>
</Card>
<Card title="Dry-run and simulation">
<p className="muted">Runs never apply changes. They retain redacted, revision-pinned evidence for review.</p>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Sample input rows" hint="JSON array, bounded by the definition's maximum.">
<textarea rows={9} value={sampleJson} disabled={!canExecute || busy} onChange={(event) => setSampleJson(event.target.value)} />
</FormField>
<FormField label="External revision" hint="Optional provider ETag, cursor, or snapshot revision.">
<input value={externalRevision} disabled={!canExecute || busy} onChange={(event) => setExternalRevision(event.target.value)} />
</FormField>
</FormGrid>
<div className="connector-run-actions">
<Button onClick={() => void run("dry-runs")} disabled={!canExecute || busy || dirty}>Run dry-run</Button>
<Button variant="primary" onClick={() => void run("simulations")} disabled={!canExecute || busy || dirty}>Run simulation</Button>
{dirty ? <span className="muted">Save or discard configuration changes before running.</span> : null}
</div>
</Card>
<Card title="Simulation evidence">
<DataGrid
id="connector-simulation-runs"
rows={runs}
columns={runColumns}
getRowKey={(row) => row.id}
initialFit="container"
emptyText="No dry-runs or simulations have been recorded."
/>
</Card>
<Card title="Effective governed definition">
<pre className="connector-json-preview">{JSON.stringify(selected.effective_configuration, null, 2)}</pre>
</Card>
</div>}
</WorkspaceLayout>
<Dialog
open={definitionOpen}
title="Create or revise connector definition"
onClose={() => !busy && setDefinitionOpen(false)}
closeDisabled={busy}
footer={<>
<Button onClick={() => setDefinitionOpen(false)} disabled={busy}>Cancel</Button>
<Button variant="primary" onClick={() => void createDefinition()} disabled={!canAdmin || busy}>Save definition revision</Button>
</>}
>
<p className="muted">The definition is schema-validated and every changed specification creates an immutable revision. Package definitions must name their package reference.</p>
<FormField label="Governed definition JSON">
<textarea className="connector-definition-editor" rows={24} value={definitionJson} disabled={busy} onChange={(event) => setDefinitionJson(event.target.value)} />
</FormField>
</Dialog>
<Dialog
open={configurationOpen}
title="Create connector configuration"
onClose={() => !busy && setConfigurationOpen(false)}
closeDisabled={busy}
footer={<>
<Button onClick={() => setConfigurationOpen(false)} disabled={busy}>Cancel</Button>
<Button variant="primary" onClick={() => void createConfiguration()} disabled={busy || !newDefinitionId || !newDraft.name.trim()}>Create configuration</Button>
</>}
>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Definition">
<select value={newDefinitionId} disabled={busy} onChange={(event) => setNewDefinitionId(event.target.value)}>
{definitions.map((item) => <option key={item.id} value={item.id}>{item.name} · revision {item.current_revision}</option>)}
</select>
</FormField>
<FormField label="Name">
<input value={newDraft.name} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, name: event.target.value })} />
</FormField>
<FormField label="Endpoint URL">
<input value={newDraft.endpoint_url} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, endpoint_url: event.target.value })} />
</FormField>
<FormField label="Credential reference">
<input value={newDraft.credential_ref} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, credential_ref: event.target.value })} />
</FormField>
<FormField label="Ambiguous-result policy">
<select value={newDraft.ambiguity_policy} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, ambiguity_policy: event.target.value as ConnectorConfigurationDraft["ambiguity_policy"] })}>
<option value="manual_review">Manual review</option>
<option value="quarantine">Quarantine</option>
<option value="reject">Reject</option>
</select>
</FormField>
<FormField label="Local overrides">
<textarea rows={7} value={newDraft.local_overrides} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, local_overrides: event.target.value })} />
</FormField>
</FormGrid>
</Dialog>
<Dialog
open={Boolean(reviewRun)}
title="Review ambiguous connector result"
onClose={() => !busy && setReviewRun(null)}
closeDisabled={busy}
footer={<>
<Button onClick={() => setReviewRun(null)} disabled={busy}>Cancel</Button>
<Button variant="danger" onClick={() => void decideReview("rejected")} disabled={busy || reviewReason.trim().length < 5}>Reject</Button>
<Button variant="primary" onClick={() => void decideReview("approved")} disabled={busy || reviewReason.trim().length < 5}>Approve</Button>
</>}
>
<p>Review {reviewRun?.summary.ambiguous ?? 0} ambiguous effects against the retained input and configuration hashes before deciding.</p>
<FormField label="Decision reason">
<textarea rows={4} value={reviewReason} disabled={busy} onChange={(event) => setReviewReason(event.target.value)} />
</FormField>
<pre className="connector-json-preview">{JSON.stringify(reviewRun?.diagnostics ?? [], null, 2)}</pre>
</Dialog>
</AdminPageLayout>;
}
function draftFromConfiguration(item: ConnectorConfiguration): ConnectorConfigurationDraft {
return {
name: item.name,
status: item.status,
endpoint_url: item.endpoint_url ?? "",
credential_ref: item.credential_ref ?? "",
local_overrides: JSON.stringify(item.local_overrides, null, 2),
ambiguity_policy: item.ambiguity_policy
};
}
function draftKey(value: ConnectorConfigurationDraft): string {
return JSON.stringify({
name: value.name.trim(),
status: value.status,
endpoint_url: value.endpoint_url.trim(),
credential_ref: value.credential_ref.trim(),
local_overrides: normalizeJson(value.local_overrides),
ambiguity_policy: value.ambiguity_policy
});
}
function normalizeJson(value: string): unknown {
try {
return JSON.parse(value);
} catch {
return value.trim();
}
}
function parseObject(value: string, label: string): Record<string, unknown> {
const parsed: unknown = JSON.parse(value);
if (!parsed || Array.isArray(parsed) || typeof parsed !== "object") {
throw new Error(`${label} must be a JSON object.`);
}
return parsed as Record<string, unknown>;
}
function parseRows(value: string): Array<Record<string, unknown>> {
const parsed: unknown = JSON.parse(value);
if (!Array.isArray(parsed) || parsed.some((item) => !item || Array.isArray(item) || typeof item !== "object")) {
throw new Error("Sample input must be a JSON array of objects.");
}
return parsed as Array<Record<string, unknown>>;
}
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
@@ -0,0 +1,561 @@
import { useCallback, useEffect, useMemo, useState } from "react";
import {
AdminPageLayout,
Button,
Card,
Dialog,
FilterBar,
FormField,
FormGrid,
MetricCard,
MetricGrid,
PageActionBar,
SelectionList,
SelectionListItem,
SelectionListItemContent,
StatePanel,
StatusBadge,
WorkspaceLayout,
formatDateTime,
hasScope,
useUnsavedChanges,
useUnsavedDraftGuard,
type ApiSettings,
type AuthInfo
} from "@govoplan/core-webui";
import {
createKnowledgeProfile,
discoverKnowledgeProfile,
listKnowledgeObjects,
listKnowledgeProfiles,
listKnowledgeRuns,
previewKnowledgeMigration,
publishKnowledgePage,
synchronizeKnowledgeProfile,
updateKnowledgeProfile,
type KnowledgeMigrationPreview,
type KnowledgeObject,
type KnowledgeProfile,
type KnowledgeRun
} from "../api/externalKnowledge";
type Props = {
settings: ApiSettings;
auth: AuthInfo;
};
type ProfileDraft = {
status: "active" | "paused";
desired_maturity: KnowledgeProfile["desired_maturity"];
source_authority_mode: KnowledgeProfile["source_authority_mode"];
default_visibility: KnowledgeProfile["default_visibility"];
default_acl_tokens: string;
namespace_mappings: string;
};
const EMPTY_DRAFT: ProfileDraft = {
status: "active",
desired_maturity: "migrate",
source_authority_mode: "external_mirror",
default_visibility: "restricted",
default_acl_tokens: "scope:connectors:knowledge:read",
namespace_mappings: JSON.stringify([{
source_namespace_id: 0,
source_name: "",
target_space_ref: "external-knowledge",
target_path_prefix: "",
include: true,
acl_tokens: []
}], null, 2)
};
export default function ExternalKnowledgePage({ settings, auth }: Props) {
const [profiles, setProfiles] = useState<KnowledgeProfile[]>([]);
const [objects, setObjects] = useState<KnowledgeObject[]>([]);
const [runs, setRuns] = useState<KnowledgeRun[]>([]);
const [selectedId, setSelectedId] = useState("");
const [draft, setDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
const [savedKey, setSavedKey] = useState("");
const [search, setSearch] = useState("");
const [loading, setLoading] = useState(true);
const [busy, setBusy] = useState(false);
const [error, setError] = useState("");
const [success, setSuccess] = useState("");
const [createOpen, setCreateOpen] = useState(false);
const [configurationId, setConfigurationId] = useState("");
const [newDraft, setNewDraft] = useState<ProfileDraft>(EMPTY_DRAFT);
const [migrationOpen, setMigrationOpen] = useState(false);
const [targetSpace, setTargetSpace] = useState("external-knowledge");
const [supportedMacros, setSupportedMacros] = useState("");
const [existingTargets, setExistingTargets] = useState("[]");
const [migration, setMigration] = useState<KnowledgeMigrationPreview | null>(null);
const [publishOpen, setPublishOpen] = useState(false);
const [externalPageId, setExternalPageId] = useState("");
const [publishTitle, setPublishTitle] = useState("");
const [publishBody, setPublishBody] = useState("");
const [publishSummary, setPublishSummary] = useState("");
const [publishRevision, setPublishRevision] = useState("");
const { requestDiscard } = useUnsavedChanges();
const selected = profiles.find((item) => item.id === selectedId) ?? null;
const canAdmin = hasScope(auth, "connectors:knowledge:admin");
const canSync = hasScope(auth, "connectors:knowledge:sync");
const canMigrate = hasScope(auth, "connectors:knowledge:migrate");
const canPublish = hasScope(auth, "connectors:knowledge:publish");
const dirty = Boolean(selected && draftKey(draft) !== savedKey);
const applyProfile = useCallback((profile: KnowledgeProfile | null) => {
const next = profile ? draftFromProfile(profile) : EMPTY_DRAFT;
setDraft(next);
setSavedKey(profile ? draftKey(next) : "");
}, []);
const reload = useCallback(async (preferredId?: string) => {
setLoading(true);
setError("");
try {
const nextProfiles = await listKnowledgeProfiles(settings);
const nextId = preferredId && nextProfiles.some((item) => item.id === preferredId)
? preferredId
: nextProfiles.some((item) => item.id === selectedId)
? selectedId
: nextProfiles[0]?.id ?? "";
const [nextObjects, nextRuns] = nextId
? await Promise.all([
listKnowledgeObjects(settings, nextId),
listKnowledgeRuns(settings, nextId)
])
: [[], []];
setProfiles(nextProfiles);
setSelectedId(nextId);
setObjects(nextObjects);
setRuns(nextRuns);
applyProfile(nextProfiles.find((item) => item.id === nextId) ?? null);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setLoading(false);
}
}, [applyProfile, selectedId, settings]);
useEffect(() => {
void reload();
}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
const save = async (): Promise<boolean> => {
if (!selected || !canAdmin) return false;
setBusy(true);
setError("");
try {
const updated = await updateKnowledgeProfile(settings, selected.id, {
expected_resource_revision: selected.resource_revision,
status: draft.status,
desired_maturity: draft.desired_maturity,
source_authority_mode: draft.source_authority_mode,
default_visibility: draft.default_visibility,
default_acl_tokens: lines(draft.default_acl_tokens),
namespace_mappings: parseArray(draft.namespace_mappings, "Namespace mappings")
});
setSuccess("External knowledge profile saved; fallback ACLs and Search projections were refreshed.");
await reload(updated.id);
return true;
} catch (caught) {
setError(errorMessage(caught));
return false;
} finally {
setBusy(false);
}
};
useUnsavedDraftGuard({
dirty,
onSave: save,
onDiscard: () => applyProfile(selected),
title: "Unsaved knowledge profile changes",
message: "Save or discard the profile changes before continuing."
});
const selectProfile = (profile: KnowledgeProfile) => {
if (profile.id === selectedId) return;
requestDiscard(() => {
setSelectedId(profile.id);
applyProfile(profile);
setObjects([]);
setRuns([]);
setMigration(null);
void Promise.all([
listKnowledgeObjects(settings, profile.id),
listKnowledgeRuns(settings, profile.id)
]).then(([nextObjects, nextRuns]) => {
setObjects(nextObjects);
setRuns(nextRuns);
}).catch((caught) => setError(errorMessage(caught)));
});
};
const createProfile = async () => {
if (!configurationId.trim()) return;
setBusy(true);
setError("");
try {
const created = await createKnowledgeProfile(settings, {
configuration_id: configurationId.trim(),
desired_maturity: newDraft.desired_maturity,
source_authority_mode: newDraft.source_authority_mode,
default_visibility: newDraft.default_visibility,
default_acl_tokens: lines(newDraft.default_acl_tokens),
namespace_mappings: parseArray(newDraft.namespace_mappings, "Namespace mappings")
});
setCreateOpen(false);
setConfigurationId("");
setNewDraft(EMPTY_DRAFT);
setSuccess("External knowledge profile created. Run discovery before synchronization.");
await reload(created.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const discover = async () => {
if (!selected || dirty) return;
setBusy(true);
setError("");
try {
const result = await discoverKnowledgeProfile(settings, selected.id);
setSuccess(`Discovered ${result.product} ${result.product_version ?? ""} at ${result.maturity} maturity with ${result.diagnostics.length} diagnostics.`);
await reload(selected.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const sync = async (forceFull: boolean) => {
if (!selected || dirty) return;
setBusy(true);
setError("");
try {
const run = await synchronizeKnowledgeProfile(settings, selected.id, {
idempotency_key: `knowledge-${forceFull ? "backfill" : "delta"}-${crypto.randomUUID()}`,
force_full: forceFull,
limit: 100
});
setSuccess(`${forceFull ? "Backfill" : "Delta"} completed with ${effectTotal(run)} effects.`);
await reload(selected.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const previewMigration = async () => {
if (!selected || !targetSpace.trim()) return;
setBusy(true);
setError("");
try {
const result = await previewKnowledgeMigration(settings, selected.id, {
idempotency_key: `knowledge-migration-${crypto.randomUUID()}`,
target_space_ref: targetSpace.trim(),
max_items: 100,
supported_macros: lines(supportedMacros),
existing_targets: parseArray(existingTargets, "Existing targets")
});
setMigration(result);
setMigrationOpen(false);
setSuccess(result.can_apply
? "Migration preview is complete and contains no blocking conflict. It did not write Wiki pages."
: "Migration preview found conflicts, errors, or truncation. It did not write Wiki pages.");
await reload(selected.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const publish = async () => {
if (!selected || !externalPageId.trim() || !publishTitle.trim()) return;
setBusy(true);
setError("");
try {
const result = await publishKnowledgePage(
settings,
selected.id,
externalPageId.trim(),
{
idempotency_key: `knowledge-publish-${crypto.randomUUID()}`,
title: publishTitle.trim(),
body: publishBody,
summary: publishSummary.trim(),
expected_external_revision: publishRevision.trim() || null,
minor: false
}
);
setPublishOpen(false);
setSuccess(result.outcome_unknown
? "Publication outcome is unknown. Reconcile the provider revision before retrying."
: "Provider accepted the page revision and durable recovery evidence was recorded.");
await reload(selected.id);
} catch (caught) {
setError(errorMessage(caught));
} finally {
setBusy(false);
}
};
const visibleProfiles = useMemo(() => {
const needle = search.trim().toLocaleLowerCase();
return profiles.filter((item) => !needle ||
`${item.product} ${item.product_version ?? ""} ${item.health_status} ${item.configuration_id}`
.toLocaleLowerCase()
.includes(needle));
}, [profiles, search]);
const actionBar = <PageActionBar
variant="editor"
state={busy ? "saving" : dirty ? "dirty" : "clean"}
refreshable
reloadAction={{ onReload: () => void reload(selectedId), loading }}
primaryActions={<>
<Button onClick={() => setCreateOpen(true)} disabled={!canAdmin || busy}>New profile</Button>
<Button variant="secondary" onClick={() => void discover()} disabled={!selected || !canAdmin || busy || dirty}>Discover</Button>
<Button variant="secondary" onClick={() => void sync(false)} disabled={!selected || !canSync || busy || dirty}>Run delta</Button>
<Button variant="secondary" onClick={() => void sync(true)} disabled={!selected || !canSync || busy || dirty}>Run full backfill</Button>
<Button variant="secondary" onClick={() => setMigrationOpen(true)} disabled={!selected || !canMigrate || busy || dirty}>Preview migration</Button>
<Button variant="primary" onClick={() => setPublishOpen(true)} disabled={!selected || !canPublish || busy || dirty}>Publish page</Button>
</>}
discardAction={{
label: "Discard changes",
disabled: !selected,
onClick: () => applyProfile(selected)
}}
saveAction={{
label: "Save",
disabled: !selected || !canAdmin || busy,
disabledReason: !canAdmin ? "External knowledge administration permission is required." : undefined,
onClick: () => void save()
}}
/>;
return <AdminPageLayout
archetype="workspace"
title="External knowledge"
description="Discover and synchronize MediaWiki or BlueSpice, preserve stable identity and permissions, and preview migration into native Wiki."
loading={loading && !profiles.length}
error={error}
success={success}
actions={actionBar}
className="connector-knowledge-page"
helpContextId="connectors.admin.external-knowledge"
>
<MetricGrid columns={4} density="compact" minimum="compact">
<MetricCard label="Profiles" value={profiles.length} />
<MetricCard label="Active pages" value={objects.filter((item) => item.status !== "deleted").length} />
<MetricCard label="Unhealthy profiles" value={profiles.filter((item) => !["healthy", "unknown"].includes(item.health_status)).length} tone="warning" />
<MetricCard label="Unresolved runs" value={runs.filter((item) => ["failed", "outcome_unknown"].includes(item.status)).length} tone="warning" />
</MetricGrid>
<WorkspaceLayout
variant="split"
primarySize="compact"
surface="contained"
primaryScrollable={false}
contentScrollable={false}
primaryLabel="Knowledge profiles"
contentLabel="Profile details"
primary={<div className="connector-knowledge-list">
<FilterBar surface="panel">
<input type="search" value={search} onChange={(event) => setSearch(event.target.value)} placeholder="Search profiles" aria-label="Search external knowledge profiles" />
</FilterBar>
<SelectionList variant="navigation" label="External knowledge profiles">
{visibleProfiles.map((profile) => <SelectionListItem key={profile.id} selected={profile.id === selectedId} onClick={() => selectProfile(profile)}>
<SelectionListItemContent
title={`${profile.product}${profile.product_version ? ` ${profile.product_version}` : ""}`}
description={`${profile.discovered_maturity} · ${profile.configuration_id}`}
/>
<StatusBadge status={profile.status === "paused" ? "inactive" : profile.health_status} />
</SelectionListItem>)}
{!visibleProfiles.length ? <StatePanel size="compact" description="No matching knowledge profiles." /> : null}
</SelectionList>
</div>}
>
{!selected ? <StatePanel size="fill" title="External knowledge profiles" description="Create or select a profile to discover provider capabilities and inspect synchronization evidence." /> : <div className="connector-knowledge-detail">
<Card title={`${selected.product}${selected.product_version ? ` ${selected.product_version}` : ""}`}>
<div className="connector-revision-line">
<StatusBadge status={selected.status} />
<StatusBadge status={selected.health_status} />
<span>Discovered maturity: {selected.discovered_maturity}</span>
<code title={selected.discovery_revision ?? undefined}>r{selected.resource_revision}</code>
</div>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Status" hint="Pausing immediately makes Search authorization fail closed.">
<select value={draft.status} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, status: event.target.value as ProfileDraft["status"] })}>
<option value="active">Active</option>
<option value="paused">Paused</option>
</select>
</FormField>
<FormField label="Desired maturity" hint="This is the operator ceiling even when the provider offers more.">
<select value={draft.desired_maturity} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>
{['discover', 'link', 'search', 'read', 'publish', 'synchronize', 'migrate'].map((value) => <option key={value} value={value}>{value}</option>)}
</select>
</FormField>
<FormField label="Source authority">
<select value={draft.source_authority_mode} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, source_authority_mode: event.target.value as ProfileDraft["source_authority_mode"] })}>
<option value="external_authoritative">External authoritative</option>
<option value="external_mirror">External mirror</option>
<option value="governed_sync">Governed sync</option>
<option value="linked_reference">Linked reference</option>
</select>
</FormField>
<FormField label="Fallback visibility">
<select value={draft.default_visibility} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}>
<option value="restricted">Restricted</option>
<option value="tenant">Tenant</option>
</select>
</FormField>
<FormField label="Fallback ACL tokens" hint="One account, membership, identity, group, role, function, or scope token per line.">
<textarea rows={6} value={draft.default_acl_tokens} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, default_acl_tokens: event.target.value })} />
</FormField>
<FormField label="Namespace mappings" hint="JSON array; maps source namespace ids to target Wiki space references and path prefixes.">
<textarea rows={12} value={draft.namespace_mappings} disabled={!canAdmin || busy} onChange={(event) => setDraft({ ...draft, namespace_mappings: event.target.value })} />
</FormField>
</FormGrid>
<p className="muted">Capabilities: {selected.capabilities.length ? selected.capabilities.join(", ") : "run discovery"}</p>
<p className="muted">Last high-watermark: {selected.last_high_watermark ?? "none"} · credential reference: {selected.credential_reference_present ? "configured" : "not configured"}</p>
</Card>
<Card title="Synchronized pages">
<SelectionList variant="static" label="Synchronized external knowledge pages">
{objects.slice(0, 100).map((item) => <SelectionListItem key={item.id}>
<SelectionListItemContent title={item.title} description={`${item.status} · revision ${item.source_revision} · ${item.visibility}`} />
{item.canonical_url ? <a href={item.canonical_url} target="_blank" rel="noreferrer">Open source</a> : null}
</SelectionListItem>)}
{!objects.length ? <StatePanel size="compact" description="No synchronized pages. Run a full backfill after discovery." /> : null}
</SelectionList>
</Card>
<Card title="Synchronization and migration evidence">
<SelectionList variant="static" label="Knowledge connector runs">
{runs.map((run) => <SelectionListItem key={run.id}>
<SelectionListItemContent title={`${run.mode.replaceAll("_", " ")} · ${run.status}`} description={`${formatDateTime(run.started_at)} · ${effectTotal(run)} effects · ${run.diagnostics.length} diagnostics`} />
<StatusBadge status={run.status} />
</SelectionListItem>)}
{!runs.length ? <StatePanel size="compact" description="No knowledge connector runs have been recorded." /> : null}
</SelectionList>
</Card>
{migration ? <Card title="Latest migration preview">
<div className="connector-revision-line">
<StatusBadge status={migration.can_apply ? "success" : "warning"} label={migration.can_apply ? "No blocking conflict" : "Review required"} />
<span>{migration.effects.length} effects</span>
<span>{migration.diagnostics.length} diagnostics</span>
<code title={migration.source_fingerprint}>{migration.source_fingerprint.slice(0, 12)}</code>
</div>
<p className="muted">Preview only: no native Wiki page was written.</p>
<pre className="connector-json-preview">{JSON.stringify({ summary: migration.summary, diagnostics: migration.diagnostics, truncated: migration.truncated }, null, 2)}</pre>
</Card> : null}
</div>}
</WorkspaceLayout>
<Dialog open={createOpen} title="Create external knowledge profile" onClose={() => !busy && setCreateOpen(false)} closeDisabled={busy} footer={<>
<Button onClick={() => setCreateOpen(false)} disabled={busy}>Cancel</Button>
<Button variant="primary" onClick={() => void createProfile()} disabled={busy || !configurationId.trim()}>Create profile</Button>
</>}>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Governed configuration id" hint="Select an active MediaWiki Action API configuration from Connector governance.">
<input value={configurationId} disabled={busy} onChange={(event) => setConfigurationId(event.target.value)} />
</FormField>
<FormField label="Desired maturity">
<select value={newDraft.desired_maturity} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, desired_maturity: event.target.value as ProfileDraft["desired_maturity"] })}>
{['discover', 'link', 'search', 'read', 'publish', 'synchronize', 'migrate'].map((value) => <option key={value} value={value}>{value}</option>)}
</select>
</FormField>
<FormField label="Fallback visibility">
<select value={newDraft.default_visibility} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_visibility: event.target.value as ProfileDraft["default_visibility"] })}>
<option value="restricted">Restricted</option>
<option value="tenant">Tenant</option>
</select>
</FormField>
<FormField label="Fallback ACL tokens">
<textarea rows={5} value={newDraft.default_acl_tokens} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, default_acl_tokens: event.target.value })} />
</FormField>
<FormField label="Namespace mappings">
<textarea rows={12} value={newDraft.namespace_mappings} disabled={busy} onChange={(event) => setNewDraft({ ...newDraft, namespace_mappings: event.target.value })} />
</FormField>
</FormGrid>
</Dialog>
<Dialog open={migrationOpen} title="Preview migration into native Wiki" onClose={() => !busy && setMigrationOpen(false)} closeDisabled={busy} footer={<>
<Button onClick={() => setMigrationOpen(false)} disabled={busy}>Cancel</Button>
<Button variant="primary" onClick={() => void previewMigration()} disabled={busy || !targetSpace.trim()}>Run migration preview</Button>
</>}>
<p className="muted">This dry-run detects path, attachment, macro, and truncation problems. It never writes Wiki pages.</p>
<FormField label="Target Wiki space reference"><input value={targetSpace} disabled={busy} onChange={(event) => setTargetSpace(event.target.value)} /></FormField>
<FormField label="Supported macros" hint="One macro name per line."><textarea rows={5} value={supportedMacros} disabled={busy} onChange={(event) => setSupportedMacros(event.target.value)} /></FormField>
<FormField label="Existing targets" hint="JSON array with path, optional source_external_id, and attachment_names."><textarea rows={10} value={existingTargets} disabled={busy} onChange={(event) => setExistingTargets(event.target.value)} /></FormField>
</Dialog>
<Dialog open={publishOpen} title="Publish provider page revision" onClose={() => !busy && setPublishOpen(false)} closeDisabled={busy} footer={<>
<Button onClick={() => setPublishOpen(false)} disabled={busy}>Cancel</Button>
<Button variant="primary" onClick={() => void publish()} disabled={busy || !externalPageId.trim() || !publishTitle.trim()}>Publish revision</Button>
</>}>
<p className="muted">Publication is an external effect. Supply the current provider revision where possible; an unknown outcome blocks blind retry.</p>
<FormGrid columns={2} collapseAt="standard" className="">
<FormField label="Stable external page id"><input value={externalPageId} disabled={busy} onChange={(event) => setExternalPageId(event.target.value)} /></FormField>
<FormField label="Expected provider revision"><input value={publishRevision} disabled={busy} onChange={(event) => setPublishRevision(event.target.value)} /></FormField>
<FormField label="Title"><input value={publishTitle} disabled={busy} onChange={(event) => setPublishTitle(event.target.value)} /></FormField>
<FormField label="Edit summary"><input value={publishSummary} disabled={busy} onChange={(event) => setPublishSummary(event.target.value)} /></FormField>
<FormField label="Wikitext body"><textarea rows={16} value={publishBody} disabled={busy} onChange={(event) => setPublishBody(event.target.value)} /></FormField>
</FormGrid>
</Dialog>
</AdminPageLayout>;
}
function draftFromProfile(profile: KnowledgeProfile): ProfileDraft {
return {
status: profile.status,
desired_maturity: profile.desired_maturity,
source_authority_mode: profile.source_authority_mode,
default_visibility: profile.default_visibility,
default_acl_tokens: profile.default_acl_tokens.join("\n"),
namespace_mappings: JSON.stringify(profile.namespace_mappings, null, 2)
};
}
function draftKey(draft: ProfileDraft): string {
return JSON.stringify({
...draft,
default_acl_tokens: lines(draft.default_acl_tokens),
namespace_mappings: normalizeJson(draft.namespace_mappings)
});
}
function lines(value: string): string[] {
return [...new Set(value.split(/\r?\n/).map((item) => item.trim()).filter(Boolean))];
}
function normalizeJson(value: string): unknown {
try {
return JSON.parse(value);
} catch {
return value.trim();
}
}
function parseArray(value: string, label: string): unknown[] {
const parsed: unknown = JSON.parse(value);
if (!Array.isArray(parsed)) throw new Error(`${label} must be a JSON array.`);
return parsed;
}
function effectTotal(run: KnowledgeRun): number {
return Object.values(run.counts).reduce((total, value) => total + Number(value || 0), 0);
}
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
+1
View File
@@ -0,0 +1 @@
export { connectorsModule as default, connectorsModule } from "./module";
+89
View File
@@ -0,0 +1,89 @@
import { createElement, lazy } from "react";
import type {
AdminSectionsUiCapability,
PlatformWebModule
} from "@govoplan/core-webui";
import "./styles/connectors.css";
const ConnectorGovernancePage = lazy(
() => import("./features/ConnectorGovernancePage")
);
const ExternalKnowledgePage = lazy(
() => import("./features/ExternalKnowledgePage")
);
const readScopes = [
"connectors:source:read",
"connectors:source:admin"
];
const knowledgeReadScopes = [
"connectors:knowledge:read",
"connectors:knowledge:admin"
];
const adminSections: AdminSectionsUiCapability = {
sections: [
{
id: "connector-governance",
moduleId: "connectors",
kind: "management",
surfaceId: "connectors.admin.governed-configurations",
label: "Connector governance",
group: "SYSTEM",
order: 45,
anyOf: readScopes,
render: ({ settings, auth }) =>
createElement(ConnectorGovernancePage, { settings, auth })
},
{
id: "connector-external-knowledge",
moduleId: "connectors",
kind: "management",
surfaceId: "connectors.admin.external-knowledge",
label: "External knowledge",
group: "SYSTEM",
order: 46,
anyOf: knowledgeReadScopes,
render: ({ settings, auth }) =>
createElement(ExternalKnowledgePage, { settings, auth })
}
]
};
export const connectorsModule: PlatformWebModule = {
id: "connectors",
label: "Connectors",
version: "0.1.21",
dependencies: [],
optionalDependencies: ["access", "audit", "policy", "ops", "search", "wiki"],
viewSurfaces: [
{
id: "connectors.admin.governed-configurations",
moduleId: "connectors",
kind: "section",
label: "Connector governance",
order: 45
},
{
id: "connectors.admin.simulation-review",
moduleId: "connectors",
kind: "section",
label: "Connector simulation review",
parentId: "connectors.admin.governed-configurations",
order: 20
},
{
id: "connectors.admin.external-knowledge",
moduleId: "connectors",
kind: "section",
label: "External knowledge",
parentId: "connectors.admin.governed-configurations",
order: 30
}
],
uiCapabilities: {
"admin.sections": adminSections
}
};
export default connectorsModule;
+40
View File
@@ -0,0 +1,40 @@
.connector-governance-page .connector-governance-list,
.connector-governance-page .connector-governance-detail {
display: grid;
gap: var(--space-4, 1rem);
}
.connector-knowledge-page .connector-knowledge-list,
.connector-knowledge-page .connector-knowledge-detail {
display: grid;
gap: var(--space-4, 1rem);
}
.connector-governance-page .connector-revision-line,
.connector-governance-page .connector-run-actions,
.connector-knowledge-page .connector-revision-line {
align-items: center;
display: flex;
flex-wrap: wrap;
gap: var(--space-3, 0.75rem);
margin-bottom: var(--space-4, 1rem);
}
.connector-governance-page textarea,
.connector-knowledge-page textarea {
font-family: var(--font-family-mono, ui-monospace, monospace);
resize: vertical;
}
.connector-governance-page .connector-definition-editor,
.connector-governance-page .connector-json-preview {
max-height: 34rem;
overflow: auto;
white-space: pre-wrap;
}
.connector-governance-page .connector-json-preview {
background: var(--surface-subtle);
border-radius: var(--radius-md, 0.5rem);
padding: var(--space-4, 1rem);
}
@@ -0,0 +1,40 @@
import assert from "node:assert/strict";
import { readFileSync } from "node:fs";
const moduleSource = readFileSync("src/module.ts", "utf8");
const page = readFileSync("src/features/ConnectorGovernancePage.tsx", "utf8");
const api = readFileSync("src/api/governedConnectors.ts", "utf8");
const knowledgePage = readFileSync("src/features/ExternalKnowledgePage.tsx", "utf8");
const knowledgeApi = readFileSync("src/api/externalKnowledge.ts", "utf8");
assert.match(moduleSource, /"admin.sections": adminSections/);
assert.match(moduleSource, /connectors\.admin\.governed-configurations/);
assert.match(page, /<AdminPageLayout/);
assert.match(page, /<PageActionBar/);
assert.match(page, /refreshable/);
assert.match(page, /saveAction=/);
assert.match(page, /useUnsavedDraftGuard/);
assert.match(page, /<WorkspaceLayout/);
assert.match(page, /Adopt package revision/);
assert.match(page, /Protected paths/);
assert.match(page, /manual_review/);
assert.match(page, /quarantine/);
assert.match(page, /Run simulation/);
assert.match(page, /Review ambiguous connector result/);
assert.match(api, /idempotency_key/);
assert.match(api, /credential_ref/);
assert.match(moduleSource, /connectors\.admin\.external-knowledge/);
assert.match(knowledgePage, /<AdminPageLayout/);
assert.match(knowledgePage, /<PageActionBar/);
assert.match(knowledgePage, /refreshable/);
assert.match(knowledgePage, /saveAction=/);
assert.match(knowledgePage, /useUnsavedDraftGuard/);
assert.match(knowledgePage, /<WorkspaceLayout/);
assert.match(knowledgePage, /Run full backfill/);
assert.match(knowledgePage, /Preview migration/);
assert.match(knowledgePage, /outcome is unknown/);
assert.match(knowledgeApi, /\/knowledge/);
assert.match(knowledgeApi, /migration-dry-runs/);
assert.match(knowledgeApi, /\/publish/);
console.log("Connector governance UI structural contract passed.");