3.4 KiB
govoplan-datasources
Repository type: module (platform).
govoplan-datasources is GovOPlaN's provider-neutral data access layer. It owns
the tenant catalogue, lifecycle, staging, immutable materializations, frozen
states, schema and fingerprint metadata, provenance, and bounded reads.
The module deliberately separates two layers:
- Datasources describe governed data that modules can consume consistently.
- Connectors acquire data from external systems and expose provider origins.
Dataflow, Workflow, Reporting, Risk Compliance, and other consumers depend on Datasource contracts rather than connector implementations.
The first executable slice supports tabular static uploads, connector-backed live and cached sources, staging and promotion, refresh, immutable snapshots, explicit frozen states, previews, retirement, and atomic producer publication. Origin discovery retains each provider's source mode, structured health, and declared pushdown support. Live previews preserve the provider's effective row, serialized-byte, and elapsed-time limits and its redacted diagnostics. Producer modules can append a bounded inline tabular result or pin a larger durable artifact through the same idempotent capability. Artifact references declare a backend, locator, SHA-256 checksum, schema, fingerprint, row and byte counts; the installed provider verifies integrity and serves bounded reads. The publication ledger retains the producer run, output materialization, provenance, and replay identity. On PostgreSQL, a transaction-scoped advisory lock serializes each tenant, producer, and idempotency identity before any output side effect, so retries from multiple application or worker nodes resolve to one durable publication.
Tabular staging also evaluates governed quality rules and classifies schema changes before promotion. Blocking stages remain inspectable, and successful promotion preserves the exact policy hash and validation result in immutable materialization provenance. The supported contract is documented in docs/QUALITY_POLICY.md.
The contracts already model database, HTTP/REST, directory, file, feed,
document, binary, directory, and stream sources so providers can be added
without changing consumers. Storage modules contribute artifact backends
through the provider-neutral datasources.artifactBackends capability; the
Datasources module never imports their internals.
See docs/CONCEPT.md for ownership and lifecycle details.
Data-subject requests
Datasources publishes privacy.dsar.datasources for exact catalogue,
governance-reference, materialization, payload, stage, and publication
references and for minimized operator attribution. It never exports connector
references, locators, credentials, arbitrary rows, schemas, validation
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
The module does not guess subject identity by scanning schema-dependent tabular
payloads; the authoritative source module locates and corrects those facts.
Unpromoted stages and unreferenced payloads can be deleted idempotently. Published or referenced state, immutable materializations, governance evidence, holds, and operator attribution require data-steward review. Dataflow and Reporting derivatives must be refreshed after the source correction.