Metadata-Version: 2.4
Name: govoplan-datasources
Version: 0.1.24
Summary: Governed datasource catalogue, staging, and materialization lifecycle for GovOPlaN.
Author: GovOPlaN
License-Expression: AGPL-3.0-or-later
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: govoplan-core>=0.1.20

# govoplan-datasources

<!-- govoplan-repository-type:start -->
**Repository type:** module (platform).
<!-- govoplan-repository-type:end -->

`govoplan-datasources` is GovOPlaN's provider-neutral data access layer. It owns
the tenant catalogue, lifecycle, staging, immutable materializations, frozen
states, schema and fingerprint metadata, provenance, and bounded reads.

The module deliberately separates two layers:

- **Datasources** describe governed data that modules can consume consistently.
- **Connectors** acquire data from external systems and expose provider origins.

Dataflow, Workflow, Reporting, Risk Compliance, and other consumers depend on
Datasource contracts rather than connector implementations.

The first executable slice supports tabular static uploads, connector-backed
live and cached sources, staging and promotion, refresh, immutable snapshots,
explicit frozen states, previews, retirement, and atomic producer publication.
Origin discovery retains each provider's source mode, structured health, and
declared pushdown support. Live previews preserve the provider's effective row,
serialized-byte, and elapsed-time limits and its redacted diagnostics.
Producer modules can append a bounded inline tabular result or pin a larger
durable artifact through the same idempotent capability. Artifact references
declare a backend, locator, SHA-256 checksum, schema, fingerprint, row and byte
counts; the installed provider verifies integrity and serves bounded reads.
The publication ledger retains the
producer run, output materialization, provenance, and replay identity. On
PostgreSQL, a transaction-scoped advisory lock serializes each tenant, producer,
and idempotency identity before any output side effect, so retries from multiple
application or worker nodes resolve to one durable publication.

Tabular staging also evaluates governed quality rules and classifies schema
changes before promotion. Blocking stages remain inspectable, and successful
promotion preserves the exact policy hash and validation result in immutable
materialization provenance. The supported contract is documented in
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).

The same lifecycle contract can require an attributable, separated approval
quorum before a stage or cached refresh becomes current. Versioned retention
rules cover transient stages, ordinary materializations, and frozen evidence.
Retention remains an explicit preview-and-apply operation: current revisions,
legal holds, pending approvals, and publication evidence are blocked, while
every decision, promotion, and disposition is preserved in hash-chained local
evidence even when the optional Audit, Access, or Policy modules are absent.

The contracts already model database, HTTP/REST, directory, file, feed,
document, binary, directory, and stream sources so providers can be added
without changing consumers. Storage modules contribute artifact backends
through the provider-neutral `datasources.artifactBackends` capability; the
Datasources module never imports their internals.

See [docs/CONCEPT.md](docs/CONCEPT.md) for ownership and lifecycle details.

## Data-subject requests

Datasources publishes `privacy.dsar.datasources` for exact catalogue,
governance-reference, materialization, payload, stage, publication, and
lifecycle-evidence
references and for minimized operator attribution. It never exports connector
references, locators, credentials, arbitrary rows, schemas, validation
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
The module does not guess subject identity by scanning schema-dependent tabular
payloads; the authoritative source module locates and corrects those facts.

Unpromoted stages and unreferenced payloads can be deleted idempotently.
Published or referenced state, immutable materializations, lifecycle evidence,
governance evidence,
holds, and operator attribution require data-steward review. Dataflow and
Reporting derivatives must be refreshed after the source correction.
