91 lines
5.0 KiB
Markdown
91 lines
5.0 KiB
Markdown
# govoplan-datasources
|
|
|
|
<!-- govoplan-repository-type:start -->
|
|
**Repository type:** module (platform).
|
|
<!-- govoplan-repository-type:end -->
|
|
|
|
`govoplan-datasources` is GovOPlaN's provider-neutral data access layer. It owns
|
|
the tenant catalogue, lifecycle, staging, immutable materializations, frozen
|
|
states, schema and fingerprint metadata, provenance, and bounded reads.
|
|
|
|
The module deliberately separates two layers:
|
|
|
|
- **Datasources** describe governed data that modules can consume consistently.
|
|
- **Connectors** acquire data from external systems and expose provider origins.
|
|
|
|
Dataflow, Workflow, Reporting, Risk Compliance, and other consumers depend on
|
|
Datasource contracts rather than connector implementations.
|
|
|
|
The first executable slice supports tabular static uploads, connector-backed
|
|
live and cached sources, staging and promotion, refresh, immutable snapshots,
|
|
explicit frozen states, previews, retirement, and atomic producer publication.
|
|
Origin discovery retains each provider's source mode, structured health, and
|
|
declared pushdown support. Live previews preserve the provider's effective row,
|
|
serialized-byte, and elapsed-time limits and its redacted diagnostics.
|
|
Producer modules can append a bounded inline tabular result or pin a larger
|
|
durable artifact through the same idempotent capability. Artifact references
|
|
declare a backend, locator, SHA-256 checksum, schema, fingerprint, row and byte
|
|
counts; the installed provider verifies integrity and serves bounded reads.
|
|
The publication ledger retains the
|
|
producer run, output materialization, provenance, and replay identity. On
|
|
PostgreSQL, a transaction-scoped advisory lock serializes each tenant, producer,
|
|
and idempotency identity before any output side effect, so retries from multiple
|
|
application or worker nodes resolve to one durable publication.
|
|
|
|
Tabular staging also evaluates governed quality rules and classifies schema
|
|
changes before promotion. Blocking stages remain inspectable, and successful
|
|
promotion preserves the exact policy hash and validation result in immutable
|
|
materialization provenance. The supported contract is documented in
|
|
[docs/QUALITY_POLICY.md](docs/QUALITY_POLICY.md).
|
|
|
|
The same lifecycle contract can require an attributable, separated approval
|
|
quorum before a stage or cached refresh becomes current. Versioned retention
|
|
rules cover transient stages, ordinary materializations, and frozen evidence.
|
|
Retention remains an explicit preview-and-apply operation: current revisions,
|
|
legal holds, pending approvals, and publication evidence are blocked, while
|
|
every decision, promotion, and disposition is preserved in hash-chained local
|
|
evidence even when the optional Audit, Access, or Policy modules are absent.
|
|
|
|
The contracts already model database, HTTP/REST, directory, file, feed,
|
|
document, binary, directory, and stream sources so providers can be added
|
|
without changing consumers. Storage modules contribute artifact backends
|
|
through the provider-neutral `datasources.artifactBackends` capability; the
|
|
Datasources module never imports their internals.
|
|
|
|
See [docs/CONCEPT.md](docs/CONCEPT.md) for ownership and lifecycle details.
|
|
|
|
## Data-subject requests
|
|
|
|
Datasources publishes `privacy.dsar.datasources` for exact catalogue,
|
|
governance-reference, materialization, payload, stage, publication, and
|
|
lifecycle-evidence
|
|
references and for minimized operator attribution. It never exports connector
|
|
references, locators, credentials, arbitrary rows, schemas, validation
|
|
samples, metadata, provenance bodies, checkpoints, replay material, or hashes.
|
|
The module does not guess subject identity by scanning schema-dependent tabular
|
|
payloads; the authoritative source module locates and corrects those facts.
|
|
|
|
Unpromoted stages and unreferenced payloads can be deleted idempotently.
|
|
Published or referenced state, immutable materializations, lifecycle evidence,
|
|
governance evidence,
|
|
holds, and operator attribution require data-steward review. Dataflow and
|
|
Reporting derivatives must be refreshed after the source correction.
|
|
|
|
## Git-source WebUI package
|
|
|
|
The repository root exposes `@govoplan/datasources-webui` for Git-tagged release
|
|
dependencies. It mirrors the owning `webui/package.json` version, public
|
|
TypeScript/CSS exports and peer requirements, with entry paths under
|
|
`webui/src`. Consumers provide the shared Core/React peers; the facade runs no
|
|
development or install scripts. The source archive contains `webui/src`, this
|
|
README and any repository license file. Run module development checks from `webui/`; Python
|
|
installation remains governed by `pyproject.toml`.
|
|
|
|
Das Repository stellt `@govoplan/datasources-webui` am Wurzelpfad für versionierte
|
|
Git-Abhängigkeiten bereit. Version, öffentliche TypeScript-/CSS-Exporte und
|
|
Peer-Anforderungen entsprechen `webui/package.json`; die Einstiegspfade liegen
|
|
unter `webui/src`. Gemeinsame Core-/React-Peers stellt die einbindende Anwendung
|
|
bereit. Die Fassade führt keine Entwicklungs- oder Installationsskripte aus.
|
|
Entwicklungsprüfungen bleiben in `webui/`, die Python-Installation weiterhin in
|
|
`pyproject.toml` definiert.
|