govoplan-connectors/docs/CONNECTOR_SOURCE_LIFECYCLE.md

Connector Source Lifecycle

GovOPlaN modules should treat external systems as sources with explicit lifecycle state. A connector profile can consume records from a source, publish records into a source, or do both. The lifecycle below keeps connectors predictable and avoids hidden module imports.

Source Directions

• consume: GovOPlaN reads external records, normalizes them, and exposes them to modules as external references, events, or staged imports.
• publish: GovOPlaN creates or updates external records and stores the external identifiers as immutable references.
• bidirectional: GovOPlaN supports both directions with conflict detection and reconciliation rules.

Direction describes transport. Every binding also needs a source-authority mode:

• native_authoritative
• external_authoritative
• external_mirror
• governed_sync
• governance_overlay
• linked_reference

The authority mode and the connector's integration maturity are orthogonal. A bidirectional connector may be configured as an external mirror, and a native GovOPlaN object may publish to an external target without transferring authority. The effective binding must identify its scope and provenance rather than relying on a profile-wide sync boolean.

Source Data Lifecycle

Connector profiles have operational states, while individual external records or source datasets move through a data lifecycle:

1. discovered A source, record, file, feed item, webhook event, or remote object is known but not yet trusted for domain use.
2. connected GovOPlaN can authenticate and fetch or publish against the source profile.
3. imported Minimal source data has been staged with external id, version/ETag, source timestamp, and provenance.
4. validated Shape, permissions, freshness, and required fields passed connector and domain validation.
5. transformed A dataflow, workflow, or domain module normalized the staged payload into a domain-specific form.
6. published GovOPlaN exposed or wrote an output through API, RSS, report, export, or a downstream connector.
7. archived The source/output is no longer active but remains available under retention, audit, and external-reference rules.
8. deprecated The source/output remains readable for history but must not be used for new workflows.

Every transition must preserve provenance, permissions context, freshness, and audit trace. Domain modules may add stricter states, but they should map back to this lifecycle when a connector publishes status.

Lifecycle States

1. draft Profile exists but is not used by runtime jobs.
2. configured Required endpoint and credential references are present.
3. tested A health/test run succeeded and recorded non-secret diagnostics.
4. active Runtime jobs may consume or publish data.
5. degraded The connector is active but health checks or recent jobs show failures.
6. paused Operators intentionally stop scheduled connector activity.
7. retiring The connector is being removed from active workflows while references remain readable.
8. retired No new runtime activity is allowed. Historical references remain available.

State Transition Gates

Transition	Required Evidence	Blockers
draft -> configured	endpoint fields are valid, credential references exist, owner/tenant scope is set	plaintext secret in profile payload, unsupported connector type
configured -> tested	latest profile test succeeded and diagnostics were redacted	failed auth, unreachable endpoint, TLS/policy error
tested -> active	operator enabled runtime use, required modules/capabilities are present, schedule/webhook is valid	missing module, missing permission, no idempotency strategy for publish jobs
active -> degraded	health check or job telemetry reports failures	none; this is automatic diagnostic state
degraded -> active	health/test succeeds or failed jobs are reconciled	unresolved conflict or repeated failure threshold
any running state -> paused	operator pause request or maintenance preflight	active critical transaction that cannot be interrupted
paused -> active	successful re-test when credentials/endpoints changed	failed profile test
any state -> retiring	uninstall/disable plan accepted, schedulers/workers stopped	active domain references that require operator decision
retiring -> retired	non-destructive retirement complete, references remain readable	destructive retirement requested without provider and backup

Consume Flow

1. Discover changes through polling, webhook, batch upload, or manual operator action.
2. Fetch only the minimal remote data required for the declared use case.
3. Normalize into a connector-owned staging payload.
4. Validate shape, required fields, and source trust level.
5. Publish data-shaped inputs as versioned datasource origins.
6. Let Datasources register live/cached origins or stage immutable snapshots.
7. Let domain modules consume governed datasource references through capabilities, not imports.
8. Emit a core-mediated event such as connector.record_discovered.
9. Store external references with source system, object type, object id, version or ETag, and last-seen timestamp.

Tabular source discovery and refresh

Immutable JSON/CSV snapshots, exact managed Files versions, and live PostgreSQL tables share the same catalogue and bounded-preview contract. A managed-file origin stores only the Files asset id, exact immutable version id, checksum, parser settings, reviewed schema, and discovery fingerprint. Files re-authorizes the current principal and verifies storage integrity and any encryption envelope on every preview. A newer current version produces a warning; only an explicit source refresh changes the pinned version and increments the discovery revision.

A PostgreSQL origin references an active governed connector configuration. Its endpoint must contain no credentials. Connectors resolves the referenced Core credential envelope for the current tenant, scope, module, and server, opens a read-only connection, reflects a simple schema/table identifier, and records the configuration hash/revision, credential revision, schema, and discovery fingerprint. Configuration, credential, or schema drift blocks preview until an explicit refresh. Missing Files capability, revoked file access, quarantine, oversized or malformed content, inactive/stale credentials, unreachable SQL, and timeout failures produce sanitized unavailable/validation diagnostics.

XLSX archive checks and workbook parsing run in a fresh disposable Core worker. Each invocation allows 15 seconds wall time, 10 CPU seconds and 512 MiB virtual address space, with no file output. The typed transport allows 8 MiB input and 64 MiB result bytes, at most 64 nesting levels and 1,000,000 value nodes. These transport bounds include serialization overhead. The existing workbook bounds remain 5,000,000 raw bytes, 50,000,000 expanded bytes, 5,000 archive entries, 100:1 compression ratio, 500 columns and 10,000 row positions after the header. Authorization and exact-version file reads happen in the parent; credentials, SQL sessions and durable changes are never passed to the parser.

Users receive an explicit failure rather than a partial source when resource or transport limits are exceeded; reduce workbook size or complexity before retrying. The Core GOVOPLAN_ISOLATED_PROCESS_CONCURRENCY setting limits active work per API/worker process without queuing. Busy capacity may be retried later. Missing POSIX resource controls, cancellation and worker failure produce sanitized unavailable diagnostics; there is no in-process fallback. Operators must keep the Core worker API available and account for the aggregate memory of all active worker slots across API/worker replicas.

All three current providers declare projection and pagination pushdown only. Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly declares and tests those operations.

Publish Flow

1. Domain module requests publish through a core-mediated connector capability.
2. Connector validates profile state, permission, idempotency key, and payload shape.
3. Connector sends the remote request.
4. Connector stores the remote id, version/ETag, and response diagnostics.
5. A timeout or lost acknowledgement after dispatch becomes outcome-unknown, not an ordinary failure or permission to duplicate the command.
6. Connector emits a confirmed, retryable, outcome-unknown, reconciled, or corrected result event.
7. Domain module stores only the external-reference DTO and any domain result.

Reconciliation

Every connector that writes to an external system needs a reconciliation story:

• idempotency key for create/update jobs
• remote object version, ETag, or last-modified value where available
• conflict state when local and remote records diverge
• retry policy for temporary failures
• explicit operator action for destructive overwrite or deletion
• audit trace from GovOPlaN record to external request and response summary
• explicit requested, approved, dispatched, possibly-executed, confirmed, and reconciled/corrected effect states

Durable recovery operations

Connectors declares two recovery classes. A read-only acquisition into an immutable snapshot is atomic: the source revision or conditional cursor, redacted dry-run decision, canonical request digest, and distributed tenant/provider lease are durable before the fetch. The acquired domain rows and terminal Core recovery checkpoint commit in one PostgreSQL transaction. A caller-supplied Idempotency-Key replays that committed result without a second provider request. A failed or stale transaction has no remote mutation and may be repeated only as a new deliberate acquisition.

An external create, update, publish, or delete is forward_recovery. It must start through the connector mutation recovery contract with a stable idempotency key, SHA-256 request digest, source revision/cursor, and dry-run evidence. Definitive rejection is terminal. A timeout or lost acknowledgement after dispatch is outcome_unknown and blocks replay until the owning connector verifies provider state. The MediaWiki/BlueSpice page publisher implements this path for revision-checked edits. Other connector types do not thereby acquire a write capability; each adapter must declare and prove its own recovery and reconciliation behavior.

Provider Declaration

An executable connector type should publish machine-readable metadata for:

• owned object and field groups, plus supported authority modes;
• supported discovery, link, search, read, publish, synchronize, migrate, and replacement maturity;
• read/write/delete/preview/dry-run operations and bounded response limits;
• revision/concurrency token, freshness, health, timeout, retry, and conflict semantics;
• idempotency and outcome-unknown handling;
• evidence, rollback/compensation, correction, and reconciliation paths;
• classification, purpose, retention, secret, degraded, and outage behavior.

This declaration composes Core contracts. It does not move protocol behavior or domain semantics into Core or Connectors.

Capability Boundary

Domain modules must not import connector implementation packages directly. They should ask core for capabilities such as:

• connectors.catalog
• connectors.profileTester
• connectors.health
• connectors.externalReferences
• connectors.datasourceOrigins
• connectors.sourceConsumer
• connectors.sourcePublisher

Connector payloads should be DTOs or protocol objects from kernel/core contracts. Protocol-specific clients stay inside the connector module that owns them.

Safety Rules

• Secret values never leave the secret contract and are never stored in test result payloads.
• Runtime jobs must include profile id, connector type, direction, idempotency key, and triggering principal/system actor.
• Profile tests must redact tokens, passwords, cookies, authorization headers, and remote personal data not needed for diagnostics.
• Deactivation must stop schedulers/workers before profile removal.
• Uninstall defaults to non-destructive retirement; domain data and external references remain readable.
• Destructive retirement requires a module-owned retirement provider and an explicit operator choice.

Release Checklist

Before shipping an executable connector type:

• Add catalogue metadata and capability names.
• Add profile schema validation that rejects plaintext secrets.
• Add redaction tests for success and failure diagnostics.
• Add unavailable-optional-module tests for every consuming domain module.
• Add profile test and health status fixtures.
• Add external-reference DTO tests.
• Add source-authority and provider-declaration validation tests.
• Add lifecycle transition tests for pause, retry, retirement, and uninstall guard behavior.