govoplan-connectors/docs/CONNECTOR_SOURCE_LIFECYCLE.md

# Connector Source Lifecycle

GovOPlaN modules should treat external systems as sources with explicit
lifecycle state. A connector profile can consume records from a source, publish
records into a source, or do both. The lifecycle below keeps connectors
predictable and avoids hidden module imports.

## Source Directions

- `consume`: GovOPlaN reads external records, normalizes them, and exposes them
  to modules as external references, events, or staged imports.
- `publish`: GovOPlaN creates or updates external records and stores the external
  identifiers as immutable references.
- `bidirectional`: GovOPlaN supports both directions with conflict detection and
  reconciliation rules.

Direction describes transport. Every binding also needs a source-authority
mode:

- `native_authoritative`
- `external_authoritative`
- `external_mirror`
- `governed_sync`
- `governance_overlay`
- `linked_reference`

The authority mode and the connector's integration maturity are orthogonal. A
bidirectional connector may be configured as an external mirror, and a native
GovOPlaN object may publish to an external target without transferring
authority. The effective binding must identify its scope and provenance rather
than relying on a profile-wide `sync` boolean.

## Source Data Lifecycle

Connector profiles have operational states, while individual external records
or source datasets move through a data lifecycle:

1. `discovered`
   A source, record, file, feed item, webhook event, or remote object is known
   but not yet trusted for domain use.
2. `connected`
   GovOPlaN can authenticate and fetch or publish against the source profile.
3. `imported`
   Minimal source data has been staged with external id, version/ETag, source
   timestamp, and provenance.
4. `validated`
   Shape, permissions, freshness, and required fields passed connector and
   domain validation.
5. `transformed`
   A dataflow, workflow, or domain module normalized the staged payload into a
   domain-specific form.
6. `published`
   GovOPlaN exposed or wrote an output through API, RSS, report, export, or a
   downstream connector.
7. `archived`
   The source/output is no longer active but remains available under retention,
   audit, and external-reference rules.
8. `deprecated`
   The source/output remains readable for history but must not be used for new
   workflows.

Every transition must preserve provenance, permissions context, freshness, and
audit trace. Domain modules may add stricter states, but they should map back to
this lifecycle when a connector publishes status.

## Lifecycle States

1. `draft`
   Profile exists but is not used by runtime jobs.
2. `configured`
   Required endpoint and credential references are present.
3. `tested`
   A health/test run succeeded and recorded non-secret diagnostics.
4. `active`
   Runtime jobs may consume or publish data.
5. `degraded`
   The connector is active but health checks or recent jobs show failures.
6. `paused`
   Operators intentionally stop scheduled connector activity.
7. `retiring`
   The connector is being removed from active workflows while references remain
   readable.
8. `retired`
   No new runtime activity is allowed. Historical references remain available.

## State Transition Gates

| Transition | Required Evidence | Blockers |
| --- | --- | --- |
| `draft` -> `configured` | endpoint fields are valid, credential references exist, owner/tenant scope is set | plaintext secret in profile payload, unsupported connector type |
| `configured` -> `tested` | latest profile test succeeded and diagnostics were redacted | failed auth, unreachable endpoint, TLS/policy error |
| `tested` -> `active` | operator enabled runtime use, required modules/capabilities are present, schedule/webhook is valid | missing module, missing permission, no idempotency strategy for publish jobs |
| `active` -> `degraded` | health check or job telemetry reports failures | none; this is automatic diagnostic state |
| `degraded` -> `active` | health/test succeeds or failed jobs are reconciled | unresolved conflict or repeated failure threshold |
| any running state -> `paused` | operator pause request or maintenance preflight | active critical transaction that cannot be interrupted |
| `paused` -> `active` | successful re-test when credentials/endpoints changed | failed profile test |
| any state -> `retiring` | uninstall/disable plan accepted, schedulers/workers stopped | active domain references that require operator decision |
| `retiring` -> `retired` | non-destructive retirement complete, references remain readable | destructive retirement requested without provider and backup |

## Consume Flow

1. Discover changes through polling, webhook, batch upload, or manual operator
   action.
2. Fetch only the minimal remote data required for the declared use case.
3. Normalize into a connector-owned staging payload.
4. Validate shape, required fields, and source trust level.
5. Publish data-shaped inputs as versioned datasource origins.
6. Let Datasources register live/cached origins or stage immutable snapshots.
7. Let domain modules consume governed datasource references through
   capabilities, not imports.
8. Emit a core-mediated event such as `connector.record_discovered`.
9. Store external references with source system, object type, object id, version
   or ETag, and last-seen timestamp.

## Tabular source discovery and refresh

Immutable JSON/CSV snapshots, exact managed Files versions, and live PostgreSQL
tables share the same catalogue and bounded-preview contract. A managed-file
origin stores only the Files asset id, exact immutable version id, checksum,
parser settings, reviewed schema, and discovery fingerprint. Files re-authorizes
the current principal and verifies storage integrity and any encryption envelope
on every preview. A newer current version produces a warning; only an explicit
source refresh changes the pinned version and increments the discovery revision.

A PostgreSQL origin references an active governed connector configuration. Its
endpoint must contain no credentials. Connectors resolves the referenced Core
credential envelope for the current tenant, scope, module, and server, opens a
read-only connection, reflects a simple schema/table identifier, and records the
configuration hash/revision, credential revision, schema, and discovery
fingerprint. Configuration, credential, or schema drift blocks preview until an
explicit refresh. Missing Files capability, revoked file access, quarantine,
oversized or malformed content, inactive/stale credentials, unreachable SQL,
and timeout failures produce sanitized unavailable/validation diagnostics.

All three current providers declare projection and pagination pushdown only.
Filters, aggregations, and sorting remain in Dataflow until an adapter explicitly
declares and tests those operations.

## Publish Flow

1. Domain module requests publish through a core-mediated connector capability.
2. Connector validates profile state, permission, idempotency key, and payload
   shape.
3. Connector sends the remote request.
4. Connector stores the remote id, version/ETag, and response diagnostics.
5. A timeout or lost acknowledgement after dispatch becomes outcome-unknown,
   not an ordinary failure or permission to duplicate the command.
6. Connector emits a confirmed, retryable, outcome-unknown, reconciled, or
   corrected result event.
7. Domain module stores only the external-reference DTO and any domain result.

## Reconciliation

Every connector that writes to an external system needs a reconciliation story:

- idempotency key for create/update jobs
- remote object version, ETag, or last-modified value where available
- conflict state when local and remote records diverge
- retry policy for temporary failures
- explicit operator action for destructive overwrite or deletion
- audit trace from GovOPlaN record to external request and response summary
- explicit requested, approved, dispatched, possibly-executed, confirmed, and
  reconciled/corrected effect states

## Durable recovery operations

Connectors declares two recovery classes. A read-only acquisition into an
immutable snapshot is `atomic`: the source revision or conditional cursor,
redacted dry-run decision, canonical request digest, and distributed
tenant/provider lease are durable before the fetch. The acquired domain rows
and terminal Core recovery checkpoint commit in one PostgreSQL transaction. A
caller-supplied `Idempotency-Key` replays that committed result without a second
provider request. A failed or stale transaction has no remote mutation and may
be repeated only as a new deliberate acquisition.

An external create, update, publish, or delete is `forward_recovery`. It must
start through the connector mutation recovery contract with a stable
idempotency key, SHA-256 request digest, source revision/cursor, and dry-run
evidence. Definitive rejection is terminal. A timeout or lost acknowledgement
after dispatch is `outcome_unknown` and blocks replay until the owning connector
verifies provider state. The contract and conformance tests exist; no current
connector advertises a production external mutation, so write/delete adoption
remains explicitly planned rather than implied.

## Provider Declaration

An executable connector type should publish machine-readable metadata for:

- owned object and field groups, plus supported authority modes;
- supported discovery, link, search, read, publish, synchronize, migrate, and
  replacement maturity;
- read/write/delete/preview/dry-run operations and bounded response limits;
- revision/concurrency token, freshness, health, timeout, retry, and conflict
  semantics;
- idempotency and outcome-unknown handling;
- evidence, rollback/compensation, correction, and reconciliation paths;
- classification, purpose, retention, secret, degraded, and outage behavior.

This declaration composes Core contracts. It does not move protocol behavior
or domain semantics into Core or Connectors.

## Capability Boundary

Domain modules must not import connector implementation packages directly. They
should ask core for capabilities such as:

- `connectors.catalog`
- `connectors.profileTester`
- `connectors.health`
- `connectors.externalReferences`
- `connectors.datasourceOrigins`
- `connectors.sourceConsumer`
- `connectors.sourcePublisher`

Connector payloads should be DTOs or protocol objects from kernel/core
contracts. Protocol-specific clients stay inside the connector module that owns
them.

## Safety Rules

- Secret values never leave the secret contract and are never stored in test
  result payloads.
- Runtime jobs must include profile id, connector type, direction, idempotency
  key, and triggering principal/system actor.
- Profile tests must redact tokens, passwords, cookies, authorization headers,
  and remote personal data not needed for diagnostics.
- Deactivation must stop schedulers/workers before profile removal.
- Uninstall defaults to non-destructive retirement; domain data and external
  references remain readable.
- Destructive retirement requires a module-owned retirement provider and an
  explicit operator choice.

## Release Checklist

Before shipping an executable connector type:

- Add catalogue metadata and capability names.
- Add profile schema validation that rejects plaintext secrets.
- Add redaction tests for success and failure diagnostics.
- Add unavailable-optional-module tests for every consuming domain module.
- Add profile test and health status fixtures.
- Add external-reference DTO tests.
- Add source-authority and provider-declaration validation tests.
- Add lifecycle transition tests for pause, retry, retirement, and uninstall
  guard behavior.