govoplan-connectors/docs/CONNECTOR_SOURCE_LIFECYCLE.md

Connector Source Lifecycle

GovOPlaN modules should treat external systems as sources with explicit lifecycle state. A connector profile can consume records from a source, publish records into a source, or do both. The lifecycle below keeps connectors predictable and avoids hidden module imports.

Source Directions

• consume: GovOPlaN reads external records, normalizes them, and exposes them to modules as external references, events, or staged imports.
• publish: GovOPlaN creates or updates external records and stores the external identifiers as immutable references.
• bidirectional: GovOPlaN supports both directions with conflict detection and reconciliation rules.

Source Data Lifecycle

Connector profiles have operational states, while individual external records or source datasets move through a data lifecycle:

1. discovered A source, record, file, feed item, webhook event, or remote object is known but not yet trusted for domain use.
2. connected GovOPlaN can authenticate and fetch or publish against the source profile.
3. imported Minimal source data has been staged with external id, version/ETag, source timestamp, and provenance.
4. validated Shape, permissions, freshness, and required fields passed connector and domain validation.
5. transformed A dataflow, workflow, or domain module normalized the staged payload into a domain-specific form.
6. published GovOPlaN exposed or wrote an output through API, RSS, report, export, or a downstream connector.
7. archived The source/output is no longer active but remains available under retention, audit, and external-reference rules.
8. deprecated The source/output remains readable for history but must not be used for new workflows.

Every transition must preserve provenance, permissions context, freshness, and audit trace. Domain modules may add stricter states, but they should map back to this lifecycle when a connector publishes status.

Lifecycle States

1. draft Profile exists but is not used by runtime jobs.
2. configured Required endpoint and credential references are present.
3. tested A health/test run succeeded and recorded non-secret diagnostics.
4. active Runtime jobs may consume or publish data.
5. degraded The connector is active but health checks or recent jobs show failures.
6. paused Operators intentionally stop scheduled connector activity.
7. retiring The connector is being removed from active workflows while references remain readable.
8. retired No new runtime activity is allowed. Historical references remain available.

State Transition Gates

Transition	Required Evidence	Blockers
draft -> configured	endpoint fields are valid, credential references exist, owner/tenant scope is set	plaintext secret in profile payload, unsupported connector type
configured -> tested	latest profile test succeeded and diagnostics were redacted	failed auth, unreachable endpoint, TLS/policy error
tested -> active	operator enabled runtime use, required modules/capabilities are present, schedule/webhook is valid	missing module, missing permission, no idempotency strategy for publish jobs
active -> degraded	health check or job telemetry reports failures	none; this is automatic diagnostic state
degraded -> active	health/test succeeds or failed jobs are reconciled	unresolved conflict or repeated failure threshold
any running state -> paused	operator pause request or maintenance preflight	active critical transaction that cannot be interrupted
paused -> active	successful re-test when credentials/endpoints changed	failed profile test
any state -> retiring	uninstall/disable plan accepted, schedulers/workers stopped	active domain references that require operator decision
retiring -> retired	non-destructive retirement complete, references remain readable	destructive retirement requested without provider and backup

Consume Flow

1. Discover changes through polling, webhook, batch upload, or manual operator action.
2. Fetch only the minimal remote data required for the declared use case.
3. Normalize into a connector-owned staging payload.
4. Validate shape, required fields, and source trust level.
5. Emit a core-mediated event such as connector.record_discovered.
6. Let domain modules claim or transform staged data through capabilities, not imports.
7. Store external references with source system, object type, object id, version or ETag, and last-seen timestamp.

Publish Flow

1. Domain module requests publish through a core-mediated connector capability.
2. Connector validates profile state, permission, idempotency key, and payload shape.
3. Connector sends the remote request.
4. Connector stores the remote id, version/ETag, and response diagnostics.
5. Connector emits connector.record_published or connector.publish_failed.
6. Domain module stores only the external-reference DTO and any domain result.

Reconciliation

Every connector that writes to an external system needs a reconciliation story:

• idempotency key for create/update jobs
• remote object version, ETag, or last-modified value where available
• conflict state when local and remote records diverge
• retry policy for temporary failures
• explicit operator action for destructive overwrite or deletion
• audit trace from GovOPlaN record to external request and response summary

Capability Boundary

Domain modules must not import connector implementation packages directly. They should ask core for capabilities such as:

• connectors.catalog
• connectors.profileTester
• connectors.health
• connectors.externalReferences
• connectors.sourceConsumer
• connectors.sourcePublisher

Connector payloads should be DTOs or protocol objects from kernel/core contracts. Protocol-specific clients stay inside the connector module that owns them.

Safety Rules

• Secret values never leave the secret contract and are never stored in test result payloads.
• Runtime jobs must include profile id, connector type, direction, idempotency key, and triggering principal/system actor.
• Profile tests must redact tokens, passwords, cookies, authorization headers, and remote personal data not needed for diagnostics.
• Deactivation must stop schedulers/workers before profile removal.
• Uninstall defaults to non-destructive retirement; domain data and external references remain readable.
• Destructive retirement requires a module-owned retirement provider and an explicit operator choice.

Release Checklist

Before shipping an executable connector type:

• Add catalogue metadata and capability names.
• Add profile schema validation that rejects plaintext secrets.
• Add redaction tests for success and failure diagnostics.
• Add unavailable-optional-module tests for every consuming domain module.
• Add profile test and health status fixtures.
• Add external-reference DTO tests.
• Add lifecycle transition tests for pause, retry, retirement, and uninstall guard behavior.