160 lines
7.0 KiB
Markdown
160 lines
7.0 KiB
Markdown
# Connector Source Lifecycle
|
|
|
|
GovOPlaN modules should treat external systems as sources with explicit
|
|
lifecycle state. A connector profile can consume records from a source, publish
|
|
records into a source, or do both. The lifecycle below keeps connectors
|
|
predictable and avoids hidden module imports.
|
|
|
|
## Source Directions
|
|
|
|
- `consume`: GovOPlaN reads external records, normalizes them, and exposes them
|
|
to modules as external references, events, or staged imports.
|
|
- `publish`: GovOPlaN creates or updates external records and stores the external
|
|
identifiers as immutable references.
|
|
- `bidirectional`: GovOPlaN supports both directions with conflict detection and
|
|
reconciliation rules.
|
|
|
|
## Source Data Lifecycle
|
|
|
|
Connector profiles have operational states, while individual external records
|
|
or source datasets move through a data lifecycle:
|
|
|
|
1. `discovered`
|
|
A source, record, file, feed item, webhook event, or remote object is known
|
|
but not yet trusted for domain use.
|
|
2. `connected`
|
|
GovOPlaN can authenticate and fetch or publish against the source profile.
|
|
3. `imported`
|
|
Minimal source data has been staged with external id, version/ETag, source
|
|
timestamp, and provenance.
|
|
4. `validated`
|
|
Shape, permissions, freshness, and required fields passed connector and
|
|
domain validation.
|
|
5. `transformed`
|
|
A dataflow, workflow, or domain module normalized the staged payload into a
|
|
domain-specific form.
|
|
6. `published`
|
|
GovOPlaN exposed or wrote an output through API, RSS, report, export, or a
|
|
downstream connector.
|
|
7. `archived`
|
|
The source/output is no longer active but remains available under retention,
|
|
audit, and external-reference rules.
|
|
8. `deprecated`
|
|
The source/output remains readable for history but must not be used for new
|
|
workflows.
|
|
|
|
Every transition must preserve provenance, permissions context, freshness, and
|
|
audit trace. Domain modules may add stricter states, but they should map back to
|
|
this lifecycle when a connector publishes status.
|
|
|
|
## Lifecycle States
|
|
|
|
1. `draft`
|
|
Profile exists but is not used by runtime jobs.
|
|
2. `configured`
|
|
Required endpoint and credential references are present.
|
|
3. `tested`
|
|
A health/test run succeeded and recorded non-secret diagnostics.
|
|
4. `active`
|
|
Runtime jobs may consume or publish data.
|
|
5. `degraded`
|
|
The connector is active but health checks or recent jobs show failures.
|
|
6. `paused`
|
|
Operators intentionally stop scheduled connector activity.
|
|
7. `retiring`
|
|
The connector is being removed from active workflows while references remain
|
|
readable.
|
|
8. `retired`
|
|
No new runtime activity is allowed. Historical references remain available.
|
|
|
|
## State Transition Gates
|
|
|
|
| Transition | Required Evidence | Blockers |
|
|
| --- | --- | --- |
|
|
| `draft` -> `configured` | endpoint fields are valid, credential references exist, owner/tenant scope is set | plaintext secret in profile payload, unsupported connector type |
|
|
| `configured` -> `tested` | latest profile test succeeded and diagnostics were redacted | failed auth, unreachable endpoint, TLS/policy error |
|
|
| `tested` -> `active` | operator enabled runtime use, required modules/capabilities are present, schedule/webhook is valid | missing module, missing permission, no idempotency strategy for publish jobs |
|
|
| `active` -> `degraded` | health check or job telemetry reports failures | none; this is automatic diagnostic state |
|
|
| `degraded` -> `active` | health/test succeeds or failed jobs are reconciled | unresolved conflict or repeated failure threshold |
|
|
| any running state -> `paused` | operator pause request or maintenance preflight | active critical transaction that cannot be interrupted |
|
|
| `paused` -> `active` | successful re-test when credentials/endpoints changed | failed profile test |
|
|
| any state -> `retiring` | uninstall/disable plan accepted, schedulers/workers stopped | active domain references that require operator decision |
|
|
| `retiring` -> `retired` | non-destructive retirement complete, references remain readable | destructive retirement requested without provider and backup |
|
|
|
|
## Consume Flow
|
|
|
|
1. Discover changes through polling, webhook, batch upload, or manual operator
|
|
action.
|
|
2. Fetch only the minimal remote data required for the declared use case.
|
|
3. Normalize into a connector-owned staging payload.
|
|
4. Validate shape, required fields, and source trust level.
|
|
5. Emit a core-mediated event such as `connector.record_discovered`.
|
|
6. Let domain modules claim or transform staged data through capabilities, not
|
|
imports.
|
|
7. Store external references with source system, object type, object id, version
|
|
or ETag, and last-seen timestamp.
|
|
|
|
## Publish Flow
|
|
|
|
1. Domain module requests publish through a core-mediated connector capability.
|
|
2. Connector validates profile state, permission, idempotency key, and payload
|
|
shape.
|
|
3. Connector sends the remote request.
|
|
4. Connector stores the remote id, version/ETag, and response diagnostics.
|
|
5. Connector emits `connector.record_published` or `connector.publish_failed`.
|
|
6. Domain module stores only the external-reference DTO and any domain result.
|
|
|
|
## Reconciliation
|
|
|
|
Every connector that writes to an external system needs a reconciliation story:
|
|
|
|
- idempotency key for create/update jobs
|
|
- remote object version, ETag, or last-modified value where available
|
|
- conflict state when local and remote records diverge
|
|
- retry policy for temporary failures
|
|
- explicit operator action for destructive overwrite or deletion
|
|
- audit trace from GovOPlaN record to external request and response summary
|
|
|
|
## Capability Boundary
|
|
|
|
Domain modules must not import connector implementation packages directly. They
|
|
should ask core for capabilities such as:
|
|
|
|
- `connectors.catalog`
|
|
- `connectors.profileTester`
|
|
- `connectors.health`
|
|
- `connectors.externalReferences`
|
|
- `connectors.sourceConsumer`
|
|
- `connectors.sourcePublisher`
|
|
|
|
Connector payloads should be DTOs or protocol objects from kernel/core
|
|
contracts. Protocol-specific clients stay inside the connector module that owns
|
|
them.
|
|
|
|
## Safety Rules
|
|
|
|
- Secret values never leave the secret contract and are never stored in test
|
|
result payloads.
|
|
- Runtime jobs must include profile id, connector type, direction, idempotency
|
|
key, and triggering principal/system actor.
|
|
- Profile tests must redact tokens, passwords, cookies, authorization headers,
|
|
and remote personal data not needed for diagnostics.
|
|
- Deactivation must stop schedulers/workers before profile removal.
|
|
- Uninstall defaults to non-destructive retirement; domain data and external
|
|
references remain readable.
|
|
- Destructive retirement requires a module-owned retirement provider and an
|
|
explicit operator choice.
|
|
|
|
## Release Checklist
|
|
|
|
Before shipping an executable connector type:
|
|
|
|
- Add catalogue metadata and capability names.
|
|
- Add profile schema validation that rejects plaintext secrets.
|
|
- Add redaction tests for success and failure diagnostics.
|
|
- Add unavailable-optional-module tests for every consuming domain module.
|
|
- Add profile test and health status fixtures.
|
|
- Add external-reference DTO tests.
|
|
- Add lifecycle transition tests for pause, retry, retirement, and uninstall
|
|
guard behavior.
|