diff --git a/Codex-Project-Index.md b/Codex-Project-Index.md index 774178a..3f2d20a 100644 --- a/Codex-Project-Index.md +++ b/Codex-Project-Index.md @@ -15,6 +15,7 @@ This page is generated from repository and product-directory project files. - [Repo-docs-DEPLOYMENT-OPERATOR-GUIDE](Repo-docs-DEPLOYMENT-OPERATOR-GUIDE) - `/mnt/DATA/git/govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md` - [Repo-docs-DOCUMENTATION-MAP](Repo-docs-DOCUMENTATION-MAP) - `/mnt/DATA/git/govoplan-core/docs/DOCUMENTATION_MAP.md` - [Repo-docs-EVENTS-AND-AUDIT](Repo-docs-EVENTS-AND-AUDIT) - `/mnt/DATA/git/govoplan-core/docs/EVENTS_AND_AUDIT.md` +- [Repo-docs-EXTERNAL-REFERENCES-AND-INTEGRATION-MATURITY](Repo-docs-EXTERNAL-REFERENCES-AND-INTEGRATION-MATURITY) - `/mnt/DATA/git/govoplan-core/docs/EXTERNAL_REFERENCES_AND_INTEGRATION_MATURITY.md` - [Repo-docs-GITEA-ISSUES](Repo-docs-GITEA-ISSUES) - `/mnt/DATA/git/govoplan-core/docs/GITEA_ISSUES.md` - [Repo-docs-GOVERNANCE-MODEL](Repo-docs-GOVERNANCE-MODEL) - `/mnt/DATA/git/govoplan-core/docs/GOVERNANCE_MODEL.md` - [Repo-docs-GOVOPLAN-MASTER-ROADMAP](Repo-docs-GOVOPLAN-MASTER-ROADMAP) - `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MASTER_ROADMAP.md` diff --git a/Repo-README.md b/Repo-README.md index 3192147..b24f2fa 100644 --- a/Repo-README.md +++ b/Repo-README.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/README.md`. > Origin: `repository`. @@ -54,7 +54,7 @@ python3 -m venv .venv ./.venv/bin/python -m pip install -r requirements-dev.txt ``` -Run the platform server from core through the module-aware development runner. The default config reads `ENABLED_MODULES` and discovers installed module entry points. Local development defaults to `tenancy,organizations,identity,access,admin,dashboard,policy,audit,campaigns,files,mail,calendar,docs,ops`; set `ENABLED_MODULES` explicitly when testing a smaller module permutation. +Run the platform server from core through the module-aware development runner. The default config reads `ENABLED_MODULES` and discovers installed module entry points. Local development defaults to the modules listed by `govoplan_core.settings.Settings.enabled_modules`, including Views when its package is installed; set `ENABLED_MODULES` explicitly when testing a smaller module permutation. ```bash cd /mnt/DATA/git/govoplan-core diff --git a/Repo-docs-AUTOMATION-CONTRACTS.md b/Repo-docs-AUTOMATION-CONTRACTS.md index 97aa314..7f5a1d5 100644 --- a/Repo-docs-AUTOMATION-CONTRACTS.md +++ b/Repo-docs-AUTOMATION-CONTRACTS.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/AUTOMATION_CONTRACTS.md`. > Origin: `repository`. @@ -49,8 +49,18 @@ inheritance, reuse, derivation, and automation are unavailable. ## Delivery Durability Domain trigger implementations persist idempotent deliveries before running. -The existing `PlatformEvent` bus is process-local and is not a durable -automation source. A transactional Core event/outbox bridge is still required -to guarantee capture across commits, restarts, and multiple workers. Until -that bridge exists, direct event ingestion must be authenticated, tenant -scoped, and limited to public or internal event envelopes. +`emit_platform_event` binds event delivery to the producer's SQLAlchemy +transaction. When an enabled module provides `platform.eventOutbox`, the event +is stored in that transaction and a dispatcher may retry it across restarts and +workers. The Audit module provides the current SQL outbox implementation; the +Celery `govoplan.events.dispatch_outbox` task drains it through the Dataflow +event-ingestion capability and the local event bus. + +The outbox capability remains optional so reduced module combinations can +start. Without it, Core queues events on the SQLAlchemy transaction and +publishes them to the process-local bus only after the outer commit. A rollback, +including a nested savepoint rollback, discards the corresponding events. This +fallback is suitable for local or non-critical reactions, but it is not a +durable multi-worker automation source. Deployments that rely on event-triggered +work must enable the outbox provider and run the `events` worker queue and +periodic dispatcher. diff --git a/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md b/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md index 2b9ea21..5f9740c 100644 --- a/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md +++ b/Repo-docs-DEPLOYMENT-OPERATOR-GUIDE.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/DEPLOYMENT_OPERATOR_GUIDE.md`. > Origin: `repository`. @@ -216,9 +216,13 @@ prefer `FILE_STORAGE_*`. Interactive password login is enabled with fixed-window limits of 10 failures per normalized identity and 100 failures per direct client over 900 seconds. `AUTH_LOGIN_THROTTLE_*` settings change those limits. Counters use `REDIS_URL` -when Redis is reachable so replicas share state; a bounded process-local -fallback keeps development and Redis outages functional, with per-process -enforcement until Redis recovers. +when Redis is reachable so replicas share state. Production-like startup fails +when throttling is enabled without `REDIS_URL`. Set +`GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE=true` only as an explicit +single-process risk acceptance. A bounded process-local fallback keeps +development and temporary Redis outages functional, with per-process +enforcement until Redis recovers; monitor Redis because protection is weaker +during that fallback. ### Outbound Connector Egress diff --git a/Repo-docs-EVENTS-AND-AUDIT.md b/Repo-docs-EVENTS-AND-AUDIT.md index b1ce7e4..9779870 100644 --- a/Repo-docs-EVENTS-AND-AUDIT.md +++ b/Repo-docs-EVENTS-AND-AUDIT.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/EVENTS_AND_AUDIT.md`. > Origin: `repository`. @@ -15,25 +15,32 @@ module reactions, and operator diagnostics. ## Production Transport Decision -The first production target is a **database outbox plus in-process immediate -dispatch**: +The production transport is a **transactional database outbox plus retrying +dispatcher**: - Use `govoplan_core.core.events.PlatformEvent` for domain and platform events. -- Use `EventBus` as the in-process dispatch contract for same-process module - reactions that are safe to run inline. +- Call `emit_platform_event(session, event)` to bind event delivery to the + domain transaction. +- The optional `platform.eventOutbox` capability persists events atomically. + The Audit module provides the current SQL implementation. +- Without an outbox provider, Core publishes to `EventBus` only after the outer + transaction commits. This preserves reduced installations but is not durable + across process failure or multiple workers. +- Use `EventBus` as the in-process dispatch contract for module reactions + invoked by the outbox dispatcher or for non-critical fallback reactions. - Use the shared `audit_event` / `audit_from_principal` helper for audited - module actions. The helper persists the audit row and immediately publishes a + module actions. The helper persists the audit row and transactionally emits a governed `PlatformEvent` whose `type` is the audit action. - Use `record_change` for module delta feeds. It persists the change-sequence - row and immediately publishes a generic module change event such as + row and transactionally emits a generic module change event such as `mail.profile.updated`. -- Persist durable integration/workflow events through a database outbox before - acknowledging the state change that produced them. -- Drain the outbox through a small dispatcher process. The dispatcher may call - in-process handlers in the same deployment first, but its storage contract is - database-backed. +- Drain the outbox with the Celery `govoplan.events.dispatch_outbox` task on the + `events` queue. The periodic schedule also retries pending rows. +- The dispatcher invokes Dataflow event ingestion when that capability is + active, then publishes to the process-local bus. - Treat Redis/Celery as worker/job infrastructure, not as the authoritative - first event transport. A Celery dispatcher can consume the outbox later. + first event transport. PostgreSQL remains authoritative until dispatch is + recorded. - Keep the dispatch implementation pluggable behind the `PlatformEvent` envelope so a future message broker can be added without changing event producers. @@ -51,17 +58,18 @@ Event producers should write their domain state and outbox event in the same database transaction wherever possible. Handlers must be idempotent because the outbox dispatcher can retry after a crash or timeout. -Recommended first outbox columns: +The current outbox stores: - `event_id`, `event_type`, `module_id` - `correlation_id`, `causation_id` -- `payload`, `occurred_at` -- `available_at`, `attempt_count`, `claimed_at`, `claim_token` -- `processed_at`, `last_error` +- `classification`, serialized event `payload` +- `status`, `attempts`, `next_attempt_at` +- `dispatched_at`, `last_error`, timestamps -Inline `EventBus` handlers are allowed only for non-critical local reactions. -Anything that must survive process failure, restart, package update, or worker -redeployment belongs in the outbox. +Handlers must be idempotent: a worker may complete an external effect and fail +before marking its outbox row dispatched. Anything that must survive process +failure, restart, package update, or worker redeployment requires the outbox +provider and dispatcher. ## Trace IDs diff --git a/Repo-docs-EXTERNAL-REFERENCES-AND-INTEGRATION-MATURITY.md b/Repo-docs-EXTERNAL-REFERENCES-AND-INTEGRATION-MATURITY.md new file mode 100644 index 0000000..9d795dd --- /dev/null +++ b/Repo-docs-EXTERNAL-REFERENCES-AND-INTEGRATION-MATURITY.md @@ -0,0 +1,54 @@ + + +> Mirrored from `/mnt/DATA/git/govoplan-core/docs/EXTERNAL_REFERENCES_AND_INTEGRATION_MATURITY.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# External References And Integration Maturity + +GovOPlaN integrations use a shared external-reference contract instead of +storing connector-specific URLs and identifiers in every module. + +An external reference identifies an object by: + +- external system instance +- object type +- stable external object ID +- optional connector configuration +- canonical HTTP(S) URL without embedded credentials +- optional source version, ETag, observation time, and non-secret metadata + +The identity key is `system:object_type:object_id`. A GovOPlaN object may retain +multiple references, but one reference must never silently change its identity. +Moving or escalating work creates a new object and an explicit relationship; it +does not rewrite either object's history. + +## Integration Maturity + +Maturity is cumulative: + +1. `discover`: identify configured external systems and their health. +2. `link`: retain and open stable external references. +3. `search`: include authorized external objects in GovOPlaN search. +4. `read`: display authoritative external content. +5. `publish`: create or update external content from GovOPlaN. +6. `synchronize`: reconcile changes in both directions with conflict handling. +7. `migrate`: perform a governed, verifiable transfer into GovOPlaN. +8. `replace`: provide the native operational capability without the external tool. + +Connectors must declare and document the maturity they actually implement. +`synchronize` requires durable cursors, idempotency, provenance, conflict +handling, deletion semantics, and observable failures. A link-only connector +must not imply that GovOPlaN holds an authoritative copy. + +## Domain Ownership + +- Domain modules own native GovOPlaN objects and their authorization. +- Connectors own protocols, credentials, discovery, transport, and sync state. +- Search owns indexing and result aggregation, but source modules remain + responsible for authorization. +- Core owns only the stable DTOs and extension contracts. + +The Python contract is +`govoplan_core.core.external_references.ExternalObjectReference`. diff --git a/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md b/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md index a879186..56f6067 100644 --- a/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md +++ b/Repo-docs-GOVOPLAN-MASTER-ROADMAP.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/GOVOPLAN_MASTER_ROADMAP.md`. > Origin: `repository`. @@ -419,7 +419,7 @@ Goal: cover internal support and public issue reporting. Create or refine in this order: -1. `govoplan-issue-reporting`: public/internal reports, categories, intake, +1. `govoplan-tickets`: public/internal reports, requests, incidents, queues, location, evidence, and triage. 2. `govoplan-helpdesk`: service desk tickets, queues, SLAs, assignments, escalation, and resolution evidence. diff --git a/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md b/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md index 46465f5..ebbdae3 100644 --- a/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md +++ b/Repo-docs-PUBLIC-SECTOR-INTEGRATION-STRATEGY.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/PUBLIC_SECTOR_INTEGRATION_STRATEGY.md`. > Origin: `repository`. @@ -12,6 +12,9 @@ before deciding to replace specialist workflows. This document is the core strategy index. The executable connector catalogue lives in `govoplan-connectors/docs/PUBLIC_SECTOR_INTEGRATION_CATALOGUE.md`. +The canonical cumulative maturity model and external-object DTO are documented +in [EXTERNAL_REFERENCES_AND_INTEGRATION_MATURITY.md](EXTERNAL_REFERENCES_AND_INTEGRATION_MATURITY.md). + ## Strategy Labels Use one or more of these labels for every external system family: diff --git a/Repo-docs-THROTTLING.md b/Repo-docs-THROTTLING.md index b2e7123..34cf85a 100644 --- a/Repo-docs-THROTTLING.md +++ b/Repo-docs-THROTTLING.md @@ -1,4 +1,4 @@ - + > Mirrored from `/mnt/DATA/git/govoplan-core/docs/THROTTLING.md`. > Origin: `repository`. @@ -10,10 +10,13 @@ Core exposes `govoplan_core.core.throttling` for security-sensitive endpoints that need bounded fixed-window counters. Subjects are SHA-256 hashed before they become store keys. A configured Redis instance provides atomic counters shared -across API workers; development, a missing Redis configuration, and temporary -Redis outages use a bounded process-local fallback. When Redis fails, local -attempts are still mirrored so losing the distributed store does not reset the -active worker's protection window. +across API workers. Development and temporary Redis outages use a bounded +process-local fallback. Production-like startup rejects an enabled login +throttle without `REDIS_URL` unless +`GOVOPLAN_ALLOW_PROCESS_LOCAL_LOGIN_THROTTLE=true` explicitly acknowledges the +single-process limitation. When Redis fails at runtime, local attempts are still +mirrored so losing the distributed store does not reset the active worker's +protection window. Callers define one or more `ThrottleDimension` values with a controlled namespace, a subject and a positive limit. They must call `check` before an