feat: expose institutional architecture and runtime state
This commit is contained in:
@@ -1,5 +1,11 @@
|
||||
# GovOPlaN Ops Codex Guide
|
||||
|
||||
## Documentation Contract
|
||||
|
||||
- Treat documentation as part of every behavior change. Update this module's manifest-driven `DocumentationTopic` contributions for affected user and administrator behavior.
|
||||
- Keep feature content here; `govoplan-docs` projects it without importing Ops internals.
|
||||
- Maintain a static user/admin baseline and run `/mnt/DATA/git/govoplan/tools/checks/check-manifest-shapes.py` after behavior or manifest changes.
|
||||
|
||||
## Scope
|
||||
|
||||
This repository owns the GovOPlaN operations module seed: runtime health,
|
||||
|
||||
@@ -13,9 +13,15 @@ This repository owns:
|
||||
|
||||
- backend module manifest `ops`
|
||||
- operator-facing status APIs
|
||||
- runtime-node registration, heartbeat, composition, stale-node, and expected
|
||||
replica projection
|
||||
- audited API and worker drain/cancel controls
|
||||
- recovery-operation status and evidence-chain summaries
|
||||
- governance inventory for module-declared permissions, roles, capabilities,
|
||||
policies, documentation, access-control hooks, search providers, and
|
||||
migration ownership
|
||||
- sanitized runtime health, freshness, conflict, and recovery state for
|
||||
configured external-provider bindings
|
||||
- deployment profile and sizing assumption summaries
|
||||
- WebUI route contribution `@govoplan/ops-webui`
|
||||
- future operational runbooks that describe the configured platform rather than
|
||||
@@ -28,7 +34,19 @@ Core exposes the registry contract but does not own an operations dashboard.
|
||||
Ops projects the provider-neutral registry metadata and runtime checks into the
|
||||
operator-facing governance surface.
|
||||
|
||||
Provider declarations describe supported behavior; module-owned runtime-state
|
||||
providers describe the currently configured bindings. An optional provider
|
||||
failure is isolated and reported as an attention state without suppressing the
|
||||
rest of the governance inventory. Secrets, endpoints, and raw provider errors
|
||||
are not part of this projection.
|
||||
|
||||
## Runbooks
|
||||
|
||||
- `docs/SCALABILITY_PROFILES.md` explains how to use the Ops page with the
|
||||
canonical sizing matrix, readiness model, and profile-selection worksheet.
|
||||
|
||||
The Runtime cluster panel is backed by Core's shared PostgreSQL coordination
|
||||
tables. A drain request is durable and is observed on the node heartbeat: API
|
||||
readiness closes and workers stop taking new queue work. The Recovery panel
|
||||
shows operations requiring forward recovery or manual intervention; it does not
|
||||
claim that a production database backup exists.
|
||||
|
||||
@@ -15,6 +15,12 @@ live Ops page.
|
||||
campaigns, imports, exports, or workflow automation.
|
||||
6. Record the current profile and open measurements before moving to a larger
|
||||
topology.
|
||||
7. In a replicated profile, compare active non-stale API and worker counts with
|
||||
configured expectations, and resolve composition skew before rollout.
|
||||
8. Drain a node before replacement, then verify it is no longer ready/consuming
|
||||
before terminating it.
|
||||
9. Inspect recovery-required and outcome-unknown operations; follow the recorded
|
||||
recovery mode rather than retrying or rolling back blindly.
|
||||
|
||||
## Live Profile Signals
|
||||
|
||||
@@ -30,10 +36,24 @@ The Ops API reports:
|
||||
- HTTP/certificate deployment posture through the `deployment_security` check
|
||||
- readiness blockers
|
||||
- reference deployment profiles and sizing assumptions
|
||||
- runtime node identity, role, software/module composition, queues, heartbeat,
|
||||
stale state, and drain state
|
||||
- configured versus active API and worker replica counts
|
||||
- recovery operation status, mode, checkpoint count, and last update
|
||||
|
||||
These values are intentionally diagnostic. They do not replace deployment
|
||||
configuration management, backups, monitoring, or restore drills.
|
||||
|
||||
Drain controls are cooperative. API and worker processes observe the request on
|
||||
their next heartbeat. API readiness then fails; workers cancel their configured
|
||||
queue consumers but may still be finishing already claimed work. Confirm the
|
||||
process state and queue evidence before forcefully terminating a node.
|
||||
|
||||
Recovery evidence is similarly conservative. A forward-recovery or
|
||||
manual-intervention record means an operator must repair the current release or
|
||||
restore a separately verified coordinated backup. Ops does not convert that
|
||||
state into a safe rollback.
|
||||
|
||||
`deployment_security` is inactive for local/test profiles. In staging or pilot
|
||||
profiles it warns when secure cookies or CORS origins still look local. In
|
||||
production it becomes readiness-critical because TLS certificates, proxy
|
||||
@@ -93,23 +113,26 @@ Stateless and horizontally replicable:
|
||||
- API workers when `MASTER_KEY_B64`, `DATABASE_URL`, storage, queue, and module
|
||||
configuration are shared
|
||||
- Background workers when queues and idempotency keys are used
|
||||
- Scheduler replicas only when leader election or an external lock exists
|
||||
- Scheduler processes only through Core's renewable PostgreSQL lease and
|
||||
fencing-token runner; deploy one desired scheduler replica
|
||||
|
||||
Stateful or singleton-sensitive:
|
||||
|
||||
- PostgreSQL
|
||||
- local file storage when not replaced by object storage
|
||||
- local file storage when not replaced by object storage (or a one-host shared
|
||||
volume under the `host-shared` profile)
|
||||
- Redis/queue state
|
||||
- module installer daemon and package mutation operations
|
||||
- migration execution
|
||||
- scheduler without distributed locking
|
||||
- migration execution, although competing jobs are serialized by a PostgreSQL
|
||||
advisory lock
|
||||
- outgoing campaign append/send jobs unless claim tokens are enforced
|
||||
|
||||
## Readiness And Degraded Modes
|
||||
|
||||
| Component | Ready When | Degraded Mode |
|
||||
| --- | --- | --- |
|
||||
| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood | Read-only/admin-only where routes allow it; otherwise fail closed. |
|
||||
| API | Database reachable, migrations current, enabled module registry builds, maintenance mode understood, and the node is not draining | Read-only/admin-only where routes allow it; otherwise fail closed. |
|
||||
| WebUI | Static assets match backend module metadata contract | Show unavailable modules/routes with reason; do not invent routes. |
|
||||
| PostgreSQL | Accepts connections and migration head is current | Block writes and package changes if migration state is unknown. |
|
||||
| Storage | Configured backend is reachable and writable for write flows | Read-only file views may continue if storage is read-only but reachable. |
|
||||
|
||||
+1
-1
@@ -10,7 +10,7 @@ readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
authors = [{ name = "GovOPlaN" }]
|
||||
dependencies = [
|
||||
"govoplan-core>=0.1.8",
|
||||
"govoplan-core>=0.1.14",
|
||||
"govoplan-access>=0.1.8",
|
||||
]
|
||||
|
||||
|
||||
@@ -9,9 +9,11 @@ from typing import Any
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import SQLAlchemyError
|
||||
|
||||
from govoplan_core.audit.logging import audit_event
|
||||
from govoplan_core.auth import ApiPrincipal, require_any_scope
|
||||
from govoplan_core.core.maintenance import saved_maintenance_mode
|
||||
from govoplan_core.core.module_installer import (
|
||||
@@ -20,7 +22,20 @@ from govoplan_core.core.module_installer import (
|
||||
read_module_installer_run,
|
||||
)
|
||||
from govoplan_core.core.operations import OperationalCheckProviderRegistration
|
||||
from govoplan_core.core.recovery import (
|
||||
RecoveryOperation,
|
||||
)
|
||||
from govoplan_core.core.registry import PlatformRegistry
|
||||
from govoplan_core.core.provider_governance import (
|
||||
ExternalProviderStateContext,
|
||||
collect_external_provider_states,
|
||||
)
|
||||
from govoplan_core.core.runtime_coordination import (
|
||||
RuntimeCoordinationError,
|
||||
cancel_runtime_node_drain,
|
||||
list_runtime_nodes,
|
||||
request_runtime_node_drain,
|
||||
)
|
||||
from govoplan_core.db.session import get_database
|
||||
from govoplan_core.settings import settings as core_settings
|
||||
|
||||
@@ -33,6 +48,10 @@ _worker_check_cache: tuple[float, dict[str, Any]] | None = None
|
||||
_worker_check_cache_lock = threading.Lock()
|
||||
|
||||
|
||||
class RuntimeDrainRequest(BaseModel):
|
||||
reason: str = Field(default="operator request", min_length=1, max_length=500)
|
||||
|
||||
|
||||
@router.get("/status")
|
||||
def ops_status(
|
||||
request: Request,
|
||||
@@ -64,6 +83,91 @@ def run_ops_checks(
|
||||
return _ops_status_payload(request, force_module_checks=True)
|
||||
|
||||
|
||||
@router.get("/runtime/nodes")
|
||||
def runtime_nodes(
|
||||
principal: ApiPrincipal = Depends(require_any_scope(*OPS_READ_SCOPES)),
|
||||
) -> dict[str, Any]:
|
||||
del principal
|
||||
return _runtime_cluster_status()
|
||||
|
||||
|
||||
@router.post("/runtime/nodes/{node_id}/drain")
|
||||
def drain_runtime_node(
|
||||
node_id: str,
|
||||
body: RuntimeDrainRequest,
|
||||
principal: ApiPrincipal = Depends(require_any_scope(*OPS_RUN_SCOPES)),
|
||||
) -> dict[str, Any]:
|
||||
try:
|
||||
with get_database().SessionLocal() as session:
|
||||
node = request_runtime_node_drain(
|
||||
session,
|
||||
installation_id=core_settings.installation_id,
|
||||
node_id=node_id,
|
||||
reason=body.reason,
|
||||
)
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=None,
|
||||
scope="system",
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="ops.runtime_node.drain_requested",
|
||||
object_type="runtime_node",
|
||||
object_id=node.node_id,
|
||||
details={
|
||||
"installation_id": core_settings.installation_id,
|
||||
"reason": node.drain_reason,
|
||||
"incarnation": node.incarnation,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return {
|
||||
"node_id": node.node_id,
|
||||
"state": node.state,
|
||||
"drain_reason": node.drain_reason,
|
||||
}
|
||||
except RuntimeCoordinationError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
|
||||
|
||||
@router.delete("/runtime/nodes/{node_id}/drain")
|
||||
def cancel_runtime_drain(
|
||||
node_id: str,
|
||||
principal: ApiPrincipal = Depends(require_any_scope(*OPS_RUN_SCOPES)),
|
||||
) -> dict[str, Any]:
|
||||
try:
|
||||
with get_database().SessionLocal() as session:
|
||||
node = cancel_runtime_node_drain(
|
||||
session,
|
||||
installation_id=core_settings.installation_id,
|
||||
node_id=node_id,
|
||||
)
|
||||
audit_event(
|
||||
session,
|
||||
tenant_id=None,
|
||||
scope="system",
|
||||
user_id=getattr(principal.user, "id", None),
|
||||
api_key_id=principal.api_key_id,
|
||||
action="ops.runtime_node.drain_cancelled",
|
||||
object_type="runtime_node",
|
||||
object_id=node.node_id,
|
||||
details={
|
||||
"installation_id": core_settings.installation_id,
|
||||
"incarnation": node.incarnation,
|
||||
},
|
||||
)
|
||||
session.commit()
|
||||
return {"node_id": node.node_id, "state": node.state}
|
||||
except RuntimeCoordinationError as exc:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=str(exc),
|
||||
) from exc
|
||||
|
||||
|
||||
def _ops_status_payload(
|
||||
request: Request,
|
||||
*,
|
||||
@@ -75,6 +179,7 @@ def _ops_status_payload(
|
||||
redis_check = _redis_check()
|
||||
current_profile = _current_profile()
|
||||
worker_check = _worker_check()
|
||||
runtime_cluster = _runtime_cluster_status()
|
||||
module_checks = _module_operational_checks(
|
||||
registry,
|
||||
force=force_module_checks,
|
||||
@@ -85,13 +190,17 @@ def _ops_status_payload(
|
||||
_maintenance_check(maintenance_mode),
|
||||
redis_check,
|
||||
worker_check,
|
||||
_runtime_cluster_check(runtime_cluster),
|
||||
_storage_check(),
|
||||
_backup_restore_check(),
|
||||
_deployment_security_check(current_profile),
|
||||
*module_checks,
|
||||
]
|
||||
readiness = _readiness(checks, maintenance_mode)
|
||||
governance = _governance_inventory(registry)
|
||||
governance = _governance_inventory(
|
||||
registry,
|
||||
external_provider_states=_external_provider_states(registry),
|
||||
)
|
||||
return {
|
||||
"summary": {
|
||||
"app_env": core_settings.app_env,
|
||||
@@ -105,15 +214,137 @@ def _ops_status_payload(
|
||||
"file_storage_backend": core_settings.file_storage_backend,
|
||||
"worker_metrics": worker_check.get("metrics", {}),
|
||||
"operational_probe_count": len(module_checks),
|
||||
"runtime_node_count": len(runtime_cluster.get("nodes", [])),
|
||||
"recovery_required_count": runtime_cluster.get("recovery", {}).get("requires_attention", 0),
|
||||
},
|
||||
"readiness": readiness,
|
||||
"checks": checks,
|
||||
"governance": governance,
|
||||
"deployment_profiles": _deployment_profiles(current_profile),
|
||||
"sizing": _sizing_assumptions(),
|
||||
"runtime_cluster": runtime_cluster,
|
||||
}
|
||||
|
||||
|
||||
def _runtime_cluster_status() -> dict[str, Any]:
|
||||
try:
|
||||
with get_database().SessionLocal() as session:
|
||||
nodes = list_runtime_nodes(
|
||||
session,
|
||||
installation_id=core_settings.installation_id,
|
||||
stale_after_seconds=core_settings.runtime_stale_after_seconds,
|
||||
)
|
||||
operations = (
|
||||
session.query(RecoveryOperation)
|
||||
.filter(
|
||||
RecoveryOperation.installation_id
|
||||
== core_settings.installation_id
|
||||
)
|
||||
.order_by(RecoveryOperation.updated_at.desc())
|
||||
.limit(50)
|
||||
.all()
|
||||
)
|
||||
except SQLAlchemyError as exc:
|
||||
return {
|
||||
"available": False,
|
||||
"detail": f"Runtime coordination query failed: {exc}",
|
||||
"state_profile": core_settings.state_profile,
|
||||
"nodes": [],
|
||||
"recovery": {"operations": [], "requires_attention": 0},
|
||||
}
|
||||
active_nodes = [
|
||||
node
|
||||
for node in nodes
|
||||
if node["state"] != "stopped" and not node["stale"]
|
||||
]
|
||||
active_api = sum(node["role"] == "api" for node in active_nodes)
|
||||
active_workers = sum(node["role"] == "worker" for node in active_nodes)
|
||||
operation_payloads = [
|
||||
{
|
||||
"id": operation.id,
|
||||
"module_id": operation.module_id,
|
||||
"operation_type": operation.operation_type,
|
||||
"resource_type": operation.resource_type,
|
||||
"resource_id": operation.resource_id,
|
||||
"mode": operation.mode,
|
||||
"status": operation.status,
|
||||
"checkpoint_count": operation.checkpoint_count,
|
||||
"evidence_head_sha256": operation.evidence_head_sha256,
|
||||
"failure_summary": operation.failure_summary,
|
||||
"updated_at": operation.updated_at.isoformat(),
|
||||
}
|
||||
for operation in operations
|
||||
]
|
||||
requires_attention = sum(
|
||||
operation["status"]
|
||||
in {
|
||||
"outcome_unknown",
|
||||
"recovery_required",
|
||||
"recovering",
|
||||
"manual_intervention",
|
||||
}
|
||||
for operation in operation_payloads
|
||||
)
|
||||
return {
|
||||
"available": True,
|
||||
"detail": "Shared runtime directory is available.",
|
||||
"installation_id": core_settings.installation_id,
|
||||
"state_profile": core_settings.state_profile,
|
||||
"expected": {
|
||||
"api": core_settings.runtime_expected_api_replicas,
|
||||
"worker": core_settings.runtime_expected_worker_replicas,
|
||||
},
|
||||
"active": {"api": active_api, "worker": active_workers},
|
||||
"nodes": nodes,
|
||||
"recovery": {
|
||||
"operations": operation_payloads,
|
||||
"requires_attention": requires_attention,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _runtime_cluster_check(cluster: dict[str, Any]) -> dict[str, Any]:
|
||||
if not cluster.get("available"):
|
||||
return _check(
|
||||
"runtime_cluster",
|
||||
"Runtime cluster",
|
||||
"error",
|
||||
str(cluster.get("detail") or "Runtime directory unavailable."),
|
||||
readiness_critical=True,
|
||||
)
|
||||
nodes = cluster.get("nodes") or []
|
||||
stale = [node for node in nodes if node.get("stale")]
|
||||
expected = cluster.get("expected") or {}
|
||||
active = cluster.get("active") or {}
|
||||
missing_api = max(0, int(expected.get("api") or 0) - int(active.get("api") or 0))
|
||||
missing_workers = max(0, int(expected.get("worker") or 0) - int(active.get("worker") or 0))
|
||||
state = "ok"
|
||||
detail = f"{len(nodes)} runtime node record(s); {len(stale)} stale."
|
||||
readiness_critical = False
|
||||
if missing_api or missing_workers:
|
||||
state = "error" if cluster.get("state_profile") == "shared" else "warning"
|
||||
readiness_critical = state == "error"
|
||||
detail += f" Missing expected replicas: api={missing_api}, worker={missing_workers}."
|
||||
elif stale:
|
||||
state = "warning"
|
||||
return _check(
|
||||
"runtime_cluster",
|
||||
"Runtime cluster",
|
||||
state,
|
||||
detail,
|
||||
readiness_critical=readiness_critical,
|
||||
metrics={
|
||||
"nodes": len(nodes),
|
||||
"stale": len(stale),
|
||||
"active_api": int(active.get("api") or 0),
|
||||
"active_workers": int(active.get("worker") or 0),
|
||||
"recovery_required": int(
|
||||
cluster.get("recovery", {}).get("requires_attention") or 0
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _registry(request: Request) -> PlatformRegistry:
|
||||
registry = getattr(request.app.state, "govoplan_registry", None)
|
||||
if not isinstance(registry, PlatformRegistry):
|
||||
@@ -121,10 +352,27 @@ def _registry(request: Request) -> PlatformRegistry:
|
||||
return registry
|
||||
|
||||
|
||||
def _governance_inventory(registry: PlatformRegistry) -> dict[str, Any]:
|
||||
def _governance_inventory(
|
||||
registry: PlatformRegistry,
|
||||
*,
|
||||
external_provider_states: Mapping[str, Mapping[str, object]] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
modules: list[dict[str, Any]] = []
|
||||
for manifest in registry.manifests():
|
||||
capability_names = tuple(sorted(manifest.capability_factories))
|
||||
architecture = (
|
||||
manifest.architecture.to_dict() if manifest.architecture else None
|
||||
)
|
||||
external_providers = tuple(
|
||||
{
|
||||
**declaration.to_dict(),
|
||||
"runtime_state": dict(
|
||||
(external_provider_states or {}).get(declaration.id, {})
|
||||
)
|
||||
or None,
|
||||
}
|
||||
for declaration in manifest.external_providers
|
||||
)
|
||||
access_control_count = (
|
||||
len(manifest.resource_acl_providers)
|
||||
+ len(manifest.ownership_providers)
|
||||
@@ -144,6 +392,9 @@ def _governance_inventory(registry: PlatformRegistry) -> dict[str, Any]:
|
||||
"access_control_count": access_control_count,
|
||||
"search_provider_count": len(manifest.search_providers) + len(manifest.search_sources),
|
||||
"migration_managed": manifest.migration_spec is not None,
|
||||
"architecture": architecture,
|
||||
"external_provider_count": len(external_providers),
|
||||
"external_providers": external_providers,
|
||||
})
|
||||
return {
|
||||
"summary": {
|
||||
@@ -159,11 +410,62 @@ def _governance_inventory(registry: PlatformRegistry) -> dict[str, Any]:
|
||||
"access_control_count": sum(item["access_control_count"] for item in modules),
|
||||
"search_provider_count": sum(item["search_provider_count"] for item in modules),
|
||||
"migration_module_count": sum(bool(item["migration_managed"]) for item in modules),
|
||||
"architecture_declared_module_count": sum(
|
||||
bool(item["architecture"]) for item in modules
|
||||
),
|
||||
"external_provider_count": sum(
|
||||
item["external_provider_count"] for item in modules
|
||||
),
|
||||
"configured_external_provider_count": sum(
|
||||
bool(
|
||||
provider.get("runtime_state", {}).get("configured")
|
||||
if isinstance(provider.get("runtime_state"), Mapping)
|
||||
else False
|
||||
)
|
||||
for item in modules
|
||||
for provider in item["external_providers"]
|
||||
),
|
||||
"provider_attention_count": sum(
|
||||
(
|
||||
str(provider.get("runtime_state", {}).get("health"))
|
||||
in {"warning", "error", "unknown"}
|
||||
or str(provider.get("runtime_state", {}).get("recovery"))
|
||||
== "attention"
|
||||
)
|
||||
if isinstance(provider.get("runtime_state"), Mapping)
|
||||
else False
|
||||
for item in modules
|
||||
for provider in item["external_providers"]
|
||||
),
|
||||
"supported_module_count": sum(
|
||||
bool(item["architecture"])
|
||||
and item["architecture"]["maturity"] in {"supported", "lts"}
|
||||
for item in modules
|
||||
),
|
||||
},
|
||||
"modules": modules,
|
||||
}
|
||||
|
||||
|
||||
def _external_provider_states(
|
||||
registry: PlatformRegistry,
|
||||
) -> dict[str, dict[str, object]]:
|
||||
registrations = registry.external_provider_state_providers()
|
||||
if not registrations:
|
||||
return {}
|
||||
try:
|
||||
with get_database().session() as session:
|
||||
return collect_external_provider_states(
|
||||
registrations,
|
||||
ExternalProviderStateContext(session=session),
|
||||
)
|
||||
except Exception: # noqa: BLE001 - preserve Ops inventory during DB outages.
|
||||
return collect_external_provider_states(
|
||||
registrations,
|
||||
ExternalProviderStateContext(session=None),
|
||||
)
|
||||
|
||||
|
||||
def _database_status() -> dict[str, Any]:
|
||||
try:
|
||||
with get_database().session() as session:
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from govoplan_core.core.access import CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PRINCIPAL_RESOLVER
|
||||
from govoplan_core.core.modules import FrontendModule, FrontendRoute, ModuleContext, ModuleManifest, NavItem, PermissionDefinition, RoleTemplate
|
||||
from govoplan_core.core.modules import DocumentationCondition, DocumentationTopic, FrontendModule, FrontendRoute, ModuleContext, ModuleManifest, NavItem, PermissionDefinition, RoleTemplate
|
||||
from govoplan_core.core.provider_governance import ModuleArchitectureDeclaration, ModuleArchitectureDocumentation, ModuleMaturityEvidence
|
||||
from govoplan_core.core.views import ViewSurface
|
||||
|
||||
OPS_READ_SCOPE = "ops:operations:read"
|
||||
@@ -9,6 +10,33 @@ OPS_READ_SCOPES = (OPS_READ_SCOPE, "system:settings:read", "admin:settings:read"
|
||||
OPS_RUN_SCOPE = "ops:operations:run"
|
||||
OPS_RUN_SCOPES = (OPS_RUN_SCOPE, "system:settings:write")
|
||||
|
||||
ARCHITECTURE = ModuleArchitectureDeclaration(
|
||||
layer="runtime_meta",
|
||||
kind="operations",
|
||||
maturity="vertical_slice",
|
||||
evidence=(
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_governance_inventory.py",
|
||||
summary="Tests safe operational projection of module governance declarations.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="documentation",
|
||||
reference="docs/SCALABILITY_PROFILES.md",
|
||||
summary="Documents operational topology and scaling posture.",
|
||||
),
|
||||
),
|
||||
known_limits=(
|
||||
"Provider health observations depend on module-owned operational probes and may be unavailable until configured.",
|
||||
),
|
||||
owned_concepts=("operations status projection", "bounded operational probes"),
|
||||
non_owned_concepts=("domain repair", "external provider credentials"),
|
||||
documentation=ModuleArchitectureDocumentation(
|
||||
recovery=("docs/SCALABILITY_PROFILES.md",),
|
||||
operations=("docs/SCALABILITY_PROFILES.md",),
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _permission(scope: str, label: str, description: str) -> PermissionDefinition:
|
||||
module_id, resource, action = scope.split(":", 2)
|
||||
@@ -65,6 +93,48 @@ manifest = ModuleManifest(
|
||||
level="system",
|
||||
),
|
||||
),
|
||||
documentation=(
|
||||
DocumentationTopic(
|
||||
id="ops.health-governance-and-sizing",
|
||||
title="Inspect platform health and deployment posture",
|
||||
summary="Ops combines module-owned health checks with deployment profile, governance inventory, worker assumptions, and sizing guidance.",
|
||||
body="Read-only status distinguishes configured capabilities from healthy integrations. Authorized operators can run bounded probes; a probe must not perform unbounded business work or silently repair data. Use readiness and worker results when diagnosing a node, and use the deployment profile and sizing assumptions when planning horizontal capacity.",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("operator", "system_admin"),
|
||||
related_modules=("audit", "docs", "notifications"),
|
||||
metadata={"kind": "reference"},
|
||||
),
|
||||
DocumentationTopic(
|
||||
id="ops.runtime-coordination-and-recovery",
|
||||
title="Drain runtime nodes and inspect recovery evidence",
|
||||
summary="Ops projects shared runtime heartbeats, replica gaps, drain controls, and recovery states that require operator attention.",
|
||||
body="Use the runtime table to identify stale or composition-skewed API and worker replicas. Drain before replacement so API readiness closes and workers stop taking new queue work; cancellation is available while the node is still draining. The recovery table reports durable Core recovery operations, but a recorded forward-recovery or manual-intervention state is not an automatic database restore.",
|
||||
documentation_types=("admin", "user"),
|
||||
audience=("operator", "system_admin"),
|
||||
conditions=(
|
||||
DocumentationCondition(
|
||||
required_modules=("ops",),
|
||||
any_scopes=OPS_READ_SCOPES,
|
||||
),
|
||||
),
|
||||
related_modules=("audit", "notifications"),
|
||||
metadata={
|
||||
"kind": "workflow",
|
||||
"route": "/ops",
|
||||
"screen": "Runtime cluster and recovery evidence",
|
||||
"steps": [
|
||||
"Compare active non-stale nodes with the configured API and worker replica expectations.",
|
||||
"Request drain and wait for the node to report draining before replacing it.",
|
||||
"Inspect every recovery-required, outcome-unknown, or manual-intervention operation and follow its recorded recovery mode.",
|
||||
"Verify replacement composition, readiness, queue consumers, and recovery evidence before closing the operation.",
|
||||
],
|
||||
"limitations": [
|
||||
"Drain is observed on the runtime heartbeat interval and does not forcibly terminate active work.",
|
||||
"Ops does not create database backups or make an unsafe post-migration rollback reversible.",
|
||||
],
|
||||
},
|
||||
),
|
||||
),
|
||||
route_factory=_route_factory,
|
||||
nav_items=(NavItem(path="/ops", label="Ops", icon="activity", required_any=OPS_READ_SCOPES, order=890),),
|
||||
frontend=FrontendModule(
|
||||
@@ -76,6 +146,7 @@ manifest = ModuleManifest(
|
||||
ViewSurface(id="ops.widget.health", module_id="ops", kind="section", label="Operations health widget", order=100),
|
||||
),
|
||||
),
|
||||
architecture=ARCHITECTURE,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -11,6 +11,14 @@ from govoplan_core.core.modules import (
|
||||
RoleTemplate,
|
||||
)
|
||||
from govoplan_core.core.registry import PlatformRegistry
|
||||
from govoplan_core.core.provider_governance import (
|
||||
ExternalProviderDeclaration,
|
||||
ModuleArchitectureDeclaration,
|
||||
ModuleArchitectureDocumentation,
|
||||
ModuleMaturityEvidence,
|
||||
ProviderBehaviorDeclaration,
|
||||
ProviderObjectDeclaration,
|
||||
)
|
||||
|
||||
from govoplan_ops.backend.api.v1.routes import _governance_inventory
|
||||
|
||||
@@ -60,9 +68,70 @@ class GovernanceInventoryTests(unittest.TestCase):
|
||||
),
|
||||
),
|
||||
migration_spec=MigrationSpec(module_id="example"),
|
||||
architecture=ModuleArchitectureDeclaration(
|
||||
layer="governance_accountability",
|
||||
kind="governance",
|
||||
maturity="vertical_slice",
|
||||
evidence=(
|
||||
ModuleMaturityEvidence(
|
||||
kind="test",
|
||||
reference="tests/test_governance_inventory.py",
|
||||
summary="Exercises the governance projection.",
|
||||
),
|
||||
ModuleMaturityEvidence(
|
||||
kind="documentation",
|
||||
reference="example.reference",
|
||||
summary="Documents the example provider.",
|
||||
),
|
||||
),
|
||||
known_limits=("The example has no runtime implementation.",),
|
||||
supported_authority_modes=("external_mirror",),
|
||||
owned_concepts=("example projections",),
|
||||
documentation=ModuleArchitectureDocumentation(
|
||||
operations=("example.reference",),
|
||||
),
|
||||
),
|
||||
external_providers=(
|
||||
ExternalProviderDeclaration(
|
||||
id="example.read",
|
||||
module_id="example",
|
||||
label="Example reader",
|
||||
maturity="read",
|
||||
operations=("read", "search"),
|
||||
objects=(
|
||||
ProviderObjectDeclaration(
|
||||
object_type="example.record",
|
||||
field_groups=("identity", "summary"),
|
||||
authority_modes=("external_mirror",),
|
||||
default_authority_mode="external_mirror",
|
||||
),
|
||||
),
|
||||
behavior=ProviderBehaviorDeclaration(
|
||||
freshness="Request-time read with a five-minute cache.",
|
||||
health="Reported through example.health.",
|
||||
max_read_items=100,
|
||||
outage="Reads fail closed and retain the last inspected snapshot.",
|
||||
classifications=("internal",),
|
||||
purposes=("operations",),
|
||||
retention="No provider-owned payload is retained.",
|
||||
),
|
||||
documentation_topic_ids=("example.reference",),
|
||||
),
|
||||
),
|
||||
))
|
||||
|
||||
payload = _governance_inventory(registry)
|
||||
payload = _governance_inventory(
|
||||
registry,
|
||||
external_provider_states={
|
||||
"example.read": {
|
||||
"configured": True,
|
||||
"active": True,
|
||||
"health": "warning",
|
||||
"freshness": "stale",
|
||||
"recovery": "attention",
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
self.assertEqual(payload["summary"]["module_count"], 1)
|
||||
self.assertEqual(payload["summary"]["permission_count"], 1)
|
||||
@@ -71,6 +140,21 @@ class GovernanceInventoryTests(unittest.TestCase):
|
||||
self.assertEqual(payload["summary"]["policy_count"], 1)
|
||||
self.assertEqual(payload["summary"]["documented_module_count"], 1)
|
||||
self.assertEqual(payload["summary"]["migration_module_count"], 1)
|
||||
self.assertEqual(payload["summary"]["architecture_declared_module_count"], 1)
|
||||
self.assertEqual(payload["summary"]["external_provider_count"], 1)
|
||||
self.assertEqual(
|
||||
payload["summary"]["configured_external_provider_count"],
|
||||
1,
|
||||
)
|
||||
self.assertEqual(payload["summary"]["provider_attention_count"], 1)
|
||||
self.assertEqual(payload["modules"][0]["architecture"]["maturity"], "vertical_slice")
|
||||
self.assertEqual(payload["modules"][0]["external_providers"][0]["id"], "example.read")
|
||||
self.assertEqual(
|
||||
payload["modules"][0]["external_providers"][0]["runtime_state"][
|
||||
"freshness"
|
||||
],
|
||||
"stale",
|
||||
)
|
||||
self.assertEqual(payload["modules"][0]["module_id"], "example")
|
||||
self.assertNotIn("capability_factories", payload["modules"][0])
|
||||
self.assertFalse(any(
|
||||
|
||||
@@ -63,3 +63,22 @@ def test_module_operational_check_failure_is_isolated() -> None:
|
||||
assert result["state"] == "error"
|
||||
assert "secret detail" not in result["detail"]
|
||||
|
||||
|
||||
def test_shared_runtime_cluster_missing_replicas_blocks_readiness() -> None:
|
||||
check = routes._runtime_cluster_check(
|
||||
{
|
||||
"available": True,
|
||||
"state_profile": "shared",
|
||||
"nodes": [
|
||||
{"role": "api", "state": "active", "stale": False},
|
||||
{"role": "worker", "state": "active", "stale": True},
|
||||
],
|
||||
"expected": {"api": 2, "worker": 1},
|
||||
"active": {"api": 1, "worker": 0},
|
||||
"recovery": {"requires_attention": 1},
|
||||
}
|
||||
)
|
||||
|
||||
assert check["state"] == "error"
|
||||
assert check["readiness_critical"] is True
|
||||
assert check["metrics"]["recovery_required"] == 1
|
||||
|
||||
@@ -37,6 +37,91 @@ export type OpsGovernanceModule = {
|
||||
access_control_count: number;
|
||||
search_provider_count: number;
|
||||
migration_managed: boolean;
|
||||
architecture?: {
|
||||
contract_version: string;
|
||||
layer: string;
|
||||
kind: string;
|
||||
maturity: string;
|
||||
known_limits: string[];
|
||||
supported_authority_modes: string[];
|
||||
owned_concepts: string[];
|
||||
non_owned_concepts: string[];
|
||||
evidence: Array<{ kind: string; reference: string; summary: string }>;
|
||||
} | null;
|
||||
external_provider_count: number;
|
||||
external_providers: Array<{
|
||||
id: string;
|
||||
label: string;
|
||||
maturity: string;
|
||||
authority_modes: string[];
|
||||
operations: string[];
|
||||
behavior: { outage?: string | null };
|
||||
runtime_state?: {
|
||||
configured: boolean;
|
||||
active: boolean;
|
||||
authority_mode?: string | null;
|
||||
authority_modes?: string[];
|
||||
health: string;
|
||||
freshness: string;
|
||||
conflict: string;
|
||||
recovery: string;
|
||||
observed_at: string;
|
||||
last_success_at?: string | null;
|
||||
bindings?: Array<{
|
||||
binding_ref: string;
|
||||
authority_mode: string;
|
||||
active: boolean;
|
||||
health: string;
|
||||
freshness: string;
|
||||
conflict: string;
|
||||
recovery: string;
|
||||
}>;
|
||||
} | null;
|
||||
}>;
|
||||
};
|
||||
|
||||
export type OpsRuntimeNode = {
|
||||
node_id: string;
|
||||
incarnation: string;
|
||||
role: string;
|
||||
software_version: string;
|
||||
composition_hash: string;
|
||||
queues: string[];
|
||||
state: "active" | "draining" | "stopped" | string;
|
||||
started_at: string;
|
||||
last_heartbeat_at: string;
|
||||
drain_requested_at?: string | null;
|
||||
drain_reason?: string | null;
|
||||
stopped_at?: string | null;
|
||||
stale: boolean;
|
||||
};
|
||||
|
||||
export type OpsRecoveryOperation = {
|
||||
id: string;
|
||||
module_id: string;
|
||||
operation_type: string;
|
||||
resource_type?: string | null;
|
||||
resource_id?: string | null;
|
||||
mode: string;
|
||||
status: string;
|
||||
checkpoint_count: number;
|
||||
evidence_head_sha256?: string | null;
|
||||
failure_summary?: string | null;
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type OpsRuntimeCluster = {
|
||||
available: boolean;
|
||||
detail: string;
|
||||
installation_id?: string;
|
||||
state_profile: string;
|
||||
expected?: { api: number; worker: number };
|
||||
active?: { api: number; worker: number };
|
||||
nodes: OpsRuntimeNode[];
|
||||
recovery: {
|
||||
operations: OpsRecoveryOperation[];
|
||||
requires_attention: number;
|
||||
};
|
||||
};
|
||||
|
||||
export type OpsStatus = {
|
||||
@@ -62,6 +147,8 @@ export type OpsStatus = {
|
||||
queue_depths?: Record<string, number>;
|
||||
};
|
||||
operational_probe_count: number;
|
||||
runtime_node_count: number;
|
||||
recovery_required_count: number;
|
||||
};
|
||||
readiness: {
|
||||
ready: boolean;
|
||||
@@ -85,11 +172,17 @@ export type OpsStatus = {
|
||||
access_control_count: number;
|
||||
search_provider_count: number;
|
||||
migration_module_count: number;
|
||||
architecture_declared_module_count: number;
|
||||
external_provider_count: number;
|
||||
configured_external_provider_count: number;
|
||||
provider_attention_count: number;
|
||||
supported_module_count: number;
|
||||
};
|
||||
modules: OpsGovernanceModule[];
|
||||
};
|
||||
deployment_profiles: OpsDeploymentProfile[];
|
||||
sizing: OpsSizingAssumption[];
|
||||
runtime_cluster: OpsRuntimeCluster;
|
||||
};
|
||||
|
||||
export function fetchOpsStatus(settings: ApiSettings): Promise<OpsStatus> {
|
||||
@@ -99,3 +192,30 @@ export function fetchOpsStatus(settings: ApiSettings): Promise<OpsStatus> {
|
||||
export function runOpsChecks(settings: ApiSettings): Promise<OpsStatus> {
|
||||
return apiFetch(settings, "/api/v1/ops/checks/run", { method: "POST" });
|
||||
}
|
||||
|
||||
export function drainRuntimeNode(
|
||||
settings: ApiSettings,
|
||||
nodeId: string,
|
||||
reason = "operator request"
|
||||
): Promise<{ node_id: string; state: string; drain_reason: string }> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/ops/runtime/nodes/${encodeURIComponent(nodeId)}/drain`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ reason })
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
export function cancelRuntimeNodeDrain(
|
||||
settings: ApiSettings,
|
||||
nodeId: string
|
||||
): Promise<{ node_id: string; state: string }> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/ops/runtime/nodes/${encodeURIComponent(nodeId)}/drain`,
|
||||
{ method: "DELETE" }
|
||||
);
|
||||
}
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
import { useEffect, useState } from "react";
|
||||
import { RefreshCw } from "lucide-react";
|
||||
import { PauseCircle, PlayCircle, RefreshCw } from "lucide-react";
|
||||
import {
|
||||
Button,
|
||||
Card,
|
||||
ConfirmDialog,
|
||||
DataGrid,
|
||||
DismissibleAlert,
|
||||
LoadingFrame,
|
||||
@@ -18,10 +19,14 @@ import {
|
||||
"@govoplan/core-webui";
|
||||
import {
|
||||
fetchOpsStatus,
|
||||
cancelRuntimeNodeDrain,
|
||||
drainRuntimeNode,
|
||||
runOpsChecks,
|
||||
type OpsCheck,
|
||||
type OpsDeploymentProfile,
|
||||
type OpsGovernanceModule,
|
||||
type OpsRecoveryOperation,
|
||||
type OpsRuntimeNode,
|
||||
type OpsSizingAssumption,
|
||||
type OpsStatus
|
||||
} from "../../api/ops";
|
||||
@@ -30,6 +35,8 @@ export default function OpsPage({ settings, auth }: {settings: ApiSettings;auth:
|
||||
const [status, setStatus] = useState<OpsStatus | null>(null);
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState("");
|
||||
const [drainTarget, setDrainTarget] = useState<OpsRuntimeNode | null>(null);
|
||||
const [nodeActionId, setNodeActionId] = useState("");
|
||||
|
||||
async function load() {
|
||||
setLoading(true);
|
||||
@@ -55,6 +62,34 @@ export default function OpsPage({ settings, auth }: {settings: ApiSettings;auth:
|
||||
}
|
||||
}
|
||||
|
||||
async function confirmDrain() {
|
||||
if (!drainTarget) return;
|
||||
setNodeActionId(drainTarget.node_id);
|
||||
setError("");
|
||||
try {
|
||||
await drainRuntimeNode(settings, drainTarget.node_id);
|
||||
setDrainTarget(null);
|
||||
setStatus(await fetchOpsStatus(settings));
|
||||
} catch (err) {
|
||||
setError(adminErrorMessage(err));
|
||||
} finally {
|
||||
setNodeActionId("");
|
||||
}
|
||||
}
|
||||
|
||||
async function cancelDrain(node: OpsRuntimeNode) {
|
||||
setNodeActionId(node.node_id);
|
||||
setError("");
|
||||
try {
|
||||
await cancelRuntimeNodeDrain(settings, node.node_id);
|
||||
setStatus(await fetchOpsStatus(settings));
|
||||
} catch (err) {
|
||||
setError(adminErrorMessage(err));
|
||||
} finally {
|
||||
setNodeActionId("");
|
||||
}
|
||||
}
|
||||
|
||||
useEffect(() => {void load();}, [settings.apiBaseUrl, settings.apiKey, settings.accessToken]);
|
||||
|
||||
const checks = status?.checks ?? [];
|
||||
@@ -89,6 +124,9 @@ export default function OpsPage({ settings, auth }: {settings: ApiSettings;auth:
|
||||
<MetricCard label="i18n:govoplan-ops.workers.b6ef3acd" value={status?.summary.celery_enabled ? status.summary.worker_metrics.workers ?? 0 : "off"} tone={status?.summary.celery_enabled && !(status.summary.worker_metrics.missing_queues?.length) ? "good" : "warning"} detail={workerMetricDetail(status)} />
|
||||
<MetricCard label="i18n:govoplan-ops.redis.5eaa1f2f" value={status?.summary.redis_url ? "configured" : "-"} tone={status?.summary.celery_enabled ? "info" : "neutral"} detail={status?.summary.redis_url ?? "-"} />
|
||||
<MetricCard label="i18n:govoplan-ops.warnings.1430f976" value={warningCount + errorCount} tone={errorCount ? "danger" : warningCount ? "warning" : "good"} detail="i18n:govoplan-ops.current_health_checks.7830bccf" />
|
||||
<MetricCard label="Runtime nodes" value={status?.summary.runtime_node_count ?? 0} tone={status?.runtime_cluster.available ? "good" : "danger"} detail={runtimeNodeMetricDetail(status)} />
|
||||
<MetricCard label="Recovery" value={status?.summary.recovery_required_count ?? 0} tone={status?.summary.recovery_required_count ? "danger" : "good"} detail="Operations requiring recovery attention" />
|
||||
<MetricCard label="Provider bindings" value={status?.governance.summary.configured_external_provider_count ?? 0} tone={status?.governance.summary.provider_attention_count ? "warning" : "good"} detail={`${status?.governance.summary.provider_attention_count ?? 0} requiring attention`} />
|
||||
</div>
|
||||
|
||||
<div className="dashboard-grid">
|
||||
@@ -96,6 +134,20 @@ export default function OpsPage({ settings, auth }: {settings: ApiSettings;auth:
|
||||
<CheckList checks={checks} />
|
||||
</Card>
|
||||
|
||||
<Card title="Runtime cluster">
|
||||
<RuntimeNodeTable
|
||||
nodes={status?.runtime_cluster.nodes ?? []}
|
||||
canManage={canRunChecks}
|
||||
busyNodeId={nodeActionId}
|
||||
onDrain={setDrainTarget}
|
||||
onCancelDrain={(node) => void cancelDrain(node)}
|
||||
/>
|
||||
</Card>
|
||||
|
||||
<Card title="Recovery evidence">
|
||||
<RecoveryTable operations={status?.runtime_cluster.recovery.operations ?? []} />
|
||||
</Card>
|
||||
|
||||
<Card title="i18n:govoplan-ops.governance_inventory.835d8e57">
|
||||
<GovernanceTable modules={status?.governance.modules ?? []} />
|
||||
</Card>
|
||||
@@ -109,11 +161,159 @@ export default function OpsPage({ settings, auth }: {settings: ApiSettings;auth:
|
||||
</Card>
|
||||
</div>
|
||||
</LoadingFrame>
|
||||
<ConfirmDialog
|
||||
open={Boolean(drainTarget)}
|
||||
title="Drain runtime node"
|
||||
message={`Stop routing new work to ${drainTarget?.node_id ?? "this node"}. In-flight work is allowed to finish.`}
|
||||
confirmLabel="Drain node"
|
||||
busy={Boolean(nodeActionId)}
|
||||
onCancel={() => setDrainTarget(null)}
|
||||
onConfirm={() => void confirmDrain()}
|
||||
/>
|
||||
</div>
|
||||
</PageScrollViewport>);
|
||||
|
||||
}
|
||||
|
||||
function RuntimeNodeTable({
|
||||
nodes,
|
||||
canManage,
|
||||
busyNodeId,
|
||||
onDrain,
|
||||
onCancelDrain
|
||||
}: {
|
||||
nodes: OpsRuntimeNode[];
|
||||
canManage: boolean;
|
||||
busyNodeId: string;
|
||||
onDrain: (node: OpsRuntimeNode) => void;
|
||||
onCancelDrain: (node: OpsRuntimeNode) => void;
|
||||
}) {
|
||||
const columns: DataGridColumn<OpsRuntimeNode>[] = [
|
||||
{
|
||||
id: "node",
|
||||
header: "Node",
|
||||
width: "minmax(220px, 1fr)",
|
||||
minWidth: 200,
|
||||
resizable: true,
|
||||
sortable: true,
|
||||
filterable: true,
|
||||
value: (node) => `${node.node_id} ${node.role}`,
|
||||
render: (node) => <div><strong>{node.node_id}</strong><span className="muted block">{node.role} · {node.software_version}</span></div>
|
||||
},
|
||||
{
|
||||
id: "state",
|
||||
header: "State",
|
||||
width: 150,
|
||||
sortable: true,
|
||||
filterable: true,
|
||||
value: (node) => node.stale ? "stale" : node.state,
|
||||
render: (node) => <StatusBadge status={node.stale ? "error" : stateTone(node.state)} label={node.stale ? "stale" : node.state} />
|
||||
},
|
||||
{
|
||||
id: "heartbeat",
|
||||
header: "Last heartbeat",
|
||||
width: "minmax(210px, .8fr)",
|
||||
minWidth: 190,
|
||||
resizable: true,
|
||||
sortable: true,
|
||||
value: (node) => node.last_heartbeat_at,
|
||||
render: (node) => new Date(node.last_heartbeat_at).toLocaleString()
|
||||
},
|
||||
{
|
||||
id: "queues",
|
||||
header: "Queues",
|
||||
width: "minmax(180px, .7fr)",
|
||||
minWidth: 160,
|
||||
resizable: true,
|
||||
filterable: true,
|
||||
value: (node) => node.queues.join(" "),
|
||||
render: (node) => node.queues.join(", ") || "-"
|
||||
},
|
||||
{
|
||||
id: "actions",
|
||||
header: "",
|
||||
width: 56,
|
||||
value: (node) => node.state,
|
||||
render: (node) => {
|
||||
if (!canManage || node.state === "stopped") return null;
|
||||
const busy = busyNodeId === node.node_id;
|
||||
return node.state === "draining" ? (
|
||||
<Button
|
||||
variant="ghost"
|
||||
title="Cancel drain"
|
||||
aria-label={`Cancel drain for ${node.node_id}`}
|
||||
disabled={busy}
|
||||
onClick={() => onCancelDrain(node)}
|
||||
><PlayCircle size={16} /></Button>
|
||||
) : (
|
||||
<Button
|
||||
variant="ghost"
|
||||
title="Drain node"
|
||||
aria-label={`Drain ${node.node_id}`}
|
||||
disabled={busy || node.stale}
|
||||
onClick={() => onDrain(node)}
|
||||
><PauseCircle size={16} /></Button>
|
||||
);
|
||||
}
|
||||
}
|
||||
];
|
||||
return <DataGrid id="ops-runtime-nodes" rows={nodes} columns={columns} getRowKey={(node) => `${node.node_id}:${node.incarnation}`} emptyText="No runtime nodes reported." />;
|
||||
}
|
||||
|
||||
function RecoveryTable({ operations }: { operations: OpsRecoveryOperation[] }) {
|
||||
const columns: DataGridColumn<OpsRecoveryOperation>[] = [
|
||||
{
|
||||
id: "operation",
|
||||
header: "Operation",
|
||||
width: "minmax(240px, 1fr)",
|
||||
minWidth: 220,
|
||||
resizable: true,
|
||||
sortable: true,
|
||||
filterable: true,
|
||||
value: (operation) => `${operation.module_id} ${operation.operation_type}`,
|
||||
render: (operation) => <div><strong>{operation.operation_type}</strong><span className="muted block">{operation.module_id} · {operation.mode}</span></div>
|
||||
},
|
||||
{
|
||||
id: "resource",
|
||||
header: "Resource",
|
||||
width: "minmax(180px, .8fr)",
|
||||
minWidth: 160,
|
||||
resizable: true,
|
||||
filterable: true,
|
||||
value: (operation) => `${operation.resource_type ?? ""} ${operation.resource_id ?? ""}`,
|
||||
render: (operation) => operation.resource_type ? `${operation.resource_type}: ${operation.resource_id ?? "-"}` : "-"
|
||||
},
|
||||
{
|
||||
id: "status",
|
||||
header: "Status",
|
||||
width: 170,
|
||||
sortable: true,
|
||||
filterable: true,
|
||||
value: (operation) => operation.status,
|
||||
render: (operation) => <StatusBadge status={recoveryTone(operation.status)} label={operation.status.replaceAll("_", " ")} />
|
||||
},
|
||||
{
|
||||
id: "evidence",
|
||||
header: "Evidence",
|
||||
width: 150,
|
||||
sortable: true,
|
||||
value: (operation) => operation.checkpoint_count,
|
||||
render: (operation) => `${operation.checkpoint_count} checkpoint(s)`
|
||||
},
|
||||
{
|
||||
id: "updated",
|
||||
header: "Updated",
|
||||
width: "minmax(210px, .8fr)",
|
||||
minWidth: 190,
|
||||
resizable: true,
|
||||
sortable: true,
|
||||
value: (operation) => operation.updated_at,
|
||||
render: (operation) => new Date(operation.updated_at).toLocaleString()
|
||||
}
|
||||
];
|
||||
return <DataGrid id="ops-recovery-operations" rows={operations} columns={columns} getRowKey={(operation) => operation.id} emptyText="No recovery operations recorded." />;
|
||||
}
|
||||
|
||||
function GovernanceTable({ modules }: { modules: OpsGovernanceModule[] }) {
|
||||
const columns: DataGridColumn<OpsGovernanceModule>[] = [
|
||||
{
|
||||
@@ -145,6 +345,44 @@ function GovernanceTable({ modules }: { modules: OpsGovernanceModule[] }) {
|
||||
value: (module) => `${module.capability_count} ${module.policy_count}`,
|
||||
render: (module) => `${module.capability_count} capabilities · ${module.policy_count} policies`
|
||||
},
|
||||
{
|
||||
id: "architecture",
|
||||
header: "Architecture",
|
||||
width: "minmax(210px, .9fr)",
|
||||
minWidth: 190,
|
||||
resizable: true,
|
||||
sortable: true,
|
||||
filterable: true,
|
||||
value: (module) => module.architecture
|
||||
? `${module.architecture.layer} ${module.architecture.kind} ${module.architecture.maturity}`
|
||||
: "undeclared",
|
||||
render: (module) => module.architecture ? (
|
||||
<div>
|
||||
<strong>{module.architecture.maturity}</strong>
|
||||
<span className="muted block">{module.architecture.layer} · {module.architecture.kind}</span>
|
||||
{module.architecture.known_limits.length ? <span className="muted block">{module.architecture.known_limits.length} known limit(s)</span> : null}
|
||||
</div>
|
||||
) : <span className="muted">staged declaration pending</span>
|
||||
},
|
||||
{
|
||||
id: "providers",
|
||||
header: "External providers",
|
||||
width: "minmax(230px, 1fr)",
|
||||
minWidth: 210,
|
||||
resizable: true,
|
||||
filterable: true,
|
||||
value: (module) => `${module.external_provider_count} ${module.external_providers.map((provider) => `${provider.id} ${provider.maturity} ${provider.authority_modes.join(" ")} ${provider.runtime_state?.health ?? "unobserved"} ${provider.runtime_state?.freshness ?? ""}`).join(" ")}`,
|
||||
render: (module) => module.external_provider_count ? (
|
||||
<div>
|
||||
<strong>{module.external_provider_count} declared</strong>
|
||||
{module.external_providers.map((provider) => (
|
||||
<span className="muted block" key={provider.id}>
|
||||
{provider.label}: {provider.maturity} · {provider.runtime_state ? `${provider.runtime_state.health}/${provider.runtime_state.freshness} · ${provider.runtime_state.bindings?.length ?? 0} binding(s)` : "unobserved"}
|
||||
</span>
|
||||
))}
|
||||
</div>
|
||||
) : <span className="muted">none declared</span>
|
||||
},
|
||||
{
|
||||
id: "controls",
|
||||
header: "i18n:govoplan-ops.controls.0cdb80fb",
|
||||
@@ -227,6 +465,21 @@ function stateTone(state: string): string {
|
||||
return "inactive";
|
||||
}
|
||||
|
||||
function recoveryTone(state: string): string {
|
||||
if (["succeeded", "recovered"].includes(state)) return "success";
|
||||
if (["failed", "recovery_required"].includes(state)) return "error";
|
||||
if (["running", "recovering", "prepared"].includes(state)) return "warning";
|
||||
return "inactive";
|
||||
}
|
||||
|
||||
function runtimeNodeMetricDetail(status: OpsStatus | null): string {
|
||||
const cluster = status?.runtime_cluster;
|
||||
if (!cluster?.available) return cluster?.detail ?? "Runtime directory unavailable";
|
||||
const active = cluster.active ?? { api: 0, worker: 0 };
|
||||
const expected = cluster.expected ?? { api: 0, worker: 0 };
|
||||
return `${active.api}/${expected.api} API · ${active.worker}/${expected.worker} workers`;
|
||||
}
|
||||
|
||||
function workerMetricDetail(status: OpsStatus | null): string {
|
||||
if (!status?.summary.celery_enabled) return "i18n:govoplan-ops.celery_worker_setting.323d7737";
|
||||
const metrics = status.summary.worker_metrics;
|
||||
|
||||
Reference in New Issue
Block a user