From 9bb2c808a6118b03304156794c458a81ff48d202 Mon Sep 17 00:00:00 2001 From: Albrecht Degering Date: Tue, 4 Aug 2026 05:20:47 +0200 Subject: [PATCH] Enforce declared platform interface inventory --- .gitea/workflows/module-matrix.yml | 4 +- docs/INTERFACE_SURFACE_INVENTORY.md | 2 +- docs/PLATFORM_CONTROL_PLANE.md | 72 +++- tests/test_platform_interface_inventory.py | 108 +++++ tools/checks/check-focused.sh | 1 + tools/checks/check-release-integration.sh | 5 + .../endpoint-surface-declarations.json | 7 + tools/inventory/extract-webui-structure.mjs | 237 +++++++++- .../inventory/platform-interface-inventory.py | 407 +++++++++++++++++- 9 files changed, 818 insertions(+), 25 deletions(-) diff --git a/.gitea/workflows/module-matrix.yml b/.gitea/workflows/module-matrix.yml index da50128..1c3694d 100644 --- a/.gitea/workflows/module-matrix.yml +++ b/.gitea/workflows/module-matrix.yml @@ -55,9 +55,9 @@ jobs: - name: Install WebUI release dependencies with test scripts working-directory: govoplan run: bash tools/release/install-webui-release-dependencies.sh ../govoplan-core/webui - - name: Validate platform endpoint surface declarations + - name: Validate platform interface and endpoint declarations working-directory: govoplan - run: .venv/bin/python tools/inventory/platform-interface-inventory.py --strict-endpoints + run: .venv/bin/python tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints - name: Validate Search against PostgreSQL working-directory: govoplan env: diff --git a/docs/INTERFACE_SURFACE_INVENTORY.md b/docs/INTERFACE_SURFACE_INVENTORY.md index e7cd360..1525cd9 100644 --- a/docs/INTERFACE_SURFACE_INVENTORY.md +++ b/docs/INTERFACE_SURFACE_INVENTORY.md @@ -345,7 +345,7 @@ make every module symmetrical. | 11 | Configured-system pattern help | Role/config-aware workflow, reference, pattern, and system topics are projected by Docs; shared route, field, blocker, and action links resolve to configured Docs or the hosted fallback | Stable configured-system guidance without feature-to-Docs imports | Docs #15 | Docs suite, shared component tests, Campaign review tests, 46 module permutations, full-product bundle budget | Complete 2026-08-03 (Docs `abe2f78`; Core `b823a22`; Campaign `d635f3a`) | | 12 | Admin/configuration family | Core host/settings/credential/retention contracts, shared primitives, module lifecycle, Files, Mail, Policy, Access, Admin, Tenancy, Views, and Organizations are integrated and verified | Continue the same consequence/provenance grammar only through bounded module-owned migrations | Core #225 and module children | Per-surface state/accessibility/consequence evidence | Core #225 complete `fa32cca`; Access `1409dbf`; Files `d8ae506`; Mail `7844d9c`; Policy `f964ed7`; Admin `d428f33`; Tenancy `e76fe16`; Views `c125f33`; Organizations `97acfcb` | | 13 | Remaining module surfaces | 33 bounded module-owned issues cover every WebUI contributor not already tracked by Campaign #74 or completed Docs #15 | Per-module audit and migration, ordered by user task and consequence rather than a bulk rewrite | Issues linked in the direct-route and composed-surface sections | Module-focused tests, manifest shapes, contextual Docs, and applicable definition-of-done gates | Complete: prior 28 recorded commits plus Workflow #15, Search #4, Reporting #8, Projects #2 and Portal #2 verified 2026-08-03 | -| 14 | Manifest/runtime alignment | Authenticated canonical routes align; public signed-token and compatibility routes are explicit exceptions | Reconcile stable declarations with source and runtime continuously | [Meta #25](https://git.add-ideas.de/GovOPlaN/govoplan/issues/25) | Generated static/runtime comparison and duplicate/stale declaration CI | Independent control-plane hardening follow-up; not a rollout closure blocker | +| 14 | Manifest/runtime alignment | Authenticated canonical routes align; public signed-token and compatibility routes are explicit exceptions | Stable declarations reconcile with source and any effective runtime module combination | [Meta #25](https://git.add-ideas.de/GovOPlaN/govoplan/issues/25) | Strict duplicate/stale/undeclared declaration CI, per-module digests, and authorized read-only runtime inventory | Complete 2026-08-04 | Workflow remains outside this rollout matrix because it has its own runtime and editor workstream, not because it is postponed. Focused views can be specified, diff --git a/docs/PLATFORM_CONTROL_PLANE.md b/docs/PLATFORM_CONTROL_PLANE.md index ea0509d..46e5829 100644 --- a/docs/PLATFORM_CONTROL_PLANE.md +++ b/docs/PLATFORM_CONTROL_PLANE.md @@ -25,6 +25,7 @@ releases, module boundaries, migrations, and security controls. | Labels and translations | Generated translation catalogs plus source usage | | Fields and help coverage | Shared form components plus generated TypeScript AST inventory | | API use by the WebUI | Typed API clients plus generated static reference inventory | +| Stable platform interface IDs | Typed manifest/WebUI declarations plus line-independent source anchors for low-level controls | | Effective configuration | Owning module data plus Policy provenance | Runtime introspection is authoritative for an installed system. Static source @@ -45,7 +46,9 @@ The command writes: - `audit-reports/platform-inventory/platform-interface-inventory.json` - `audit-reports/platform-inventory/platform-interface-inventory.md` -Use `--strict` for the combined translation and endpoint audit. Use +Use `--strict` for the combined translation, endpoint, and declaration audit. +Use `--strict-declarations` for duplicate/stale/undeclared interface checks +without making existing translation coverage a release blocker. Use `--strict-endpoints` in the endpoint-surface CI gate so unrelated translation catalog work cannot disable route classification enforcement. Both strict modes require every backend endpoint without a statically visible WebUI path to have @@ -75,6 +78,16 @@ It combines: 2. TypeScript AST extraction of fields, label attributes, visible text, translations, frontend routes, navigation, capabilities, and API references 3. Python AST extraction of FastAPI route decorators and router prefixes +4. normalized runtime declarations from every loaded `ModuleManifest` + +The declaration set covers routes, navigation, View surfaces, fields, actions, +help references, translations, admin/settings sections, widgets, search +objects, permissions, provided interfaces, and backend capabilities. Typed +module contributions keep their declared IDs. Shared controls may declare +`interfaceId` and `helpTopicId`; otherwise the extractor assigns a deterministic +source anchor based on repository, file, component context, control type, and +semantic label rather than a line number. The JSON records which identity +source was used. The JSON includes exact repository, file, and line evidence. A missing-help entry is a review candidate because dynamic parent components may supply help. @@ -82,11 +95,41 @@ A backend route without a static frontend reference is also a review candidate: public APIs, workers, callbacks, health checks, connectors, and dynamic URL assembly are valid explanations. -The module matrix enforces endpoint declarations with `--strict-endpoints`. +The module matrix enforces endpoint and interface declarations with +`--strict-endpoints --strict-declarations`. Combined `--strict` additionally fails when used translation keys are absent from generated locale catalogs. Help-text findings remain review candidates rather than a release gate because dynamic parent components can supply help. +## Runtime Comparison + +Core exposes a sanitized read-only catalog at +`GET /api/v1/platform/interface-catalog`. Access requires +`admin:module:read` or `system:settings:read`. Tenant module entitlements are +applied before serialization, so the response describes only the effective +installed combination. It contains IDs, paths, authorization metadata, +versions, counts, and canonical digests; it excludes factories, callbacks, +credentials, and mutable runtime state. + +Capture and compare a running installation: + +```bash +curl --fail --silent \ + -H "Authorization: Bearer $GOVOPLAN_ACCESS_TOKEN" \ + "$GOVOPLAN_URL/api/v1/platform/interface-catalog" \ + > /tmp/govoplan-runtime-interface.json + +./.venv/bin/python tools/inventory/platform-interface-inventory.py \ + --runtime-snapshot /tmp/govoplan-runtime-interface.json \ + --strict-declarations \ + --strict-endpoints +``` + +The comparison accepts any installed subset. Every module present in the +runtime response must have the same contract version, module version, and +declaration digest as the static release inventory. Unknown, duplicate, or +mismatched runtime modules fail strict declaration mode. + ## Admin Information Architecture The Admin host uses a tree because system, tenant, group, user, and module @@ -139,13 +182,20 @@ Custom code, new routes, arbitrary SQL, and executable workflow nodes remain release artifacts. Modeling them as ordinary configuration would create an unreviewed code-execution and migration channel. -## Next Enforcement Slices +## Enforced Contract -1. Require every WebUI module route and admin/settings contribution to have - matching manifest metadata or a reviewed exception. -2. Add stable field IDs and optional help-topic IDs to shared field components. -3. Classify each statically unreferenced backend endpoint by consumer type. -4. Compare a running installation's OpenAPI and module registry against the - release inventory. -5. Publish the sanitized installed-system structure through Ops/Docs for - authorized administrators. +1. Public WebUI routes and View surfaces must reconcile with runtime manifest + metadata; stale runtime routes and source-only public surfaces fail CI. +2. Duplicate stable IDs fail CI. Shared controls support explicit field/action + and help-topic identities; fallback anchors remain visible review evidence. +3. Every statically unreferenced backend endpoint has an exact reviewed + consumer classification, and stale classifications fail CI. +4. Runtime module combinations can be compared exactly with static release + evidence through versioned per-module digests. +5. Runtime introspection is authorized, tenant-filtered, and read-only. It is + safe for Ops/Docs projection but is not a generic configuration or code + mutation channel. + +Generated JSON and Markdown remain build/audit artifacts. Do not hand-edit or +use them as a backlog; change the owning manifest, typed WebUI contribution, +translation/help declaration, or exact endpoint classification instead. diff --git a/tests/test_platform_interface_inventory.py b/tests/test_platform_interface_inventory.py index 269f313..12ab793 100644 --- a/tests/test_platform_interface_inventory.py +++ b/tests/test_platform_interface_inventory.py @@ -199,6 +199,114 @@ def read_item(item_id: str): }, ) + def test_source_declarations_normalize_stable_control_and_contribution_ids( + self, + ) -> None: + webui = { + "fields": [ + { + "repository": "govoplan-example", + "file": "webui/src/Example.tsx", + "line": 12, + "column": 3, + "id": "govoplan-example.field.example.name.abc123", + "idSource": "source_anchor", + "explicitId": None, + "context": "Example", + "helpId": "govoplan-example.field.example.name.abc123.help", + "helpDynamic": False, + } + ], + "actions": [], + "contributions": [ + { + "repository": "govoplan-example", + "file": "webui/src/module.ts", + "line": 20, + "column": 5, + "kind": "frontend_route", + "id": "/examples/:exampleId", + "path": "/examples/:exampleId", + } + ], + "translationCatalog": {"en": {}, "de": {}}, + } + manifests = [{"repository": "govoplan-example", "id": "examples"}] + + declarations = inventory._source_interface_declarations(webui, manifests) + keys = {item["key"] for item in declarations} + + self.assertIn("field:examples.field.example.name.abc123", keys) + self.assertIn( + "help:examples.field.example.name.abc123.help", + keys, + ) + self.assertIn( + "frontend_route:examples.route.examples.exampleid", + keys, + ) + + def test_declaration_health_rejects_duplicate_and_undeclared_source_ids( + self, + ) -> None: + declaration = { + "key": "frontend_route:example.route.unlisted", + "id": "example.route.unlisted", + "module_id": "example", + "kind": "frontend_route", + "origin": "webui_contribution", + } + manifests = [ + { + "id": "example", + "repository": "govoplan-example", + "interface_catalog": {"declarations": []}, + } + ] + + health = inventory._declaration_health( + [declaration, dict(declaration)], + manifests, + ) + + self.assertEqual(1, len(health["duplicate_ids"])) + self.assertEqual(1, len(health["undeclared_source_surfaces"])) + + def test_runtime_snapshot_comparison_accepts_an_installed_subset(self) -> None: + manifests = [ + { + "id": "one", + "interface_catalog": { + "contract_version": "1", + "module_id": "one", + "module_version": "1.0.0", + "digest": "sha256:one", + }, + }, + { + "id": "two", + "interface_catalog": { + "contract_version": "1", + "module_id": "two", + "module_version": "1.0.0", + "digest": "sha256:two", + }, + }, + ] + snapshot = { + "contract_version": "1", + "modules": [dict(manifests[1]["interface_catalog"])], + } + + comparison = inventory._compare_runtime_snapshot(snapshot, manifests) + + self.assertEqual(["two"], comparison["matched_modules"]) + self.assertEqual([], comparison["mismatches"]) + + snapshot["modules"][0]["digest"] = "sha256:changed" + comparison = inventory._compare_runtime_snapshot(snapshot, manifests) + self.assertEqual("digest_mismatch", comparison["mismatches"][0]["reason"]) + if __name__ == "__main__": unittest.main() diff --git a/tools/checks/check-focused.sh b/tools/checks/check-focused.sh index 26f4d8d..381add2 100644 --- a/tools/checks/check-focused.sh +++ b/tools/checks/check-focused.sh @@ -40,6 +40,7 @@ GOVOPLAN_CORE_ROOT="$ROOT" PYTHON="$PYTHON" CHECK_TESTCLIENT_DEPRECATIONS=1 bash PYTHONDONTWRITEBYTECODE=1 "$PYTHON" "$META_ROOT/tools/checks/check-manifest-shapes.py" --require-architecture cd "$META_ROOT" +"$PYTHON" tools/inventory/platform-interface-inventory.py --strict-declarations --strict-endpoints "$PYTHON" tools/repo/sync-module-package-workflows.py --check "$PYTHON" tools/release/generate-developer-meta-package.py --check "$PYTHON" -m unittest tests.test_module_package_workflows tests.test_package_registry_release diff --git a/tools/checks/check-release-integration.sh b/tools/checks/check-release-integration.sh index 24d441f..a242059 100644 --- a/tools/checks/check-release-integration.sh +++ b/tools/checks/check-release-integration.sh @@ -182,6 +182,11 @@ run_step "Validate installed module manifests and registry" "$PYTHON" "$META_ROOT/tools/checks/release_integration.py" artifacts \ --requirements "$META_ROOT/requirements-release.txt" +run_step "Validate platform interface and endpoint declarations" +"$PYTHON" "$META_ROOT/tools/inventory/platform-interface-inventory.py" \ + --strict-declarations \ + --strict-endpoints + run_step "Generate release dependency provenance" "$PYTHON" "$META_ROOT/tools/release/generate-release-sbom.py" \ --python "$PYTHON" \ diff --git a/tools/inventory/endpoint-surface-declarations.json b/tools/inventory/endpoint-surface-declarations.json index 76d3ee4..c76ac9e 100644 --- a/tools/inventory/endpoint-surface-declarations.json +++ b/tools/inventory/endpoint-surface-declarations.json @@ -567,6 +567,13 @@ "rationale": "The shared ownership UI uses a dynamic transfer-action path that the static string scan cannot resolve.", "repository": "govoplan-core" }, + { + "category": "intentionally_headless", + "method": "GET", + "path": "/platform/interface-catalog", + "rationale": "Authorized module and system administrators consume the read-only control-plane inventory directly or through Ops/Docs projections.", + "repository": "govoplan-core" + }, { "category": "compatibility", "method": "GET", diff --git a/tools/inventory/extract-webui-structure.mjs b/tools/inventory/extract-webui-structure.mjs index 9f50e67..933828c 100644 --- a/tools/inventory/extract-webui-structure.mjs +++ b/tools/inventory/extract-webui-structure.mjs @@ -2,6 +2,7 @@ import fs from "node:fs"; import path from "node:path"; +import { createHash } from "node:crypto"; import { pathToFileURL } from "node:url"; const [metaRootArgument] = process.argv.slice(2); @@ -60,9 +61,19 @@ const helpAttributes = new Set([ "helperText", "helpText" ]); +const actionComponentPattern = /(?:Action|Button|Link)$/; +const contributionTypes = new Map([ + ["AdminSectionsUiCapability", "admin_section"], + ["DashboardWidgetsUiCapability", "widget"], + ["OrganizationFunctionActionsUiCapability", "action"], + ["SearchContextsUiCapability", "search_object"], + ["SettingsSectionsUiCapability", "setting"], + ["WizardDirectoriesUiCapability", "workflow_directory"] +]); const result = { fields: [], + actions: [], labels: [], visibleText: [], translationCatalog: {}, @@ -71,7 +82,8 @@ const result = { routes: [], navigation: [], frontendApiReferences: [], - uiCapabilities: [] + uiCapabilities: [], + contributions: [] }; for (const repository of repositoryCatalog.repositories) { @@ -115,6 +127,7 @@ function inspectSource(repository, sourceRoot, sourcePath) { sourcePath.endsWith(".tsx") ? ts.ScriptKind.TSX : ts.ScriptKind.TS ); const relativeFile = path.relative(path.join(workspaceRoot, repository), sourcePath); + const identityCounters = new Map(); function location(node) { const position = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)); @@ -178,6 +191,7 @@ function inspectSource(repository, sourceRoot, sourcePath) { const isField = fieldComponents.has(component) || (fieldComponentPattern.test(component) && component !== "FormField"); + inspectAction(node, component, attributes, locate); if (!isField) return; const parentFormField = nearestFormField(node); @@ -191,8 +205,25 @@ function inspectSource(repository, sourceRoot, sourcePath) { null; const help = firstAttribute(attributes, helpAttributes) ?? firstAttribute(parentAttributes, helpAttributes); + const hasHelp = hasAnyAttribute(attributes, helpAttributes) || + hasAnyAttribute(parentAttributes, helpAttributes); + const explicitId = firstAttribute( + attributes, + new Set(["interfaceId", "data-interface-id", "id", "name", "field"]) + ); + const context = nearestNamedContext(node); + const stableId = sourceIdentity( + "field", + node, + component, + explicitId ?? label ?? attributes.get("placeholder") ?? "field" + ); result.fields.push({ ...locate(node), + id: stableId, + explicitId, + idSource: explicitId === null ? "source_anchor" : "explicit", + context, component, name: attributes.get("name") ?? @@ -202,8 +233,102 @@ function inspectSource(repository, sourceRoot, sourcePath) { label, placeholder: attributes.get("placeholder") ?? null, help: help ?? null, - helpCandidate: help === null + helpId: hasHelp ? `${stableId}.help` : null, + helpDynamic: hasHelp && help === null, + helpCandidate: !hasHelp }); + + } + + function inspectAction(node, component, attributes, locate) { + const lowerComponent = component.toLowerCase(); + const inputType = attributes.get("type")?.toLowerCase(); + const isAction = lowerComponent === "button" || + lowerComponent === "a" || + actionComponentPattern.test(component) || + (lowerComponent === "input" && ["button", "reset", "submit"].includes(inputType)); + if (!isAction) return; + + const label = attributes.get("aria-label") ?? + attributes.get("title") ?? + attributes.get("label") ?? + staticJsxChildText(node) ?? + null; + const explicitId = firstAttribute( + attributes, + new Set(["interfaceId", "data-interface-id", "id", "name"]) + ); + const context = nearestNamedContext(node); + result.actions.push({ + ...locate(node), + id: sourceIdentity( + "action", + node, + component, + explicitId ?? label ?? "action" + ), + explicitId, + idSource: explicitId === null ? "source_anchor" : "explicit", + context, + component, + label + }); + } + + function nearestNamedContext(node) { + let current = node.parent; + while (current) { + if (ts.isFunctionDeclaration(current) && current.name) { + return current.name.text; + } + if (ts.isMethodDeclaration(current) && current.name) { + return current.name.getText(sourceFile); + } + if ( + (ts.isArrowFunction(current) || ts.isFunctionExpression(current)) && + ts.isVariableDeclaration(current.parent) && + ts.isIdentifier(current.parent.name) + ) { + return current.parent.name.text; + } + if ( + (ts.isArrowFunction(current) || ts.isFunctionExpression(current)) && + ts.isPropertyAssignment(current.parent) + ) { + return propertyNameText(current.parent.name) ?? "anonymous"; + } + current = current.parent; + } + return path.basename(relativeFile).replace(/\.[^.]+$/, ""); + } + + function sourceIdentity(kind, node, component, semantic) { + const context = nearestNamedContext(node); + const normalizedSemantic = slug(String(semantic)); + const counterKey = `${kind}:${context}:${component}:${normalizedSemantic}`; + const occurrence = (identityCounters.get(counterKey) ?? 0) + 1; + identityCounters.set(counterKey, occurrence); + const anchor = [relativeFile, context, component, normalizedSemantic, occurrence].join(":"); + const digest = createHash("sha256").update(anchor).digest("hex").slice(0, 12); + return `${repository}.${kind}.${slug(context)}.${normalizedSemantic}.${digest}`; + } + + function staticJsxChildText(node) { + if (!ts.isJsxOpeningElement(node) || !ts.isJsxElement(node.parent)) return null; + const values = []; + for (const child of node.parent.children) { + if (ts.isJsxText(child)) { + const value = child.getText(sourceFile).replace(/\s+/g, " ").trim(); + if (value) values.push(value); + } else if ( + ts.isJsxExpression(child) && + child.expression && + ts.isStringLiteralLike(child.expression) + ) { + values.push(child.expression.text); + } + } + return values.length > 0 ? values.join(" ") : null; } function nearestFormField(node) { @@ -275,22 +400,103 @@ function inspectSource(repository, sourceRoot, sourcePath) { function inspectProperty(node, locate) { const propertyName = propertyNameText(node.name); + if (isDirectUiCapabilityProperty(node) && typeof propertyName === "string") { + result.uiCapabilities.push({ ...locate(node), name: propertyName }); + result.contributions.push({ + ...locate(node), + kind: "ui_capability", + id: propertyName + }); + } const value = staticExpressionText(node.initializer); if (value === null) return; if (propertyName === "path" && value.startsWith("/") && !value.includes("/api/")) { result.routes.push({ ...locate(node), path: value }); + const routeCollection = nearestCollectionProperty(node); + if (routeCollection === "routes" || routeCollection === "publicRoutes") { + result.contributions.push({ + ...locate(node), + kind: routeCollection === "publicRoutes" ? "public_route" : "frontend_route", + id: value, + path: value + }); + } } if (propertyName === "to" && value.startsWith("/")) { result.navigation.push({ ...locate(node), path: value }); + if (nearestCollectionProperty(node) === "navItems") { + result.contributions.push({ + ...locate(node), + kind: "navigation", + id: value, + path: value + }); + } } - if ( - ancestorPropertyName(node, "uiCapabilities") && - typeof propertyName === "string" - ) { - result.uiCapabilities.push({ ...locate(node), name: propertyName }); + if (propertyName === "id") { + const contributionKind = contributionKindFor(node); + if (contributionKind !== null) { + result.contributions.push({ + ...locate(node), + kind: contributionKind, + id: value + }); + } } } + function contributionKindFor(node) { + const collection = nearestCollectionProperty(node); + if (collection === "viewSurfaces") return "view_surface"; + if (collection === "widgets") return "widget"; + if (collection === "contexts") return "search_object"; + if (collection === "actions") return "action"; + if (collection === "directories") return "workflow_directory"; + if (collection !== "sections") return null; + const variableType = nearestVariableType(node); + for (const [typeName, kind] of contributionTypes) { + if (variableType.includes(typeName)) return kind; + } + return ancestorPropertyName(node, "admin.sections") + ? "admin_section" + : ancestorPropertyName(node, "settings.sections") + ? "setting" + : "section"; + } + + function nearestCollectionProperty(node) { + let current = node.parent; + while (current) { + if ( + ts.isArrayLiteralExpression(current) && + ts.isPropertyAssignment(current.parent) + ) { + return propertyNameText(current.parent.name); + } + current = current.parent; + } + return null; + } + + function nearestVariableType(node) { + let current = node.parent; + while (current) { + if (ts.isVariableDeclaration(current)) { + return current.type?.getText(sourceFile) ?? ""; + } + current = current.parent; + } + return ""; + } + + function isDirectUiCapabilityProperty(node) { + const parent = node.parent; + const uiCapabilities = parent?.parent; + return ts.isObjectLiteralExpression(parent) && + ts.isPropertyAssignment(uiCapabilities) && + propertyNameText(uiCapabilities.name) === "uiCapabilities"; + } + function inspectTranslationProperty(node, locate) { const key = propertyNameText(node.name); if (!key?.startsWith("i18n:")) return; @@ -343,6 +549,23 @@ function firstAttribute(attributes, names) { return null; } +function hasAnyAttribute(attributes, names) { + for (const name of names) { + if (attributes.has(name)) return true; + } + return false; +} + +function slug(value) { + const normalized = value + .toLowerCase() + .replace(/^i18n:/, "") + .replace(/\.[a-f0-9]{8}$/, "") + .replace(/[^a-z0-9]+/g, ".") + .replace(/^\.+|\.+$/g, ""); + return (normalized || "unnamed").slice(0, 72); +} + function propertyNameText(name) { if ( ts.isIdentifier(name) || diff --git a/tools/inventory/platform-interface-inventory.py b/tools/inventory/platform-interface-inventory.py index e47cc01..47a28d9 100644 --- a/tools/inventory/platform-interface-inventory.py +++ b/tools/inventory/platform-interface-inventory.py @@ -50,6 +50,23 @@ def main() -> int: action="store_true", help="Fail only on incomplete or stale endpoint-surface declarations.", ) + parser.add_argument( + "--strict-declarations", + action="store_true", + help=( + "Fail on duplicate stable IDs, WebUI surfaces absent from runtime " + "metadata, or stale runtime route declarations." + ), + ) + parser.add_argument( + "--runtime-snapshot", + type=Path, + help=( + "Compare a saved /api/v1/platform/interface-catalog response with " + "the static manifest inventory. Any installed module combination " + "is accepted; every module present in the snapshot must match." + ), + ) parser.add_argument( "--endpoint-declarations", type=Path, @@ -71,6 +88,11 @@ def main() -> int: backend_endpoints=backend_endpoints, manifests=manifests, endpoint_declarations=endpoint_declarations, + runtime_snapshot=( + _load_runtime_snapshot(args.runtime_snapshot.resolve()) + if args.runtime_snapshot is not None + else None + ), ) output_dir = args.output_dir.resolve() @@ -85,11 +107,12 @@ def main() -> int: print(f"Platform inventory JSON: {json_path}") print(f"Platform inventory summary: {markdown_path}") - if args.strict or args.strict_endpoints: + if args.strict or args.strict_endpoints or args.strict_declarations: failures = _strict_failures( inventory, check_translations=args.strict, - check_endpoints=True, + check_endpoints=args.strict or args.strict_endpoints, + check_declarations=args.strict or args.strict_declarations, ) if failures: print( @@ -105,6 +128,7 @@ def _strict_failures( *, check_translations: bool, check_endpoints: bool, + check_declarations: bool = False, ) -> list[str]: failures: list[str] = [] if ( @@ -122,6 +146,32 @@ def _strict_failures( f"{len(inventory['api']['stale_endpoint_declarations'])} " "endpoint declarations do not match a backend endpoint" ) + declaration_health = inventory.get("declaration_health", {}) + if check_declarations and declaration_health.get("duplicate_ids"): + failures.append( + f"{len(declaration_health['duplicate_ids'])} platform interface " + "IDs are declared more than once" + ) + if check_declarations and declaration_health.get("undeclared_source_surfaces"): + failures.append( + f"{len(declaration_health['undeclared_source_surfaces'])} public " + "WebUI surfaces have no runtime manifest declaration" + ) + if check_declarations and declaration_health.get("stale_runtime_routes"): + failures.append( + f"{len(declaration_health['stale_runtime_routes'])} runtime route " + "declarations have no WebUI implementation" + ) + runtime_comparison = inventory.get("runtime_comparison") + if ( + check_declarations + and runtime_comparison is not None + and runtime_comparison.get("mismatches") + ): + failures.append( + f"{len(runtime_comparison['mismatches'])} runtime catalog entries " + "do not match the release inventory" + ) return failures @@ -265,6 +315,10 @@ def _extract_manifests( if (workspace_root / repository["path"] / "src").is_dir() ] sys.path[:0] = [str(path) for path in source_roots] + from govoplan_core.core.platform_interfaces import ( # noqa: PLC0415 + manifest_interface_catalog, + ) + manifests: list[dict[str, Any]] = [] for repository in catalog["repositories"]: source_root = workspace_root / repository["path"] / "src" @@ -299,15 +353,24 @@ def _extract_manifests( } for permission in manifest.permissions ], + "interface_catalog": manifest_interface_catalog(manifest), "frontend": ( { "package": frontend.package_name, "routes": [ _plain_value(route) for route in frontend.routes ], + "public_routes": [ + _plain_value(route) + for route in frontend.public_routes + ], "nav_items": [ _plain_value(item) for item in frontend.nav_items ], + "settings_routes": [ + _plain_value(route) + for route in frontend.settings_routes + ], "view_surfaces": [ _plain_value(surface) for surface in frontend.view_surfaces @@ -327,6 +390,7 @@ def _assemble_inventory( backend_endpoints: list[dict[str, Any]], manifests: list[dict[str, Any]], endpoint_declarations: dict[tuple[str, str, str], dict[str, Any]], + runtime_snapshot: dict[str, Any] | None = None, ) -> dict[str, Any]: frontend_refs = webui["frontendApiReferences"] frontend_paths = { @@ -396,25 +460,40 @@ def _assemble_inventory( ] fields = webui["fields"] help_candidates = [field for field in fields if field["helpCandidate"]] + dynamic_help = [field for field in fields if field.get("helpDynamic")] + source_declarations = _source_interface_declarations(webui, manifests) + declaration_health = _declaration_health(source_declarations, manifests) + runtime_comparison = ( + _compare_runtime_snapshot(runtime_snapshot, manifests) + if runtime_snapshot is not None + else None + ) return { - "schema_version": 1, + "schema_version": 2, "scope": { "source": "local GovOPlaN repository catalog", "limitations": [ "Static extraction cannot resolve runtime-computed labels, routes, or API paths.", "A backend endpoint without a static WebUI reference may intentionally serve public clients, workers, connectors, or external integrations.", "A field marked as a help candidate may receive contextual help from a surrounding dynamic component.", + "Low-level field and action IDs use line-independent source anchors unless an explicit interfaceId, DOM id, or name is declared.", ], }, "modules": manifests, + "interface_declarations": source_declarations, + "declaration_health": declaration_health, + "runtime_comparison": runtime_comparison, "ui": { "fields": fields, + "actions": webui.get("actions", []), "labels": webui["labels"], "visible_text": webui["visibleText"], "routes": webui["routes"], "navigation": webui["navigation"], "ui_capabilities": webui["uiCapabilities"], "help_candidates": help_candidates, + "dynamic_help": dynamic_help, + "contributions": webui.get("contributions", []), }, "translations": { "catalog": catalogs, @@ -439,6 +518,16 @@ def _assemble_inventory( "ui_fields": len(fields), "ui_fields_with_static_help": len(fields) - len(help_candidates), "help_review_candidates": len(help_candidates), + "dynamic_help_references": len(dynamic_help), + "ui_actions": len(webui.get("actions", [])), + "interface_declarations": len(source_declarations), + "duplicate_interface_ids": len(declaration_health["duplicate_ids"]), + "undeclared_source_surfaces": len( + declaration_health["undeclared_source_surfaces"] + ), + "stale_runtime_routes": len( + declaration_health["stale_runtime_routes"] + ), "label_attributes": len(webui["labels"]), "visible_text_nodes": len(webui["visibleText"]), "frontend_routes": len(webui["routes"]), @@ -451,6 +540,299 @@ def _assemble_inventory( } +def _source_interface_declarations( + webui: dict[str, Any], + manifests: list[dict[str, Any]], +) -> list[dict[str, Any]]: + module_by_repository = { + str(manifest["repository"]): str(manifest["id"]) + for manifest in manifests + } + declarations: list[dict[str, Any]] = [] + + def module_id(repository: str) -> str: + if repository in module_by_repository: + return module_by_repository[repository] + if repository == "govoplan-core": + return "core" + return repository.removeprefix("govoplan-").replace("-", "_") + + def source_evidence(item: dict[str, Any]) -> dict[str, Any]: + return { + key: item[key] + for key in ("repository", "file", "line", "column") + if key in item + } + + for kind, items in ( + ("field", webui["fields"]), + ("action", webui.get("actions", [])), + ): + for item in items: + repository = str(item["repository"]) + owner = module_id(repository) + raw_id = str(item["id"]) + stable_id = ( + f"{owner}.{raw_id[len(repository) + 1:]}" + if raw_id.startswith(f"{repository}.") + else _namespaced_interface_id(owner, kind, raw_id) + ) + declarations.append( + { + "key": f"{kind}:{stable_id}", + "id": stable_id, + "module_id": owner, + "kind": kind, + "origin": "webui_source", + "id_source": item.get("idSource", "source_anchor"), + "explicit_id": item.get("explicitId"), + "context": item.get("context"), + **source_evidence(item), + } + ) + if kind == "field" and item.get("helpId"): + help_raw_id = str(item["helpId"]) + help_id = ( + f"{owner}.{help_raw_id[len(repository) + 1:]}" + if help_raw_id.startswith(f"{repository}.") + else _namespaced_interface_id(owner, "help", help_raw_id) + ) + declarations.append( + { + "key": f"help:{help_id}", + "id": help_id, + "module_id": owner, + "kind": "help", + "origin": "webui_source", + "field_id": stable_id, + "dynamic": bool(item.get("helpDynamic")), + **source_evidence(item), + } + ) + + for item in webui.get("contributions", []): + repository = str(item["repository"]) + owner = module_id(repository) + kind = str(item["kind"]) + raw_id = str(item["id"]) + path = item.get("path") + if kind == "frontend_route" and isinstance(path, str): + stable_id = f"{owner}.route.{_surface_slug(path)}" + elif kind == "public_route" and isinstance(path, str): + stable_id = f"{owner}.public.{_surface_slug(path)}" + elif kind == "navigation" and isinstance(path, str): + stable_id = f"{owner}.nav.{_surface_slug(path)}" + else: + stable_id = _namespaced_interface_id(owner, kind, raw_id) + declarations.append( + { + "key": f"{kind}:{stable_id}", + "id": stable_id, + "module_id": owner, + "kind": kind, + "origin": "webui_contribution", + "declared_value": raw_id, + **({"path": path} if isinstance(path, str) else {}), + **source_evidence(item), + } + ) + + translation_entries: dict[tuple[str, str], dict[str, Any]] = {} + for locale, entries in webui["translationCatalog"].items(): + for key, item in entries.items(): + repository = str(item["repository"]) + owner = module_id(repository) + declaration_key = (owner, str(key)) + declaration = translation_entries.setdefault( + declaration_key, + { + "key": f"translation:{key}", + "id": key, + "module_id": owner, + "kind": "translation", + "origin": "translation_catalog", + "locales": [], + **source_evidence(item), + }, + ) + declaration["locales"].append(locale) + declarations.extend(translation_entries.values()) + + return sorted( + declarations, + key=lambda item: ( + item["module_id"], + item["kind"], + item["id"], + item.get("file", ""), + item.get("line", 0), + ), + ) + + +def _declaration_health( + source_declarations: list[dict[str, Any]], + manifests: list[dict[str, Any]], +) -> dict[str, Any]: + grouped: dict[tuple[str, str, str], list[dict[str, Any]]] = {} + for declaration in source_declarations: + key = ( + str(declaration["module_id"]), + str(declaration["kind"]), + str(declaration["id"]), + ) + grouped.setdefault(key, []).append(declaration) + duplicate_ids = [ + { + "module_id": key[0], + "kind": key[1], + "id": key[2], + "evidence": values, + } + for key, values in sorted(grouped.items()) + if len(values) > 1 + ] + + comparable_kinds = { + "frontend_route", + "navigation", + "public_route", + "view_surface", + } + source_surfaces = { + (str(item["module_id"]), str(item["kind"]), str(item["id"])): item + for item in source_declarations + if item["origin"] == "webui_contribution" + and item["kind"] in comparable_kinds + } + runtime_surfaces: dict[tuple[str, str, str], dict[str, Any]] = {} + for manifest in manifests: + catalog = manifest["interface_catalog"] + for declaration in catalog["declarations"]: + if declaration["kind"] not in comparable_kinds: + continue + key = ( + str(manifest["id"]), + str(declaration["kind"]), + str(declaration["id"]), + ) + runtime_surfaces[key] = { + "repository": manifest["repository"], + **declaration, + } + surface_id = declaration.get("metadata", {}).get("surface_id") + if ( + declaration["kind"] + in {"frontend_route", "navigation", "settings_route"} + and isinstance(surface_id, str) + and surface_id + ): + runtime_surfaces[ + (str(manifest["id"]), "view_surface", surface_id) + ] = { + "repository": manifest["repository"], + "key": f"view_surface:{surface_id}", + "id": surface_id, + "module_id": manifest["id"], + "kind": "view_surface", + "metadata": { + "derived_from": declaration["kind"], + "path": declaration.get("path"), + }, + } + + undeclared_source_surfaces = [ + source_surfaces[key] + for key in sorted(source_surfaces.keys() - runtime_surfaces.keys()) + ] + route_kinds = {"frontend_route", "public_route"} + stale_runtime_routes = [ + runtime_surfaces[key] + for key in sorted(runtime_surfaces.keys() - source_surfaces.keys()) + if key[1] in route_kinds + ] + return { + "duplicate_ids": duplicate_ids, + "undeclared_source_surfaces": undeclared_source_surfaces, + "stale_runtime_routes": stale_runtime_routes, + "source_declaration_count": len(source_declarations), + "runtime_declaration_count": sum( + len(manifest["interface_catalog"]["declarations"]) + for manifest in manifests + ), + } + + +def _compare_runtime_snapshot( + snapshot: dict[str, Any], + manifests: list[dict[str, Any]], +) -> dict[str, Any]: + static_by_module = { + str(manifest["id"]): manifest["interface_catalog"] + for manifest in manifests + } + modules = snapshot.get("modules") + if not isinstance(modules, list): + raise ValueError("Runtime interface snapshot must contain a modules list.") + mismatches: list[dict[str, Any]] = [] + seen: set[str] = set() + matched: list[str] = [] + for item in modules: + if not isinstance(item, dict) or not isinstance(item.get("module_id"), str): + raise ValueError("Runtime interface snapshot has an invalid module entry.") + module_id = item["module_id"] + if module_id in seen: + mismatches.append({"module_id": module_id, "reason": "duplicate_module"}) + continue + seen.add(module_id) + expected = static_by_module.get(module_id) + if expected is None: + mismatches.append({"module_id": module_id, "reason": "unknown_module"}) + continue + for field in ("contract_version", "module_version", "digest"): + if item.get(field) != expected.get(field): + mismatches.append( + { + "module_id": module_id, + "reason": f"{field}_mismatch", + "expected": expected.get(field), + "actual": item.get(field), + } + ) + if not any( + mismatch["module_id"] == module_id for mismatch in mismatches + ): + matched.append(module_id) + return { + "contract_version": snapshot.get("contract_version"), + "matched_modules": sorted(matched), + "mismatches": mismatches, + } + + +def _load_runtime_snapshot(path: Path) -> dict[str, Any]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise ValueError(f"Runtime interface snapshot does not exist: {path}") from exc + except json.JSONDecodeError as exc: + raise ValueError(f"Runtime interface snapshot is invalid JSON: {exc}") from exc + if not isinstance(payload, dict): + raise ValueError("Runtime interface snapshot must be a JSON object.") + return payload + + +def _namespaced_interface_id(module_id: str, kind: str, value: str) -> str: + if value.startswith(f"{module_id}."): + return value + return f"{module_id}.{kind}.{_surface_slug(value)}" + + +def _surface_slug(value: str) -> str: + normalized = re.sub(r"[^a-z0-9]+", ".", value.strip().lower()).strip(".") + return normalized or "root" + + def _render_markdown(inventory: dict[str, Any]) -> str: summary = inventory["summary"] missing = inventory["translation_health"]["missing_catalog_entries"] @@ -472,8 +854,14 @@ def _render_markdown(inventory: dict[str, Any]) -> str: "", f"- Modules: {summary['modules']}", f"- UI fields: {summary['ui_fields']}", + f"- UI actions: {summary['ui_actions']}", f"- Fields with statically associated help: {summary['ui_fields_with_static_help']}", + f"- Fields with dynamic help references: {summary['dynamic_help_references']}", f"- Help review candidates: {summary['help_review_candidates']}", + f"- Stable interface declarations: {summary['interface_declarations']}", + f"- Duplicate interface IDs: {summary['duplicate_interface_ids']}", + f"- WebUI surfaces missing runtime declarations: {summary['undeclared_source_surfaces']}", + f"- Runtime routes missing WebUI implementations: {summary['stale_runtime_routes']}", f"- Label attributes: {summary['label_attributes']}", f"- Frontend routes: {summary['frontend_routes']}", f"- Backend endpoints: {summary['backend_endpoints']}", @@ -527,13 +915,24 @@ def _render_markdown(inventory: dict[str, Any]) -> str: ) lines.extend( [ + "", + "## Declaration Reconciliation", + "", + "Routes, navigation, View surfaces, fields, actions, help references,", + "translations, settings, widgets, search objects, and backend", + "capabilities use normalized stable IDs. CI rejects duplicate IDs,", + "WebUI public surfaces absent from runtime metadata, and runtime routes", + "without a WebUI implementation.", + "", "", "## Interpretation", "", "Use the JSON artifact for exact file and line evidence. Missing help is", "a triage list, not an automatic defect. Endpoint coverage requires an", "owner classification before enforcement. Runtime-computed structures", - "need explicit manifest metadata to become canonically visible.", + "need explicit manifest or typed PlatformWebModule metadata to become", + "canonically visible. The generated files are release evidence, not an", + "editable source of platform behavior.", "", ] )