diff --git a/docs/INTERFACE_SURFACE_INVENTORY.md b/docs/INTERFACE_SURFACE_INVENTORY.md index d7adc4f..06b0765 100644 --- a/docs/INTERFACE_SURFACE_INVENTORY.md +++ b/docs/INTERFACE_SURFACE_INVENTORY.md @@ -9,6 +9,12 @@ The applicable design contract is ## Snapshot And Method +The source-derived inventory command is documented in +[`PLATFORM_CONTROL_PLANE.md`](PLATFORM_CONTROL_PLANE.md). It produces +machine-readable field, label, translation, route, API-reference, and module +manifest evidence. This hand-maintained document remains the reviewed product +interpretation and rollout ledger; generated evidence does not replace it. + Snapshot refreshed: 2026-07-22. Evidence was read from tracked Git `HEAD` in the local GovOPlaN checkouts: @@ -215,10 +221,11 @@ backend-only modules may remain intentionally headless. | 13 | Remaining direct routes | Routes are contributed; most are unreviewed | Per-module bounded audit and migration plan | New module issues derived from this inventory | Applicable definition-of-done gates | P2 after Campaign, not a bulk rewrite | | 14 | Manifest/runtime alignment | Several executable routes are absent from manifest metadata | Declared alignment or explicit validated exception | Core contract issue to create | Automated manifest/module route check and configured Docs verification | Discovery follow-up | -Workflow/user-story implementation is postponed. It is not on the critical path -for this rollout matrix. Focused views can be specified, manually selected, and -tested through core composition contracts; a later workflow step may become one -activation source without changing the proven surface patterns. +Workflow remains outside this rollout matrix because it has its own runtime and +editor workstream, not because it is postponed. Focused views can be specified, +manually selected, and tested through core composition contracts today. +Workflow steps may activate those views through the same contract without +changing the proven surface patterns. ## Inventory Maintenance diff --git a/docs/PLATFORM_CONTROL_PLANE.md b/docs/PLATFORM_CONTROL_PLANE.md new file mode 100644 index 0000000..5afff27 --- /dev/null +++ b/docs/PLATFORM_CONTROL_PLANE.md @@ -0,0 +1,126 @@ +# Platform Control Plane And Self-Description + +## Objective + +GovOPlaN should be able to describe its installed structure without becoming a +self-modifying application. The platform model is a declarative control plane: +module manifests, UI contributions, schemas, policy provenance, runtime +capabilities, and generated source evidence describe what can be configured. +Ordinary administrators edit validated data through those contracts; they do +not edit Python, TypeScript, routes, or database code from the product UI. + +This distinction provides the requested overview while preserving reviewable +releases, module boundaries, migrations, and security controls. + +## Canonical Sources + +| Concern | Canonical source | +| --- | --- | +| Installed modules and dependency graph | Runtime `ModuleManifest` registry | +| Backend routes | Registered FastAPI application; Python AST is build-time evidence | +| Frontend routes and navigation | `PlatformWebModule` contributions | +| View-filterable regions | Versioned `viewSurfaces` declarations | +| Admin sections and module settings | `admin.sections`, including `moduleId`, `kind`, scope group, permission guards, and surface ID | +| User settings | `settings.sections` and core settings schemas | +| Labels and translations | Generated translation catalogs plus source usage | +| Fields and help coverage | Shared form components plus generated TypeScript AST inventory | +| API use by the WebUI | Typed API clients plus generated static reference inventory | +| Effective configuration | Owning module data plus Policy provenance | + +Runtime introspection is authoritative for an installed system. Static source +inventory is authoritative evidence for a checkout or release candidate. The +two should be compared in CI and by Ops, not conflated. + +## Generated Inventory + +Run: + +```bash +cd /mnt/DATA/git/govoplan +./.venv/bin/python tools/inventory/platform-interface-inventory.py +``` + +The command writes: + +- `audit-reports/platform-inventory/platform-interface-inventory.json` +- `audit-reports/platform-inventory/platform-interface-inventory.md` + +It combines: + +1. loaded module manifests +2. TypeScript AST extraction of fields, label attributes, visible text, + translations, frontend routes, navigation, capabilities, and API references +3. Python AST extraction of FastAPI route decorators and router prefixes + +The JSON includes exact repository, file, and line evidence. A missing-help +entry is a review candidate because dynamic parent components may supply help. +A backend route without a static frontend reference is also a review candidate: +public APIs, workers, callbacks, health checks, connectors, and dynamic URL +assembly are valid explanations. + +`--strict` currently enforces only translation-catalog completeness. Endpoint +and help classifications need narrow reviewed baselines before they can become +release gates. + +## Admin Information Architecture + +The Admin host uses a tree because system, tenant, group, user, and module +settings form a hierarchy rather than one flat list. Every contributed section +can identify: + +- its owning `moduleId` +- whether it is `management` or `settings` +- its system/tenant/group/user scope group +- an optional future `parentId` +- permission and View visibility requirements + +Existing panels remain their own render owners. The tree only changes discovery +and grouping. A later embedded-settings contract may add named slots inside an +owning page; it must not allow one module to import another module's private +component. + +## Navigation And Workflow + +The intended maximum visible navigation stack is: + +1. global shell context +2. one task/object navigation surface +3. one workflow stage surface when a workflow is active + +Workflow instance pages should reuse the Campaign stage language: clear stage +state, optional/skipped/blocked semantics, partial progress, and a stable current +step. Workflow definition pages remain graph editors. Views may activate a +focused workflow view that suppresses unrelated shell and module surfaces while +retaining an explicit way out. + +Nested module submenus should not be added merely because a data hierarchy +exists. Prefer a tree inside configuration/directory surfaces, tabs for sibling +views, and the workflow stage rail for ordered work. + +## Safe Meta-Configuration + +The platform can eventually render many configuration editors from versioned +JSON Schema and UI Schema supplied by modules. Generated editors remain bounded +by: + +- explicit typed schemas and migrations +- module-owned validation and preview +- Policy locks and provenance +- permission and View filtering +- preflight, consequence, and rollback information +- auditable apply operations + +Custom code, new routes, arbitrary SQL, and executable workflow nodes remain +release artifacts. Modeling them as ordinary configuration would create an +unreviewed code-execution and migration channel. + +## Next Enforcement Slices + +1. Require every WebUI module route and admin/settings contribution to have + matching manifest metadata or a reviewed exception. +2. Add stable field IDs and optional help-topic IDs to shared field components. +3. Classify each statically unreferenced backend endpoint by consumer type. +4. Compare a running installation's OpenAPI and module registry against the + release inventory. +5. Publish the sanitized installed-system structure through Ops/Docs for + authorized administrators. diff --git a/docs/REPOSITORY_INDEX.md b/docs/REPOSITORY_INDEX.md index f780b5e..9f41f2e 100644 --- a/docs/REPOSITORY_INDEX.md +++ b/docs/REPOSITORY_INDEX.md @@ -34,6 +34,7 @@ Generated from `repositories.json`. Use that JSON file as the machine-readable s | `govoplan-dms` | `domain` | `../govoplan-dms` | [govoplan-dms](https://git.add-ideas.de/GovOPlaN/govoplan-dms) | | `govoplan-dist-lists` | `domain` | `../govoplan-dist-lists` | [govoplan-dist-lists](https://git.add-ideas.de/GovOPlaN/govoplan-dist-lists) | | `govoplan-docs` | `platform` | `../govoplan-docs` | [govoplan-docs](https://git.add-ideas.de/GovOPlaN/govoplan-docs) | +| `govoplan-encryption` | `platform` | `../govoplan-encryption` | [govoplan-encryption](https://git.add-ideas.de/GovOPlaN/govoplan-encryption) | | `govoplan-erp` | `domain` | `../govoplan-erp` | [govoplan-erp](https://git.add-ideas.de/GovOPlaN/govoplan-erp) | | `govoplan-evaluation` | `domain` | `../govoplan-evaluation` | [govoplan-evaluation](https://git.add-ideas.de/GovOPlaN/govoplan-evaluation) | | `govoplan-facilities` | `domain` | `../govoplan-facilities` | [govoplan-facilities](https://git.add-ideas.de/GovOPlaN/govoplan-facilities) | diff --git a/docs/gitea-labels.json b/docs/gitea-labels.json index 775a288..52d10b7 100644 --- a/docs/gitea-labels.json +++ b/docs/gitea-labels.json @@ -191,6 +191,12 @@ "description": "GovOPlaN Erp module behavior or integration.", "exclusive": false }, + { + "name": "module/encryption", + "color": "b60205", + "description": "GovOPlaN Encryption key custody, cryptographic policy, and E2EE integration.", + "exclusive": false + }, { "name": "module/evaluation", "color": "bfdadc", diff --git a/repositories.json b/repositories.json index 39b56a9..f212a0f 100644 --- a/repositories.json +++ b/repositories.json @@ -27,6 +27,7 @@ {"name": "govoplan-dms", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dms.git", "path": "govoplan-dms"}, {"name": "govoplan-dist-lists", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-dist-lists.git", "path": "govoplan-dist-lists"}, {"name": "govoplan-docs", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-docs.git", "path": "govoplan-docs"}, + {"name": "govoplan-encryption", "category": "module", "subtype": "platform", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-encryption.git", "path": "govoplan-encryption"}, {"name": "govoplan-erp", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-erp.git", "path": "govoplan-erp"}, {"name": "govoplan-evaluation", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-evaluation.git", "path": "govoplan-evaluation"}, {"name": "govoplan-facilities", "category": "module", "subtype": "domain", "remote": "git@git.add-ideas.de:GovOPlaN/govoplan-facilities.git", "path": "govoplan-facilities"}, diff --git a/requirements-dev.txt b/requirements-dev.txt index 8b69139..9b839f5 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -32,6 +32,7 @@ -e ../govoplan-notifications -e ../govoplan-evaluation -e ../govoplan-docs +-e ../govoplan-encryption -e ../govoplan-ops httpx==0.28.1 httpx2>=2.5,<3 diff --git a/tests/test_platform_interface_inventory.py b/tests/test_platform_interface_inventory.py new file mode 100644 index 0000000..a498573 --- /dev/null +++ b/tests/test_platform_interface_inventory.py @@ -0,0 +1,66 @@ +from __future__ import annotations + +import ast +import importlib.util +from pathlib import Path +import unittest + + +SCRIPT = ( + Path(__file__).resolve().parents[1] + / "tools" + / "inventory" + / "platform-interface-inventory.py" +) +SPEC = importlib.util.spec_from_file_location("platform_interface_inventory", SCRIPT) +assert SPEC is not None and SPEC.loader is not None +inventory = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(inventory) + + +class PlatformInterfaceInventoryTests(unittest.TestCase): + def test_canonical_api_path_normalizes_versions_and_parameters(self) -> None: + self.assertEqual( + inventory.canonical_api_path( + "http://localhost/api/v1/campaigns/${campaignId}?limit=10" + ), + "/campaigns/{}", + ) + self.assertEqual( + inventory.canonical_api_path("/api/v2/campaigns/{campaign_id}"), + "/campaigns/{}", + ) + + def test_fastapi_route_scanner_includes_router_prefix(self) -> None: + tree = ast.parse( + """ +from fastapi import APIRouter +router = APIRouter(prefix="/api/v1/items") + +@router.get("/{item_id}") +def read_item(item_id: str): + return item_id +""" + ) + prefixes = inventory._router_prefixes(tree) + function = next( + node for node in ast.walk(tree) if isinstance(node, ast.FunctionDef) + ) + + route = inventory._endpoint_from_decorator( + function.decorator_list[0], + prefixes=prefixes, + ) + + self.assertEqual( + route, + { + "method": "GET", + "path": "/api/v1/items/{item_id}", + "router": "router", + }, + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/tools/inventory/extract-webui-structure.mjs b/tools/inventory/extract-webui-structure.mjs new file mode 100644 index 0000000..99f3c4a --- /dev/null +++ b/tools/inventory/extract-webui-structure.mjs @@ -0,0 +1,366 @@ +#!/usr/bin/env node + +import fs from "node:fs"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; + +const [metaRootArgument] = process.argv.slice(2); +if (!metaRootArgument) { + throw new Error("Usage: extract-webui-structure.mjs META_ROOT"); +} + +const metaRoot = path.resolve(metaRootArgument); +const repositoryCatalog = JSON.parse( + fs.readFileSync(path.join(metaRoot, "repositories.json"), "utf8") +); +const workspaceRoot = path.resolve(repositoryCatalog.default_parent); +const typescriptPath = path.join( + workspaceRoot, + "govoplan-core", + "webui", + "node_modules", + "typescript", + "lib", + "typescript.js" +); +const ts = await import(pathToFileURL(typescriptPath).href); + +const fieldComponents = new Set([ + "input", + "select", + "textarea", + "Checkbox", + "DateTimeField", + "EmailAddressInput", + "ReferenceSelect", + "SearchableSelect", + "ToggleSwitch" +]); +const fieldComponentPattern = + /(?:Input|Field|Select|Picker|Toggle|Checkbox|Radio|Editor)$/; +const labelAttributes = new Set([ + "aria-label", + "description", + "help", + "helperText", + "helpText", + "label", + "placeholder", + "title" +]); +const helpAttributes = new Set([ + "description", + "help", + "helperText", + "helpText" +]); + +const result = { + fields: [], + labels: [], + visibleText: [], + translationCatalog: {}, + translationUsages: [], + dynamicTranslationUsages: [], + routes: [], + navigation: [], + frontendApiReferences: [], + uiCapabilities: [] +}; + +for (const repository of repositoryCatalog.repositories) { + const sourceRoot = path.join(workspaceRoot, repository.path, "webui", "src"); + if (!fs.existsSync(sourceRoot)) continue; + for (const sourcePath of sourceFiles(sourceRoot)) { + inspectSource(repository.name, sourceRoot, sourcePath); + } +} + +process.stdout.write(`${JSON.stringify(result)}\n`); + +function sourceFiles(root) { + const files = []; + const pending = [root]; + while (pending.length > 0) { + const current = pending.pop(); + for (const entry of fs.readdirSync(current, { withFileTypes: true })) { + if ( + entry.name === "node_modules" || + entry.name.startsWith(".") || + entry.name === "dist" + ) { + continue; + } + const candidate = path.join(current, entry.name); + if (entry.isDirectory()) pending.push(candidate); + else if (/\.(?:ts|tsx)$/.test(entry.name)) files.push(candidate); + } + } + return files.sort(); +} + +function inspectSource(repository, sourceRoot, sourcePath) { + const sourceText = fs.readFileSync(sourcePath, "utf8"); + const sourceFile = ts.createSourceFile( + sourcePath, + sourceText, + ts.ScriptTarget.Latest, + true, + sourcePath.endsWith(".tsx") ? ts.ScriptKind.TSX : ts.ScriptKind.TS + ); + const relativeFile = path.relative(path.join(workspaceRoot, repository), sourcePath); + + function location(node) { + const position = sourceFile.getLineAndCharacterOfPosition(node.getStart(sourceFile)); + return { + repository, + file: relativeFile, + line: position.line + 1, + column: position.character + 1 + }; + } + + function visit(node) { + if (ts.isJsxOpeningElement(node) || ts.isJsxSelfClosingElement(node)) { + inspectJsxOpening(node, location); + } + if (ts.isJsxText(node)) { + const value = node.getText(sourceFile).replace(/\s+/g, " ").trim(); + if (value) { + result.visibleText.push({ + ...location(node), + value, + translationKey: value.startsWith("i18n:") ? value : null + }); + } + } + if (ts.isStringLiteralLike(node)) { + inspectString(node.text, node, location); + } else if (ts.isTemplateExpression(node)) { + inspectString(templateText(node), node, location); + } + if (ts.isPropertyAssignment(node)) { + inspectProperty(node, location); + inspectTranslationProperty(node, location); + } + ts.forEachChild(node, visit); + } + + visit(sourceFile); + + function inspectJsxOpening(node, locate) { + const component = node.tagName.getText(sourceFile); + const attributes = new Map(); + for (const attribute of node.attributes.properties) { + if (!ts.isJsxAttribute(attribute)) continue; + attributes.set( + attribute.name.getText(sourceFile), + jsxAttributeValue(attribute) + ); + } + for (const [attribute, value] of attributes) { + if (!labelAttributes.has(attribute) || value === null) continue; + result.labels.push({ + ...locate(node), + component, + attribute, + value, + translationKey: value.startsWith("i18n:") ? value : null + }); + } + + const isField = + fieldComponents.has(component) || + (fieldComponentPattern.test(component) && component !== "FormField"); + if (!isField) return; + + const parentFormField = nearestFormField(node); + const parentAttributes = parentFormField + ? jsxAttributes(parentFormField) + : new Map(); + const label = + attributes.get("label") ?? + attributes.get("aria-label") ?? + parentAttributes.get("label") ?? + null; + const help = firstAttribute(attributes, helpAttributes) ?? + firstAttribute(parentAttributes, helpAttributes); + result.fields.push({ + ...locate(node), + component, + name: + attributes.get("name") ?? + attributes.get("id") ?? + attributes.get("field") ?? + null, + label, + placeholder: attributes.get("placeholder") ?? null, + help: help ?? null, + helpCandidate: help === null + }); + } + + function nearestFormField(node) { + let current = node.parent; + while (current) { + if ( + ts.isJsxElement(current) && + current.openingElement.tagName.getText(sourceFile) === "FormField" + ) { + return current.openingElement; + } + if ( + ts.isJsxOpeningElement(current) || + ts.isJsxSelfClosingElement(current) + ) { + const name = current.tagName.getText(sourceFile); + if (name !== "FormField") return null; + } + current = current.parent; + } + return null; + } + + function jsxAttributes(node) { + const mapped = new Map(); + for (const attribute of node.attributes.properties) { + if (!ts.isJsxAttribute(attribute)) continue; + mapped.set( + attribute.name.getText(sourceFile), + jsxAttributeValue(attribute) + ); + } + return mapped; + } + + function jsxAttributeValue(attribute) { + if (!attribute.initializer) return "true"; + if (ts.isStringLiteral(attribute.initializer)) return attribute.initializer.text; + if (!ts.isJsxExpression(attribute.initializer)) return null; + const expression = attribute.initializer.expression; + if (!expression) return null; + if (ts.isStringLiteralLike(expression)) return expression.text; + if (ts.isNoSubstitutionTemplateLiteral(expression)) return expression.text; + if (ts.isTemplateExpression(expression)) return templateText(expression); + return null; + } + + function inspectString(value, node, locate) { + if (value.startsWith("i18n:") && value.length > "i18n:".length) { + const target = value.includes("${}") || isStringPrefixCheck(node) + ? result.dynamicTranslationUsages + : result.translationUsages; + target.push({ ...locate(node), key: value }); + } + if (value.includes("/api/")) { + result.frontendApiReferences.push({ ...locate(node), path: value }); + } + } + + function isStringPrefixCheck(node) { + const call = node.parent; + if (!ts.isCallExpression(call) || !ts.isPropertyAccessExpression(call.expression)) { + return false; + } + return ["endsWith", "includes", "startsWith"].includes( + call.expression.name.text + ); + } + + function inspectProperty(node, locate) { + const propertyName = propertyNameText(node.name); + const value = staticExpressionText(node.initializer); + if (value === null) return; + if (propertyName === "path" && value.startsWith("/") && !value.includes("/api/")) { + result.routes.push({ ...locate(node), path: value }); + } + if (propertyName === "to" && value.startsWith("/")) { + result.navigation.push({ ...locate(node), path: value }); + } + if ( + ancestorPropertyName(node, "uiCapabilities") && + typeof propertyName === "string" + ) { + result.uiCapabilities.push({ ...locate(node), name: propertyName }); + } + } + + function inspectTranslationProperty(node, locate) { + const key = propertyNameText(node.name); + if (!key?.startsWith("i18n:")) return; + const value = staticExpressionText(node.initializer); + if (value === null) return; + const locale = nearestLocaleProperty(node); + if (!locale) return; + result.translationCatalog[locale] ??= {}; + result.translationCatalog[locale][key] = { + value, + ...locate(node) + }; + } + + function nearestLocaleProperty(node) { + let current = node.parent; + while (current) { + if ( + ts.isPropertyAssignment(current) && + (propertyNameText(current.name) === "en" || + propertyNameText(current.name) === "de") + ) { + return propertyNameText(current.name); + } + current = current.parent; + } + return null; + } + + function ancestorPropertyName(node, expected) { + let current = node.parent; + while (current) { + if ( + ts.isPropertyAssignment(current) && + propertyNameText(current.name) === expected + ) { + return true; + } + current = current.parent; + } + return false; + } +} + +function firstAttribute(attributes, names) { + for (const name of names) { + const value = attributes.get(name); + if (value !== undefined && value !== null) return value; + } + return null; +} + +function propertyNameText(name) { + if ( + ts.isIdentifier(name) || + ts.isStringLiteral(name) || + ts.isNumericLiteral(name) + ) { + return name.text; + } + return null; +} + +function staticExpressionText(node) { + if (ts.isStringLiteralLike(node) || ts.isNoSubstitutionTemplateLiteral(node)) { + return node.text; + } + if (ts.isTemplateExpression(node)) return templateText(node); + return null; +} + +function templateText(node) { + return ( + node.head.text + + node.templateSpans + .map((span) => "${}" + span.literal.text) + .join("") + ); +} diff --git a/tools/inventory/platform-interface-inventory.py b/tools/inventory/platform-interface-inventory.py new file mode 100644 index 0000000..1e607a5 --- /dev/null +++ b/tools/inventory/platform-interface-inventory.py @@ -0,0 +1,462 @@ +#!/usr/bin/env python3 +"""Extract a source-derived GovOPlaN UI, translation, module, and API inventory.""" + +from __future__ import annotations + +import argparse +import ast +from collections import Counter +from dataclasses import asdict, is_dataclass +import importlib +import json +from pathlib import Path +import re +import subprocess +import sys +from typing import Any + + +META_ROOT = Path(__file__).resolve().parents[2] +HTTP_METHODS = {"delete", "get", "head", "options", "patch", "post", "put"} +PATH_PARAMETER = re.compile(r"\$\{[^{}]*\}|\{[^{}]+\}") + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--output-dir", + type=Path, + default=META_ROOT / "audit-reports" / "platform-inventory", + ) + parser.add_argument( + "--strict", + action="store_true", + help="Fail when a used translation key is absent from a generated locale catalog.", + ) + args = parser.parse_args() + + catalog = json.loads( + (META_ROOT / "repositories.json").read_text(encoding="utf-8") + ) + workspace_root = Path(catalog["default_parent"]).resolve() + webui = _extract_webui() + backend_endpoints = _extract_backend_endpoints(catalog, workspace_root) + manifests = _extract_manifests(catalog, workspace_root) + inventory = _assemble_inventory( + webui=webui, + backend_endpoints=backend_endpoints, + manifests=manifests, + ) + + output_dir = args.output_dir.resolve() + output_dir.mkdir(parents=True, exist_ok=True) + json_path = output_dir / "platform-interface-inventory.json" + markdown_path = output_dir / "platform-interface-inventory.md" + json_path.write_text( + json.dumps(inventory, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + markdown_path.write_text(_render_markdown(inventory), encoding="utf-8") + print(f"Platform inventory JSON: {json_path}") + print(f"Platform inventory summary: {markdown_path}") + + if args.strict and inventory["translation_health"]["missing_catalog_entries"]: + print( + "Used translation keys are missing from generated catalogs.", + file=sys.stderr, + ) + return 1 + return 0 + + +def _extract_webui() -> dict[str, Any]: + helper = META_ROOT / "tools" / "inventory" / "extract-webui-structure.mjs" + completed = subprocess.run( + ["node", str(helper), str(META_ROOT)], + check=True, + capture_output=True, + text=True, + ) + return json.loads(completed.stdout) + + +def _extract_backend_endpoints( + catalog: dict[str, Any], + workspace_root: Path, +) -> list[dict[str, Any]]: + endpoints: list[dict[str, Any]] = [] + for repository in catalog["repositories"]: + repository_root = workspace_root / repository["path"] + source_root = repository_root / "src" + if not source_root.is_dir(): + continue + for source_path in sorted(source_root.rglob("*.py")): + try: + tree = ast.parse( + source_path.read_text(encoding="utf-8"), + filename=str(source_path), + ) + except (OSError, SyntaxError, UnicodeDecodeError): + continue + prefixes = _router_prefixes(tree) + for node in ast.walk(tree): + if not isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)): + continue + for decorator in node.decorator_list: + endpoint = _endpoint_from_decorator( + decorator, + prefixes=prefixes, + ) + if endpoint is None: + continue + endpoints.append( + { + "repository": repository["name"], + "file": str(source_path.relative_to(repository_root)), + "line": node.lineno, + "handler": node.name, + **endpoint, + } + ) + return sorted( + endpoints, + key=lambda item: ( + item["repository"], + item["path"], + item["method"], + item["file"], + item["line"], + ), + ) + + +def _router_prefixes(tree: ast.AST) -> dict[str, str]: + prefixes: dict[str, str] = {} + for node in ast.walk(tree): + if not isinstance(node, (ast.Assign, ast.AnnAssign)): + continue + value = node.value + if not isinstance(value, ast.Call): + continue + function_name = _call_name(value.func) + if function_name not in {"APIRouter", "fastapi.APIRouter"}: + continue + prefix = "" + for keyword in value.keywords: + if keyword.arg == "prefix": + prefix = _static_string(keyword.value) or "" + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + for target in targets: + if isinstance(target, ast.Name): + prefixes[target.id] = prefix + return prefixes + + +def _endpoint_from_decorator( + decorator: ast.expr, + *, + prefixes: dict[str, str], +) -> dict[str, str] | None: + if not isinstance(decorator, ast.Call) or not isinstance( + decorator.func, ast.Attribute + ): + return None + method = decorator.func.attr.lower() + if method not in HTTP_METHODS or not decorator.args: + return None + route = _static_string(decorator.args[0]) + if route is None: + return None + owner = decorator.func.value.id if isinstance(decorator.func.value, ast.Name) else "" + prefix = prefixes.get(owner, "") + return { + "method": method.upper(), + "path": _join_route(prefix, route), + "router": owner, + } + + +def _extract_manifests( + catalog: dict[str, Any], + workspace_root: Path, +) -> list[dict[str, Any]]: + source_roots = [ + workspace_root / repository["path"] / "src" + for repository in catalog["repositories"] + if (workspace_root / repository["path"] / "src").is_dir() + ] + sys.path[:0] = [str(path) for path in source_roots] + manifests: list[dict[str, Any]] = [] + for repository in catalog["repositories"]: + source_root = workspace_root / repository["path"] / "src" + if not source_root.is_dir(): + continue + for manifest_path in sorted(source_root.glob("*/backend/manifest.py")): + module_name = ".".join( + manifest_path.relative_to(source_root).with_suffix("").parts + ) + loaded = importlib.import_module(module_name) + manifest = loaded.get_manifest() + frontend = manifest.frontend + manifests.append( + { + "repository": repository["name"], + "id": manifest.id, + "name": manifest.name, + "version": manifest.version, + "dependencies": list(manifest.dependencies), + "optional_dependencies": list(manifest.optional_dependencies), + "required_capabilities": list(manifest.required_capabilities), + "provided_interfaces": [ + {"name": item.name, "version": item.version} + for item in manifest.provides_interfaces + ], + "runtime_capabilities": sorted(manifest.capability_factories), + "permissions": [ + { + "scope": permission.scope, + "label": permission.label, + "level": permission.level, + } + for permission in manifest.permissions + ], + "frontend": ( + { + "package": frontend.package_name, + "routes": [ + _plain_value(route) for route in frontend.routes + ], + "nav_items": [ + _plain_value(item) for item in frontend.nav_items + ], + "view_surfaces": [ + _plain_value(surface) + for surface in frontend.view_surfaces + ], + } + if frontend is not None + else None + ), + } + ) + return sorted(manifests, key=lambda item: item["id"]) + + +def _assemble_inventory( + *, + webui: dict[str, Any], + backend_endpoints: list[dict[str, Any]], + manifests: list[dict[str, Any]], +) -> dict[str, Any]: + frontend_refs = webui["frontendApiReferences"] + frontend_paths = { + canonical_api_path(reference["path"]) + for reference in frontend_refs + if canonical_api_path(reference["path"]) + } + unreferenced = [ + endpoint + for endpoint in backend_endpoints + if canonical_api_path(endpoint["path"]) not in frontend_paths + ] + usages = {item["key"] for item in webui["translationUsages"]} + catalogs = webui["translationCatalog"] + catalog_keys = { + locale: set(entries) + for locale, entries in catalogs.items() + } + expected_locales = sorted(catalog_keys) + missing_catalog_entries = [ + { + "key": key, + "missing_locales": [ + locale + for locale in expected_locales + if key not in catalog_keys[locale] + ], + } + for key in sorted(usages) + if any(key not in catalog_keys[locale] for locale in expected_locales) + ] + fields = webui["fields"] + help_candidates = [field for field in fields if field["helpCandidate"]] + return { + "schema_version": 1, + "scope": { + "source": "local GovOPlaN repository catalog", + "limitations": [ + "Static extraction cannot resolve runtime-computed labels, routes, or API paths.", + "A backend endpoint without a static WebUI reference may intentionally serve public clients, workers, connectors, or external integrations.", + "A field marked as a help candidate may receive contextual help from a surrounding dynamic component.", + ], + }, + "modules": manifests, + "ui": { + "fields": fields, + "labels": webui["labels"], + "visible_text": webui["visibleText"], + "routes": webui["routes"], + "navigation": webui["navigation"], + "ui_capabilities": webui["uiCapabilities"], + "help_candidates": help_candidates, + }, + "translations": { + "catalog": catalogs, + "usages": webui["translationUsages"], + "dynamic_usages": webui["dynamicTranslationUsages"], + }, + "translation_health": { + "locales": expected_locales, + "used_keys": len(usages), + "missing_catalog_entries": missing_catalog_entries, + }, + "api": { + "backend_endpoints": backend_endpoints, + "frontend_references": frontend_refs, + "unreferenced_by_static_webui_scan": unreferenced, + }, + "summary": { + "modules": len(manifests), + "ui_fields": len(fields), + "ui_fields_with_static_help": len(fields) - len(help_candidates), + "help_review_candidates": len(help_candidates), + "label_attributes": len(webui["labels"]), + "visible_text_nodes": len(webui["visibleText"]), + "frontend_routes": len(webui["routes"]), + "backend_endpoints": len(backend_endpoints), + "frontend_api_references": len(frontend_refs), + "backend_endpoints_without_static_webui_reference": len(unreferenced), + }, + } + + +def _render_markdown(inventory: dict[str, Any]) -> str: + summary = inventory["summary"] + missing = inventory["translation_health"]["missing_catalog_entries"] + help_by_repository = Counter( + item["repository"] for item in inventory["ui"]["help_candidates"] + ) + endpoint_by_repository = Counter( + item["repository"] + for item in inventory["api"]["unreferenced_by_static_webui_scan"] + ) + lines = [ + "# GovOPlaN Platform Interface Inventory", + "", + "Generated from runtime module manifests plus TypeScript and Python ASTs.", + "Counts are source evidence, not a claim that every surface is enabled or reachable.", + "", + "## Summary", + "", + f"- Modules: {summary['modules']}", + f"- UI fields: {summary['ui_fields']}", + f"- Fields with statically associated help: {summary['ui_fields_with_static_help']}", + f"- Help review candidates: {summary['help_review_candidates']}", + f"- Label attributes: {summary['label_attributes']}", + f"- Frontend routes: {summary['frontend_routes']}", + f"- Backend endpoints: {summary['backend_endpoints']}", + f"- Frontend API references: {summary['frontend_api_references']}", + ( + "- Backend endpoints without a static WebUI reference: " + f"{summary['backend_endpoints_without_static_webui_reference']}" + ), + f"- Used translation keys missing from a locale catalog: {len(missing)}", + "", + "## Help Review Candidates", + "", + "| Repository | Fields |", + "| --- | ---: |", + ] + lines.extend( + f"| `{repository}` | {count} |" + for repository, count in sorted(help_by_repository.items()) + ) + lines.extend( + [ + "", + "## Endpoints Without Static WebUI References", + "", + "These are review candidates. Public APIs, worker callbacks, connector", + "endpoints, health checks, and dynamically assembled paths are legitimate", + "reasons for appearing here.", + "", + "| Repository | Endpoints |", + "| --- | ---: |", + ] + ) + lines.extend( + f"| `{repository}` | {count} |" + for repository, count in sorted(endpoint_by_repository.items()) + ) + lines.extend( + [ + "", + "## Interpretation", + "", + "Use the JSON artifact for exact file and line evidence. Missing help is", + "a triage list, not an automatic defect. Endpoint coverage requires an", + "owner classification before enforcement. Runtime-computed structures", + "need explicit manifest metadata to become canonically visible.", + "", + ] + ) + return "\n".join(lines) + + +def canonical_api_path(value: str) -> str: + path = value.split("?", 1)[0].strip() + if "/api/" in path: + path = path[path.index("/api/") :] + path = re.sub(r"^/api/v\d+", "", path) + path = PATH_PARAMETER.sub("{}", path) + path = re.sub(r"/+", "/", path) + return path.rstrip("/") or "/" + + +def _join_route(prefix: str, route: str) -> str: + return f"/{prefix.strip('/')}/{route.strip('/')}".replace("//", "/") + + +def _static_string(node: ast.AST) -> str | None: + if isinstance(node, ast.Constant) and isinstance(node.value, str): + return node.value + if isinstance(node, ast.JoinedStr): + pieces: list[str] = [] + for value in node.values: + if isinstance(value, ast.Constant) and isinstance(value.value, str): + pieces.append(value.value) + elif isinstance(value, ast.FormattedValue): + pieces.append("${}") + else: + return None + return "".join(pieces) + return None + + +def _call_name(node: ast.AST) -> str: + if isinstance(node, ast.Name): + return node.id + if isinstance(node, ast.Attribute): + parent = _call_name(node.value) + return f"{parent}.{node.attr}" if parent else node.attr + return "" + + +def _plain_value(value: Any) -> Any: + if is_dataclass(value): + return { + key: _plain_value(item) + for key, item in asdict(value).items() + } + if isinstance(value, tuple): + return [_plain_value(item) for item in value] + if isinstance(value, dict): + return { + str(key): _plain_value(item) + for key, item in value.items() + } + return value + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/release/push-release-tag.sh b/tools/release/push-release-tag.sh index 707b72d..6a92f50 100644 --- a/tools/release/push-release-tag.sh +++ b/tools/release/push-release-tag.sh @@ -91,6 +91,7 @@ PACKAGE_MODULE_REPOS=( "$PARENT/govoplan-dataflow" "$PARENT/govoplan-datasources" "$PARENT/govoplan-docs" + "$PARENT/govoplan-encryption" "$PARENT/govoplan-facilities" "$PARENT/govoplan-files" "$PARENT/govoplan-forms-runtime"