diff --git a/README.md b/README.md index 2670c86..da8294d 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,9 @@ The module now provides an executable governed semantic-reporting vertical: arbitrary report SQL; - quality gates, saved views, interval/scheduled runs, CSV/JSON export, provider-neutral publication targets, and import activation assessments; +- a versioned, provider-neutral cross-module report contract with source-owned + authorization, declared result schemas, privacy transforms, effective scope, + source revisions, purpose, retention, export history, and audit provenance; - a full-height Reporting workspace for running, inspecting, saving, scheduling, visualizing, and exporting authorized reports. @@ -27,6 +30,10 @@ Reporting consumes Dataflow outputs or provider-owned read models. It does not read another module's ORM tables or take ownership of ingestion and transformation. +The canonical global route is `/reports`. `/reporting` remains a +Reporting-owned compatibility route for saved links. Domain modules may keep +their own operational report routes, but do not register `/reports`. + See [docs/REPORTING_BOUNDARY.md](docs/REPORTING_BOUNDARY.md) for the boundary decision. The behavior-level comparison with the supplied SuperX module set is recorded in diff --git a/docs/ADMIN_GUIDE.md b/docs/ADMIN_GUIDE.md index eb11673..0a4b9a8 100644 --- a/docs/ADMIN_GUIDE.md +++ b/docs/ADMIN_GUIDE.md @@ -32,6 +32,36 @@ the Reporting publication-target contract. The target receives one immutable execution payload and an idempotency key. It must return bounded evidence and must not expose credentials in that evidence. +## Cross-module provider governance + +Source modules register `reporting.report_provider.` capabilities; +do not grant Reporting direct table access to those modules. Review every +descriptor's result schema, required privacy transforms, retention class, +export formats, and re-identification risk before enabling it in production. +Provider authorization remains mandatory even when the Reporting role allows +the user to run reports. + +When Policy is enabled, configure `reporting_governance_policy` in system or +tenant settings. Tenant settings may only tighten the system result. Supported +fields are: + +```json +{ + "reporting_governance_policy": { + "allow_exports": true, + "allowed_export_formats": ["json"], + "allow_high_reidentification_risk": false, + "max_retention_days": 30, + "required_privacy_transforms": ["small_cell_suppression"] + } +} +``` + +Malformed explicit policy fails closed. The ordinary privacy-retention setting +`stored_report_detail_retention_days` is an additional ceiling. A tenant cannot +re-enable an export format, high-risk report, or longer retention period that +the system policy denied. + ## Import assessments Import assessment accepts declarative metadata only. Native datasets, diff --git a/docs/OPERATIONS.md b/docs/OPERATIONS.md index f9c3217..502de82 100644 --- a/docs/OPERATIONS.md +++ b/docs/OPERATIONS.md @@ -12,6 +12,15 @@ claim bounded batches, retain success or failure state, and advance interval time from the claimed occurrence. Providers must use the same tenant and principal context and must enforce their source-side authorization. +Provider-report executions and exports are database evidence. Each execution +stores the exact provider/report revision, request hash, source revisions, +effective scope, applied privacy transforms, governance decision provenance, +generation actor/time, retention expiry, and output hash. Export rows retain +their own purpose, audience, actor/time, format, and hash. Expired provider +results are rejected by retrieval and export paths; run the platform privacy +retention process on schedule to remove or minimize expired stored detail in +accordance with the active Policy. + ## Security controls - The semantic engine never evaluates Python, browser code, raw SQL, stored @@ -25,6 +34,10 @@ principal context and must enforce their source-side authorization. - Freshness, schema, fingerprints, quality gates, definition hashes, source provenance, actor, and output hash are retained with every run. - CSV formula injection is neutralized before direct download. +- Provider exceptions are isolated in catalogue diagnostics; an unavailable + optional provider cannot take down the global report catalogue. +- High re-identification-risk provider reports fail closed unless an installed + Policy capability explicitly permits them. ## Backup and restore diff --git a/docs/REPORTING_BOUNDARY.md b/docs/REPORTING_BOUNDARY.md index ca2cc17..6983fe2 100644 --- a/docs/REPORTING_BOUNDARY.md +++ b/docs/REPORTING_BOUNDARY.md @@ -87,6 +87,29 @@ Reporting does not own: - `reporting.publication_target:*` capabilities can accept immutable result payloads without Reporting importing the target module. +### Cross-module report providers + +Source modules contribute aggregate or otherwise minimized reports through +`reporting.report_provider.`. Core owns contract version `1.0` and +the provider-neutral DTOs. A provider must declare stable report and revision +IDs, typed parameters, the complete result schema, privacy transforms, +retention class, export formats, and re-identification risk. It must enforce +source access before listing options or producing a result and must return +source revisions, effective tenant scope, applied transforms, generation time, +and bounded provenance. + +Reporting discovers these capabilities by prefix. It validates descriptors, +rejects undeclared output fields, requires every mandatory transform, records +purpose and effective audience, and persists immutable execution and export +evidence. The optional `policy.reporting_governance` capability can tighten +retention, export formats, required transforms, and high-risk handling. If +Policy is absent, Reporting applies its restrictive built-in baseline and +denies high-risk provider reports. + +This direction is intentionally one-way: source modules import only Core's +contract and remain usable when Reporting is absent. Reporting never imports +the source module or reads its ORM tables. + ## Implemented Vertical The first complete vertical persists datasets, semantic models, reports, @@ -120,6 +143,11 @@ unsupported executable behavior remains. The WebUI uses the platform module loader and common controls. It exposes a report catalogue, parameter and semantic-query controls, result visualization and table views, history/provenance, saved views, schedules, and downloads. +The global `/reports` route is owned only by Reporting. `/reporting` is a +documented compatibility path. Campaign's module-local aggregate view remains +at `/campaigns/reports`; when both modules are enabled, the same safe aggregate +projection is also contributed to the global catalogue through the provider +contract. ## Remaining Product Depth diff --git a/docs/USER_GUIDE.md b/docs/USER_GUIDE.md index 4e7ccec..e09cf41 100644 --- a/docs/USER_GUIDE.md +++ b/docs/USER_GUIDE.md @@ -2,8 +2,9 @@ ## Open and run a report -Open **Reporting** and choose an active report from the left catalogue. The -header identifies the immutable report revision that will be used. Select a +Open **Reporting** at `/reports` and choose a report from the left catalogue. +The catalogue separates source-module reports from Reporting semantic reports. +The header identifies the immutable report revision that will be used. Select a summary, detail, or pivot shape, choose dimensions and measures, enter any report parameters, and select **Run**. @@ -30,3 +31,18 @@ spreadsheet formula marker are escaped. Users with scheduling permission can create an hourly, daily, weekly, or 30-day interval from the current revision, parameters, and query. Scheduled runs continue to use those exact pins until the schedule is edited. + +## Run a module report + +Module reports retain their source module's access rules. Select the source +object and any optional parameters, enter the concrete purpose for the run, +and verify the effective tenant audience before selecting **Run**. Reporting +will refuse a result that does not carry the declared privacy transforms, +source revision, and tenant scope. + +The result groups declared metrics and details without exposing undeclared +provider data. The Governance panel explains the risk class, permitted export +formats, retention ceiling, contract version, and required privacy transforms. +An export records a separate purpose, actor, audience, time, format, and output +hash. A missing or restricted module report is not replaced with a less +protected client-side query. diff --git a/src/govoplan_reporting/backend/db/models.py b/src/govoplan_reporting/backend/db/models.py index 638232e..140dfe5 100644 --- a/src/govoplan_reporting/backend/db/models.py +++ b/src/govoplan_reporting/backend/db/models.py @@ -218,6 +218,125 @@ class ReportingExecution(Base, TimestampMixin): actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) +class ReportingProviderExecution(Base, TimestampMixin): + __tablename__ = "reporting_provider_executions" + __table_args__ = ( + UniqueConstraint( + "tenant_id", + "execution_id", + name="uq_reporting_provider_execution", + ), + UniqueConstraint( + "tenant_id", + "idempotency_key", + name="uq_reporting_provider_execution_idempotency", + ), + Index( + "ix_reporting_provider_execution_history", + "tenant_id", + "provider_id", + "report_id", + "generated_at", + ), + Index( + "ix_reporting_provider_execution_retention", + "tenant_id", + "expires_at", + ), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + provider_id: Mapped[str] = mapped_column(String(120), nullable=False, index=True) + report_id: Mapped[str] = mapped_column(String(255), nullable=False, index=True) + report_revision: Mapped[str] = mapped_column(String(255), nullable=False) + contract_version: Mapped[str] = mapped_column(String(30), nullable=False) + idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False) + request_sha256: Mapped[str] = mapped_column(String(64), nullable=False) + purpose: Mapped[str] = mapped_column(Text, nullable=False) + audience_scope: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + parameters: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + result_schema: Mapped[list[dict[str, Any]]] = mapped_column( + JSON, default=list, nullable=False + ) + result_payload: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + source_revisions: Mapped[list[dict[str, Any]]] = mapped_column( + JSON, default=list, nullable=False + ) + effective_scope: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + privacy_transforms: Mapped[list[str]] = mapped_column( + JSON, default=list, nullable=False + ) + provenance: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + governance_provenance: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + retention_class: Mapped[str] = mapped_column(String(120), nullable=False) + retention_days: Mapped[int | None] = mapped_column(Integer, nullable=True) + expires_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True, index=True + ) + retention_redacted_at: Mapped[datetime | None] = mapped_column( + DateTime(timezone=True), nullable=True, index=True + ) + output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True) + generated_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, index=True + ) + actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) + + +class ReportingProviderExport(Base, TimestampMixin): + __tablename__ = "reporting_provider_exports" + __table_args__ = ( + UniqueConstraint( + "tenant_id", + "export_id", + name="uq_reporting_provider_export", + ), + Index( + "ix_reporting_provider_export_history", + "tenant_id", + "provider_execution_id", + "exported_at", + ), + ) + + id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid) + tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + export_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + provider_execution_id: Mapped[str] = mapped_column( + ForeignKey( + "reporting_provider_executions.id", + ondelete="RESTRICT", + ), + nullable=False, + index=True, + ) + execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True) + format: Mapped[str] = mapped_column(String(30), nullable=False) + purpose: Mapped[str] = mapped_column(Text, nullable=False) + audience_scope: Mapped[dict[str, Any]] = mapped_column( + JSON, default=dict, nullable=False + ) + output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True) + exported_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), nullable=False, index=True + ) + actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) + + class ReportingDefinitionGrant(Base, TimestampMixin): __tablename__ = "reporting_definition_grants" __table_args__ = ( @@ -435,6 +554,8 @@ __all__ = [ "ReportingExecution", "ReportingImportAssessment", "ReportingPublication", + "ReportingProviderExecution", + "ReportingProviderExport", "ReportingQualityResult", "ReportingSavedView", "ReportingSchedule", diff --git a/src/govoplan_reporting/backend/manifest.py b/src/govoplan_reporting/backend/manifest.py index ad49fc3..368b343 100644 --- a/src/govoplan_reporting/backend/manifest.py +++ b/src/govoplan_reporting/backend/manifest.py @@ -31,6 +31,10 @@ from govoplan_core.core.provider_governance import ( ModuleArchitectureDocumentation, ModuleMaturityEvidence, ) +from govoplan_core.core.reporting import ( + CAPABILITY_POLICY_REPORTING_GOVERNANCE, + CAPABILITY_REPORTING_RETENTION, +) from govoplan_core.core.search import SearchSourceProviderRegistration from govoplan_core.core.views import ViewSurface from govoplan_core.db.base import Base @@ -171,6 +175,13 @@ def _chart_renderer(context: ModuleContext) -> DefaultChartRenderer: return DefaultChartRenderer() +def _retention(context: ModuleContext): + del context + from govoplan_reporting.backend.retention import ReportingRetentionService + + return ReportingRetentionService() + + def _tenant_summary(session, tenant_id: str) -> dict[str, int]: definitions = ( session.query(reporting_models.ReportingDefinitionRevision) @@ -232,17 +243,21 @@ manifest = ModuleManifest( CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PERMISSION_EVALUATOR, ), - optional_capabilities=(CAPABILITY_DATAFLOW_DATASET_OUTPUT,), + optional_capabilities=( + CAPABILITY_DATAFLOW_DATASET_OUTPUT, + CAPABILITY_POLICY_REPORTING_GOVERNANCE, + ), permissions=PERMISSIONS, role_templates=ROLE_TEMPLATES, route_factory=_router, nav_items=( NavItem( - path="/reporting", + path="/reports", label="Reporting", icon="clipboard-pen-line", required_any=(READ_SCOPE,), order=74, + surface_id="reporting.navigation", ), ), frontend=FrontendModule( @@ -250,36 +265,31 @@ manifest = ModuleManifest( package_name="@govoplan/reporting-webui", routes=( FrontendRoute( - path="/reporting", + path="/reports", component="ReportingPage", required_any=(READ_SCOPE,), order=74, + surface_id="reporting.workspace", + ), + FrontendRoute( + path="/reporting", + component="ReportingPage", + required_any=(READ_SCOPE,), + order=175, + surface_id="reporting.compatibility", ), ), nav_items=( NavItem( - path="/reporting", + path="/reports", label="Reporting", icon="clipboard-pen-line", required_any=(READ_SCOPE,), order=74, + surface_id="reporting.navigation", ), ), view_surfaces=( - ViewSurface( - id="reporting.navigation", - module_id=MODULE_ID, - kind="navigation", - label="Reporting navigation", - order=10, - ), - ViewSurface( - id="reporting.workspace", - module_id=MODULE_ID, - kind="route", - label="Reporting workspace", - order=20, - ), ViewSurface( id="reporting.parameters", module_id=MODULE_ID, @@ -303,6 +313,7 @@ manifest = ModuleManifest( ModuleInterfaceProvider(name="reporting.runner", version="0.1.0"), ModuleInterfaceProvider(name="reporting.scheduler", version="0.1.0"), ModuleInterfaceProvider(name="reporting.chart_renderer", version="0.1.0"), + ModuleInterfaceProvider(name=CAPABILITY_REPORTING_RETENTION, version="1.0.0"), ), requires_interfaces=( ModuleInterfaceRequirement( @@ -311,12 +322,19 @@ manifest = ModuleManifest( version_max_exclusive="0.2.0", optional=True, ), + ModuleInterfaceRequirement( + name=CAPABILITY_POLICY_REPORTING_GOVERNANCE, + version_min="1.0.0", + version_max_exclusive="2.0.0", + optional=True, + ), ), capability_factories={ CAPABILITY_REPORTING_REGISTRY: _registry, CAPABILITY_REPORTING_RUNNER: _runner, CAPABILITY_REPORTING_SCHEDULER: _scheduler, CAPABILITY_REPORTING_CHART_RENDERER: _chart_renderer, + CAPABILITY_REPORTING_RETENTION: _retention, }, capability_documentation={ CAPABILITY_REPORTING_REGISTRY: CapabilityDocumentation( @@ -339,6 +357,13 @@ manifest = ModuleManifest( summary="Builds provider-neutral chart models with an accessible tabular fallback.", contract_version="0.1.0", ), + CAPABILITY_REPORTING_RETENTION: CapabilityDocumentation( + label="Reporting result retention", + summary="Minimizes expired provider-report detail while retaining audit hashes and provenance.", + contract_version="1.0", + documentation_types=("admin",), + audience=("privacy_officer", "operator", "system_admin"), + ), }, search_sources=( SearchSourceProviderRegistration( @@ -359,6 +384,8 @@ manifest = ModuleManifest( reporting_models.ReportingSavedView, reporting_models.ReportingDefinitionGrant, reporting_models.ReportingExecution, + reporting_models.ReportingProviderExport, + reporting_models.ReportingProviderExecution, reporting_models.ReportingDefinitionRevision, reporting_models.ReportingDefinitionIdentity, label="Reporting", @@ -374,6 +401,8 @@ manifest = ModuleManifest( reporting_models.ReportingDefinitionRevision, reporting_models.ReportingDefinitionGrant, reporting_models.ReportingExecution, + reporting_models.ReportingProviderExecution, + reporting_models.ReportingProviderExport, reporting_models.ReportingSavedView, reporting_models.ReportingSchedule, reporting_models.ReportingPublication, diff --git a/src/govoplan_reporting/backend/migrations/versions/b7c4e1a9d2f6_provider_reporting.py b/src/govoplan_reporting/backend/migrations/versions/b7c4e1a9d2f6_provider_reporting.py new file mode 100644 index 0000000..a60c8ea --- /dev/null +++ b/src/govoplan_reporting/backend/migrations/versions/b7c4e1a9d2f6_provider_reporting.py @@ -0,0 +1,142 @@ +"""Add governed cross-module report executions and export history. + +Revision ID: b7c4e1a9d2f6 +Revises: e5b2c9d4f7a1 +""" + +from __future__ import annotations + +from alembic import op +import sqlalchemy as sa + + +revision = "b7c4e1a9d2f6" +down_revision = "e5b2c9d4f7a1" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.create_table( + "reporting_provider_executions", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=False), + sa.Column("execution_id", sa.String(length=36), nullable=False), + sa.Column("provider_id", sa.String(length=120), nullable=False), + sa.Column("report_id", sa.String(length=255), nullable=False), + sa.Column("report_revision", sa.String(length=255), nullable=False), + sa.Column("contract_version", sa.String(length=30), nullable=False), + sa.Column("idempotency_key", sa.String(length=255), nullable=False), + sa.Column("request_sha256", sa.String(length=64), nullable=False), + sa.Column("purpose", sa.Text(), nullable=False), + sa.Column("audience_scope", sa.JSON(), nullable=False), + sa.Column("parameters", sa.JSON(), nullable=False), + sa.Column("result_schema", sa.JSON(), nullable=False), + sa.Column("result_payload", sa.JSON(), nullable=False), + sa.Column("source_revisions", sa.JSON(), nullable=False), + sa.Column("effective_scope", sa.JSON(), nullable=False), + sa.Column("privacy_transforms", sa.JSON(), nullable=False), + sa.Column("provenance", sa.JSON(), nullable=False), + sa.Column("governance_provenance", sa.JSON(), nullable=False), + sa.Column("retention_class", sa.String(length=120), nullable=False), + sa.Column("retention_days", sa.Integer(), nullable=True), + sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("retention_redacted_at", sa.DateTime(timezone=True), nullable=True), + sa.Column("output_hash", sa.String(length=64), nullable=False), + sa.Column("generated_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("actor_id", sa.String(length=255), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_executions")), + sa.UniqueConstraint( + "tenant_id", + "execution_id", + name="uq_reporting_provider_execution", + ), + sa.UniqueConstraint( + "tenant_id", + "idempotency_key", + name="uq_reporting_provider_execution_idempotency", + ), + ) + _indexes( + "reporting_provider_executions", + "tenant_id", + "execution_id", + "provider_id", + "report_id", + "expires_at", + "retention_redacted_at", + "output_hash", + "generated_at", + "actor_id", + ) + op.create_index( + "ix_reporting_provider_execution_history", + "reporting_provider_executions", + ["tenant_id", "provider_id", "report_id", "generated_at"], + unique=False, + ) + op.create_index( + "ix_reporting_provider_execution_retention", + "reporting_provider_executions", + ["tenant_id", "expires_at"], + unique=False, + ) + + op.create_table( + "reporting_provider_exports", + sa.Column("id", sa.String(length=36), nullable=False), + sa.Column("tenant_id", sa.String(length=36), nullable=False), + sa.Column("export_id", sa.String(length=36), nullable=False), + sa.Column("provider_execution_id", sa.String(length=36), nullable=False), + sa.Column("execution_id", sa.String(length=36), nullable=False), + sa.Column("format", sa.String(length=30), nullable=False), + sa.Column("purpose", sa.Text(), nullable=False), + sa.Column("audience_scope", sa.JSON(), nullable=False), + sa.Column("output_hash", sa.String(length=64), nullable=False), + sa.Column("exported_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("actor_id", sa.String(length=255), nullable=True), + sa.Column("created_at", sa.DateTime(timezone=True), nullable=False), + sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False), + sa.ForeignKeyConstraint( + ["provider_execution_id"], + ["reporting_provider_executions.id"], + name=op.f( + "fk_reporting_provider_exports_provider_execution_id_reporting_provider_executions" + ), + ondelete="RESTRICT", + ), + sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_exports")), + sa.UniqueConstraint( + "tenant_id", + "export_id", + name="uq_reporting_provider_export", + ), + ) + _indexes( + "reporting_provider_exports", + "tenant_id", + "export_id", + "provider_execution_id", + "execution_id", + "output_hash", + "exported_at", + "actor_id", + ) + op.create_index( + "ix_reporting_provider_export_history", + "reporting_provider_exports", + ["tenant_id", "provider_execution_id", "exported_at"], + unique=False, + ) + + +def downgrade() -> None: + op.drop_table("reporting_provider_exports") + op.drop_table("reporting_provider_executions") + + +def _indexes(table: str, *columns: str) -> None: + for column in columns: + op.create_index(op.f(f"ix_{table}_{column}"), table, [column], unique=False) diff --git a/src/govoplan_reporting/backend/provider_reports.py b/src/govoplan_reporting/backend/provider_reports.py new file mode 100644 index 0000000..313c418 --- /dev/null +++ b/src/govoplan_reporting/backend/provider_reports.py @@ -0,0 +1,835 @@ +"""Governed execution boundary for module-contributed reports.""" + +from __future__ import annotations + +from collections.abc import Mapping +import csv +from datetime import UTC, date, datetime, timedelta +from hashlib import sha256 +import io +import json +import uuid + +from sqlalchemy.orm import Session + +from govoplan_core.audit.logging import audit_from_principal +from govoplan_core.core.reporting import ( + ReportDescriptor, + ReportParameterOption, + ReportProvider, + ReportProviderRequest, + ReportingGovernanceDecision, + ReportingGovernanceRequest, + report_providers, + reporting_governance_provider, +) +from govoplan_reporting.backend.db.models import ( + ReportingProviderExecution, + ReportingProviderExport, +) + + +class ProviderReportError(RuntimeError): + pass + + +def list_provider_reports( + session: Session, + principal: object, + *, + registry: object | None, +) -> dict[str, object]: + reports: list[dict[str, object]] = [] + diagnostics: list[dict[str, str]] = [] + for provider_id, provider in report_providers(registry): + try: + for descriptor in provider.list_reports(session, principal): + _validate_descriptor(provider_id, descriptor) + decision = _governance_decision( + session, + principal, + registry=registry, + descriptor=descriptor, + action="catalogue", + purpose=None, + audience_scope={}, + ) + reports.append( + { + **descriptor.to_dict(), + "available": decision.allowed, + "unavailable_reason": decision.reason, + "governance": _decision_payload(decision), + } + ) + except Exception as exc: # noqa: BLE001 - isolate optional providers. + diagnostics.append( + { + "provider_id": provider_id, + "code": "provider_catalogue_failed", + "message": "The provider catalogue could not be loaded.", + "error_type": type(exc).__name__, + } + ) + return { + "reports": sorted( + reports, + key=lambda item: (str(item["title"]), str(item["provider_id"])), + ), + "diagnostics": diagnostics, + } + + +def provider_parameter_options( + session: Session, + principal: object, + *, + registry: object | None, + provider_id: str, + report_id: str, + parameter_key: str, + query: str, + limit: int, +) -> tuple[ReportParameterOption, ...]: + provider, descriptor = _provider_report( + session, + principal, + registry=registry, + provider_id=provider_id, + report_id=report_id, + ) + parameter = next( + (item for item in descriptor.parameters if item.key == parameter_key), + None, + ) + if parameter is None or not parameter.options_from_provider: + raise ProviderReportError("This report parameter has no provider options") + return provider.parameter_options( + session, + principal, + report_id=report_id, + parameter_key=parameter_key, + query=query, + limit=max(1, min(limit, 200)), + ) + + +def execute_provider_report( + session: Session, + principal: object, + *, + registry: object | None, + provider_id: str, + report_id: str, + parameters: Mapping[str, object], + purpose: str, + audience_scope: Mapping[str, object], + idempotency_key: str, +) -> dict[str, object]: + provider, descriptor = _provider_report( + session, + principal, + registry=registry, + provider_id=provider_id, + report_id=report_id, + ) + clean_purpose = purpose.strip() + clean_idempotency_key = idempotency_key.strip() + clean_parameters = _validated_parameters(descriptor, parameters) + clean_audience = _json_mapping(audience_scope, "audience scope") + if descriptor.purpose_required and not clean_purpose: + raise ProviderReportError("A report purpose is required") + if descriptor.audience_scope_required and not clean_audience: + raise ProviderReportError("An effective audience scope is required") + request_hash = _hash( + { + "provider_id": provider_id, + "report_id": report_id, + "revision": descriptor.revision, + "parameters": clean_parameters, + "purpose": clean_purpose, + "audience_scope": clean_audience, + } + ) + replay = ( + session.query(ReportingProviderExecution) + .filter( + ReportingProviderExecution.tenant_id == _tenant(principal), + ReportingProviderExecution.idempotency_key == clean_idempotency_key, + ) + .one_or_none() + ) + if replay is not None: + if replay.request_sha256 != request_hash: + raise ProviderReportError( + "The provider-report idempotency key belongs to another request" + ) + if _expired(replay) or replay.retention_redacted_at is not None: + raise ProviderReportError( + "The replayed provider-report result has expired; use a new idempotency key" + ) + _require_result_access(provider, session, principal, replay) + return provider_execution_payload(replay) + + preflight = _governance_decision( + session, + principal, + registry=registry, + descriptor=descriptor, + action="execute", + purpose=clean_purpose, + audience_scope=clean_audience, + ) + _require_allowed(preflight) + result = provider.execute_report( + session, + principal, + request=ReportProviderRequest( + report_id=report_id, + parameters=clean_parameters, + purpose=clean_purpose, + audience_scope=clean_audience, + ), + ) + _validate_result(descriptor, result, tenant_id=_tenant(principal)) + decision = _governance_decision( + session, + principal, + registry=registry, + descriptor=descriptor, + action="execute", + purpose=clean_purpose, + audience_scope=clean_audience, + applied_privacy_transforms=result.applied_privacy_transforms, + ) + _require_allowed(decision) + missing_policy_transforms = set(decision.required_privacy_transforms) - set( + result.applied_privacy_transforms + ) + if missing_policy_transforms: + raise ProviderReportError( + "The report omitted Policy-required privacy transformations: " + + ", ".join(sorted(missing_policy_transforms)) + ) + generated_at = _aware(result.generated_at) + result_payload = _json_mapping(result.payload, "provider report payload") + expires_at = ( + generated_at + timedelta(days=decision.retention_days) + if decision.retention_days is not None + else None + ) + row = ReportingProviderExecution( + tenant_id=_tenant(principal), + execution_id=str(uuid.uuid4()), + provider_id=provider_id, + report_id=report_id, + report_revision=descriptor.revision, + contract_version=descriptor.contract_version, + idempotency_key=clean_idempotency_key, + request_sha256=request_hash, + purpose=clean_purpose, + audience_scope=clean_audience, + parameters=clean_parameters, + result_schema=[item.to_dict() for item in descriptor.result_schema], + result_payload=result_payload, + source_revisions=[dict(item) for item in result.source_revisions], + effective_scope=dict(result.effective_scope), + privacy_transforms=list(result.applied_privacy_transforms), + provenance=dict(result.provenance), + governance_provenance=dict(decision.provenance), + retention_class=descriptor.retention_class, + retention_days=decision.retention_days, + expires_at=expires_at, + output_hash=_hash(result_payload), + generated_at=generated_at, + actor_id=_actor(principal), + ) + session.add(row) + session.flush() + audit_from_principal( + session, + principal, + action="reporting.provider_report.executed", + object_type="reporting_provider_execution", + object_id=row.execution_id, + details={ + "provider_id": provider_id, + "report_id": report_id, + "report_revision": descriptor.revision, + "purpose": clean_purpose, + "audience_scope": clean_audience, + "source_revision_count": len(row.source_revisions), + "privacy_transforms": row.privacy_transforms, + "retention_class": row.retention_class, + "retention_days": row.retention_days, + "output_hash": row.output_hash, + }, + ) + return provider_execution_payload(row) + + +def get_provider_execution( + session: Session, + principal: object, + *, + registry: object | None, + execution_id: str, +) -> dict[str, object] | None: + row = _provider_execution(session, principal, execution_id=execution_id) + if row is None or _expired(row): + return None + provider, _descriptor = _provider_report( + session, + principal, + registry=registry, + provider_id=row.provider_id, + report_id=row.report_id, + ) + _require_result_access(provider, session, principal, row) + return provider_execution_payload(row) + + +def export_provider_execution( + session: Session, + principal: object, + *, + registry: object | None, + execution_id: str, + format: str, + purpose: str, + audience_scope: Mapping[str, object], +) -> tuple[bytes, str, str]: + row = _provider_execution(session, principal, execution_id=execution_id) + if row is None or _expired(row): + raise LookupError("Provider report execution not found") + provider, descriptor = _provider_report( + session, + principal, + registry=registry, + provider_id=row.provider_id, + report_id=row.report_id, + ) + _require_result_access(provider, session, principal, row) + clean_format = format.strip().lower() + if clean_format not in descriptor.export_formats: + raise ProviderReportError("This report does not support the export format") + clean_purpose = purpose.strip() + clean_audience = _json_mapping(audience_scope, "audience scope") + decision = _governance_decision( + session, + principal, + registry=registry, + descriptor=descriptor, + action="export", + purpose=clean_purpose, + audience_scope=clean_audience, + export_format=clean_format, + applied_privacy_transforms=tuple(row.privacy_transforms or ()), + ) + _require_allowed(decision) + if clean_format not in decision.export_formats: + raise ProviderReportError("Policy does not allow this export format") + content, media_type, extension = _serialize_export( + clean_format, + row.result_payload, + row.result_schema, + ) + output_hash = sha256(content).hexdigest() + export = ReportingProviderExport( + tenant_id=row.tenant_id, + export_id=str(uuid.uuid4()), + provider_execution_id=row.id, + execution_id=row.execution_id, + format=clean_format, + purpose=clean_purpose, + audience_scope=clean_audience, + output_hash=output_hash, + exported_at=datetime.now(UTC), + actor_id=_actor(principal), + ) + session.add(export) + session.flush() + audit_from_principal( + session, + principal, + action="reporting.provider_report.exported", + object_type="reporting_provider_execution", + object_id=row.execution_id, + details={ + "export_id": export.export_id, + "format": clean_format, + "purpose": clean_purpose, + "audience_scope": clean_audience, + "output_hash": output_hash, + }, + ) + filename = f"{row.provider_id}-{row.report_id}-{row.execution_id}.{extension}" + return content, media_type, filename + + +def list_provider_exports( + session: Session, + principal: object, + *, + registry: object | None, + execution_id: str, +) -> tuple[dict[str, object], ...]: + row = _provider_execution(session, principal, execution_id=execution_id) + if row is None or _expired(row): + return () + provider, _descriptor = _provider_report( + session, + principal, + registry=registry, + provider_id=row.provider_id, + report_id=row.report_id, + ) + _require_result_access(provider, session, principal, row) + exports = ( + session.query(ReportingProviderExport) + .filter( + ReportingProviderExport.tenant_id == row.tenant_id, + ReportingProviderExport.provider_execution_id == row.id, + ) + .order_by(ReportingProviderExport.exported_at.desc()) + .all() + ) + return tuple( + { + "export_id": item.export_id, + "execution_id": item.execution_id, + "format": item.format, + "purpose": item.purpose, + "audience_scope": dict(item.audience_scope or {}), + "output_hash": item.output_hash, + "exported_at": item.exported_at.isoformat(), + "actor_id": item.actor_id, + } + for item in exports + ) + + +def provider_execution_payload(row: ReportingProviderExecution) -> dict[str, object]: + return { + "execution_id": row.execution_id, + "provider_id": row.provider_id, + "report_id": row.report_id, + "report_revision": row.report_revision, + "contract_version": row.contract_version, + "purpose": row.purpose, + "audience_scope": dict(row.audience_scope or {}), + "parameters": dict(row.parameters or {}), + "result_schema": list(row.result_schema or []), + "result": dict(row.result_payload or {}), + "source_revisions": list(row.source_revisions or []), + "effective_scope": dict(row.effective_scope or {}), + "privacy_transforms": list(row.privacy_transforms or []), + "provenance": dict(row.provenance or {}), + "governance_provenance": dict(row.governance_provenance or {}), + "retention_class": row.retention_class, + "retention_days": row.retention_days, + "expires_at": row.expires_at.isoformat() if row.expires_at else None, + "output_hash": row.output_hash, + "generated_at": row.generated_at.isoformat(), + "actor_id": row.actor_id, + } + + +def _provider_report( + session: Session, + principal: object, + *, + registry: object | None, + provider_id: str, + report_id: str, +) -> tuple[ReportProvider, ReportDescriptor]: + provider = dict(report_providers(registry)).get(provider_id) + if provider is None: + raise LookupError("Report provider is unavailable") + descriptor = next( + ( + item + for item in provider.list_reports(session, principal) + if item.report_id == report_id + ), + None, + ) + if descriptor is None: + raise LookupError("Provider report is unavailable") + _validate_descriptor(provider_id, descriptor) + return provider, descriptor + + +def _validate_descriptor(provider_id: str, descriptor: ReportDescriptor) -> None: + if descriptor.provider_id != provider_id: + raise ProviderReportError("Report descriptor provider id differs") + if not descriptor.report_id or not descriptor.revision: + raise ProviderReportError("Report descriptors require stable ids and revisions") + parameter_keys = [item.key for item in descriptor.parameters] + field_paths = [item.path for item in descriptor.result_schema] + transform_ids = [item.id for item in descriptor.privacy_transforms] + if len(parameter_keys) != len(set(parameter_keys)): + raise ProviderReportError("Report parameter keys must be unique") + if not field_paths or len(field_paths) != len(set(field_paths)): + raise ProviderReportError("Report result paths must be non-empty and unique") + if len(transform_ids) != len(set(transform_ids)): + raise ProviderReportError("Report privacy transforms must be unique") + if any(item.sensitive for item in descriptor.result_schema) and ( + descriptor.reidentification_risk != "high" + ): + raise ProviderReportError( + "Sensitive result fields require high risk classification" + ) + + +def _validate_result( + descriptor: ReportDescriptor, + result: object, + *, + tenant_id: str, +) -> None: + if not hasattr(result, "report_id") or result.report_id != descriptor.report_id: + raise ProviderReportError("Provider result report id differs") + if not result.source_revisions: + raise ProviderReportError("Provider result has no source revision provenance") + if str(result.effective_scope.get("tenant_id") or "") != tenant_id: + raise ProviderReportError( + "Provider result effective scope differs from the tenant" + ) + required = {item.id for item in descriptor.privacy_transforms if item.required} + missing = required - set(result.applied_privacy_transforms) + if missing: + raise ProviderReportError( + "Provider result omitted required privacy transformations: " + + ", ".join(sorted(missing)) + ) + declared = {item.path: item for item in descriptor.result_schema} + for field in descriptor.result_schema: + if field.type != "suppressed_count": + continue + value = _path_value(result.payload, field.path) + if not isinstance(value, Mapping) or set(value) != {"value", "suppressed"}: + raise ProviderReportError( + f"Provider result has an invalid suppressed count: {field.path}" + ) + if value["value"] is not None and ( + not isinstance(value["value"], int) or isinstance(value["value"], bool) + ): + raise ProviderReportError( + f"Provider result has an invalid suppressed-count value: {field.path}" + ) + if not isinstance(value["suppressed"], bool): + raise ProviderReportError( + f"Provider result has an invalid suppression flag: {field.path}" + ) + for path in _leaf_paths(result.payload): + if path in declared: + continue + if any( + path.startswith(declared_path + ".") + and field.type in {"object", "suppressed_count"} + for declared_path, field in declared.items() + ): + continue + raise ProviderReportError(f"Provider result contains undeclared field: {path}") + + +def _validated_parameters( + descriptor: ReportDescriptor, + parameters: Mapping[str, object], +) -> dict[str, object]: + clean = _json_mapping(parameters, "report parameters") + declared = {item.key: item for item in descriptor.parameters} + unknown = set(clean) - set(declared) + if unknown: + raise ProviderReportError( + "Unknown report parameters: " + ", ".join(sorted(unknown)) + ) + missing = [ + item.key + for item in descriptor.parameters + if item.required and clean.get(item.key) in (None, "") + ] + if missing: + raise ProviderReportError( + "Missing report parameters: " + ", ".join(sorted(missing)) + ) + invalid = [ + item.key + for item in descriptor.parameters + if item.key in clean + and clean[item.key] is not None + and not _valid_parameter_value(item.type, clean[item.key]) + ] + if invalid: + raise ProviderReportError( + "Invalid report parameter types: " + ", ".join(sorted(invalid)) + ) + return clean + + +def _valid_parameter_value(parameter_type: str, value: object) -> bool: + if parameter_type in {"string", "reference"}: + return isinstance(value, str) + if parameter_type == "boolean": + return isinstance(value, bool) + if parameter_type == "integer": + return isinstance(value, int) and not isinstance(value, bool) + if parameter_type == "number": + return isinstance(value, (int, float)) and not isinstance(value, bool) + if parameter_type == "date": + if not isinstance(value, str): + return False + try: + date.fromisoformat(value) + except ValueError: + return False + return True + if parameter_type == "datetime": + if not isinstance(value, str): + return False + try: + datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return False + return True + return False + + +def _governance_decision( + session: Session, + principal: object, + *, + registry: object | None, + descriptor: ReportDescriptor, + action: str, + purpose: str | None, + audience_scope: Mapping[str, object], + export_format: str | None = None, + applied_privacy_transforms: tuple[str, ...] = (), +) -> ReportingGovernanceDecision: + required = tuple(item.id for item in descriptor.privacy_transforms if item.required) + provider = reporting_governance_provider(registry) + if provider is None: + allowed = descriptor.reidentification_risk != "high" + if descriptor.purpose_required and action != "catalogue" and not purpose: + allowed = False + if ( + descriptor.audience_scope_required + and action != "catalogue" + and not audience_scope + ): + allowed = False + if action == "export" and export_format not in descriptor.export_formats: + allowed = False + return ReportingGovernanceDecision( + allowed=allowed, + reason=None + if allowed + else "The built-in Reporting privacy baseline denied this action.", + retention_days=30, + export_formats=descriptor.export_formats, + required_privacy_transforms=required, + provenance={"provider": "reporting.built_in_baseline", "version": "1"}, + ) + decision = provider.decide_reporting_action( + session, + principal, + request=ReportingGovernanceRequest( + action=action, # type: ignore[arg-type] + tenant_id=_tenant(principal), + provider_id=descriptor.provider_id, + report_id=descriptor.report_id, + purpose=purpose, + audience_scope=audience_scope, + retention_class=descriptor.retention_class, + export_format=export_format, + reidentification_risk=descriptor.reidentification_risk, + declared_privacy_transforms=required, + applied_privacy_transforms=applied_privacy_transforms, + ), + ) + unsupported = set(decision.required_privacy_transforms) - { + item.id for item in descriptor.privacy_transforms + } + if not unsupported: + return decision + return ReportingGovernanceDecision( + allowed=False, + reason=( + "Policy requires privacy transformations this report does not support: " + + ", ".join(sorted(unsupported)) + ), + retention_days=decision.retention_days, + export_formats=decision.export_formats, + required_privacy_transforms=decision.required_privacy_transforms, + provenance={ + **dict(decision.provenance), + "unsupported_required_privacy_transforms": sorted(unsupported), + }, + ) + + +def _serialize_export( + format: str, + payload: Mapping[str, object], + schema: list[dict[str, object]], +) -> tuple[bytes, str, str]: + if format == "json": + return ( + json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True).encode( + "utf-8" + ), + "application/json", + "json", + ) + if format != "csv": + raise ProviderReportError("Unsupported provider report export format") + output = io.StringIO(newline="") + writer = csv.writer(output) + writer.writerow([str(item["label"]) for item in schema]) + writer.writerow( + [_csv_value(_path_value(payload, str(item["path"]))) for item in schema] + ) + return output.getvalue().encode("utf-8-sig"), "text/csv; charset=utf-8", "csv" + + +def _csv_value(value: object) -> object: + if isinstance(value, Mapping): + if value.get("suppressed") is True: + return "suppressed" + if set(value).issuperset({"value", "suppressed"}): + value = value.get("value") + else: + value = json.dumps(value, ensure_ascii=False, sort_keys=True) + if isinstance(value, (list, tuple)): + value = json.dumps(value, ensure_ascii=False, sort_keys=True) + if isinstance(value, str) and value.startswith(("=", "+", "-", "@", "\t", "\r")): + return "'" + value + return "" if value is None else value + + +def _path_value(payload: Mapping[str, object], path: str) -> object: + value: object = payload + for part in path.split("."): + if not isinstance(value, Mapping): + return None + value = value.get(part) + return value + + +def _leaf_paths(value: object, prefix: str = "") -> tuple[str, ...]: + if isinstance(value, Mapping): + rows: list[str] = [] + for key, item in value.items(): + path = f"{prefix}.{key}" if prefix else str(key) + rows.extend(_leaf_paths(item, path)) + return tuple(rows) + if isinstance(value, (list, tuple)): + return (prefix,) + return (prefix,) + + +def _provider_execution( + session: Session, + principal: object, + *, + execution_id: str, +) -> ReportingProviderExecution | None: + return ( + session.query(ReportingProviderExecution) + .filter( + ReportingProviderExecution.tenant_id == _tenant(principal), + ReportingProviderExecution.execution_id == execution_id, + ) + .one_or_none() + ) + + +def _require_result_access( + provider: ReportProvider, + session: Session, + principal: object, + row: ReportingProviderExecution, +) -> None: + if not provider.authorize_result( + session, + principal, + report_id=row.report_id, + source_revisions=tuple(row.source_revisions or ()), + effective_scope=dict(row.effective_scope or {}), + ): + raise PermissionError( + "The source module no longer permits access to this report result" + ) + + +def _expired(row: ReportingProviderExecution) -> bool: + if row.expires_at is None: + return False + return _aware(row.expires_at) <= datetime.now(UTC) + + +def _require_allowed(decision: ReportingGovernanceDecision) -> None: + if not decision.allowed: + raise PermissionError(decision.reason or "Reporting Policy denied this action") + + +def _decision_payload(decision: ReportingGovernanceDecision) -> dict[str, object]: + return { + "allowed": decision.allowed, + "reason": decision.reason, + "retention_days": decision.retention_days, + "export_formats": list(decision.export_formats), + "required_privacy_transforms": list(decision.required_privacy_transforms), + "provenance": dict(decision.provenance), + } + + +def _json_mapping(value: Mapping[str, object], label: str) -> dict[str, object]: + try: + payload = json.loads(json.dumps(value, sort_keys=True, separators=(",", ":"))) + except (TypeError, ValueError) as exc: + raise ProviderReportError(f"The {label} must contain JSON values") from exc + if not isinstance(payload, dict): + raise ProviderReportError(f"The {label} must be an object") + return payload + + +def _hash(value: object) -> str: + return sha256( + json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + ).hexdigest() + + +def _tenant(principal: object) -> str: + tenant_id = str(getattr(principal, "tenant_id", "") or "").strip() + if not tenant_id: + raise PermissionError("A tenant principal is required") + return tenant_id + + +def _actor(principal: object) -> str | None: + user = getattr(principal, "user", None) + actor_id = getattr(user, "id", None) + return str(actor_id) if actor_id else None + + +def _aware(value: datetime) -> datetime: + return value if value.tzinfo is not None else value.replace(tzinfo=UTC) + + +__all__ = [ + "ProviderReportError", + "execute_provider_report", + "export_provider_execution", + "get_provider_execution", + "list_provider_exports", + "list_provider_reports", + "provider_parameter_options", +] diff --git a/src/govoplan_reporting/backend/retention.py b/src/govoplan_reporting/backend/retention.py new file mode 100644 index 0000000..d026569 --- /dev/null +++ b/src/govoplan_reporting/backend/retention.py @@ -0,0 +1,69 @@ +"""Retention minimization for governed provider-report results.""" + +from __future__ import annotations + +from datetime import UTC, datetime + +from sqlalchemy.orm import Session + +from govoplan_reporting.backend.db.models import ReportingProviderExecution + + +class ReportingRetentionService: + def apply_retention( + self, + session: object, + *, + dry_run: bool, + now: datetime, + limit: int = 500, + ) -> dict[str, int]: + if not isinstance(session, Session): + raise TypeError("Reporting retention requires a SQLAlchemy Session") + observed_at = _aware(now) + rows = ( + session.query(ReportingProviderExecution) + .filter( + ReportingProviderExecution.expires_at.is_not(None), + ReportingProviderExecution.expires_at <= observed_at, + ReportingProviderExecution.retention_redacted_at.is_(None), + ) + .order_by( + ReportingProviderExecution.expires_at.asc(), + ReportingProviderExecution.id.asc(), + ) + .limit(max(1, min(int(limit), 5_000))) + .all() + ) + counts = { + "eligible": len(rows), + "redacted": 0, + "remaining_in_batch": 0, + } + if dry_run: + return counts + for row in rows: + # Keep immutable request/output hashes and provenance as audit + # evidence while removing the retained report detail itself. + row.result_payload = {} + row.retention_redacted_at = observed_at + counts["redacted"] += 1 + session.flush() + counts["remaining_in_batch"] = ( + session.query(ReportingProviderExecution.id) + .filter( + ReportingProviderExecution.expires_at.is_not(None), + ReportingProviderExecution.expires_at <= observed_at, + ReportingProviderExecution.retention_redacted_at.is_(None), + ) + .limit(1) + .count() + ) + return counts + + +def _aware(value: datetime) -> datetime: + return value if value.tzinfo is not None else value.replace(tzinfo=UTC) + + +__all__ = ["ReportingRetentionService"] diff --git a/src/govoplan_reporting/backend/router.py b/src/govoplan_reporting/backend/router.py index e5f8aec..03f55dd 100644 --- a/src/govoplan_reporting/backend/router.py +++ b/src/govoplan_reporting/backend/router.py @@ -44,12 +44,23 @@ from govoplan_reporting.backend.operations import ( upsert_saved_view, upsert_schedule, ) +from govoplan_reporting.backend.provider_reports import ( + ProviderReportError, + execute_provider_report, + export_provider_execution, + get_provider_execution, + list_provider_exports, + list_provider_reports, + provider_parameter_options, +) from govoplan_reporting.backend.query_engine import ReportingQueryError from govoplan_reporting.backend.schemas import ( DefinitionUpdateRequest, DefinitionWriteRequest, ImportAssessmentRequest, PublicationRequest, + ProviderReportExecutionRequest, + ProviderReportExportRequest, QualityRunRequest, ReportExecutionRequest, SavedViewWriteRequest, @@ -60,6 +71,138 @@ from govoplan_reporting.backend.schemas import ( def create_router(registry: object | None) -> APIRouter: router = APIRouter(prefix="/reporting", tags=["reporting"]) + @router.get("/provider-reports") + def api_list_provider_reports( + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> dict[str, object]: + _require(principal, READ_SCOPE) + try: + return list_provider_reports(session, principal, registry=registry) + except (ProviderReportError, TypeError, ValueError) as exc: + raise _error(exc) from exc + + @router.get( + "/provider-reports/{provider_id}/{report_id}/parameters/{parameter_key}/options" + ) + def api_provider_parameter_options( + provider_id: str, + report_id: str, + parameter_key: str, + query: str = "", + limit: int = Query(default=100, ge=1, le=200), + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> dict[str, object]: + _require(principal, READ_SCOPE) + try: + options = provider_parameter_options( + session, + principal, + registry=registry, + provider_id=provider_id, + report_id=report_id, + parameter_key=parameter_key, + query=query, + limit=limit, + ) + except (ProviderReportError, PermissionError, LookupError) as exc: + raise _error(exc) from exc + return {"options": [item.to_dict() for item in options]} + + @router.post("/provider-reports/{provider_id}/{report_id}/executions") + def api_execute_provider_report( + provider_id: str, + report_id: str, + payload: ProviderReportExecutionRequest, + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> dict[str, object]: + _require(principal, RUN_SCOPE) + try: + result = execute_provider_report( + session, + principal, + registry=registry, + provider_id=provider_id, + report_id=report_id, + **payload.model_dump(), + ) + session.commit() + except (ProviderReportError, PermissionError, LookupError, ValueError) as exc: + session.rollback() + raise _error(exc) from exc + return result + + @router.get("/provider-executions/{execution_id}") + def api_get_provider_execution( + execution_id: str, + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> dict[str, object]: + _require(principal, RUN_SCOPE) + try: + result = get_provider_execution( + session, + principal, + registry=registry, + execution_id=execution_id, + ) + except (ProviderReportError, PermissionError, LookupError) as exc: + raise _error(exc) from exc + if result is None: + raise HTTPException( + status_code=404, detail="Provider report execution not found" + ) + return result + + @router.get("/provider-executions/{execution_id}/exports") + def api_list_provider_exports( + execution_id: str, + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> dict[str, object]: + _require(principal, RUN_SCOPE) + try: + return { + "exports": list( + list_provider_exports( + session, + principal, + registry=registry, + execution_id=execution_id, + ) + ) + } + except (ProviderReportError, PermissionError, LookupError) as exc: + raise _error(exc) from exc + + @router.post("/provider-executions/{execution_id}/exports") + def api_export_provider_execution( + execution_id: str, + payload: ProviderReportExportRequest, + session: Session = Depends(get_session), + principal: ApiPrincipal = Depends(get_api_principal), + ) -> Response: + _require(principal, RUN_SCOPE) + try: + content, media_type, filename = export_provider_execution( + session, + principal, + registry=registry, + execution_id=execution_id, + **payload.model_dump(), + ) + session.commit() + except (ProviderReportError, PermissionError, LookupError) as exc: + session.rollback() + raise _error(exc) from exc + return Response( + content=content, + media_type=media_type, + headers={"Content-Disposition": f'attachment; filename="{filename}"'}, + ) + @router.get("/definitions") def api_list_definitions( definition_kind: list[str] | None = Query(default=None), diff --git a/src/govoplan_reporting/backend/schemas.py b/src/govoplan_reporting/backend/schemas.py index ec211b7..237e686 100644 --- a/src/govoplan_reporting/backend/schemas.py +++ b/src/govoplan_reporting/backend/schemas.py @@ -422,6 +422,23 @@ class ReportExecutionRequest(BaseModel): idempotency_key: str = Field(min_length=1, max_length=255) +class ProviderReportExecutionRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + parameters: dict[str, Any] = Field(default_factory=dict) + purpose: str = Field(min_length=1, max_length=1_000) + audience_scope: dict[str, Any] = Field(min_length=1, max_length=50) + idempotency_key: str = Field(min_length=1, max_length=255) + + +class ProviderReportExportRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + + format: Literal["json", "csv"] = "json" + purpose: str = Field(min_length=1, max_length=1_000) + audience_scope: dict[str, Any] = Field(min_length=1, max_length=50) + + class SavedViewWriteRequest(BaseModel): model_config = ConfigDict(extra="forbid") @@ -489,6 +506,8 @@ __all__ = [ "ImportAssessmentRequest", "MeasureDefinition", "PublicationRequest", + "ProviderReportExecutionRequest", + "ProviderReportExportRequest", "QualityPlanDefinition", "QualityRunRequest", "ReportDefinition", diff --git a/tests/test_manifest.py b/tests/test_manifest.py index 2fdf948..80855a0 100644 --- a/tests/test_manifest.py +++ b/tests/test_manifest.py @@ -14,9 +14,15 @@ class ReportingManifestTests(unittest.TestCase): self.assertEqual("@govoplan/reporting-webui", manifest.frontend.package_name) self.assertIsNotNone(manifest.route_factory) self.assertIsNotNone(manifest.migration_spec) - self.assertEqual(4, len(manifest.provides_interfaces)) + self.assertEqual(5, len(manifest.provides_interfaces)) self.assertEqual(1, len(manifest.search_sources)) self.assertIn("dataflow", manifest.optional_dependencies) + self.assertIn("policy", manifest.optional_dependencies) + self.assertEqual("/reports", manifest.nav_items[0].path) + self.assertEqual( + {"/reports", "/reporting"}, + {route.path for route in manifest.frontend.routes}, + ) self.assertIn( "report execution and publication evidence", manifest.architecture.owned_concepts, diff --git a/tests/test_migrations.py b/tests/test_migrations.py new file mode 100644 index 0000000..b0994e0 --- /dev/null +++ b/tests/test_migrations.py @@ -0,0 +1,42 @@ +from __future__ import annotations + +from pathlib import Path +import tempfile + +from alembic.runtime.migration import MigrationContext +from sqlalchemy import create_engine, inspect + +from govoplan_core.db.migrations import migrate_database +from govoplan_reporting.backend.manifest import get_manifest + + +def test_fresh_migration_creates_provider_evidence_tables_and_current_head() -> None: + with tempfile.TemporaryDirectory( + prefix="govoplan-reporting-migration-" + ) as directory: + url = f"sqlite:///{Path(directory) / 'reporting.db'}" + migrate_database( + database_url=url, + enabled_modules=("reporting",), + manifest_factories=(get_manifest,), + ) + engine = create_engine(url) + try: + with engine.connect() as connection: + tables = set(inspect(connection).get_table_names()) + assert { + "reporting_provider_executions", + "reporting_provider_exports", + }.issubset(tables) + assert "b7c4e1a9d2f6" in set( + MigrationContext.configure(connection).get_current_heads() + ) + columns = { + item["name"] + for item in inspect(connection).get_columns( + "reporting_provider_executions" + ) + } + assert "retention_redacted_at" in columns + finally: + engine.dispose() diff --git a/tests/test_module_permutations.py b/tests/test_module_permutations.py new file mode 100644 index 0000000..dea2d82 --- /dev/null +++ b/tests/test_module_permutations.py @@ -0,0 +1,91 @@ +from __future__ import annotations + +import os +from pathlib import Path +import subprocess +import sys + +_ROOT = Path(__file__).resolve().parents[1] + + +def test_campaign_starts_and_contributes_its_provider_without_reporting() -> None: + script = """ +import importlib.abc +import sys + +class Blocker(importlib.abc.MetaPathFinder): + def find_spec(self, fullname, path=None, target=None): + if fullname == 'govoplan_reporting' or fullname.startswith('govoplan_reporting.'): + raise ModuleNotFoundError("Reporting is physically absent", name=fullname) + return None + +sys.meta_path.insert(0, Blocker()) +from govoplan_campaign.backend.manifest import get_manifest +manifest = get_manifest() +assert 'reporting' not in manifest.dependencies +assert 'reporting' in manifest.optional_dependencies +assert 'reporting.report_provider.campaigns' in manifest.capability_factories +""" + _run_probe(script) + + +def test_reporting_starts_without_campaign_and_owns_the_global_route() -> None: + script = """ +import importlib.abc +import sys + +class Blocker(importlib.abc.MetaPathFinder): + def find_spec(self, fullname, path=None, target=None): + if fullname == 'govoplan_campaign' or fullname.startswith('govoplan_campaign.'): + raise ModuleNotFoundError("Campaign is physically absent", name=fullname) + return None + +sys.meta_path.insert(0, Blocker()) +from govoplan_reporting.backend.manifest import get_manifest +manifest = get_manifest() +assert 'campaigns' not in manifest.dependencies +assert manifest.nav_items[0].path == '/reports' +assert {item.path for item in manifest.frontend.routes} == {'/reports', '/reporting'} +""" + _run_probe(script) + + +def test_reporting_with_campaign_uses_only_the_provider_contract() -> None: + script = """ +from govoplan_campaign.backend.manifest import get_manifest as campaign_manifest +from govoplan_core.core.reporting import REPORT_PROVIDER_CAPABILITY_PREFIX +from govoplan_reporting.backend.manifest import get_manifest as reporting_manifest + +reporting = reporting_manifest() +campaign = campaign_manifest() +provider_name = REPORT_PROVIDER_CAPABILITY_PREFIX + 'campaigns' +assert provider_name in campaign.capability_factories +assert provider_name in {item.name for item in campaign.provides_interfaces} +assert 'campaigns' not in reporting.dependencies +assert '/reports' not in {route.path for route in campaign.frontend.routes} +provider = campaign.capability_factories[provider_name](None) +assert provider.provider_id == 'campaigns' +assert provider.contract_version == '1.0' +""" + _run_probe(script) + + +def _run_probe(source: str) -> None: + environment = dict(os.environ) + environment["PYTHONPATH"] = os.pathsep.join( + ( + str(_ROOT / "src"), + str(_ROOT.parent / "govoplan-campaign" / "src"), + str(_ROOT.parent / "govoplan-core" / "src"), + environment.get("PYTHONPATH", ""), + ) + ) + result = subprocess.run( + [sys.executable, "-c", source], + cwd=_ROOT, + env=environment, + text=True, + capture_output=True, + check=False, + ) + assert result.returncode == 0, result.stderr or result.stdout diff --git a/tests/test_provider_reports.py b/tests/test_provider_reports.py new file mode 100644 index 0000000..647267b --- /dev/null +++ b/tests/test_provider_reports.py @@ -0,0 +1,336 @@ +from __future__ import annotations + +from datetime import UTC, datetime, timedelta +from types import SimpleNamespace + +from sqlalchemy import create_engine +from sqlalchemy.orm import Session + +from govoplan_core.core.reporting import ( + REPORT_PROVIDER_CAPABILITY_PREFIX, + REPORT_PROVIDER_CONTRACT_VERSION, + ReportDescriptor, + ReportParameterDescriptor, + ReportParameterOption, + ReportPrivacyTransform, + ReportProviderRequest, + ReportProviderResult, + ReportResultField, +) +from govoplan_core.db.base import Base +from govoplan_reporting.backend.db.models import ( + ReportingProviderExecution, + ReportingProviderExport, +) +from govoplan_reporting.backend.provider_reports import ( + ProviderReportError, + execute_provider_report, + export_provider_execution, + list_provider_reports, +) +from govoplan_reporting.backend.retention import ReportingRetentionService + + +class _Principal: + tenant_id = "tenant-1" + user = SimpleNamespace(id="analyst-1") + api_key = None + + +class _Provider: + provider_id = "example" + contract_version = REPORT_PROVIDER_CONTRACT_VERSION + + def __init__(self) -> None: + self.result_access = True + + def list_reports(self, session, principal): + del session, principal + return ( + ReportDescriptor( + provider_id=self.provider_id, + report_id="summary", + revision="example.summary.v1", + title="Example summary", + summary="A minimized example report.", + parameters=( + ReportParameterDescriptor( + key="source_id", + label="Source", + type="reference", + required=True, + options_from_provider=True, + ), + ), + result_schema=( + ReportResultField( + path="metric", + label="Metric", + type="integer", + group="Outcome", + ), + ), + privacy_transforms=( + ReportPrivacyTransform(id="aggregate", label="Aggregate"), + ), + export_formats=("json",), + reidentification_risk="low", + ), + ) + + def parameter_options(self, session, principal, **kwargs): + del session, principal, kwargs + return (ReportParameterOption(value="source-1", label="Source 1"),) + + def execute_report( + self, + session, + principal, + *, + request: ReportProviderRequest, + ): + del session + return ReportProviderResult( + report_id=request.report_id, + generated_at=datetime(2026, 8, 2, 10, 0, tzinfo=UTC), + payload={"metric": 12}, + source_revisions=( + { + "module_id": "example", + "resource_id": request.parameters["source_id"], + "revision_id": "7", + }, + ), + effective_scope={ + "tenant_id": principal.tenant_id, + "source_id": request.parameters["source_id"], + }, + applied_privacy_transforms=("aggregate",), + provenance={"executor": "example-v1"}, + ) + + def authorize_result(self, session, principal, **kwargs): + del session, principal, kwargs + return self.result_access + + +class _Registry: + def __init__(self, provider=None): + self.provider = provider or _Provider() + + def capability_names(self): + return (REPORT_PROVIDER_CAPABILITY_PREFIX + "example",) + + def has_capability(self, name): + return name == REPORT_PROVIDER_CAPABILITY_PREFIX + "example" + + def capability(self, name): + return self.provider if self.has_capability(name) else None + + +def test_provider_report_execution_is_governed_replayable_and_audited() -> None: + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + principal = _Principal() + registry = _Registry() + with Session(engine, expire_on_commit=False) as session: + catalogue = list_provider_reports(session, principal, registry=registry) + assert catalogue["reports"][0]["available"] is True + + execution = execute_provider_report( + session, + principal, + registry=registry, + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="provider-run-1", + ) + replay = execute_provider_report( + session, + principal, + registry=registry, + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="provider-run-1", + ) + assert replay["execution_id"] == execution["execution_id"] + assert execution["source_revisions"][0]["revision_id"] == "7" + assert execution["privacy_transforms"] == ["aggregate"] + assert execution["retention_days"] == 30 + + content, media_type, filename = export_provider_execution( + session, + principal, + registry=registry, + execution_id=str(execution["execution_id"]), + format="json", + purpose="Archive approved aggregate", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + ) + assert b'"metric": 12' in content + assert media_type == "application/json" + assert filename.endswith(".json") + assert session.query(ReportingProviderExport).count() == 1 + engine.dispose() + + +def test_provider_result_missing_a_required_transform_is_rejected() -> None: + class _UnsafeProvider(_Provider): + def execute_report(self, session, principal, *, request): + result = super().execute_report( + session, + principal, + request=request, + ) + return ReportProviderResult( + report_id=result.report_id, + generated_at=result.generated_at, + payload=result.payload, + source_revisions=result.source_revisions, + effective_scope=result.effective_scope, + applied_privacy_transforms=(), + provenance=result.provenance, + ) + + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + with Session(engine) as session: + try: + execute_provider_report( + session, + _Principal(), + registry=_Registry(_UnsafeProvider()), + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant"}, + idempotency_key="unsafe-provider-run", + ) + except ProviderReportError as exc: + assert "privacy transformations" in str(exc) + else: + raise AssertionError("unsafe provider report was accepted") + engine.dispose() + + +def test_expired_provider_result_is_minimized_but_evidence_is_retained() -> None: + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + with Session(engine, expire_on_commit=False) as session: + execution = execute_provider_report( + session, + _Principal(), + registry=_Registry(), + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="expired-provider-run", + ) + row = session.query(ReportingProviderExecution).one() + original_hash = row.output_hash + row.expires_at = datetime.now(UTC) - timedelta(seconds=1) + session.flush() + + preview = ReportingRetentionService().apply_retention( + session, + dry_run=True, + now=datetime.now(UTC), + ) + assert preview["eligible"] == 1 + assert row.result_payload == {"metric": 12} + + applied = ReportingRetentionService().apply_retention( + session, + dry_run=False, + now=datetime.now(UTC), + ) + assert applied["redacted"] == 1 + assert row.result_payload == {} + assert row.retention_redacted_at is not None + assert row.output_hash == original_hash == execution["output_hash"] + assert row.source_revisions[0]["revision_id"] == "7" + try: + execute_provider_report( + session, + _Principal(), + registry=_Registry(), + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="expired-provider-run", + ) + except ProviderReportError as exc: + assert "expired" in str(exc) + else: + raise AssertionError("expired provider-report detail was replayed") + engine.dispose() + + +def test_provider_parameters_are_checked_against_the_declared_type() -> None: + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + with Session(engine) as session: + try: + execute_provider_report( + session, + _Principal(), + registry=_Registry(), + provider_id="example", + report_id="summary", + parameters={"source_id": 123}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="invalid-provider-parameters", + ) + except ProviderReportError as exc: + assert "parameter types" in str(exc) + else: + raise AssertionError("invalid provider-report parameters were accepted") + engine.dispose() + + +def test_source_access_is_rechecked_before_retained_result_export() -> None: + engine = create_engine("sqlite+pysqlite:///:memory:") + Base.metadata.create_all(engine) + provider = _Provider() + registry = _Registry(provider) + with Session(engine) as session: + execution = execute_provider_report( + session, + _Principal(), + registry=registry, + provider_id="example", + report_id="summary", + parameters={"source_id": "source-1"}, + purpose="Operational overview", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + idempotency_key="source-access-provider-run", + ) + provider.result_access = False + + try: + export_provider_execution( + session, + _Principal(), + registry=registry, + execution_id=str(execution["execution_id"]), + format="json", + purpose="Archive aggregate", + audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"}, + ) + except PermissionError as exc: + assert "source module" in str(exc) + else: + raise AssertionError("revoked source result was exported") + engine.dispose() diff --git a/webui/src/api/reporting.ts b/webui/src/api/reporting.ts index 7349ef0..9efe755 100644 --- a/webui/src/api/reporting.ts +++ b/webui/src/api/reporting.ts @@ -119,6 +119,74 @@ export type ReportingSavedView = { updated_at: string; }; +export type ProviderReportParameter = { + key: string; + label: string; + type: string; + required: boolean; + description?: string | null; + options_from_provider: boolean; +}; + +export type ProviderReportField = { + path: string; + label: string; + type: string; + group: string; + nullable: boolean; + sensitive: boolean; +}; + +export type ProviderReportDescriptor = { + contract_version: string; + provider_id: string; + report_id: string; + revision: string; + title: string; + summary: string; + parameters: ProviderReportParameter[]; + result_schema: ProviderReportField[]; + privacy_transforms: Array<{ id: string; label: string; required: boolean }>; + purpose_required: boolean; + audience_scope_required: boolean; + retention_class: string; + export_formats: Array<"json" | "csv">; + reidentification_risk: "low" | "moderate" | "high"; + presentation: { kind?: string }; + available: boolean; + unavailable_reason?: string | null; + governance: { + retention_days?: number | null; + export_formats: string[]; + required_privacy_transforms: string[]; + provenance: Record; + }; +}; + +export type ProviderReportExecution = { + execution_id: string; + provider_id: string; + report_id: string; + report_revision: string; + contract_version: string; + purpose: string; + audience_scope: Record; + parameters: Record; + result_schema: ProviderReportField[]; + result: Record; + source_revisions: Array>; + effective_scope: Record; + privacy_transforms: string[]; + provenance: Record; + governance_provenance: Record; + retention_class: string; + retention_days?: number | null; + expires_at?: string | null; + output_hash: string; + generated_at: string; + actor_id?: string | null; +}; + export function listDefinitions( settings: ApiSettings, options: { kinds?: ReportingDefinitionKind[]; status?: string[]; query?: string; limit?: number }, @@ -132,6 +200,75 @@ export function listDefinitions( }), { signal }); } +export function listProviderReports( + settings: ApiSettings, + signal?: AbortSignal +): Promise<{ reports: ProviderReportDescriptor[]; diagnostics: Array> }> { + return apiFetch(settings, "/api/v1/reporting/provider-reports", { signal }); +} + +export function listProviderParameterOptions( + settings: ApiSettings, + report: ProviderReportDescriptor, + parameterKey: string, + query = "", + signal?: AbortSignal +): Promise<{ options: Array<{ value: string; label: string; description?: string | null }> }> { + return apiFetch(settings, apiPath( + `/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/parameters/${encodeURIComponent(parameterKey)}/options`, + { query, limit: 200 } + ), { signal }); +} + +export function runProviderReport( + settings: ApiSettings, + report: ProviderReportDescriptor, + parameters: Record, + purpose: string, + audienceScope: Record +): Promise { + return apiFetch( + settings, + `/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/executions`, + { + method: "POST", + body: JSON.stringify({ + parameters, + purpose, + audience_scope: audienceScope, + idempotency_key: crypto.randomUUID() + }) + } + ); +} + +export async function downloadProviderExecution( + settings: ApiSettings, + execution: ProviderReportExecution, + format: "json" | "csv", + purpose: string, + audienceScope: Record +): Promise { + const response = await fetch( + apiUrl(settings, `/api/v1/reporting/provider-executions/${encodeURIComponent(execution.execution_id)}/exports`), + { + method: "POST", + headers: { ...authHeaders(settings), "Content-Type": "application/json" }, + credentials: "include", + body: JSON.stringify({ format, purpose, audience_scope: audienceScope }) + } + ); + if (!response.ok) throw new Error(await response.text()); + const blob = await response.blob(); + const disposition = response.headers.get("Content-Disposition") ?? ""; + const filename = disposition.match(/filename="?([^";]+)"?/)?.[1] ?? `provider-report.${format}`; + const link = document.createElement("a"); + link.href = URL.createObjectURL(blob); + link.download = filename; + link.click(); + URL.revokeObjectURL(link.href); +} + export function getDefinition( settings: ApiSettings, kind: ReportingDefinitionKind, diff --git a/webui/src/features/reporting/ProviderReportWorkspace.tsx b/webui/src/features/reporting/ProviderReportWorkspace.tsx new file mode 100644 index 0000000..49ba14b --- /dev/null +++ b/webui/src/features/reporting/ProviderReportWorkspace.tsx @@ -0,0 +1,295 @@ +import { Download, FileJson, Play, ShieldCheck } from "lucide-react"; +import { useEffect, useMemo, useState } from "react"; +import { + Button, + Card, + DismissibleAlert, + IconButton, + MetricCard, + StatusBadge, + type ApiSettings, + type AuthInfo +} from "@govoplan/core-webui"; +import { + downloadProviderExecution, + listProviderParameterOptions, + runProviderReport, + type ProviderReportDescriptor, + type ProviderReportExecution, + type ProviderReportField +} from "../../api/reporting"; + + +export function ProviderReportWorkspace({ settings, auth, report }: { + settings: ApiSettings; + auth: AuthInfo; + report: ProviderReportDescriptor; +}) { + const [parameters, setParameters] = useState>({}); + const [options, setOptions] = useState>>({}); + const [purpose, setPurpose] = useState(""); + const [execution, setExecution] = useState(null); + const [running, setRunning] = useState(false); + const [error, setError] = useState(""); + const tenant = auth.active_tenant ?? auth.tenant; + const audienceScope = useMemo(() => ({ + scope_type: "tenant", + scope_id: tenant.id, + label: tenant.name + }), [tenant.id, tenant.name]); + + useEffect(() => { + setParameters({}); + setPurpose(""); + setExecution(null); + setError(""); + const controller = new AbortController(); + const optionParameters = report.parameters.filter((item) => item.options_from_provider); + void Promise.all(optionParameters.map(async (parameter) => { + const result = await listProviderParameterOptions( + settings, + report, + parameter.key, + "", + controller.signal + ); + return [parameter.key, result.options] as const; + })).then((entries) => { + setOptions(Object.fromEntries(entries)); + setParameters(Object.fromEntries(entries.flatMap(([key, values]) => + values[0] ? [[key, values[0].value]] : [] + ))); + }).catch((reason) => { + if ((reason as Error).name !== "AbortError") setError(message(reason)); + }); + return () => controller.abort(); + }, [settings, report.provider_id, report.report_id, report.revision]); + + async function run() { + setRunning(true); + setError(""); + try { + setExecution(await runProviderReport( + settings, + report, + parameters, + purpose.trim(), + audienceScope + )); + } catch (reason) { + setError(message(reason)); + } finally { + setRunning(false); + } + } + + async function download(format: "json" | "csv") { + if (!execution) return; + try { + await downloadProviderExecution( + settings, + execution, + format, + purpose.trim(), + audienceScope + ); + } catch (reason) { + setError(message(reason)); + } + } + + const missingRequired = report.parameters.some((item) => + item.required && (parameters[item.key] === undefined || parameters[item.key] === "") + ); + return ( + <> +
+
+ {report.provider_id} · {report.revision} +

{report.title}

+

{report.summary}

+
+
+ +
+
+ {error && {error}} + {!report.available && + + {report.unavailable_reason ?? "Policy does not allow this report."} + + } +
+ {report.parameters.map((parameter) => + + )} + + +
+ {execution ? + <> +
+ Generated {formatDateTime(execution.generated_at)} + {report.export_formats.includes("csv") && + } variant="ghost" onClick={() => void download("csv")} /> + } + {report.export_formats.includes("json") && + } variant="ghost" onClick={() => void download("json")} /> + } +
+
+ +
+ : +
Select the parameters and run this governed report.
+ } + + ); +} + + +export function ProviderReportInspector({ report }: { report: ProviderReportDescriptor }) { + return ( +
+
+

Governance

+
+
Risk
+
Retention
{report.governance.retention_days == null ? "Policy managed" : `${report.governance.retention_days} days`}
+
Exports
{report.governance.export_formats.join(", ") || "Disabled"}
+
Contract
{report.contract_version}
+
+
+
+

Privacy transforms

+ {report.privacy_transforms.map((item) => +

{item.label}{item.required ? " · required" : ""}

+ )} +
+
+ ); +} + + +function ProviderResult({ execution }: { execution: ProviderReportExecution }) { + const groups = groupFields(execution.result_schema); + return ( + <> + {[...groups].map(([group, fields]) => { + const metrics = fields.filter((field) => field.type === "suppressed_count"); + const details = fields.filter((field) => field.type !== "suppressed_count"); + return ( + + {metrics.length > 0 && +
+ {metrics.map((field) => + + )} +
+ } + {details.length > 0 && +
+ {details.map((field) => +
+
{field.label}
+
{displayValue(pathValue(execution.result, field.path), field)}
+
+ )} +
+ } + + ); + })} + +
+
Purpose
{execution.purpose}
+
Output hash
{shortHash(execution.output_hash)}
+
Source revisions
{execution.source_revisions.length}
+
Privacy transforms
{execution.privacy_transforms.join(", ")}
+
Expires
{execution.expires_at ? formatDateTime(execution.expires_at) : "Policy managed"}
+
+ + + ); +} + + +function groupFields(fields: ProviderReportField[]): Map { + const groups = new Map(); + for (const field of fields) groups.set(field.group, [...(groups.get(field.group) ?? []), field]); + return groups; +} + +function pathValue(payload: Record, path: string): unknown { + let value: unknown = payload; + for (const part of path.split(".")) { + if (!value || typeof value !== "object") return null; + value = (value as Record)[part]; + } + return value; +} + +function displayValue(value: unknown, field?: ProviderReportField): string | number { + if (value && typeof value === "object") { + const count = value as { value?: unknown; suppressed?: boolean }; + if (count.suppressed) return "Suppressed"; + if ("value" in count) return displayValue(count.value); + return JSON.stringify(value); + } + if (value === null || value === undefined || value === "") return "—"; + if (field?.type === "datetime" || field?.type === "date") return formatDateTime(String(value)); + if (typeof value === "number") return new Intl.NumberFormat().format(value); + if (typeof value === "boolean") return value ? "Yes" : "No"; + return String(value); +} + +function formatDateTime(value: string): string { + const date = new Date(value); + return Number.isNaN(date.valueOf()) ? value : new Intl.DateTimeFormat(undefined, { dateStyle: "medium", timeStyle: "short" }).format(date); +} + +function humanize(value: string): string { + return value.replaceAll("_", " ").replace(/\b\w/g, (letter) => letter.toUpperCase()); +} + +function shortHash(value: string): string { + return `${value.slice(0, 10)}…${value.slice(-6)}`; +} + +function message(reason: unknown): string { + return reason instanceof Error ? reason.message : "The provider report could not be loaded."; +} diff --git a/webui/src/features/reporting/ReportingPage.tsx b/webui/src/features/reporting/ReportingPage.tsx index daf0992..3ac4d93 100644 --- a/webui/src/features/reporting/ReportingPage.tsx +++ b/webui/src/features/reporting/ReportingPage.tsx @@ -37,6 +37,7 @@ import { getDefinition, listDefinitions, listExecutions, + listProviderReports, listSavedViews, reportPayload, runReport, @@ -47,8 +48,13 @@ import { type ReportingQuery, type ReportingQueryMode, type ReportingSavedView, + type ProviderReportDescriptor, type SemanticModelPayload } from "../../api/reporting"; +import { + ProviderReportInspector, + ProviderReportWorkspace +} from "./ProviderReportWorkspace"; type OutputMode = "visual" | "table"; @@ -57,7 +63,9 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext) const [search, setSearch] = useState(""); const [submittedSearch, setSubmittedSearch] = useState(""); const [reports, setReports] = useState([]); + const [providerReports, setProviderReports] = useState([]); const [selectedId, setSelectedId] = useState(""); + const [selectedProviderKey, setSelectedProviderKey] = useState(""); const [semantic, setSemantic] = useState(null); const [query, setQuery] = useState(emptyQuery()); const [parameters, setParameters] = useState>({}); @@ -77,21 +85,42 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext) () => reports.find((item) => item.definition_id === selectedId) ?? null, [reports, selectedId] ); + const selectedProvider = useMemo( + () => providerReports.find((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey) ?? null, + [providerReports, selectedProviderKey] + ); const report = selected ? reportPayload(selected) : null; function reload(signal?: AbortSignal) { setLoading(true); setError(""); - return listDefinitions(settings, { - kinds: ["report"], - status: ["active"], - query: submittedSearch, - limit: 200 - }, signal). - then((result) => { + return Promise.all([ + listDefinitions(settings, { + kinds: ["report"], + status: ["active"], + query: submittedSearch, + limit: 200 + }, signal), + listProviderReports(settings, signal) + ]). + then(([result, providerResult]) => { + const providerRows = providerResult.reports.filter((item) => { + const query = submittedSearch.toLocaleLowerCase(); + return !query || `${item.title} ${item.summary} ${item.provider_id}`.toLocaleLowerCase().includes(query); + }); setReports(result.definitions); - setSelectedId((current) => result.definitions.some((item) => item.definition_id === current) ? - current : result.definitions[0]?.definition_id ?? ""); + setProviderReports(providerRows); + const currentSemanticAvailable = result.definitions.some((item) => item.definition_id === selectedId); + const currentProviderAvailable = providerRows.some((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey); + if (!currentSemanticAvailable && !currentProviderAvailable) { + if (providerRows[0]) { + setSelectedProviderKey(`${providerRows[0].provider_id}:${providerRows[0].report_id}`); + setSelectedId(""); + } else { + setSelectedId(result.definitions[0]?.definition_id ?? ""); + setSelectedProviderKey(""); + } + } }). catch((reason) => { if ((reason as Error).name !== "AbortError") setError(message(reason)); @@ -172,7 +201,7 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext) placeholder="Search reports" /> - {reports.length} reports + {reports.length + providerReports.length} reports } @@ -188,15 +217,32 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
{loading && } - {!loading && reports.length === 0 &&
No active reports are available.
} + {!loading && reports.length + providerReports.length === 0 &&
No active reports are available.
}
+ {providerReports.length > 0 &&
Module reports
} + {providerReports.map((item) => { + const key = `${item.provider_id}:${item.report_id}`; + return ( + + ); + })} + {reports.length > 0 &&
Semantic reports
} {reports.map((item) =>
- {selected && report ? + {selectedProvider ? + : + selected && report ? <>
@@ -258,14 +306,16 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext) }
- : + + />}
diff --git a/webui/src/module.ts b/webui/src/module.ts index 74001cc..efdbdd5 100644 --- a/webui/src/module.ts +++ b/webui/src/module.ts @@ -24,16 +24,23 @@ export const reportingModule: PlatformWebModule = { ], routes: [ { - path: "/reporting", + path: "/reports", anyOf: ["reporting:definition:read"], order: 74, surfaceId: "reporting.workspace", render: (context) => createElement(ReportingPage, context) + }, + { + path: "/reporting", + anyOf: ["reporting:definition:read"], + order: 175, + surfaceId: "reporting.compatibility", + render: (context) => createElement(ReportingPage, context) } ], navItems: [ { - to: "/reporting", + to: "/reports", label: "Reporting", iconName: "clipboard-pen-line", anyOf: ["reporting:definition:read"], diff --git a/webui/src/styles/reporting.css b/webui/src/styles/reporting.css index fc1edd2..95588d0 100644 --- a/webui/src/styles/reporting.css +++ b/webui/src/styles/reporting.css @@ -74,6 +74,16 @@ background: var(--surface-raised); } +.reporting-list-heading { + padding: 7px 10px; + border-bottom: 1px solid var(--border); + background: var(--surface-subtle, var(--surface)); + color: var(--text-soft); + font-size: 0.7rem; + font-weight: 700; + text-transform: uppercase; +} + .reporting-report-row, .reporting-inspector-content section > button { display: grid; @@ -162,6 +172,16 @@ white-space: nowrap; } +.reporting-title p { + max-width: 70ch; + margin: 3px 0 0; + overflow: hidden; + color: var(--text-soft); + font-size: 0.76rem; + text-overflow: ellipsis; + white-space: nowrap; +} + .reporting-run-actions { display: flex; align-items: center; @@ -223,6 +243,14 @@ gap: 4px; } +.reporting-provider-controls { + align-items: end; +} + +.reporting-provider-purpose { + min-width: min(320px, 35vw); +} + .reporting-output-toolbar { min-height: 48px; padding: 6px 12px; @@ -241,6 +269,21 @@ padding: 10px; } +.reporting-provider-output { + display: flex; + min-height: 0; + flex: 1; + flex-direction: column; + gap: 10px; + overflow: auto; + padding: 10px; +} + +.reporting-provider-metrics { + margin: 0; + padding: 0; +} + .reporting-output .data-grid-shell { min-width: 100%; }