feat: add governed cross-module reporting

This commit is contained in:
2026-08-02 05:29:38 +02:00
parent eba35edd3c
commit 0abcc2455e
21 changed files with 2497 additions and 38 deletions
+7
View File
@@ -20,6 +20,9 @@ The module now provides an executable governed semantic-reporting vertical:
arbitrary report SQL; arbitrary report SQL;
- quality gates, saved views, interval/scheduled runs, CSV/JSON export, - quality gates, saved views, interval/scheduled runs, CSV/JSON export,
provider-neutral publication targets, and import activation assessments; provider-neutral publication targets, and import activation assessments;
- a versioned, provider-neutral cross-module report contract with source-owned
authorization, declared result schemas, privacy transforms, effective scope,
source revisions, purpose, retention, export history, and audit provenance;
- a full-height Reporting workspace for running, inspecting, saving, - a full-height Reporting workspace for running, inspecting, saving,
scheduling, visualizing, and exporting authorized reports. scheduling, visualizing, and exporting authorized reports.
@@ -27,6 +30,10 @@ Reporting consumes Dataflow outputs or provider-owned read models. It does not
read another module's ORM tables or take ownership of ingestion and read another module's ORM tables or take ownership of ingestion and
transformation. transformation.
The canonical global route is `/reports`. `/reporting` remains a
Reporting-owned compatibility route for saved links. Domain modules may keep
their own operational report routes, but do not register `/reports`.
See [docs/REPORTING_BOUNDARY.md](docs/REPORTING_BOUNDARY.md) for the boundary See [docs/REPORTING_BOUNDARY.md](docs/REPORTING_BOUNDARY.md) for the boundary
decision. The behavior-level comparison with the supplied SuperX module set is decision. The behavior-level comparison with the supplied SuperX module set is
recorded in recorded in
+30
View File
@@ -32,6 +32,36 @@ the Reporting publication-target contract. The target receives one immutable
execution payload and an idempotency key. It must return bounded evidence and execution payload and an idempotency key. It must return bounded evidence and
must not expose credentials in that evidence. must not expose credentials in that evidence.
## Cross-module provider governance
Source modules register `reporting.report_provider.<provider-id>` capabilities;
do not grant Reporting direct table access to those modules. Review every
descriptor's result schema, required privacy transforms, retention class,
export formats, and re-identification risk before enabling it in production.
Provider authorization remains mandatory even when the Reporting role allows
the user to run reports.
When Policy is enabled, configure `reporting_governance_policy` in system or
tenant settings. Tenant settings may only tighten the system result. Supported
fields are:
```json
{
"reporting_governance_policy": {
"allow_exports": true,
"allowed_export_formats": ["json"],
"allow_high_reidentification_risk": false,
"max_retention_days": 30,
"required_privacy_transforms": ["small_cell_suppression"]
}
}
```
Malformed explicit policy fails closed. The ordinary privacy-retention setting
`stored_report_detail_retention_days` is an additional ceiling. A tenant cannot
re-enable an export format, high-risk report, or longer retention period that
the system policy denied.
## Import assessments ## Import assessments
Import assessment accepts declarative metadata only. Native datasets, Import assessment accepts declarative metadata only. Native datasets,
+13
View File
@@ -12,6 +12,15 @@ claim bounded batches, retain success or failure state, and advance interval
time from the claimed occurrence. Providers must use the same tenant and time from the claimed occurrence. Providers must use the same tenant and
principal context and must enforce their source-side authorization. principal context and must enforce their source-side authorization.
Provider-report executions and exports are database evidence. Each execution
stores the exact provider/report revision, request hash, source revisions,
effective scope, applied privacy transforms, governance decision provenance,
generation actor/time, retention expiry, and output hash. Export rows retain
their own purpose, audience, actor/time, format, and hash. Expired provider
results are rejected by retrieval and export paths; run the platform privacy
retention process on schedule to remove or minimize expired stored detail in
accordance with the active Policy.
## Security controls ## Security controls
- The semantic engine never evaluates Python, browser code, raw SQL, stored - The semantic engine never evaluates Python, browser code, raw SQL, stored
@@ -25,6 +34,10 @@ principal context and must enforce their source-side authorization.
- Freshness, schema, fingerprints, quality gates, definition hashes, source - Freshness, schema, fingerprints, quality gates, definition hashes, source
provenance, actor, and output hash are retained with every run. provenance, actor, and output hash are retained with every run.
- CSV formula injection is neutralized before direct download. - CSV formula injection is neutralized before direct download.
- Provider exceptions are isolated in catalogue diagnostics; an unavailable
optional provider cannot take down the global report catalogue.
- High re-identification-risk provider reports fail closed unless an installed
Policy capability explicitly permits them.
## Backup and restore ## Backup and restore
+28
View File
@@ -87,6 +87,29 @@ Reporting does not own:
- `reporting.publication_target:*` capabilities can accept immutable result - `reporting.publication_target:*` capabilities can accept immutable result
payloads without Reporting importing the target module. payloads without Reporting importing the target module.
### Cross-module report providers
Source modules contribute aggregate or otherwise minimized reports through
`reporting.report_provider.<provider-id>`. Core owns contract version `1.0` and
the provider-neutral DTOs. A provider must declare stable report and revision
IDs, typed parameters, the complete result schema, privacy transforms,
retention class, export formats, and re-identification risk. It must enforce
source access before listing options or producing a result and must return
source revisions, effective tenant scope, applied transforms, generation time,
and bounded provenance.
Reporting discovers these capabilities by prefix. It validates descriptors,
rejects undeclared output fields, requires every mandatory transform, records
purpose and effective audience, and persists immutable execution and export
evidence. The optional `policy.reporting_governance` capability can tighten
retention, export formats, required transforms, and high-risk handling. If
Policy is absent, Reporting applies its restrictive built-in baseline and
denies high-risk provider reports.
This direction is intentionally one-way: source modules import only Core's
contract and remain usable when Reporting is absent. Reporting never imports
the source module or reads its ORM tables.
## Implemented Vertical ## Implemented Vertical
The first complete vertical persists datasets, semantic models, reports, The first complete vertical persists datasets, semantic models, reports,
@@ -120,6 +143,11 @@ unsupported executable behavior remains.
The WebUI uses the platform module loader and common controls. It exposes a The WebUI uses the platform module loader and common controls. It exposes a
report catalogue, parameter and semantic-query controls, result visualization report catalogue, parameter and semantic-query controls, result visualization
and table views, history/provenance, saved views, schedules, and downloads. and table views, history/provenance, saved views, schedules, and downloads.
The global `/reports` route is owned only by Reporting. `/reporting` is a
documented compatibility path. Campaign's module-local aggregate view remains
at `/campaigns/reports`; when both modules are enabled, the same safe aggregate
projection is also contributed to the global catalogue through the provider
contract.
## Remaining Product Depth ## Remaining Product Depth
+18 -2
View File
@@ -2,8 +2,9 @@
## Open and run a report ## Open and run a report
Open **Reporting** and choose an active report from the left catalogue. The Open **Reporting** at `/reports` and choose a report from the left catalogue.
header identifies the immutable report revision that will be used. Select a The catalogue separates source-module reports from Reporting semantic reports.
The header identifies the immutable report revision that will be used. Select a
summary, detail, or pivot shape, choose dimensions and measures, enter any summary, detail, or pivot shape, choose dimensions and measures, enter any
report parameters, and select **Run**. report parameters, and select **Run**.
@@ -30,3 +31,18 @@ spreadsheet formula marker are escaped.
Users with scheduling permission can create an hourly, daily, weekly, or Users with scheduling permission can create an hourly, daily, weekly, or
30-day interval from the current revision, parameters, and query. Scheduled 30-day interval from the current revision, parameters, and query. Scheduled
runs continue to use those exact pins until the schedule is edited. runs continue to use those exact pins until the schedule is edited.
## Run a module report
Module reports retain their source module's access rules. Select the source
object and any optional parameters, enter the concrete purpose for the run,
and verify the effective tenant audience before selecting **Run**. Reporting
will refuse a result that does not carry the declared privacy transforms,
source revision, and tenant scope.
The result groups declared metrics and details without exposing undeclared
provider data. The Governance panel explains the risk class, permitted export
formats, retention ceiling, contract version, and required privacy transforms.
An export records a separate purpose, actor, audience, time, format, and output
hash. A missing or restricted module report is not replaced with a less
protected client-side query.
+121
View File
@@ -218,6 +218,125 @@ class ReportingExecution(Base, TimestampMixin):
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True) actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
class ReportingProviderExecution(Base, TimestampMixin):
__tablename__ = "reporting_provider_executions"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"execution_id",
name="uq_reporting_provider_execution",
),
UniqueConstraint(
"tenant_id",
"idempotency_key",
name="uq_reporting_provider_execution_idempotency",
),
Index(
"ix_reporting_provider_execution_history",
"tenant_id",
"provider_id",
"report_id",
"generated_at",
),
Index(
"ix_reporting_provider_execution_retention",
"tenant_id",
"expires_at",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
provider_id: Mapped[str] = mapped_column(String(120), nullable=False, index=True)
report_id: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
report_revision: Mapped[str] = mapped_column(String(255), nullable=False)
contract_version: Mapped[str] = mapped_column(String(30), nullable=False)
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
purpose: Mapped[str] = mapped_column(Text, nullable=False)
audience_scope: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
parameters: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
result_schema: Mapped[list[dict[str, Any]]] = mapped_column(
JSON, default=list, nullable=False
)
result_payload: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
source_revisions: Mapped[list[dict[str, Any]]] = mapped_column(
JSON, default=list, nullable=False
)
effective_scope: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
privacy_transforms: Mapped[list[str]] = mapped_column(
JSON, default=list, nullable=False
)
provenance: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
governance_provenance: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
retention_class: Mapped[str] = mapped_column(String(120), nullable=False)
retention_days: Mapped[int | None] = mapped_column(Integer, nullable=True)
expires_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True
)
retention_redacted_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True), nullable=True, index=True
)
output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
generated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True
)
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
class ReportingProviderExport(Base, TimestampMixin):
__tablename__ = "reporting_provider_exports"
__table_args__ = (
UniqueConstraint(
"tenant_id",
"export_id",
name="uq_reporting_provider_export",
),
Index(
"ix_reporting_provider_export_history",
"tenant_id",
"provider_execution_id",
"exported_at",
),
)
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
export_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
provider_execution_id: Mapped[str] = mapped_column(
ForeignKey(
"reporting_provider_executions.id",
ondelete="RESTRICT",
),
nullable=False,
index=True,
)
execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
format: Mapped[str] = mapped_column(String(30), nullable=False)
purpose: Mapped[str] = mapped_column(Text, nullable=False)
audience_scope: Mapped[dict[str, Any]] = mapped_column(
JSON, default=dict, nullable=False
)
output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
exported_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), nullable=False, index=True
)
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
class ReportingDefinitionGrant(Base, TimestampMixin): class ReportingDefinitionGrant(Base, TimestampMixin):
__tablename__ = "reporting_definition_grants" __tablename__ = "reporting_definition_grants"
__table_args__ = ( __table_args__ = (
@@ -435,6 +554,8 @@ __all__ = [
"ReportingExecution", "ReportingExecution",
"ReportingImportAssessment", "ReportingImportAssessment",
"ReportingPublication", "ReportingPublication",
"ReportingProviderExecution",
"ReportingProviderExport",
"ReportingQualityResult", "ReportingQualityResult",
"ReportingSavedView", "ReportingSavedView",
"ReportingSchedule", "ReportingSchedule",
+47 -18
View File
@@ -31,6 +31,10 @@ from govoplan_core.core.provider_governance import (
ModuleArchitectureDocumentation, ModuleArchitectureDocumentation,
ModuleMaturityEvidence, ModuleMaturityEvidence,
) )
from govoplan_core.core.reporting import (
CAPABILITY_POLICY_REPORTING_GOVERNANCE,
CAPABILITY_REPORTING_RETENTION,
)
from govoplan_core.core.search import SearchSourceProviderRegistration from govoplan_core.core.search import SearchSourceProviderRegistration
from govoplan_core.core.views import ViewSurface from govoplan_core.core.views import ViewSurface
from govoplan_core.db.base import Base from govoplan_core.db.base import Base
@@ -171,6 +175,13 @@ def _chart_renderer(context: ModuleContext) -> DefaultChartRenderer:
return DefaultChartRenderer() return DefaultChartRenderer()
def _retention(context: ModuleContext):
del context
from govoplan_reporting.backend.retention import ReportingRetentionService
return ReportingRetentionService()
def _tenant_summary(session, tenant_id: str) -> dict[str, int]: def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
definitions = ( definitions = (
session.query(reporting_models.ReportingDefinitionRevision) session.query(reporting_models.ReportingDefinitionRevision)
@@ -232,17 +243,21 @@ manifest = ModuleManifest(
CAPABILITY_AUTH_PRINCIPAL_RESOLVER, CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
CAPABILITY_AUTH_PERMISSION_EVALUATOR, CAPABILITY_AUTH_PERMISSION_EVALUATOR,
), ),
optional_capabilities=(CAPABILITY_DATAFLOW_DATASET_OUTPUT,), optional_capabilities=(
CAPABILITY_DATAFLOW_DATASET_OUTPUT,
CAPABILITY_POLICY_REPORTING_GOVERNANCE,
),
permissions=PERMISSIONS, permissions=PERMISSIONS,
role_templates=ROLE_TEMPLATES, role_templates=ROLE_TEMPLATES,
route_factory=_router, route_factory=_router,
nav_items=( nav_items=(
NavItem( NavItem(
path="/reporting", path="/reports",
label="Reporting", label="Reporting",
icon="clipboard-pen-line", icon="clipboard-pen-line",
required_any=(READ_SCOPE,), required_any=(READ_SCOPE,),
order=74, order=74,
surface_id="reporting.navigation",
), ),
), ),
frontend=FrontendModule( frontend=FrontendModule(
@@ -250,36 +265,31 @@ manifest = ModuleManifest(
package_name="@govoplan/reporting-webui", package_name="@govoplan/reporting-webui",
routes=( routes=(
FrontendRoute( FrontendRoute(
path="/reporting", path="/reports",
component="ReportingPage", component="ReportingPage",
required_any=(READ_SCOPE,), required_any=(READ_SCOPE,),
order=74, order=74,
surface_id="reporting.workspace",
),
FrontendRoute(
path="/reporting",
component="ReportingPage",
required_any=(READ_SCOPE,),
order=175,
surface_id="reporting.compatibility",
), ),
), ),
nav_items=( nav_items=(
NavItem( NavItem(
path="/reporting", path="/reports",
label="Reporting", label="Reporting",
icon="clipboard-pen-line", icon="clipboard-pen-line",
required_any=(READ_SCOPE,), required_any=(READ_SCOPE,),
order=74, order=74,
surface_id="reporting.navigation",
), ),
), ),
view_surfaces=( view_surfaces=(
ViewSurface(
id="reporting.navigation",
module_id=MODULE_ID,
kind="navigation",
label="Reporting navigation",
order=10,
),
ViewSurface(
id="reporting.workspace",
module_id=MODULE_ID,
kind="route",
label="Reporting workspace",
order=20,
),
ViewSurface( ViewSurface(
id="reporting.parameters", id="reporting.parameters",
module_id=MODULE_ID, module_id=MODULE_ID,
@@ -303,6 +313,7 @@ manifest = ModuleManifest(
ModuleInterfaceProvider(name="reporting.runner", version="0.1.0"), ModuleInterfaceProvider(name="reporting.runner", version="0.1.0"),
ModuleInterfaceProvider(name="reporting.scheduler", version="0.1.0"), ModuleInterfaceProvider(name="reporting.scheduler", version="0.1.0"),
ModuleInterfaceProvider(name="reporting.chart_renderer", version="0.1.0"), ModuleInterfaceProvider(name="reporting.chart_renderer", version="0.1.0"),
ModuleInterfaceProvider(name=CAPABILITY_REPORTING_RETENTION, version="1.0.0"),
), ),
requires_interfaces=( requires_interfaces=(
ModuleInterfaceRequirement( ModuleInterfaceRequirement(
@@ -311,12 +322,19 @@ manifest = ModuleManifest(
version_max_exclusive="0.2.0", version_max_exclusive="0.2.0",
optional=True, optional=True,
), ),
ModuleInterfaceRequirement(
name=CAPABILITY_POLICY_REPORTING_GOVERNANCE,
version_min="1.0.0",
version_max_exclusive="2.0.0",
optional=True,
),
), ),
capability_factories={ capability_factories={
CAPABILITY_REPORTING_REGISTRY: _registry, CAPABILITY_REPORTING_REGISTRY: _registry,
CAPABILITY_REPORTING_RUNNER: _runner, CAPABILITY_REPORTING_RUNNER: _runner,
CAPABILITY_REPORTING_SCHEDULER: _scheduler, CAPABILITY_REPORTING_SCHEDULER: _scheduler,
CAPABILITY_REPORTING_CHART_RENDERER: _chart_renderer, CAPABILITY_REPORTING_CHART_RENDERER: _chart_renderer,
CAPABILITY_REPORTING_RETENTION: _retention,
}, },
capability_documentation={ capability_documentation={
CAPABILITY_REPORTING_REGISTRY: CapabilityDocumentation( CAPABILITY_REPORTING_REGISTRY: CapabilityDocumentation(
@@ -339,6 +357,13 @@ manifest = ModuleManifest(
summary="Builds provider-neutral chart models with an accessible tabular fallback.", summary="Builds provider-neutral chart models with an accessible tabular fallback.",
contract_version="0.1.0", contract_version="0.1.0",
), ),
CAPABILITY_REPORTING_RETENTION: CapabilityDocumentation(
label="Reporting result retention",
summary="Minimizes expired provider-report detail while retaining audit hashes and provenance.",
contract_version="1.0",
documentation_types=("admin",),
audience=("privacy_officer", "operator", "system_admin"),
),
}, },
search_sources=( search_sources=(
SearchSourceProviderRegistration( SearchSourceProviderRegistration(
@@ -359,6 +384,8 @@ manifest = ModuleManifest(
reporting_models.ReportingSavedView, reporting_models.ReportingSavedView,
reporting_models.ReportingDefinitionGrant, reporting_models.ReportingDefinitionGrant,
reporting_models.ReportingExecution, reporting_models.ReportingExecution,
reporting_models.ReportingProviderExport,
reporting_models.ReportingProviderExecution,
reporting_models.ReportingDefinitionRevision, reporting_models.ReportingDefinitionRevision,
reporting_models.ReportingDefinitionIdentity, reporting_models.ReportingDefinitionIdentity,
label="Reporting", label="Reporting",
@@ -374,6 +401,8 @@ manifest = ModuleManifest(
reporting_models.ReportingDefinitionRevision, reporting_models.ReportingDefinitionRevision,
reporting_models.ReportingDefinitionGrant, reporting_models.ReportingDefinitionGrant,
reporting_models.ReportingExecution, reporting_models.ReportingExecution,
reporting_models.ReportingProviderExecution,
reporting_models.ReportingProviderExport,
reporting_models.ReportingSavedView, reporting_models.ReportingSavedView,
reporting_models.ReportingSchedule, reporting_models.ReportingSchedule,
reporting_models.ReportingPublication, reporting_models.ReportingPublication,
@@ -0,0 +1,142 @@
"""Add governed cross-module report executions and export history.
Revision ID: b7c4e1a9d2f6
Revises: e5b2c9d4f7a1
"""
from __future__ import annotations
from alembic import op
import sqlalchemy as sa
revision = "b7c4e1a9d2f6"
down_revision = "e5b2c9d4f7a1"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"reporting_provider_executions",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("execution_id", sa.String(length=36), nullable=False),
sa.Column("provider_id", sa.String(length=120), nullable=False),
sa.Column("report_id", sa.String(length=255), nullable=False),
sa.Column("report_revision", sa.String(length=255), nullable=False),
sa.Column("contract_version", sa.String(length=30), nullable=False),
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
sa.Column("request_sha256", sa.String(length=64), nullable=False),
sa.Column("purpose", sa.Text(), nullable=False),
sa.Column("audience_scope", sa.JSON(), nullable=False),
sa.Column("parameters", sa.JSON(), nullable=False),
sa.Column("result_schema", sa.JSON(), nullable=False),
sa.Column("result_payload", sa.JSON(), nullable=False),
sa.Column("source_revisions", sa.JSON(), nullable=False),
sa.Column("effective_scope", sa.JSON(), nullable=False),
sa.Column("privacy_transforms", sa.JSON(), nullable=False),
sa.Column("provenance", sa.JSON(), nullable=False),
sa.Column("governance_provenance", sa.JSON(), nullable=False),
sa.Column("retention_class", sa.String(length=120), nullable=False),
sa.Column("retention_days", sa.Integer(), nullable=True),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("retention_redacted_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("output_hash", sa.String(length=64), nullable=False),
sa.Column("generated_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("actor_id", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_executions")),
sa.UniqueConstraint(
"tenant_id",
"execution_id",
name="uq_reporting_provider_execution",
),
sa.UniqueConstraint(
"tenant_id",
"idempotency_key",
name="uq_reporting_provider_execution_idempotency",
),
)
_indexes(
"reporting_provider_executions",
"tenant_id",
"execution_id",
"provider_id",
"report_id",
"expires_at",
"retention_redacted_at",
"output_hash",
"generated_at",
"actor_id",
)
op.create_index(
"ix_reporting_provider_execution_history",
"reporting_provider_executions",
["tenant_id", "provider_id", "report_id", "generated_at"],
unique=False,
)
op.create_index(
"ix_reporting_provider_execution_retention",
"reporting_provider_executions",
["tenant_id", "expires_at"],
unique=False,
)
op.create_table(
"reporting_provider_exports",
sa.Column("id", sa.String(length=36), nullable=False),
sa.Column("tenant_id", sa.String(length=36), nullable=False),
sa.Column("export_id", sa.String(length=36), nullable=False),
sa.Column("provider_execution_id", sa.String(length=36), nullable=False),
sa.Column("execution_id", sa.String(length=36), nullable=False),
sa.Column("format", sa.String(length=30), nullable=False),
sa.Column("purpose", sa.Text(), nullable=False),
sa.Column("audience_scope", sa.JSON(), nullable=False),
sa.Column("output_hash", sa.String(length=64), nullable=False),
sa.Column("exported_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("actor_id", sa.String(length=255), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.ForeignKeyConstraint(
["provider_execution_id"],
["reporting_provider_executions.id"],
name=op.f(
"fk_reporting_provider_exports_provider_execution_id_reporting_provider_executions"
),
ondelete="RESTRICT",
),
sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_exports")),
sa.UniqueConstraint(
"tenant_id",
"export_id",
name="uq_reporting_provider_export",
),
)
_indexes(
"reporting_provider_exports",
"tenant_id",
"export_id",
"provider_execution_id",
"execution_id",
"output_hash",
"exported_at",
"actor_id",
)
op.create_index(
"ix_reporting_provider_export_history",
"reporting_provider_exports",
["tenant_id", "provider_execution_id", "exported_at"],
unique=False,
)
def downgrade() -> None:
op.drop_table("reporting_provider_exports")
op.drop_table("reporting_provider_executions")
def _indexes(table: str, *columns: str) -> None:
for column in columns:
op.create_index(op.f(f"ix_{table}_{column}"), table, [column], unique=False)
@@ -0,0 +1,835 @@
"""Governed execution boundary for module-contributed reports."""
from __future__ import annotations
from collections.abc import Mapping
import csv
from datetime import UTC, date, datetime, timedelta
from hashlib import sha256
import io
import json
import uuid
from sqlalchemy.orm import Session
from govoplan_core.audit.logging import audit_from_principal
from govoplan_core.core.reporting import (
ReportDescriptor,
ReportParameterOption,
ReportProvider,
ReportProviderRequest,
ReportingGovernanceDecision,
ReportingGovernanceRequest,
report_providers,
reporting_governance_provider,
)
from govoplan_reporting.backend.db.models import (
ReportingProviderExecution,
ReportingProviderExport,
)
class ProviderReportError(RuntimeError):
pass
def list_provider_reports(
session: Session,
principal: object,
*,
registry: object | None,
) -> dict[str, object]:
reports: list[dict[str, object]] = []
diagnostics: list[dict[str, str]] = []
for provider_id, provider in report_providers(registry):
try:
for descriptor in provider.list_reports(session, principal):
_validate_descriptor(provider_id, descriptor)
decision = _governance_decision(
session,
principal,
registry=registry,
descriptor=descriptor,
action="catalogue",
purpose=None,
audience_scope={},
)
reports.append(
{
**descriptor.to_dict(),
"available": decision.allowed,
"unavailable_reason": decision.reason,
"governance": _decision_payload(decision),
}
)
except Exception as exc: # noqa: BLE001 - isolate optional providers.
diagnostics.append(
{
"provider_id": provider_id,
"code": "provider_catalogue_failed",
"message": "The provider catalogue could not be loaded.",
"error_type": type(exc).__name__,
}
)
return {
"reports": sorted(
reports,
key=lambda item: (str(item["title"]), str(item["provider_id"])),
),
"diagnostics": diagnostics,
}
def provider_parameter_options(
session: Session,
principal: object,
*,
registry: object | None,
provider_id: str,
report_id: str,
parameter_key: str,
query: str,
limit: int,
) -> tuple[ReportParameterOption, ...]:
provider, descriptor = _provider_report(
session,
principal,
registry=registry,
provider_id=provider_id,
report_id=report_id,
)
parameter = next(
(item for item in descriptor.parameters if item.key == parameter_key),
None,
)
if parameter is None or not parameter.options_from_provider:
raise ProviderReportError("This report parameter has no provider options")
return provider.parameter_options(
session,
principal,
report_id=report_id,
parameter_key=parameter_key,
query=query,
limit=max(1, min(limit, 200)),
)
def execute_provider_report(
session: Session,
principal: object,
*,
registry: object | None,
provider_id: str,
report_id: str,
parameters: Mapping[str, object],
purpose: str,
audience_scope: Mapping[str, object],
idempotency_key: str,
) -> dict[str, object]:
provider, descriptor = _provider_report(
session,
principal,
registry=registry,
provider_id=provider_id,
report_id=report_id,
)
clean_purpose = purpose.strip()
clean_idempotency_key = idempotency_key.strip()
clean_parameters = _validated_parameters(descriptor, parameters)
clean_audience = _json_mapping(audience_scope, "audience scope")
if descriptor.purpose_required and not clean_purpose:
raise ProviderReportError("A report purpose is required")
if descriptor.audience_scope_required and not clean_audience:
raise ProviderReportError("An effective audience scope is required")
request_hash = _hash(
{
"provider_id": provider_id,
"report_id": report_id,
"revision": descriptor.revision,
"parameters": clean_parameters,
"purpose": clean_purpose,
"audience_scope": clean_audience,
}
)
replay = (
session.query(ReportingProviderExecution)
.filter(
ReportingProviderExecution.tenant_id == _tenant(principal),
ReportingProviderExecution.idempotency_key == clean_idempotency_key,
)
.one_or_none()
)
if replay is not None:
if replay.request_sha256 != request_hash:
raise ProviderReportError(
"The provider-report idempotency key belongs to another request"
)
if _expired(replay) or replay.retention_redacted_at is not None:
raise ProviderReportError(
"The replayed provider-report result has expired; use a new idempotency key"
)
_require_result_access(provider, session, principal, replay)
return provider_execution_payload(replay)
preflight = _governance_decision(
session,
principal,
registry=registry,
descriptor=descriptor,
action="execute",
purpose=clean_purpose,
audience_scope=clean_audience,
)
_require_allowed(preflight)
result = provider.execute_report(
session,
principal,
request=ReportProviderRequest(
report_id=report_id,
parameters=clean_parameters,
purpose=clean_purpose,
audience_scope=clean_audience,
),
)
_validate_result(descriptor, result, tenant_id=_tenant(principal))
decision = _governance_decision(
session,
principal,
registry=registry,
descriptor=descriptor,
action="execute",
purpose=clean_purpose,
audience_scope=clean_audience,
applied_privacy_transforms=result.applied_privacy_transforms,
)
_require_allowed(decision)
missing_policy_transforms = set(decision.required_privacy_transforms) - set(
result.applied_privacy_transforms
)
if missing_policy_transforms:
raise ProviderReportError(
"The report omitted Policy-required privacy transformations: "
+ ", ".join(sorted(missing_policy_transforms))
)
generated_at = _aware(result.generated_at)
result_payload = _json_mapping(result.payload, "provider report payload")
expires_at = (
generated_at + timedelta(days=decision.retention_days)
if decision.retention_days is not None
else None
)
row = ReportingProviderExecution(
tenant_id=_tenant(principal),
execution_id=str(uuid.uuid4()),
provider_id=provider_id,
report_id=report_id,
report_revision=descriptor.revision,
contract_version=descriptor.contract_version,
idempotency_key=clean_idempotency_key,
request_sha256=request_hash,
purpose=clean_purpose,
audience_scope=clean_audience,
parameters=clean_parameters,
result_schema=[item.to_dict() for item in descriptor.result_schema],
result_payload=result_payload,
source_revisions=[dict(item) for item in result.source_revisions],
effective_scope=dict(result.effective_scope),
privacy_transforms=list(result.applied_privacy_transforms),
provenance=dict(result.provenance),
governance_provenance=dict(decision.provenance),
retention_class=descriptor.retention_class,
retention_days=decision.retention_days,
expires_at=expires_at,
output_hash=_hash(result_payload),
generated_at=generated_at,
actor_id=_actor(principal),
)
session.add(row)
session.flush()
audit_from_principal(
session,
principal,
action="reporting.provider_report.executed",
object_type="reporting_provider_execution",
object_id=row.execution_id,
details={
"provider_id": provider_id,
"report_id": report_id,
"report_revision": descriptor.revision,
"purpose": clean_purpose,
"audience_scope": clean_audience,
"source_revision_count": len(row.source_revisions),
"privacy_transforms": row.privacy_transforms,
"retention_class": row.retention_class,
"retention_days": row.retention_days,
"output_hash": row.output_hash,
},
)
return provider_execution_payload(row)
def get_provider_execution(
session: Session,
principal: object,
*,
registry: object | None,
execution_id: str,
) -> dict[str, object] | None:
row = _provider_execution(session, principal, execution_id=execution_id)
if row is None or _expired(row):
return None
provider, _descriptor = _provider_report(
session,
principal,
registry=registry,
provider_id=row.provider_id,
report_id=row.report_id,
)
_require_result_access(provider, session, principal, row)
return provider_execution_payload(row)
def export_provider_execution(
session: Session,
principal: object,
*,
registry: object | None,
execution_id: str,
format: str,
purpose: str,
audience_scope: Mapping[str, object],
) -> tuple[bytes, str, str]:
row = _provider_execution(session, principal, execution_id=execution_id)
if row is None or _expired(row):
raise LookupError("Provider report execution not found")
provider, descriptor = _provider_report(
session,
principal,
registry=registry,
provider_id=row.provider_id,
report_id=row.report_id,
)
_require_result_access(provider, session, principal, row)
clean_format = format.strip().lower()
if clean_format not in descriptor.export_formats:
raise ProviderReportError("This report does not support the export format")
clean_purpose = purpose.strip()
clean_audience = _json_mapping(audience_scope, "audience scope")
decision = _governance_decision(
session,
principal,
registry=registry,
descriptor=descriptor,
action="export",
purpose=clean_purpose,
audience_scope=clean_audience,
export_format=clean_format,
applied_privacy_transforms=tuple(row.privacy_transforms or ()),
)
_require_allowed(decision)
if clean_format not in decision.export_formats:
raise ProviderReportError("Policy does not allow this export format")
content, media_type, extension = _serialize_export(
clean_format,
row.result_payload,
row.result_schema,
)
output_hash = sha256(content).hexdigest()
export = ReportingProviderExport(
tenant_id=row.tenant_id,
export_id=str(uuid.uuid4()),
provider_execution_id=row.id,
execution_id=row.execution_id,
format=clean_format,
purpose=clean_purpose,
audience_scope=clean_audience,
output_hash=output_hash,
exported_at=datetime.now(UTC),
actor_id=_actor(principal),
)
session.add(export)
session.flush()
audit_from_principal(
session,
principal,
action="reporting.provider_report.exported",
object_type="reporting_provider_execution",
object_id=row.execution_id,
details={
"export_id": export.export_id,
"format": clean_format,
"purpose": clean_purpose,
"audience_scope": clean_audience,
"output_hash": output_hash,
},
)
filename = f"{row.provider_id}-{row.report_id}-{row.execution_id}.{extension}"
return content, media_type, filename
def list_provider_exports(
session: Session,
principal: object,
*,
registry: object | None,
execution_id: str,
) -> tuple[dict[str, object], ...]:
row = _provider_execution(session, principal, execution_id=execution_id)
if row is None or _expired(row):
return ()
provider, _descriptor = _provider_report(
session,
principal,
registry=registry,
provider_id=row.provider_id,
report_id=row.report_id,
)
_require_result_access(provider, session, principal, row)
exports = (
session.query(ReportingProviderExport)
.filter(
ReportingProviderExport.tenant_id == row.tenant_id,
ReportingProviderExport.provider_execution_id == row.id,
)
.order_by(ReportingProviderExport.exported_at.desc())
.all()
)
return tuple(
{
"export_id": item.export_id,
"execution_id": item.execution_id,
"format": item.format,
"purpose": item.purpose,
"audience_scope": dict(item.audience_scope or {}),
"output_hash": item.output_hash,
"exported_at": item.exported_at.isoformat(),
"actor_id": item.actor_id,
}
for item in exports
)
def provider_execution_payload(row: ReportingProviderExecution) -> dict[str, object]:
return {
"execution_id": row.execution_id,
"provider_id": row.provider_id,
"report_id": row.report_id,
"report_revision": row.report_revision,
"contract_version": row.contract_version,
"purpose": row.purpose,
"audience_scope": dict(row.audience_scope or {}),
"parameters": dict(row.parameters or {}),
"result_schema": list(row.result_schema or []),
"result": dict(row.result_payload or {}),
"source_revisions": list(row.source_revisions or []),
"effective_scope": dict(row.effective_scope or {}),
"privacy_transforms": list(row.privacy_transforms or []),
"provenance": dict(row.provenance or {}),
"governance_provenance": dict(row.governance_provenance or {}),
"retention_class": row.retention_class,
"retention_days": row.retention_days,
"expires_at": row.expires_at.isoformat() if row.expires_at else None,
"output_hash": row.output_hash,
"generated_at": row.generated_at.isoformat(),
"actor_id": row.actor_id,
}
def _provider_report(
session: Session,
principal: object,
*,
registry: object | None,
provider_id: str,
report_id: str,
) -> tuple[ReportProvider, ReportDescriptor]:
provider = dict(report_providers(registry)).get(provider_id)
if provider is None:
raise LookupError("Report provider is unavailable")
descriptor = next(
(
item
for item in provider.list_reports(session, principal)
if item.report_id == report_id
),
None,
)
if descriptor is None:
raise LookupError("Provider report is unavailable")
_validate_descriptor(provider_id, descriptor)
return provider, descriptor
def _validate_descriptor(provider_id: str, descriptor: ReportDescriptor) -> None:
if descriptor.provider_id != provider_id:
raise ProviderReportError("Report descriptor provider id differs")
if not descriptor.report_id or not descriptor.revision:
raise ProviderReportError("Report descriptors require stable ids and revisions")
parameter_keys = [item.key for item in descriptor.parameters]
field_paths = [item.path for item in descriptor.result_schema]
transform_ids = [item.id for item in descriptor.privacy_transforms]
if len(parameter_keys) != len(set(parameter_keys)):
raise ProviderReportError("Report parameter keys must be unique")
if not field_paths or len(field_paths) != len(set(field_paths)):
raise ProviderReportError("Report result paths must be non-empty and unique")
if len(transform_ids) != len(set(transform_ids)):
raise ProviderReportError("Report privacy transforms must be unique")
if any(item.sensitive for item in descriptor.result_schema) and (
descriptor.reidentification_risk != "high"
):
raise ProviderReportError(
"Sensitive result fields require high risk classification"
)
def _validate_result(
descriptor: ReportDescriptor,
result: object,
*,
tenant_id: str,
) -> None:
if not hasattr(result, "report_id") or result.report_id != descriptor.report_id:
raise ProviderReportError("Provider result report id differs")
if not result.source_revisions:
raise ProviderReportError("Provider result has no source revision provenance")
if str(result.effective_scope.get("tenant_id") or "") != tenant_id:
raise ProviderReportError(
"Provider result effective scope differs from the tenant"
)
required = {item.id for item in descriptor.privacy_transforms if item.required}
missing = required - set(result.applied_privacy_transforms)
if missing:
raise ProviderReportError(
"Provider result omitted required privacy transformations: "
+ ", ".join(sorted(missing))
)
declared = {item.path: item for item in descriptor.result_schema}
for field in descriptor.result_schema:
if field.type != "suppressed_count":
continue
value = _path_value(result.payload, field.path)
if not isinstance(value, Mapping) or set(value) != {"value", "suppressed"}:
raise ProviderReportError(
f"Provider result has an invalid suppressed count: {field.path}"
)
if value["value"] is not None and (
not isinstance(value["value"], int) or isinstance(value["value"], bool)
):
raise ProviderReportError(
f"Provider result has an invalid suppressed-count value: {field.path}"
)
if not isinstance(value["suppressed"], bool):
raise ProviderReportError(
f"Provider result has an invalid suppression flag: {field.path}"
)
for path in _leaf_paths(result.payload):
if path in declared:
continue
if any(
path.startswith(declared_path + ".")
and field.type in {"object", "suppressed_count"}
for declared_path, field in declared.items()
):
continue
raise ProviderReportError(f"Provider result contains undeclared field: {path}")
def _validated_parameters(
descriptor: ReportDescriptor,
parameters: Mapping[str, object],
) -> dict[str, object]:
clean = _json_mapping(parameters, "report parameters")
declared = {item.key: item for item in descriptor.parameters}
unknown = set(clean) - set(declared)
if unknown:
raise ProviderReportError(
"Unknown report parameters: " + ", ".join(sorted(unknown))
)
missing = [
item.key
for item in descriptor.parameters
if item.required and clean.get(item.key) in (None, "")
]
if missing:
raise ProviderReportError(
"Missing report parameters: " + ", ".join(sorted(missing))
)
invalid = [
item.key
for item in descriptor.parameters
if item.key in clean
and clean[item.key] is not None
and not _valid_parameter_value(item.type, clean[item.key])
]
if invalid:
raise ProviderReportError(
"Invalid report parameter types: " + ", ".join(sorted(invalid))
)
return clean
def _valid_parameter_value(parameter_type: str, value: object) -> bool:
if parameter_type in {"string", "reference"}:
return isinstance(value, str)
if parameter_type == "boolean":
return isinstance(value, bool)
if parameter_type == "integer":
return isinstance(value, int) and not isinstance(value, bool)
if parameter_type == "number":
return isinstance(value, (int, float)) and not isinstance(value, bool)
if parameter_type == "date":
if not isinstance(value, str):
return False
try:
date.fromisoformat(value)
except ValueError:
return False
return True
if parameter_type == "datetime":
if not isinstance(value, str):
return False
try:
datetime.fromisoformat(value.replace("Z", "+00:00"))
except ValueError:
return False
return True
return False
def _governance_decision(
session: Session,
principal: object,
*,
registry: object | None,
descriptor: ReportDescriptor,
action: str,
purpose: str | None,
audience_scope: Mapping[str, object],
export_format: str | None = None,
applied_privacy_transforms: tuple[str, ...] = (),
) -> ReportingGovernanceDecision:
required = tuple(item.id for item in descriptor.privacy_transforms if item.required)
provider = reporting_governance_provider(registry)
if provider is None:
allowed = descriptor.reidentification_risk != "high"
if descriptor.purpose_required and action != "catalogue" and not purpose:
allowed = False
if (
descriptor.audience_scope_required
and action != "catalogue"
and not audience_scope
):
allowed = False
if action == "export" and export_format not in descriptor.export_formats:
allowed = False
return ReportingGovernanceDecision(
allowed=allowed,
reason=None
if allowed
else "The built-in Reporting privacy baseline denied this action.",
retention_days=30,
export_formats=descriptor.export_formats,
required_privacy_transforms=required,
provenance={"provider": "reporting.built_in_baseline", "version": "1"},
)
decision = provider.decide_reporting_action(
session,
principal,
request=ReportingGovernanceRequest(
action=action, # type: ignore[arg-type]
tenant_id=_tenant(principal),
provider_id=descriptor.provider_id,
report_id=descriptor.report_id,
purpose=purpose,
audience_scope=audience_scope,
retention_class=descriptor.retention_class,
export_format=export_format,
reidentification_risk=descriptor.reidentification_risk,
declared_privacy_transforms=required,
applied_privacy_transforms=applied_privacy_transforms,
),
)
unsupported = set(decision.required_privacy_transforms) - {
item.id for item in descriptor.privacy_transforms
}
if not unsupported:
return decision
return ReportingGovernanceDecision(
allowed=False,
reason=(
"Policy requires privacy transformations this report does not support: "
+ ", ".join(sorted(unsupported))
),
retention_days=decision.retention_days,
export_formats=decision.export_formats,
required_privacy_transforms=decision.required_privacy_transforms,
provenance={
**dict(decision.provenance),
"unsupported_required_privacy_transforms": sorted(unsupported),
},
)
def _serialize_export(
format: str,
payload: Mapping[str, object],
schema: list[dict[str, object]],
) -> tuple[bytes, str, str]:
if format == "json":
return (
json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True).encode(
"utf-8"
),
"application/json",
"json",
)
if format != "csv":
raise ProviderReportError("Unsupported provider report export format")
output = io.StringIO(newline="")
writer = csv.writer(output)
writer.writerow([str(item["label"]) for item in schema])
writer.writerow(
[_csv_value(_path_value(payload, str(item["path"]))) for item in schema]
)
return output.getvalue().encode("utf-8-sig"), "text/csv; charset=utf-8", "csv"
def _csv_value(value: object) -> object:
if isinstance(value, Mapping):
if value.get("suppressed") is True:
return "suppressed"
if set(value).issuperset({"value", "suppressed"}):
value = value.get("value")
else:
value = json.dumps(value, ensure_ascii=False, sort_keys=True)
if isinstance(value, (list, tuple)):
value = json.dumps(value, ensure_ascii=False, sort_keys=True)
if isinstance(value, str) and value.startswith(("=", "+", "-", "@", "\t", "\r")):
return "'" + value
return "" if value is None else value
def _path_value(payload: Mapping[str, object], path: str) -> object:
value: object = payload
for part in path.split("."):
if not isinstance(value, Mapping):
return None
value = value.get(part)
return value
def _leaf_paths(value: object, prefix: str = "") -> tuple[str, ...]:
if isinstance(value, Mapping):
rows: list[str] = []
for key, item in value.items():
path = f"{prefix}.{key}" if prefix else str(key)
rows.extend(_leaf_paths(item, path))
return tuple(rows)
if isinstance(value, (list, tuple)):
return (prefix,)
return (prefix,)
def _provider_execution(
session: Session,
principal: object,
*,
execution_id: str,
) -> ReportingProviderExecution | None:
return (
session.query(ReportingProviderExecution)
.filter(
ReportingProviderExecution.tenant_id == _tenant(principal),
ReportingProviderExecution.execution_id == execution_id,
)
.one_or_none()
)
def _require_result_access(
provider: ReportProvider,
session: Session,
principal: object,
row: ReportingProviderExecution,
) -> None:
if not provider.authorize_result(
session,
principal,
report_id=row.report_id,
source_revisions=tuple(row.source_revisions or ()),
effective_scope=dict(row.effective_scope or {}),
):
raise PermissionError(
"The source module no longer permits access to this report result"
)
def _expired(row: ReportingProviderExecution) -> bool:
if row.expires_at is None:
return False
return _aware(row.expires_at) <= datetime.now(UTC)
def _require_allowed(decision: ReportingGovernanceDecision) -> None:
if not decision.allowed:
raise PermissionError(decision.reason or "Reporting Policy denied this action")
def _decision_payload(decision: ReportingGovernanceDecision) -> dict[str, object]:
return {
"allowed": decision.allowed,
"reason": decision.reason,
"retention_days": decision.retention_days,
"export_formats": list(decision.export_formats),
"required_privacy_transforms": list(decision.required_privacy_transforms),
"provenance": dict(decision.provenance),
}
def _json_mapping(value: Mapping[str, object], label: str) -> dict[str, object]:
try:
payload = json.loads(json.dumps(value, sort_keys=True, separators=(",", ":")))
except (TypeError, ValueError) as exc:
raise ProviderReportError(f"The {label} must contain JSON values") from exc
if not isinstance(payload, dict):
raise ProviderReportError(f"The {label} must be an object")
return payload
def _hash(value: object) -> str:
return sha256(
json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
).hexdigest()
def _tenant(principal: object) -> str:
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
if not tenant_id:
raise PermissionError("A tenant principal is required")
return tenant_id
def _actor(principal: object) -> str | None:
user = getattr(principal, "user", None)
actor_id = getattr(user, "id", None)
return str(actor_id) if actor_id else None
def _aware(value: datetime) -> datetime:
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
__all__ = [
"ProviderReportError",
"execute_provider_report",
"export_provider_execution",
"get_provider_execution",
"list_provider_exports",
"list_provider_reports",
"provider_parameter_options",
]
@@ -0,0 +1,69 @@
"""Retention minimization for governed provider-report results."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy.orm import Session
from govoplan_reporting.backend.db.models import ReportingProviderExecution
class ReportingRetentionService:
def apply_retention(
self,
session: object,
*,
dry_run: bool,
now: datetime,
limit: int = 500,
) -> dict[str, int]:
if not isinstance(session, Session):
raise TypeError("Reporting retention requires a SQLAlchemy Session")
observed_at = _aware(now)
rows = (
session.query(ReportingProviderExecution)
.filter(
ReportingProviderExecution.expires_at.is_not(None),
ReportingProviderExecution.expires_at <= observed_at,
ReportingProviderExecution.retention_redacted_at.is_(None),
)
.order_by(
ReportingProviderExecution.expires_at.asc(),
ReportingProviderExecution.id.asc(),
)
.limit(max(1, min(int(limit), 5_000)))
.all()
)
counts = {
"eligible": len(rows),
"redacted": 0,
"remaining_in_batch": 0,
}
if dry_run:
return counts
for row in rows:
# Keep immutable request/output hashes and provenance as audit
# evidence while removing the retained report detail itself.
row.result_payload = {}
row.retention_redacted_at = observed_at
counts["redacted"] += 1
session.flush()
counts["remaining_in_batch"] = (
session.query(ReportingProviderExecution.id)
.filter(
ReportingProviderExecution.expires_at.is_not(None),
ReportingProviderExecution.expires_at <= observed_at,
ReportingProviderExecution.retention_redacted_at.is_(None),
)
.limit(1)
.count()
)
return counts
def _aware(value: datetime) -> datetime:
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
__all__ = ["ReportingRetentionService"]
+143
View File
@@ -44,12 +44,23 @@ from govoplan_reporting.backend.operations import (
upsert_saved_view, upsert_saved_view,
upsert_schedule, upsert_schedule,
) )
from govoplan_reporting.backend.provider_reports import (
ProviderReportError,
execute_provider_report,
export_provider_execution,
get_provider_execution,
list_provider_exports,
list_provider_reports,
provider_parameter_options,
)
from govoplan_reporting.backend.query_engine import ReportingQueryError from govoplan_reporting.backend.query_engine import ReportingQueryError
from govoplan_reporting.backend.schemas import ( from govoplan_reporting.backend.schemas import (
DefinitionUpdateRequest, DefinitionUpdateRequest,
DefinitionWriteRequest, DefinitionWriteRequest,
ImportAssessmentRequest, ImportAssessmentRequest,
PublicationRequest, PublicationRequest,
ProviderReportExecutionRequest,
ProviderReportExportRequest,
QualityRunRequest, QualityRunRequest,
ReportExecutionRequest, ReportExecutionRequest,
SavedViewWriteRequest, SavedViewWriteRequest,
@@ -60,6 +71,138 @@ from govoplan_reporting.backend.schemas import (
def create_router(registry: object | None) -> APIRouter: def create_router(registry: object | None) -> APIRouter:
router = APIRouter(prefix="/reporting", tags=["reporting"]) router = APIRouter(prefix="/reporting", tags=["reporting"])
@router.get("/provider-reports")
def api_list_provider_reports(
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> dict[str, object]:
_require(principal, READ_SCOPE)
try:
return list_provider_reports(session, principal, registry=registry)
except (ProviderReportError, TypeError, ValueError) as exc:
raise _error(exc) from exc
@router.get(
"/provider-reports/{provider_id}/{report_id}/parameters/{parameter_key}/options"
)
def api_provider_parameter_options(
provider_id: str,
report_id: str,
parameter_key: str,
query: str = "",
limit: int = Query(default=100, ge=1, le=200),
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> dict[str, object]:
_require(principal, READ_SCOPE)
try:
options = provider_parameter_options(
session,
principal,
registry=registry,
provider_id=provider_id,
report_id=report_id,
parameter_key=parameter_key,
query=query,
limit=limit,
)
except (ProviderReportError, PermissionError, LookupError) as exc:
raise _error(exc) from exc
return {"options": [item.to_dict() for item in options]}
@router.post("/provider-reports/{provider_id}/{report_id}/executions")
def api_execute_provider_report(
provider_id: str,
report_id: str,
payload: ProviderReportExecutionRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> dict[str, object]:
_require(principal, RUN_SCOPE)
try:
result = execute_provider_report(
session,
principal,
registry=registry,
provider_id=provider_id,
report_id=report_id,
**payload.model_dump(),
)
session.commit()
except (ProviderReportError, PermissionError, LookupError, ValueError) as exc:
session.rollback()
raise _error(exc) from exc
return result
@router.get("/provider-executions/{execution_id}")
def api_get_provider_execution(
execution_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> dict[str, object]:
_require(principal, RUN_SCOPE)
try:
result = get_provider_execution(
session,
principal,
registry=registry,
execution_id=execution_id,
)
except (ProviderReportError, PermissionError, LookupError) as exc:
raise _error(exc) from exc
if result is None:
raise HTTPException(
status_code=404, detail="Provider report execution not found"
)
return result
@router.get("/provider-executions/{execution_id}/exports")
def api_list_provider_exports(
execution_id: str,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> dict[str, object]:
_require(principal, RUN_SCOPE)
try:
return {
"exports": list(
list_provider_exports(
session,
principal,
registry=registry,
execution_id=execution_id,
)
)
}
except (ProviderReportError, PermissionError, LookupError) as exc:
raise _error(exc) from exc
@router.post("/provider-executions/{execution_id}/exports")
def api_export_provider_execution(
execution_id: str,
payload: ProviderReportExportRequest,
session: Session = Depends(get_session),
principal: ApiPrincipal = Depends(get_api_principal),
) -> Response:
_require(principal, RUN_SCOPE)
try:
content, media_type, filename = export_provider_execution(
session,
principal,
registry=registry,
execution_id=execution_id,
**payload.model_dump(),
)
session.commit()
except (ProviderReportError, PermissionError, LookupError) as exc:
session.rollback()
raise _error(exc) from exc
return Response(
content=content,
media_type=media_type,
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
)
@router.get("/definitions") @router.get("/definitions")
def api_list_definitions( def api_list_definitions(
definition_kind: list[str] | None = Query(default=None), definition_kind: list[str] | None = Query(default=None),
+19
View File
@@ -422,6 +422,23 @@ class ReportExecutionRequest(BaseModel):
idempotency_key: str = Field(min_length=1, max_length=255) idempotency_key: str = Field(min_length=1, max_length=255)
class ProviderReportExecutionRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
parameters: dict[str, Any] = Field(default_factory=dict)
purpose: str = Field(min_length=1, max_length=1_000)
audience_scope: dict[str, Any] = Field(min_length=1, max_length=50)
idempotency_key: str = Field(min_length=1, max_length=255)
class ProviderReportExportRequest(BaseModel):
model_config = ConfigDict(extra="forbid")
format: Literal["json", "csv"] = "json"
purpose: str = Field(min_length=1, max_length=1_000)
audience_scope: dict[str, Any] = Field(min_length=1, max_length=50)
class SavedViewWriteRequest(BaseModel): class SavedViewWriteRequest(BaseModel):
model_config = ConfigDict(extra="forbid") model_config = ConfigDict(extra="forbid")
@@ -489,6 +506,8 @@ __all__ = [
"ImportAssessmentRequest", "ImportAssessmentRequest",
"MeasureDefinition", "MeasureDefinition",
"PublicationRequest", "PublicationRequest",
"ProviderReportExecutionRequest",
"ProviderReportExportRequest",
"QualityPlanDefinition", "QualityPlanDefinition",
"QualityRunRequest", "QualityRunRequest",
"ReportDefinition", "ReportDefinition",
+7 -1
View File
@@ -14,9 +14,15 @@ class ReportingManifestTests(unittest.TestCase):
self.assertEqual("@govoplan/reporting-webui", manifest.frontend.package_name) self.assertEqual("@govoplan/reporting-webui", manifest.frontend.package_name)
self.assertIsNotNone(manifest.route_factory) self.assertIsNotNone(manifest.route_factory)
self.assertIsNotNone(manifest.migration_spec) self.assertIsNotNone(manifest.migration_spec)
self.assertEqual(4, len(manifest.provides_interfaces)) self.assertEqual(5, len(manifest.provides_interfaces))
self.assertEqual(1, len(manifest.search_sources)) self.assertEqual(1, len(manifest.search_sources))
self.assertIn("dataflow", manifest.optional_dependencies) self.assertIn("dataflow", manifest.optional_dependencies)
self.assertIn("policy", manifest.optional_dependencies)
self.assertEqual("/reports", manifest.nav_items[0].path)
self.assertEqual(
{"/reports", "/reporting"},
{route.path for route in manifest.frontend.routes},
)
self.assertIn( self.assertIn(
"report execution and publication evidence", "report execution and publication evidence",
manifest.architecture.owned_concepts, manifest.architecture.owned_concepts,
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
from pathlib import Path
import tempfile
from alembic.runtime.migration import MigrationContext
from sqlalchemy import create_engine, inspect
from govoplan_core.db.migrations import migrate_database
from govoplan_reporting.backend.manifest import get_manifest
def test_fresh_migration_creates_provider_evidence_tables_and_current_head() -> None:
with tempfile.TemporaryDirectory(
prefix="govoplan-reporting-migration-"
) as directory:
url = f"sqlite:///{Path(directory) / 'reporting.db'}"
migrate_database(
database_url=url,
enabled_modules=("reporting",),
manifest_factories=(get_manifest,),
)
engine = create_engine(url)
try:
with engine.connect() as connection:
tables = set(inspect(connection).get_table_names())
assert {
"reporting_provider_executions",
"reporting_provider_exports",
}.issubset(tables)
assert "b7c4e1a9d2f6" in set(
MigrationContext.configure(connection).get_current_heads()
)
columns = {
item["name"]
for item in inspect(connection).get_columns(
"reporting_provider_executions"
)
}
assert "retention_redacted_at" in columns
finally:
engine.dispose()
+91
View File
@@ -0,0 +1,91 @@
from __future__ import annotations
import os
from pathlib import Path
import subprocess
import sys
_ROOT = Path(__file__).resolve().parents[1]
def test_campaign_starts_and_contributes_its_provider_without_reporting() -> None:
script = """
import importlib.abc
import sys
class Blocker(importlib.abc.MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
if fullname == 'govoplan_reporting' or fullname.startswith('govoplan_reporting.'):
raise ModuleNotFoundError("Reporting is physically absent", name=fullname)
return None
sys.meta_path.insert(0, Blocker())
from govoplan_campaign.backend.manifest import get_manifest
manifest = get_manifest()
assert 'reporting' not in manifest.dependencies
assert 'reporting' in manifest.optional_dependencies
assert 'reporting.report_provider.campaigns' in manifest.capability_factories
"""
_run_probe(script)
def test_reporting_starts_without_campaign_and_owns_the_global_route() -> None:
script = """
import importlib.abc
import sys
class Blocker(importlib.abc.MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
if fullname == 'govoplan_campaign' or fullname.startswith('govoplan_campaign.'):
raise ModuleNotFoundError("Campaign is physically absent", name=fullname)
return None
sys.meta_path.insert(0, Blocker())
from govoplan_reporting.backend.manifest import get_manifest
manifest = get_manifest()
assert 'campaigns' not in manifest.dependencies
assert manifest.nav_items[0].path == '/reports'
assert {item.path for item in manifest.frontend.routes} == {'/reports', '/reporting'}
"""
_run_probe(script)
def test_reporting_with_campaign_uses_only_the_provider_contract() -> None:
script = """
from govoplan_campaign.backend.manifest import get_manifest as campaign_manifest
from govoplan_core.core.reporting import REPORT_PROVIDER_CAPABILITY_PREFIX
from govoplan_reporting.backend.manifest import get_manifest as reporting_manifest
reporting = reporting_manifest()
campaign = campaign_manifest()
provider_name = REPORT_PROVIDER_CAPABILITY_PREFIX + 'campaigns'
assert provider_name in campaign.capability_factories
assert provider_name in {item.name for item in campaign.provides_interfaces}
assert 'campaigns' not in reporting.dependencies
assert '/reports' not in {route.path for route in campaign.frontend.routes}
provider = campaign.capability_factories[provider_name](None)
assert provider.provider_id == 'campaigns'
assert provider.contract_version == '1.0'
"""
_run_probe(script)
def _run_probe(source: str) -> None:
environment = dict(os.environ)
environment["PYTHONPATH"] = os.pathsep.join(
(
str(_ROOT / "src"),
str(_ROOT.parent / "govoplan-campaign" / "src"),
str(_ROOT.parent / "govoplan-core" / "src"),
environment.get("PYTHONPATH", ""),
)
)
result = subprocess.run(
[sys.executable, "-c", source],
cwd=_ROOT,
env=environment,
text=True,
capture_output=True,
check=False,
)
assert result.returncode == 0, result.stderr or result.stdout
+336
View File
@@ -0,0 +1,336 @@
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from types import SimpleNamespace
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from govoplan_core.core.reporting import (
REPORT_PROVIDER_CAPABILITY_PREFIX,
REPORT_PROVIDER_CONTRACT_VERSION,
ReportDescriptor,
ReportParameterDescriptor,
ReportParameterOption,
ReportPrivacyTransform,
ReportProviderRequest,
ReportProviderResult,
ReportResultField,
)
from govoplan_core.db.base import Base
from govoplan_reporting.backend.db.models import (
ReportingProviderExecution,
ReportingProviderExport,
)
from govoplan_reporting.backend.provider_reports import (
ProviderReportError,
execute_provider_report,
export_provider_execution,
list_provider_reports,
)
from govoplan_reporting.backend.retention import ReportingRetentionService
class _Principal:
tenant_id = "tenant-1"
user = SimpleNamespace(id="analyst-1")
api_key = None
class _Provider:
provider_id = "example"
contract_version = REPORT_PROVIDER_CONTRACT_VERSION
def __init__(self) -> None:
self.result_access = True
def list_reports(self, session, principal):
del session, principal
return (
ReportDescriptor(
provider_id=self.provider_id,
report_id="summary",
revision="example.summary.v1",
title="Example summary",
summary="A minimized example report.",
parameters=(
ReportParameterDescriptor(
key="source_id",
label="Source",
type="reference",
required=True,
options_from_provider=True,
),
),
result_schema=(
ReportResultField(
path="metric",
label="Metric",
type="integer",
group="Outcome",
),
),
privacy_transforms=(
ReportPrivacyTransform(id="aggregate", label="Aggregate"),
),
export_formats=("json",),
reidentification_risk="low",
),
)
def parameter_options(self, session, principal, **kwargs):
del session, principal, kwargs
return (ReportParameterOption(value="source-1", label="Source 1"),)
def execute_report(
self,
session,
principal,
*,
request: ReportProviderRequest,
):
del session
return ReportProviderResult(
report_id=request.report_id,
generated_at=datetime(2026, 8, 2, 10, 0, tzinfo=UTC),
payload={"metric": 12},
source_revisions=(
{
"module_id": "example",
"resource_id": request.parameters["source_id"],
"revision_id": "7",
},
),
effective_scope={
"tenant_id": principal.tenant_id,
"source_id": request.parameters["source_id"],
},
applied_privacy_transforms=("aggregate",),
provenance={"executor": "example-v1"},
)
def authorize_result(self, session, principal, **kwargs):
del session, principal, kwargs
return self.result_access
class _Registry:
def __init__(self, provider=None):
self.provider = provider or _Provider()
def capability_names(self):
return (REPORT_PROVIDER_CAPABILITY_PREFIX + "example",)
def has_capability(self, name):
return name == REPORT_PROVIDER_CAPABILITY_PREFIX + "example"
def capability(self, name):
return self.provider if self.has_capability(name) else None
def test_provider_report_execution_is_governed_replayable_and_audited() -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
principal = _Principal()
registry = _Registry()
with Session(engine, expire_on_commit=False) as session:
catalogue = list_provider_reports(session, principal, registry=registry)
assert catalogue["reports"][0]["available"] is True
execution = execute_provider_report(
session,
principal,
registry=registry,
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="provider-run-1",
)
replay = execute_provider_report(
session,
principal,
registry=registry,
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="provider-run-1",
)
assert replay["execution_id"] == execution["execution_id"]
assert execution["source_revisions"][0]["revision_id"] == "7"
assert execution["privacy_transforms"] == ["aggregate"]
assert execution["retention_days"] == 30
content, media_type, filename = export_provider_execution(
session,
principal,
registry=registry,
execution_id=str(execution["execution_id"]),
format="json",
purpose="Archive approved aggregate",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
)
assert b'"metric": 12' in content
assert media_type == "application/json"
assert filename.endswith(".json")
assert session.query(ReportingProviderExport).count() == 1
engine.dispose()
def test_provider_result_missing_a_required_transform_is_rejected() -> None:
class _UnsafeProvider(_Provider):
def execute_report(self, session, principal, *, request):
result = super().execute_report(
session,
principal,
request=request,
)
return ReportProviderResult(
report_id=result.report_id,
generated_at=result.generated_at,
payload=result.payload,
source_revisions=result.source_revisions,
effective_scope=result.effective_scope,
applied_privacy_transforms=(),
provenance=result.provenance,
)
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
with Session(engine) as session:
try:
execute_provider_report(
session,
_Principal(),
registry=_Registry(_UnsafeProvider()),
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant"},
idempotency_key="unsafe-provider-run",
)
except ProviderReportError as exc:
assert "privacy transformations" in str(exc)
else:
raise AssertionError("unsafe provider report was accepted")
engine.dispose()
def test_expired_provider_result_is_minimized_but_evidence_is_retained() -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
with Session(engine, expire_on_commit=False) as session:
execution = execute_provider_report(
session,
_Principal(),
registry=_Registry(),
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="expired-provider-run",
)
row = session.query(ReportingProviderExecution).one()
original_hash = row.output_hash
row.expires_at = datetime.now(UTC) - timedelta(seconds=1)
session.flush()
preview = ReportingRetentionService().apply_retention(
session,
dry_run=True,
now=datetime.now(UTC),
)
assert preview["eligible"] == 1
assert row.result_payload == {"metric": 12}
applied = ReportingRetentionService().apply_retention(
session,
dry_run=False,
now=datetime.now(UTC),
)
assert applied["redacted"] == 1
assert row.result_payload == {}
assert row.retention_redacted_at is not None
assert row.output_hash == original_hash == execution["output_hash"]
assert row.source_revisions[0]["revision_id"] == "7"
try:
execute_provider_report(
session,
_Principal(),
registry=_Registry(),
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="expired-provider-run",
)
except ProviderReportError as exc:
assert "expired" in str(exc)
else:
raise AssertionError("expired provider-report detail was replayed")
engine.dispose()
def test_provider_parameters_are_checked_against_the_declared_type() -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
with Session(engine) as session:
try:
execute_provider_report(
session,
_Principal(),
registry=_Registry(),
provider_id="example",
report_id="summary",
parameters={"source_id": 123},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="invalid-provider-parameters",
)
except ProviderReportError as exc:
assert "parameter types" in str(exc)
else:
raise AssertionError("invalid provider-report parameters were accepted")
engine.dispose()
def test_source_access_is_rechecked_before_retained_result_export() -> None:
engine = create_engine("sqlite+pysqlite:///:memory:")
Base.metadata.create_all(engine)
provider = _Provider()
registry = _Registry(provider)
with Session(engine) as session:
execution = execute_provider_report(
session,
_Principal(),
registry=registry,
provider_id="example",
report_id="summary",
parameters={"source_id": "source-1"},
purpose="Operational overview",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
idempotency_key="source-access-provider-run",
)
provider.result_access = False
try:
export_provider_execution(
session,
_Principal(),
registry=registry,
execution_id=str(execution["execution_id"]),
format="json",
purpose="Archive aggregate",
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
)
except PermissionError as exc:
assert "source module" in str(exc)
else:
raise AssertionError("revoked source result was exported")
engine.dispose()
+137
View File
@@ -119,6 +119,74 @@ export type ReportingSavedView = {
updated_at: string; updated_at: string;
}; };
export type ProviderReportParameter = {
key: string;
label: string;
type: string;
required: boolean;
description?: string | null;
options_from_provider: boolean;
};
export type ProviderReportField = {
path: string;
label: string;
type: string;
group: string;
nullable: boolean;
sensitive: boolean;
};
export type ProviderReportDescriptor = {
contract_version: string;
provider_id: string;
report_id: string;
revision: string;
title: string;
summary: string;
parameters: ProviderReportParameter[];
result_schema: ProviderReportField[];
privacy_transforms: Array<{ id: string; label: string; required: boolean }>;
purpose_required: boolean;
audience_scope_required: boolean;
retention_class: string;
export_formats: Array<"json" | "csv">;
reidentification_risk: "low" | "moderate" | "high";
presentation: { kind?: string };
available: boolean;
unavailable_reason?: string | null;
governance: {
retention_days?: number | null;
export_formats: string[];
required_privacy_transforms: string[];
provenance: Record<string, unknown>;
};
};
export type ProviderReportExecution = {
execution_id: string;
provider_id: string;
report_id: string;
report_revision: string;
contract_version: string;
purpose: string;
audience_scope: Record<string, unknown>;
parameters: Record<string, unknown>;
result_schema: ProviderReportField[];
result: Record<string, unknown>;
source_revisions: Array<Record<string, unknown>>;
effective_scope: Record<string, unknown>;
privacy_transforms: string[];
provenance: Record<string, unknown>;
governance_provenance: Record<string, unknown>;
retention_class: string;
retention_days?: number | null;
expires_at?: string | null;
output_hash: string;
generated_at: string;
actor_id?: string | null;
};
export function listDefinitions( export function listDefinitions(
settings: ApiSettings, settings: ApiSettings,
options: { kinds?: ReportingDefinitionKind[]; status?: string[]; query?: string; limit?: number }, options: { kinds?: ReportingDefinitionKind[]; status?: string[]; query?: string; limit?: number },
@@ -132,6 +200,75 @@ export function listDefinitions(
}), { signal }); }), { signal });
} }
export function listProviderReports(
settings: ApiSettings,
signal?: AbortSignal
): Promise<{ reports: ProviderReportDescriptor[]; diagnostics: Array<Record<string, string>> }> {
return apiFetch(settings, "/api/v1/reporting/provider-reports", { signal });
}
export function listProviderParameterOptions(
settings: ApiSettings,
report: ProviderReportDescriptor,
parameterKey: string,
query = "",
signal?: AbortSignal
): Promise<{ options: Array<{ value: string; label: string; description?: string | null }> }> {
return apiFetch(settings, apiPath(
`/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/parameters/${encodeURIComponent(parameterKey)}/options`,
{ query, limit: 200 }
), { signal });
}
export function runProviderReport(
settings: ApiSettings,
report: ProviderReportDescriptor,
parameters: Record<string, unknown>,
purpose: string,
audienceScope: Record<string, unknown>
): Promise<ProviderReportExecution> {
return apiFetch(
settings,
`/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/executions`,
{
method: "POST",
body: JSON.stringify({
parameters,
purpose,
audience_scope: audienceScope,
idempotency_key: crypto.randomUUID()
})
}
);
}
export async function downloadProviderExecution(
settings: ApiSettings,
execution: ProviderReportExecution,
format: "json" | "csv",
purpose: string,
audienceScope: Record<string, unknown>
): Promise<void> {
const response = await fetch(
apiUrl(settings, `/api/v1/reporting/provider-executions/${encodeURIComponent(execution.execution_id)}/exports`),
{
method: "POST",
headers: { ...authHeaders(settings), "Content-Type": "application/json" },
credentials: "include",
body: JSON.stringify({ format, purpose, audience_scope: audienceScope })
}
);
if (!response.ok) throw new Error(await response.text());
const blob = await response.blob();
const disposition = response.headers.get("Content-Disposition") ?? "";
const filename = disposition.match(/filename="?([^";]+)"?/)?.[1] ?? `provider-report.${format}`;
const link = document.createElement("a");
link.href = URL.createObjectURL(blob);
link.download = filename;
link.click();
URL.revokeObjectURL(link.href);
}
export function getDefinition( export function getDefinition(
settings: ApiSettings, settings: ApiSettings,
kind: ReportingDefinitionKind, kind: ReportingDefinitionKind,
@@ -0,0 +1,295 @@
import { Download, FileJson, Play, ShieldCheck } from "lucide-react";
import { useEffect, useMemo, useState } from "react";
import {
Button,
Card,
DismissibleAlert,
IconButton,
MetricCard,
StatusBadge,
type ApiSettings,
type AuthInfo
} from "@govoplan/core-webui";
import {
downloadProviderExecution,
listProviderParameterOptions,
runProviderReport,
type ProviderReportDescriptor,
type ProviderReportExecution,
type ProviderReportField
} from "../../api/reporting";
export function ProviderReportWorkspace({ settings, auth, report }: {
settings: ApiSettings;
auth: AuthInfo;
report: ProviderReportDescriptor;
}) {
const [parameters, setParameters] = useState<Record<string, unknown>>({});
const [options, setOptions] = useState<Record<string, Array<{ value: string; label: string; description?: string | null }>>>({});
const [purpose, setPurpose] = useState("");
const [execution, setExecution] = useState<ProviderReportExecution | null>(null);
const [running, setRunning] = useState(false);
const [error, setError] = useState("");
const tenant = auth.active_tenant ?? auth.tenant;
const audienceScope = useMemo(() => ({
scope_type: "tenant",
scope_id: tenant.id,
label: tenant.name
}), [tenant.id, tenant.name]);
useEffect(() => {
setParameters({});
setPurpose("");
setExecution(null);
setError("");
const controller = new AbortController();
const optionParameters = report.parameters.filter((item) => item.options_from_provider);
void Promise.all(optionParameters.map(async (parameter) => {
const result = await listProviderParameterOptions(
settings,
report,
parameter.key,
"",
controller.signal
);
return [parameter.key, result.options] as const;
})).then((entries) => {
setOptions(Object.fromEntries(entries));
setParameters(Object.fromEntries(entries.flatMap(([key, values]) =>
values[0] ? [[key, values[0].value]] : []
)));
}).catch((reason) => {
if ((reason as Error).name !== "AbortError") setError(message(reason));
});
return () => controller.abort();
}, [settings, report.provider_id, report.report_id, report.revision]);
async function run() {
setRunning(true);
setError("");
try {
setExecution(await runProviderReport(
settings,
report,
parameters,
purpose.trim(),
audienceScope
));
} catch (reason) {
setError(message(reason));
} finally {
setRunning(false);
}
}
async function download(format: "json" | "csv") {
if (!execution) return;
try {
await downloadProviderExecution(
settings,
execution,
format,
purpose.trim(),
audienceScope
);
} catch (reason) {
setError(message(reason));
}
}
const missingRequired = report.parameters.some((item) =>
item.required && (parameters[item.key] === undefined || parameters[item.key] === "")
);
return (
<>
<header className="reporting-result-header">
<div className="reporting-title">
<span>{report.provider_id} · {report.revision}</span>
<h1>{report.title}</h1>
<p>{report.summary}</p>
</div>
<div className="reporting-run-actions">
<Button
variant="primary"
onClick={() => void run()}
disabled={!report.available || running || missingRequired || !purpose.trim()}>
<Play size={16} aria-hidden="true" /> {running ? "Running" : "Run"}
</Button>
</div>
</header>
{error && <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert>}
{!report.available &&
<DismissibleAlert tone="warning" dismissible={false}>
{report.unavailable_reason ?? "Policy does not allow this report."}
</DismissibleAlert>
}
<div className="reporting-query-controls reporting-provider-controls">
{report.parameters.map((parameter) =>
<label className="reporting-parameter" key={parameter.key}>
<span>{parameter.label}{parameter.required ? " *" : ""}</span>
{parameter.options_from_provider ?
<select
value={String(parameters[parameter.key] ?? "")}
onChange={(event) => setParameters((current) => ({ ...current, [parameter.key]: event.target.value }))}>
{!parameter.required && <option value="">Current/default</option>}
{(options[parameter.key] ?? []).map((item) =>
<option value={item.value} key={item.value}>{item.label}{item.description ? ` · ${item.description}` : ""}</option>
)}
</select> :
<input
type={parameter.type === "integer" || parameter.type === "number" ? "number" : parameter.type === "date" ? "date" : "text"}
value={String(parameters[parameter.key] ?? "")}
onChange={(event) => setParameters((current) => ({ ...current, [parameter.key]: event.target.value }))}
/>
}
{parameter.description && <small>{parameter.description}</small>}
</label>
)}
<label className="reporting-parameter reporting-provider-purpose">
<span>Purpose *</span>
<input
value={purpose}
maxLength={1000}
onChange={(event) => setPurpose(event.target.value)}
placeholder="Purpose recorded with this execution"
/>
</label>
<label className="reporting-parameter">
<span>Effective audience</span>
<input value={tenant.name} readOnly />
</label>
</div>
{execution ?
<>
<div className="reporting-output-toolbar">
<span>Generated {formatDateTime(execution.generated_at)}</span>
{report.export_formats.includes("csv") &&
<IconButton label="Download CSV" icon={<Download size={17} />} variant="ghost" onClick={() => void download("csv")} />
}
{report.export_formats.includes("json") &&
<IconButton label="Download JSON" icon={<FileJson size={17} />} variant="ghost" onClick={() => void download("json")} />
}
</div>
<div className="reporting-provider-output">
<ProviderResult execution={execution} />
</div>
</> :
<div className="reporting-empty">Select the parameters and run this governed report.</div>
}
</>
);
}
export function ProviderReportInspector({ report }: { report: ProviderReportDescriptor }) {
return (
<div className="reporting-inspector-content">
<section className="reporting-provenance">
<h2><ShieldCheck size={16} /> Governance</h2>
<dl>
<dt>Risk</dt><dd><StatusBadge status={report.reidentification_risk === "high" ? "warning" : "active"} label={humanize(report.reidentification_risk)} /></dd>
<dt>Retention</dt><dd>{report.governance.retention_days == null ? "Policy managed" : `${report.governance.retention_days} days`}</dd>
<dt>Exports</dt><dd>{report.governance.export_formats.join(", ") || "Disabled"}</dd>
<dt>Contract</dt><dd>{report.contract_version}</dd>
</dl>
</section>
<section>
<h2>Privacy transforms</h2>
{report.privacy_transforms.map((item) =>
<p key={item.id}>{item.label}{item.required ? " · required" : ""}</p>
)}
</section>
</div>
);
}
function ProviderResult({ execution }: { execution: ProviderReportExecution }) {
const groups = groupFields(execution.result_schema);
return (
<>
{[...groups].map(([group, fields]) => {
const metrics = fields.filter((field) => field.type === "suppressed_count");
const details = fields.filter((field) => field.type !== "suppressed_count");
return (
<Card title={group} key={group}>
{metrics.length > 0 &&
<div className="dashboard-grid reporting-provider-metrics">
{metrics.map((field) =>
<MetricCard key={field.path} label={field.label} value={displayValue(pathValue(execution.result, field.path))} />
)}
</div>
}
{details.length > 0 &&
<dl className="detail-list">
{details.map((field) =>
<div key={field.path}>
<dt>{field.label}</dt>
<dd>{displayValue(pathValue(execution.result, field.path), field)}</dd>
</div>
)}
</dl>
}
</Card>
);
})}
<Card title="Provenance">
<dl className="detail-list">
<div><dt>Purpose</dt><dd>{execution.purpose}</dd></div>
<div><dt>Output hash</dt><dd title={execution.output_hash}>{shortHash(execution.output_hash)}</dd></div>
<div><dt>Source revisions</dt><dd>{execution.source_revisions.length}</dd></div>
<div><dt>Privacy transforms</dt><dd>{execution.privacy_transforms.join(", ")}</dd></div>
<div><dt>Expires</dt><dd>{execution.expires_at ? formatDateTime(execution.expires_at) : "Policy managed"}</dd></div>
</dl>
</Card>
</>
);
}
function groupFields(fields: ProviderReportField[]): Map<string, ProviderReportField[]> {
const groups = new Map<string, ProviderReportField[]>();
for (const field of fields) groups.set(field.group, [...(groups.get(field.group) ?? []), field]);
return groups;
}
function pathValue(payload: Record<string, unknown>, path: string): unknown {
let value: unknown = payload;
for (const part of path.split(".")) {
if (!value || typeof value !== "object") return null;
value = (value as Record<string, unknown>)[part];
}
return value;
}
function displayValue(value: unknown, field?: ProviderReportField): string | number {
if (value && typeof value === "object") {
const count = value as { value?: unknown; suppressed?: boolean };
if (count.suppressed) return "Suppressed";
if ("value" in count) return displayValue(count.value);
return JSON.stringify(value);
}
if (value === null || value === undefined || value === "") return "—";
if (field?.type === "datetime" || field?.type === "date") return formatDateTime(String(value));
if (typeof value === "number") return new Intl.NumberFormat().format(value);
if (typeof value === "boolean") return value ? "Yes" : "No";
return String(value);
}
function formatDateTime(value: string): string {
const date = new Date(value);
return Number.isNaN(date.valueOf()) ? value : new Intl.DateTimeFormat(undefined, { dateStyle: "medium", timeStyle: "short" }).format(date);
}
function humanize(value: string): string {
return value.replaceAll("_", " ").replace(/\b\w/g, (letter) => letter.toUpperCase());
}
function shortHash(value: string): string {
return `${value.slice(0, 10)}${value.slice(-6)}`;
}
function message(reason: unknown): string {
return reason instanceof Error ? reason.message : "The provider report could not be loaded.";
}
+65 -15
View File
@@ -37,6 +37,7 @@ import {
getDefinition, getDefinition,
listDefinitions, listDefinitions,
listExecutions, listExecutions,
listProviderReports,
listSavedViews, listSavedViews,
reportPayload, reportPayload,
runReport, runReport,
@@ -47,8 +48,13 @@ import {
type ReportingQuery, type ReportingQuery,
type ReportingQueryMode, type ReportingQueryMode,
type ReportingSavedView, type ReportingSavedView,
type ProviderReportDescriptor,
type SemanticModelPayload type SemanticModelPayload
} from "../../api/reporting"; } from "../../api/reporting";
import {
ProviderReportInspector,
ProviderReportWorkspace
} from "./ProviderReportWorkspace";
type OutputMode = "visual" | "table"; type OutputMode = "visual" | "table";
@@ -57,7 +63,9 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
const [search, setSearch] = useState(""); const [search, setSearch] = useState("");
const [submittedSearch, setSubmittedSearch] = useState(""); const [submittedSearch, setSubmittedSearch] = useState("");
const [reports, setReports] = useState<ReportingDefinition[]>([]); const [reports, setReports] = useState<ReportingDefinition[]>([]);
const [providerReports, setProviderReports] = useState<ProviderReportDescriptor[]>([]);
const [selectedId, setSelectedId] = useState(""); const [selectedId, setSelectedId] = useState("");
const [selectedProviderKey, setSelectedProviderKey] = useState("");
const [semantic, setSemantic] = useState<SemanticModelPayload | null>(null); const [semantic, setSemantic] = useState<SemanticModelPayload | null>(null);
const [query, setQuery] = useState<ReportingQuery>(emptyQuery()); const [query, setQuery] = useState<ReportingQuery>(emptyQuery());
const [parameters, setParameters] = useState<Record<string, unknown>>({}); const [parameters, setParameters] = useState<Record<string, unknown>>({});
@@ -77,21 +85,42 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
() => reports.find((item) => item.definition_id === selectedId) ?? null, () => reports.find((item) => item.definition_id === selectedId) ?? null,
[reports, selectedId] [reports, selectedId]
); );
const selectedProvider = useMemo(
() => providerReports.find((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey) ?? null,
[providerReports, selectedProviderKey]
);
const report = selected ? reportPayload(selected) : null; const report = selected ? reportPayload(selected) : null;
function reload(signal?: AbortSignal) { function reload(signal?: AbortSignal) {
setLoading(true); setLoading(true);
setError(""); setError("");
return listDefinitions(settings, { return Promise.all([
kinds: ["report"], listDefinitions(settings, {
status: ["active"], kinds: ["report"],
query: submittedSearch, status: ["active"],
limit: 200 query: submittedSearch,
}, signal). limit: 200
then((result) => { }, signal),
listProviderReports(settings, signal)
]).
then(([result, providerResult]) => {
const providerRows = providerResult.reports.filter((item) => {
const query = submittedSearch.toLocaleLowerCase();
return !query || `${item.title} ${item.summary} ${item.provider_id}`.toLocaleLowerCase().includes(query);
});
setReports(result.definitions); setReports(result.definitions);
setSelectedId((current) => result.definitions.some((item) => item.definition_id === current) ? setProviderReports(providerRows);
current : result.definitions[0]?.definition_id ?? ""); const currentSemanticAvailable = result.definitions.some((item) => item.definition_id === selectedId);
const currentProviderAvailable = providerRows.some((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey);
if (!currentSemanticAvailable && !currentProviderAvailable) {
if (providerRows[0]) {
setSelectedProviderKey(`${providerRows[0].provider_id}:${providerRows[0].report_id}`);
setSelectedId("");
} else {
setSelectedId(result.definitions[0]?.definition_id ?? "");
setSelectedProviderKey("");
}
}
}). }).
catch((reason) => { catch((reason) => {
if ((reason as Error).name !== "AbortError") setError(message(reason)); if ((reason as Error).name !== "AbortError") setError(message(reason));
@@ -172,7 +201,7 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
placeholder="Search reports" placeholder="Search reports"
/> />
</form> </form>
<span className="reporting-count">{reports.length} reports</span> <span className="reporting-count">{reports.length + providerReports.length} reports</span>
<IconButton <IconButton
label="Reload reports" label="Reload reports"
icon={<RefreshCw size={17} />} icon={<RefreshCw size={17} />}
@@ -188,15 +217,32 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
<div className="reporting-workspace"> <div className="reporting-workspace">
<PageScrollViewport className="reporting-catalogue"> <PageScrollViewport className="reporting-catalogue">
{loading && <LoadingIndicator label="Loading reports" />} {loading && <LoadingIndicator label="Loading reports" />}
{!loading && reports.length === 0 && <div className="reporting-empty">No active reports are available.</div>} {!loading && reports.length + providerReports.length === 0 && <div className="reporting-empty">No active reports are available.</div>}
<div className="reporting-report-list" role="list"> <div className="reporting-report-list" role="list">
{providerReports.length > 0 && <div className="reporting-list-heading">Module reports</div>}
{providerReports.map((item) => {
const key = `${item.provider_id}:${item.report_id}`;
return (
<button
type="button"
role="listitem"
key={key}
className={`reporting-report-row${selectedProviderKey === key ? " is-selected" : ""}`}
onClick={() => { setSelectedProviderKey(key); setSelectedId(""); }}>
<BarChart3 size={17} aria-hidden="true" />
<span><strong>{item.title}</strong><small>{item.provider_id} · {item.revision}</small></span>
<StatusBadge status={item.available ? "active" : "locked"} label={item.available ? "Available" : "Restricted"} />
</button>
);
})}
{reports.length > 0 && <div className="reporting-list-heading">Semantic reports</div>}
{reports.map((item) => {reports.map((item) =>
<button <button
type="button" type="button"
role="listitem" role="listitem"
key={item.definition_id} key={item.definition_id}
className={`reporting-report-row${selectedId === item.definition_id ? " is-selected" : ""}`} className={`reporting-report-row${selectedId === item.definition_id ? " is-selected" : ""}`}
onClick={() => setSelectedId(item.definition_id)}> onClick={() => { setSelectedId(item.definition_id); setSelectedProviderKey(""); }}>
<BarChart3 size={17} aria-hidden="true" /> <BarChart3 size={17} aria-hidden="true" />
<span><strong>{item.name}</strong><small>{item.definition_key} · r{item.revision}</small></span> <span><strong>{item.name}</strong><small>{item.definition_key} · r{item.revision}</small></span>
<StatusBadge status="active" label="Active" /> <StatusBadge status="active" label="Active" />
@@ -205,7 +251,9 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
</div> </div>
</PageScrollViewport> </PageScrollViewport>
<section className="reporting-result-region"> <section className="reporting-result-region">
{selected && report ? {selectedProvider ?
<ProviderReportWorkspace settings={settings} auth={auth} report={selectedProvider} /> :
selected && report ?
<> <>
<header className="reporting-result-header"> <header className="reporting-result-header">
<div className="reporting-title"> <div className="reporting-title">
@@ -258,14 +306,16 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
} }
</section> </section>
<PageScrollViewport className="reporting-inspector"> <PageScrollViewport className="reporting-inspector">
<Inspector {selectedProvider ?
<ProviderReportInspector report={selectedProvider} /> :
<Inspector
selected={selected} selected={selected}
execution={execution} execution={execution}
history={history} history={history}
savedViews={savedViews} savedViews={savedViews}
onSelectExecution={setExecution} onSelectExecution={setExecution}
onApplySavedView={applySavedView} onApplySavedView={applySavedView}
/> />}
</PageScrollViewport> </PageScrollViewport>
</div> </div>
</div> </div>
+9 -2
View File
@@ -24,16 +24,23 @@ export const reportingModule: PlatformWebModule = {
], ],
routes: [ routes: [
{ {
path: "/reporting", path: "/reports",
anyOf: ["reporting:definition:read"], anyOf: ["reporting:definition:read"],
order: 74, order: 74,
surfaceId: "reporting.workspace", surfaceId: "reporting.workspace",
render: (context) => createElement(ReportingPage, context) render: (context) => createElement(ReportingPage, context)
},
{
path: "/reporting",
anyOf: ["reporting:definition:read"],
order: 175,
surfaceId: "reporting.compatibility",
render: (context) => createElement(ReportingPage, context)
} }
], ],
navItems: [ navItems: [
{ {
to: "/reporting", to: "/reports",
label: "Reporting", label: "Reporting",
iconName: "clipboard-pen-line", iconName: "clipboard-pen-line",
anyOf: ["reporting:definition:read"], anyOf: ["reporting:definition:read"],
+43
View File
@@ -74,6 +74,16 @@
background: var(--surface-raised); background: var(--surface-raised);
} }
.reporting-list-heading {
padding: 7px 10px;
border-bottom: 1px solid var(--border);
background: var(--surface-subtle, var(--surface));
color: var(--text-soft);
font-size: 0.7rem;
font-weight: 700;
text-transform: uppercase;
}
.reporting-report-row, .reporting-report-row,
.reporting-inspector-content section > button { .reporting-inspector-content section > button {
display: grid; display: grid;
@@ -162,6 +172,16 @@
white-space: nowrap; white-space: nowrap;
} }
.reporting-title p {
max-width: 70ch;
margin: 3px 0 0;
overflow: hidden;
color: var(--text-soft);
font-size: 0.76rem;
text-overflow: ellipsis;
white-space: nowrap;
}
.reporting-run-actions { .reporting-run-actions {
display: flex; display: flex;
align-items: center; align-items: center;
@@ -223,6 +243,14 @@
gap: 4px; gap: 4px;
} }
.reporting-provider-controls {
align-items: end;
}
.reporting-provider-purpose {
min-width: min(320px, 35vw);
}
.reporting-output-toolbar { .reporting-output-toolbar {
min-height: 48px; min-height: 48px;
padding: 6px 12px; padding: 6px 12px;
@@ -241,6 +269,21 @@
padding: 10px; padding: 10px;
} }
.reporting-provider-output {
display: flex;
min-height: 0;
flex: 1;
flex-direction: column;
gap: 10px;
overflow: auto;
padding: 10px;
}
.reporting-provider-metrics {
margin: 0;
padding: 0;
}
.reporting-output .data-grid-shell { .reporting-output .data-grid-shell {
min-width: 100%; min-width: 100%;
} }