feat: add governed cross-module reporting
This commit is contained in:
@@ -20,6 +20,9 @@ The module now provides an executable governed semantic-reporting vertical:
|
||||
arbitrary report SQL;
|
||||
- quality gates, saved views, interval/scheduled runs, CSV/JSON export,
|
||||
provider-neutral publication targets, and import activation assessments;
|
||||
- a versioned, provider-neutral cross-module report contract with source-owned
|
||||
authorization, declared result schemas, privacy transforms, effective scope,
|
||||
source revisions, purpose, retention, export history, and audit provenance;
|
||||
- a full-height Reporting workspace for running, inspecting, saving,
|
||||
scheduling, visualizing, and exporting authorized reports.
|
||||
|
||||
@@ -27,6 +30,10 @@ Reporting consumes Dataflow outputs or provider-owned read models. It does not
|
||||
read another module's ORM tables or take ownership of ingestion and
|
||||
transformation.
|
||||
|
||||
The canonical global route is `/reports`. `/reporting` remains a
|
||||
Reporting-owned compatibility route for saved links. Domain modules may keep
|
||||
their own operational report routes, but do not register `/reports`.
|
||||
|
||||
See [docs/REPORTING_BOUNDARY.md](docs/REPORTING_BOUNDARY.md) for the boundary
|
||||
decision. The behavior-level comparison with the supplied SuperX module set is
|
||||
recorded in
|
||||
|
||||
@@ -32,6 +32,36 @@ the Reporting publication-target contract. The target receives one immutable
|
||||
execution payload and an idempotency key. It must return bounded evidence and
|
||||
must not expose credentials in that evidence.
|
||||
|
||||
## Cross-module provider governance
|
||||
|
||||
Source modules register `reporting.report_provider.<provider-id>` capabilities;
|
||||
do not grant Reporting direct table access to those modules. Review every
|
||||
descriptor's result schema, required privacy transforms, retention class,
|
||||
export formats, and re-identification risk before enabling it in production.
|
||||
Provider authorization remains mandatory even when the Reporting role allows
|
||||
the user to run reports.
|
||||
|
||||
When Policy is enabled, configure `reporting_governance_policy` in system or
|
||||
tenant settings. Tenant settings may only tighten the system result. Supported
|
||||
fields are:
|
||||
|
||||
```json
|
||||
{
|
||||
"reporting_governance_policy": {
|
||||
"allow_exports": true,
|
||||
"allowed_export_formats": ["json"],
|
||||
"allow_high_reidentification_risk": false,
|
||||
"max_retention_days": 30,
|
||||
"required_privacy_transforms": ["small_cell_suppression"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Malformed explicit policy fails closed. The ordinary privacy-retention setting
|
||||
`stored_report_detail_retention_days` is an additional ceiling. A tenant cannot
|
||||
re-enable an export format, high-risk report, or longer retention period that
|
||||
the system policy denied.
|
||||
|
||||
## Import assessments
|
||||
|
||||
Import assessment accepts declarative metadata only. Native datasets,
|
||||
|
||||
@@ -12,6 +12,15 @@ claim bounded batches, retain success or failure state, and advance interval
|
||||
time from the claimed occurrence. Providers must use the same tenant and
|
||||
principal context and must enforce their source-side authorization.
|
||||
|
||||
Provider-report executions and exports are database evidence. Each execution
|
||||
stores the exact provider/report revision, request hash, source revisions,
|
||||
effective scope, applied privacy transforms, governance decision provenance,
|
||||
generation actor/time, retention expiry, and output hash. Export rows retain
|
||||
their own purpose, audience, actor/time, format, and hash. Expired provider
|
||||
results are rejected by retrieval and export paths; run the platform privacy
|
||||
retention process on schedule to remove or minimize expired stored detail in
|
||||
accordance with the active Policy.
|
||||
|
||||
## Security controls
|
||||
|
||||
- The semantic engine never evaluates Python, browser code, raw SQL, stored
|
||||
@@ -25,6 +34,10 @@ principal context and must enforce their source-side authorization.
|
||||
- Freshness, schema, fingerprints, quality gates, definition hashes, source
|
||||
provenance, actor, and output hash are retained with every run.
|
||||
- CSV formula injection is neutralized before direct download.
|
||||
- Provider exceptions are isolated in catalogue diagnostics; an unavailable
|
||||
optional provider cannot take down the global report catalogue.
|
||||
- High re-identification-risk provider reports fail closed unless an installed
|
||||
Policy capability explicitly permits them.
|
||||
|
||||
## Backup and restore
|
||||
|
||||
|
||||
@@ -87,6 +87,29 @@ Reporting does not own:
|
||||
- `reporting.publication_target:*` capabilities can accept immutable result
|
||||
payloads without Reporting importing the target module.
|
||||
|
||||
### Cross-module report providers
|
||||
|
||||
Source modules contribute aggregate or otherwise minimized reports through
|
||||
`reporting.report_provider.<provider-id>`. Core owns contract version `1.0` and
|
||||
the provider-neutral DTOs. A provider must declare stable report and revision
|
||||
IDs, typed parameters, the complete result schema, privacy transforms,
|
||||
retention class, export formats, and re-identification risk. It must enforce
|
||||
source access before listing options or producing a result and must return
|
||||
source revisions, effective tenant scope, applied transforms, generation time,
|
||||
and bounded provenance.
|
||||
|
||||
Reporting discovers these capabilities by prefix. It validates descriptors,
|
||||
rejects undeclared output fields, requires every mandatory transform, records
|
||||
purpose and effective audience, and persists immutable execution and export
|
||||
evidence. The optional `policy.reporting_governance` capability can tighten
|
||||
retention, export formats, required transforms, and high-risk handling. If
|
||||
Policy is absent, Reporting applies its restrictive built-in baseline and
|
||||
denies high-risk provider reports.
|
||||
|
||||
This direction is intentionally one-way: source modules import only Core's
|
||||
contract and remain usable when Reporting is absent. Reporting never imports
|
||||
the source module or reads its ORM tables.
|
||||
|
||||
## Implemented Vertical
|
||||
|
||||
The first complete vertical persists datasets, semantic models, reports,
|
||||
@@ -120,6 +143,11 @@ unsupported executable behavior remains.
|
||||
The WebUI uses the platform module loader and common controls. It exposes a
|
||||
report catalogue, parameter and semantic-query controls, result visualization
|
||||
and table views, history/provenance, saved views, schedules, and downloads.
|
||||
The global `/reports` route is owned only by Reporting. `/reporting` is a
|
||||
documented compatibility path. Campaign's module-local aggregate view remains
|
||||
at `/campaigns/reports`; when both modules are enabled, the same safe aggregate
|
||||
projection is also contributed to the global catalogue through the provider
|
||||
contract.
|
||||
|
||||
## Remaining Product Depth
|
||||
|
||||
|
||||
+18
-2
@@ -2,8 +2,9 @@
|
||||
|
||||
## Open and run a report
|
||||
|
||||
Open **Reporting** and choose an active report from the left catalogue. The
|
||||
header identifies the immutable report revision that will be used. Select a
|
||||
Open **Reporting** at `/reports` and choose a report from the left catalogue.
|
||||
The catalogue separates source-module reports from Reporting semantic reports.
|
||||
The header identifies the immutable report revision that will be used. Select a
|
||||
summary, detail, or pivot shape, choose dimensions and measures, enter any
|
||||
report parameters, and select **Run**.
|
||||
|
||||
@@ -30,3 +31,18 @@ spreadsheet formula marker are escaped.
|
||||
Users with scheduling permission can create an hourly, daily, weekly, or
|
||||
30-day interval from the current revision, parameters, and query. Scheduled
|
||||
runs continue to use those exact pins until the schedule is edited.
|
||||
|
||||
## Run a module report
|
||||
|
||||
Module reports retain their source module's access rules. Select the source
|
||||
object and any optional parameters, enter the concrete purpose for the run,
|
||||
and verify the effective tenant audience before selecting **Run**. Reporting
|
||||
will refuse a result that does not carry the declared privacy transforms,
|
||||
source revision, and tenant scope.
|
||||
|
||||
The result groups declared metrics and details without exposing undeclared
|
||||
provider data. The Governance panel explains the risk class, permitted export
|
||||
formats, retention ceiling, contract version, and required privacy transforms.
|
||||
An export records a separate purpose, actor, audience, time, format, and output
|
||||
hash. A missing or restricted module report is not replaced with a less
|
||||
protected client-side query.
|
||||
|
||||
@@ -218,6 +218,125 @@ class ReportingExecution(Base, TimestampMixin):
|
||||
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
|
||||
|
||||
class ReportingProviderExecution(Base, TimestampMixin):
|
||||
__tablename__ = "reporting_provider_executions"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"execution_id",
|
||||
name="uq_reporting_provider_execution",
|
||||
),
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_reporting_provider_execution_idempotency",
|
||||
),
|
||||
Index(
|
||||
"ix_reporting_provider_execution_history",
|
||||
"tenant_id",
|
||||
"provider_id",
|
||||
"report_id",
|
||||
"generated_at",
|
||||
),
|
||||
Index(
|
||||
"ix_reporting_provider_execution_retention",
|
||||
"tenant_id",
|
||||
"expires_at",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
provider_id: Mapped[str] = mapped_column(String(120), nullable=False, index=True)
|
||||
report_id: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
|
||||
report_revision: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
contract_version: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
idempotency_key: Mapped[str] = mapped_column(String(255), nullable=False)
|
||||
request_sha256: Mapped[str] = mapped_column(String(64), nullable=False)
|
||||
purpose: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
audience_scope: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
parameters: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
result_schema: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
result_payload: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
source_revisions: Mapped[list[dict[str, Any]]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
effective_scope: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
privacy_transforms: Mapped[list[str]] = mapped_column(
|
||||
JSON, default=list, nullable=False
|
||||
)
|
||||
provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
governance_provenance: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
retention_class: Mapped[str] = mapped_column(String(120), nullable=False)
|
||||
retention_days: Mapped[int | None] = mapped_column(Integer, nullable=True)
|
||||
expires_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
retention_redacted_at: Mapped[datetime | None] = mapped_column(
|
||||
DateTime(timezone=True), nullable=True, index=True
|
||||
)
|
||||
output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
generated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
|
||||
|
||||
class ReportingProviderExport(Base, TimestampMixin):
|
||||
__tablename__ = "reporting_provider_exports"
|
||||
__table_args__ = (
|
||||
UniqueConstraint(
|
||||
"tenant_id",
|
||||
"export_id",
|
||||
name="uq_reporting_provider_export",
|
||||
),
|
||||
Index(
|
||||
"ix_reporting_provider_export_history",
|
||||
"tenant_id",
|
||||
"provider_execution_id",
|
||||
"exported_at",
|
||||
),
|
||||
)
|
||||
|
||||
id: Mapped[str] = mapped_column(String(36), primary_key=True, default=new_uuid)
|
||||
tenant_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
export_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
provider_execution_id: Mapped[str] = mapped_column(
|
||||
ForeignKey(
|
||||
"reporting_provider_executions.id",
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
nullable=False,
|
||||
index=True,
|
||||
)
|
||||
execution_id: Mapped[str] = mapped_column(String(36), nullable=False, index=True)
|
||||
format: Mapped[str] = mapped_column(String(30), nullable=False)
|
||||
purpose: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
audience_scope: Mapped[dict[str, Any]] = mapped_column(
|
||||
JSON, default=dict, nullable=False
|
||||
)
|
||||
output_hash: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
|
||||
exported_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True), nullable=False, index=True
|
||||
)
|
||||
actor_id: Mapped[str | None] = mapped_column(String(255), nullable=True, index=True)
|
||||
|
||||
|
||||
class ReportingDefinitionGrant(Base, TimestampMixin):
|
||||
__tablename__ = "reporting_definition_grants"
|
||||
__table_args__ = (
|
||||
@@ -435,6 +554,8 @@ __all__ = [
|
||||
"ReportingExecution",
|
||||
"ReportingImportAssessment",
|
||||
"ReportingPublication",
|
||||
"ReportingProviderExecution",
|
||||
"ReportingProviderExport",
|
||||
"ReportingQualityResult",
|
||||
"ReportingSavedView",
|
||||
"ReportingSchedule",
|
||||
|
||||
@@ -31,6 +31,10 @@ from govoplan_core.core.provider_governance import (
|
||||
ModuleArchitectureDocumentation,
|
||||
ModuleMaturityEvidence,
|
||||
)
|
||||
from govoplan_core.core.reporting import (
|
||||
CAPABILITY_POLICY_REPORTING_GOVERNANCE,
|
||||
CAPABILITY_REPORTING_RETENTION,
|
||||
)
|
||||
from govoplan_core.core.search import SearchSourceProviderRegistration
|
||||
from govoplan_core.core.views import ViewSurface
|
||||
from govoplan_core.db.base import Base
|
||||
@@ -171,6 +175,13 @@ def _chart_renderer(context: ModuleContext) -> DefaultChartRenderer:
|
||||
return DefaultChartRenderer()
|
||||
|
||||
|
||||
def _retention(context: ModuleContext):
|
||||
del context
|
||||
from govoplan_reporting.backend.retention import ReportingRetentionService
|
||||
|
||||
return ReportingRetentionService()
|
||||
|
||||
|
||||
def _tenant_summary(session, tenant_id: str) -> dict[str, int]:
|
||||
definitions = (
|
||||
session.query(reporting_models.ReportingDefinitionRevision)
|
||||
@@ -232,17 +243,21 @@ manifest = ModuleManifest(
|
||||
CAPABILITY_AUTH_PRINCIPAL_RESOLVER,
|
||||
CAPABILITY_AUTH_PERMISSION_EVALUATOR,
|
||||
),
|
||||
optional_capabilities=(CAPABILITY_DATAFLOW_DATASET_OUTPUT,),
|
||||
optional_capabilities=(
|
||||
CAPABILITY_DATAFLOW_DATASET_OUTPUT,
|
||||
CAPABILITY_POLICY_REPORTING_GOVERNANCE,
|
||||
),
|
||||
permissions=PERMISSIONS,
|
||||
role_templates=ROLE_TEMPLATES,
|
||||
route_factory=_router,
|
||||
nav_items=(
|
||||
NavItem(
|
||||
path="/reporting",
|
||||
path="/reports",
|
||||
label="Reporting",
|
||||
icon="clipboard-pen-line",
|
||||
required_any=(READ_SCOPE,),
|
||||
order=74,
|
||||
surface_id="reporting.navigation",
|
||||
),
|
||||
),
|
||||
frontend=FrontendModule(
|
||||
@@ -250,36 +265,31 @@ manifest = ModuleManifest(
|
||||
package_name="@govoplan/reporting-webui",
|
||||
routes=(
|
||||
FrontendRoute(
|
||||
path="/reporting",
|
||||
path="/reports",
|
||||
component="ReportingPage",
|
||||
required_any=(READ_SCOPE,),
|
||||
order=74,
|
||||
surface_id="reporting.workspace",
|
||||
),
|
||||
FrontendRoute(
|
||||
path="/reporting",
|
||||
component="ReportingPage",
|
||||
required_any=(READ_SCOPE,),
|
||||
order=175,
|
||||
surface_id="reporting.compatibility",
|
||||
),
|
||||
),
|
||||
nav_items=(
|
||||
NavItem(
|
||||
path="/reporting",
|
||||
path="/reports",
|
||||
label="Reporting",
|
||||
icon="clipboard-pen-line",
|
||||
required_any=(READ_SCOPE,),
|
||||
order=74,
|
||||
surface_id="reporting.navigation",
|
||||
),
|
||||
),
|
||||
view_surfaces=(
|
||||
ViewSurface(
|
||||
id="reporting.navigation",
|
||||
module_id=MODULE_ID,
|
||||
kind="navigation",
|
||||
label="Reporting navigation",
|
||||
order=10,
|
||||
),
|
||||
ViewSurface(
|
||||
id="reporting.workspace",
|
||||
module_id=MODULE_ID,
|
||||
kind="route",
|
||||
label="Reporting workspace",
|
||||
order=20,
|
||||
),
|
||||
ViewSurface(
|
||||
id="reporting.parameters",
|
||||
module_id=MODULE_ID,
|
||||
@@ -303,6 +313,7 @@ manifest = ModuleManifest(
|
||||
ModuleInterfaceProvider(name="reporting.runner", version="0.1.0"),
|
||||
ModuleInterfaceProvider(name="reporting.scheduler", version="0.1.0"),
|
||||
ModuleInterfaceProvider(name="reporting.chart_renderer", version="0.1.0"),
|
||||
ModuleInterfaceProvider(name=CAPABILITY_REPORTING_RETENTION, version="1.0.0"),
|
||||
),
|
||||
requires_interfaces=(
|
||||
ModuleInterfaceRequirement(
|
||||
@@ -311,12 +322,19 @@ manifest = ModuleManifest(
|
||||
version_max_exclusive="0.2.0",
|
||||
optional=True,
|
||||
),
|
||||
ModuleInterfaceRequirement(
|
||||
name=CAPABILITY_POLICY_REPORTING_GOVERNANCE,
|
||||
version_min="1.0.0",
|
||||
version_max_exclusive="2.0.0",
|
||||
optional=True,
|
||||
),
|
||||
),
|
||||
capability_factories={
|
||||
CAPABILITY_REPORTING_REGISTRY: _registry,
|
||||
CAPABILITY_REPORTING_RUNNER: _runner,
|
||||
CAPABILITY_REPORTING_SCHEDULER: _scheduler,
|
||||
CAPABILITY_REPORTING_CHART_RENDERER: _chart_renderer,
|
||||
CAPABILITY_REPORTING_RETENTION: _retention,
|
||||
},
|
||||
capability_documentation={
|
||||
CAPABILITY_REPORTING_REGISTRY: CapabilityDocumentation(
|
||||
@@ -339,6 +357,13 @@ manifest = ModuleManifest(
|
||||
summary="Builds provider-neutral chart models with an accessible tabular fallback.",
|
||||
contract_version="0.1.0",
|
||||
),
|
||||
CAPABILITY_REPORTING_RETENTION: CapabilityDocumentation(
|
||||
label="Reporting result retention",
|
||||
summary="Minimizes expired provider-report detail while retaining audit hashes and provenance.",
|
||||
contract_version="1.0",
|
||||
documentation_types=("admin",),
|
||||
audience=("privacy_officer", "operator", "system_admin"),
|
||||
),
|
||||
},
|
||||
search_sources=(
|
||||
SearchSourceProviderRegistration(
|
||||
@@ -359,6 +384,8 @@ manifest = ModuleManifest(
|
||||
reporting_models.ReportingSavedView,
|
||||
reporting_models.ReportingDefinitionGrant,
|
||||
reporting_models.ReportingExecution,
|
||||
reporting_models.ReportingProviderExport,
|
||||
reporting_models.ReportingProviderExecution,
|
||||
reporting_models.ReportingDefinitionRevision,
|
||||
reporting_models.ReportingDefinitionIdentity,
|
||||
label="Reporting",
|
||||
@@ -374,6 +401,8 @@ manifest = ModuleManifest(
|
||||
reporting_models.ReportingDefinitionRevision,
|
||||
reporting_models.ReportingDefinitionGrant,
|
||||
reporting_models.ReportingExecution,
|
||||
reporting_models.ReportingProviderExecution,
|
||||
reporting_models.ReportingProviderExport,
|
||||
reporting_models.ReportingSavedView,
|
||||
reporting_models.ReportingSchedule,
|
||||
reporting_models.ReportingPublication,
|
||||
|
||||
+142
@@ -0,0 +1,142 @@
|
||||
"""Add governed cross-module report executions and export history.
|
||||
|
||||
Revision ID: b7c4e1a9d2f6
|
||||
Revises: e5b2c9d4f7a1
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
revision = "b7c4e1a9d2f6"
|
||||
down_revision = "e5b2c9d4f7a1"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"reporting_provider_executions",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("execution_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("provider_id", sa.String(length=120), nullable=False),
|
||||
sa.Column("report_id", sa.String(length=255), nullable=False),
|
||||
sa.Column("report_revision", sa.String(length=255), nullable=False),
|
||||
sa.Column("contract_version", sa.String(length=30), nullable=False),
|
||||
sa.Column("idempotency_key", sa.String(length=255), nullable=False),
|
||||
sa.Column("request_sha256", sa.String(length=64), nullable=False),
|
||||
sa.Column("purpose", sa.Text(), nullable=False),
|
||||
sa.Column("audience_scope", sa.JSON(), nullable=False),
|
||||
sa.Column("parameters", sa.JSON(), nullable=False),
|
||||
sa.Column("result_schema", sa.JSON(), nullable=False),
|
||||
sa.Column("result_payload", sa.JSON(), nullable=False),
|
||||
sa.Column("source_revisions", sa.JSON(), nullable=False),
|
||||
sa.Column("effective_scope", sa.JSON(), nullable=False),
|
||||
sa.Column("privacy_transforms", sa.JSON(), nullable=False),
|
||||
sa.Column("provenance", sa.JSON(), nullable=False),
|
||||
sa.Column("governance_provenance", sa.JSON(), nullable=False),
|
||||
sa.Column("retention_class", sa.String(length=120), nullable=False),
|
||||
sa.Column("retention_days", sa.Integer(), nullable=True),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("retention_redacted_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("output_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("generated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("actor_id", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_executions")),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"execution_id",
|
||||
name="uq_reporting_provider_execution",
|
||||
),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"idempotency_key",
|
||||
name="uq_reporting_provider_execution_idempotency",
|
||||
),
|
||||
)
|
||||
_indexes(
|
||||
"reporting_provider_executions",
|
||||
"tenant_id",
|
||||
"execution_id",
|
||||
"provider_id",
|
||||
"report_id",
|
||||
"expires_at",
|
||||
"retention_redacted_at",
|
||||
"output_hash",
|
||||
"generated_at",
|
||||
"actor_id",
|
||||
)
|
||||
op.create_index(
|
||||
"ix_reporting_provider_execution_history",
|
||||
"reporting_provider_executions",
|
||||
["tenant_id", "provider_id", "report_id", "generated_at"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_index(
|
||||
"ix_reporting_provider_execution_retention",
|
||||
"reporting_provider_executions",
|
||||
["tenant_id", "expires_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
op.create_table(
|
||||
"reporting_provider_exports",
|
||||
sa.Column("id", sa.String(length=36), nullable=False),
|
||||
sa.Column("tenant_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("export_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("provider_execution_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("execution_id", sa.String(length=36), nullable=False),
|
||||
sa.Column("format", sa.String(length=30), nullable=False),
|
||||
sa.Column("purpose", sa.Text(), nullable=False),
|
||||
sa.Column("audience_scope", sa.JSON(), nullable=False),
|
||||
sa.Column("output_hash", sa.String(length=64), nullable=False),
|
||||
sa.Column("exported_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("actor_id", sa.String(length=255), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.ForeignKeyConstraint(
|
||||
["provider_execution_id"],
|
||||
["reporting_provider_executions.id"],
|
||||
name=op.f(
|
||||
"fk_reporting_provider_exports_provider_execution_id_reporting_provider_executions"
|
||||
),
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id", name=op.f("pk_reporting_provider_exports")),
|
||||
sa.UniqueConstraint(
|
||||
"tenant_id",
|
||||
"export_id",
|
||||
name="uq_reporting_provider_export",
|
||||
),
|
||||
)
|
||||
_indexes(
|
||||
"reporting_provider_exports",
|
||||
"tenant_id",
|
||||
"export_id",
|
||||
"provider_execution_id",
|
||||
"execution_id",
|
||||
"output_hash",
|
||||
"exported_at",
|
||||
"actor_id",
|
||||
)
|
||||
op.create_index(
|
||||
"ix_reporting_provider_export_history",
|
||||
"reporting_provider_exports",
|
||||
["tenant_id", "provider_execution_id", "exported_at"],
|
||||
unique=False,
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("reporting_provider_exports")
|
||||
op.drop_table("reporting_provider_executions")
|
||||
|
||||
|
||||
def _indexes(table: str, *columns: str) -> None:
|
||||
for column in columns:
|
||||
op.create_index(op.f(f"ix_{table}_{column}"), table, [column], unique=False)
|
||||
@@ -0,0 +1,835 @@
|
||||
"""Governed execution boundary for module-contributed reports."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Mapping
|
||||
import csv
|
||||
from datetime import UTC, date, datetime, timedelta
|
||||
from hashlib import sha256
|
||||
import io
|
||||
import json
|
||||
import uuid
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.audit.logging import audit_from_principal
|
||||
from govoplan_core.core.reporting import (
|
||||
ReportDescriptor,
|
||||
ReportParameterOption,
|
||||
ReportProvider,
|
||||
ReportProviderRequest,
|
||||
ReportingGovernanceDecision,
|
||||
ReportingGovernanceRequest,
|
||||
report_providers,
|
||||
reporting_governance_provider,
|
||||
)
|
||||
from govoplan_reporting.backend.db.models import (
|
||||
ReportingProviderExecution,
|
||||
ReportingProviderExport,
|
||||
)
|
||||
|
||||
|
||||
class ProviderReportError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def list_provider_reports(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
) -> dict[str, object]:
|
||||
reports: list[dict[str, object]] = []
|
||||
diagnostics: list[dict[str, str]] = []
|
||||
for provider_id, provider in report_providers(registry):
|
||||
try:
|
||||
for descriptor in provider.list_reports(session, principal):
|
||||
_validate_descriptor(provider_id, descriptor)
|
||||
decision = _governance_decision(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
descriptor=descriptor,
|
||||
action="catalogue",
|
||||
purpose=None,
|
||||
audience_scope={},
|
||||
)
|
||||
reports.append(
|
||||
{
|
||||
**descriptor.to_dict(),
|
||||
"available": decision.allowed,
|
||||
"unavailable_reason": decision.reason,
|
||||
"governance": _decision_payload(decision),
|
||||
}
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 - isolate optional providers.
|
||||
diagnostics.append(
|
||||
{
|
||||
"provider_id": provider_id,
|
||||
"code": "provider_catalogue_failed",
|
||||
"message": "The provider catalogue could not be loaded.",
|
||||
"error_type": type(exc).__name__,
|
||||
}
|
||||
)
|
||||
return {
|
||||
"reports": sorted(
|
||||
reports,
|
||||
key=lambda item: (str(item["title"]), str(item["provider_id"])),
|
||||
),
|
||||
"diagnostics": diagnostics,
|
||||
}
|
||||
|
||||
|
||||
def provider_parameter_options(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
provider_id: str,
|
||||
report_id: str,
|
||||
parameter_key: str,
|
||||
query: str,
|
||||
limit: int,
|
||||
) -> tuple[ReportParameterOption, ...]:
|
||||
provider, descriptor = _provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=provider_id,
|
||||
report_id=report_id,
|
||||
)
|
||||
parameter = next(
|
||||
(item for item in descriptor.parameters if item.key == parameter_key),
|
||||
None,
|
||||
)
|
||||
if parameter is None or not parameter.options_from_provider:
|
||||
raise ProviderReportError("This report parameter has no provider options")
|
||||
return provider.parameter_options(
|
||||
session,
|
||||
principal,
|
||||
report_id=report_id,
|
||||
parameter_key=parameter_key,
|
||||
query=query,
|
||||
limit=max(1, min(limit, 200)),
|
||||
)
|
||||
|
||||
|
||||
def execute_provider_report(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
provider_id: str,
|
||||
report_id: str,
|
||||
parameters: Mapping[str, object],
|
||||
purpose: str,
|
||||
audience_scope: Mapping[str, object],
|
||||
idempotency_key: str,
|
||||
) -> dict[str, object]:
|
||||
provider, descriptor = _provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=provider_id,
|
||||
report_id=report_id,
|
||||
)
|
||||
clean_purpose = purpose.strip()
|
||||
clean_idempotency_key = idempotency_key.strip()
|
||||
clean_parameters = _validated_parameters(descriptor, parameters)
|
||||
clean_audience = _json_mapping(audience_scope, "audience scope")
|
||||
if descriptor.purpose_required and not clean_purpose:
|
||||
raise ProviderReportError("A report purpose is required")
|
||||
if descriptor.audience_scope_required and not clean_audience:
|
||||
raise ProviderReportError("An effective audience scope is required")
|
||||
request_hash = _hash(
|
||||
{
|
||||
"provider_id": provider_id,
|
||||
"report_id": report_id,
|
||||
"revision": descriptor.revision,
|
||||
"parameters": clean_parameters,
|
||||
"purpose": clean_purpose,
|
||||
"audience_scope": clean_audience,
|
||||
}
|
||||
)
|
||||
replay = (
|
||||
session.query(ReportingProviderExecution)
|
||||
.filter(
|
||||
ReportingProviderExecution.tenant_id == _tenant(principal),
|
||||
ReportingProviderExecution.idempotency_key == clean_idempotency_key,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
if replay is not None:
|
||||
if replay.request_sha256 != request_hash:
|
||||
raise ProviderReportError(
|
||||
"The provider-report idempotency key belongs to another request"
|
||||
)
|
||||
if _expired(replay) or replay.retention_redacted_at is not None:
|
||||
raise ProviderReportError(
|
||||
"The replayed provider-report result has expired; use a new idempotency key"
|
||||
)
|
||||
_require_result_access(provider, session, principal, replay)
|
||||
return provider_execution_payload(replay)
|
||||
|
||||
preflight = _governance_decision(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
descriptor=descriptor,
|
||||
action="execute",
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
)
|
||||
_require_allowed(preflight)
|
||||
result = provider.execute_report(
|
||||
session,
|
||||
principal,
|
||||
request=ReportProviderRequest(
|
||||
report_id=report_id,
|
||||
parameters=clean_parameters,
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
),
|
||||
)
|
||||
_validate_result(descriptor, result, tenant_id=_tenant(principal))
|
||||
decision = _governance_decision(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
descriptor=descriptor,
|
||||
action="execute",
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
applied_privacy_transforms=result.applied_privacy_transforms,
|
||||
)
|
||||
_require_allowed(decision)
|
||||
missing_policy_transforms = set(decision.required_privacy_transforms) - set(
|
||||
result.applied_privacy_transforms
|
||||
)
|
||||
if missing_policy_transforms:
|
||||
raise ProviderReportError(
|
||||
"The report omitted Policy-required privacy transformations: "
|
||||
+ ", ".join(sorted(missing_policy_transforms))
|
||||
)
|
||||
generated_at = _aware(result.generated_at)
|
||||
result_payload = _json_mapping(result.payload, "provider report payload")
|
||||
expires_at = (
|
||||
generated_at + timedelta(days=decision.retention_days)
|
||||
if decision.retention_days is not None
|
||||
else None
|
||||
)
|
||||
row = ReportingProviderExecution(
|
||||
tenant_id=_tenant(principal),
|
||||
execution_id=str(uuid.uuid4()),
|
||||
provider_id=provider_id,
|
||||
report_id=report_id,
|
||||
report_revision=descriptor.revision,
|
||||
contract_version=descriptor.contract_version,
|
||||
idempotency_key=clean_idempotency_key,
|
||||
request_sha256=request_hash,
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
parameters=clean_parameters,
|
||||
result_schema=[item.to_dict() for item in descriptor.result_schema],
|
||||
result_payload=result_payload,
|
||||
source_revisions=[dict(item) for item in result.source_revisions],
|
||||
effective_scope=dict(result.effective_scope),
|
||||
privacy_transforms=list(result.applied_privacy_transforms),
|
||||
provenance=dict(result.provenance),
|
||||
governance_provenance=dict(decision.provenance),
|
||||
retention_class=descriptor.retention_class,
|
||||
retention_days=decision.retention_days,
|
||||
expires_at=expires_at,
|
||||
output_hash=_hash(result_payload),
|
||||
generated_at=generated_at,
|
||||
actor_id=_actor(principal),
|
||||
)
|
||||
session.add(row)
|
||||
session.flush()
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="reporting.provider_report.executed",
|
||||
object_type="reporting_provider_execution",
|
||||
object_id=row.execution_id,
|
||||
details={
|
||||
"provider_id": provider_id,
|
||||
"report_id": report_id,
|
||||
"report_revision": descriptor.revision,
|
||||
"purpose": clean_purpose,
|
||||
"audience_scope": clean_audience,
|
||||
"source_revision_count": len(row.source_revisions),
|
||||
"privacy_transforms": row.privacy_transforms,
|
||||
"retention_class": row.retention_class,
|
||||
"retention_days": row.retention_days,
|
||||
"output_hash": row.output_hash,
|
||||
},
|
||||
)
|
||||
return provider_execution_payload(row)
|
||||
|
||||
|
||||
def get_provider_execution(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
execution_id: str,
|
||||
) -> dict[str, object] | None:
|
||||
row = _provider_execution(session, principal, execution_id=execution_id)
|
||||
if row is None or _expired(row):
|
||||
return None
|
||||
provider, _descriptor = _provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=row.provider_id,
|
||||
report_id=row.report_id,
|
||||
)
|
||||
_require_result_access(provider, session, principal, row)
|
||||
return provider_execution_payload(row)
|
||||
|
||||
|
||||
def export_provider_execution(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
execution_id: str,
|
||||
format: str,
|
||||
purpose: str,
|
||||
audience_scope: Mapping[str, object],
|
||||
) -> tuple[bytes, str, str]:
|
||||
row = _provider_execution(session, principal, execution_id=execution_id)
|
||||
if row is None or _expired(row):
|
||||
raise LookupError("Provider report execution not found")
|
||||
provider, descriptor = _provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=row.provider_id,
|
||||
report_id=row.report_id,
|
||||
)
|
||||
_require_result_access(provider, session, principal, row)
|
||||
clean_format = format.strip().lower()
|
||||
if clean_format not in descriptor.export_formats:
|
||||
raise ProviderReportError("This report does not support the export format")
|
||||
clean_purpose = purpose.strip()
|
||||
clean_audience = _json_mapping(audience_scope, "audience scope")
|
||||
decision = _governance_decision(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
descriptor=descriptor,
|
||||
action="export",
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
export_format=clean_format,
|
||||
applied_privacy_transforms=tuple(row.privacy_transforms or ()),
|
||||
)
|
||||
_require_allowed(decision)
|
||||
if clean_format not in decision.export_formats:
|
||||
raise ProviderReportError("Policy does not allow this export format")
|
||||
content, media_type, extension = _serialize_export(
|
||||
clean_format,
|
||||
row.result_payload,
|
||||
row.result_schema,
|
||||
)
|
||||
output_hash = sha256(content).hexdigest()
|
||||
export = ReportingProviderExport(
|
||||
tenant_id=row.tenant_id,
|
||||
export_id=str(uuid.uuid4()),
|
||||
provider_execution_id=row.id,
|
||||
execution_id=row.execution_id,
|
||||
format=clean_format,
|
||||
purpose=clean_purpose,
|
||||
audience_scope=clean_audience,
|
||||
output_hash=output_hash,
|
||||
exported_at=datetime.now(UTC),
|
||||
actor_id=_actor(principal),
|
||||
)
|
||||
session.add(export)
|
||||
session.flush()
|
||||
audit_from_principal(
|
||||
session,
|
||||
principal,
|
||||
action="reporting.provider_report.exported",
|
||||
object_type="reporting_provider_execution",
|
||||
object_id=row.execution_id,
|
||||
details={
|
||||
"export_id": export.export_id,
|
||||
"format": clean_format,
|
||||
"purpose": clean_purpose,
|
||||
"audience_scope": clean_audience,
|
||||
"output_hash": output_hash,
|
||||
},
|
||||
)
|
||||
filename = f"{row.provider_id}-{row.report_id}-{row.execution_id}.{extension}"
|
||||
return content, media_type, filename
|
||||
|
||||
|
||||
def list_provider_exports(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
execution_id: str,
|
||||
) -> tuple[dict[str, object], ...]:
|
||||
row = _provider_execution(session, principal, execution_id=execution_id)
|
||||
if row is None or _expired(row):
|
||||
return ()
|
||||
provider, _descriptor = _provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=row.provider_id,
|
||||
report_id=row.report_id,
|
||||
)
|
||||
_require_result_access(provider, session, principal, row)
|
||||
exports = (
|
||||
session.query(ReportingProviderExport)
|
||||
.filter(
|
||||
ReportingProviderExport.tenant_id == row.tenant_id,
|
||||
ReportingProviderExport.provider_execution_id == row.id,
|
||||
)
|
||||
.order_by(ReportingProviderExport.exported_at.desc())
|
||||
.all()
|
||||
)
|
||||
return tuple(
|
||||
{
|
||||
"export_id": item.export_id,
|
||||
"execution_id": item.execution_id,
|
||||
"format": item.format,
|
||||
"purpose": item.purpose,
|
||||
"audience_scope": dict(item.audience_scope or {}),
|
||||
"output_hash": item.output_hash,
|
||||
"exported_at": item.exported_at.isoformat(),
|
||||
"actor_id": item.actor_id,
|
||||
}
|
||||
for item in exports
|
||||
)
|
||||
|
||||
|
||||
def provider_execution_payload(row: ReportingProviderExecution) -> dict[str, object]:
|
||||
return {
|
||||
"execution_id": row.execution_id,
|
||||
"provider_id": row.provider_id,
|
||||
"report_id": row.report_id,
|
||||
"report_revision": row.report_revision,
|
||||
"contract_version": row.contract_version,
|
||||
"purpose": row.purpose,
|
||||
"audience_scope": dict(row.audience_scope or {}),
|
||||
"parameters": dict(row.parameters or {}),
|
||||
"result_schema": list(row.result_schema or []),
|
||||
"result": dict(row.result_payload or {}),
|
||||
"source_revisions": list(row.source_revisions or []),
|
||||
"effective_scope": dict(row.effective_scope or {}),
|
||||
"privacy_transforms": list(row.privacy_transforms or []),
|
||||
"provenance": dict(row.provenance or {}),
|
||||
"governance_provenance": dict(row.governance_provenance or {}),
|
||||
"retention_class": row.retention_class,
|
||||
"retention_days": row.retention_days,
|
||||
"expires_at": row.expires_at.isoformat() if row.expires_at else None,
|
||||
"output_hash": row.output_hash,
|
||||
"generated_at": row.generated_at.isoformat(),
|
||||
"actor_id": row.actor_id,
|
||||
}
|
||||
|
||||
|
||||
def _provider_report(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
provider_id: str,
|
||||
report_id: str,
|
||||
) -> tuple[ReportProvider, ReportDescriptor]:
|
||||
provider = dict(report_providers(registry)).get(provider_id)
|
||||
if provider is None:
|
||||
raise LookupError("Report provider is unavailable")
|
||||
descriptor = next(
|
||||
(
|
||||
item
|
||||
for item in provider.list_reports(session, principal)
|
||||
if item.report_id == report_id
|
||||
),
|
||||
None,
|
||||
)
|
||||
if descriptor is None:
|
||||
raise LookupError("Provider report is unavailable")
|
||||
_validate_descriptor(provider_id, descriptor)
|
||||
return provider, descriptor
|
||||
|
||||
|
||||
def _validate_descriptor(provider_id: str, descriptor: ReportDescriptor) -> None:
|
||||
if descriptor.provider_id != provider_id:
|
||||
raise ProviderReportError("Report descriptor provider id differs")
|
||||
if not descriptor.report_id or not descriptor.revision:
|
||||
raise ProviderReportError("Report descriptors require stable ids and revisions")
|
||||
parameter_keys = [item.key for item in descriptor.parameters]
|
||||
field_paths = [item.path for item in descriptor.result_schema]
|
||||
transform_ids = [item.id for item in descriptor.privacy_transforms]
|
||||
if len(parameter_keys) != len(set(parameter_keys)):
|
||||
raise ProviderReportError("Report parameter keys must be unique")
|
||||
if not field_paths or len(field_paths) != len(set(field_paths)):
|
||||
raise ProviderReportError("Report result paths must be non-empty and unique")
|
||||
if len(transform_ids) != len(set(transform_ids)):
|
||||
raise ProviderReportError("Report privacy transforms must be unique")
|
||||
if any(item.sensitive for item in descriptor.result_schema) and (
|
||||
descriptor.reidentification_risk != "high"
|
||||
):
|
||||
raise ProviderReportError(
|
||||
"Sensitive result fields require high risk classification"
|
||||
)
|
||||
|
||||
|
||||
def _validate_result(
|
||||
descriptor: ReportDescriptor,
|
||||
result: object,
|
||||
*,
|
||||
tenant_id: str,
|
||||
) -> None:
|
||||
if not hasattr(result, "report_id") or result.report_id != descriptor.report_id:
|
||||
raise ProviderReportError("Provider result report id differs")
|
||||
if not result.source_revisions:
|
||||
raise ProviderReportError("Provider result has no source revision provenance")
|
||||
if str(result.effective_scope.get("tenant_id") or "") != tenant_id:
|
||||
raise ProviderReportError(
|
||||
"Provider result effective scope differs from the tenant"
|
||||
)
|
||||
required = {item.id for item in descriptor.privacy_transforms if item.required}
|
||||
missing = required - set(result.applied_privacy_transforms)
|
||||
if missing:
|
||||
raise ProviderReportError(
|
||||
"Provider result omitted required privacy transformations: "
|
||||
+ ", ".join(sorted(missing))
|
||||
)
|
||||
declared = {item.path: item for item in descriptor.result_schema}
|
||||
for field in descriptor.result_schema:
|
||||
if field.type != "suppressed_count":
|
||||
continue
|
||||
value = _path_value(result.payload, field.path)
|
||||
if not isinstance(value, Mapping) or set(value) != {"value", "suppressed"}:
|
||||
raise ProviderReportError(
|
||||
f"Provider result has an invalid suppressed count: {field.path}"
|
||||
)
|
||||
if value["value"] is not None and (
|
||||
not isinstance(value["value"], int) or isinstance(value["value"], bool)
|
||||
):
|
||||
raise ProviderReportError(
|
||||
f"Provider result has an invalid suppressed-count value: {field.path}"
|
||||
)
|
||||
if not isinstance(value["suppressed"], bool):
|
||||
raise ProviderReportError(
|
||||
f"Provider result has an invalid suppression flag: {field.path}"
|
||||
)
|
||||
for path in _leaf_paths(result.payload):
|
||||
if path in declared:
|
||||
continue
|
||||
if any(
|
||||
path.startswith(declared_path + ".")
|
||||
and field.type in {"object", "suppressed_count"}
|
||||
for declared_path, field in declared.items()
|
||||
):
|
||||
continue
|
||||
raise ProviderReportError(f"Provider result contains undeclared field: {path}")
|
||||
|
||||
|
||||
def _validated_parameters(
|
||||
descriptor: ReportDescriptor,
|
||||
parameters: Mapping[str, object],
|
||||
) -> dict[str, object]:
|
||||
clean = _json_mapping(parameters, "report parameters")
|
||||
declared = {item.key: item for item in descriptor.parameters}
|
||||
unknown = set(clean) - set(declared)
|
||||
if unknown:
|
||||
raise ProviderReportError(
|
||||
"Unknown report parameters: " + ", ".join(sorted(unknown))
|
||||
)
|
||||
missing = [
|
||||
item.key
|
||||
for item in descriptor.parameters
|
||||
if item.required and clean.get(item.key) in (None, "")
|
||||
]
|
||||
if missing:
|
||||
raise ProviderReportError(
|
||||
"Missing report parameters: " + ", ".join(sorted(missing))
|
||||
)
|
||||
invalid = [
|
||||
item.key
|
||||
for item in descriptor.parameters
|
||||
if item.key in clean
|
||||
and clean[item.key] is not None
|
||||
and not _valid_parameter_value(item.type, clean[item.key])
|
||||
]
|
||||
if invalid:
|
||||
raise ProviderReportError(
|
||||
"Invalid report parameter types: " + ", ".join(sorted(invalid))
|
||||
)
|
||||
return clean
|
||||
|
||||
|
||||
def _valid_parameter_value(parameter_type: str, value: object) -> bool:
|
||||
if parameter_type in {"string", "reference"}:
|
||||
return isinstance(value, str)
|
||||
if parameter_type == "boolean":
|
||||
return isinstance(value, bool)
|
||||
if parameter_type == "integer":
|
||||
return isinstance(value, int) and not isinstance(value, bool)
|
||||
if parameter_type == "number":
|
||||
return isinstance(value, (int, float)) and not isinstance(value, bool)
|
||||
if parameter_type == "date":
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
try:
|
||||
date.fromisoformat(value)
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
if parameter_type == "datetime":
|
||||
if not isinstance(value, str):
|
||||
return False
|
||||
try:
|
||||
datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _governance_decision(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
registry: object | None,
|
||||
descriptor: ReportDescriptor,
|
||||
action: str,
|
||||
purpose: str | None,
|
||||
audience_scope: Mapping[str, object],
|
||||
export_format: str | None = None,
|
||||
applied_privacy_transforms: tuple[str, ...] = (),
|
||||
) -> ReportingGovernanceDecision:
|
||||
required = tuple(item.id for item in descriptor.privacy_transforms if item.required)
|
||||
provider = reporting_governance_provider(registry)
|
||||
if provider is None:
|
||||
allowed = descriptor.reidentification_risk != "high"
|
||||
if descriptor.purpose_required and action != "catalogue" and not purpose:
|
||||
allowed = False
|
||||
if (
|
||||
descriptor.audience_scope_required
|
||||
and action != "catalogue"
|
||||
and not audience_scope
|
||||
):
|
||||
allowed = False
|
||||
if action == "export" and export_format not in descriptor.export_formats:
|
||||
allowed = False
|
||||
return ReportingGovernanceDecision(
|
||||
allowed=allowed,
|
||||
reason=None
|
||||
if allowed
|
||||
else "The built-in Reporting privacy baseline denied this action.",
|
||||
retention_days=30,
|
||||
export_formats=descriptor.export_formats,
|
||||
required_privacy_transforms=required,
|
||||
provenance={"provider": "reporting.built_in_baseline", "version": "1"},
|
||||
)
|
||||
decision = provider.decide_reporting_action(
|
||||
session,
|
||||
principal,
|
||||
request=ReportingGovernanceRequest(
|
||||
action=action, # type: ignore[arg-type]
|
||||
tenant_id=_tenant(principal),
|
||||
provider_id=descriptor.provider_id,
|
||||
report_id=descriptor.report_id,
|
||||
purpose=purpose,
|
||||
audience_scope=audience_scope,
|
||||
retention_class=descriptor.retention_class,
|
||||
export_format=export_format,
|
||||
reidentification_risk=descriptor.reidentification_risk,
|
||||
declared_privacy_transforms=required,
|
||||
applied_privacy_transforms=applied_privacy_transforms,
|
||||
),
|
||||
)
|
||||
unsupported = set(decision.required_privacy_transforms) - {
|
||||
item.id for item in descriptor.privacy_transforms
|
||||
}
|
||||
if not unsupported:
|
||||
return decision
|
||||
return ReportingGovernanceDecision(
|
||||
allowed=False,
|
||||
reason=(
|
||||
"Policy requires privacy transformations this report does not support: "
|
||||
+ ", ".join(sorted(unsupported))
|
||||
),
|
||||
retention_days=decision.retention_days,
|
||||
export_formats=decision.export_formats,
|
||||
required_privacy_transforms=decision.required_privacy_transforms,
|
||||
provenance={
|
||||
**dict(decision.provenance),
|
||||
"unsupported_required_privacy_transforms": sorted(unsupported),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _serialize_export(
|
||||
format: str,
|
||||
payload: Mapping[str, object],
|
||||
schema: list[dict[str, object]],
|
||||
) -> tuple[bytes, str, str]:
|
||||
if format == "json":
|
||||
return (
|
||||
json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True).encode(
|
||||
"utf-8"
|
||||
),
|
||||
"application/json",
|
||||
"json",
|
||||
)
|
||||
if format != "csv":
|
||||
raise ProviderReportError("Unsupported provider report export format")
|
||||
output = io.StringIO(newline="")
|
||||
writer = csv.writer(output)
|
||||
writer.writerow([str(item["label"]) for item in schema])
|
||||
writer.writerow(
|
||||
[_csv_value(_path_value(payload, str(item["path"]))) for item in schema]
|
||||
)
|
||||
return output.getvalue().encode("utf-8-sig"), "text/csv; charset=utf-8", "csv"
|
||||
|
||||
|
||||
def _csv_value(value: object) -> object:
|
||||
if isinstance(value, Mapping):
|
||||
if value.get("suppressed") is True:
|
||||
return "suppressed"
|
||||
if set(value).issuperset({"value", "suppressed"}):
|
||||
value = value.get("value")
|
||||
else:
|
||||
value = json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
if isinstance(value, (list, tuple)):
|
||||
value = json.dumps(value, ensure_ascii=False, sort_keys=True)
|
||||
if isinstance(value, str) and value.startswith(("=", "+", "-", "@", "\t", "\r")):
|
||||
return "'" + value
|
||||
return "" if value is None else value
|
||||
|
||||
|
||||
def _path_value(payload: Mapping[str, object], path: str) -> object:
|
||||
value: object = payload
|
||||
for part in path.split("."):
|
||||
if not isinstance(value, Mapping):
|
||||
return None
|
||||
value = value.get(part)
|
||||
return value
|
||||
|
||||
|
||||
def _leaf_paths(value: object, prefix: str = "") -> tuple[str, ...]:
|
||||
if isinstance(value, Mapping):
|
||||
rows: list[str] = []
|
||||
for key, item in value.items():
|
||||
path = f"{prefix}.{key}" if prefix else str(key)
|
||||
rows.extend(_leaf_paths(item, path))
|
||||
return tuple(rows)
|
||||
if isinstance(value, (list, tuple)):
|
||||
return (prefix,)
|
||||
return (prefix,)
|
||||
|
||||
|
||||
def _provider_execution(
|
||||
session: Session,
|
||||
principal: object,
|
||||
*,
|
||||
execution_id: str,
|
||||
) -> ReportingProviderExecution | None:
|
||||
return (
|
||||
session.query(ReportingProviderExecution)
|
||||
.filter(
|
||||
ReportingProviderExecution.tenant_id == _tenant(principal),
|
||||
ReportingProviderExecution.execution_id == execution_id,
|
||||
)
|
||||
.one_or_none()
|
||||
)
|
||||
|
||||
|
||||
def _require_result_access(
|
||||
provider: ReportProvider,
|
||||
session: Session,
|
||||
principal: object,
|
||||
row: ReportingProviderExecution,
|
||||
) -> None:
|
||||
if not provider.authorize_result(
|
||||
session,
|
||||
principal,
|
||||
report_id=row.report_id,
|
||||
source_revisions=tuple(row.source_revisions or ()),
|
||||
effective_scope=dict(row.effective_scope or {}),
|
||||
):
|
||||
raise PermissionError(
|
||||
"The source module no longer permits access to this report result"
|
||||
)
|
||||
|
||||
|
||||
def _expired(row: ReportingProviderExecution) -> bool:
|
||||
if row.expires_at is None:
|
||||
return False
|
||||
return _aware(row.expires_at) <= datetime.now(UTC)
|
||||
|
||||
|
||||
def _require_allowed(decision: ReportingGovernanceDecision) -> None:
|
||||
if not decision.allowed:
|
||||
raise PermissionError(decision.reason or "Reporting Policy denied this action")
|
||||
|
||||
|
||||
def _decision_payload(decision: ReportingGovernanceDecision) -> dict[str, object]:
|
||||
return {
|
||||
"allowed": decision.allowed,
|
||||
"reason": decision.reason,
|
||||
"retention_days": decision.retention_days,
|
||||
"export_formats": list(decision.export_formats),
|
||||
"required_privacy_transforms": list(decision.required_privacy_transforms),
|
||||
"provenance": dict(decision.provenance),
|
||||
}
|
||||
|
||||
|
||||
def _json_mapping(value: Mapping[str, object], label: str) -> dict[str, object]:
|
||||
try:
|
||||
payload = json.loads(json.dumps(value, sort_keys=True, separators=(",", ":")))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise ProviderReportError(f"The {label} must contain JSON values") from exc
|
||||
if not isinstance(payload, dict):
|
||||
raise ProviderReportError(f"The {label} must be an object")
|
||||
return payload
|
||||
|
||||
|
||||
def _hash(value: object) -> str:
|
||||
return sha256(
|
||||
json.dumps(
|
||||
value,
|
||||
ensure_ascii=False,
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
).hexdigest()
|
||||
|
||||
|
||||
def _tenant(principal: object) -> str:
|
||||
tenant_id = str(getattr(principal, "tenant_id", "") or "").strip()
|
||||
if not tenant_id:
|
||||
raise PermissionError("A tenant principal is required")
|
||||
return tenant_id
|
||||
|
||||
|
||||
def _actor(principal: object) -> str | None:
|
||||
user = getattr(principal, "user", None)
|
||||
actor_id = getattr(user, "id", None)
|
||||
return str(actor_id) if actor_id else None
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"ProviderReportError",
|
||||
"execute_provider_report",
|
||||
"export_provider_execution",
|
||||
"get_provider_execution",
|
||||
"list_provider_exports",
|
||||
"list_provider_reports",
|
||||
"provider_parameter_options",
|
||||
]
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Retention minimization for governed provider-report results."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime
|
||||
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_reporting.backend.db.models import ReportingProviderExecution
|
||||
|
||||
|
||||
class ReportingRetentionService:
|
||||
def apply_retention(
|
||||
self,
|
||||
session: object,
|
||||
*,
|
||||
dry_run: bool,
|
||||
now: datetime,
|
||||
limit: int = 500,
|
||||
) -> dict[str, int]:
|
||||
if not isinstance(session, Session):
|
||||
raise TypeError("Reporting retention requires a SQLAlchemy Session")
|
||||
observed_at = _aware(now)
|
||||
rows = (
|
||||
session.query(ReportingProviderExecution)
|
||||
.filter(
|
||||
ReportingProviderExecution.expires_at.is_not(None),
|
||||
ReportingProviderExecution.expires_at <= observed_at,
|
||||
ReportingProviderExecution.retention_redacted_at.is_(None),
|
||||
)
|
||||
.order_by(
|
||||
ReportingProviderExecution.expires_at.asc(),
|
||||
ReportingProviderExecution.id.asc(),
|
||||
)
|
||||
.limit(max(1, min(int(limit), 5_000)))
|
||||
.all()
|
||||
)
|
||||
counts = {
|
||||
"eligible": len(rows),
|
||||
"redacted": 0,
|
||||
"remaining_in_batch": 0,
|
||||
}
|
||||
if dry_run:
|
||||
return counts
|
||||
for row in rows:
|
||||
# Keep immutable request/output hashes and provenance as audit
|
||||
# evidence while removing the retained report detail itself.
|
||||
row.result_payload = {}
|
||||
row.retention_redacted_at = observed_at
|
||||
counts["redacted"] += 1
|
||||
session.flush()
|
||||
counts["remaining_in_batch"] = (
|
||||
session.query(ReportingProviderExecution.id)
|
||||
.filter(
|
||||
ReportingProviderExecution.expires_at.is_not(None),
|
||||
ReportingProviderExecution.expires_at <= observed_at,
|
||||
ReportingProviderExecution.retention_redacted_at.is_(None),
|
||||
)
|
||||
.limit(1)
|
||||
.count()
|
||||
)
|
||||
return counts
|
||||
|
||||
|
||||
def _aware(value: datetime) -> datetime:
|
||||
return value if value.tzinfo is not None else value.replace(tzinfo=UTC)
|
||||
|
||||
|
||||
__all__ = ["ReportingRetentionService"]
|
||||
@@ -44,12 +44,23 @@ from govoplan_reporting.backend.operations import (
|
||||
upsert_saved_view,
|
||||
upsert_schedule,
|
||||
)
|
||||
from govoplan_reporting.backend.provider_reports import (
|
||||
ProviderReportError,
|
||||
execute_provider_report,
|
||||
export_provider_execution,
|
||||
get_provider_execution,
|
||||
list_provider_exports,
|
||||
list_provider_reports,
|
||||
provider_parameter_options,
|
||||
)
|
||||
from govoplan_reporting.backend.query_engine import ReportingQueryError
|
||||
from govoplan_reporting.backend.schemas import (
|
||||
DefinitionUpdateRequest,
|
||||
DefinitionWriteRequest,
|
||||
ImportAssessmentRequest,
|
||||
PublicationRequest,
|
||||
ProviderReportExecutionRequest,
|
||||
ProviderReportExportRequest,
|
||||
QualityRunRequest,
|
||||
ReportExecutionRequest,
|
||||
SavedViewWriteRequest,
|
||||
@@ -60,6 +71,138 @@ from govoplan_reporting.backend.schemas import (
|
||||
def create_router(registry: object | None) -> APIRouter:
|
||||
router = APIRouter(prefix="/reporting", tags=["reporting"])
|
||||
|
||||
@router.get("/provider-reports")
|
||||
def api_list_provider_reports(
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> dict[str, object]:
|
||||
_require(principal, READ_SCOPE)
|
||||
try:
|
||||
return list_provider_reports(session, principal, registry=registry)
|
||||
except (ProviderReportError, TypeError, ValueError) as exc:
|
||||
raise _error(exc) from exc
|
||||
|
||||
@router.get(
|
||||
"/provider-reports/{provider_id}/{report_id}/parameters/{parameter_key}/options"
|
||||
)
|
||||
def api_provider_parameter_options(
|
||||
provider_id: str,
|
||||
report_id: str,
|
||||
parameter_key: str,
|
||||
query: str = "",
|
||||
limit: int = Query(default=100, ge=1, le=200),
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> dict[str, object]:
|
||||
_require(principal, READ_SCOPE)
|
||||
try:
|
||||
options = provider_parameter_options(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=provider_id,
|
||||
report_id=report_id,
|
||||
parameter_key=parameter_key,
|
||||
query=query,
|
||||
limit=limit,
|
||||
)
|
||||
except (ProviderReportError, PermissionError, LookupError) as exc:
|
||||
raise _error(exc) from exc
|
||||
return {"options": [item.to_dict() for item in options]}
|
||||
|
||||
@router.post("/provider-reports/{provider_id}/{report_id}/executions")
|
||||
def api_execute_provider_report(
|
||||
provider_id: str,
|
||||
report_id: str,
|
||||
payload: ProviderReportExecutionRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> dict[str, object]:
|
||||
_require(principal, RUN_SCOPE)
|
||||
try:
|
||||
result = execute_provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id=provider_id,
|
||||
report_id=report_id,
|
||||
**payload.model_dump(),
|
||||
)
|
||||
session.commit()
|
||||
except (ProviderReportError, PermissionError, LookupError, ValueError) as exc:
|
||||
session.rollback()
|
||||
raise _error(exc) from exc
|
||||
return result
|
||||
|
||||
@router.get("/provider-executions/{execution_id}")
|
||||
def api_get_provider_execution(
|
||||
execution_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> dict[str, object]:
|
||||
_require(principal, RUN_SCOPE)
|
||||
try:
|
||||
result = get_provider_execution(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
execution_id=execution_id,
|
||||
)
|
||||
except (ProviderReportError, PermissionError, LookupError) as exc:
|
||||
raise _error(exc) from exc
|
||||
if result is None:
|
||||
raise HTTPException(
|
||||
status_code=404, detail="Provider report execution not found"
|
||||
)
|
||||
return result
|
||||
|
||||
@router.get("/provider-executions/{execution_id}/exports")
|
||||
def api_list_provider_exports(
|
||||
execution_id: str,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> dict[str, object]:
|
||||
_require(principal, RUN_SCOPE)
|
||||
try:
|
||||
return {
|
||||
"exports": list(
|
||||
list_provider_exports(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
execution_id=execution_id,
|
||||
)
|
||||
)
|
||||
}
|
||||
except (ProviderReportError, PermissionError, LookupError) as exc:
|
||||
raise _error(exc) from exc
|
||||
|
||||
@router.post("/provider-executions/{execution_id}/exports")
|
||||
def api_export_provider_execution(
|
||||
execution_id: str,
|
||||
payload: ProviderReportExportRequest,
|
||||
session: Session = Depends(get_session),
|
||||
principal: ApiPrincipal = Depends(get_api_principal),
|
||||
) -> Response:
|
||||
_require(principal, RUN_SCOPE)
|
||||
try:
|
||||
content, media_type, filename = export_provider_execution(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
execution_id=execution_id,
|
||||
**payload.model_dump(),
|
||||
)
|
||||
session.commit()
|
||||
except (ProviderReportError, PermissionError, LookupError) as exc:
|
||||
session.rollback()
|
||||
raise _error(exc) from exc
|
||||
return Response(
|
||||
content=content,
|
||||
media_type=media_type,
|
||||
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
|
||||
)
|
||||
|
||||
@router.get("/definitions")
|
||||
def api_list_definitions(
|
||||
definition_kind: list[str] | None = Query(default=None),
|
||||
|
||||
@@ -422,6 +422,23 @@ class ReportExecutionRequest(BaseModel):
|
||||
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||
|
||||
|
||||
class ProviderReportExecutionRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
parameters: dict[str, Any] = Field(default_factory=dict)
|
||||
purpose: str = Field(min_length=1, max_length=1_000)
|
||||
audience_scope: dict[str, Any] = Field(min_length=1, max_length=50)
|
||||
idempotency_key: str = Field(min_length=1, max_length=255)
|
||||
|
||||
|
||||
class ProviderReportExportRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
format: Literal["json", "csv"] = "json"
|
||||
purpose: str = Field(min_length=1, max_length=1_000)
|
||||
audience_scope: dict[str, Any] = Field(min_length=1, max_length=50)
|
||||
|
||||
|
||||
class SavedViewWriteRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
@@ -489,6 +506,8 @@ __all__ = [
|
||||
"ImportAssessmentRequest",
|
||||
"MeasureDefinition",
|
||||
"PublicationRequest",
|
||||
"ProviderReportExecutionRequest",
|
||||
"ProviderReportExportRequest",
|
||||
"QualityPlanDefinition",
|
||||
"QualityRunRequest",
|
||||
"ReportDefinition",
|
||||
|
||||
@@ -14,9 +14,15 @@ class ReportingManifestTests(unittest.TestCase):
|
||||
self.assertEqual("@govoplan/reporting-webui", manifest.frontend.package_name)
|
||||
self.assertIsNotNone(manifest.route_factory)
|
||||
self.assertIsNotNone(manifest.migration_spec)
|
||||
self.assertEqual(4, len(manifest.provides_interfaces))
|
||||
self.assertEqual(5, len(manifest.provides_interfaces))
|
||||
self.assertEqual(1, len(manifest.search_sources))
|
||||
self.assertIn("dataflow", manifest.optional_dependencies)
|
||||
self.assertIn("policy", manifest.optional_dependencies)
|
||||
self.assertEqual("/reports", manifest.nav_items[0].path)
|
||||
self.assertEqual(
|
||||
{"/reports", "/reporting"},
|
||||
{route.path for route in manifest.frontend.routes},
|
||||
)
|
||||
self.assertIn(
|
||||
"report execution and publication evidence",
|
||||
manifest.architecture.owned_concepts,
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
|
||||
from alembic.runtime.migration import MigrationContext
|
||||
from sqlalchemy import create_engine, inspect
|
||||
|
||||
from govoplan_core.db.migrations import migrate_database
|
||||
from govoplan_reporting.backend.manifest import get_manifest
|
||||
|
||||
|
||||
def test_fresh_migration_creates_provider_evidence_tables_and_current_head() -> None:
|
||||
with tempfile.TemporaryDirectory(
|
||||
prefix="govoplan-reporting-migration-"
|
||||
) as directory:
|
||||
url = f"sqlite:///{Path(directory) / 'reporting.db'}"
|
||||
migrate_database(
|
||||
database_url=url,
|
||||
enabled_modules=("reporting",),
|
||||
manifest_factories=(get_manifest,),
|
||||
)
|
||||
engine = create_engine(url)
|
||||
try:
|
||||
with engine.connect() as connection:
|
||||
tables = set(inspect(connection).get_table_names())
|
||||
assert {
|
||||
"reporting_provider_executions",
|
||||
"reporting_provider_exports",
|
||||
}.issubset(tables)
|
||||
assert "b7c4e1a9d2f6" in set(
|
||||
MigrationContext.configure(connection).get_current_heads()
|
||||
)
|
||||
columns = {
|
||||
item["name"]
|
||||
for item in inspect(connection).get_columns(
|
||||
"reporting_provider_executions"
|
||||
)
|
||||
}
|
||||
assert "retention_redacted_at" in columns
|
||||
finally:
|
||||
engine.dispose()
|
||||
@@ -0,0 +1,91 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
_ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def test_campaign_starts_and_contributes_its_provider_without_reporting() -> None:
|
||||
script = """
|
||||
import importlib.abc
|
||||
import sys
|
||||
|
||||
class Blocker(importlib.abc.MetaPathFinder):
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname == 'govoplan_reporting' or fullname.startswith('govoplan_reporting.'):
|
||||
raise ModuleNotFoundError("Reporting is physically absent", name=fullname)
|
||||
return None
|
||||
|
||||
sys.meta_path.insert(0, Blocker())
|
||||
from govoplan_campaign.backend.manifest import get_manifest
|
||||
manifest = get_manifest()
|
||||
assert 'reporting' not in manifest.dependencies
|
||||
assert 'reporting' in manifest.optional_dependencies
|
||||
assert 'reporting.report_provider.campaigns' in manifest.capability_factories
|
||||
"""
|
||||
_run_probe(script)
|
||||
|
||||
|
||||
def test_reporting_starts_without_campaign_and_owns_the_global_route() -> None:
|
||||
script = """
|
||||
import importlib.abc
|
||||
import sys
|
||||
|
||||
class Blocker(importlib.abc.MetaPathFinder):
|
||||
def find_spec(self, fullname, path=None, target=None):
|
||||
if fullname == 'govoplan_campaign' or fullname.startswith('govoplan_campaign.'):
|
||||
raise ModuleNotFoundError("Campaign is physically absent", name=fullname)
|
||||
return None
|
||||
|
||||
sys.meta_path.insert(0, Blocker())
|
||||
from govoplan_reporting.backend.manifest import get_manifest
|
||||
manifest = get_manifest()
|
||||
assert 'campaigns' not in manifest.dependencies
|
||||
assert manifest.nav_items[0].path == '/reports'
|
||||
assert {item.path for item in manifest.frontend.routes} == {'/reports', '/reporting'}
|
||||
"""
|
||||
_run_probe(script)
|
||||
|
||||
|
||||
def test_reporting_with_campaign_uses_only_the_provider_contract() -> None:
|
||||
script = """
|
||||
from govoplan_campaign.backend.manifest import get_manifest as campaign_manifest
|
||||
from govoplan_core.core.reporting import REPORT_PROVIDER_CAPABILITY_PREFIX
|
||||
from govoplan_reporting.backend.manifest import get_manifest as reporting_manifest
|
||||
|
||||
reporting = reporting_manifest()
|
||||
campaign = campaign_manifest()
|
||||
provider_name = REPORT_PROVIDER_CAPABILITY_PREFIX + 'campaigns'
|
||||
assert provider_name in campaign.capability_factories
|
||||
assert provider_name in {item.name for item in campaign.provides_interfaces}
|
||||
assert 'campaigns' not in reporting.dependencies
|
||||
assert '/reports' not in {route.path for route in campaign.frontend.routes}
|
||||
provider = campaign.capability_factories[provider_name](None)
|
||||
assert provider.provider_id == 'campaigns'
|
||||
assert provider.contract_version == '1.0'
|
||||
"""
|
||||
_run_probe(script)
|
||||
|
||||
|
||||
def _run_probe(source: str) -> None:
|
||||
environment = dict(os.environ)
|
||||
environment["PYTHONPATH"] = os.pathsep.join(
|
||||
(
|
||||
str(_ROOT / "src"),
|
||||
str(_ROOT.parent / "govoplan-campaign" / "src"),
|
||||
str(_ROOT.parent / "govoplan-core" / "src"),
|
||||
environment.get("PYTHONPATH", ""),
|
||||
)
|
||||
)
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-c", source],
|
||||
cwd=_ROOT,
|
||||
env=environment,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr or result.stdout
|
||||
@@ -0,0 +1,336 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from types import SimpleNamespace
|
||||
|
||||
from sqlalchemy import create_engine
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from govoplan_core.core.reporting import (
|
||||
REPORT_PROVIDER_CAPABILITY_PREFIX,
|
||||
REPORT_PROVIDER_CONTRACT_VERSION,
|
||||
ReportDescriptor,
|
||||
ReportParameterDescriptor,
|
||||
ReportParameterOption,
|
||||
ReportPrivacyTransform,
|
||||
ReportProviderRequest,
|
||||
ReportProviderResult,
|
||||
ReportResultField,
|
||||
)
|
||||
from govoplan_core.db.base import Base
|
||||
from govoplan_reporting.backend.db.models import (
|
||||
ReportingProviderExecution,
|
||||
ReportingProviderExport,
|
||||
)
|
||||
from govoplan_reporting.backend.provider_reports import (
|
||||
ProviderReportError,
|
||||
execute_provider_report,
|
||||
export_provider_execution,
|
||||
list_provider_reports,
|
||||
)
|
||||
from govoplan_reporting.backend.retention import ReportingRetentionService
|
||||
|
||||
|
||||
class _Principal:
|
||||
tenant_id = "tenant-1"
|
||||
user = SimpleNamespace(id="analyst-1")
|
||||
api_key = None
|
||||
|
||||
|
||||
class _Provider:
|
||||
provider_id = "example"
|
||||
contract_version = REPORT_PROVIDER_CONTRACT_VERSION
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.result_access = True
|
||||
|
||||
def list_reports(self, session, principal):
|
||||
del session, principal
|
||||
return (
|
||||
ReportDescriptor(
|
||||
provider_id=self.provider_id,
|
||||
report_id="summary",
|
||||
revision="example.summary.v1",
|
||||
title="Example summary",
|
||||
summary="A minimized example report.",
|
||||
parameters=(
|
||||
ReportParameterDescriptor(
|
||||
key="source_id",
|
||||
label="Source",
|
||||
type="reference",
|
||||
required=True,
|
||||
options_from_provider=True,
|
||||
),
|
||||
),
|
||||
result_schema=(
|
||||
ReportResultField(
|
||||
path="metric",
|
||||
label="Metric",
|
||||
type="integer",
|
||||
group="Outcome",
|
||||
),
|
||||
),
|
||||
privacy_transforms=(
|
||||
ReportPrivacyTransform(id="aggregate", label="Aggregate"),
|
||||
),
|
||||
export_formats=("json",),
|
||||
reidentification_risk="low",
|
||||
),
|
||||
)
|
||||
|
||||
def parameter_options(self, session, principal, **kwargs):
|
||||
del session, principal, kwargs
|
||||
return (ReportParameterOption(value="source-1", label="Source 1"),)
|
||||
|
||||
def execute_report(
|
||||
self,
|
||||
session,
|
||||
principal,
|
||||
*,
|
||||
request: ReportProviderRequest,
|
||||
):
|
||||
del session
|
||||
return ReportProviderResult(
|
||||
report_id=request.report_id,
|
||||
generated_at=datetime(2026, 8, 2, 10, 0, tzinfo=UTC),
|
||||
payload={"metric": 12},
|
||||
source_revisions=(
|
||||
{
|
||||
"module_id": "example",
|
||||
"resource_id": request.parameters["source_id"],
|
||||
"revision_id": "7",
|
||||
},
|
||||
),
|
||||
effective_scope={
|
||||
"tenant_id": principal.tenant_id,
|
||||
"source_id": request.parameters["source_id"],
|
||||
},
|
||||
applied_privacy_transforms=("aggregate",),
|
||||
provenance={"executor": "example-v1"},
|
||||
)
|
||||
|
||||
def authorize_result(self, session, principal, **kwargs):
|
||||
del session, principal, kwargs
|
||||
return self.result_access
|
||||
|
||||
|
||||
class _Registry:
|
||||
def __init__(self, provider=None):
|
||||
self.provider = provider or _Provider()
|
||||
|
||||
def capability_names(self):
|
||||
return (REPORT_PROVIDER_CAPABILITY_PREFIX + "example",)
|
||||
|
||||
def has_capability(self, name):
|
||||
return name == REPORT_PROVIDER_CAPABILITY_PREFIX + "example"
|
||||
|
||||
def capability(self, name):
|
||||
return self.provider if self.has_capability(name) else None
|
||||
|
||||
|
||||
def test_provider_report_execution_is_governed_replayable_and_audited() -> None:
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
principal = _Principal()
|
||||
registry = _Registry()
|
||||
with Session(engine, expire_on_commit=False) as session:
|
||||
catalogue = list_provider_reports(session, principal, registry=registry)
|
||||
assert catalogue["reports"][0]["available"] is True
|
||||
|
||||
execution = execute_provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="provider-run-1",
|
||||
)
|
||||
replay = execute_provider_report(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="provider-run-1",
|
||||
)
|
||||
assert replay["execution_id"] == execution["execution_id"]
|
||||
assert execution["source_revisions"][0]["revision_id"] == "7"
|
||||
assert execution["privacy_transforms"] == ["aggregate"]
|
||||
assert execution["retention_days"] == 30
|
||||
|
||||
content, media_type, filename = export_provider_execution(
|
||||
session,
|
||||
principal,
|
||||
registry=registry,
|
||||
execution_id=str(execution["execution_id"]),
|
||||
format="json",
|
||||
purpose="Archive approved aggregate",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
)
|
||||
assert b'"metric": 12' in content
|
||||
assert media_type == "application/json"
|
||||
assert filename.endswith(".json")
|
||||
assert session.query(ReportingProviderExport).count() == 1
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def test_provider_result_missing_a_required_transform_is_rejected() -> None:
|
||||
class _UnsafeProvider(_Provider):
|
||||
def execute_report(self, session, principal, *, request):
|
||||
result = super().execute_report(
|
||||
session,
|
||||
principal,
|
||||
request=request,
|
||||
)
|
||||
return ReportProviderResult(
|
||||
report_id=result.report_id,
|
||||
generated_at=result.generated_at,
|
||||
payload=result.payload,
|
||||
source_revisions=result.source_revisions,
|
||||
effective_scope=result.effective_scope,
|
||||
applied_privacy_transforms=(),
|
||||
provenance=result.provenance,
|
||||
)
|
||||
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
with Session(engine) as session:
|
||||
try:
|
||||
execute_provider_report(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=_Registry(_UnsafeProvider()),
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant"},
|
||||
idempotency_key="unsafe-provider-run",
|
||||
)
|
||||
except ProviderReportError as exc:
|
||||
assert "privacy transformations" in str(exc)
|
||||
else:
|
||||
raise AssertionError("unsafe provider report was accepted")
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def test_expired_provider_result_is_minimized_but_evidence_is_retained() -> None:
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
with Session(engine, expire_on_commit=False) as session:
|
||||
execution = execute_provider_report(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=_Registry(),
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="expired-provider-run",
|
||||
)
|
||||
row = session.query(ReportingProviderExecution).one()
|
||||
original_hash = row.output_hash
|
||||
row.expires_at = datetime.now(UTC) - timedelta(seconds=1)
|
||||
session.flush()
|
||||
|
||||
preview = ReportingRetentionService().apply_retention(
|
||||
session,
|
||||
dry_run=True,
|
||||
now=datetime.now(UTC),
|
||||
)
|
||||
assert preview["eligible"] == 1
|
||||
assert row.result_payload == {"metric": 12}
|
||||
|
||||
applied = ReportingRetentionService().apply_retention(
|
||||
session,
|
||||
dry_run=False,
|
||||
now=datetime.now(UTC),
|
||||
)
|
||||
assert applied["redacted"] == 1
|
||||
assert row.result_payload == {}
|
||||
assert row.retention_redacted_at is not None
|
||||
assert row.output_hash == original_hash == execution["output_hash"]
|
||||
assert row.source_revisions[0]["revision_id"] == "7"
|
||||
try:
|
||||
execute_provider_report(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=_Registry(),
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="expired-provider-run",
|
||||
)
|
||||
except ProviderReportError as exc:
|
||||
assert "expired" in str(exc)
|
||||
else:
|
||||
raise AssertionError("expired provider-report detail was replayed")
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def test_provider_parameters_are_checked_against_the_declared_type() -> None:
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
with Session(engine) as session:
|
||||
try:
|
||||
execute_provider_report(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=_Registry(),
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": 123},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="invalid-provider-parameters",
|
||||
)
|
||||
except ProviderReportError as exc:
|
||||
assert "parameter types" in str(exc)
|
||||
else:
|
||||
raise AssertionError("invalid provider-report parameters were accepted")
|
||||
engine.dispose()
|
||||
|
||||
|
||||
def test_source_access_is_rechecked_before_retained_result_export() -> None:
|
||||
engine = create_engine("sqlite+pysqlite:///:memory:")
|
||||
Base.metadata.create_all(engine)
|
||||
provider = _Provider()
|
||||
registry = _Registry(provider)
|
||||
with Session(engine) as session:
|
||||
execution = execute_provider_report(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=registry,
|
||||
provider_id="example",
|
||||
report_id="summary",
|
||||
parameters={"source_id": "source-1"},
|
||||
purpose="Operational overview",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
idempotency_key="source-access-provider-run",
|
||||
)
|
||||
provider.result_access = False
|
||||
|
||||
try:
|
||||
export_provider_execution(
|
||||
session,
|
||||
_Principal(),
|
||||
registry=registry,
|
||||
execution_id=str(execution["execution_id"]),
|
||||
format="json",
|
||||
purpose="Archive aggregate",
|
||||
audience_scope={"scope_type": "tenant", "scope_id": "tenant-1"},
|
||||
)
|
||||
except PermissionError as exc:
|
||||
assert "source module" in str(exc)
|
||||
else:
|
||||
raise AssertionError("revoked source result was exported")
|
||||
engine.dispose()
|
||||
@@ -119,6 +119,74 @@ export type ReportingSavedView = {
|
||||
updated_at: string;
|
||||
};
|
||||
|
||||
export type ProviderReportParameter = {
|
||||
key: string;
|
||||
label: string;
|
||||
type: string;
|
||||
required: boolean;
|
||||
description?: string | null;
|
||||
options_from_provider: boolean;
|
||||
};
|
||||
|
||||
export type ProviderReportField = {
|
||||
path: string;
|
||||
label: string;
|
||||
type: string;
|
||||
group: string;
|
||||
nullable: boolean;
|
||||
sensitive: boolean;
|
||||
};
|
||||
|
||||
export type ProviderReportDescriptor = {
|
||||
contract_version: string;
|
||||
provider_id: string;
|
||||
report_id: string;
|
||||
revision: string;
|
||||
title: string;
|
||||
summary: string;
|
||||
parameters: ProviderReportParameter[];
|
||||
result_schema: ProviderReportField[];
|
||||
privacy_transforms: Array<{ id: string; label: string; required: boolean }>;
|
||||
purpose_required: boolean;
|
||||
audience_scope_required: boolean;
|
||||
retention_class: string;
|
||||
export_formats: Array<"json" | "csv">;
|
||||
reidentification_risk: "low" | "moderate" | "high";
|
||||
presentation: { kind?: string };
|
||||
available: boolean;
|
||||
unavailable_reason?: string | null;
|
||||
governance: {
|
||||
retention_days?: number | null;
|
||||
export_formats: string[];
|
||||
required_privacy_transforms: string[];
|
||||
provenance: Record<string, unknown>;
|
||||
};
|
||||
};
|
||||
|
||||
export type ProviderReportExecution = {
|
||||
execution_id: string;
|
||||
provider_id: string;
|
||||
report_id: string;
|
||||
report_revision: string;
|
||||
contract_version: string;
|
||||
purpose: string;
|
||||
audience_scope: Record<string, unknown>;
|
||||
parameters: Record<string, unknown>;
|
||||
result_schema: ProviderReportField[];
|
||||
result: Record<string, unknown>;
|
||||
source_revisions: Array<Record<string, unknown>>;
|
||||
effective_scope: Record<string, unknown>;
|
||||
privacy_transforms: string[];
|
||||
provenance: Record<string, unknown>;
|
||||
governance_provenance: Record<string, unknown>;
|
||||
retention_class: string;
|
||||
retention_days?: number | null;
|
||||
expires_at?: string | null;
|
||||
output_hash: string;
|
||||
generated_at: string;
|
||||
actor_id?: string | null;
|
||||
};
|
||||
|
||||
export function listDefinitions(
|
||||
settings: ApiSettings,
|
||||
options: { kinds?: ReportingDefinitionKind[]; status?: string[]; query?: string; limit?: number },
|
||||
@@ -132,6 +200,75 @@ export function listDefinitions(
|
||||
}), { signal });
|
||||
}
|
||||
|
||||
export function listProviderReports(
|
||||
settings: ApiSettings,
|
||||
signal?: AbortSignal
|
||||
): Promise<{ reports: ProviderReportDescriptor[]; diagnostics: Array<Record<string, string>> }> {
|
||||
return apiFetch(settings, "/api/v1/reporting/provider-reports", { signal });
|
||||
}
|
||||
|
||||
export function listProviderParameterOptions(
|
||||
settings: ApiSettings,
|
||||
report: ProviderReportDescriptor,
|
||||
parameterKey: string,
|
||||
query = "",
|
||||
signal?: AbortSignal
|
||||
): Promise<{ options: Array<{ value: string; label: string; description?: string | null }> }> {
|
||||
return apiFetch(settings, apiPath(
|
||||
`/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/parameters/${encodeURIComponent(parameterKey)}/options`,
|
||||
{ query, limit: 200 }
|
||||
), { signal });
|
||||
}
|
||||
|
||||
export function runProviderReport(
|
||||
settings: ApiSettings,
|
||||
report: ProviderReportDescriptor,
|
||||
parameters: Record<string, unknown>,
|
||||
purpose: string,
|
||||
audienceScope: Record<string, unknown>
|
||||
): Promise<ProviderReportExecution> {
|
||||
return apiFetch(
|
||||
settings,
|
||||
`/api/v1/reporting/provider-reports/${encodeURIComponent(report.provider_id)}/${encodeURIComponent(report.report_id)}/executions`,
|
||||
{
|
||||
method: "POST",
|
||||
body: JSON.stringify({
|
||||
parameters,
|
||||
purpose,
|
||||
audience_scope: audienceScope,
|
||||
idempotency_key: crypto.randomUUID()
|
||||
})
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
export async function downloadProviderExecution(
|
||||
settings: ApiSettings,
|
||||
execution: ProviderReportExecution,
|
||||
format: "json" | "csv",
|
||||
purpose: string,
|
||||
audienceScope: Record<string, unknown>
|
||||
): Promise<void> {
|
||||
const response = await fetch(
|
||||
apiUrl(settings, `/api/v1/reporting/provider-executions/${encodeURIComponent(execution.execution_id)}/exports`),
|
||||
{
|
||||
method: "POST",
|
||||
headers: { ...authHeaders(settings), "Content-Type": "application/json" },
|
||||
credentials: "include",
|
||||
body: JSON.stringify({ format, purpose, audience_scope: audienceScope })
|
||||
}
|
||||
);
|
||||
if (!response.ok) throw new Error(await response.text());
|
||||
const blob = await response.blob();
|
||||
const disposition = response.headers.get("Content-Disposition") ?? "";
|
||||
const filename = disposition.match(/filename="?([^";]+)"?/)?.[1] ?? `provider-report.${format}`;
|
||||
const link = document.createElement("a");
|
||||
link.href = URL.createObjectURL(blob);
|
||||
link.download = filename;
|
||||
link.click();
|
||||
URL.revokeObjectURL(link.href);
|
||||
}
|
||||
|
||||
export function getDefinition(
|
||||
settings: ApiSettings,
|
||||
kind: ReportingDefinitionKind,
|
||||
|
||||
@@ -0,0 +1,295 @@
|
||||
import { Download, FileJson, Play, ShieldCheck } from "lucide-react";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import {
|
||||
Button,
|
||||
Card,
|
||||
DismissibleAlert,
|
||||
IconButton,
|
||||
MetricCard,
|
||||
StatusBadge,
|
||||
type ApiSettings,
|
||||
type AuthInfo
|
||||
} from "@govoplan/core-webui";
|
||||
import {
|
||||
downloadProviderExecution,
|
||||
listProviderParameterOptions,
|
||||
runProviderReport,
|
||||
type ProviderReportDescriptor,
|
||||
type ProviderReportExecution,
|
||||
type ProviderReportField
|
||||
} from "../../api/reporting";
|
||||
|
||||
|
||||
export function ProviderReportWorkspace({ settings, auth, report }: {
|
||||
settings: ApiSettings;
|
||||
auth: AuthInfo;
|
||||
report: ProviderReportDescriptor;
|
||||
}) {
|
||||
const [parameters, setParameters] = useState<Record<string, unknown>>({});
|
||||
const [options, setOptions] = useState<Record<string, Array<{ value: string; label: string; description?: string | null }>>>({});
|
||||
const [purpose, setPurpose] = useState("");
|
||||
const [execution, setExecution] = useState<ProviderReportExecution | null>(null);
|
||||
const [running, setRunning] = useState(false);
|
||||
const [error, setError] = useState("");
|
||||
const tenant = auth.active_tenant ?? auth.tenant;
|
||||
const audienceScope = useMemo(() => ({
|
||||
scope_type: "tenant",
|
||||
scope_id: tenant.id,
|
||||
label: tenant.name
|
||||
}), [tenant.id, tenant.name]);
|
||||
|
||||
useEffect(() => {
|
||||
setParameters({});
|
||||
setPurpose("");
|
||||
setExecution(null);
|
||||
setError("");
|
||||
const controller = new AbortController();
|
||||
const optionParameters = report.parameters.filter((item) => item.options_from_provider);
|
||||
void Promise.all(optionParameters.map(async (parameter) => {
|
||||
const result = await listProviderParameterOptions(
|
||||
settings,
|
||||
report,
|
||||
parameter.key,
|
||||
"",
|
||||
controller.signal
|
||||
);
|
||||
return [parameter.key, result.options] as const;
|
||||
})).then((entries) => {
|
||||
setOptions(Object.fromEntries(entries));
|
||||
setParameters(Object.fromEntries(entries.flatMap(([key, values]) =>
|
||||
values[0] ? [[key, values[0].value]] : []
|
||||
)));
|
||||
}).catch((reason) => {
|
||||
if ((reason as Error).name !== "AbortError") setError(message(reason));
|
||||
});
|
||||
return () => controller.abort();
|
||||
}, [settings, report.provider_id, report.report_id, report.revision]);
|
||||
|
||||
async function run() {
|
||||
setRunning(true);
|
||||
setError("");
|
||||
try {
|
||||
setExecution(await runProviderReport(
|
||||
settings,
|
||||
report,
|
||||
parameters,
|
||||
purpose.trim(),
|
||||
audienceScope
|
||||
));
|
||||
} catch (reason) {
|
||||
setError(message(reason));
|
||||
} finally {
|
||||
setRunning(false);
|
||||
}
|
||||
}
|
||||
|
||||
async function download(format: "json" | "csv") {
|
||||
if (!execution) return;
|
||||
try {
|
||||
await downloadProviderExecution(
|
||||
settings,
|
||||
execution,
|
||||
format,
|
||||
purpose.trim(),
|
||||
audienceScope
|
||||
);
|
||||
} catch (reason) {
|
||||
setError(message(reason));
|
||||
}
|
||||
}
|
||||
|
||||
const missingRequired = report.parameters.some((item) =>
|
||||
item.required && (parameters[item.key] === undefined || parameters[item.key] === "")
|
||||
);
|
||||
return (
|
||||
<>
|
||||
<header className="reporting-result-header">
|
||||
<div className="reporting-title">
|
||||
<span>{report.provider_id} · {report.revision}</span>
|
||||
<h1>{report.title}</h1>
|
||||
<p>{report.summary}</p>
|
||||
</div>
|
||||
<div className="reporting-run-actions">
|
||||
<Button
|
||||
variant="primary"
|
||||
onClick={() => void run()}
|
||||
disabled={!report.available || running || missingRequired || !purpose.trim()}>
|
||||
<Play size={16} aria-hidden="true" /> {running ? "Running" : "Run"}
|
||||
</Button>
|
||||
</div>
|
||||
</header>
|
||||
{error && <DismissibleAlert tone="danger" resetKey={error}>{error}</DismissibleAlert>}
|
||||
{!report.available &&
|
||||
<DismissibleAlert tone="warning" dismissible={false}>
|
||||
{report.unavailable_reason ?? "Policy does not allow this report."}
|
||||
</DismissibleAlert>
|
||||
}
|
||||
<div className="reporting-query-controls reporting-provider-controls">
|
||||
{report.parameters.map((parameter) =>
|
||||
<label className="reporting-parameter" key={parameter.key}>
|
||||
<span>{parameter.label}{parameter.required ? " *" : ""}</span>
|
||||
{parameter.options_from_provider ?
|
||||
<select
|
||||
value={String(parameters[parameter.key] ?? "")}
|
||||
onChange={(event) => setParameters((current) => ({ ...current, [parameter.key]: event.target.value }))}>
|
||||
{!parameter.required && <option value="">Current/default</option>}
|
||||
{(options[parameter.key] ?? []).map((item) =>
|
||||
<option value={item.value} key={item.value}>{item.label}{item.description ? ` · ${item.description}` : ""}</option>
|
||||
)}
|
||||
</select> :
|
||||
<input
|
||||
type={parameter.type === "integer" || parameter.type === "number" ? "number" : parameter.type === "date" ? "date" : "text"}
|
||||
value={String(parameters[parameter.key] ?? "")}
|
||||
onChange={(event) => setParameters((current) => ({ ...current, [parameter.key]: event.target.value }))}
|
||||
/>
|
||||
}
|
||||
{parameter.description && <small>{parameter.description}</small>}
|
||||
</label>
|
||||
)}
|
||||
<label className="reporting-parameter reporting-provider-purpose">
|
||||
<span>Purpose *</span>
|
||||
<input
|
||||
value={purpose}
|
||||
maxLength={1000}
|
||||
onChange={(event) => setPurpose(event.target.value)}
|
||||
placeholder="Purpose recorded with this execution"
|
||||
/>
|
||||
</label>
|
||||
<label className="reporting-parameter">
|
||||
<span>Effective audience</span>
|
||||
<input value={tenant.name} readOnly />
|
||||
</label>
|
||||
</div>
|
||||
{execution ?
|
||||
<>
|
||||
<div className="reporting-output-toolbar">
|
||||
<span>Generated {formatDateTime(execution.generated_at)}</span>
|
||||
{report.export_formats.includes("csv") &&
|
||||
<IconButton label="Download CSV" icon={<Download size={17} />} variant="ghost" onClick={() => void download("csv")} />
|
||||
}
|
||||
{report.export_formats.includes("json") &&
|
||||
<IconButton label="Download JSON" icon={<FileJson size={17} />} variant="ghost" onClick={() => void download("json")} />
|
||||
}
|
||||
</div>
|
||||
<div className="reporting-provider-output">
|
||||
<ProviderResult execution={execution} />
|
||||
</div>
|
||||
</> :
|
||||
<div className="reporting-empty">Select the parameters and run this governed report.</div>
|
||||
}
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
export function ProviderReportInspector({ report }: { report: ProviderReportDescriptor }) {
|
||||
return (
|
||||
<div className="reporting-inspector-content">
|
||||
<section className="reporting-provenance">
|
||||
<h2><ShieldCheck size={16} /> Governance</h2>
|
||||
<dl>
|
||||
<dt>Risk</dt><dd><StatusBadge status={report.reidentification_risk === "high" ? "warning" : "active"} label={humanize(report.reidentification_risk)} /></dd>
|
||||
<dt>Retention</dt><dd>{report.governance.retention_days == null ? "Policy managed" : `${report.governance.retention_days} days`}</dd>
|
||||
<dt>Exports</dt><dd>{report.governance.export_formats.join(", ") || "Disabled"}</dd>
|
||||
<dt>Contract</dt><dd>{report.contract_version}</dd>
|
||||
</dl>
|
||||
</section>
|
||||
<section>
|
||||
<h2>Privacy transforms</h2>
|
||||
{report.privacy_transforms.map((item) =>
|
||||
<p key={item.id}>{item.label}{item.required ? " · required" : ""}</p>
|
||||
)}
|
||||
</section>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
function ProviderResult({ execution }: { execution: ProviderReportExecution }) {
|
||||
const groups = groupFields(execution.result_schema);
|
||||
return (
|
||||
<>
|
||||
{[...groups].map(([group, fields]) => {
|
||||
const metrics = fields.filter((field) => field.type === "suppressed_count");
|
||||
const details = fields.filter((field) => field.type !== "suppressed_count");
|
||||
return (
|
||||
<Card title={group} key={group}>
|
||||
{metrics.length > 0 &&
|
||||
<div className="dashboard-grid reporting-provider-metrics">
|
||||
{metrics.map((field) =>
|
||||
<MetricCard key={field.path} label={field.label} value={displayValue(pathValue(execution.result, field.path))} />
|
||||
)}
|
||||
</div>
|
||||
}
|
||||
{details.length > 0 &&
|
||||
<dl className="detail-list">
|
||||
{details.map((field) =>
|
||||
<div key={field.path}>
|
||||
<dt>{field.label}</dt>
|
||||
<dd>{displayValue(pathValue(execution.result, field.path), field)}</dd>
|
||||
</div>
|
||||
)}
|
||||
</dl>
|
||||
}
|
||||
</Card>
|
||||
);
|
||||
})}
|
||||
<Card title="Provenance">
|
||||
<dl className="detail-list">
|
||||
<div><dt>Purpose</dt><dd>{execution.purpose}</dd></div>
|
||||
<div><dt>Output hash</dt><dd title={execution.output_hash}>{shortHash(execution.output_hash)}</dd></div>
|
||||
<div><dt>Source revisions</dt><dd>{execution.source_revisions.length}</dd></div>
|
||||
<div><dt>Privacy transforms</dt><dd>{execution.privacy_transforms.join(", ")}</dd></div>
|
||||
<div><dt>Expires</dt><dd>{execution.expires_at ? formatDateTime(execution.expires_at) : "Policy managed"}</dd></div>
|
||||
</dl>
|
||||
</Card>
|
||||
</>
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
function groupFields(fields: ProviderReportField[]): Map<string, ProviderReportField[]> {
|
||||
const groups = new Map<string, ProviderReportField[]>();
|
||||
for (const field of fields) groups.set(field.group, [...(groups.get(field.group) ?? []), field]);
|
||||
return groups;
|
||||
}
|
||||
|
||||
function pathValue(payload: Record<string, unknown>, path: string): unknown {
|
||||
let value: unknown = payload;
|
||||
for (const part of path.split(".")) {
|
||||
if (!value || typeof value !== "object") return null;
|
||||
value = (value as Record<string, unknown>)[part];
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
function displayValue(value: unknown, field?: ProviderReportField): string | number {
|
||||
if (value && typeof value === "object") {
|
||||
const count = value as { value?: unknown; suppressed?: boolean };
|
||||
if (count.suppressed) return "Suppressed";
|
||||
if ("value" in count) return displayValue(count.value);
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
if (value === null || value === undefined || value === "") return "—";
|
||||
if (field?.type === "datetime" || field?.type === "date") return formatDateTime(String(value));
|
||||
if (typeof value === "number") return new Intl.NumberFormat().format(value);
|
||||
if (typeof value === "boolean") return value ? "Yes" : "No";
|
||||
return String(value);
|
||||
}
|
||||
|
||||
function formatDateTime(value: string): string {
|
||||
const date = new Date(value);
|
||||
return Number.isNaN(date.valueOf()) ? value : new Intl.DateTimeFormat(undefined, { dateStyle: "medium", timeStyle: "short" }).format(date);
|
||||
}
|
||||
|
||||
function humanize(value: string): string {
|
||||
return value.replaceAll("_", " ").replace(/\b\w/g, (letter) => letter.toUpperCase());
|
||||
}
|
||||
|
||||
function shortHash(value: string): string {
|
||||
return `${value.slice(0, 10)}…${value.slice(-6)}`;
|
||||
}
|
||||
|
||||
function message(reason: unknown): string {
|
||||
return reason instanceof Error ? reason.message : "The provider report could not be loaded.";
|
||||
}
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
getDefinition,
|
||||
listDefinitions,
|
||||
listExecutions,
|
||||
listProviderReports,
|
||||
listSavedViews,
|
||||
reportPayload,
|
||||
runReport,
|
||||
@@ -47,8 +48,13 @@ import {
|
||||
type ReportingQuery,
|
||||
type ReportingQueryMode,
|
||||
type ReportingSavedView,
|
||||
type ProviderReportDescriptor,
|
||||
type SemanticModelPayload
|
||||
} from "../../api/reporting";
|
||||
import {
|
||||
ProviderReportInspector,
|
||||
ProviderReportWorkspace
|
||||
} from "./ProviderReportWorkspace";
|
||||
|
||||
|
||||
type OutputMode = "visual" | "table";
|
||||
@@ -57,7 +63,9 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
const [search, setSearch] = useState("");
|
||||
const [submittedSearch, setSubmittedSearch] = useState("");
|
||||
const [reports, setReports] = useState<ReportingDefinition[]>([]);
|
||||
const [providerReports, setProviderReports] = useState<ProviderReportDescriptor[]>([]);
|
||||
const [selectedId, setSelectedId] = useState("");
|
||||
const [selectedProviderKey, setSelectedProviderKey] = useState("");
|
||||
const [semantic, setSemantic] = useState<SemanticModelPayload | null>(null);
|
||||
const [query, setQuery] = useState<ReportingQuery>(emptyQuery());
|
||||
const [parameters, setParameters] = useState<Record<string, unknown>>({});
|
||||
@@ -77,21 +85,42 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
() => reports.find((item) => item.definition_id === selectedId) ?? null,
|
||||
[reports, selectedId]
|
||||
);
|
||||
const selectedProvider = useMemo(
|
||||
() => providerReports.find((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey) ?? null,
|
||||
[providerReports, selectedProviderKey]
|
||||
);
|
||||
const report = selected ? reportPayload(selected) : null;
|
||||
|
||||
function reload(signal?: AbortSignal) {
|
||||
setLoading(true);
|
||||
setError("");
|
||||
return listDefinitions(settings, {
|
||||
return Promise.all([
|
||||
listDefinitions(settings, {
|
||||
kinds: ["report"],
|
||||
status: ["active"],
|
||||
query: submittedSearch,
|
||||
limit: 200
|
||||
}, signal).
|
||||
then((result) => {
|
||||
}, signal),
|
||||
listProviderReports(settings, signal)
|
||||
]).
|
||||
then(([result, providerResult]) => {
|
||||
const providerRows = providerResult.reports.filter((item) => {
|
||||
const query = submittedSearch.toLocaleLowerCase();
|
||||
return !query || `${item.title} ${item.summary} ${item.provider_id}`.toLocaleLowerCase().includes(query);
|
||||
});
|
||||
setReports(result.definitions);
|
||||
setSelectedId((current) => result.definitions.some((item) => item.definition_id === current) ?
|
||||
current : result.definitions[0]?.definition_id ?? "");
|
||||
setProviderReports(providerRows);
|
||||
const currentSemanticAvailable = result.definitions.some((item) => item.definition_id === selectedId);
|
||||
const currentProviderAvailable = providerRows.some((item) => `${item.provider_id}:${item.report_id}` === selectedProviderKey);
|
||||
if (!currentSemanticAvailable && !currentProviderAvailable) {
|
||||
if (providerRows[0]) {
|
||||
setSelectedProviderKey(`${providerRows[0].provider_id}:${providerRows[0].report_id}`);
|
||||
setSelectedId("");
|
||||
} else {
|
||||
setSelectedId(result.definitions[0]?.definition_id ?? "");
|
||||
setSelectedProviderKey("");
|
||||
}
|
||||
}
|
||||
}).
|
||||
catch((reason) => {
|
||||
if ((reason as Error).name !== "AbortError") setError(message(reason));
|
||||
@@ -172,7 +201,7 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
placeholder="Search reports"
|
||||
/>
|
||||
</form>
|
||||
<span className="reporting-count">{reports.length} reports</span>
|
||||
<span className="reporting-count">{reports.length + providerReports.length} reports</span>
|
||||
<IconButton
|
||||
label="Reload reports"
|
||||
icon={<RefreshCw size={17} />}
|
||||
@@ -188,15 +217,32 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
<div className="reporting-workspace">
|
||||
<PageScrollViewport className="reporting-catalogue">
|
||||
{loading && <LoadingIndicator label="Loading reports" />}
|
||||
{!loading && reports.length === 0 && <div className="reporting-empty">No active reports are available.</div>}
|
||||
{!loading && reports.length + providerReports.length === 0 && <div className="reporting-empty">No active reports are available.</div>}
|
||||
<div className="reporting-report-list" role="list">
|
||||
{providerReports.length > 0 && <div className="reporting-list-heading">Module reports</div>}
|
||||
{providerReports.map((item) => {
|
||||
const key = `${item.provider_id}:${item.report_id}`;
|
||||
return (
|
||||
<button
|
||||
type="button"
|
||||
role="listitem"
|
||||
key={key}
|
||||
className={`reporting-report-row${selectedProviderKey === key ? " is-selected" : ""}`}
|
||||
onClick={() => { setSelectedProviderKey(key); setSelectedId(""); }}>
|
||||
<BarChart3 size={17} aria-hidden="true" />
|
||||
<span><strong>{item.title}</strong><small>{item.provider_id} · {item.revision}</small></span>
|
||||
<StatusBadge status={item.available ? "active" : "locked"} label={item.available ? "Available" : "Restricted"} />
|
||||
</button>
|
||||
);
|
||||
})}
|
||||
{reports.length > 0 && <div className="reporting-list-heading">Semantic reports</div>}
|
||||
{reports.map((item) =>
|
||||
<button
|
||||
type="button"
|
||||
role="listitem"
|
||||
key={item.definition_id}
|
||||
className={`reporting-report-row${selectedId === item.definition_id ? " is-selected" : ""}`}
|
||||
onClick={() => setSelectedId(item.definition_id)}>
|
||||
onClick={() => { setSelectedId(item.definition_id); setSelectedProviderKey(""); }}>
|
||||
<BarChart3 size={17} aria-hidden="true" />
|
||||
<span><strong>{item.name}</strong><small>{item.definition_key} · r{item.revision}</small></span>
|
||||
<StatusBadge status="active" label="Active" />
|
||||
@@ -205,7 +251,9 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
</div>
|
||||
</PageScrollViewport>
|
||||
<section className="reporting-result-region">
|
||||
{selected && report ?
|
||||
{selectedProvider ?
|
||||
<ProviderReportWorkspace settings={settings} auth={auth} report={selectedProvider} /> :
|
||||
selected && report ?
|
||||
<>
|
||||
<header className="reporting-result-header">
|
||||
<div className="reporting-title">
|
||||
@@ -258,6 +306,8 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
}
|
||||
</section>
|
||||
<PageScrollViewport className="reporting-inspector">
|
||||
{selectedProvider ?
|
||||
<ProviderReportInspector report={selectedProvider} /> :
|
||||
<Inspector
|
||||
selected={selected}
|
||||
execution={execution}
|
||||
@@ -265,7 +315,7 @@ export default function ReportingPage({ settings, auth }: PlatformRouteContext)
|
||||
savedViews={savedViews}
|
||||
onSelectExecution={setExecution}
|
||||
onApplySavedView={applySavedView}
|
||||
/>
|
||||
/>}
|
||||
</PageScrollViewport>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
+9
-2
@@ -24,16 +24,23 @@ export const reportingModule: PlatformWebModule = {
|
||||
],
|
||||
routes: [
|
||||
{
|
||||
path: "/reporting",
|
||||
path: "/reports",
|
||||
anyOf: ["reporting:definition:read"],
|
||||
order: 74,
|
||||
surfaceId: "reporting.workspace",
|
||||
render: (context) => createElement(ReportingPage, context)
|
||||
},
|
||||
{
|
||||
path: "/reporting",
|
||||
anyOf: ["reporting:definition:read"],
|
||||
order: 175,
|
||||
surfaceId: "reporting.compatibility",
|
||||
render: (context) => createElement(ReportingPage, context)
|
||||
}
|
||||
],
|
||||
navItems: [
|
||||
{
|
||||
to: "/reporting",
|
||||
to: "/reports",
|
||||
label: "Reporting",
|
||||
iconName: "clipboard-pen-line",
|
||||
anyOf: ["reporting:definition:read"],
|
||||
|
||||
@@ -74,6 +74,16 @@
|
||||
background: var(--surface-raised);
|
||||
}
|
||||
|
||||
.reporting-list-heading {
|
||||
padding: 7px 10px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--surface-subtle, var(--surface));
|
||||
color: var(--text-soft);
|
||||
font-size: 0.7rem;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.reporting-report-row,
|
||||
.reporting-inspector-content section > button {
|
||||
display: grid;
|
||||
@@ -162,6 +172,16 @@
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.reporting-title p {
|
||||
max-width: 70ch;
|
||||
margin: 3px 0 0;
|
||||
overflow: hidden;
|
||||
color: var(--text-soft);
|
||||
font-size: 0.76rem;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.reporting-run-actions {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
@@ -223,6 +243,14 @@
|
||||
gap: 4px;
|
||||
}
|
||||
|
||||
.reporting-provider-controls {
|
||||
align-items: end;
|
||||
}
|
||||
|
||||
.reporting-provider-purpose {
|
||||
min-width: min(320px, 35vw);
|
||||
}
|
||||
|
||||
.reporting-output-toolbar {
|
||||
min-height: 48px;
|
||||
padding: 6px 12px;
|
||||
@@ -241,6 +269,21 @@
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.reporting-provider-output {
|
||||
display: flex;
|
||||
min-height: 0;
|
||||
flex: 1;
|
||||
flex-direction: column;
|
||||
gap: 10px;
|
||||
overflow: auto;
|
||||
padding: 10px;
|
||||
}
|
||||
|
||||
.reporting-provider-metrics {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.reporting-output .data-grid-shell {
|
||||
min-width: 100%;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user