feat: implement governed reporting vertical
This commit is contained in:
@@ -0,0 +1,42 @@
|
||||
# Reporting Administration Guide
|
||||
|
||||
## Definition graph
|
||||
|
||||
Reporting definitions form an exact graph:
|
||||
|
||||
```text
|
||||
Dataset revision -> Semantic-model revision -> Report revision
|
||||
-> Quality-plan revision
|
||||
```
|
||||
|
||||
Create parents before children. An active child may reference only an active,
|
||||
existing parent revision. Editing creates a new immutable revision and
|
||||
requires the currently observed revision number. Existing runs continue to
|
||||
reference the historical revisions they used.
|
||||
|
||||
Datasets may bind a static fixture, a pinned Dataflow output, or a capability
|
||||
published by a source-owning module. Do not expose another module's ORM or an
|
||||
unbounded SQL connection as a report source. Configure an explicit schema,
|
||||
freshness policy, source fingerprint expectations, purpose, privacy,
|
||||
retention, and a row-policy provider where source access alone is not enough.
|
||||
|
||||
## Access and publication
|
||||
|
||||
Tenant-visible definitions are readable by principals with Reporting read
|
||||
permission. Restricted reports use normalized account, identity, group, role,
|
||||
function, assignment, organization-unit, or service-account grants. The
|
||||
creator and Reporting administrators retain management access.
|
||||
|
||||
Scheduled output publication requires an installed capability implementing
|
||||
the Reporting publication-target contract. The target receives one immutable
|
||||
execution payload and an idempotency key. It must return bounded evidence and
|
||||
must not expose credentials in that evidence.
|
||||
|
||||
## Import assessments
|
||||
|
||||
Import assessment accepts declarative metadata only. Native datasets,
|
||||
dimensions, hierarchies, measures, parameters, tables, pivots, charts,
|
||||
quality assertions, and saved views map exactly. Provider-specific formatting,
|
||||
dialect functions, and dashboard layouts require explicit approximation
|
||||
acceptance. Raw SQL, procedures, scripts, implicit authorization, unchecked
|
||||
functions, and unknown features block activation.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Reporting Operations, Security, and Recovery
|
||||
|
||||
## Runtime behavior
|
||||
|
||||
Reporting API nodes are stateless. Definitions, executions, schedules,
|
||||
publication evidence, and import/quality evidence live in the shared platform
|
||||
database. Dataset bytes remain with their source owner; large durable output
|
||||
files belong in a publication provider backed by shared object storage.
|
||||
|
||||
Run and schedule idempotency keys make retries replay-safe. Schedule workers
|
||||
claim bounded batches, retain success or failure state, and advance interval
|
||||
time from the claimed occurrence. Providers must use the same tenant and
|
||||
principal context and must enforce their source-side authorization.
|
||||
|
||||
## Security controls
|
||||
|
||||
- The semantic engine never evaluates Python, browser code, raw SQL, stored
|
||||
procedures, or imported runtime scripts.
|
||||
- Definition and execution APIs require explicit module scopes and a
|
||||
tenant-bound principal.
|
||||
- Restricted access is normalized into grants and rechecked before definitions
|
||||
or execution history are returned.
|
||||
- Row-policy providers may only reduce a dataset; returning more rows than the
|
||||
source read fails closed.
|
||||
- Freshness, schema, fingerprints, quality gates, definition hashes, source
|
||||
provenance, actor, and output hash are retained with every run.
|
||||
- CSV formula injection is neutralized before direct download.
|
||||
|
||||
## Backup and restore
|
||||
|
||||
Back up the shared database and every object-storage publication target as one
|
||||
recovery set. Before destructive module retirement, take and verify a database
|
||||
snapshot; retirement removes definitions, result evidence, schedules, and
|
||||
publication records. Restore the database first, then target object storage,
|
||||
then reconcile provider publications by their idempotency/evidence references.
|
||||
|
||||
After restore, verify migrations, definition parent pins, schedule enablement,
|
||||
execution counts and hashes, and publication target health before dispatching
|
||||
due schedules. Do not claim successful recovery until an operator has run and
|
||||
recorded the deployment-specific drill.
|
||||
+71
-18
@@ -15,6 +15,9 @@ Reporting owns:
|
||||
- curated query/view inputs from modules, connectors, datasources, or
|
||||
configuration packages
|
||||
- report execution history, generated output evidence, and scheduled runs
|
||||
- institutional question, goal/obligation, owner, calculation-version,
|
||||
freshness/quality, and decision/action-consumption references for each
|
||||
material report or indicator
|
||||
- report permissions and sharing rules
|
||||
- export targets such as file, DMS, mail, API, RSS/Atom publication, and
|
||||
downstream connector handoff
|
||||
@@ -28,13 +31,27 @@ Initial source categories:
|
||||
- curated SQL/query views exposed through a controlled capability
|
||||
- connector-staged datasets and external references
|
||||
- generated file/csv/xlsx inputs from files/connectors
|
||||
- future `govoplan-datasources` catalog entries, if that module is created
|
||||
- future `govoplan-dataflow` outputs, if that module is created
|
||||
- governed `govoplan-datasources` catalogue entries and immutable
|
||||
materializations
|
||||
- versioned `govoplan-dataflow` outputs and run lineage
|
||||
|
||||
Reporting should not reach into module ORM internals directly. Module-owned
|
||||
data must be exposed through capabilities, DTOs, events, read models, or
|
||||
controlled query views.
|
||||
|
||||
Every reproducible report definition should retain or resolve:
|
||||
|
||||
- the institutional question, obligation, goal, or service measure it serves;
|
||||
- owner and responsible organization/function;
|
||||
- datasource/materialization and Dataflow definition/run revisions;
|
||||
- source freshness, quality state, known limitations, and calculation version;
|
||||
- purpose, visibility, privacy, retention, and publication decisions;
|
||||
- reports, decisions, controls, projects, or actions that consumed the result.
|
||||
|
||||
Reporting owns execution and presentation of the measure. Mandates, Services,
|
||||
Projects, Risk Compliance, Decisions, and other domains retain ownership of
|
||||
the referenced institutional concepts.
|
||||
|
||||
## Output Targets
|
||||
|
||||
- dashboard/widget view
|
||||
@@ -53,24 +70,60 @@ Reporting does not own:
|
||||
- reusable template rendering; that belongs to `govoplan-templates`
|
||||
- DMS lifecycle, collaborative editing, legal hold, or records management
|
||||
- raw file/blob storage and provider connectors
|
||||
- general-purpose ingestion/transformation pipeline ownership unless a future
|
||||
`govoplan-dataflow` module is justified
|
||||
- ingestion, governed source identity, staging, or transformation pipeline
|
||||
ownership; those belong to Connectors, Datasources, and Dataflow
|
||||
- cross-module search indexing; that belongs to `govoplan-search`
|
||||
|
||||
## Candidate Capabilities
|
||||
## Capability Contracts
|
||||
|
||||
- `reporting.catalog`
|
||||
- `reporting.run`
|
||||
- `reporting.schedules`
|
||||
- `reporting.exports`
|
||||
- `reporting.readModels`
|
||||
- `reporting.dashboardContributions`
|
||||
- `reporting.registry` owns immutable definition registration and lookup.
|
||||
- `reporting.runner` executes exact report graphs and returns governed result
|
||||
evidence.
|
||||
- `reporting.scheduler` claims due schedules and preserves run/publication
|
||||
outcomes.
|
||||
- `reporting.chart_renderer` renders provider-neutral visual models with an
|
||||
accessible table fallback.
|
||||
- `reporting.read_model:*` capabilities can expose bounded source-owned rows.
|
||||
- `reporting.publication_target:*` capabilities can accept immutable result
|
||||
payloads without Reporting importing the target module.
|
||||
|
||||
## First Implementation Slice
|
||||
## Implemented Vertical
|
||||
|
||||
1. Define manifest metadata, permissions, and capability names.
|
||||
2. Add report definition and parameter DTOs.
|
||||
3. Add one read-only report source contract over module-provided summary DTOs.
|
||||
4. Add export target DTOs without implementing every target.
|
||||
5. Add tests that report consumers/producers use capability lookup rather than
|
||||
direct imports.
|
||||
The first complete vertical persists datasets, semantic models, reports,
|
||||
quality plans, grants, executions, saved views, schedules, publications,
|
||||
quality results, and import assessments. Each definition update creates an
|
||||
immutable revision and requires optimistic concurrency. Active child
|
||||
definitions require the exact pinned parent revision to be active.
|
||||
|
||||
Execution resolves the report, semantic model, and dataset graph before
|
||||
reading data. Static fixtures, Dataflow output, and provider-owned read models
|
||||
share one bounded read contract. Source fingerprints, freshness, schema,
|
||||
row-policy provenance, blocking quality plans, definition hashes, executor
|
||||
version, output hash, diagnostics, and authorized rows are retained with the
|
||||
execution. Failed runs also retain evidence.
|
||||
|
||||
The query engine deliberately implements a typed expression and semantic
|
||||
query language rather than `eval`, arbitrary SQL, stored procedures, or
|
||||
runtime scripts. It supports detail, grouped summary, pivot, dimensions,
|
||||
hierarchies, common aggregates, calculated measures, filters, sorting,
|
||||
pagination, totals, and a provider-neutral visualization model. A saved chart
|
||||
that is incompatible with an ad-hoc query degrades to its mandatory table
|
||||
fallback instead of failing a valid report run.
|
||||
|
||||
Direct export supports UTF-8 CSV and JSON. CSV cells that spreadsheet software
|
||||
could interpret as formulas are escaped. Additional formats and delivery
|
||||
destinations use an optional publication capability and preserve idempotent
|
||||
evidence. Import assessments classify exact, approximated, and unsupported
|
||||
semantics and block activation until every approximation is accepted and no
|
||||
unsupported executable behavior remains.
|
||||
|
||||
The WebUI uses the platform module loader and common controls. It exposes a
|
||||
report catalogue, parameter and semantic-query controls, result visualization
|
||||
and table views, history/provenance, saved views, schedules, and downloads.
|
||||
|
||||
## Remaining Product Depth
|
||||
|
||||
The architecture boundary is implemented. Further work is additive product
|
||||
depth: richer visualization providers, drill-through navigation, packaged
|
||||
domain report catalogues, XLSX/PDF formatting through optional providers, and
|
||||
target-environment evidence for a maturity claim above `vertical_slice`.
|
||||
|
||||
@@ -42,13 +42,13 @@ The supplied modules collectively provide five relevant layers:
|
||||
| Scope inheritance and provenance | Available foundation | Policy/Core |
|
||||
| Configurable dashboard widgets | Available foundation | Dashboard |
|
||||
| Reusable parameterized subflows | Available foundation | Dataflow |
|
||||
| BI dimensions, measures, and hierarchies | Missing | Reporting |
|
||||
| Pivot/drill-down analytical view | Missing | Reporting |
|
||||
| Parameterized report masks and saved views | Missing | Reporting |
|
||||
| Chart renderer/provider contract | Missing | Reporting |
|
||||
| Scheduled report publication/export | Partially modelled, not integrated | Reporting |
|
||||
| Report-specific access explanations | Missing | Reporting/Policy |
|
||||
| Data-quality test plans and evidence UI | Partial | Dataflow/Reporting |
|
||||
| BI dimensions, measures, and hierarchies | Available | Reporting |
|
||||
| Pivot analytical view | Available; drill-through is product depth | Reporting |
|
||||
| Parameterized report masks and saved views | Available | Reporting |
|
||||
| Chart renderer/provider contract | Available with tabular fallback | Reporting |
|
||||
| Scheduled report publication/export | Available through provider contract | Reporting |
|
||||
| Report-specific access enforcement | Available; richer explanations are product depth | Reporting/Policy |
|
||||
| Data-quality test plans and evidence | Available foundation | Dataflow/Reporting |
|
||||
| Packaged domain report catalogues | Missing | Reporting plus domain modules |
|
||||
|
||||
GovOPlaN already has most platform primitives below the BI layer. The largest
|
||||
@@ -102,9 +102,9 @@ accepts a documented approximation. Imported definitions remain ordinary
|
||||
versioned GovOPlaN definitions after migration; the runtime must not depend on
|
||||
the original SuperX installation.
|
||||
|
||||
## Delivery Plan
|
||||
## Delivery State
|
||||
|
||||
### Phase 1: Semantic Core
|
||||
### Phase 1: Semantic Core - complete
|
||||
|
||||
- scaffold the Reporting module and manifest
|
||||
- define analytical dataset, dimension, hierarchy, measure, parameter, and
|
||||
@@ -112,7 +112,7 @@ the original SuperX installation.
|
||||
- consume one Dataflow-published dataset through a capability
|
||||
- enforce scope, Policy provenance, and source freshness
|
||||
|
||||
### Phase 2: Interactive BI
|
||||
### Phase 2: Interactive BI - vertical complete
|
||||
|
||||
- parameter panel and saved views
|
||||
- table, pivot, drill-down, totals, calculated measures, and export of the
|
||||
@@ -120,20 +120,24 @@ the original SuperX installation.
|
||||
- stable URL/reference contract for report state
|
||||
- dashboard contribution for saved reports
|
||||
|
||||
### Phase 3: Visualization And Publication
|
||||
### Phase 3: Visualization And Publication - vertical complete
|
||||
|
||||
- chart renderer provider contract with an initial bounded chart catalogue
|
||||
- scheduled runs and publication through Files, Mail, or another declared
|
||||
target capability
|
||||
- accessible tabular fallback and report-specific access explanations
|
||||
|
||||
### Phase 4: Quality And Domain Packs
|
||||
### Phase 4: Quality And Domain Packs - foundation complete
|
||||
|
||||
- reusable report assertions and dataset comparison plans
|
||||
- execution evidence and trend views
|
||||
- versioned domain report packages owned by the relevant domain modules
|
||||
- import tooling for metadata where licensing and semantic mapping permit it
|
||||
|
||||
Packaged domain catalogues and an importer that translates actual source
|
||||
metadata remain product depth. The native contracts, quality evidence, and
|
||||
blocking import assessment required to implement them are present.
|
||||
|
||||
## Product Boundary
|
||||
|
||||
Dataflow remains responsible for producing governed tabular datasets.
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Reporting User Guide
|
||||
|
||||
## Open and run a report
|
||||
|
||||
Open **Reporting** and choose an active report from the left catalogue. The
|
||||
header identifies the immutable report revision that will be used. Select a
|
||||
summary, detail, or pivot shape, choose dimensions and measures, enter any
|
||||
report parameters, and select **Run**.
|
||||
|
||||
Only rows authorized for the current tenant and principal reach the query
|
||||
engine. A successful result shows its authorized row count, visualization,
|
||||
and table. When a saved chart does not match an ad-hoc query, Reporting shows
|
||||
the accessible table fallback instead of changing or rejecting the query.
|
||||
|
||||
## Inspect evidence
|
||||
|
||||
The right panel lists previous runs and definition/source pins. Select an
|
||||
earlier run to inspect its retained result. Content and output hashes identify
|
||||
the exact definition and output. Warnings explain freshness, inferred schema,
|
||||
or provider diagnostics. A failed quality gate records a failed execution and
|
||||
does not publish a result.
|
||||
|
||||
## Save and export
|
||||
|
||||
Use **Save current view** to keep the current query under your account. Saved
|
||||
views do not alter the report definition. CSV and JSON downloads contain only
|
||||
the authorized result of the selected execution. CSV values that begin with a
|
||||
spreadsheet formula marker are escaped.
|
||||
|
||||
Users with scheduling permission can create an hourly, daily, weekly, or
|
||||
30-day interval from the current revision, parameters, and query. Scheduled
|
||||
runs continue to use those exact pins until the schedule is edited.
|
||||
Reference in New Issue
Block a user