From 430a9d8ef9f355385da23779690cc66f2e42f539 Mon Sep 17 00:00:00 2001 From: zemion Date: Mon, 6 Jul 2026 14:37:24 +0200 Subject: [PATCH] Sync Repo-docs-gtfs-harmonization from project files --- Repo-docs-gtfs-harmonization.-.md | 251 ++++++++++++++++++++++++++++++ 1 file changed, 251 insertions(+) create mode 100644 Repo-docs-gtfs-harmonization.-.md diff --git a/Repo-docs-gtfs-harmonization.-.md b/Repo-docs-gtfs-harmonization.-.md new file mode 100644 index 0000000..f00a576 --- /dev/null +++ b/Repo-docs-gtfs-harmonization.-.md @@ -0,0 +1,251 @@ + + +> Mirrored from `/mnt/DATA/git/meubility-workbench/docs/gtfs_harmonization.md`. +> Origin: `repository`. +> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context. + +--- +# GTFS Harmonization and QA Concept + +Last updated: 2026-07-02 + +## Decision + +Run harmonization inside the existing Mobility Workbench for now: + +- Same FastAPI server. +- Same operator/data-engineering UI. +- Same PostgreSQL/PostGIS database. +- Separate GTFS Harmonization and Mapping Data UI modules backed by the existing source/job tables. +- Separate QA/harmonization API surface starting with `/api/qa/*`. +- Separate canonical export concept, but no separate public API backend yet. + +Split this into a separate service later when one of these becomes true: + +- third-party API consumers need independent uptime, auth, quotas, billing, or SLA boundaries; +- export jobs need independent workers, storage, or scaling; +- canonical data publication needs immutable release management independent of the editing workbench; +- commercial/public API concerns start slowing down internal QA and import workflows. + +The public/API product should not expose raw workbench tables directly. It should consume versioned canonical snapshots. + +The journey/routing interface should consume the active harmonized transit snapshot. It should not expose raw GTFS feed selection as a normal traveller-facing routing control. Feed-specific filters remain useful for QA, layer inspection, diagnostics, and source review. + +## Target Pipeline + +```text +source catalog + -> raw feed snapshots + -> validation reports + -> normalized staging tables + -> canonical matching and deduplication + -> conflict review and reusable rules + -> versioned canonical snapshot + -> GTFS/API/GeoParquet exports +``` + +The pan-European output should be a canonical mobility dataset first, not one giant internal GTFS feed. GTFS should be one export format from that canonical snapshot. + +## Core Concepts + +### Layered Source Model + +The target model has four distinct layers: + +1. **Source directories / discovery indexes**: places where sources are found, such as Mobility Database, PTNA, OpenData ÖPNV, NAP registries, Transitland, or manually curated seed files. +2. **Sources / publishers / authorities**: the organization or authority that owns or publishes data, such as DELFI, VBB, RNV, a regional transport authority, or an operator. +3. **Feeds**: concrete machine-readable products from a source, such as a static GTFS ZIP, NeTEx package, GTFS-RT TripUpdates endpoint, GTFS-RT ServiceAlerts endpoint, or operator-specific API endpoint. +4. **Feed snapshots / datasets**: immutable downloaded/imported versions of a feed with hash, timestamp, parser version, validation report, activation status, and imported rows. + +The current database does not fully separate these layers yet: + +- `source_catalog_entries` is a lightweight discovery/backlog catalog. +- `sources` currently represents a registered importable feed, even though the name is broader. +- `datasets` represents imported feed snapshots. +- generated discovery files under `docs/generated/` contain many more feed candidates than are currently promoted into the database review workflow. + +Because of this, the discovery catalog should be much larger than GTFS Feed QA. Feed QA should only show registered GTFS feeds that have been selected for review/import, not every source directory or every discovered candidate row. + +### Source Registry + +Track every identified source, including feeds not yet imported: + +- source URL and publisher; +- country/region/mode coverage; +- source authority and priority; +- update cadence and freshness; +- importability; +- license and redistribution status. + +Mobility Database can be used as a broad discovery connector. Prefer the full `feeds_v2.csv` catalog/API over validator acceptance-test feed lists because it includes feed status, official/source flags, latest/direct URLs, license URLs, features, and bounding boxes. Treat it as candidate metadata: the catalog metadata is reusable, but each transit feed still needs its own provider licence review and authority ranking. + +PTNA can be used as a GTFS/OSM QA and crosswalk connector. Its country pages expose feed IDs, provider names, release dates, validity windows, route-analysis links, detail pages, and original release-page links. Detail pages can add license text, OSM permission notes, `network:guid`, and route matching hints. PTNA should not become the canonical publisher for a feed; the harmonizer should follow the original provider URL where possible and keep PTNA as evidence. + +OpenData ÖPNV is a Germany-focused source-registry connector. It lists DELFI national GTFS/NeTEx/ZHV datasets and many regional/operator datasets. Some downloads, including the Germany-wide DELFI GTFS on the portal, require registration, so those rows must stay in review until credentials and license decisions are recorded. + +The generated discovery files live under `docs/generated/`: + +- `gtfs_feed_candidates.csv` keeps every discovered feed/evidence row. +- `gtfs_ingestable_sources.csv` keeps rows that can be imported as GTFS sources after review. +- `gtfs_test_run_sources.csv` keeps a smaller multi-source set for deduplication tests. + +Required license flags before publication: + +- `can_import` +- `can_derive` +- `can_redistribute` +- `requires_attribution` +- `commercial_restrictions` + +### Source Hierarchy + +The harmonizer should not import every overlapping GTFS source into one active canonical snapshot. Each discovered source gets hierarchy metadata: + +- `national_official` + `national` -> canonical baseline candidate. +- `regional_authority` -> QA or override candidate for a bounded region, not additive by default when a national baseline already covers the same services. +- `operator_feed` -> operator detail candidate, useful for QA, freshest local data, or replacing a bounded subset. +- `mirror` -> bootstrap only unless the original source is unavailable and license review allows use. +- `secondary_discovery` -> evidence metadata only. + +Default import policy: + +- Use one baseline per coverage universe. +- Add regional/operator feeds only as QA inputs or explicit replacements for a defined subset. +- Keep mirrors and aggregators out of canonical publication unless they are the only legally usable source and the provenance is explicit. +- Record every override decision so later imports can replay the same source precedence. + +### Raw Snapshots + +Every update should preserve immutable raw input: + +- source id; +- fetch time; +- source hash; +- upstream metadata; +- parser/import version; +- validator report; +- previous active snapshot. + +This keeps deduplication and conflict decisions reproducible. + +### Update and Diff Policy + +Source updates should not reset reviewed work. A new feed or map import should create a new raw snapshot, compute entity-level diffs, replay existing matching rules, and open review items only for changed, new, conflicting, or no-longer-valid evidence. + +Initial diff units: + +- feed metadata, hash, freshness, and service horizon; +- agencies/operators/networks; +- stops/platforms/station complexes; +- routes/lines; +- route patterns and stop sequences; +- trips, calendars, and stop times; +- route geometry evidence from GTFS shapes and OSM relations; +- OSM stop/platform/route relation candidates. + +Unchanged raw rows may still be stored in the immutable snapshot tables for auditability. The expensive derived work should be incremental: canonical links, route geometry, route-layer rows, and QA review queues should rebuild only where the diff says something changed. + +### Canonical Entities + +Stable meubility IDs should be the internal truth. Source IDs remain aliases. + +Initial canonical entity families: + +- operators/agencies/authorities/networks; +- stop places and station complexes; +- platforms, tracks, bus bays, entrances; +- routes/lines; +- route patterns and trip patterns; +- calendars/service validity; +- shapes/geometries; +- fares/ticketing references later. + +### GTFS to Map Matching Workbench + +The workbench should turn raw source evidence into reviewed, reusable map/timetable links: + +- match GTFS stops to concrete OSM stop/platform/bay nodes or ways, preserving side-of-street and platform detail; +- link nearby but distinct stops with explicit transfer edges instead of merging them into one physical stop; +- match GTFS routes and route patterns to OSM route relations where the evidence is strong; +- when OSM route relations are missing or weak, propose stop-to-stop paths over mode-compatible OSM infrastructure; +- keep GTFS shapes as geometry evidence, especially for detours, temporary routes, and missing OSM relations; +- allow reviewer decisions such as accept OSM relation, use GTFS shape, use generated stop-to-stop map path, ignore visual geometry, split variant, mark detour, or create manual override; +- persist every decision as a reusable rule keyed by canonical entity, source scope, upstream IDs, normalized route keys, geometry hashes, and validity window. + +Candidate generation must be constrained before expensive comparison: + +- compare only OSM route features for route matching; +- filter by mode, network/operator/ref/name evidence, and source authority; +- use PostGIS bbox and distance filters before loading geometry into Python; +- prefer route-pattern and stop-sequence signatures over whole-feed scans; +- rerun candidate generation only for new or changed routes, stops, or map features. + +### Authority Ranking + +Conflict resolution needs explicit source authority: + +- manual review decision; +- national official feed or registry; +- regional authority feed; +- operator feed; +- broad aggregator feed; +- OSM as visual/gap evidence, not timetable authority. + +Authority can differ by entity type. A source can be authoritative for timetable but weak for route geometry or operator identity. + +### Conflict Review + +The QA dashboard should expose review queues for: + +- duplicate operators/agencies; +- duplicate stop places/station complexes; +- GTFS stops without canonical links; +- OSM stops without GTFS/canonical links; +- canonical stop groups with large spatial disagreement; +- routes with missing, weak, or conflicting OSM links; +- routes with missing shapes or route-pattern geometry; +- map-matching proposals that need reviewer approval; +- update diffs that invalidate a previous manual match; +- stale calendars and short service horizons; +- license/redistribution blockers. + +Manual resolutions must become reusable rules so source updates do not reintroduce the same conflict. + +## Export Strategy + +Do not start with one giant Europe GTFS zip as the only product. Produce: + +- versioned canonical snapshot tables; +- country/region GTFS exports; +- network/operator GTFS exports; +- full-Europe analytical dumps such as GeoParquet; +- API-ready entity endpoints later. + +Each export needs: + +- snapshot id; +- source feed versions; +- generation time; +- validation summary; +- license/attribution manifest; +- conflict/review status. + +## Current Implementation Step + +The current implementation is a lightweight harmonization boundary: + +- `/api/qa/summary`; +- source discovery metrics; +- import health metrics; +- GTFS validation counters; +- canonical stop/link coverage; +- route matching and geometry counters; +- publication-readiness warnings. +- GTFS source add/import/review controls live in the `GTFS Harmonization` sidebar module. +- Feed review decisions store review status, note, source authority ranking, and explicit publication flags: + can import, can derive, can redistribute, attribution required, and commercial restrictions. +- OSM/route-layer source controls live in the `Mapping Data` sidebar module. +- The journey panel displays the active harmonized transit snapshot instead of a GTFS source picker. +- The current active harmonized snapshot is computed from active GTFS datasets, review authority, source priority, and route-key overlap; lower-precedence feeds that are mostly covered by a higher-precedence feed are shadowed for default routing while remaining available for QA/source-specific inspection. + +This is intentionally still a skeleton. The next implementation step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions, then run the generated 24-source test set through review, import, validation, and deduplication.