245 lines
12 KiB
Markdown
245 lines
12 KiB
Markdown
# GTFS Harmonization and QA Concept
|
|
|
|
Last updated: 2026-07-02
|
|
|
|
## Decision
|
|
|
|
Run harmonization inside the existing Mobility Workbench for now:
|
|
|
|
- Same FastAPI server.
|
|
- Same operator/data-engineering UI.
|
|
- Same PostgreSQL/PostGIS database.
|
|
- Separate GTFS Harmonization and Mapping Data UI modules backed by the existing source/job tables.
|
|
- Separate QA/harmonization API surface starting with `/api/qa/*`.
|
|
- Separate canonical export concept, but no separate public API backend yet.
|
|
|
|
Split this into a separate service later when one of these becomes true:
|
|
|
|
- third-party API consumers need independent uptime, auth, quotas, billing, or SLA boundaries;
|
|
- export jobs need independent workers, storage, or scaling;
|
|
- canonical data publication needs immutable release management independent of the editing workbench;
|
|
- commercial/public API concerns start slowing down internal QA and import workflows.
|
|
|
|
The public/API product should not expose raw workbench tables directly. It should consume versioned canonical snapshots.
|
|
|
|
The journey/routing interface should consume the active harmonized transit snapshot. It should not expose raw GTFS feed selection as a normal traveller-facing routing control. Feed-specific filters remain useful for QA, layer inspection, diagnostics, and source review.
|
|
|
|
## Target Pipeline
|
|
|
|
```text
|
|
source catalog
|
|
-> raw feed snapshots
|
|
-> validation reports
|
|
-> normalized staging tables
|
|
-> canonical matching and deduplication
|
|
-> conflict review and reusable rules
|
|
-> versioned canonical snapshot
|
|
-> GTFS/API/GeoParquet exports
|
|
```
|
|
|
|
The pan-European output should be a canonical mobility dataset first, not one giant internal GTFS feed. GTFS should be one export format from that canonical snapshot.
|
|
|
|
## Core Concepts
|
|
|
|
### Layered Source Model
|
|
|
|
The target model has four distinct layers:
|
|
|
|
1. **Source directories / discovery indexes**: places where sources are found, such as Mobility Database, PTNA, OpenData ÖPNV, NAP registries, Transitland, or manually curated seed files.
|
|
2. **Sources / publishers / authorities**: the organization or authority that owns or publishes data, such as DELFI, VBB, RNV, a regional transport authority, or an operator.
|
|
3. **Feeds**: concrete machine-readable products from a source, such as a static GTFS ZIP, NeTEx package, GTFS-RT TripUpdates endpoint, GTFS-RT ServiceAlerts endpoint, or operator-specific API endpoint.
|
|
4. **Feed snapshots / datasets**: immutable downloaded/imported versions of a feed with hash, timestamp, parser version, validation report, activation status, and imported rows.
|
|
|
|
The current database does not fully separate these layers yet:
|
|
|
|
- `source_catalog_entries` is a lightweight discovery/backlog catalog.
|
|
- `sources` currently represents a registered importable feed, even though the name is broader.
|
|
- `datasets` represents imported feed snapshots.
|
|
- generated discovery files under `docs/generated/` contain many more feed candidates than are currently promoted into the database review workflow.
|
|
|
|
Because of this, the discovery catalog should be much larger than GTFS Feed QA. Feed QA should only show registered GTFS feeds that have been selected for review/import, not every source directory or every discovered candidate row.
|
|
|
|
### Source Registry
|
|
|
|
Track every identified source, including feeds not yet imported:
|
|
|
|
- source URL and publisher;
|
|
- country/region/mode coverage;
|
|
- source authority and priority;
|
|
- update cadence and freshness;
|
|
- importability;
|
|
- license and redistribution status.
|
|
|
|
Mobility Database can be used as a broad discovery connector. Prefer the full `feeds_v2.csv` catalog/API over validator acceptance-test feed lists because it includes feed status, official/source flags, latest/direct URLs, license URLs, features, and bounding boxes. Treat it as candidate metadata: the catalog metadata is reusable, but each transit feed still needs its own provider licence review and authority ranking.
|
|
|
|
PTNA can be used as a GTFS/OSM QA and crosswalk connector. Its country pages expose feed IDs, provider names, release dates, validity windows, route-analysis links, detail pages, and original release-page links. Detail pages can add license text, OSM permission notes, `network:guid`, and route matching hints. PTNA should not become the canonical publisher for a feed; the harmonizer should follow the original provider URL where possible and keep PTNA as evidence.
|
|
|
|
OpenData ÖPNV is a Germany-focused source-registry connector. It lists DELFI national GTFS/NeTEx/ZHV datasets and many regional/operator datasets. Some downloads, including the Germany-wide DELFI GTFS on the portal, require registration, so those rows must stay in review until credentials and license decisions are recorded.
|
|
|
|
The generated discovery files live under `docs/generated/`:
|
|
|
|
- `gtfs_feed_candidates.csv` keeps every discovered feed/evidence row.
|
|
- `gtfs_ingestable_sources.csv` keeps rows that can be imported as GTFS sources after review.
|
|
- `gtfs_test_run_sources.csv` keeps a smaller multi-source set for deduplication tests.
|
|
|
|
Required license flags before publication:
|
|
|
|
- `can_import`
|
|
- `can_derive`
|
|
- `can_redistribute`
|
|
- `requires_attribution`
|
|
- `commercial_restrictions`
|
|
|
|
### Source Hierarchy
|
|
|
|
The harmonizer should not import every overlapping GTFS source into one active canonical snapshot. Each discovered source gets hierarchy metadata:
|
|
|
|
- `national_official` + `national` -> canonical baseline candidate.
|
|
- `regional_authority` -> QA or override candidate for a bounded region, not additive by default when a national baseline already covers the same services.
|
|
- `operator_feed` -> operator detail candidate, useful for QA, freshest local data, or replacing a bounded subset.
|
|
- `mirror` -> bootstrap only unless the original source is unavailable and license review allows use.
|
|
- `secondary_discovery` -> evidence metadata only.
|
|
|
|
Default import policy:
|
|
|
|
- Use one baseline per coverage universe.
|
|
- Add regional/operator feeds only as QA inputs or explicit replacements for a defined subset.
|
|
- Keep mirrors and aggregators out of canonical publication unless they are the only legally usable source and the provenance is explicit.
|
|
- Record every override decision so later imports can replay the same source precedence.
|
|
|
|
### Raw Snapshots
|
|
|
|
Every update should preserve immutable raw input:
|
|
|
|
- source id;
|
|
- fetch time;
|
|
- source hash;
|
|
- upstream metadata;
|
|
- parser/import version;
|
|
- validator report;
|
|
- previous active snapshot.
|
|
|
|
This keeps deduplication and conflict decisions reproducible.
|
|
|
|
### Update and Diff Policy
|
|
|
|
Source updates should not reset reviewed work. A new feed or map import should create a new raw snapshot, compute entity-level diffs, replay existing matching rules, and open review items only for changed, new, conflicting, or no-longer-valid evidence.
|
|
|
|
Initial diff units:
|
|
|
|
- feed metadata, hash, freshness, and service horizon;
|
|
- agencies/operators/networks;
|
|
- stops/platforms/station complexes;
|
|
- routes/lines;
|
|
- route patterns and stop sequences;
|
|
- trips, calendars, and stop times;
|
|
- route geometry evidence from GTFS shapes and OSM relations;
|
|
- OSM stop/platform/route relation candidates.
|
|
|
|
Unchanged raw rows may still be stored in the immutable snapshot tables for auditability. The expensive derived work should be incremental: canonical links, route geometry, route-layer rows, and QA review queues should rebuild only where the diff says something changed.
|
|
|
|
### Canonical Entities
|
|
|
|
Stable meubility IDs should be the internal truth. Source IDs remain aliases.
|
|
|
|
Initial canonical entity families:
|
|
|
|
- operators/agencies/authorities/networks;
|
|
- stop places and station complexes;
|
|
- platforms, tracks, bus bays, entrances;
|
|
- routes/lines;
|
|
- route patterns and trip patterns;
|
|
- calendars/service validity;
|
|
- shapes/geometries;
|
|
- fares/ticketing references later.
|
|
|
|
### GTFS to Map Matching Workbench
|
|
|
|
The workbench should turn raw source evidence into reviewed, reusable map/timetable links:
|
|
|
|
- match GTFS stops to concrete OSM stop/platform/bay nodes or ways, preserving side-of-street and platform detail;
|
|
- link nearby but distinct stops with explicit transfer edges instead of merging them into one physical stop;
|
|
- match GTFS routes and route patterns to OSM route relations where the evidence is strong;
|
|
- when OSM route relations are missing or weak, propose stop-to-stop paths over mode-compatible OSM infrastructure;
|
|
- keep GTFS shapes as geometry evidence, especially for detours, temporary routes, and missing OSM relations;
|
|
- allow reviewer decisions such as accept OSM relation, use GTFS shape, use generated stop-to-stop map path, ignore visual geometry, split variant, mark detour, or create manual override;
|
|
- persist every decision as a reusable rule keyed by canonical entity, source scope, upstream IDs, normalized route keys, geometry hashes, and validity window.
|
|
|
|
Candidate generation must be constrained before expensive comparison:
|
|
|
|
- compare only OSM route features for route matching;
|
|
- filter by mode, network/operator/ref/name evidence, and source authority;
|
|
- use PostGIS bbox and distance filters before loading geometry into Python;
|
|
- prefer route-pattern and stop-sequence signatures over whole-feed scans;
|
|
- rerun candidate generation only for new or changed routes, stops, or map features.
|
|
|
|
### Authority Ranking
|
|
|
|
Conflict resolution needs explicit source authority:
|
|
|
|
- manual review decision;
|
|
- national official feed or registry;
|
|
- regional authority feed;
|
|
- operator feed;
|
|
- broad aggregator feed;
|
|
- OSM as visual/gap evidence, not timetable authority.
|
|
|
|
Authority can differ by entity type. A source can be authoritative for timetable but weak for route geometry or operator identity.
|
|
|
|
### Conflict Review
|
|
|
|
The QA dashboard should expose review queues for:
|
|
|
|
- duplicate operators/agencies;
|
|
- duplicate stop places/station complexes;
|
|
- GTFS stops without canonical links;
|
|
- OSM stops without GTFS/canonical links;
|
|
- canonical stop groups with large spatial disagreement;
|
|
- routes with missing, weak, or conflicting OSM links;
|
|
- routes with missing shapes or route-pattern geometry;
|
|
- map-matching proposals that need reviewer approval;
|
|
- update diffs that invalidate a previous manual match;
|
|
- stale calendars and short service horizons;
|
|
- license/redistribution blockers.
|
|
|
|
Manual resolutions must become reusable rules so source updates do not reintroduce the same conflict.
|
|
|
|
## Export Strategy
|
|
|
|
Do not start with one giant Europe GTFS zip as the only product. Produce:
|
|
|
|
- versioned canonical snapshot tables;
|
|
- country/region GTFS exports;
|
|
- network/operator GTFS exports;
|
|
- full-Europe analytical dumps such as GeoParquet;
|
|
- API-ready entity endpoints later.
|
|
|
|
Each export needs:
|
|
|
|
- snapshot id;
|
|
- source feed versions;
|
|
- generation time;
|
|
- validation summary;
|
|
- license/attribution manifest;
|
|
- conflict/review status.
|
|
|
|
## Current Implementation Step
|
|
|
|
The current implementation is a lightweight harmonization boundary:
|
|
|
|
- `/api/qa/summary`;
|
|
- source discovery metrics;
|
|
- import health metrics;
|
|
- GTFS validation counters;
|
|
- canonical stop/link coverage;
|
|
- route matching and geometry counters;
|
|
- publication-readiness warnings.
|
|
- GTFS source add/import/review controls live in the `GTFS Harmonization` sidebar module.
|
|
- Feed review decisions store review status, note, source authority ranking, and explicit publication flags:
|
|
can import, can derive, can redistribute, attribution required, and commercial restrictions.
|
|
- OSM/route-layer source controls live in the `Mapping Data` sidebar module.
|
|
- The journey panel displays the active harmonized transit snapshot instead of a GTFS source picker.
|
|
- The current active harmonized snapshot is computed from active GTFS datasets, review authority, source priority, and route-key overlap; lower-precedence feeds that are mostly covered by a higher-precedence feed are shadowed for default routing while remaining available for QA/source-specific inspection.
|
|
|
|
This is intentionally still a skeleton. The next implementation step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions, then run the generated 24-source test set through review, import, validation, and deduplication.
|