add-ideas/meubility-workbench
1
0
Fork 0
Code Issues 122 Pull Requests Actions Packages Projects Releases Wiki Activity

Next alpha stage commit

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-06 20:54:46 +02:00
parent e23387738b
commit 1d7ec956d2
42 changed files with 11603 additions and 2837 deletions
Download Patch File Download Diff File Expand all files Collapse all files

85
docs/gtfs_harmonization.md
View File

@@ -1,6 +1,6 @@
# GTFS Harmonization and QA Concept
Last updated: 2026-07-01
Last updated: 2026-07-02
## Decision
@@ -41,6 +41,24 @@ The pan-European output should be a canonical mobility dataset first, not one gi
## Core Concepts
### Layered Source Model
The target model has four distinct layers:
1. **Source directories / discovery indexes**: places where sources are found, such as Mobility Database, PTNA, OpenData ÖPNV, NAP registries, Transitland, or manually curated seed files.
2. **Sources / publishers / authorities**: the organization or authority that owns or publishes data, such as DELFI, VBB, RNV, a regional transport authority, or an operator.
3. **Feeds**: concrete machine-readable products from a source, such as a static GTFS ZIP, NeTEx package, GTFS-RT TripUpdates endpoint, GTFS-RT ServiceAlerts endpoint, or operator-specific API endpoint.
4. **Feed snapshots / datasets**: immutable downloaded/imported versions of a feed with hash, timestamp, parser version, validation report, activation status, and imported rows.
The current database does not fully separate these layers yet:
- `source_catalog_entries` is a lightweight discovery/backlog catalog.
- `sources` currently represents a registered importable feed, even though the name is broader.
- `datasets` represents imported feed snapshots.
- generated discovery files under `docs/generated/` contain many more feed candidates than are currently promoted into the database review workflow.
Because of this, the discovery catalog should be much larger than GTFS Feed QA. Feed QA should only show registered GTFS feeds that have been selected for review/import, not every source directory or every discovered candidate row.
### Source Registry
Track every identified source, including feeds not yet imported:
@@ -56,6 +74,8 @@ Mobility Database can be used as a broad discovery connector. Prefer the full `f
PTNA can be used as a GTFS/OSM QA and crosswalk connector. Its country pages expose feed IDs, provider names, release dates, validity windows, route-analysis links, detail pages, and original release-page links. Detail pages can add license text, OSM permission notes, `network:guid`, and route matching hints. PTNA should not become the canonical publisher for a feed; the harmonizer should follow the original provider URL where possible and keep PTNA as evidence.
OpenData ÖPNV is a Germany-focused source-registry connector. It lists DELFI national GTFS/NeTEx/ZHV datasets and many regional/operator datasets. Some downloads, including the Germany-wide DELFI GTFS on the portal, require registration, so those rows must stay in review until credentials and license decisions are recorded.
The generated discovery files live under `docs/generated/`:
- `gtfs_feed_candidates.csv` keeps every discovered feed/evidence row.
@@ -70,6 +90,23 @@ Required license flags before publication:
- `requires_attribution`
- `commercial_restrictions`
### Source Hierarchy
The harmonizer should not import every overlapping GTFS source into one active canonical snapshot. Each discovered source gets hierarchy metadata:
- `national_official` + `national` -> canonical baseline candidate.
- `regional_authority` -> QA or override candidate for a bounded region, not additive by default when a national baseline already covers the same services.
- `operator_feed` -> operator detail candidate, useful for QA, freshest local data, or replacing a bounded subset.
- `mirror` -> bootstrap only unless the original source is unavailable and license review allows use.
- `secondary_discovery` -> evidence metadata only.
Default import policy:
- Use one baseline per coverage universe.
- Add regional/operator feeds only as QA inputs or explicit replacements for a defined subset.
- Keep mirrors and aggregators out of canonical publication unless they are the only legally usable source and the provenance is explicit.
- Record every override decision so later imports can replay the same source precedence.
### Raw Snapshots
Every update should preserve immutable raw input:
@@ -84,6 +121,23 @@ Every update should preserve immutable raw input:
This keeps deduplication and conflict decisions reproducible.
### Update and Diff Policy
Source updates should not reset reviewed work. A new feed or map import should create a new raw snapshot, compute entity-level diffs, replay existing matching rules, and open review items only for changed, new, conflicting, or no-longer-valid evidence.
Initial diff units:
- feed metadata, hash, freshness, and service horizon;
- agencies/operators/networks;
- stops/platforms/station complexes;
- routes/lines;
- route patterns and stop sequences;
- trips, calendars, and stop times;
- route geometry evidence from GTFS shapes and OSM relations;
- OSM stop/platform/route relation candidates.
Unchanged raw rows may still be stored in the immutable snapshot tables for auditability. The expensive derived work should be incremental: canonical links, route geometry, route-layer rows, and QA review queues should rebuild only where the diff says something changed.
### Canonical Entities
Stable meubility IDs should be the internal truth. Source IDs remain aliases.
@@ -99,6 +153,26 @@ Initial canonical entity families:
- shapes/geometries;
- fares/ticketing references later.
### GTFS to Map Matching Workbench
The workbench should turn raw source evidence into reviewed, reusable map/timetable links:
- match GTFS stops to concrete OSM stop/platform/bay nodes or ways, preserving side-of-street and platform detail;
- link nearby but distinct stops with explicit transfer edges instead of merging them into one physical stop;
- match GTFS routes and route patterns to OSM route relations where the evidence is strong;
- when OSM route relations are missing or weak, propose stop-to-stop paths over mode-compatible OSM infrastructure;
- keep GTFS shapes as geometry evidence, especially for detours, temporary routes, and missing OSM relations;
- allow reviewer decisions such as accept OSM relation, use GTFS shape, use generated stop-to-stop map path, ignore visual geometry, split variant, mark detour, or create manual override;
- persist every decision as a reusable rule keyed by canonical entity, source scope, upstream IDs, normalized route keys, geometry hashes, and validity window.
Candidate generation must be constrained before expensive comparison:
- compare only OSM route features for route matching;
- filter by mode, network/operator/ref/name evidence, and source authority;
- use PostGIS bbox and distance filters before loading geometry into Python;
- prefer route-pattern and stop-sequence signatures over whole-feed scans;
- rerun candidate generation only for new or changed routes, stops, or map features.
### Authority Ranking
Conflict resolution needs explicit source authority:
@@ -123,6 +197,8 @@ The QA dashboard should expose review queues for:
- canonical stop groups with large spatial disagreement;
- routes with missing, weak, or conflicting OSM links;
- routes with missing shapes or route-pattern geometry;
- map-matching proposals that need reviewer approval;
- update diffs that invalidate a previous manual match;
- stale calendars and short service horizons;
- license/redistribution blockers.
@@ -149,7 +225,7 @@ Each export needs:
## Current Implementation Step
The first implementation is a lightweight harmonization boundary:
The current implementation is a lightweight harmonization boundary:
- `/api/qa/summary`;
- source discovery metrics;
@@ -159,7 +235,10 @@ The first implementation is a lightweight harmonization boundary:
- route matching and geometry counters;
- publication-readiness warnings.
- GTFS source add/import/review controls live in the `GTFS Harmonization` sidebar module.
- Feed review decisions store review status, note, source authority ranking, and explicit publication flags:
can import, can derive, can redistribute, attribution required, and commercial restrictions.
- OSM/route-layer source controls live in the `Mapping Data` sidebar module.
- The journey panel displays the active harmonized transit snapshot instead of a GTFS source picker.
- The current active harmonized snapshot is computed from active GTFS datasets, review authority, source priority, and route-key overlap; lower-precedence feeds that are mostly covered by a higher-precedence feed are shadowed for default routing while remaining available for QA/source-specific inspection.
This is intentionally a skeleton. The next step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions.
This is intentionally still a skeleton. The next implementation step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions, then run the generated 24-source test set through review, import, validation, and deduplication.