Next alpha stage commit
This commit is contained in:
202
docs/backlog.md
202
docs/backlog.md
@@ -1,202 +0,0 @@
|
||||
# Product and Engineering Backlog
|
||||
|
||||
Last updated: 2026-07-01
|
||||
|
||||
This backlog reflects the current Germany-scale PostGIS prototype. The target remains a Europe-scale mobility data workbench that builds canonical stops, stations, routes, route geometry, timetable links, transfer rules, routing graph data, address search, and coverage evidence from many public sources.
|
||||
|
||||
OSM-derived geometry is the preferred visual authority. GTFS, NeTEx, realtime, and official APIs are timetable, validation, routing, and gap-detection inputs. GTFS shapes are still valuable evidence, especially for missing OSM relations and temporary detours.
|
||||
|
||||
## Current State
|
||||
|
||||
- PostgreSQL/PostGIS is the active development database path; SQLite remains a legacy/test fallback.
|
||||
- Germany OSM and Germany GTFS/DELFI-scale imports are supported.
|
||||
- OSM address indexing is available and address search is bbox-aware without being bbox-limited.
|
||||
- Jobs and job events exist for imports, route matching, route-layer rebuilds, address indexing, relabeling, deletes, and maintenance.
|
||||
- Job rows expose a generic details overlay with planned/current/done phases, event log, metadata, and a compact queue snapshot.
|
||||
- A first QA dashboard skeleton exists for source discovery, import health, GTFS validation, canonical stop/link coverage, route matching, and publication readiness.
|
||||
- The GTFS harmonization target architecture is documented in `docs/gtfs_harmonization.md`.
|
||||
- GTFS source management is presented as a separate `GTFS Harmonization` UI module; OSM/map inputs are presented as a separate `Mapping Data` module.
|
||||
- Journey search consumes the active harmonized transit snapshot instead of exposing a raw GTFS source selector.
|
||||
- Route-layer rebuild runs through the queue, but it is still coarse-grained and can take minutes on national datasets.
|
||||
- The route-layer builder links canonical GTFS stops, OSM stops, OSM route relations, GTFS route patterns, and trip-pattern links.
|
||||
- Journey search is progressive and can publish intermediate results, but the underlying routing algorithm is still a prototype.
|
||||
- Walk and drive routing use the OSM-derived routing layer when available.
|
||||
|
||||
## Current Caveats
|
||||
|
||||
- Journey search is not yet a full RAPTOR/CSA-style router.
|
||||
- Address endpoints can multiply the search space: current behavior can use up to 4 access stops and 4 egress stops, creating up to 16 transit stop-pair searches per transfer stage.
|
||||
- Progressive stages still recompute too much. Searching `up to 2 transfers` repeats direct and one-transfer work before deeper expansion.
|
||||
- Walking access/egress legs are represented separately in journey output, but the search engine still needs a cleaner transfer budget model where access/egress walking never consumes public-transport transfer count.
|
||||
- Route-search caches are in-process only. They do not survive server restart, do not deduplicate identical searches already running in another thread/process, and only help once a stage/search has completed.
|
||||
- Route-layer rebuild currently clears/rebuilds derived tables. Until the rebuild completes, visual route-pattern link tables can be incomplete.
|
||||
- Timetable reachability should not depend on visual route-pattern links. The code has been patched in this checkout, but a running server must reload before using that fix.
|
||||
- Canonical stop extraction on national feeds is CPU/memory heavy and does too much Python-side grouping.
|
||||
- OSM stop-linking and OSM route-candidate indexing are still large spatial/batch operations.
|
||||
- GTFS detours are not classified as first-class route variants yet.
|
||||
- Local-transport-only routing is not a first-class profile yet.
|
||||
- Proper Alembic migrations are still missing; runtime schema maintenance should be reduced to an explicit migration/maintenance path.
|
||||
- The source and job database tables are still shared between harmonization, mapping, and routing; the current split is a product/UI boundary, not a separate service or database boundary yet.
|
||||
|
||||
## P0: Routing Performance and Correctness
|
||||
|
||||
These items directly address slow or failed searches such as `Berlin, Alexanderplatz` to `Heidelberg, Blumenstrasse 36`.
|
||||
|
||||
- Replace the demo round-expansion router with a timetable-native algorithm.
|
||||
Preferred direction: RAPTOR or CSA over preloaded arrays/tables, with rounds representing public-transport boardings rather than ad hoc SQL expansion.
|
||||
- Precompute a transfer graph.
|
||||
Store station-internal transfers, nearby walking transfers, platform/stop-place links, and allowed transfer times by mode/source/station.
|
||||
- Separate access/egress from transfer count.
|
||||
Walking from an address to the first stop, and from the last stop to an address, should never count as a vehicle transfer.
|
||||
- Add a durable journey cache.
|
||||
Cache normalized requests, address-to-stop candidates, stop-to-stop stage results, common station-pair results, and in-flight request deduplication in PostgreSQL.
|
||||
- Add hub-aware long-distance routing.
|
||||
For long-distance OD pairs, search local access to likely hubs, trunk rail/regional candidates, then local egress. Candidate hubs can be ranked by station importance, service frequency, route scope, distance, and direction.
|
||||
- Add a local-transport-only profile.
|
||||
Implement a Deutschlandticket-like profile that excludes long-distance route scopes and still supports regional rail, S-Bahn, subway, tram, bus, ferry, and walking transfers.
|
||||
- Add admissible pruning.
|
||||
Bound exploration by best known arrival, remaining distance, direction/off-course penalty, transfer budget, service frequency, and maximum tolerated detour.
|
||||
- Add journey diagnostics.
|
||||
Return searched stages, candidate counts, pruned reasons, access/egress stops, service date, source feeds, transfer stops, and whether no-route means no timetable path or a search limit was hit.
|
||||
- Add arrive-by search.
|
||||
This is important for route quality and for comparing against operator/DB route planners.
|
||||
- Add route profile controls in the UI.
|
||||
`fastest`, `earliest arrival`, `fewest transfers`, `local only`, `walk`, `drive`, `arrive by`, `via`, `avoid`, and transfer buffer controls.
|
||||
|
||||
## P0: Queue and Rebuild Robustness
|
||||
|
||||
- Move runtime schema maintenance out of normal app startup.
|
||||
The current checkout avoids redundant PostgreSQL DDL, but explicit migrations are still needed.
|
||||
- Add Alembic migrations.
|
||||
Use migrations for PostGIS columns, indexes, route-layer tables, routing tables, and cache tables.
|
||||
- Make route-layer rebuild use shadow tables or versioned rows.
|
||||
Build replacement rows without deleting the readable active layer first; atomically promote the new version when complete.
|
||||
- Make route-layer rebuild incremental.
|
||||
Rebuild only affected route patterns after new matches, stop-link decisions, source updates, or OSM diffs.
|
||||
- Add stale worker and stale pid reconciliation.
|
||||
Worker status should never report a pid as running unless the current server can verify it.
|
||||
- Improve cancellation.
|
||||
Long PostgreSQL statements need cancellable phases and visible progress rather than only a queued/running state.
|
||||
- Improve progress granularity and timings.
|
||||
The UI can display job events now, but long PostgreSQL statements still need finer checkpoints, elapsed times, estimated remaining work, and cancellable sub-phases.
|
||||
|
||||
## P1: Route Layer, Detours, and Geometry Provenance
|
||||
|
||||
- Classify GTFS route variants.
|
||||
Group trips by route, direction, shape, stop sequence, service date span, and trip frequency. Mark rare/temporary shapes as detours or temporary variants rather than replacing the canonical visual route.
|
||||
- Add stop-by-stop OSM path fallback.
|
||||
When an OSM route relation is missing or a GTFS shape is a detour, assemble geometry between matched consecutive stops using mode-constrained OSM paths.
|
||||
- Cache stop-to-stop route geometry.
|
||||
Key by mode, from canonical stop, to canonical stop, direction constraints, and graph version.
|
||||
- Store geometry provenance per route pattern.
|
||||
Examples: `osm_route_relation`, `gtfs_shape`, `stop_to_stop_osm_path`, `manual_override`, `detour_variant`.
|
||||
- Respect directionality.
|
||||
Bus/car paths need oneway handling; tram/rail paths need topology and direction evidence; reverse links must not be assumed valid.
|
||||
- Add route-pattern detail inspection.
|
||||
Show OSM geometry, GTFS shapes, linked trips, linked stops, direction evidence, confidence, and variant/detour status.
|
||||
- Add generalized route geometries.
|
||||
Store high-detail inspection geometry and simplified map geometry.
|
||||
|
||||
## P1: Canonical Stops, Stations, and Addresses
|
||||
|
||||
- Optimize canonical stop extraction.
|
||||
Push more grouping/linking into SQL, avoid loading all scheduled stops into Python, batch inserts, and keep stable canonical IDs when possible.
|
||||
- Build a canonical stop alias table.
|
||||
Persist normalized names, multilingual names, station codes, IBNR/EVA/UIC/IFOPT, stop_area IDs, OSM IDs, and source-specific aliases.
|
||||
- Improve station-complex modeling.
|
||||
Separate public stop place, station complex, platforms/tracks, entrances, bus bays, and nearby stop groups.
|
||||
- Add canonical stop detail overlay.
|
||||
Show linked GTFS stops, linked OSM stops/stations, source names, confidence, distances, and manual overrides.
|
||||
- Add manual canonical stop link/unlink decisions.
|
||||
Persist stop matching decisions like route matching decisions, so source updates do not overwrite reviewed links.
|
||||
- Improve address result folding.
|
||||
Prefer street-level suggestions for dense house-number ranges, but preserve exact address selection when a full address is typed.
|
||||
- Precompute address access candidates.
|
||||
Store nearest useful public-transport stops per address/street point, with mode/source/radius metadata.
|
||||
|
||||
## P1: More GTFS Sources and Deduplication
|
||||
|
||||
- Import more GTFS feeds where they improve authority or coverage.
|
||||
DB long-distance/regional feeds, state feeds, and neighboring-country feeds touching Germany are useful test cases.
|
||||
- Add source priority and authority ranking.
|
||||
Decide which source is more authoritative for stops, operators, routes, calendars, and geometry evidence.
|
||||
- Deduplicate operators/agencies.
|
||||
Merge agency/operator records with provenance and aliases instead of treating each GTFS `agency.txt` row as a separate operator.
|
||||
- Turn QA summary counters into review queues.
|
||||
Drill down from each bad/warn metric into concrete sources, stops, routes, links, and conflicts.
|
||||
- Add GTFS feed QA reports.
|
||||
Calendar coverage, stale feeds, missing shapes, impossible stop times, duplicate routes, route direction coverage, stop coordinate outliers.
|
||||
- Add conflict dashboards and reusable resolution workflows.
|
||||
Show canonical stops/routes with competing source claims, weak matches, missing visual geometry, authority-rule conflicts, and license blockers.
|
||||
|
||||
## P1: Scalable OSM and Map Outputs
|
||||
|
||||
- Keep OSM PBF import chunked and resumable.
|
||||
Keep previous active visual datasets available while the next import builds.
|
||||
- Add vector tile or PMTiles export.
|
||||
Needed for Germany/Europe route layers and dense editing views.
|
||||
- Add route-scope and mode-specific map generalization.
|
||||
Different zooms should use different detail levels and route classes.
|
||||
- Improve OSM route candidate indexing.
|
||||
Use stronger SQL/PostGIS filtering before loading route geometry into Python.
|
||||
- Add OSM diffs later.
|
||||
Minutely/hourly/daily diffs can update route and address layers without full country rebuilds.
|
||||
|
||||
## P2: Data Platform Hardening
|
||||
|
||||
- Add explicit read/write transaction boundaries for all long requests and jobs.
|
||||
- Add API pagination for large result sets.
|
||||
- Add import logs and source-run history.
|
||||
- Add database maintenance commands: analyze, vacuum, reindex, orphan cleanup.
|
||||
- Add test fixtures that do not mutate the live development database.
|
||||
- Add observability: query timings, job timings, row counts, cache hit rates, and per-stage routing metrics.
|
||||
|
||||
## P2: Better Map and Editing Workflows
|
||||
|
||||
- Add canonical stop and route detail side panels.
|
||||
- Add candidate map preview for stop matching, not only route matching.
|
||||
- Add unmatched/matched/weak/proposed visual layers with source filters.
|
||||
- Keep calculated journey geometry and stop markers always on top.
|
||||
- Add editable match queues for stops, station complexes, routes, and operators.
|
||||
- Add route-layer diff view after rebuilds.
|
||||
|
||||
## P3: Additional Formats and Live Data
|
||||
|
||||
- Add NeTEx import.
|
||||
- Add GTFS-Realtime ingestion for service alerts and trip updates.
|
||||
- Add SIRI profile support where national APIs expose it.
|
||||
- Add GBFS/shared mobility only after core public transport data is stable.
|
||||
- Model temporary closures and disruptions as validity-windowed events, not modifications to base route geometry.
|
||||
|
||||
## Open Optimization List
|
||||
|
||||
Not yet implemented, or only partially implemented:
|
||||
|
||||
- RAPTOR/CSA routing core.
|
||||
- Precomputed public-transport transfer graph.
|
||||
- Durable PostgreSQL route-search cache.
|
||||
- In-flight identical search coalescing.
|
||||
- Hub-aware long-distance routing.
|
||||
- Local-transport-only routing profile.
|
||||
- Access/egress legs excluded from transfer budget at the search-state level.
|
||||
- Better pruning for off-course exploration and dominated labels.
|
||||
- SQL/array-based canonical stop extraction.
|
||||
- Incremental route-layer rebuild.
|
||||
- Route-layer shadow tables/versioned activation.
|
||||
- Stop-to-stop OSM route fallback for missing routes and detours.
|
||||
- Detour/temporary variant classification.
|
||||
- PostGIS-first OSM route candidate filtering.
|
||||
- Vector tiles or PMTiles for large route layers.
|
||||
- Alembic migrations.
|
||||
- Persistent query/stage timing diagnostics.
|
||||
|
||||
## Recommended Next Sprint
|
||||
|
||||
1. Finish the route-layer rebuild currently in progress and verify route-pattern/trip-pattern link counts.
|
||||
2. Restart/reload the server so it picks up the current checkout fixes.
|
||||
3. Add route-search diagnostics and timing instrumentation around address access, direct, one-transfer, and round-search stages.
|
||||
4. Implement transfer graph precomputation and exclude access/egress walking from transfer count.
|
||||
5. Add a hub-aware city-to-city search path for long-distance requests.
|
||||
6. Add a local-only routing profile using route scopes.
|
||||
7. Convert route-layer rebuild to shadow/versioned tables or incremental updates.
|
||||
8. Add Alembic migrations and stop doing routine schema checks during normal app/worker startup.
|
||||
@@ -1,5 +1,5 @@
|
||||
{
|
||||
"generated_at": "2026-07-01T13:22:51.928761+00:00",
|
||||
"generated_at": "2026-07-02T10:38:44.832065+00:00",
|
||||
"countries": [
|
||||
"DE",
|
||||
"AT",
|
||||
@@ -18,27 +18,30 @@
|
||||
"sources": {
|
||||
"mobility_database": "https://files.mobilitydatabase.org/feeds_v2.csv",
|
||||
"mobility_acceptance_test_list": "https://raw.githubusercontent.com/MobilityData/gtfs-validator/master/scripts/mobility-database-harvester/acceptance_test_feed_list.csv",
|
||||
"ptna": "https://ptna.openstreetmap.de/gtfs/index.html"
|
||||
"ptna": "https://ptna.openstreetmap.de/gtfs/index.html",
|
||||
"opendata_oepnv": "https://www.opendata-oepnv.de/ht/de/datensaetze"
|
||||
},
|
||||
"counts": {
|
||||
"candidates": 1090,
|
||||
"ingestable": 990,
|
||||
"candidates": 1131,
|
||||
"ingestable": 998,
|
||||
"test_run": 24,
|
||||
"by_source": {
|
||||
"curated_seed": 9,
|
||||
"curated_seed; mobility_database": 2,
|
||||
"curated_seed; mobility_database; mobility_validator_acceptance": 2,
|
||||
"mobility_database": 805,
|
||||
"mobility_database; mobility_validator_acceptance": 149,
|
||||
"mobility_database; mobility_validator_acceptance": 148,
|
||||
"mobility_database; mobility_validator_acceptance; opendata_oepnv": 1,
|
||||
"mobility_database; mobility_validator_acceptance; ptna": 1,
|
||||
"mobility_validator_acceptance": 3,
|
||||
"opendata_oepnv": 41,
|
||||
"ptna": 119
|
||||
},
|
||||
"ingestable_by_country": {
|
||||
"AT": 6,
|
||||
"BE": 15,
|
||||
"CH": 10,
|
||||
"DE": 68,
|
||||
"DE": 76,
|
||||
"DK": 1,
|
||||
"EU": 1,
|
||||
"FI": 31,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -1,25 +1,25 @@
|
||||
name,kind,url,country,license,mode_scope,source_basis,priority,notes
|
||||
CH Swiss national GTFS,gtfs,https://gtfs.geops.ch/dl/gtfs_complete.zip,CH,verify at opentransportdata.swiss,"rail,tram,metro,bus,ferry",European transport feeds / official Swiss OTD derivative,P0,geOps feed is derived from official Swiss Open Transport Data; verify production terms.
|
||||
NL OpenOV national GTFS,gtfs,http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip,NL,verify OpenOV/NDOV terms,"rail,tram,metro,bus,ferry",European transport feeds / OpenOV,P0,Use NDOV/OVapi for production and realtime.
|
||||
NL OVapi GTFS,gtfs,http://gtfs.ovapi.nl/gtfs-nl.zip,NL,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip
|
||||
DK Rejseplanen GTFS,gtfs,https://www.rejseplanen.info/labs/GTFS.zip,DK,verify Rejseplanen Labs terms,"rail,bus",Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip
|
||||
FI Helsingin seudun liikenne (HSL) GTFS,gtfs,http://dev.hsl.fi/gtfs/hsl.zip,FI,see http://developer.reittiopas.fi/pages/en/home.php,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip
|
||||
NO Entur national aggregated GTFS,gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip,NO,verify Entur terms/NLOD,"rail,tram,metro,bus,ferry",Entur; Mobility Database feed catalog,P0,GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip
|
||||
IE Aircoach GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip
|
||||
"IE BK & Sons, JJ/Bernard Kavanagh GTFS",gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip,IE,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip
|
||||
IE Bus Éireann GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip
|
||||
GB BODS national GTFS,gtfs,https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/,GB,OGL/verify BODS terms,"rail,bus",BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip
|
||||
DE DB Long-distance Rail GTFS.DE,gtfs,https://download.gtfs.de/germany/fv_free/latest.zip,DE,Creative Commons 4.0,rail,GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog,P1,Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip
|
||||
DE Public Transport Germany GTFS,gtfs,https://download.gtfs.de/germany/nv_free/latest.zip,DE,see https://www.nvbw.de/open-data/lizenz,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1090/latest.zip
|
||||
DE Regional Rail Transport Germany GTFS,gtfs,https://download.gtfs.de/germany/rv_free/latest.zip,DE,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1089/latest.zip
|
||||
DE Verkehrsverbund Berlin-Brandenburg (VBB) GTFS,gtfs,http://vbb.de/vbbgtfs,DE,see http://vbb.de/vbbgtfs,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-782/latest.zip
|
||||
DE Verkehrsverbund Rhein-Neckar GTFS,gtfs,https://geoportal.vrn.de/services/sharing/rest/content/items/4ec4b1d131eb46a6bb8e216ce9b90eff/data,DE,see https://www.vrn.de/opendata/datasets/soll-fahrplandaten-gtfs-aktuell,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1173/latest.zip
|
||||
"AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS",gtfs,http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip
|
||||
NL OVapi GTFS,gtfs,https://gtfs.ovapi.nl/nl/gtfs-nl.zip,NL,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi,P2,PTNA candidate; use original publisher URL where available.
|
||||
FI Helsingin seudun liikenne GTFS,gtfs,https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip,FI,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL,P2,PTNA candidate; use original publisher URL where available.
|
||||
CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip
|
||||
CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip
|
||||
AT Bean Shuttle GTFS,gtfs,https://www.beanshuttle.com/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip
|
||||
AT Optima Express GTFS,gtfs,https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip,AT,,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip
|
||||
"FI Haarasilta Toivo Samuli, Järvisen Liikenne Oy, Koiviston Auto Oy, Lehtimäen Liikenne Oy, Bus Travel Oy Reissu Ruoti, Tilausliikenne Kuisma Ky GTFS",gtfs,https://tvv.fra1.digitaloceanspaces.com/223.zip,FI,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1129/latest.zip
|
||||
"NO Vestfold Kollektivtrafikk, Bastø Fosen GTFS",gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_vkt-aggregated-gtfs.zip,NO,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-1048/latest.zip
|
||||
CH Swiss national GTFS,gtfs,https://gtfs.geops.ch/dl/gtfs_complete.zip,CH,verify at opentransportdata.swiss,"rail,tram,metro,bus,ferry",European transport feeds / official Swiss OTD derivative,P0,geOps feed is derived from official Swiss Open Transport Data; verify production terms.; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
DE Rhein-Neckar-Verkehr GTFS,gtfs,https://gtfs-sandbox-dds.rnv-online.de/latest/gtfs.zip,DE,Creative Commons,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list; OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-rnv&cHash=6b4f1a9d1a4c981c2703f4446d234027; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details,P0,Selected Mobility Database latest.zip mirror because the catalog direct URL is known to be stale.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Selected non-mirror publisher URL from merged source evidence.; Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-777/latest.zip; Import policy: qa_or_override_not_additive; overlap role: operator_detail_candidate.
|
||||
NL OpenOV national GTFS,gtfs,http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip,NL,verify OpenOV/NDOV terms,"rail,tram,metro,bus,ferry",European transport feeds / OpenOV,P0,Use NDOV/OVapi for production and realtime.; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
NL OVapi GTFS,gtfs,http://gtfs.ovapi.nl/gtfs-nl.zip,NL,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
DK Rejseplanen GTFS,gtfs,https://www.rejseplanen.info/labs/GTFS.zip,DK,verify Rejseplanen Labs terms,"rail,bus",Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
FI Helsingin seudun liikenne (HSL) GTFS,gtfs,http://dev.hsl.fi/gtfs/hsl.zip,FI,see http://developer.reittiopas.fi/pages/en/home.php,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
NO Entur national aggregated GTFS,gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip,NO,verify Entur terms/NLOD,"rail,tram,metro,bus,ferry",Entur; Mobility Database feed catalog,P0,GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
IE Aircoach GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
"IE BK & Sons, JJ/Bernard Kavanagh GTFS",gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip,IE,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
IE Bus Éireann GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
GB BODS national GTFS,gtfs,https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/,GB,OGL/verify BODS terms,"rail,bus",BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
DE Aachener Verkehrsverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/dede57c7-9601-43cd-9fb6-8ab4c27412d2/resource/94744341-7116-402a-b0d5-9517d6c2060f/download/avv_gtfs_masten_mit_spnv_january_2020.zip,DE,Creative Commons Attribution,bus,OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-avv&cHash=b604ba50698ef0a26cd2a2685f168041; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
|
||||
DE DB Long-distance Rail GTFS.DE,gtfs,https://download.gtfs.de/germany/fv_free/latest.zip,DE,Creative Commons 4.0,rail,GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog,P1,Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
|
||||
DE Hamburger Verkehrsverbund GTFS,gtfs,https://opendata-oepnv.de/fileadmin/datasets/hvv/10-08-2017_Soll-Fahrplandaten.zip,DE,Creative Commons,"rail,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-hvv&cHash=559eb3ae637273e6cc2da67b34bc3c04; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
|
||||
DE Münchner Verkehrs- und Tarifverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/885dfcc6-571e-47e4-85dc-e95d900fb257/resource/8f4addd8-ba55-4dde-8937-90c880fc6da0/download/250922_mvvcombi_ohneshape_google_transit.zip,DE,Creative Commons Attribution,"rail,tram,metro,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_gesamt&cHash=b3e6bda4cc9e5656172f0b0d0c120a98; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahr,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
|
||||
DE Münchner Verkehrs- und Tarifverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/17065229-c3fd-46d7-84a9-aae55aadbf40/resource/879dd863-1abc-4f64-8b27-a8b88994572d/download/mvv_gtfs.zip,DE,Creative Commons Attribution,bus,OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_reg&cHash=f25f41feaa97b6e02cb9dfcc260dd4c7; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrpla,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
|
||||
DE Nordhessischer VerkehrsVerbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/4c35703c-62d2-463a-8f8d-014594ab6e30/resource/21c5b646-6f3f-461a-9493-4603357850eb/download/gtfs-20250704.zip,DE,Creative Commons,"rail,tram,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten_nvv&cHash=1fa9e9f4ab59d86496e5f03b934b6316; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
|
||||
"AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS",gtfs,http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
NL OVapi GTFS,gtfs,https://gtfs.ovapi.nl/nl/gtfs-nl.zip,NL,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi,P2,PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
|
||||
FI Helsingin seudun liikenne GTFS,gtfs,https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip,FI,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL,P2,PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
|
||||
CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
AT Bean Shuttle GTFS,gtfs,https://www.beanshuttle.com/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
AT Optima Express GTFS,gtfs,https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip,AT,,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
|
||||
|
||||
|
@@ -1,6 +1,6 @@
|
||||
# GTFS Harmonization and QA Concept
|
||||
|
||||
Last updated: 2026-07-01
|
||||
Last updated: 2026-07-02
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -41,6 +41,24 @@ The pan-European output should be a canonical mobility dataset first, not one gi
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Layered Source Model
|
||||
|
||||
The target model has four distinct layers:
|
||||
|
||||
1. **Source directories / discovery indexes**: places where sources are found, such as Mobility Database, PTNA, OpenData ÖPNV, NAP registries, Transitland, or manually curated seed files.
|
||||
2. **Sources / publishers / authorities**: the organization or authority that owns or publishes data, such as DELFI, VBB, RNV, a regional transport authority, or an operator.
|
||||
3. **Feeds**: concrete machine-readable products from a source, such as a static GTFS ZIP, NeTEx package, GTFS-RT TripUpdates endpoint, GTFS-RT ServiceAlerts endpoint, or operator-specific API endpoint.
|
||||
4. **Feed snapshots / datasets**: immutable downloaded/imported versions of a feed with hash, timestamp, parser version, validation report, activation status, and imported rows.
|
||||
|
||||
The current database does not fully separate these layers yet:
|
||||
|
||||
- `source_catalog_entries` is a lightweight discovery/backlog catalog.
|
||||
- `sources` currently represents a registered importable feed, even though the name is broader.
|
||||
- `datasets` represents imported feed snapshots.
|
||||
- generated discovery files under `docs/generated/` contain many more feed candidates than are currently promoted into the database review workflow.
|
||||
|
||||
Because of this, the discovery catalog should be much larger than GTFS Feed QA. Feed QA should only show registered GTFS feeds that have been selected for review/import, not every source directory or every discovered candidate row.
|
||||
|
||||
### Source Registry
|
||||
|
||||
Track every identified source, including feeds not yet imported:
|
||||
@@ -56,6 +74,8 @@ Mobility Database can be used as a broad discovery connector. Prefer the full `f
|
||||
|
||||
PTNA can be used as a GTFS/OSM QA and crosswalk connector. Its country pages expose feed IDs, provider names, release dates, validity windows, route-analysis links, detail pages, and original release-page links. Detail pages can add license text, OSM permission notes, `network:guid`, and route matching hints. PTNA should not become the canonical publisher for a feed; the harmonizer should follow the original provider URL where possible and keep PTNA as evidence.
|
||||
|
||||
OpenData ÖPNV is a Germany-focused source-registry connector. It lists DELFI national GTFS/NeTEx/ZHV datasets and many regional/operator datasets. Some downloads, including the Germany-wide DELFI GTFS on the portal, require registration, so those rows must stay in review until credentials and license decisions are recorded.
|
||||
|
||||
The generated discovery files live under `docs/generated/`:
|
||||
|
||||
- `gtfs_feed_candidates.csv` keeps every discovered feed/evidence row.
|
||||
@@ -70,6 +90,23 @@ Required license flags before publication:
|
||||
- `requires_attribution`
|
||||
- `commercial_restrictions`
|
||||
|
||||
### Source Hierarchy
|
||||
|
||||
The harmonizer should not import every overlapping GTFS source into one active canonical snapshot. Each discovered source gets hierarchy metadata:
|
||||
|
||||
- `national_official` + `national` -> canonical baseline candidate.
|
||||
- `regional_authority` -> QA or override candidate for a bounded region, not additive by default when a national baseline already covers the same services.
|
||||
- `operator_feed` -> operator detail candidate, useful for QA, freshest local data, or replacing a bounded subset.
|
||||
- `mirror` -> bootstrap only unless the original source is unavailable and license review allows use.
|
||||
- `secondary_discovery` -> evidence metadata only.
|
||||
|
||||
Default import policy:
|
||||
|
||||
- Use one baseline per coverage universe.
|
||||
- Add regional/operator feeds only as QA inputs or explicit replacements for a defined subset.
|
||||
- Keep mirrors and aggregators out of canonical publication unless they are the only legally usable source and the provenance is explicit.
|
||||
- Record every override decision so later imports can replay the same source precedence.
|
||||
|
||||
### Raw Snapshots
|
||||
|
||||
Every update should preserve immutable raw input:
|
||||
@@ -84,6 +121,23 @@ Every update should preserve immutable raw input:
|
||||
|
||||
This keeps deduplication and conflict decisions reproducible.
|
||||
|
||||
### Update and Diff Policy
|
||||
|
||||
Source updates should not reset reviewed work. A new feed or map import should create a new raw snapshot, compute entity-level diffs, replay existing matching rules, and open review items only for changed, new, conflicting, or no-longer-valid evidence.
|
||||
|
||||
Initial diff units:
|
||||
|
||||
- feed metadata, hash, freshness, and service horizon;
|
||||
- agencies/operators/networks;
|
||||
- stops/platforms/station complexes;
|
||||
- routes/lines;
|
||||
- route patterns and stop sequences;
|
||||
- trips, calendars, and stop times;
|
||||
- route geometry evidence from GTFS shapes and OSM relations;
|
||||
- OSM stop/platform/route relation candidates.
|
||||
|
||||
Unchanged raw rows may still be stored in the immutable snapshot tables for auditability. The expensive derived work should be incremental: canonical links, route geometry, route-layer rows, and QA review queues should rebuild only where the diff says something changed.
|
||||
|
||||
### Canonical Entities
|
||||
|
||||
Stable meubility IDs should be the internal truth. Source IDs remain aliases.
|
||||
@@ -99,6 +153,26 @@ Initial canonical entity families:
|
||||
- shapes/geometries;
|
||||
- fares/ticketing references later.
|
||||
|
||||
### GTFS to Map Matching Workbench
|
||||
|
||||
The workbench should turn raw source evidence into reviewed, reusable map/timetable links:
|
||||
|
||||
- match GTFS stops to concrete OSM stop/platform/bay nodes or ways, preserving side-of-street and platform detail;
|
||||
- link nearby but distinct stops with explicit transfer edges instead of merging them into one physical stop;
|
||||
- match GTFS routes and route patterns to OSM route relations where the evidence is strong;
|
||||
- when OSM route relations are missing or weak, propose stop-to-stop paths over mode-compatible OSM infrastructure;
|
||||
- keep GTFS shapes as geometry evidence, especially for detours, temporary routes, and missing OSM relations;
|
||||
- allow reviewer decisions such as accept OSM relation, use GTFS shape, use generated stop-to-stop map path, ignore visual geometry, split variant, mark detour, or create manual override;
|
||||
- persist every decision as a reusable rule keyed by canonical entity, source scope, upstream IDs, normalized route keys, geometry hashes, and validity window.
|
||||
|
||||
Candidate generation must be constrained before expensive comparison:
|
||||
|
||||
- compare only OSM route features for route matching;
|
||||
- filter by mode, network/operator/ref/name evidence, and source authority;
|
||||
- use PostGIS bbox and distance filters before loading geometry into Python;
|
||||
- prefer route-pattern and stop-sequence signatures over whole-feed scans;
|
||||
- rerun candidate generation only for new or changed routes, stops, or map features.
|
||||
|
||||
### Authority Ranking
|
||||
|
||||
Conflict resolution needs explicit source authority:
|
||||
@@ -123,6 +197,8 @@ The QA dashboard should expose review queues for:
|
||||
- canonical stop groups with large spatial disagreement;
|
||||
- routes with missing, weak, or conflicting OSM links;
|
||||
- routes with missing shapes or route-pattern geometry;
|
||||
- map-matching proposals that need reviewer approval;
|
||||
- update diffs that invalidate a previous manual match;
|
||||
- stale calendars and short service horizons;
|
||||
- license/redistribution blockers.
|
||||
|
||||
@@ -149,7 +225,7 @@ Each export needs:
|
||||
|
||||
## Current Implementation Step
|
||||
|
||||
The first implementation is a lightweight harmonization boundary:
|
||||
The current implementation is a lightweight harmonization boundary:
|
||||
|
||||
- `/api/qa/summary`;
|
||||
- source discovery metrics;
|
||||
@@ -159,7 +235,10 @@ The first implementation is a lightweight harmonization boundary:
|
||||
- route matching and geometry counters;
|
||||
- publication-readiness warnings.
|
||||
- GTFS source add/import/review controls live in the `GTFS Harmonization` sidebar module.
|
||||
- Feed review decisions store review status, note, source authority ranking, and explicit publication flags:
|
||||
can import, can derive, can redistribute, attribution required, and commercial restrictions.
|
||||
- OSM/route-layer source controls live in the `Mapping Data` sidebar module.
|
||||
- The journey panel displays the active harmonized transit snapshot instead of a GTFS source picker.
|
||||
- The current active harmonized snapshot is computed from active GTFS datasets, review authority, source priority, and route-key overlap; lower-precedence feeds that are mostly covered by a higher-precedence feed are shadowed for default routing while remaining available for QA/source-specific inspection.
|
||||
|
||||
This is intentionally a skeleton. The next step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions.
|
||||
This is intentionally still a skeleton. The next implementation step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions, then run the generated 24-source test set through review, import, validation, and deduplication.
|
||||
|
||||
88
docs/product_vision.md
Normal file
88
docs/product_vision.md
Normal file
@@ -0,0 +1,88 @@
|
||||
# Meubility Product Vision
|
||||
|
||||
Last updated: 2026-07-06
|
||||
|
||||
Meubility should become an EU-wide mobility platform that combines harmonized mobility data, clear visual representation, routing, comparison, and analysis tooling.
|
||||
|
||||
The platform should support both traveller-facing use cases and institutional planning use cases. The shared foundation is a canonical, auditable mobility dataset: sources are discovered, licensed, imported, validated, harmonized, matched to map evidence, versioned, and exposed through stable APIs and visual tools.
|
||||
|
||||
## Product Goal
|
||||
|
||||
The long-term product goal is to answer:
|
||||
|
||||
```text
|
||||
How can a person or organization get from A to B, what options exist, how do those options compare, how reliable are they, what do they cost, what emissions do they produce, and what can be booked or reserved?
|
||||
```
|
||||
|
||||
The answer should be explainable. Every route, price, emission estimate, reliability prognosis, and availability signal needs provenance and confidence metadata.
|
||||
|
||||
## Target Audiences
|
||||
|
||||
- General public travellers who need understandable route options across public transport, walking, biking, driving, and shared mobility.
|
||||
- Public institutions, municipalities, states, and governments analyzing network quality, coverage gaps, accessibility, emissions, car dependency, and improvement potential.
|
||||
- Companies planning employee travel, business travel, mobility budgets, or commuting alternatives.
|
||||
- Municipalities, venues, and event agencies offering "car-free travel to destination" guidance.
|
||||
- Data teams and transport planners who need a workbench for source review, harmonization, QA, and publication.
|
||||
- Third-party developers and researchers consuming harmonized mobility data through APIs.
|
||||
|
||||
## Mobility Coverage
|
||||
|
||||
The canonical data model should eventually cover all relevant mobility options:
|
||||
|
||||
- Public transport: rail, regional rail, subway, tram, bus, coach, ferry, cableways, and demand-responsive services where available.
|
||||
- Active mobility: walking and biking.
|
||||
- Shared mobility: rental bikes, e-scooters, car sharing, station-based rental cars, free-floating vehicles, and similar services.
|
||||
- Commercial individual transport: taxi, ride-hailing, chauffeur, rental car, and transfer services.
|
||||
- Private car comparison: routing, duration, cost, parking assumptions, and emissions.
|
||||
|
||||
Booking and reservation support should be treated as a separate layer from route plausibility. The platform can show a plausible itinerary before it can guarantee or sell the full connection. Where booking exists, it should be clear whether the result is a deep link, a partner booking, multiple independent bookings, or a protected through-ticket.
|
||||
|
||||
## Traveller Comparison
|
||||
|
||||
Routing results should compare options by:
|
||||
|
||||
- Duration and arrival/departure time.
|
||||
- Number of changes and transfer complexity.
|
||||
- Walking/biking distance and accessibility constraints.
|
||||
- Price or estimated cost.
|
||||
- Emissions.
|
||||
- Reliability prognosis, including connection risk and disruption sensitivity.
|
||||
- Booking or reservation availability.
|
||||
- Comfort and policy flags where available, such as local-transport-only, bike carriage, wheelchair accessibility, or company travel rules.
|
||||
|
||||
Missed-connection simulation should be a first-class analysis capability: if a transfer fails, the platform should estimate the likely delay, fallback options, booking consequences, and whether the route remains acceptable.
|
||||
|
||||
## Institutional Analytics
|
||||
|
||||
The same harmonized dataset should support analysis for public bodies and organizations:
|
||||
|
||||
- Coverage and accessibility by region, municipality, station, stop, venue, or event site.
|
||||
- Route and timetable gaps.
|
||||
- Missing map or timetable evidence.
|
||||
- Reliability weak points and fragile transfers.
|
||||
- Emissions and modal-shift potential.
|
||||
- Car-free access plans for events, public buildings, tourism, campuses, and large employers.
|
||||
- Source-quality reporting: freshness, license confidence, validation errors, route matching confidence, stop matching confidence, and coverage by authority level.
|
||||
|
||||
## Platform Architecture Direction
|
||||
|
||||
The product should be split into clear layers:
|
||||
|
||||
1. Source discovery and licensing.
|
||||
2. Raw source snapshots and update history.
|
||||
3. Validation and QA reports.
|
||||
4. Harmonization and deduplication.
|
||||
5. Map/GTFS matching workbench and persistent reviewer decisions.
|
||||
6. Active canonical mobility snapshots for routing and analysis.
|
||||
7. Routing, comparison, reliability, pricing, emissions, and booking-readiness services.
|
||||
8. Public UI, institutional dashboard, and third-party API surfaces.
|
||||
|
||||
GTFS harmonization remains the next critical sprint because high-quality routing, comparison, and analytics depend on stable canonical timetable data. Shared mobility, pricing, emissions, reliability, and booking layers should build on that canonical base rather than bypassing it.
|
||||
|
||||
## Near-Term Product Implications
|
||||
|
||||
- Keep prioritizing the GTFS harmonization and Map/GTFS workbench tracks.
|
||||
- Treat routing improvements as a parallel technical track, especially RAPTOR/CSA routing, durable caches, transfer graph precomputation, and route diagnostics.
|
||||
- Add cost, emissions, reliability, and booking-readiness as explicit future data layers in the roadmap.
|
||||
- Keep API readiness in view: harmonized GTFS and canonical mobility snapshots should become useful outputs even before the public traveller app is complete.
|
||||
- Keep institutional analytics in scope from the start, because the workbench already stores the provenance, confidence, and quality evidence that public bodies and planners need.
|
||||
@@ -4,7 +4,7 @@ This repository now contains two seed catalogues:
|
||||
|
||||
- `docs/source_catalog_seed.csv` — broad discovery catalogue for official NAPs, feed registries, route-geometry evidence, realtime/disruption sources, rail/air registries and country notes.
|
||||
- `docs/ingestable_sources_seed.csv` — direct static feeds that the current prototype can import immediately.
|
||||
- `docs/generated/gtfs_feed_candidates.csv` — generated GTFS discovery manifest from Mobility Database, PTNA, the validator acceptance list, and curated local seeds.
|
||||
- `docs/generated/gtfs_feed_candidates.csv` — generated GTFS discovery manifest from Mobility Database, PTNA, OpenData ÖPNV, the validator acceptance list, and curated local seeds.
|
||||
- `docs/generated/gtfs_ingestable_sources.csv` — generated direct GTFS source rows suitable for source-registry import after license/source review.
|
||||
- `docs/generated/gtfs_test_run_sources.csv` — generated focused feed set for the first multi-source harmonization/deduplication run.
|
||||
|
||||
@@ -14,7 +14,14 @@ Regenerate the GTFS discovery manifests:
|
||||
python -m app.cli discover-gtfs-sources --max-ptna-details 0 --test-limit 24
|
||||
```
|
||||
|
||||
Use `--countries ALL` for the broad global Mobility Database/acceptance-list pass. Use a positive `--max-ptna-details` when you want PTNA license and OSM crosswalk fields; the country-table scrape is fast, while detail pages can be slow.
|
||||
Use `--countries ALL` for the broad global Mobility Database/acceptance-list pass. Use a positive `--max-ptna-details` when you want PTNA license and OSM crosswalk fields; the country-table scrape is fast, while detail pages can be slow. OpenData ÖPNV detail pages are fetched for Germany by default and can be skipped with `--no-opendata-oepnv`.
|
||||
|
||||
The candidate manifest includes source hierarchy fields:
|
||||
|
||||
- `authority_level` ranks the publisher/evidence source, for example `national_official`, `regional_authority`, `operator_feed`, `mirror`, or `secondary_discovery`.
|
||||
- `coverage_scope` marks whether a feed is national, regional, operator-specific, multi-country, or unknown.
|
||||
- `overlap_role` explains how to treat overlaps, for example `canonical_baseline`, `regional_override_candidate`, `operator_detail_candidate`, `bootstrap_mirror`, or `discovery_evidence`.
|
||||
- `import_policy` is the practical guardrail. `qa_or_override_not_additive` means do not import the feed into the active canonical snapshot together with a broader baseline unless it is replacing a defined subset or being used for QA.
|
||||
|
||||
Import the direct feed seed list into the source registry:
|
||||
|
||||
|
||||
@@ -28,6 +28,7 @@ Finland,FI,"rail, bus, tram, metro, ferry, bike",Fintraffic FINAP / national GTF
|
||||
France,FR,"rail, bus, tram, metro, ferry, coach, air metadata",transport.data.gouv.fr,National NAP/catalog,"GTFS, NeTEx, GTFS-RT, SIRI; SSIM reference for air",public catalog; per-feed terms/auth,"Large official catalog for public transport, road, shared vehicles, carpooling, etc.; public transit datasets describe networks, stops, routes and times.",GTFS/NeTEx shapes/stops supersede OSM; consolidated stop datasets help stop registry.,GTFS-RT and SIRI feeds including SNCF service alerts/trip updates where available.,Operators/publishers from dataset metadata and agencies.,Licence per dataset.,P0,https://transport.data.gouv.fr/,https://transport.data.gouv.fr/datasets?type=public-transit&locale=en,Crawl catalog API/search; import SNCF and regional feeds; add RT after static.
|
||||
France,FR,national rail,SNCF Open Data,Operator feed,"GTFS, NeTEx, SIRI Lite, GTFS-RT TripUpdates/ServiceAlerts",public; terms per dataset,SNCF publishes static and realtime passenger information datasets.,SNCF GTFS/NeTEx route and stop data supersede OSM for timetable layer.,GTFS-RT TripUpdates and ServiceAlerts; SIRI ET/SX Lite.,SNCF agency/operator records.,Terms per data.gouv dataset.,P0,https://ressources.data.sncf.com/,https://transport.data.gouv.fr/datasets?organization=sncf&locale=en,Create SNCF connector; add stale-feed and preview-window checks.
|
||||
Germany,DE,"rail, bus, tram, metro, ferry if in ÖPNV",DELFI / Mobilithek national NeTEx + GTFS.de,National feed / derived GTFS,"NeTEx, GTFS; GTFS-RT aggregate",public; official static via NAP; GTFS derived,National static timetable data published via Mobilithek/DELFI; GTFS.de offers daily GTFS covering DB long-distance/regional and local/urban transit.,Official NeTEx / GTFS shapes supersede OSM for planned services.,GTFS.de RT stream aggregates realtime where open/licensed: TripUpdates/ServiceAlerts.,Agencies/operators from feed; NAP publishers.,Static open; RT may depend on open licences/special agreements.,P0,https://gtfs.de/en/,https://gtfs.de/en/,"Use GTFS.de for bootstrap, plan NeTEx ingestion for higher fidelity."
|
||||
Germany,DE,"rail, bus, tram, metro, ferry if in ÖPNV",OpenData ÖPNV Deutschland,National/regional open-data portal and source registry,"GTFS, NeTEx, GTFS-RT, CSV, GeoJSON, XML, JSON, APIs",public catalog; some downloads require registration,"Portal lists DELFI national datasets plus regional/operator datasets such as VBB, VRR, MVV, HVV, VVS, RNV, AVV, RMV, NVV and others.",GTFS/NeTEx/stops can supersede OSM for planned services only after source hierarchy and license review.,GTFS-RT/TRIAS/API datasets are listed for selected publishers; access varies by dataset.,Publishers and organizations can seed authority/operator review.,Per-dataset license; common cases include CC BY and Datenlizenz Deutschland; explicit import/derivation/redistribution review required.,P0,https://www.opendata-oepnv.de/ht/de/datensaetze,https://www.opendata-oepnv.de/ht/de/datensaetze,"Use discovery connector; treat DELFI/national feeds as baseline, regional/operator feeds as QA or override candidates to avoid duplicate imports."
|
||||
Germany / Berlin-Brandenburg,DE,"rail, bus, tram, metro, ferry",VBB Berlin-Brandenburg GTFS,Regional authority feed / official GTFS,GTFS,public; updated twice weekly,Official VBB timetable feed for Berlin and Brandenburg bus and rail services; useful regional bootstrap/demo source before national-scale ingestion.,GTFS stops/shapes/timetables supersede OSM for planned service verification where present.,Static timetable only in this row; VBB GTFS-RT is a separate realtime source.,Agencies/operators from agency.txt and VBB feed metadata.,CC-BY; attribution required: VBB Verkehrsverbund Berlin-Brandenburg GmbH.,P5,https://www.vbb.de/vbbgtfs,https://daten.berlin.de/datensaetze/vbb-fahrplandaten-via-gtfs,Use for Berlin model bootstrap; pair with Geofabrik Berlin OSM PBF and validate route geometry coverage.
|
||||
Germany,DE,rail disruptions,DB Baustellen / construction works,Operator disruption/planned works,web/API where available,public web; structured access variable,Major long-distance construction works and timetable changes; useful for planned closure enrichment.,"Can supersede OSM for temporary rail service restrictions, not geometry.","Planned works, timetable changes.",DB as operator/infrastructure-related source.,Terms to verify.,P2,https://bauinfos.deutschebahn.com/,https://bauinfos.deutschebahn.com/,Research structured endpoints; otherwise link as non-ingested evidence.
|
||||
Greece,GR,"metro, tram, bus, coach, ferry",Greece NAP + OASA telematics,NAP / operator portals,web/API; GTFS unknown/fragmented,public web; machine access uncertain,"Official NAP exists; Athens OASA publishes line schedules/stops and telematics app information, but open machine-readable coverage needs verification.",Official feeds if found supersede OSM; otherwise OSM remains strong existence layer.,Telematics app includes real-time arrivals/locations; open reuse unclear.,Operators from NAP/operator pages and OSM.,Likely access/licence work required.,P3,https://www.nap.gov.gr/,https://telematics.oasa.gr/,Investigate NAP catalog API and OASA endpoints/licence.
|
||||
|
||||
|
Reference in New Issue
Block a user