add-ideas/meubility-workbench
1
0
Fork 0
Code Issues 122 Pull Requests Actions Packages Projects Releases Wiki Activity
Page: Repo-MVP-ROADMAP
Pages
Codex-Project-Index Repo-MVP-ROADMAP Repo-README Repo-docs-backlog Repo-docs-gtfs-harmonization Repo-docs-ingestable-sources-seed Repo-docs-product-vision Repo-docs-source-acquisition Repo-docs-source-catalog-seed
Clone
2
Repo-MVP-ROADMAP
zemion edited this page 2026-07-06 20:38:28 +02:00
Table of Contents

Mirrored from /mnt/DATA/git/meubility-workbench/MVP_ROADMAP.md. Origin: repository. Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.

MVP roadmap

Last updated: 2026-07-02

See also docs/product_vision.md for the long-term EU-wide platform vision and docs/backlog.md for the prioritized engineering backlog, caveats, and open optimization list.

Objective

Build an internal management workbench that turns public mobility data into a normalized, auditable, coverage-scored dataset for a future EU-wide mobility platform.

The workbench stays distinct from the public app. Its users are data engineers, analysts, and operations staff who need to ingest, inspect, link, correct, route against, and publish mobility data.

The broader platform direction is documented in docs/product_vision.md: harmonized mobility data, map/routing visualization, traveller-facing comparison by price, emissions, duration, changes, and reliability, booking-readiness where possible, and institutional analytics for communities, municipalities, states, governments, companies, and event planners.

Current prototype: implemented

The repository has moved beyond the original SQLite/Berlin prototype. The current development path is Germany-scale and PostGIS-first, while SQLite remains useful as a legacy/test fallback.

Implemented:

source registry and source catalog
local source cache
job queue with job events and worker process
PostgreSQL/PostGIS runtime support with SQLite fallback
GTFS static importer for large national feeds
OSM PBF import path for Germany-scale extracts
OSM address index and address-aware journey endpoints
canonical stop/station linking from GTFS and OSM
automatic GTFS <-> OSM route matching
manual route and canonical-stop rule persistence
visual route-layer builder from OSM routes and GTFS shapes
walk/drive routing layer from OSM-derived routing graph
bike route mode from the OSM-derived routing graph
progressive journey-search API and UI polling
map right-click "from here" / "to here"
management UI with map, sources, stats, jobs, matches, search, and journeys
separate GTFS Harmonization and Mapping Data source modules in the UI
generic job-details overlay with phase timeline, event log, and queue snapshot
QA dashboard skeleton for source/import/link/route/publication health
GTFS harmonization concept and service-boundary decision
discovery catalog, registered feeds, and imported datasets represented in the UI as separate prototype panels
CLI commands
tests and syntax checks for changed modules

Recent fixes:

PostgreSQL startup avoids unnecessary DDL when PostGIS columns/indexes already exist.
Queue route-layer rebuild can be claimed by a real worker instead of staying queued behind a stale worker pid.
Timetable routing no longer requires visual route-pattern trip links.
Walk-leg route geometry has a short-lived in-process cache.
Address search is bbox-aware without being bbox-limited.
Job rows expose a details overlay that polls job events only while open.
Journey routing consumes the active harmonized GTFS snapshot instead of a raw feed picker.
GTFS Harmonization feed review records structured license and authority decisions.
Active harmonized GTFS snapshot selection is currently computed from active datasets, authority/priority, and route-key overlap; persisted versioned canonical snapshot tables are still outstanding.
Walk-only journey options reuse routed OSM walking geometry instead of straight-line display geometry.

Current prototype: known limits

The app can import and inspect Germany-scale OSM and GTFS, but the routing and route-layer rebuild paths are still prototype-grade.

Important limits:

journey search is not yet RAPTOR/CSA or connection-scan based
address endpoints can multiply transit searches through several nearby access/egress stops
progressive transfer stages still recompute too much
route-layer rebuild is coarse-grained and rewrites derived link tables
visual route-pattern links are not yet incrementally updated
canonical stop extraction is CPU/memory heavy on national feeds
route geometry cannot yet classify temporary GTFS detours as separate variants
local-transport-only routing is not a first-class query mode
route-search caches are process-local and not persisted
Alembic migrations are still missing
source discovery directories, publishers/authorities, concrete feeds, and immutable feed snapshots are not yet separate first-class tables
GTFS harmonization still lives in the map sidebar and should move to a dedicated workbench view as review queues grow
manual GTFS/OSM stop and route decisions are not yet replayed as stable rules across feed/map updates
source updates do not yet generate focused entity-level review diffs

MVP 1: stable Germany data workbench

Backend

  • Add proper Alembic migrations for PostgreSQL and keep SQLite test support.
  • Promote GTFS harmonization to the primary data workflow: source discovery, review, import, validation, deduplication, and publication readiness.
  • Build the feed review workflow: discovered feed -> license/authority review -> approve -> import.
  • Add the first-class source hierarchy: source directories -> sources/publishers/authorities -> concrete feeds -> immutable feed snapshots/datasets.
  • Store explicit license/publication flags per feed: can import, can derive, can redistribute, attribution required, and commercial restrictions.
  • Add immutable raw feed snapshots and source-run history.
  • Add validation report storage per feed snapshot.
  • Add first dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
  • Add persistent GTFS/OSM stop and route matching decisions that survive source updates.
  • Add feed/map update diffs so only changed stops, routes, patterns, calendars, and geometry evidence need review.
  • Add constrained candidate generation for route matching: OSM route features only, mode/ref/name filters, PostGIS bbox/spatial pruning, and geometry loading only for plausible candidates.
  • Add constrained candidate generation for stop matching: concrete OSM stop/platform/bay candidates near each GTFS stop, with explicit transfer links for nearby but distinct stops.
  • Add generated stop-to-stop map-path proposals for missing OSM route relations and GTFS detours.
  • Produce an active harmonized transit snapshot that routing consumes.
  • Run the generated 24-source GTFS test set and measure conflicts before scaling to more Europe feeds.
  • Add source-run history and dataset-version comparison.
  • Make route-layer rebuild incremental: update only affected matches/patterns/stops.
  • Keep old route-layer tables readable while a rebuild prepares replacement rows.
  • Add source health checks: download success, hash change, feed freshness, calendar validity.
  • Expand the QA dashboard into drill-down review queues for source health, GTFS validation, canonical stop conflicts, route conflicts, and publication blockers.
  • Add GTFS validation summary reports: service dates, route direction coverage, stop coordinate outliers, bad stop_times, missing shapes.
  • Add database maintenance jobs: analyze, vacuum, stale job recovery, orphan cleanup.
  • Add durable cache tables for journey stages, nearest stops, address access candidates, and common station-to-station searches.

Routing

  • Replace the demo round-expansion router with a GTFS-appropriate algorithm such as RAPTOR or CSA.
  • Precompute transfer graph edges: station-internal transfers, nearby walking transfers, and access/egress stop candidates.
  • Add routing profiles:
fastest public transport
fewest transfers
local transport only / Deutschlandticket-like
walk only
bike
drive
car comparison
  • Treat access/egress walking as access legs, not as public-transport transfers.
  • Add bounded hub-aware long-distance routing for city-to-city requests: local access to likely hubs, long-distance/regional trunk, local egress.
  • Add arrive-by search and better stop conditions for "good enough" results.
  • Add route diagnostics that explain why a route was found or pruned.

Frontend

  • Add source detail page.
  • Add dataset detail page.
  • Add match-review queue with filters by mode, operator, country, confidence, and source scope.
  • Add route detail inspection: GTFS geometry, OSM geometry, candidate matches, stops, evidence, and route-pattern provenance.
  • Add canonical stop/station detail overlay.
  • Add searchable route/stop review queues with filters for source, mode, operator, region, status, confidence, changed-since-last-run, missing geometry, and detour candidates.
  • Add map inspection that overlays competing GTFS shapes, OSM route relations, generated path proposals, concrete stop candidates, and previous reviewer decisions.
  • Add persistent rule editor.
  • Add routing controls for profile, transfer buffer, avoid/prefer modes, arrive-by, via, and local-only.
  • Show partial/progressive route results with clear stage labels.

Data outputs

  • GeoJSON exports for small regions.
  • GeoParquet exports for analysis.
  • PMTiles/vector-tile export for map display.
  • Coverage CSV/API for downstream services.

MVP 2: Europe-scale coverage map

  • Use Geofabrik country/Europe extracts and reproducible OSM PBF jobs.
  • Store OSM transport features, addresses, and routing graph in PostGIS.
  • Generate ranked/generalized transport route layers by zoom level.
  • Serve tiles with Martin or export PMTiles.
  • Add coverage statuses:
existing_in_osm
static_timetable_covered
live_data_covered
fare_data_covered
booking_covered
missing_static
stale_feed
restricted_license
low_confidence_match
detour_or_temporary_variant
  • Add coverage metrics:
operator coverage
route coverage
route-km coverage
stop coverage
live-data coverage
feed freshness
license confidence
booking coverage
route-layer provenance coverage

MVP 3: more source formats

Add importers:

NeTEx
TransXChange
SIRI discovery/live endpoints
GTFS-Realtime
GBFS for shared mobility, optional
operator CSV/API adapters

Target data model:

canonical operators
canonical stops/stations/terminals
canonical routes
route variants
trip patterns
calendar/service validity
transfers
access/egress legs
coverage observations
source evidence
manual rules

MVP 4: production journey-planning dataset

  • Build a canonical stop/station graph with transfer rules and transfer-time profiles.
  • Generate timetable-routing input for RAPTOR/CSA.
  • Add first/last-mile routing from OSM walk/drive graph.
  • Add emissions factors per mode/operator/country.
  • Add fare/ticket placeholders and booking/deep-link metadata.
  • Add reliability prognosis and missed-connection simulation inputs.
  • Add confidence and provenance to every derived route/journey.

MVP 5: booking-readiness layer

  • Track booking availability separately from timetable coverage.
  • Add deep-link metadata per operator/route.
  • Add partner API adapters later.
  • Distinguish clearly:
travel-plausible itinerary
bookable itinerary
single-interface multi-booking
protected through-ticket

Recommended next implementation sprint

Primary product/data track: GTFS Harmonization.

  1. Turn discovered GTFS feeds into a review queue with license, redistribution, authority, freshness, and coverage fields.
  2. Approve/import the generated 24-source GTFS test set and keep immutable raw feed snapshots.
  3. Persist source-run history and validation reports for every feed snapshot.
  4. Add entity-level feed diffs and matching-rule replay so updates do not reset reviewed work.
  5. Add GTFS/OSM stop and route review queues with map inspection.
  6. Add dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
  7. Generate an active harmonized transit snapshot and make route search consume that snapshot by default.
  8. Measure conflicts and authority decisions from the test set before scaling to broader Europe coverage.

Parallel technical track:

  1. Add route-search diagnostics and timing instrumentation.
  2. Implement a precomputed transfer graph and cleaner access/egress transfer-budget model.
  3. Add durable PostgreSQL-backed route-search caches.
  4. Convert route-layer rebuild to shadow/versioned rows or incremental updates.
  5. Add Alembic migrations and remove routine schema checks from normal app/worker startup.
  6. Add vector-tile output for large route layers.