Sync Repo-MVP-ROADMAP from project files
@@ -0,0 +1,264 @@
|
|||||||
|
<!-- codex-wiki-sync:7387d96c1a541f1bbd448e71 -->
|
||||||
|
|
||||||
|
> Mirrored from `/mnt/DATA/git/meubility-workbench/MVP_ROADMAP.md`.
|
||||||
|
> Origin: `repository`.
|
||||||
|
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
|
||||||
|
|
||||||
|
---
|
||||||
|
# MVP roadmap
|
||||||
|
|
||||||
|
Last updated: 2026-07-02
|
||||||
|
|
||||||
|
See also `docs/backlog.md` for the prioritized engineering backlog, caveats, and open optimization list.
|
||||||
|
|
||||||
|
## Objective
|
||||||
|
|
||||||
|
Build an internal management workbench that turns public mobility data into a normalized, auditable, coverage-scored dataset for a future traveller-facing web/native app.
|
||||||
|
|
||||||
|
The workbench stays distinct from the public app. Its users are data engineers, analysts, and operations staff who need to ingest, inspect, link, correct, route against, and publish mobility data.
|
||||||
|
|
||||||
|
## Current prototype: implemented
|
||||||
|
|
||||||
|
The repository has moved beyond the original SQLite/Berlin prototype. The current development path is Germany-scale and PostGIS-first, while SQLite remains useful as a legacy/test fallback.
|
||||||
|
|
||||||
|
Implemented:
|
||||||
|
|
||||||
|
```text
|
||||||
|
source registry and source catalog
|
||||||
|
local source cache
|
||||||
|
job queue with job events and worker process
|
||||||
|
PostgreSQL/PostGIS runtime support with SQLite fallback
|
||||||
|
GTFS static importer for large national feeds
|
||||||
|
OSM PBF import path for Germany-scale extracts
|
||||||
|
OSM address index and address-aware journey endpoints
|
||||||
|
canonical stop/station linking from GTFS and OSM
|
||||||
|
automatic GTFS <-> OSM route matching
|
||||||
|
manual route and canonical-stop rule persistence
|
||||||
|
visual route-layer builder from OSM routes and GTFS shapes
|
||||||
|
walk/drive routing layer from OSM-derived routing graph
|
||||||
|
progressive journey-search API and UI polling
|
||||||
|
map right-click "from here" / "to here"
|
||||||
|
management UI with map, sources, stats, jobs, matches, search, and journeys
|
||||||
|
separate GTFS Harmonization and Mapping Data source modules in the UI
|
||||||
|
generic job-details overlay with phase timeline, event log, and queue snapshot
|
||||||
|
QA dashboard skeleton for source/import/link/route/publication health
|
||||||
|
GTFS harmonization concept and service-boundary decision
|
||||||
|
discovery catalog, registered feeds, and imported datasets represented in the UI as separate prototype panels
|
||||||
|
CLI commands
|
||||||
|
tests and syntax checks for changed modules
|
||||||
|
```
|
||||||
|
|
||||||
|
Recent fixes:
|
||||||
|
|
||||||
|
```text
|
||||||
|
PostgreSQL startup avoids unnecessary DDL when PostGIS columns/indexes already exist.
|
||||||
|
Queue route-layer rebuild can be claimed by a real worker instead of staying queued behind a stale worker pid.
|
||||||
|
Timetable routing no longer requires visual route-pattern trip links.
|
||||||
|
Walk-leg route geometry has a short-lived in-process cache.
|
||||||
|
Address search is bbox-aware without being bbox-limited.
|
||||||
|
Job rows expose a details overlay that polls job events only while open.
|
||||||
|
Journey routing consumes the active harmonized GTFS snapshot instead of a raw feed picker.
|
||||||
|
GTFS Harmonization feed review records structured license and authority decisions.
|
||||||
|
Active harmonized GTFS snapshot selection is currently computed from active datasets, authority/priority, and route-key overlap; persisted versioned canonical snapshot tables are still outstanding.
|
||||||
|
Walk-only journey options reuse routed OSM walking geometry instead of straight-line display geometry.
|
||||||
|
```
|
||||||
|
|
||||||
|
## Current prototype: known limits
|
||||||
|
|
||||||
|
The app can import and inspect Germany-scale OSM and GTFS, but the routing and route-layer rebuild paths are still prototype-grade.
|
||||||
|
|
||||||
|
Important limits:
|
||||||
|
|
||||||
|
```text
|
||||||
|
journey search is not yet RAPTOR/CSA or connection-scan based
|
||||||
|
address endpoints can multiply transit searches through several nearby access/egress stops
|
||||||
|
progressive transfer stages still recompute too much
|
||||||
|
route-layer rebuild is coarse-grained and rewrites derived link tables
|
||||||
|
visual route-pattern links are not yet incrementally updated
|
||||||
|
canonical stop extraction is CPU/memory heavy on national feeds
|
||||||
|
route geometry cannot yet classify temporary GTFS detours as separate variants
|
||||||
|
local-transport-only routing is not a first-class query mode
|
||||||
|
route-search caches are process-local and not persisted
|
||||||
|
Alembic migrations are still missing
|
||||||
|
source discovery directories, publishers/authorities, concrete feeds, and immutable feed snapshots are not yet separate first-class tables
|
||||||
|
GTFS harmonization still lives in the map sidebar and should move to a dedicated workbench view as review queues grow
|
||||||
|
manual GTFS/OSM stop and route decisions are not yet replayed as stable rules across feed/map updates
|
||||||
|
source updates do not yet generate focused entity-level review diffs
|
||||||
|
```
|
||||||
|
|
||||||
|
## MVP 1: stable Germany data workbench
|
||||||
|
|
||||||
|
### Backend
|
||||||
|
|
||||||
|
- Add proper Alembic migrations for PostgreSQL and keep SQLite test support.
|
||||||
|
- Promote GTFS harmonization to the primary data workflow: source discovery, review, import, validation, deduplication, and publication readiness.
|
||||||
|
- Build the feed review workflow: discovered feed -> license/authority review -> approve -> import.
|
||||||
|
- Add the first-class source hierarchy: source directories -> sources/publishers/authorities -> concrete feeds -> immutable feed snapshots/datasets.
|
||||||
|
- Store explicit license/publication flags per feed: can import, can derive, can redistribute, attribution required, and commercial restrictions.
|
||||||
|
- Add immutable raw feed snapshots and source-run history.
|
||||||
|
- Add validation report storage per feed snapshot.
|
||||||
|
- Add first dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
|
||||||
|
- Add persistent GTFS/OSM stop and route matching decisions that survive source updates.
|
||||||
|
- Add feed/map update diffs so only changed stops, routes, patterns, calendars, and geometry evidence need review.
|
||||||
|
- Add constrained candidate generation for route matching: OSM route features only, mode/ref/name filters, PostGIS bbox/spatial pruning, and geometry loading only for plausible candidates.
|
||||||
|
- Add constrained candidate generation for stop matching: concrete OSM stop/platform/bay candidates near each GTFS stop, with explicit transfer links for nearby but distinct stops.
|
||||||
|
- Add generated stop-to-stop map-path proposals for missing OSM route relations and GTFS detours.
|
||||||
|
- Produce an active harmonized transit snapshot that routing consumes.
|
||||||
|
- Run the generated 24-source GTFS test set and measure conflicts before scaling to more Europe feeds.
|
||||||
|
- Add source-run history and dataset-version comparison.
|
||||||
|
- Make route-layer rebuild incremental: update only affected matches/patterns/stops.
|
||||||
|
- Keep old route-layer tables readable while a rebuild prepares replacement rows.
|
||||||
|
- Add source health checks: download success, hash change, feed freshness, calendar validity.
|
||||||
|
- Expand the QA dashboard into drill-down review queues for source health, GTFS validation, canonical stop conflicts, route conflicts, and publication blockers.
|
||||||
|
- Add GTFS validation summary reports: service dates, route direction coverage, stop coordinate outliers, bad stop_times, missing shapes.
|
||||||
|
- Add database maintenance jobs: analyze, vacuum, stale job recovery, orphan cleanup.
|
||||||
|
- Add durable cache tables for journey stages, nearest stops, address access candidates, and common station-to-station searches.
|
||||||
|
|
||||||
|
### Routing
|
||||||
|
|
||||||
|
- Replace the demo round-expansion router with a GTFS-appropriate algorithm such as RAPTOR or CSA.
|
||||||
|
- Precompute transfer graph edges: station-internal transfers, nearby walking transfers, and access/egress stop candidates.
|
||||||
|
- Add routing profiles:
|
||||||
|
|
||||||
|
```text
|
||||||
|
fastest public transport
|
||||||
|
fewest transfers
|
||||||
|
local transport only / Deutschlandticket-like
|
||||||
|
walk only
|
||||||
|
drive
|
||||||
|
car comparison
|
||||||
|
```
|
||||||
|
|
||||||
|
- Treat access/egress walking as access legs, not as public-transport transfers.
|
||||||
|
- Add bounded hub-aware long-distance routing for city-to-city requests: local access to likely hubs, long-distance/regional trunk, local egress.
|
||||||
|
- Add arrive-by search and better stop conditions for "good enough" results.
|
||||||
|
- Add route diagnostics that explain why a route was found or pruned.
|
||||||
|
|
||||||
|
### Frontend
|
||||||
|
|
||||||
|
- Add source detail page.
|
||||||
|
- Add dataset detail page.
|
||||||
|
- Add match-review queue with filters by mode, operator, country, confidence, and source scope.
|
||||||
|
- Add route detail inspection: GTFS geometry, OSM geometry, candidate matches, stops, evidence, and route-pattern provenance.
|
||||||
|
- Add canonical stop/station detail overlay.
|
||||||
|
- Add searchable route/stop review queues with filters for source, mode, operator, region, status, confidence, changed-since-last-run, missing geometry, and detour candidates.
|
||||||
|
- Add map inspection that overlays competing GTFS shapes, OSM route relations, generated path proposals, concrete stop candidates, and previous reviewer decisions.
|
||||||
|
- Add persistent rule editor.
|
||||||
|
- Add routing controls for profile, transfer buffer, avoid/prefer modes, arrive-by, via, and local-only.
|
||||||
|
- Show partial/progressive route results with clear stage labels.
|
||||||
|
|
||||||
|
### Data outputs
|
||||||
|
|
||||||
|
- GeoJSON exports for small regions.
|
||||||
|
- GeoParquet exports for analysis.
|
||||||
|
- PMTiles/vector-tile export for map display.
|
||||||
|
- Coverage CSV/API for downstream services.
|
||||||
|
|
||||||
|
## MVP 2: Europe-scale coverage map
|
||||||
|
|
||||||
|
- Use Geofabrik country/Europe extracts and reproducible OSM PBF jobs.
|
||||||
|
- Store OSM transport features, addresses, and routing graph in PostGIS.
|
||||||
|
- Generate ranked/generalized transport route layers by zoom level.
|
||||||
|
- Serve tiles with Martin or export PMTiles.
|
||||||
|
- Add coverage statuses:
|
||||||
|
|
||||||
|
```text
|
||||||
|
existing_in_osm
|
||||||
|
static_timetable_covered
|
||||||
|
live_data_covered
|
||||||
|
fare_data_covered
|
||||||
|
booking_covered
|
||||||
|
missing_static
|
||||||
|
stale_feed
|
||||||
|
restricted_license
|
||||||
|
low_confidence_match
|
||||||
|
detour_or_temporary_variant
|
||||||
|
```
|
||||||
|
|
||||||
|
- Add coverage metrics:
|
||||||
|
|
||||||
|
```text
|
||||||
|
operator coverage
|
||||||
|
route coverage
|
||||||
|
route-km coverage
|
||||||
|
stop coverage
|
||||||
|
live-data coverage
|
||||||
|
feed freshness
|
||||||
|
license confidence
|
||||||
|
booking coverage
|
||||||
|
route-layer provenance coverage
|
||||||
|
```
|
||||||
|
|
||||||
|
## MVP 3: more source formats
|
||||||
|
|
||||||
|
Add importers:
|
||||||
|
|
||||||
|
```text
|
||||||
|
NeTEx
|
||||||
|
TransXChange
|
||||||
|
SIRI discovery/live endpoints
|
||||||
|
GTFS-Realtime
|
||||||
|
GBFS for shared mobility, optional
|
||||||
|
operator CSV/API adapters
|
||||||
|
```
|
||||||
|
|
||||||
|
Target data model:
|
||||||
|
|
||||||
|
```text
|
||||||
|
canonical operators
|
||||||
|
canonical stops/stations/terminals
|
||||||
|
canonical routes
|
||||||
|
route variants
|
||||||
|
trip patterns
|
||||||
|
calendar/service validity
|
||||||
|
transfers
|
||||||
|
access/egress legs
|
||||||
|
coverage observations
|
||||||
|
source evidence
|
||||||
|
manual rules
|
||||||
|
```
|
||||||
|
|
||||||
|
## MVP 4: production journey-planning dataset
|
||||||
|
|
||||||
|
- Build a canonical stop/station graph with transfer rules and transfer-time profiles.
|
||||||
|
- Generate timetable-routing input for RAPTOR/CSA.
|
||||||
|
- Add first/last-mile routing from OSM walk/drive graph.
|
||||||
|
- Add emissions factors per mode/operator/country.
|
||||||
|
- Add fare/ticket placeholders and booking/deep-link metadata.
|
||||||
|
- Add confidence and provenance to every derived route/journey.
|
||||||
|
|
||||||
|
## MVP 5: booking-readiness layer
|
||||||
|
|
||||||
|
- Track booking availability separately from timetable coverage.
|
||||||
|
- Add deep-link metadata per operator/route.
|
||||||
|
- Add partner API adapters later.
|
||||||
|
- Distinguish clearly:
|
||||||
|
|
||||||
|
```text
|
||||||
|
travel-plausible itinerary
|
||||||
|
bookable itinerary
|
||||||
|
single-interface multi-booking
|
||||||
|
protected through-ticket
|
||||||
|
```
|
||||||
|
|
||||||
|
## Recommended next implementation sprint
|
||||||
|
|
||||||
|
Primary product/data track: GTFS Harmonization.
|
||||||
|
|
||||||
|
1. Turn discovered GTFS feeds into a review queue with license, redistribution, authority, freshness, and coverage fields.
|
||||||
|
2. Approve/import the generated 24-source GTFS test set and keep immutable raw feed snapshots.
|
||||||
|
3. Persist source-run history and validation reports for every feed snapshot.
|
||||||
|
4. Add entity-level feed diffs and matching-rule replay so updates do not reset reviewed work.
|
||||||
|
5. Add GTFS/OSM stop and route review queues with map inspection.
|
||||||
|
6. Add dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
|
||||||
|
7. Generate an active harmonized transit snapshot and make route search consume that snapshot by default.
|
||||||
|
8. Measure conflicts and authority decisions from the test set before scaling to broader Europe coverage.
|
||||||
|
|
||||||
|
Parallel technical track:
|
||||||
|
|
||||||
|
1. Add route-search diagnostics and timing instrumentation.
|
||||||
|
2. Implement a precomputed transfer graph and cleaner access/egress transfer-budget model.
|
||||||
|
3. Add durable PostgreSQL-backed route-search caches.
|
||||||
|
4. Convert route-layer rebuild to shadow/versioned rows or incremental updates.
|
||||||
|
5. Add Alembic migrations and remove routine schema checks from normal app/worker startup.
|
||||||
|
6. Add vector-tile output for large route layers.
|
||||||
Reference in New Issue
Block a user