Sync Repo-MVP-ROADMAP from project files
@@ -0,0 +1,264 @@
|
||||
<!-- codex-wiki-sync:7387d96c1a541f1bbd448e71 -->
|
||||
|
||||
> Mirrored from `/mnt/DATA/git/meubility-workbench/MVP_ROADMAP.md`.
|
||||
> Origin: `repository`.
|
||||
> Active tasks and changing state belong in Gitea issues; this wiki page is durable project context.
|
||||
|
||||
---
|
||||
# MVP roadmap
|
||||
|
||||
Last updated: 2026-07-02
|
||||
|
||||
See also `docs/backlog.md` for the prioritized engineering backlog, caveats, and open optimization list.
|
||||
|
||||
## Objective
|
||||
|
||||
Build an internal management workbench that turns public mobility data into a normalized, auditable, coverage-scored dataset for a future traveller-facing web/native app.
|
||||
|
||||
The workbench stays distinct from the public app. Its users are data engineers, analysts, and operations staff who need to ingest, inspect, link, correct, route against, and publish mobility data.
|
||||
|
||||
## Current prototype: implemented
|
||||
|
||||
The repository has moved beyond the original SQLite/Berlin prototype. The current development path is Germany-scale and PostGIS-first, while SQLite remains useful as a legacy/test fallback.
|
||||
|
||||
Implemented:
|
||||
|
||||
```text
|
||||
source registry and source catalog
|
||||
local source cache
|
||||
job queue with job events and worker process
|
||||
PostgreSQL/PostGIS runtime support with SQLite fallback
|
||||
GTFS static importer for large national feeds
|
||||
OSM PBF import path for Germany-scale extracts
|
||||
OSM address index and address-aware journey endpoints
|
||||
canonical stop/station linking from GTFS and OSM
|
||||
automatic GTFS <-> OSM route matching
|
||||
manual route and canonical-stop rule persistence
|
||||
visual route-layer builder from OSM routes and GTFS shapes
|
||||
walk/drive routing layer from OSM-derived routing graph
|
||||
progressive journey-search API and UI polling
|
||||
map right-click "from here" / "to here"
|
||||
management UI with map, sources, stats, jobs, matches, search, and journeys
|
||||
separate GTFS Harmonization and Mapping Data source modules in the UI
|
||||
generic job-details overlay with phase timeline, event log, and queue snapshot
|
||||
QA dashboard skeleton for source/import/link/route/publication health
|
||||
GTFS harmonization concept and service-boundary decision
|
||||
discovery catalog, registered feeds, and imported datasets represented in the UI as separate prototype panels
|
||||
CLI commands
|
||||
tests and syntax checks for changed modules
|
||||
```
|
||||
|
||||
Recent fixes:
|
||||
|
||||
```text
|
||||
PostgreSQL startup avoids unnecessary DDL when PostGIS columns/indexes already exist.
|
||||
Queue route-layer rebuild can be claimed by a real worker instead of staying queued behind a stale worker pid.
|
||||
Timetable routing no longer requires visual route-pattern trip links.
|
||||
Walk-leg route geometry has a short-lived in-process cache.
|
||||
Address search is bbox-aware without being bbox-limited.
|
||||
Job rows expose a details overlay that polls job events only while open.
|
||||
Journey routing consumes the active harmonized GTFS snapshot instead of a raw feed picker.
|
||||
GTFS Harmonization feed review records structured license and authority decisions.
|
||||
Active harmonized GTFS snapshot selection is currently computed from active datasets, authority/priority, and route-key overlap; persisted versioned canonical snapshot tables are still outstanding.
|
||||
Walk-only journey options reuse routed OSM walking geometry instead of straight-line display geometry.
|
||||
```
|
||||
|
||||
## Current prototype: known limits
|
||||
|
||||
The app can import and inspect Germany-scale OSM and GTFS, but the routing and route-layer rebuild paths are still prototype-grade.
|
||||
|
||||
Important limits:
|
||||
|
||||
```text
|
||||
journey search is not yet RAPTOR/CSA or connection-scan based
|
||||
address endpoints can multiply transit searches through several nearby access/egress stops
|
||||
progressive transfer stages still recompute too much
|
||||
route-layer rebuild is coarse-grained and rewrites derived link tables
|
||||
visual route-pattern links are not yet incrementally updated
|
||||
canonical stop extraction is CPU/memory heavy on national feeds
|
||||
route geometry cannot yet classify temporary GTFS detours as separate variants
|
||||
local-transport-only routing is not a first-class query mode
|
||||
route-search caches are process-local and not persisted
|
||||
Alembic migrations are still missing
|
||||
source discovery directories, publishers/authorities, concrete feeds, and immutable feed snapshots are not yet separate first-class tables
|
||||
GTFS harmonization still lives in the map sidebar and should move to a dedicated workbench view as review queues grow
|
||||
manual GTFS/OSM stop and route decisions are not yet replayed as stable rules across feed/map updates
|
||||
source updates do not yet generate focused entity-level review diffs
|
||||
```
|
||||
|
||||
## MVP 1: stable Germany data workbench
|
||||
|
||||
### Backend
|
||||
|
||||
- Add proper Alembic migrations for PostgreSQL and keep SQLite test support.
|
||||
- Promote GTFS harmonization to the primary data workflow: source discovery, review, import, validation, deduplication, and publication readiness.
|
||||
- Build the feed review workflow: discovered feed -> license/authority review -> approve -> import.
|
||||
- Add the first-class source hierarchy: source directories -> sources/publishers/authorities -> concrete feeds -> immutable feed snapshots/datasets.
|
||||
- Store explicit license/publication flags per feed: can import, can derive, can redistribute, attribution required, and commercial restrictions.
|
||||
- Add immutable raw feed snapshots and source-run history.
|
||||
- Add validation report storage per feed snapshot.
|
||||
- Add first dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
|
||||
- Add persistent GTFS/OSM stop and route matching decisions that survive source updates.
|
||||
- Add feed/map update diffs so only changed stops, routes, patterns, calendars, and geometry evidence need review.
|
||||
- Add constrained candidate generation for route matching: OSM route features only, mode/ref/name filters, PostGIS bbox/spatial pruning, and geometry loading only for plausible candidates.
|
||||
- Add constrained candidate generation for stop matching: concrete OSM stop/platform/bay candidates near each GTFS stop, with explicit transfer links for nearby but distinct stops.
|
||||
- Add generated stop-to-stop map-path proposals for missing OSM route relations and GTFS detours.
|
||||
- Produce an active harmonized transit snapshot that routing consumes.
|
||||
- Run the generated 24-source GTFS test set and measure conflicts before scaling to more Europe feeds.
|
||||
- Add source-run history and dataset-version comparison.
|
||||
- Make route-layer rebuild incremental: update only affected matches/patterns/stops.
|
||||
- Keep old route-layer tables readable while a rebuild prepares replacement rows.
|
||||
- Add source health checks: download success, hash change, feed freshness, calendar validity.
|
||||
- Expand the QA dashboard into drill-down review queues for source health, GTFS validation, canonical stop conflicts, route conflicts, and publication blockers.
|
||||
- Add GTFS validation summary reports: service dates, route direction coverage, stop coordinate outliers, bad stop_times, missing shapes.
|
||||
- Add database maintenance jobs: analyze, vacuum, stale job recovery, orphan cleanup.
|
||||
- Add durable cache tables for journey stages, nearest stops, address access candidates, and common station-to-station searches.
|
||||
|
||||
### Routing
|
||||
|
||||
- Replace the demo round-expansion router with a GTFS-appropriate algorithm such as RAPTOR or CSA.
|
||||
- Precompute transfer graph edges: station-internal transfers, nearby walking transfers, and access/egress stop candidates.
|
||||
- Add routing profiles:
|
||||
|
||||
```text
|
||||
fastest public transport
|
||||
fewest transfers
|
||||
local transport only / Deutschlandticket-like
|
||||
walk only
|
||||
drive
|
||||
car comparison
|
||||
```
|
||||
|
||||
- Treat access/egress walking as access legs, not as public-transport transfers.
|
||||
- Add bounded hub-aware long-distance routing for city-to-city requests: local access to likely hubs, long-distance/regional trunk, local egress.
|
||||
- Add arrive-by search and better stop conditions for "good enough" results.
|
||||
- Add route diagnostics that explain why a route was found or pruned.
|
||||
|
||||
### Frontend
|
||||
|
||||
- Add source detail page.
|
||||
- Add dataset detail page.
|
||||
- Add match-review queue with filters by mode, operator, country, confidence, and source scope.
|
||||
- Add route detail inspection: GTFS geometry, OSM geometry, candidate matches, stops, evidence, and route-pattern provenance.
|
||||
- Add canonical stop/station detail overlay.
|
||||
- Add searchable route/stop review queues with filters for source, mode, operator, region, status, confidence, changed-since-last-run, missing geometry, and detour candidates.
|
||||
- Add map inspection that overlays competing GTFS shapes, OSM route relations, generated path proposals, concrete stop candidates, and previous reviewer decisions.
|
||||
- Add persistent rule editor.
|
||||
- Add routing controls for profile, transfer buffer, avoid/prefer modes, arrive-by, via, and local-only.
|
||||
- Show partial/progressive route results with clear stage labels.
|
||||
|
||||
### Data outputs
|
||||
|
||||
- GeoJSON exports for small regions.
|
||||
- GeoParquet exports for analysis.
|
||||
- PMTiles/vector-tile export for map display.
|
||||
- Coverage CSV/API for downstream services.
|
||||
|
||||
## MVP 2: Europe-scale coverage map
|
||||
|
||||
- Use Geofabrik country/Europe extracts and reproducible OSM PBF jobs.
|
||||
- Store OSM transport features, addresses, and routing graph in PostGIS.
|
||||
- Generate ranked/generalized transport route layers by zoom level.
|
||||
- Serve tiles with Martin or export PMTiles.
|
||||
- Add coverage statuses:
|
||||
|
||||
```text
|
||||
existing_in_osm
|
||||
static_timetable_covered
|
||||
live_data_covered
|
||||
fare_data_covered
|
||||
booking_covered
|
||||
missing_static
|
||||
stale_feed
|
||||
restricted_license
|
||||
low_confidence_match
|
||||
detour_or_temporary_variant
|
||||
```
|
||||
|
||||
- Add coverage metrics:
|
||||
|
||||
```text
|
||||
operator coverage
|
||||
route coverage
|
||||
route-km coverage
|
||||
stop coverage
|
||||
live-data coverage
|
||||
feed freshness
|
||||
license confidence
|
||||
booking coverage
|
||||
route-layer provenance coverage
|
||||
```
|
||||
|
||||
## MVP 3: more source formats
|
||||
|
||||
Add importers:
|
||||
|
||||
```text
|
||||
NeTEx
|
||||
TransXChange
|
||||
SIRI discovery/live endpoints
|
||||
GTFS-Realtime
|
||||
GBFS for shared mobility, optional
|
||||
operator CSV/API adapters
|
||||
```
|
||||
|
||||
Target data model:
|
||||
|
||||
```text
|
||||
canonical operators
|
||||
canonical stops/stations/terminals
|
||||
canonical routes
|
||||
route variants
|
||||
trip patterns
|
||||
calendar/service validity
|
||||
transfers
|
||||
access/egress legs
|
||||
coverage observations
|
||||
source evidence
|
||||
manual rules
|
||||
```
|
||||
|
||||
## MVP 4: production journey-planning dataset
|
||||
|
||||
- Build a canonical stop/station graph with transfer rules and transfer-time profiles.
|
||||
- Generate timetable-routing input for RAPTOR/CSA.
|
||||
- Add first/last-mile routing from OSM walk/drive graph.
|
||||
- Add emissions factors per mode/operator/country.
|
||||
- Add fare/ticket placeholders and booking/deep-link metadata.
|
||||
- Add confidence and provenance to every derived route/journey.
|
||||
|
||||
## MVP 5: booking-readiness layer
|
||||
|
||||
- Track booking availability separately from timetable coverage.
|
||||
- Add deep-link metadata per operator/route.
|
||||
- Add partner API adapters later.
|
||||
- Distinguish clearly:
|
||||
|
||||
```text
|
||||
travel-plausible itinerary
|
||||
bookable itinerary
|
||||
single-interface multi-booking
|
||||
protected through-ticket
|
||||
```
|
||||
|
||||
## Recommended next implementation sprint
|
||||
|
||||
Primary product/data track: GTFS Harmonization.
|
||||
|
||||
1. Turn discovered GTFS feeds into a review queue with license, redistribution, authority, freshness, and coverage fields.
|
||||
2. Approve/import the generated 24-source GTFS test set and keep immutable raw feed snapshots.
|
||||
3. Persist source-run history and validation reports for every feed snapshot.
|
||||
4. Add entity-level feed diffs and matching-rule replay so updates do not reset reviewed work.
|
||||
5. Add GTFS/OSM stop and route review queues with map inspection.
|
||||
6. Add dedup review queues for agencies/operators, stop places, route duplicates, and calendar conflicts.
|
||||
7. Generate an active harmonized transit snapshot and make route search consume that snapshot by default.
|
||||
8. Measure conflicts and authority decisions from the test set before scaling to broader Europe coverage.
|
||||
|
||||
Parallel technical track:
|
||||
|
||||
1. Add route-search diagnostics and timing instrumentation.
|
||||
2. Implement a precomputed transfer graph and cleaner access/egress transfer-budget model.
|
||||
3. Add durable PostgreSQL-backed route-search caches.
|
||||
4. Convert route-layer rebuild to shadow/versioned rows or incremental updates.
|
||||
5. Add Alembic migrations and remove routine schema checks from normal app/worker startup.
|
||||
6. Add vector-tile output for large route layers.
|
||||
Reference in New Issue
Block a user