meubility-workbench/docs/backlog.md

Product and Engineering Backlog

Last updated: 2026-07-01

This backlog reflects the current Germany-scale PostGIS prototype. The target remains a Europe-scale mobility data workbench that builds canonical stops, stations, routes, route geometry, timetable links, transfer rules, routing graph data, address search, and coverage evidence from many public sources.

OSM-derived geometry is the preferred visual authority. GTFS, NeTEx, realtime, and official APIs are timetable, validation, routing, and gap-detection inputs. GTFS shapes are still valuable evidence, especially for missing OSM relations and temporary detours.

Current State

• PostgreSQL/PostGIS is the active development database path; SQLite remains a legacy/test fallback.
• Germany OSM and Germany GTFS/DELFI-scale imports are supported.
• OSM address indexing is available and address search is bbox-aware without being bbox-limited.
• Jobs and job events exist for imports, route matching, route-layer rebuilds, address indexing, relabeling, deletes, and maintenance.
• Job rows expose a generic details overlay with planned/current/done phases, event log, metadata, and a compact queue snapshot.
• A first QA dashboard skeleton exists for source discovery, import health, GTFS validation, canonical stop/link coverage, route matching, and publication readiness.
• The GTFS harmonization target architecture is documented in docs/gtfs_harmonization.md.
• GTFS source management is presented as a separate GTFS Harmonization UI module; OSM/map inputs are presented as a separate Mapping Data module.
• Journey search consumes the active harmonized transit snapshot instead of exposing a raw GTFS source selector.
• Route-layer rebuild runs through the queue, but it is still coarse-grained and can take minutes on national datasets.
• The route-layer builder links canonical GTFS stops, OSM stops, OSM route relations, GTFS route patterns, and trip-pattern links.
• Journey search is progressive and can publish intermediate results, but the underlying routing algorithm is still a prototype.
• Walk and drive routing use the OSM-derived routing layer when available.

Current Caveats

• Journey search is not yet a full RAPTOR/CSA-style router.
• Address endpoints can multiply the search space: current behavior can use up to 4 access stops and 4 egress stops, creating up to 16 transit stop-pair searches per transfer stage.
• Progressive stages still recompute too much. Searching up to 2 transfers repeats direct and one-transfer work before deeper expansion.
• Walking access/egress legs are represented separately in journey output, but the search engine still needs a cleaner transfer budget model where access/egress walking never consumes public-transport transfer count.
• Route-search caches are in-process only. They do not survive server restart, do not deduplicate identical searches already running in another thread/process, and only help once a stage/search has completed.
• Route-layer rebuild currently clears/rebuilds derived tables. Until the rebuild completes, visual route-pattern link tables can be incomplete.
• Timetable reachability should not depend on visual route-pattern links. The code has been patched in this checkout, but a running server must reload before using that fix.
• Canonical stop extraction on national feeds is CPU/memory heavy and does too much Python-side grouping.
• OSM stop-linking and OSM route-candidate indexing are still large spatial/batch operations.
• GTFS detours are not classified as first-class route variants yet.
• Local-transport-only routing is not a first-class profile yet.
• Proper Alembic migrations are still missing; runtime schema maintenance should be reduced to an explicit migration/maintenance path.
• The source and job database tables are still shared between harmonization, mapping, and routing; the current split is a product/UI boundary, not a separate service or database boundary yet.

P0: Routing Performance and Correctness

These items directly address slow or failed searches such as Berlin, Alexanderplatz to Heidelberg, Blumenstrasse 36.

• Replace the demo round-expansion router with a timetable-native algorithm. Preferred direction: RAPTOR or CSA over preloaded arrays/tables, with rounds representing public-transport boardings rather than ad hoc SQL expansion.
• Precompute a transfer graph. Store station-internal transfers, nearby walking transfers, platform/stop-place links, and allowed transfer times by mode/source/station.
• Separate access/egress from transfer count. Walking from an address to the first stop, and from the last stop to an address, should never count as a vehicle transfer.
• Add a durable journey cache. Cache normalized requests, address-to-stop candidates, stop-to-stop stage results, common station-pair results, and in-flight request deduplication in PostgreSQL.
• Add hub-aware long-distance routing. For long-distance OD pairs, search local access to likely hubs, trunk rail/regional candidates, then local egress. Candidate hubs can be ranked by station importance, service frequency, route scope, distance, and direction.
• Add a local-transport-only profile. Implement a Deutschlandticket-like profile that excludes long-distance route scopes and still supports regional rail, S-Bahn, subway, tram, bus, ferry, and walking transfers.
• Add admissible pruning. Bound exploration by best known arrival, remaining distance, direction/off-course penalty, transfer budget, service frequency, and maximum tolerated detour.
• Add journey diagnostics. Return searched stages, candidate counts, pruned reasons, access/egress stops, service date, source feeds, transfer stops, and whether no-route means no timetable path or a search limit was hit.
• Add arrive-by search. This is important for route quality and for comparing against operator/DB route planners.
• Add route profile controls in the UI. fastest, earliest arrival, fewest transfers, local only, walk, drive, arrive by, via, avoid, and transfer buffer controls.

P0: Queue and Rebuild Robustness

• Move runtime schema maintenance out of normal app startup. The current checkout avoids redundant PostgreSQL DDL, but explicit migrations are still needed.
• Add Alembic migrations. Use migrations for PostGIS columns, indexes, route-layer tables, routing tables, and cache tables.
• Make route-layer rebuild use shadow tables or versioned rows. Build replacement rows without deleting the readable active layer first; atomically promote the new version when complete.
• Make route-layer rebuild incremental. Rebuild only affected route patterns after new matches, stop-link decisions, source updates, or OSM diffs.
• Add stale worker and stale pid reconciliation. Worker status should never report a pid as running unless the current server can verify it.
• Improve cancellation. Long PostgreSQL statements need cancellable phases and visible progress rather than only a queued/running state.
• Improve progress granularity and timings. The UI can display job events now, but long PostgreSQL statements still need finer checkpoints, elapsed times, estimated remaining work, and cancellable sub-phases.

P1: Route Layer, Detours, and Geometry Provenance

• Classify GTFS route variants. Group trips by route, direction, shape, stop sequence, service date span, and trip frequency. Mark rare/temporary shapes as detours or temporary variants rather than replacing the canonical visual route.
• Add stop-by-stop OSM path fallback. When an OSM route relation is missing or a GTFS shape is a detour, assemble geometry between matched consecutive stops using mode-constrained OSM paths.
• Cache stop-to-stop route geometry. Key by mode, from canonical stop, to canonical stop, direction constraints, and graph version.
• Store geometry provenance per route pattern. Examples: osm_route_relation, gtfs_shape, stop_to_stop_osm_path, manual_override, detour_variant.
• Respect directionality. Bus/car paths need oneway handling; tram/rail paths need topology and direction evidence; reverse links must not be assumed valid.
• Add route-pattern detail inspection. Show OSM geometry, GTFS shapes, linked trips, linked stops, direction evidence, confidence, and variant/detour status.
• Add generalized route geometries. Store high-detail inspection geometry and simplified map geometry.

P1: Canonical Stops, Stations, and Addresses

• Optimize canonical stop extraction. Push more grouping/linking into SQL, avoid loading all scheduled stops into Python, batch inserts, and keep stable canonical IDs when possible.
• Build a canonical stop alias table. Persist normalized names, multilingual names, station codes, IBNR/EVA/UIC/IFOPT, stop_area IDs, OSM IDs, and source-specific aliases.
• Improve station-complex modeling. Separate public stop place, station complex, platforms/tracks, entrances, bus bays, and nearby stop groups.
• Add canonical stop detail overlay. Show linked GTFS stops, linked OSM stops/stations, source names, confidence, distances, and manual overrides.
• Add manual canonical stop link/unlink decisions. Persist stop matching decisions like route matching decisions, so source updates do not overwrite reviewed links.
• Improve address result folding. Prefer street-level suggestions for dense house-number ranges, but preserve exact address selection when a full address is typed.
• Precompute address access candidates. Store nearest useful public-transport stops per address/street point, with mode/source/radius metadata.

P1: More GTFS Sources and Deduplication

• Import more GTFS feeds where they improve authority or coverage. DB long-distance/regional feeds, state feeds, and neighboring-country feeds touching Germany are useful test cases.
• Add source priority and authority ranking. Decide which source is more authoritative for stops, operators, routes, calendars, and geometry evidence.
• Deduplicate operators/agencies. Merge agency/operator records with provenance and aliases instead of treating each GTFS agency.txt row as a separate operator.
• Turn QA summary counters into review queues. Drill down from each bad/warn metric into concrete sources, stops, routes, links, and conflicts.
• Add GTFS feed QA reports. Calendar coverage, stale feeds, missing shapes, impossible stop times, duplicate routes, route direction coverage, stop coordinate outliers.
• Add conflict dashboards and reusable resolution workflows. Show canonical stops/routes with competing source claims, weak matches, missing visual geometry, authority-rule conflicts, and license blockers.

P1: Scalable OSM and Map Outputs

• Keep OSM PBF import chunked and resumable. Keep previous active visual datasets available while the next import builds.
• Add vector tile or PMTiles export. Needed for Germany/Europe route layers and dense editing views.
• Add route-scope and mode-specific map generalization. Different zooms should use different detail levels and route classes.
• Improve OSM route candidate indexing. Use stronger SQL/PostGIS filtering before loading route geometry into Python.
• Add OSM diffs later. Minutely/hourly/daily diffs can update route and address layers without full country rebuilds.

P2: Data Platform Hardening

• Add explicit read/write transaction boundaries for all long requests and jobs.
• Add API pagination for large result sets.
• Add import logs and source-run history.
• Add database maintenance commands: analyze, vacuum, reindex, orphan cleanup.
• Add test fixtures that do not mutate the live development database.
• Add observability: query timings, job timings, row counts, cache hit rates, and per-stage routing metrics.

P2: Better Map and Editing Workflows

• Add canonical stop and route detail side panels.
• Add candidate map preview for stop matching, not only route matching.
• Add unmatched/matched/weak/proposed visual layers with source filters.
• Keep calculated journey geometry and stop markers always on top.
• Add editable match queues for stops, station complexes, routes, and operators.
• Add route-layer diff view after rebuilds.

P3: Additional Formats and Live Data

• Add NeTEx import.
• Add GTFS-Realtime ingestion for service alerts and trip updates.
• Add SIRI profile support where national APIs expose it.
• Add GBFS/shared mobility only after core public transport data is stable.
• Model temporary closures and disruptions as validity-windowed events, not modifications to base route geometry.

Open Optimization List

Not yet implemented, or only partially implemented:

• RAPTOR/CSA routing core.
• Precomputed public-transport transfer graph.
• Durable PostgreSQL route-search cache.
• In-flight identical search coalescing.
• Hub-aware long-distance routing.
• Local-transport-only routing profile.
• Access/egress legs excluded from transfer budget at the search-state level.
• Better pruning for off-course exploration and dominated labels.
• SQL/array-based canonical stop extraction.
• Incremental route-layer rebuild.
• Route-layer shadow tables/versioned activation.
• Stop-to-stop OSM route fallback for missing routes and detours.
• Detour/temporary variant classification.
• PostGIS-first OSM route candidate filtering.
• Vector tiles or PMTiles for large route layers.
• Alembic migrations.
• Persistent query/stage timing diagnostics.

Recommended Next Sprint

1. Finish the route-layer rebuild currently in progress and verify route-pattern/trip-pattern link counts.
2. Restart/reload the server so it picks up the current checkout fixes.
3. Add route-search diagnostics and timing instrumentation around address access, direct, one-transfer, and round-search stages.
4. Implement transfer graph precomputation and exclude access/egress walking from transfer count.
5. Add a hub-aware city-to-city search path for long-distance requests.
6. Add a local-only routing profile using route scopes.
7. Convert route-layer rebuild to shadow/versioned tables or incremental updates.
8. Add Alembic migrations and stop doing routine schema checks during normal app/worker startup.