add-ideas/meubility-workbench
1
0
Fork 0
Code Issues 122 Pull Requests Actions Packages Projects Releases Wiki Activity

Next alpha stage commit

Browse Source
This commit is contained in:
Albrecht Degering
2026-07-06 20:54:46 +02:00
parent e23387738b
commit 1d7ec956d2
42 changed files with 11603 additions and 2837 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.env.example
View File

@@ -4,7 +4,7 @@ DATABASE_URL=sqlite:///./data/workbench.sqlite
# DATABASE_URL=postgresql://USER:PASSWORD@localhost:5432/meubility
# POSTGRES_USE_SIDECARS=false
DATA_DIR=./data
GTFS_STOP_TIMES_IMPORT_LIMIT=250000
GTFS_STOP_TIMES_IMPORT_LIMIT=0
# Start separate queue worker processes from the API server lifespan.
# Workers survive normal server restarts by default; stale leases are recovered.
@@ -13,6 +13,7 @@ QUEUE_WORKER_COUNT=1
QUEUE_WORKER_POLL_INTERVAL_SECONDS=2
QUEUE_JOB_LEASE_SECONDS=7200
QUEUE_WORKER_STOP_ON_SHUTDOWN=false
JOURNEY_SEARCH_WORKER_COUNT=4
# Chunk sizes for queued data-preparation jobs.
ROUTE_MATCHING_BATCH_SIZE=100

220
MVP_ROADMAP.md
View File

@@ -1,220 +0,0 @@
# MVP roadmap
Last updated: 2026-07-01
See also `docs/backlog.md` for the prioritized engineering backlog, caveats, and open optimization list.
## Objective
Build an internal management workbench that turns public mobility data into a normalized, auditable, coverage-scored dataset for a future traveller-facing web/native app.
The workbench stays distinct from the public app. Its users are data engineers, analysts, and operations staff who need to ingest, inspect, link, correct, route against, and publish mobility data.
## Current prototype: implemented
The repository has moved beyond the original SQLite/Berlin prototype. The current development path is Germany-scale and PostGIS-first, while SQLite remains useful as a legacy/test fallback.
Implemented:
```text
source registry and source catalog
local source cache
job queue with job events and worker process
PostgreSQL/PostGIS runtime support with SQLite fallback
GTFS static importer for large national feeds
OSM PBF import path for Germany-scale extracts
OSM address index and address-aware journey endpoints
canonical stop/station linking from GTFS and OSM
automatic GTFS <-> OSM route matching
manual route and canonical-stop rule persistence
visual route-layer builder from OSM routes and GTFS shapes
walk/drive routing layer from OSM-derived routing graph
progressive journey-search API and UI polling
map right-click "from here" / "to here"
management UI with map, sources, stats, jobs, matches, search, and journeys
separate GTFS Harmonization and Mapping Data source modules in the UI
generic job-details overlay with phase timeline, event log, and queue snapshot
QA dashboard skeleton for source/import/link/route/publication health
GTFS harmonization concept and service-boundary decision
CLI commands
tests and syntax checks for changed modules
```
Recent fixes:
```text
PostgreSQL startup avoids unnecessary DDL when PostGIS columns/indexes already exist.
Queue route-layer rebuild can be claimed by a real worker instead of staying queued behind a stale worker pid.
Timetable routing no longer requires visual route-pattern trip links.
Walk-leg route geometry has a short-lived in-process cache.
Address search is bbox-aware without being bbox-limited.
Job rows expose a details overlay that polls job events only while open.
Journey routing consumes the active harmonized GTFS snapshot instead of a raw feed picker.
```
## Current prototype: known limits
The app can import and inspect Germany-scale OSM and GTFS, but the routing and route-layer rebuild paths are still prototype-grade.
Important limits:
```text
journey search is not yet RAPTOR/CSA or connection-scan based
address endpoints can multiply transit searches through several nearby access/egress stops
progressive transfer stages still recompute too much
route-layer rebuild is coarse-grained and rewrites derived link tables
visual route-pattern links are not yet incrementally updated
canonical stop extraction is CPU/memory heavy on national feeds
route geometry cannot yet classify temporary GTFS detours as separate variants
local-transport-only routing is not a first-class query mode
route-search caches are process-local and not persisted
Alembic migrations are still missing
```
## MVP 1: stable Germany data workbench
### Backend
- Add proper Alembic migrations for PostgreSQL and keep SQLite test support.
- Add source-run history and dataset-version comparison.
- Make route-layer rebuild incremental: update only affected matches/patterns/stops.
- Keep old route-layer tables readable while a rebuild prepares replacement rows.
- Add source health checks: download success, hash change, feed freshness, calendar validity.
- Expand the QA dashboard into drill-down review queues for source health, GTFS validation, canonical stop conflicts, route conflicts, and publication blockers.
- Add GTFS validation summary reports: service dates, route direction coverage, stop coordinate outliers, bad stop_times, missing shapes.
- Add database maintenance jobs: analyze, vacuum, stale job recovery, orphan cleanup.
- Add durable cache tables for journey stages, nearest stops, address access candidates, and common station-to-station searches.
### Routing
- Replace the demo round-expansion router with a GTFS-appropriate algorithm such as RAPTOR or CSA.
- Precompute transfer graph edges: station-internal transfers, nearby walking transfers, and access/egress stop candidates.
- Add routing profiles:
```text
fastest public transport
fewest transfers
local transport only / Deutschlandticket-like
walk only
drive
car comparison
```
- Treat access/egress walking as access legs, not as public-transport transfers.
- Add bounded hub-aware long-distance routing for city-to-city requests: local access to likely hubs, long-distance/regional trunk, local egress.
- Add arrive-by search and better stop conditions for "good enough" results.
- Add route diagnostics that explain why a route was found or pruned.
### Frontend
- Add source detail page.
- Add dataset detail page.
- Add match-review queue with filters by mode, operator, country, confidence, and source scope.
- Add route detail inspection: GTFS geometry, OSM geometry, candidate matches, stops, evidence, and route-pattern provenance.
- Add canonical stop/station detail overlay.
- Add persistent rule editor.
- Add routing controls for profile, transfer buffer, avoid/prefer modes, arrive-by, via, and local-only.
- Show partial/progressive route results with clear stage labels.
### Data outputs
- GeoJSON exports for small regions.
- GeoParquet exports for analysis.
- PMTiles/vector-tile export for map display.
- Coverage CSV/API for downstream services.
## MVP 2: Europe-scale coverage map
- Use Geofabrik country/Europe extracts and reproducible OSM PBF jobs.
- Store OSM transport features, addresses, and routing graph in PostGIS.
- Generate ranked/generalized transport route layers by zoom level.
- Serve tiles with Martin or export PMTiles.
- Add coverage statuses:
```text
existing_in_osm
static_timetable_covered
live_data_covered
fare_data_covered
booking_covered
missing_static
stale_feed
restricted_license
low_confidence_match
detour_or_temporary_variant
```
- Add coverage metrics:
```text
operator coverage
route coverage
route-km coverage
stop coverage
live-data coverage
feed freshness
license confidence
booking coverage
route-layer provenance coverage
```
## MVP 3: more source formats
Add importers:
```text
NeTEx
TransXChange
SIRI discovery/live endpoints
GTFS-Realtime
GBFS for shared mobility, optional
operator CSV/API adapters
```
Target data model:
```text
canonical operators
canonical stops/stations/terminals
canonical routes
route variants
trip patterns
calendar/service validity
transfers
access/egress legs
coverage observations
source evidence
manual rules
```
## MVP 4: production journey-planning dataset
- Build a canonical stop/station graph with transfer rules and transfer-time profiles.
- Generate timetable-routing input for RAPTOR/CSA.
- Add first/last-mile routing from OSM walk/drive graph.
- Add emissions factors per mode/operator/country.
- Add fare/ticket placeholders and booking/deep-link metadata.
- Add confidence and provenance to every derived route/journey.
## MVP 5: booking-readiness layer
- Track booking availability separately from timetable coverage.
- Add deep-link metadata per operator/route.
- Add partner API adapters later.
- Distinguish clearly:
```text
travel-plausible itinerary
bookable itinerary
single-interface multi-booking
protected through-ticket
```
## Recommended next implementation sprint
1. Finish route-layer rebuild resilience: incremental updates, shadow tables, and detour/provenance classification.
2. Replace or heavily optimize journey routing: precomputed transfers, hub-aware long-distance routing, local-only profile, and bounded search.
3. Add durable PostgreSQL-backed journey caches for address access, stop pairs, and repeated stage searches.
4. Add Alembic migrations and remove runtime DDL from normal request/worker startup.
5. Add route/journey diagnostics so slow or failed requests explain what was searched and pruned.
6. Add vector-tile output for route layers and large map rendering.

4
app/cli.py
View File

@@ -227,7 +227,9 @@ def discover_gtfs_sources_command(
no_mobility_database: bool = typer.Option(False, help="Skip Mobility Database feeds_v2.csv"),
no_acceptance_test_list: bool = typer.Option(False, help="Skip MobilityData validator acceptance-test feed list"),
no_ptna: bool = typer.Option(False, help="Skip PTNA GTFS analysis pages"),
no_opendata_oepnv: bool = typer.Option(False, help="Skip OpenData ÖPNV Germany dataset catalog"),
max_ptna_details: int = typer.Option(80, help="Maximum PTNA detail pages to fetch for license/crosswalk metadata"),
max_opendata_oepnv_details: int = typer.Option(80, help="Maximum OpenData ÖPNV detail pages to fetch"),
test_limit: int = typer.Option(24, help="Rows to write to the focused test-run ingestable CSV"),
check_urls: bool = typer.Option(False, help="Run HEAD/range checks for ingestable feed URLs"),
) -> None:
@@ -237,7 +239,9 @@ def discover_gtfs_sources_command(
include_mobility_database=not no_mobility_database,
include_acceptance_test_list=not no_acceptance_test_list,
include_ptna=not no_ptna,
include_opendata_oepnv=not no_opendata_oepnv,
max_ptna_details=max_ptna_details,
max_opendata_oepnv_details=max_opendata_oepnv_details,
test_limit=test_limit,
check_urls=check_urls,
)

1
app/config.py
View File

@@ -52,6 +52,7 @@ class Settings(BaseSettings):
# Keep supervised workers alive across API server restarts. Stale workers are
# detected by PID files at the next startup; stale job leases are requeued.
queue_worker_stop_on_shutdown: bool = False
journey_search_worker_count: int = 4
model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8")

116
app/data_management.py
View File

@@ -13,6 +13,8 @@ from app.models import (
GtfsAgency,
GtfsCalendar,
GtfsCalendarDate,
GtfsHarmonizedSnapshotDataset,
GtfsHarmonizedSnapshotRoute,
GtfsRoute,
GtfsRoutePatternLink,
GtfsShape,
@@ -20,6 +22,8 @@ from app.models import (
GtfsStopTime,
GtfsTripRoutePatternLink,
GtfsTrip,
MapGtfsDecision,
MapGtfsReviewItem,
OsmDiffState,
OsmFeature,
RouteMatch,
@@ -136,6 +140,8 @@ def delete_source(session: Session, source_id: int) -> dict[str, object]:
_delete_dataset_files(dataset)
session.delete(dataset)
session.execute(delete(OsmDiffState).where(OsmDiffState.source_id == source.id))
session.execute(delete(MapGtfsReviewItem).where(MapGtfsReviewItem.source_id == source.id))
session.execute(delete(MapGtfsDecision).where(MapGtfsDecision.source_id == source.id))
session.delete(source)
session.flush()
return {"deleted": True, "source_id": source_id, "datasets": dataset_results}
@@ -217,14 +223,41 @@ def prune_inactive_datasets(session: Session, dry_run: bool = True) -> dict[str,
for dataset_id in dataset_ids:
_detach_update_checks_for_dataset(session, dataset_id)
for dataset in session.scalars(select(Dataset).where(Dataset.id.in_(dataset_ids))).all():
_delete_workbench_rows_for_dataset(session, dataset)
if match_filters:
session.execute(delete(RouteMatch).where(or_(*match_filters)))
if gtfs_ids:
route_ids = select(GtfsRoute.id).where(GtfsRoute.dataset_id.in_(gtfs_ids))
pattern_ids = select(RoutePattern.id).where(RoutePattern.gtfs_route_id.in_(route_ids))
session.execute(
delete(GtfsHarmonizedSnapshotRoute).where(
or_(
GtfsHarmonizedSnapshotRoute.dataset_id.in_(gtfs_ids),
GtfsHarmonizedSnapshotRoute.shadowed_by_dataset_id.in_(gtfs_ids),
GtfsHarmonizedSnapshotRoute.gtfs_route_id.in_(route_ids),
GtfsHarmonizedSnapshotRoute.shadowed_by_gtfs_route_id.in_(route_ids),
)
)
)
session.execute(delete(GtfsHarmonizedSnapshotDataset).where(GtfsHarmonizedSnapshotDataset.dataset_id.in_(gtfs_ids)))
session.execute(delete(RoutePatternStop).where(RoutePatternStop.route_pattern_id.in_(pattern_ids)))
session.execute(delete(GtfsTripRoutePatternLink).where(GtfsTripRoutePatternLink.dataset_id.in_(gtfs_ids)))
session.execute(delete(GtfsRoutePatternLink).where(GtfsRoutePatternLink.dataset_id.in_(gtfs_ids)))
session.execute(
delete(GtfsTripRoutePatternLink).where(
or_(
GtfsTripRoutePatternLink.dataset_id.in_(gtfs_ids),
GtfsTripRoutePatternLink.route_pattern_id.in_(pattern_ids),
)
)
)
session.execute(
delete(GtfsRoutePatternLink).where(
or_(
GtfsRoutePatternLink.dataset_id.in_(gtfs_ids),
GtfsRoutePatternLink.route_pattern_id.in_(pattern_ids),
)
)
)
session.execute(delete(RoutePattern).where(RoutePattern.gtfs_route_id.in_(route_ids)))
session.execute(delete(CanonicalStopLink).where(CanonicalStopLink.dataset_id.in_(gtfs_ids), CanonicalStopLink.object_type == "gtfs_stop"))
for model in [GtfsStopTime, GtfsShape, GtfsTrip, GtfsCalendarDate, GtfsCalendar, GtfsRoute, GtfsStop, GtfsAgency]:
@@ -246,13 +279,39 @@ def prune_inactive_datasets(session: Session, dry_run: bool = True) -> dict[str,
def _delete_dataset_rows(session: Session, dataset: Dataset) -> None:
_delete_workbench_rows_for_dataset(session, dataset)
if dataset.kind == "gtfs":
route_ids = select(GtfsRoute.id).where(GtfsRoute.dataset_id == dataset.id)
pattern_ids = select(RoutePattern.id).where(RoutePattern.gtfs_route_id.in_(route_ids))
session.execute(
delete(GtfsHarmonizedSnapshotRoute).where(
or_(
GtfsHarmonizedSnapshotRoute.dataset_id == dataset.id,
GtfsHarmonizedSnapshotRoute.shadowed_by_dataset_id == dataset.id,
GtfsHarmonizedSnapshotRoute.gtfs_route_id.in_(route_ids),
GtfsHarmonizedSnapshotRoute.shadowed_by_gtfs_route_id.in_(route_ids),
)
)
)
session.execute(delete(GtfsHarmonizedSnapshotDataset).where(GtfsHarmonizedSnapshotDataset.dataset_id == dataset.id))
session.execute(delete(RouteMatch).where(RouteMatch.gtfs_route_id.in_(route_ids)))
session.execute(delete(RoutePatternStop).where(RoutePatternStop.route_pattern_id.in_(pattern_ids)))
session.execute(delete(GtfsTripRoutePatternLink).where(GtfsTripRoutePatternLink.dataset_id == dataset.id))
session.execute(delete(GtfsRoutePatternLink).where(GtfsRoutePatternLink.dataset_id == dataset.id))
session.execute(
delete(GtfsTripRoutePatternLink).where(
or_(
GtfsTripRoutePatternLink.dataset_id == dataset.id,
GtfsTripRoutePatternLink.route_pattern_id.in_(pattern_ids),
)
)
)
session.execute(
delete(GtfsRoutePatternLink).where(
or_(
GtfsRoutePatternLink.dataset_id == dataset.id,
GtfsRoutePatternLink.route_pattern_id.in_(pattern_ids),
)
)
)
session.execute(delete(RoutePattern).where(RoutePattern.gtfs_route_id.in_(route_ids)))
session.execute(delete(CanonicalStopLink).where(CanonicalStopLink.dataset_id == dataset.id, CanonicalStopLink.object_type == "gtfs_stop"))
for model in [GtfsStopTime, GtfsShape, GtfsTrip, GtfsCalendarDate, GtfsCalendar, GtfsRoute, GtfsStop, GtfsAgency]:
@@ -269,6 +328,55 @@ def _delete_dataset_rows(session: Session, dataset: Dataset) -> None:
session.execute(delete(OsmFeature).where(OsmFeature.dataset_id == dataset.id))
def _delete_workbench_rows_for_dataset(session: Session, dataset: Dataset) -> None:
review_filters = [MapGtfsReviewItem.dataset_id == dataset.id]
decision_filters = [
MapGtfsDecision.gtfs_dataset_id == dataset.id,
MapGtfsDecision.osm_dataset_id == dataset.id,
]
if dataset.kind == "gtfs":
route_ids = select(GtfsRoute.id).where(GtfsRoute.dataset_id == dataset.id)
stop_ids = select(GtfsStop.id).where(GtfsStop.dataset_id == dataset.id)
match_ids = select(RouteMatch.id).where(RouteMatch.gtfs_route_id.in_(route_ids))
link_ids = select(CanonicalStopLink.id).where(CanonicalStopLink.dataset_id == dataset.id, CanonicalStopLink.object_type == "gtfs_stop")
review_filters.extend(
[
MapGtfsReviewItem.gtfs_route_id.in_(route_ids),
MapGtfsReviewItem.gtfs_stop_id.in_(stop_ids),
MapGtfsReviewItem.route_match_id.in_(match_ids),
MapGtfsReviewItem.canonical_stop_link_id.in_(link_ids),
]
)
decision_filters.extend(
[
MapGtfsDecision.gtfs_route_id.in_(route_ids),
MapGtfsDecision.gtfs_stop_id.in_(stop_ids),
MapGtfsDecision.route_match_id.in_(match_ids),
MapGtfsDecision.canonical_stop_link_id.in_(link_ids),
]
)
elif dataset.kind == "osm_geojson":
osm_feature_ids = select(OsmFeature.id).where(OsmFeature.dataset_id == dataset.id)
match_ids = select(RouteMatch.id).where(RouteMatch.osm_feature_id.in_(osm_feature_ids))
link_ids = select(CanonicalStopLink.id).where(CanonicalStopLink.dataset_id == dataset.id, CanonicalStopLink.object_type == "osm_feature")
review_filters.extend(
[
MapGtfsReviewItem.osm_feature_id.in_(osm_feature_ids),
MapGtfsReviewItem.route_match_id.in_(match_ids),
MapGtfsReviewItem.canonical_stop_link_id.in_(link_ids),
]
)
decision_filters.extend(
[
MapGtfsDecision.osm_feature_id.in_(osm_feature_ids),
MapGtfsDecision.route_match_id.in_(match_ids),
MapGtfsDecision.canonical_stop_link_id.in_(link_ids),
]
)
session.execute(delete(MapGtfsReviewItem).where(or_(*review_filters)))
session.execute(delete(MapGtfsDecision).where(or_(*decision_filters)))
def _delete_dataset_files(dataset: Dataset) -> None:
for path in dataset_sidecar_paths(dataset):
try:

17
app/db.py
View File

@@ -184,6 +184,16 @@ def _ensure_runtime_indexes() -> None:
"CREATE INDEX IF NOT EXISTS ix_gtfs_shapes_dataset_shape ON gtfs_shapes (dataset_id, shape_id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_calendars_dataset_service_dates ON gtfs_calendars (dataset_id, service_id, start_date, end_date)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_calendar_dates_dataset_date ON gtfs_calendar_dates (dataset_id, date, service_id, exception_type)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshots_status_created ON gtfs_harmonized_snapshots (status, created_at, id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshot_datasets_role ON gtfs_harmonized_snapshot_datasets (snapshot_id, role, dataset_id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshot_datasets_source ON gtfs_harmonized_snapshot_datasets (source_id, role)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshot_routes_role ON gtfs_harmonized_snapshot_routes (snapshot_id, role, dataset_id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshot_routes_overlap ON gtfs_harmonized_snapshot_routes (snapshot_id, overlap_key, role)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_harmonized_snapshot_routes_shadow ON gtfs_harmonized_snapshot_routes (snapshot_id, shadowed_by_gtfs_route_id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_feed_diff_runs_source_created ON gtfs_feed_diff_runs (source_id, created_at, id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_feed_diff_runs_new_dataset ON gtfs_feed_diff_runs (new_dataset_id)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_feed_diff_items_run_type ON gtfs_feed_diff_items (diff_run_id, item_type, change_type, severity)",
"CREATE INDEX IF NOT EXISTS ix_gtfs_feed_diff_items_source_status ON gtfs_feed_diff_items (source_id, status, severity)",
"CREATE INDEX IF NOT EXISTS ix_canonical_stop_links_object ON canonical_stop_links (object_type, dataset_id, object_id)",
"CREATE INDEX IF NOT EXISTS ix_canonical_stop_links_external ON canonical_stop_links (object_type, dataset_id, external_id)",
"CREATE INDEX IF NOT EXISTS ix_route_patterns_ref_mode ON route_patterns (route_ref, mode, source_kind)",
@@ -210,6 +220,13 @@ def _ensure_runtime_indexes() -> None:
"CREATE INDEX IF NOT EXISTS ix_pipeline_runs_stage_source_hash ON pipeline_runs (stage, source_id, dependency_hash, status, started_at)",
"CREATE INDEX IF NOT EXISTS ix_pipeline_runs_job ON pipeline_runs (job_id, stage, status)",
"CREATE INDEX IF NOT EXISTS ix_match_rules_type_active ON match_rules (rule_type, active)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_decisions_type_status ON map_gtfs_decisions (decision_type, status, active)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_decisions_route ON map_gtfs_decisions (gtfs_route_id, status, active)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_decisions_stop ON map_gtfs_decisions (gtfs_stop_id, status, active)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_review_items_queue_status ON map_gtfs_review_items (queue, status, severity, updated_at)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_review_items_dataset_status ON map_gtfs_review_items (dataset_id, status, queue)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_review_items_route ON map_gtfs_review_items (gtfs_route_id, status)",
"CREATE INDEX IF NOT EXISTS ix_map_gtfs_review_items_stop ON map_gtfs_review_items (gtfs_stop_id, status)",
"CREATE INDEX IF NOT EXISTS ix_journey_search_cache_type_expires ON journey_search_cache (cache_type, expires_at)",
"CREATE INDEX IF NOT EXISTS ix_travel_requests_created ON travel_requests (created_at)",
"CREATE INDEX IF NOT EXISTS ix_itineraries_request_saved ON itineraries (request_id, saved, created_at)",

351
app/decision_replay.py Normal file
View File

@@ -0,0 +1,351 @@
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timezone
from typing import Any
from sqlalchemy import or_, select
from sqlalchemy.orm import Session
from app.models import Dataset, GtfsFeedDiffItem, GtfsRoute, GtfsStop, MapGtfsDecision, OsmFeature
from app.workbench import _upsert_review_item
def capture_gtfs_decision_replay(session: Session, previous_dataset: Dataset | None, *, diff_run_id: int | None = None) -> dict[str, Any]:
if previous_dataset is None or previous_dataset.kind != "gtfs":
return {"previous_dataset_id": None, "diff_run_id": diff_run_id, "decisions": []}
route_by_id = {
int(route.id): route
for route in session.scalars(select(GtfsRoute).where(GtfsRoute.dataset_id == previous_dataset.id)).all()
}
stop_by_id = {
int(stop.id): stop
for stop in session.scalars(select(GtfsStop).where(GtfsStop.dataset_id == previous_dataset.id)).all()
}
route_ids = list(route_by_id)
stop_ids = list(stop_by_id)
filters = [MapGtfsDecision.gtfs_dataset_id == previous_dataset.id]
if route_ids:
filters.append(MapGtfsDecision.gtfs_route_id.in_(route_ids))
if stop_ids:
filters.append(MapGtfsDecision.gtfs_stop_id.in_(stop_ids))
decisions = session.scalars(
select(MapGtfsDecision)
.where(MapGtfsDecision.active.is_(True), or_(*filters))
.order_by(MapGtfsDecision.updated_at.desc(), MapGtfsDecision.id.desc())
).all()
captured: list[dict[str, Any]] = []
for decision in decisions:
object_type = None
object_key = None
route = route_by_id.get(int(decision.gtfs_route_id or 0))
stop = stop_by_id.get(int(decision.gtfs_stop_id or 0))
if route is not None:
object_type = "route"
object_key = route.route_id
elif stop is not None:
object_type = "stop"
object_key = stop.stop_id
if not object_type or not object_key:
continue
captured.append(
{
"decision_id": decision.id,
"decision_key": decision.decision_key,
"decision_type": decision.decision_type,
"status": decision.status,
"source_id": decision.source_id,
"gtfs_dataset_id": decision.gtfs_dataset_id,
"object_type": object_type,
"object_key": object_key,
"selector": _json_object(decision.selector_json),
"action": _json_object(decision.action_json),
"note": decision.note,
"reviewer": decision.reviewer,
"confidence": decision.confidence,
"osm_dataset_id": decision.osm_dataset_id,
"osm_feature_id": decision.osm_feature_id,
"metadata": _json_object(decision.metadata_json),
}
)
return {
"previous_dataset_id": previous_dataset.id,
"source_id": previous_dataset.source_id,
"diff_run_id": diff_run_id,
"decisions": captured,
}
def apply_gtfs_decision_replay(
session: Session,
replay_plan: dict[str, Any] | None,
*,
new_dataset: Dataset,
diff_run_id: int | None = None,
) -> dict[str, int]:
if not replay_plan:
return {"captured": 0, "replayed": 0, "skipped": 0, "review_items": 0}
decisions = replay_plan.get("decisions") if isinstance(replay_plan.get("decisions"), list) else []
if not decisions and diff_run_id is None:
return {"captured": 0, "replayed": 0, "skipped": 0, "review_items": 0}
current_diff_run_id = diff_run_id or _optional_int(replay_plan.get("diff_run_id"))
changed_keys = _changed_or_removed_keys(session, current_diff_run_id)
routes = {
route.route_id: route
for route in session.scalars(select(GtfsRoute).where(GtfsRoute.dataset_id == new_dataset.id)).all()
}
previous_decisions_by_key: dict[tuple[str, str], list[dict[str, Any]]] = {}
replayed = 0
skipped = 0
for decision in decisions:
object_type = str(decision.get("object_type") or "")
object_key = str(decision.get("object_key") or "")
if not object_type or not object_key:
skipped += 1
continue
previous_decisions_by_key.setdefault((object_type, object_key), []).append(decision)
if (object_type, object_key) in changed_keys:
skipped += 1
continue
if object_type != "route":
# Stop-level operational replay is handled by durable canonical-stop
# MatchRule rows during route-layer rebuild. Keep this audit layer
# route-focused to avoid stale canonical-stop foreign keys.
skipped += 1
continue
route = routes.get(object_key)
if route is None:
skipped += 1
continue
if _replay_route_decision(session, decision, new_dataset=new_dataset, route=route):
replayed += 1
else:
skipped += 1
review_items = create_gtfs_update_decision_review_items(
session,
diff_run_id=current_diff_run_id,
new_dataset=new_dataset,
previous_decisions_by_key=previous_decisions_by_key,
)
session.flush()
return {"captured": len(decisions), "replayed": replayed, "skipped": skipped, "review_items": review_items}
def create_gtfs_update_decision_review_items(
session: Session,
*,
diff_run_id: int | None,
new_dataset: Dataset,
previous_decisions_by_key: dict[tuple[str, str], list[dict[str, Any]]] | None = None,
) -> int:
if diff_run_id is None:
return 0
previous_decisions_by_key = previous_decisions_by_key or {}
routes = {
route.route_id: route
for route in session.scalars(select(GtfsRoute).where(GtfsRoute.dataset_id == new_dataset.id)).all()
}
stops = {
stop.stop_id: stop
for stop in session.scalars(select(GtfsStop).where(GtfsStop.dataset_id == new_dataset.id)).all()
}
items = session.scalars(
select(GtfsFeedDiffItem)
.where(GtfsFeedDiffItem.diff_run_id == diff_run_id, GtfsFeedDiffItem.change_type.in_(["changed", "removed"]))
.order_by(GtfsFeedDiffItem.item_type, GtfsFeedDiffItem.change_type, GtfsFeedDiffItem.object_key)
).all()
created_or_updated = 0
for item in items:
object_type = "route" if item.item_type == "route" else "stop" if item.item_type == "stop" else str(item.item_type)
object_key = str(item.object_key)
previous_decisions = previous_decisions_by_key.get((object_type, object_key), [])
severity = "error" if previous_decisions and item.change_type == "removed" else "warn"
route = routes.get(object_key) if object_type == "route" and item.change_type != "removed" else None
stop = stops.get(object_key) if object_type == "stop" and item.change_type != "removed" else None
title = _diff_review_title(item, previous_decision_count=len(previous_decisions))
metadata = {
"diff_run_id": diff_run_id,
"item_id": item.id,
"item_type": item.item_type,
"object_key": object_key,
"change_type": item.change_type,
"old": _json_object(item.old_payload_json),
"new": _json_object(item.new_payload_json),
"diff": _json_object(item.diff_json),
"previous_decisions": previous_decisions,
}
result = _upsert_review_item(
session,
fingerprint=f"gtfs-update-diff:{diff_run_id}:{item.item_type}:{item.object_key}:{item.change_type}",
queue="gtfs_update_diff",
item_type=f"gtfs_{item.item_type}_update",
severity=severity,
title=title,
description=_diff_review_description(item, len(previous_decisions)),
source_id=new_dataset.source_id,
dataset_id=new_dataset.id,
gtfs_route_id=None if route is None else route.id,
gtfs_stop_id=None if stop is None else stop.id,
evidence_payload=metadata,
)
if result in {"created", "updated", "reopened"}:
created_or_updated += 1
return created_or_updated
def _replay_route_decision(session: Session, decision: dict[str, Any], *, new_dataset: Dataset, route: GtfsRoute) -> bool:
selector = _updated_route_selector(decision.get("selector"), new_dataset=new_dataset, route=route)
action = decision.get("action") if isinstance(decision.get("action"), dict) else {}
evidence = {
"selector": selector,
"action": action,
"confidence": decision.get("confidence"),
"replayed_from_decision_id": decision.get("decision_id"),
}
decision_key = _stable_key(str(decision.get("decision_type") or "route_osm_relation"), selector, action)
existing = session.scalar(select(MapGtfsDecision).where(MapGtfsDecision.decision_key == decision_key))
now = datetime.now(timezone.utc)
osm_dataset_id, osm_feature_id = _existing_osm_reference(session, decision)
metadata = decision.get("metadata") if isinstance(decision.get("metadata"), dict) else {}
replay_metadata = {
**metadata,
"replay": {
"from_decision_id": decision.get("decision_id"),
"from_decision_key": decision.get("decision_key"),
"previous_dataset_id": decision.get("gtfs_dataset_id"),
"new_dataset_id": new_dataset.id,
"object_key": route.route_id,
},
}
if existing is None:
session.add(
MapGtfsDecision(
decision_key=decision_key,
decision_type=str(decision.get("decision_type") or "route_osm_relation"),
status=str(decision.get("status") or "accepted"),
active=True,
source_id=new_dataset.source_id,
gtfs_dataset_id=new_dataset.id,
gtfs_route_id=route.id,
osm_dataset_id=osm_dataset_id,
osm_feature_id=osm_feature_id,
route_match_id=None,
confidence=_optional_float(decision.get("confidence")),
evidence_hash=_hash_payload(evidence),
selector_json=_json_dumps(selector),
action_json=_json_dumps(action),
note=decision.get("note"),
reviewer="decision_replay",
metadata_json=_json_dumps(replay_metadata),
created_at=now,
updated_at=now,
)
)
return True
existing.status = str(decision.get("status") or existing.status)
existing.active = True
existing.source_id = new_dataset.source_id
existing.gtfs_dataset_id = new_dataset.id
existing.gtfs_route_id = route.id
existing.osm_dataset_id = osm_dataset_id
existing.osm_feature_id = osm_feature_id
existing.route_match_id = None
existing.confidence = _optional_float(decision.get("confidence"))
existing.evidence_hash = _hash_payload(evidence)
existing.selector_json = _json_dumps(selector)
existing.action_json = _json_dumps(action)
existing.reviewer = "decision_replay"
existing.metadata_json = _json_dumps(replay_metadata)
existing.updated_at = now
return True
def _updated_route_selector(value: object, *, new_dataset: Dataset, route: GtfsRoute) -> dict[str, Any]:
selector = dict(value) if isinstance(value, dict) else {}
selector["gtfs"] = {
"source_id": new_dataset.source_id,
"dataset_id": new_dataset.id,
"route_id": route.route_id,
"route_key": route.route_key,
"ref": route.short_name,
"mode": route.mode,
}
selector.pop("route_match_id", None)
selector.pop("gtfs_route_id", None)
return selector
def _existing_osm_reference(session: Session, decision: dict[str, Any]) -> tuple[int | None, int | None]:
dataset_id = _optional_int(decision.get("osm_dataset_id"))
feature_id = _optional_int(decision.get("osm_feature_id"))
feature = session.get(OsmFeature, feature_id) if feature_id is not None else None
if feature is not None:
return feature.dataset_id, feature.id
if dataset_id is not None and session.get(Dataset, dataset_id) is not None:
return dataset_id, None
return None, None
def _changed_or_removed_keys(session: Session, diff_run_id: int | None) -> set[tuple[str, str]]:
if diff_run_id is None:
return set()
rows = session.execute(
select(GtfsFeedDiffItem.item_type, GtfsFeedDiffItem.object_key)
.where(GtfsFeedDiffItem.diff_run_id == diff_run_id, GtfsFeedDiffItem.change_type.in_(["changed", "removed"]))
).all()
return {
("route" if item_type == "route" else "stop" if item_type == "stop" else str(item_type), str(object_key))
for item_type, object_key in rows
}
def _diff_review_title(item: GtfsFeedDiffItem, *, previous_decision_count: int) -> str:
prefix = "Recheck previous decision for" if previous_decision_count else "Review updated"
return f"{prefix} {item.item_type} {item.object_key}: {item.change_type}"
def _diff_review_description(item: GtfsFeedDiffItem, previous_decision_count: int) -> str:
if previous_decision_count:
return f"{item.item_type.title()} changed after {previous_decision_count} previous decision(s); manual review is needed before replay."
return f"{item.item_type.title()} was {item.change_type} in the latest feed update."
def _stable_key(decision_type: str, selector: dict[str, Any], action: dict[str, Any]) -> str:
payload = {"decision_type": decision_type, "selector": selector, "action": action}
return f"{decision_type}:{_hash_payload(payload)[:40]}"
def _hash_payload(payload: dict[str, Any]) -> str:
encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def _json_object(value: str | None) -> dict[str, Any]:
if not value:
return {}
try:
payload = json.loads(value)
except json.JSONDecodeError:
return {}
return payload if isinstance(payload, dict) else {}
def _json_dumps(value: dict[str, Any]) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
def _optional_int(value: object) -> int | None:
try:
return None if value is None else int(value)
except (TypeError, ValueError):
return None
def _optional_float(value: object) -> float | None:
try:
return None if value is None else float(value)
except (TypeError, ValueError):
return None

640
app/feed_discovery.py
View File

@@ -22,6 +22,7 @@ MOBILITY_DATABASE_ACCEPTANCE_TEST_URL = (
)
PTNA_GTFS_INDEX_URL = "https://ptna.openstreetmap.de/gtfs/index.html"
PTNA_COUNTRY_URL_TEMPLATE = "https://ptna.openstreetmap.de/gtfs/{country}/index.php"
OPENDATA_OEPNV_DATASETS_URL = "https://www.opendata-oepnv.de/ht/de/datensaetze"
DEFAULT_DISCOVERY_COUNTRIES = ["DE", "AT", "CH", "NL", "DK", "FR", "BE", "LU", "NO", "SE", "FI", "IE", "GB"]
CURATED_TEST_COUNTRIES = ["DE", "CH", "AT", "NL", "DK", "FI", "NO", "SE", "IE", "GB", "FR", "BE", "LU"]
@@ -33,6 +34,11 @@ CANONICAL_HEADERS = [
"subdivision",
"provider",
"feed_name",
"authority_level",
"coverage_scope",
"overlap_role",
"import_policy",
"overlap_group",
"stable_id",
"ptna_feed_id",
"data_type",
@@ -71,6 +77,11 @@ class FeedCandidate:
subdivision: str = ""
provider: str = ""
feed_name: str = ""
authority_level: str = ""
coverage_scope: str = ""
overlap_role: str = ""
import_policy: str = ""
overlap_group: str = ""
stable_id: str = ""
ptna_feed_id: str = ""
data_type: str = "gtfs"
@@ -142,6 +153,11 @@ class FeedCandidate:
notes = _join_notes(notes, f"Mobility Database mirror: {self.latest_url}")
if self.osm_license_text:
notes = _join_notes(notes, f"OSM permission note: {_truncate(self.osm_license_text, 240)}")
if self.import_policy or self.overlap_role:
notes = _join_notes(
notes,
f"Import policy: {self.import_policy or 'review'}; overlap role: {self.overlap_role or 'review'}.",
)
return {
"name": _truncate(name, 240),
"kind": "gtfs",
@@ -166,7 +182,9 @@ def build_gtfs_discovery_manifests(
include_mobility_database: bool = True,
include_acceptance_test_list: bool = True,
include_ptna: bool = True,
include_opendata_oepnv: bool = True,
max_ptna_details: int = 80,
max_opendata_oepnv_details: int = 80,
test_limit: int = 24,
check_urls: bool = False,
timeout: float = 30.0,
@@ -183,8 +201,18 @@ def build_gtfs_discovery_manifests(
candidates.extend(fetch_mobility_acceptance_candidates(countries=selected_countries, timeout=timeout))
if include_ptna:
candidates.extend(fetch_ptna_candidates(countries=selected_countries, max_details=max_ptna_details, timeout=timeout))
if include_opendata_oepnv:
candidates.extend(
fetch_opendata_oepnv_candidates(
countries=selected_countries,
max_details=max_opendata_oepnv_details,
timeout=timeout,
)
)
merged = merge_candidates(candidates)
for candidate in merged:
apply_source_hierarchy(candidate)
ingestable = [candidate for candidate in merged if candidate.selected_url and candidate.data_type == "gtfs"]
if check_urls:
for candidate in ingestable:
@@ -209,6 +237,7 @@ def build_gtfs_discovery_manifests(
"mobility_database": MOBILITY_DATABASE_FEEDS_URL if include_mobility_database else None,
"mobility_acceptance_test_list": MOBILITY_DATABASE_ACCEPTANCE_TEST_URL if include_acceptance_test_list else None,
"ptna": PTNA_GTFS_INDEX_URL if include_ptna else None,
"opendata_oepnv": OPENDATA_OEPNV_DATASETS_URL if include_opendata_oepnv else None,
},
"counts": {
"candidates": len(merged),
@@ -266,6 +295,7 @@ def fetch_mobility_database_candidates(
)
normalize_candidate_geography(candidate)
apply_known_download_overrides(candidate)
apply_source_hierarchy(candidate)
candidate.priority = _candidate_priority(candidate)
candidates.append(candidate)
return candidates
@@ -303,6 +333,7 @@ def fetch_mobility_acceptance_candidates(
)
normalize_candidate_geography(candidate)
apply_known_download_overrides(candidate)
apply_source_hierarchy(candidate)
candidates.append(candidate)
return candidates
@@ -332,6 +363,7 @@ def fetch_ptna_candidates(
detail_fetches += 1
except requests.RequestException:
candidate.notes = _join_notes(candidate.notes, "PTNA detail page could not be fetched during discovery.")
apply_source_hierarchy(candidate)
candidate.priority = _candidate_priority(candidate)
candidates.append(candidate)
return candidates
@@ -383,6 +415,7 @@ def parse_ptna_country_page(html: str, *, country: str, page_url: str) -> list[F
)
normalize_candidate_geography(candidate)
apply_known_download_overrides(candidate)
apply_source_hierarchy(candidate)
candidates.append(candidate)
return candidates
@@ -421,6 +454,134 @@ def parse_ptna_detail_fields(html: str, page_url: str) -> dict[str, str]:
return parsed
@dataclass
class OpenDataOepnvDatasetLink:
slug: str
url: str
title: str = ""
def fetch_opendata_oepnv_candidates(
*,
countries: list[str] | None = None,
max_details: int = 80,
timeout: float = 30.0,
url: str = OPENDATA_OEPNV_DATASETS_URL,
) -> list[FeedCandidate]:
if countries and "DE" not in countries:
return []
try:
html = _fetch_text(url, timeout=timeout)
except requests.RequestException:
return []
dataset_links = parse_opendata_oepnv_dataset_links(html, url)
candidates: list[FeedCandidate] = []
for dataset_link in dataset_links[:max_details]:
try:
detail_html = _fetch_text(dataset_link.url, timeout=timeout)
except requests.RequestException:
candidate = FeedCandidate(
discovery_source="opendata_oepnv",
country="DE",
provider=_opendata_oepnv_provider(dataset_link.url, dataset_link.slug),
feed_name=dataset_link.title or _title_from_slug(dataset_link.slug),
stable_id=f"opendata-oepnv:{dataset_link.slug}",
status="detail_fetch_failed",
details_url=dataset_link.url,
source_basis="OpenData ÖPNV dataset catalog",
notes="OpenData ÖPNV detail page could not be fetched during discovery.",
priority="P4",
)
apply_source_hierarchy(candidate)
candidates.append(candidate)
continue
candidate = parse_opendata_oepnv_detail_page(
detail_html,
detail_url=dataset_link.url,
slug=dataset_link.slug,
title_hint=dataset_link.title,
)
normalize_candidate_geography(candidate)
apply_known_download_overrides(candidate)
apply_source_hierarchy(candidate)
candidate.priority = candidate.priority or _candidate_priority(candidate)
candidates.append(candidate)
return candidates
def parse_opendata_oepnv_dataset_links(html: str, page_url: str = OPENDATA_OEPNV_DATASETS_URL) -> list[OpenDataOepnvDatasetLink]:
titles_by_slug: dict[str, str] = {}
title_pattern = re.compile(
r"<a\s+[^>]*href=[\"'](?P<href>[^\"']*(?:dataset_name%5D|dataset_name\])=[^\"']+)[\"'][^>]*>\s*"
r"<span[^>]*itemprop=[\"']headline[\"'][^>]*>(?P<title>.*?)</span>",
re.IGNORECASE | re.DOTALL,
)
for match in title_pattern.finditer(html):
href = urljoin(page_url, unescape(match.group("href")))
slug = _opendata_oepnv_dataset_slug_from_url(href)
if slug:
titles_by_slug[slug] = _html_text(match.group("title"))
by_slug: dict[str, OpenDataOepnvDatasetLink] = {}
for link in _all_links(html, page_url):
slug = _opendata_oepnv_dataset_slug_from_url(link)
if not slug:
continue
existing = by_slug.get(slug)
candidate = OpenDataOepnvDatasetLink(slug=slug, url=link, title=titles_by_slug.get(slug, ""))
if existing is None or _opendata_oepnv_link_score(candidate.url) > _opendata_oepnv_link_score(existing.url):
by_slug[slug] = candidate
return sorted(by_slug.values(), key=lambda item: (item.title.lower(), item.slug))
def parse_opendata_oepnv_detail_page(
html: str,
*,
detail_url: str,
slug: str = "",
title_hint: str = "",
) -> FeedCandidate:
text = _html_text(html)
title = _opendata_oepnv_title(html) or title_hint or _title_from_slug(slug)
provider = _opendata_oepnv_provider(detail_url, slug)
data_type = _opendata_oepnv_data_type(title, text)
download_urls = _opendata_oepnv_download_urls(html, detail_url)
selected_url = _choose_opendata_oepnv_download_url(download_urls, data_type)
login_required = _opendata_oepnv_requires_login(text)
license_url, license_text = _opendata_oepnv_license(html, detail_url)
status = "active" if selected_url else "review_required"
if login_required:
status = "registration_required"
notes = "Discovered from OpenData ÖPNV; review per-dataset license and authority before publication."
if selected_url:
notes = _join_notes(notes, "Selected the first current GTFS download URL exposed on the detail page.")
if login_required:
notes = _join_notes(notes, "Download requires an OpenData ÖPNV account; do not auto-import without credentials.")
candidate = FeedCandidate(
discovery_source="opendata_oepnv",
country="DE",
subdivision=_opendata_oepnv_subdivision(detail_url, slug),
provider=provider,
feed_name=title,
stable_id=f"opendata-oepnv:{slug}" if slug else "",
data_type=data_type,
status=status,
is_official="",
selected_url=selected_url,
direct_download_url=selected_url,
original_release_url=detail_url,
license_url=license_url,
license_text=license_text,
details_url=detail_url,
features=_opendata_oepnv_features(data_type, title, text),
priority=_opendata_oepnv_priority(provider, data_type, selected_url),
source_basis="OpenData ÖPNV dataset catalog",
notes=notes,
)
apply_source_hierarchy(candidate)
return candidate
def load_curated_ingestable_seed(
*,
countries: list[str] | None = None,
@@ -452,6 +613,7 @@ def load_curated_ingestable_seed(
)
normalize_candidate_geography(candidate)
apply_known_download_overrides(candidate)
apply_source_hierarchy(candidate)
candidates.append(candidate)
return candidates
@@ -609,6 +771,275 @@ def apply_known_download_overrides(candidate: FeedCandidate) -> None:
candidate.notes,
"Selected Mobility Database latest.zip mirror because the catalog direct URL is known to be stale.",
)
if _opendata_oepnv_download_is_registration_gated(candidate):
gated_url = candidate.selected_url or candidate.direct_download_url
candidate.original_release_url = candidate.original_release_url or gated_url
candidate.selected_url = ""
candidate.direct_download_url = ""
candidate.status = "registration_required"
candidate.notes = _join_notes(
candidate.notes,
"OpenData ÖPNV marks this download as registration-required; kept as a review candidate instead of direct ingest.",
)
def apply_source_hierarchy(candidate: FeedCandidate) -> None:
text = _hierarchy_text(candidate)
authority_level = _infer_authority_level(candidate, text)
coverage_scope = _infer_coverage_scope(candidate, text)
overlap_role = _infer_overlap_role(authority_level, coverage_scope, candidate, text)
import_policy = _infer_import_policy(authority_level, overlap_role, candidate)
overlap_group = _infer_overlap_group(candidate, authority_level, coverage_scope, text)
candidate.authority_level = authority_level
candidate.coverage_scope = coverage_scope
candidate.overlap_role = overlap_role
candidate.import_policy = import_policy
candidate.overlap_group = overlap_group
def _hierarchy_text(candidate: FeedCandidate) -> str:
return " ".join(
[
candidate.discovery_source,
candidate.country,
candidate.subdivision,
candidate.provider,
candidate.feed_name,
candidate.stable_id,
candidate.ptna_feed_id,
candidate.status,
_url_hierarchy_text(candidate.selected_url),
_url_hierarchy_text(candidate.direct_download_url),
_url_hierarchy_text(candidate.latest_url),
_url_hierarchy_text(candidate.original_release_url),
_url_hierarchy_text(candidate.details_url),
candidate.source_basis,
candidate.notes,
]
).lower()
def _infer_authority_level(candidate: FeedCandidate, text: str) -> str:
if _is_mirror_candidate(candidate, text):
return "mirror"
discovery_sources = {part.strip() for part in candidate.discovery_source.split(";") if part.strip()}
if discovery_sources and discovery_sources <= {"ptna", "mobility_validator_acceptance"}:
return "secondary_discovery"
if _contains_any(
text,
[
"delfi",
"gtfs.de",
"mobilithek",
"germany-wide",
"deutschlandweite",
"entur",
"bod national",
"national gtfs",
"national feed",
"national timetable",
"opentransportdata.swiss",
"trafiklab",
"samtrafiken",
"transport for ireland",
"transportforireland",
"rejseplanen",
"gtfs.ovapi.nl",
],
):
return "national_official"
if _contains_any(text, ["operator", "verkehrsunternehmen", "rnv", "flixbus", "flixtrain", "sncf", "tfl"]):
return "operator_feed"
if _contains_any(
text,
[
"verkehrsverbund",
"transport association",
"regional authority",
"authority feed",
"vbb",
"vrr",
"vvs",
"vrs",
"mvv",
"hvv",
"rmv",
"nvv",
"nwl",
"avv",
"nah.sh",
"nah-sh",
"bayern",
"baden-württemberg",
"baden-wuerttemberg",
"nordrhein-westfalen",
"nrw",
"thüringen",
"thueringen",
],
):
return "regional_authority"
if "mobility_database" in candidate.discovery_source:
return "aggregator_catalog"
return "unknown"
def _infer_coverage_scope(candidate: FeedCandidate, text: str) -> str:
if _contains_any(
text,
[
"germany-wide",
"deutschlandweite",
"national gtfs",
"national feed",
"national timetable",
"delfi",
"gtfs.de",
"entur",
"transport for ireland",
"transportforireland",
"trafiklab",
"samtrafiken",
"opentransportdata.swiss",
"gtfs.ovapi.nl",
"rejseplanen",
],
):
return "national"
if _contains_any(text, ["europe", "eu/eea", "pan-europe", "flixbus", "flixtrain"]):
return "multi_country"
if _contains_any(
text,
[
"verkehrsverbund",
"regional",
"bundesland",
"vbb",
"vrr",
"vvs",
"vrs",
"mvv",
"hvv",
"rmv",
"nvv",
"nwl",
"avv",
"nah.sh",
"nah-sh",
],
):
return "regional"
if _contains_any(text, ["operator", "verkehrsunternehmen", "rnv"]):
return "operator"
if candidate.country:
return "unknown_country"
return "unknown"
def _infer_overlap_role(authority_level: str, coverage_scope: str, candidate: FeedCandidate, text: str) -> str:
if authority_level == "mirror":
return "bootstrap_mirror"
if authority_level == "secondary_discovery":
return "discovery_evidence"
if candidate.status == "registration_required":
return "review_baseline" if coverage_scope == "national" else "review_candidate"
if authority_level == "national_official" and coverage_scope == "national":
return "canonical_baseline"
if authority_level == "regional_authority":
return "regional_override_candidate"
if authority_level == "operator_feed":
return "operator_detail_candidate"
if authority_level == "aggregator_catalog":
return "catalog_metadata"
if _contains_any(text, ["active", "gtfs"]):
return "gap_fill_candidate"
return "review_candidate"
def _infer_import_policy(authority_level: str, overlap_role: str, candidate: FeedCandidate) -> str:
if candidate.status == "registration_required" or not candidate.selected_url:
return "review_before_import"
if overlap_role == "canonical_baseline":
return "canonical_candidate"
if overlap_role in {"regional_override_candidate", "operator_detail_candidate"}:
return "qa_or_override_not_additive"
if overlap_role == "bootstrap_mirror":
return "bootstrap_only"
if authority_level in {"secondary_discovery", "aggregator_catalog"}:
return "metadata_only"
return "review_before_import"
def _infer_overlap_group(candidate: FeedCandidate, authority_level: str, coverage_scope: str, text: str) -> str:
country = candidate.country or "unknown"
if coverage_scope == "national":
return f"{country}:national"
identity_text = _hierarchy_identity_text(candidate)
for token in [
"vbb",
"vrr",
"vvs",
"vrs",
"mvv",
"hvv",
"rmv",
"nvv",
"nwl",
"avv",
"nah-sh",
"nah.sh",
"rnv",
]:
if token in identity_text:
normalized = token.replace(".", "-")
return f"{country}:{coverage_scope}:{normalized}"
key = _source_key(candidate.provider or candidate.feed_name)
if key:
return f"{country}:{coverage_scope}:{key}"
return f"{country}:{coverage_scope}:{authority_level}"
def _is_mirror_candidate(candidate: FeedCandidate, text: str) -> bool:
if any(_url_is_mirror_url(url) for url in [candidate.selected_url, candidate.direct_download_url, candidate.latest_url if not candidate.selected_url else ""]):
return True
mirror_notes = " ".join([candidate.source_basis, candidate.notes]).lower()
return _contains_any(mirror_notes, ["european transport feeds mirror", "bootstrap mirror"])
def _source_key(value: str) -> str:
cleaned = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-")
return cleaned[:80]
def _contains_any(text: str, tokens: Iterable[str]) -> bool:
return any(token in text for token in tokens)
def _hierarchy_identity_text(candidate: FeedCandidate) -> str:
return " ".join(
[
candidate.provider,
candidate.feed_name,
candidate.stable_id,
candidate.ptna_feed_id,
_url_hierarchy_text(candidate.selected_url),
_url_hierarchy_text(candidate.direct_download_url),
_url_hierarchy_text(candidate.original_release_url),
_url_hierarchy_text(candidate.details_url),
]
).lower()
def _url_hierarchy_text(url: str) -> str:
parsed = urlparse(url or "")
return " ".join(part for part in [parsed.netloc, parsed.path] if part)
def _url_is_mirror_url(url: str) -> bool:
parsed = urlparse(url or "")
host_path = f"{parsed.netloc.lower()}{parsed.path.lower()}"
return _contains_any(host_path, ["files.mobilitydatabase.org", "openmobilitydata-data.s3", "scraped.data.public-transport.earth"])
@dataclass
@@ -732,6 +1163,211 @@ def _looks_like_download_url(url: str) -> bool:
return False
def _opendata_oepnv_dataset_slug_from_url(url: str) -> str:
query = parse_qs(urlparse(unescape(url)).query)
return _clean_text((query.get("tx_vrrkit_view[dataset_name]") or [""])[0])
def _opendata_oepnv_link_score(url: str) -> int:
parsed = urlparse(url)
score = 0
if "/organisation/" in parsed.path:
score += 2
if parsed.path.rstrip("/").endswith("startseite"):
score += 1
return score
def _opendata_oepnv_title(html: str) -> str:
matches = re.findall(r"<div[^>]*class=[\"'][^\"']*element-header[^\"']*[\"'][^>]*>(.*?)</div>", html, re.IGNORECASE | re.DOTALL)
for raw in reversed(matches):
title = _html_text(raw)
if title and title.lower() not in {"hinweise", "spielregeln", "lizenzen"}:
return title
match = re.search(r"<title>(.*?)</title>", html, re.IGNORECASE | re.DOTALL)
return _html_text(match.group(1)) if match else ""
def _opendata_oepnv_download_urls(html: str, detail_url: str) -> list[str]:
urls: list[str] = []
for link in _all_links(html, detail_url):
normalized = _normalize_feed_url(link)
if normalized and _looks_like_download_url(normalized) and normalized not in urls:
urls.append(normalized)
return urls
def _choose_opendata_oepnv_download_url(urls: list[str], data_type: str) -> str:
if data_type == "gtfs_rt":
for url in urls:
lower = url.lower()
if "realtime" in lower or "gtfs-rt" in lower or "gtfs_rt" in lower:
return url
return ""
if data_type != "gtfs":
return ""
if not urls:
return ""
static_urls = [url for url in urls if "realtime" not in url.lower() and "gtfs-rt" not in url.lower() and "gtfs_rt" not in url.lower()]
if not static_urls:
return ""
for url in static_urls:
lower = url.lower()
if "latest" in lower or "aktueller" in lower or lower.rstrip("/").endswith("/gtfs.zip"):
return url
return static_urls[0]
def _opendata_oepnv_license(html: str, detail_url: str) -> tuple[str, str]:
match = re.search(
r"id=[\"']dodp_license[\"'][^>]*>.*?<a\s+[^>]*href=[\"'](?P<href>[^\"']+)[\"'][^>]*>(?P<text>.*?)</a>",
html,
re.IGNORECASE | re.DOTALL,
)
if not match:
return "", ""
return urljoin(detail_url, unescape(match.group("href"))), _html_text(match.group("text"))
def _opendata_oepnv_requires_login(text: str) -> bool:
lower = text.lower()
return (
"available for registered users" in lower
or "download is only available for registered users" in lower
or "this download is only available for registered users" in lower
or "anmeldung erforderlich" in lower
or "nur für registrierte nutzer" in lower
or "nur fuer registrierte nutzer" in lower
)
def _opendata_oepnv_data_type(title: str, text: str) -> str:
title_lower = title.lower()
lower = f"{title} {text}".lower()
if "gtfs-rt" in lower or "gtfs realtime" in lower or "gtfs-realtime" in lower:
return "gtfs_rt"
if "gtfs" in title_lower:
return "gtfs"
if "haltestell" in title_lower or "zhv" in title_lower:
return "stops"
if "netex" in title_lower:
return "netex"
if "trias" in lower:
return "api"
if "netex" in lower:
return "netex"
if "gtfs" in lower:
return "gtfs"
if "haltestell" in lower or "zhv" in lower:
return "stops"
if "linien" in lower:
return "line_metadata"
return "dataset"
def _opendata_oepnv_features(data_type: str, title: str, text: str) -> str:
features = ["OpenData ÖPNV"]
if data_type == "gtfs":
features.extend(["GTFS", "Schedules"])
elif data_type == "netex":
features.extend(["NeTEx", "Schedules"])
elif data_type == "gtfs_rt":
features.extend(["GTFS-Realtime"])
elif data_type == "stops":
features.append("Stops")
lower = f"{title} {text}".lower()
if "regionalbahn" in lower or "s-bahn" in lower or "bahn" in lower:
features.append("Rail")
if "u-bahn" in lower or "metro" in lower:
features.append("Metro")
if "tram" in lower or "straßenbahn" in lower:
features.append("Tram")
if "bus" in lower:
features.append("Bus")
return "|".join(dict.fromkeys(features))
def _opendata_oepnv_priority(provider: str, data_type: str, selected_url: str) -> str:
if provider == "DELFI e.V." and data_type in {"gtfs", "netex", "stops"}:
return "P0"
if data_type == "gtfs" and selected_url:
return "P1"
if data_type in {"netex", "gtfs_rt", "stops"}:
return "P2"
return "P3"
def _opendata_oepnv_provider(detail_url: str, slug: str) -> str:
path = urlparse(detail_url).path.lower()
haystack = f"{path}/{slug.lower()}/"
providers = {
"delfi": "DELFI e.V.",
"nah-sh": "NAH.SH",
"avv": "Aachener Verkehrsverbund",
"baden-wuerttemberg": "Baden-Württemberg",
"bawue": "Baden-Württemberg",
"hvv": "Hamburger Verkehrsverbund",
"mvv": "Münchner Verkehrs- und Tarifverbund",
"nvv": "Nordhessischer VerkehrsVerbund",
"nwl": "Nahverkehr Westfalen-Lippe",
"nordrhein-westfalen": "Nordrhein-Westfalen",
"nrw": "Nordrhein-Westfalen",
"rmv": "Rhein-Main-Verkehrsverbund",
"rnv": "Rhein-Neckar-Verkehr",
"thueringen": "Thüringen",
"vbb": "Verkehrsverbund Berlin-Brandenburg",
"vrr": "Verkehrsverbund Rhein-Ruhr",
"vrs": "Verkehrsverbund Rhein-Sieg",
"vvs": "Verkehrs- und Tarifverbund Stuttgart",
}
for token, provider in providers.items():
if f"/{token}/" in haystack or f"-{token}" in haystack or f"_{token}" in haystack:
return provider
return "OpenData ÖPNV"
def _opendata_oepnv_subdivision(detail_url: str, slug: str) -> str:
text = f"{urlparse(detail_url).path} {slug}".lower()
subdivisions = {
"baden-wuerttemberg": "Baden-Württemberg",
"bawue": "Baden-Württemberg",
"thueringen": "Thüringen",
"nrw": "Nordrhein-Westfalen",
"nordrhein-westfalen": "Nordrhein-Westfalen",
"vbb": "Berlin-Brandenburg",
"hvv": "Hamburg",
"mvv": "Bayern",
"vrr": "Nordrhein-Westfalen",
"vrs": "Nordrhein-Westfalen",
"nwl": "Nordrhein-Westfalen",
"avv": "Nordrhein-Westfalen",
"rmv": "Hessen",
"nvv": "Hessen",
"rnv": "Baden-Württemberg/Rheinland-Pfalz/Hessen",
"vvs": "Baden-Württemberg",
}
for token, subdivision in subdivisions.items():
if token in text:
return subdivision
return ""
def _opendata_oepnv_download_is_registration_gated(candidate: FeedCandidate) -> bool:
url_text = " ".join([candidate.selected_url, candidate.direct_download_url, candidate.original_release_url]).lower()
if "opendata-oepnv.de" not in url_text:
return False
descriptive_text = " ".join([candidate.provider, candidate.feed_name, candidate.status, candidate.notes]).lower()
return "registration required" in descriptive_text or "user registration required" in descriptive_text
def _html_text(value: str) -> str:
return _clean_text(re.sub(r"<[^>]+>", " ", value or ""))
def _title_from_slug(slug: str) -> str:
return _clean_text(slug.replace("_", " ").replace("-", " ").title())
def _normalize_feed_url(url: str) -> str:
cleaned = _clean_text(url)
if not cleaned:
@@ -800,6 +1436,10 @@ def _merge_candidate(existing: FeedCandidate, incoming: FeedCandidate) -> None:
new_value = getattr(incoming, field_name, "")
if new_value:
setattr(existing, field_name, new_value)
if existing.selected_url and incoming.selected_url and _url_is_mirror_url(existing.selected_url) and not _url_is_mirror_url(incoming.selected_url):
existing.selected_url = incoming.selected_url
existing.direct_download_url = incoming.direct_download_url or incoming.selected_url
existing.notes = _join_notes(existing.notes, "Selected non-mirror publisher URL from merged source evidence.")
existing.discovery_source = _join_unique(existing.discovery_source, incoming.discovery_source)
for field_name in CANONICAL_HEADERS:
if field_name == "candidate_id":

440
app/gtfs_diff.py Normal file
View File

@@ -0,0 +1,440 @@
from __future__ import annotations
import hashlib
import json
import sqlite3
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from sqlalchemy import func, select
from sqlalchemy.orm import Session, joinedload
from app.models import Dataset, GtfsFeedDiffItem, GtfsFeedDiffRun, GtfsRoute, GtfsStop, GtfsTrip, Source
MAX_DIFF_ITEMS_PER_TYPE = 200
def create_gtfs_update_diff_from_stage(
session: Session,
*,
source: Source,
previous_dataset: Dataset | None,
new_dataset: Dataset,
stage_path: Path,
) -> dict[str, Any] | None:
if previous_dataset is None or previous_dataset.kind != "gtfs" or not stage_path.exists():
return None
old_routes = _main_routes(session, previous_dataset.id)
old_stops = _main_stops(session, previous_dataset.id)
old_counts = {
"routes": len(old_routes),
"stops": len(old_stops),
"trips": _main_count(session, GtfsTrip, previous_dataset.id),
}
with sqlite3.connect(stage_path) as connection:
connection.row_factory = sqlite3.Row
new_routes = _stage_routes(connection)
new_stops = _stage_stops(connection)
new_counts = {
"routes": len(new_routes),
"stops": len(new_stops),
"trips": _stage_count(connection, "gtfs_trips"),
}
route_summary, route_items = _diff_objects(
item_type="route",
old=old_routes,
new=new_routes,
identity_label="route",
)
stop_summary, stop_items = _diff_objects(
item_type="stop",
old=old_stops,
new=new_stops,
identity_label="stop",
)
summary = {
"source_id": source.id,
"previous_dataset_id": previous_dataset.id,
"new_dataset_id": new_dataset.id,
"counts": {"old": old_counts, "new": new_counts},
"routes": route_summary,
"stops": stop_summary,
"items_stored": {
"route": min(len(route_items), MAX_DIFF_ITEMS_PER_TYPE),
"stop": min(len(stop_items), MAX_DIFF_ITEMS_PER_TYPE),
},
}
diff_key = _diff_key(source.id, previous_dataset.id, new_dataset.id, summary)
existing = session.scalar(select(GtfsFeedDiffRun).where(GtfsFeedDiffRun.diff_key == diff_key))
if existing is not None:
return diff_run_payload(existing)
now = datetime.now(timezone.utc)
diff_run = GtfsFeedDiffRun(
source_id=source.id,
previous_dataset_id=previous_dataset.id,
new_dataset_id=new_dataset.id,
status="completed",
diff_key=diff_key,
summary_json=_json_dumps(summary),
metadata_json=_json_dumps({"stage_path": str(stage_path), "created_from": "gtfs_activation"}),
created_at=now,
)
session.add(diff_run)
session.flush()
for item in [*route_items[:MAX_DIFF_ITEMS_PER_TYPE], *stop_items[:MAX_DIFF_ITEMS_PER_TYPE]]:
session.add(
GtfsFeedDiffItem(
diff_run_id=diff_run.id,
source_id=source.id,
previous_dataset_id=previous_dataset.id,
new_dataset_id=new_dataset.id,
item_type=item["item_type"],
object_key=item["object_key"],
change_type=item["change_type"],
severity=item["severity"],
status="open",
title=item["title"],
old_payload_json=_json_dumps(item["old"]) if item.get("old") else None,
new_payload_json=_json_dumps(item["new"]) if item.get("new") else None,
diff_json=_json_dumps(item["diff"]) if item.get("diff") else None,
created_at=now,
)
)
session.flush()
return diff_run_payload(diff_run)
def latest_gtfs_update_diffs(session: Session, *, source_id: int | None = None, limit: int = 20) -> list[GtfsFeedDiffRun]:
stmt = (
select(GtfsFeedDiffRun)
.options(joinedload(GtfsFeedDiffRun.source))
.order_by(GtfsFeedDiffRun.created_at.desc(), GtfsFeedDiffRun.id.desc())
.limit(max(1, min(int(limit), 100)))
)
if source_id is not None:
stmt = stmt.where(GtfsFeedDiffRun.source_id == int(source_id))
return session.scalars(stmt).all()
def gtfs_update_diff_detail(session: Session, diff_id: int, *, item_limit: int = 200) -> dict[str, Any] | None:
diff_run = session.get(GtfsFeedDiffRun, diff_id)
if diff_run is None:
return None
items = session.scalars(
select(GtfsFeedDiffItem)
.where(GtfsFeedDiffItem.diff_run_id == diff_run.id)
.order_by(_change_type_order(), GtfsFeedDiffItem.item_type, GtfsFeedDiffItem.object_key)
.limit(max(1, min(int(item_limit), 1000)))
).all()
return {**diff_run_payload(diff_run), "items": [diff_item_payload(item) for item in items]}
def gtfs_update_diff_summary(session: Session) -> dict[str, Any]:
runs = session.scalar(select(func.count()).select_from(GtfsFeedDiffRun)) or 0
rows = session.execute(
select(GtfsFeedDiffItem.item_type, GtfsFeedDiffItem.change_type, GtfsFeedDiffItem.severity, func.count())
.select_from(GtfsFeedDiffItem)
.group_by(GtfsFeedDiffItem.item_type, GtfsFeedDiffItem.change_type, GtfsFeedDiffItem.severity)
).all()
by_type: dict[str, int] = {}
by_change: dict[str, int] = {}
by_severity: dict[str, int] = {}
for item_type, change_type, severity, count in rows:
value = int(count or 0)
by_type[str(item_type)] = by_type.get(str(item_type), 0) + value
by_change[str(change_type)] = by_change.get(str(change_type), 0) + value
by_severity[str(severity)] = by_severity.get(str(severity), 0) + value
return {"runs": int(runs), "by_type": by_type, "by_change": by_change, "by_severity": by_severity}
def diff_run_payload(diff_run: GtfsFeedDiffRun) -> dict[str, Any]:
summary = _json_object(diff_run.summary_json)
return {
"id": diff_run.id,
"source_id": diff_run.source_id,
"source_name": None if diff_run.source is None else diff_run.source.name,
"previous_dataset_id": diff_run.previous_dataset_id,
"new_dataset_id": diff_run.new_dataset_id,
"status": diff_run.status,
"diff_key": diff_run.diff_key,
"summary": summary,
"metadata": _json_object(diff_run.metadata_json),
"created_at": diff_run.created_at.isoformat(),
}
def diff_item_payload(item: GtfsFeedDiffItem) -> dict[str, Any]:
return {
"id": item.id,
"diff_run_id": item.diff_run_id,
"source_id": item.source_id,
"previous_dataset_id": item.previous_dataset_id,
"new_dataset_id": item.new_dataset_id,
"item_type": item.item_type,
"object_key": item.object_key,
"change_type": item.change_type,
"severity": item.severity,
"status": item.status,
"title": item.title,
"old": _json_object(item.old_payload_json),
"new": _json_object(item.new_payload_json),
"diff": _json_object(item.diff_json),
"created_at": item.created_at.isoformat(),
"resolved_at": None if item.resolved_at is None else item.resolved_at.isoformat(),
}
def _main_routes(session: Session, dataset_id: int) -> dict[str, dict[str, Any]]:
trip_counts = {
str(route_id): int(count or 0)
for route_id, count in session.execute(
select(GtfsTrip.route_id, func.count()).where(GtfsTrip.dataset_id == dataset_id).group_by(GtfsTrip.route_id)
).all()
}
routes = session.scalars(select(GtfsRoute).where(GtfsRoute.dataset_id == dataset_id)).all()
return {
str(route.route_id): _normalized_route_payload(
{
"route_id": route.route_id,
"short_name": route.short_name,
"long_name": route.long_name,
"mode": route.mode,
"route_scope": route.route_scope,
"operator_name": route.operator_name,
"route_key": route.route_key,
"operator_key": route.operator_key,
"has_geometry": bool(route.geometry_geojson),
"trip_count": trip_counts.get(str(route.route_id), 0),
}
)
for route in routes
}
def _stage_routes(connection: sqlite3.Connection) -> dict[str, dict[str, Any]]:
trip_counts = {
str(row["route_id"]): int(row["count"] or 0)
for row in connection.execute("SELECT route_id, COUNT(*) AS count FROM gtfs_trips GROUP BY route_id").fetchall()
}
rows = connection.execute(
"""
SELECT route_id, short_name, long_name, mode, route_scope, operator_name, route_key, operator_key, geometry_geojson
FROM gtfs_routes
"""
).fetchall()
return {
str(row["route_id"]): _normalized_route_payload(
{
"route_id": row["route_id"],
"short_name": row["short_name"],
"long_name": row["long_name"],
"mode": row["mode"],
"route_scope": row["route_scope"],
"operator_name": row["operator_name"],
"route_key": row["route_key"],
"operator_key": row["operator_key"],
"has_geometry": bool(row["geometry_geojson"]),
"trip_count": trip_counts.get(str(row["route_id"]), 0),
}
)
for row in rows
}
def _main_stops(session: Session, dataset_id: int) -> dict[str, dict[str, Any]]:
stops = session.scalars(select(GtfsStop).where(GtfsStop.dataset_id == dataset_id)).all()
return {
str(stop.stop_id): _normalized_stop_payload(
{
"stop_id": stop.stop_id,
"name": stop.name,
"lat": stop.lat,
"lon": stop.lon,
"parent_station": stop.parent_station,
}
)
for stop in stops
}
def _stage_stops(connection: sqlite3.Connection) -> dict[str, dict[str, Any]]:
rows = connection.execute("SELECT stop_id, name, lat, lon, parent_station FROM gtfs_stops").fetchall()
return {
str(row["stop_id"]): _normalized_stop_payload(
{
"stop_id": row["stop_id"],
"name": row["name"],
"lat": row["lat"],
"lon": row["lon"],
"parent_station": row["parent_station"],
}
)
for row in rows
}
def _diff_objects(
*,
item_type: str,
old: dict[str, dict[str, Any]],
new: dict[str, dict[str, Any]],
identity_label: str,
) -> tuple[dict[str, int], list[dict[str, Any]]]:
old_keys = set(old)
new_keys = set(new)
added = sorted(new_keys - old_keys)
removed = sorted(old_keys - new_keys)
common = sorted(old_keys & new_keys)
items: list[dict[str, Any]] = []
unchanged = 0
changed = 0
for key in common:
differences = _payload_differences(old[key], new[key])
if not differences:
unchanged += 1
continue
changed += 1
items.append(
{
"item_type": item_type,
"object_key": key,
"change_type": "changed",
"severity": "warn",
"title": f"{identity_label.title()} changed: {key}",
"old": old[key],
"new": new[key],
"diff": differences,
}
)
for key in added:
items.append(
{
"item_type": item_type,
"object_key": key,
"change_type": "added",
"severity": "info",
"title": f"{identity_label.title()} added: {key}",
"new": new[key],
"diff": {"added": True},
}
)
for key in removed:
items.append(
{
"item_type": item_type,
"object_key": key,
"change_type": "removed",
"severity": "warn",
"title": f"{identity_label.title()} removed: {key}",
"old": old[key],
"diff": {"removed": True},
}
)
summary = {
"old": len(old),
"new": len(new),
"added": len(added),
"removed": len(removed),
"changed": changed,
"unchanged": unchanged,
}
return summary, sorted(items, key=lambda item: (_change_type_rank(item["change_type"]), item["item_type"], item["object_key"]))
def _payload_differences(old: dict[str, Any], new: dict[str, Any]) -> dict[str, dict[str, Any]]:
differences = {}
for key in sorted(set(old) | set(new)):
if old.get(key) != new.get(key):
differences[key] = {"old": old.get(key), "new": new.get(key)}
return differences
def _normalized_route_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"route_id": str(payload.get("route_id") or ""),
"short_name": _clean_text(payload.get("short_name")),
"long_name": _clean_text(payload.get("long_name")),
"mode": _clean_text(payload.get("mode")),
"route_scope": _clean_text(payload.get("route_scope")),
"operator_name": _clean_text(payload.get("operator_name")),
"route_key": _clean_text(payload.get("route_key")),
"operator_key": _clean_text(payload.get("operator_key")),
"has_geometry": bool(payload.get("has_geometry")),
"trip_count": int(payload.get("trip_count") or 0),
}
def _normalized_stop_payload(payload: dict[str, Any]) -> dict[str, Any]:
return {
"stop_id": str(payload.get("stop_id") or ""),
"name": _clean_text(payload.get("name")),
"lat": _rounded_coord(payload.get("lat")),
"lon": _rounded_coord(payload.get("lon")),
"parent_station": _clean_text(payload.get("parent_station")),
}
def _main_count(session: Session, model, dataset_id: int) -> int:
return int(session.scalar(select(func.count()).select_from(model).where(model.dataset_id == dataset_id)) or 0)
def _stage_count(connection: sqlite3.Connection, table_name: str) -> int:
return int(connection.execute(f"SELECT COUNT(*) FROM {table_name}").fetchone()[0] or 0)
def _diff_key(source_id: int, previous_dataset_id: int, new_dataset_id: int, summary: dict[str, Any]) -> str:
payload = {
"source_id": source_id,
"previous_dataset_id": previous_dataset_id,
"new_dataset_id": new_dataset_id,
"routes": summary.get("routes"),
"stops": summary.get("stops"),
}
digest = hashlib.sha256(_json_dumps(payload).encode("utf-8")).hexdigest()
return f"gtfs-diff-{digest[:32]}"
def _change_type_order():
from sqlalchemy import case
return case(
(GtfsFeedDiffItem.change_type == "changed", 0),
(GtfsFeedDiffItem.change_type == "removed", 1),
(GtfsFeedDiffItem.change_type == "added", 2),
else_=9,
)
def _change_type_rank(value: str) -> int:
return {"changed": 0, "removed": 1, "added": 2}.get(value, 9)
def _clean_text(value: object) -> str | None:
if value is None:
return None
text = " ".join(str(value).strip().split())
return text or None
def _rounded_coord(value: object) -> float | None:
try:
return round(float(value), 7)
except (TypeError, ValueError):
return None
def _json_object(value: str | None) -> dict[str, Any]:
if not value:
return {}
try:
payload = json.loads(value)
except json.JSONDecodeError:
return {}
return payload if isinstance(payload, dict) else {}
def _json_dumps(value: dict[str, Any]) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)

1278
app/harmonization.py
View File

File diff suppressed because it is too large Load Diff

131
app/jobs.py
View File

@@ -23,6 +23,7 @@ from app.data_management import (
from app.db import SessionLocal, engine, init_db
from app.db_lock import DatabaseWriteBusy, database_write_lock
from app.gtfs_storage import missing_sidecar_paths as gtfs_missing_sidecar_paths
from app.harmonization import create_gtfs_harmonized_snapshot
from app.models import Dataset, Job, JobEvent, Source, SourceUpdateCheck
from app.osm_storage import missing_sidecar_paths as osm_missing_sidecar_paths
from app.pipeline.gtfs import backfill_gtfs_shapes
@@ -34,6 +35,7 @@ from app.pipeline.route_layer import rebuild_route_layer
from app.pipeline.run import run_source
from app.pipeline.sample_data import clear_project_data, load_sample_project
from app.source_catalog import import_ingestable_sources, import_source_catalog, source_catalog_summary
from app.workbench import generate_map_gtfs_review_items
ROUTE_MATCHING_JOB_KIND = "route_matching"
@@ -44,6 +46,8 @@ SOURCE_IMPORT_JOB_KIND = "source_import"
SOURCE_DELETE_JOB_KIND = "source_delete"
DATASET_DELETE_JOB_KIND = "dataset_delete"
MAINTENANCE_JOB_KIND = "maintenance"
MAP_GTFS_REVIEW_JOB_KIND = "map_gtfs_review"
GTFS_HARMONIZED_SNAPSHOT_JOB_KIND = "gtfs_harmonized_snapshot"
TERMINAL_JOB_STATUSES = {"completed", "failed", "cancelled"}
ACTIVE_JOB_STATUSES = {"queued", "running", "paused"}
LEASE_SECONDS = max(300, int(settings.queue_job_lease_seconds))
@@ -124,6 +128,70 @@ def create_route_matching_job(session: Session, *, priority: int = 0) -> Job:
return job
def create_map_gtfs_review_job(
session: Session,
*,
source_id: int | None = None,
dataset_id: int | None = None,
limit: int = 1000,
priority: int = 0,
) -> Job:
result = {
"source_id": source_id,
"dataset_id": dataset_id,
"limit": max(1, min(int(limit), 50_000)),
}
job = Job(
kind=MAP_GTFS_REVIEW_JOB_KIND,
status="queued",
description="Generate Map/GTFS workbench review queue",
progress_current=0,
progress_total=2,
priority=int(priority),
result_json=json.dumps(result, separators=(",", ":")),
)
session.add(job)
session.flush()
add_job_event(
session,
job,
event_type="queued",
message="Map/GTFS review generation queued.",
progress_current=0,
progress_total=job.progress_total,
metadata=result,
)
return job
def create_gtfs_harmonized_snapshot_job(session: Session, *, activate: bool = True, priority: int = 0) -> Job:
result = {
"activate": bool(activate),
"queued_pid": os.getpid(),
}
job = Job(
kind=GTFS_HARMONIZED_SNAPSHOT_JOB_KIND,
status="queued",
description="Build harmonized GTFS snapshot",
progress_current=0,
progress_total=2,
priority=int(priority),
result_json=json.dumps(result, separators=(",", ":")),
)
session.add(job)
session.flush()
add_job_event(
session,
job,
event_type="queued",
message="Harmonized GTFS snapshot build queued.",
progress_current=0,
progress_total=job.progress_total,
metadata=result,
)
return job
def create_osm_relabel_job(
session: Session,
*,
@@ -855,12 +923,14 @@ def job_queue_revision(session: Session, *, include_dismissed: bool = False) ->
def job_events(session: Session, job_id: int, *, limit: int = 100) -> list[JobEvent]:
return session.scalars(
event_limit = max(1, min(limit, 500))
events = session.scalars(
select(JobEvent)
.where(JobEvent.job_id == job_id)
.order_by(JobEvent.created_at, JobEvent.id)
.limit(max(1, min(limit, 500)))
.order_by(JobEvent.created_at.desc(), JobEvent.id.desc())
.limit(event_limit)
).all()
return list(reversed(events))
def request_job_control(job_id: int, action: str) -> dict[str, Any]:
@@ -990,6 +1060,10 @@ def run_claimed_job(job_id: int, worker_id: str) -> None:
_run_dataset_delete_job(job_id, worker_id)
elif job.kind == MAINTENANCE_JOB_KIND:
_run_maintenance_job(job_id, worker_id)
elif job.kind == MAP_GTFS_REVIEW_JOB_KIND:
_run_map_gtfs_review_job(job_id, worker_id)
elif job.kind == GTFS_HARMONIZED_SNAPSHOT_JOB_KIND:
_run_gtfs_harmonized_snapshot_job(job_id, worker_id)
else:
raise ValueError(f"unsupported job kind: {job.kind}")
except JobPaused:
@@ -1292,6 +1366,46 @@ def _run_osm_relabel_job(job_id: int, worker_id: str) -> None:
session.commit()
def _run_map_gtfs_review_job(job_id: int, worker_id: str) -> None:
init_db()
with SessionLocal() as session:
job = _job_for_worker(session, job_id, worker_id)
options = _json_object(job.result_json)
_job_running(session, job, worker_id, "started", "Generating Map/GTFS workbench review queue.", 1)
_check_job_control(session, job)
progress_callback = _job_progress_callback(session, job, worker_id, update_job_progress=False)
result = generate_map_gtfs_review_items(
session,
source_id=_optional_int(options.get("source_id")),
dataset_id=_optional_int(options.get("dataset_id")),
limit=int(options.get("limit") or 1000),
progress_callback=progress_callback,
)
job = _job_for_worker(session, job_id, worker_id)
_complete_job(session, job, "Map/GTFS workbench review queue generated.", {**options, "review_result": result})
session.commit()
def _run_gtfs_harmonized_snapshot_job(job_id: int, worker_id: str) -> None:
init_db()
with SessionLocal() as session:
job = _job_for_worker(session, job_id, worker_id)
options = _json_object(job.result_json)
options["worker_pid"] = os.getpid()
options["worker_id"] = worker_id
job.result_json = json.dumps(options, separators=(",", ":"))
_job_running(session, job, worker_id, "started", "Building harmonized GTFS snapshot.", 1)
_check_job_control(session, job)
snapshot = create_gtfs_harmonized_snapshot(
session,
activate=bool(options.get("activate", True)),
note=f"job #{job.id}",
)
job = _job_for_worker(session, job_id, worker_id)
_complete_job(session, job, "Harmonized GTFS snapshot built.", {**options, "snapshot": snapshot})
session.commit()
def _run_source_import_job(job_id: int, worker_id: str) -> None:
init_db()
with SessionLocal() as session:
@@ -1335,7 +1449,16 @@ def _run_source_import_job(job_id: int, worker_id: str) -> None:
if options.get("run_match"):
_job_running(session, job, worker_id, "matching", "Running route matcher after import.", 3)
progress_callback = _job_progress_callback(session, job, worker_id)
match_result = run_route_matching(session, progress_callback=progress_callback)
imported_gtfs_dataset_id = (
int(options["dataset_id"])
if options.get("dataset_kind") == "gtfs" and options.get("dataset_id")
else None
)
match_result = run_route_matching(
session,
progress_callback=progress_callback,
gtfs_dataset_ids=[imported_gtfs_dataset_id] if imported_gtfs_dataset_id is not None else None,
)
job = _job_for_worker(session, job_id, worker_id)
options = _json_object(job.result_json)
options["match_result"] = match_result

1116
app/journey.py
View File

File diff suppressed because it is too large Load Diff

254
app/journey_search.py
View File

@@ -13,8 +13,16 @@ from typing import Any
from sqlalchemy import select
from app.address_search import is_location_token
from app.config import settings
from app.db import SessionLocal
from app.journey import find_journeys, parse_service_date, resolve_location_summary
from app.harmonization import active_harmonized_gtfs_dataset_ids
from app.journey import (
find_journeys,
journey_duplicate_preference_key,
journey_equivalence_key,
parse_service_date,
resolve_location_summary,
)
from app.models import JourneySearchCache
from app.routing import direct_route_between_points, route_between_points
@@ -24,8 +32,15 @@ TRANSIT_STAGE_CACHE_TTL_SECONDS = 5 * 60
TRANSIT_STAGE_CACHE_MAX_ENTRIES = 256
PROGRESSIVE_SEARCH_CACHE_TTL_SECONDS = 10 * 60
PROGRESSIVE_SEARCH_CACHE_MAX_ENTRIES = 128
JOURNEY_SEARCH_CACHE_VERSION = "journey-search-v7"
_executor = ThreadPoolExecutor(max_workers=2, thread_name_prefix="journey-search")
JOURNEY_SEARCH_CACHE_VERSION = "journey-search-v13-route-shadow"
LOW_WALK_MIN_SAVINGS_M = 250
LOW_WALK_MIN_RELATIVE_SAVING = 0.35
LOW_WALK_MAX_EXTRA_SECONDS = 45 * 60
LOW_WALK_GOOD_ENOUGH_M = 350
_executor = ThreadPoolExecutor(
max_workers=max(1, min(int(settings.journey_search_worker_count), 16)),
thread_name_prefix="journey-search",
)
_lock = threading.RLock()
_searches: dict[str, "_SearchState"] = {}
_progressive_search_inflight: dict[tuple[object, ...], str] = {}
@@ -106,9 +121,9 @@ def _run_search(search_id: str) -> None:
state.updated_at = time.time()
request = dict(state.request)
try:
mode = str(request.get("mode") or "transit")
if mode in {"walk", "drive", "car"}:
_run_point_route_search(search_id, "drive" if mode == "car" else mode, request)
mode = _canonical_search_mode(request.get("mode"))
if mode in {"walk", "bike", "drive"}:
_run_point_route_search(search_id, mode, request)
else:
_run_transit_search(search_id, request)
except Exception as exc: # noqa: BLE001 - report progressive-search failure to client
@@ -129,25 +144,53 @@ def _run_transit_search(search_id: str, request: dict[str, Any]) -> None:
diagnostics: dict[str, Any] = {"stages": []}
best_count = 0
stale_stages = 0
ranked: list[dict] = []
stopped_early_reason = ""
for transfers in stages:
if _is_cancelled(search_id):
return
label = "direct" if transfers == 0 else f"up to {transfers} transfer{'s' if transfers != 1 else ''}"
_publish_status(search_id, "running", f"Searching {label}...", f"transfers_{transfers}")
stage_started_at = time.monotonic()
with SessionLocal() as db:
result = _cached_find_journeys(
db,
from_stop_id=str(request.get("from_stop_id") or ""),
to_stop_id=str(request.get("to_stop_id") or ""),
via_stop_id=request.get("via_stop_id") or None,
source_ids=source_ids,
departure=str(request.get("departure") or "08:00"),
service_date=service_date,
max_transfers=transfers,
transfer_seconds=transfer_seconds,
limit=stage_limit,
try:
with SessionLocal() as db:
result = _cached_find_journeys(
db,
from_stop_id=str(request.get("from_stop_id") or ""),
to_stop_id=str(request.get("to_stop_id") or ""),
via_stop_id=request.get("via_stop_id") or None,
source_ids=source_ids,
departure=str(request.get("departure") or "08:00"),
service_date=service_date,
max_transfers=transfers,
transfer_seconds=transfer_seconds,
limit=stage_limit,
)
except Exception as exc: # noqa: BLE001 - preserve already-published progressive results on DB timeouts
if not _is_statement_timeout_error(exc) or not ranked:
raise
elapsed_ms = int((time.monotonic() - stage_started_at) * 1000)
diagnostics["stages"].append(
{
"transfers": transfers,
"cache": "error",
"elapsed_ms": elapsed_ms,
"journeys": 0,
"timed_out": True,
"error": _short_error_message(exc),
}
)
context["diagnostics"] = diagnostics
stopped_early_reason = f"Stopped after {label} timed out; showing the best options found so far."
_publish_results(
search_id,
journeys=ranked,
context=context,
status="running",
stage=f"transfers_{transfers}",
message=stopped_early_reason,
)
break
cache_status = str(result.pop("_cache_status", "miss"))
elapsed_ms = int((time.monotonic() - stage_started_at) * 1000)
stage_diagnostics = {
@@ -179,11 +222,19 @@ def _run_transit_search(search_id: str, request: dict[str, Any]) -> None:
else:
stale_stages = 0
best_count = max(best_count, len(ranked))
if _has_sufficient_progressive_options(ranked, transfers=transfers, limit=limit):
stopped_early_reason = "Stopped after finding a balanced set of fast and low-walk options."
break
if ranked and stale_stages >= 2 and transfers >= 2:
stopped_early_reason = "Stopped after deeper transfer stages stopped improving the options."
break
if _major_hub_address_stage_is_complete(result_diagnostics, ranked, transfers=transfers, limit=limit):
stopped_early_reason = "Stopped after finding major-hub address access options."
break
complete_message = (
f"{stopped_early_reason} Found {best_count} option{'s' if best_count != 1 else ''}."
if stopped_early_reason and best_count
else
f"Search complete. Found {best_count} option{'s' if best_count != 1 else ''}."
if best_count
else "Search complete. No route found in the imported timetable."
@@ -270,6 +321,7 @@ def _cached_find_journeys(
to_stop_id,
str(via_stop_id or ""),
tuple(sorted(int(source_id) for source_id in source_ids or [])),
tuple(active_harmonized_gtfs_dataset_ids(db, source_ids=source_ids)),
departure,
str(service_date or ""),
int(max_transfers),
@@ -406,12 +458,14 @@ def _as_utc(value: datetime | None) -> datetime | None:
def _progressive_cache_key(request: dict[str, Any]) -> tuple[object, ...]:
source_ids = _csv_ints(request.get("source_id"))
mode = _canonical_search_mode(request.get("mode"))
return (
str(request.get("mode") or "transit"),
mode,
str(request.get("from_stop_id") or ""),
str(request.get("to_stop_id") or ""),
str(request.get("via_stop_id") or ""),
tuple(sorted(int(source_id) for source_id in source_ids or [])),
_active_dataset_cache_fragment(source_ids) if mode == "transit" else (),
str(request.get("departure") or "08:00"),
str(request.get("service_date") or ""),
bool(request.get("direct_only")),
@@ -421,6 +475,23 @@ def _progressive_cache_key(request: dict[str, Any]) -> tuple[object, ...]:
)
def _canonical_search_mode(mode: object) -> str:
value = str(mode or "transit").lower()
if value in {"bike", "bicycle", "cycle", "cycling"}:
return "bike"
if value == "car":
return "drive"
return value if value in {"transit", "walk", "drive"} else "transit"
def _active_dataset_cache_fragment(source_ids: list[int] | None) -> tuple[int, ...]:
try:
with SessionLocal() as session:
return tuple(active_harmonized_gtfs_dataset_ids(session, source_ids=source_ids))
except Exception: # noqa: BLE001 - cache key generation must not prevent search
return ()
def _progressive_cache_get(key: tuple[object, ...]) -> dict[str, Any] | None:
now = time.monotonic()
with _lock:
@@ -567,24 +638,7 @@ def _context_from_result(result: dict[str, Any]) -> dict[str, Any]:
def _journey_key(journey: dict[str, Any]) -> str:
parts = []
for leg in journey.get("legs") or []:
parts.append(
"|".join(
str(part or "")
for part in [
leg.get("dataset_id"),
leg.get("mode"),
leg.get("route_id"),
leg.get("trip_id"),
(leg.get("from") or {}).get("stop_id") or (leg.get("from") or {}).get("name"),
(leg.get("to") or {}).get("stop_id") or (leg.get("to") or {}).get("name"),
leg.get("departure_time"),
leg.get("arrival_time"),
]
)
)
return "||".join(parts)
return json.dumps(journey_equivalence_key(journey), ensure_ascii=False, separators=(",", ":"))
def _rank_journeys(journeys, ranking: str) -> list[dict]:
@@ -623,36 +677,47 @@ def _rank_journeys(journeys, ranking: str) -> list[dict]:
walking,
)
return sorted((dict(journey) for journey in journeys), key=key)
return sorted((dict(journey) for journey in journeys), key=lambda journey: (key(journey), journey_duplicate_preference_key(journey)))
def _select_diverse_journeys(journeys: list[dict], *, limit: int) -> list[dict]:
ranked = list(journeys)
selected: list[dict] = []
selected_exact: set[str] = set()
selected_diversity: set[tuple[object, ...]] = set()
for journey in journeys:
def append(journey: dict | None, *, force: bool = False) -> bool:
if journey is None:
return False
exact_key = _journey_key(journey)
if exact_key in selected_exact:
continue
return False
diversity_key = _journey_diversity_key(journey)
if diversity_key in selected_diversity and len(selected) >= 3:
continue
if not force and diversity_key in selected_diversity and len(selected) >= 3:
return False
selected.append(journey)
selected_exact.add(exact_key)
selected_diversity.add(diversity_key)
return True
if ranked:
append(ranked[0], force=True)
append(_best_public_transport_option(ranked), force=True)
append(_least_walking_public_transport_option(ranked), force=True)
append(_fewest_transfer_public_transport_option(ranked), force=True)
append(_walk_only_option(ranked), force=True)
for journey in ranked:
append(journey)
if len(selected) >= limit:
return selected
return _ensure_walk_only_option(selected[:limit], ranked, limit=limit)
if len(selected) >= min(3, limit):
return selected
for journey in journeys:
exact_key = _journey_key(journey)
if exact_key in selected_exact:
continue
selected.append(journey)
selected_exact.add(exact_key)
for journey in ranked:
append(journey, force=True)
if len(selected) >= min(3, limit):
break
return _ensure_walk_only_option(selected, journeys, limit=limit)
return _ensure_walk_only_option(selected[:limit], ranked, limit=limit)
def _ensure_walk_only_option(selected: list[dict], ranked: list[dict], *, limit: int) -> list[dict]:
@@ -673,6 +738,95 @@ def _journey_is_walk_only(journey: dict) -> bool:
return bool(legs) and all(leg.get("mode") == "walk" for leg in legs)
def _walk_only_option(journeys: list[dict]) -> dict | None:
return next((journey for journey in journeys if _journey_is_walk_only(journey)), None)
def _journey_has_public_transport(journey: dict) -> bool:
return any(leg.get("mode") != "walk" for leg in journey.get("legs") or [])
def _best_public_transport_option(journeys: list[dict]) -> dict | None:
return next((journey for journey in journeys if _journey_has_public_transport(journey)), None)
def _least_walking_public_transport_option(journeys: list[dict]) -> dict | None:
public = [journey for journey in journeys if _journey_has_public_transport(journey)]
if not public:
return None
fastest = public[0]
lowest = min(public, key=lambda journey: (_journey_walking_m(journey), _journey_arrival_seconds(journey), _journey_duration_seconds(journey)))
if lowest is fastest:
return lowest
fastest_walking = _journey_walking_m(fastest)
lowest_walking = _journey_walking_m(lowest)
walking_saved = fastest_walking - lowest_walking
if lowest_walking <= LOW_WALK_GOOD_ENOUGH_M:
return lowest
if walking_saved < LOW_WALK_MIN_SAVINGS_M:
return None
if fastest_walking > 0 and walking_saved / fastest_walking < LOW_WALK_MIN_RELATIVE_SAVING:
return None
extra_seconds = _journey_arrival_seconds(lowest) - _journey_arrival_seconds(fastest)
if extra_seconds > LOW_WALK_MAX_EXTRA_SECONDS:
return None
return lowest
def _fewest_transfer_public_transport_option(journeys: list[dict]) -> dict | None:
public = [journey for journey in journeys if _journey_has_public_transport(journey)]
if not public:
return None
return min(
public,
key=lambda journey: (
int(journey.get("transfers") or 0),
_journey_walking_m(journey),
_journey_arrival_seconds(journey),
),
)
def _journey_walking_m(journey: dict) -> float:
return sum(float(leg.get("distance_m") or 0) for leg in journey.get("legs") or [] if leg.get("mode") == "walk")
def _journey_arrival_seconds(journey: dict) -> float:
value = journey.get("arrival_seconds")
return float("inf") if value is None else float(value)
def _journey_duration_seconds(journey: dict) -> float:
value = journey.get("duration_seconds")
if value is not None:
return float(value)
minutes = journey.get("duration_minutes")
return float("inf") if minutes is None else float(minutes) * 60
def _has_sufficient_progressive_options(journeys: list[dict], *, transfers: int, limit: int) -> bool:
if transfers < 2 or len(journeys) < min(3, limit):
return False
public = [journey for journey in journeys if _journey_has_public_transport(journey)]
if not public:
return False
low_walk = _least_walking_public_transport_option(public)
if low_walk is None:
return False
return _journey_walking_m(low_walk) <= max(LOW_WALK_GOOD_ENOUGH_M * 2, _journey_walking_m(public[0]) * 0.75)
def _is_statement_timeout_error(exc: Exception) -> bool:
text = " ".join(str(part).lower() for part in [exc, getattr(exc, "orig", "")])
return "statement timeout" in text or "canceling statement due to statement timeout" in text or "querycanceled" in text
def _short_error_message(exc: Exception) -> str:
text = " ".join(str(part) for part in [getattr(exc, "orig", ""), exc] if part)
text = " ".join(text.split())
return text[:240] if text else exc.__class__.__name__
def _journey_diversity_key(journey: dict[str, Any]) -> tuple[object, ...]:
route_signature = tuple(
str(leg.get("route_ref") or leg.get("route_id") or leg.get("mode") or "")

418
app/main.py
View File

@@ -1,5 +1,7 @@
from __future__ import annotations
import csv
import io
import json
from contextlib import asynccontextmanager
from datetime import datetime, timezone
@@ -24,8 +26,17 @@ from app.dataset_search import search_datasets
from app.db import engine, get_db, init_db
from app.db_lock import DatabaseWriteBusy, database_write_lock, database_write_status
from app.geofabrik import create_geofabrik_source, geofabrik_catalog
from app.gtfs_diff import diff_run_payload, gtfs_update_diff_detail, gtfs_update_diff_summary, latest_gtfs_update_diffs
from app.gtfs_storage import scheduled_stop_ids
from app.harmonization import GTFS_QA_NOTE_PREFIX, gtfs_harmonization_feed_detail, gtfs_harmonization_inventory
from app.harmonization import (
GTFS_QA_NOTE_PREFIX,
gtfs_harmonization_feed_detail,
gtfs_harmonization_inventory,
gtfs_harmonized_snapshot,
gtfs_harmonized_snapshot_diagnostics,
gtfs_qa_review_payload,
list_active_harmonized_snapshot_routes,
)
from app.itineraries import generate_itineraries, itinerary_payload, recent_itineraries, set_itinerary_saved, set_leg_locked
from app.journey import find_journeys, nearest_scheduled_stops, search_scheduled_stops
from app.journey_search import cancel_journey_search, journey_search_payload, start_journey_search
@@ -37,6 +48,8 @@ from app.jobs import (
cancel_job,
create_dataset_delete_job,
create_address_index_rebuild_job,
create_gtfs_harmonized_snapshot_job,
create_map_gtfs_review_job,
create_maintenance_job,
create_osm_relabel_job,
create_route_matching_job,
@@ -69,6 +82,7 @@ from app.models import (
ItineraryLeg,
Job,
MatchRule,
MapGtfsReviewItem,
OsmFeature,
PipelineRun,
RouteMatch,
@@ -106,7 +120,21 @@ from app.source_catalog import (
source_catalog_rows,
source_catalog_summary,
)
from app.worker_supervisor import queue_worker_status, start_queue_workers, stop_queue_workers
from app.worker_supervisor import (
queue_worker_status,
restart_queue_workers,
start_queue_workers,
stop_configured_queue_workers,
stop_queue_workers,
)
from app.workbench import (
generate_map_gtfs_review_items,
list_map_gtfs_review_items,
map_gtfs_review_summary,
record_route_match_decision,
review_item_payload,
update_map_gtfs_review_item,
)
@asynccontextmanager
async def lifespan(_app: FastAPI):
@@ -125,6 +153,7 @@ async def lifespan(_app: FastAPI):
app = FastAPI(title="Mobility Workbench", version="0.1.0", lifespan=lifespan)
app.mount("/static", StaticFiles(directory=Path(__file__).parent / "static"), name="static")
templates = Jinja2Templates(directory=str(Path(__file__).parent / "templates"))
STATIC_ASSET_VERSION = "20260706-bike-mode"
MAP_FEATURE_LIMIT = 5000
MAP_FEATURE_LIMIT_MAX = 20000
@@ -146,6 +175,12 @@ class GtfsFeedReviewUpdate(BaseModel):
license: Optional[str] = None
review_status: Optional[str] = None
review_note: Optional[str] = None
can_import: Optional[str] = None
can_derive: Optional[str] = None
can_redistribute: Optional[str] = None
requires_attribution: Optional[str] = None
commercial_restrictions: Optional[str] = None
authority_level: Optional[str] = None
enabled: Optional[bool] = None
@@ -221,6 +256,11 @@ class JobPriorityRequest(BaseModel):
priority: int
class WorkbenchReviewUpdate(BaseModel):
status: Optional[str] = None
note: Optional[str] = None
def write_endpoint(operation: str):
def decorator(fn):
@wraps(fn)
@@ -249,7 +289,7 @@ async def database_operational_error_handler(_request: Request, exc: Operational
@app.get("/", response_class=HTMLResponse)
def index(request: Request) -> HTMLResponse:
return templates.TemplateResponse(request=request, name="index.html")
return templates.TemplateResponse(request=request, name="index.html", context={"static_version": STATIC_ASSET_VERSION})
@app.get("/api/sources")
@@ -623,6 +663,21 @@ def queue_route_matching(priority: int = 0, db: Session = Depends(get_db)) -> di
return job_payload(job)
@app.post("/api/jobs/map-gtfs-review")
@write_endpoint("queue Map/GTFS review generation")
def queue_map_gtfs_review(
source_id: Optional[int] = None,
dataset_id: Optional[int] = None,
limit: int = 1000,
priority: int = 0,
db: Session = Depends(get_db),
) -> dict:
job = create_map_gtfs_review_job(db, source_id=source_id, dataset_id=dataset_id, limit=limit, priority=priority)
db.commit()
db.refresh(job)
return job_payload(job)
@app.post("/api/jobs/osm-relabel")
@write_endpoint("queue OSM relabeling")
def queue_osm_relabel(
@@ -658,6 +713,28 @@ def list_jobs(
}
@app.post("/api/workers/start")
def start_workers_endpoint() -> dict:
started = start_queue_workers(force=True)
return _worker_control_payload("start", started=started)
@app.post("/api/workers/stop")
def stop_workers_endpoint() -> dict:
stopped = stop_configured_queue_workers()
return _worker_control_payload("stop", stopped=stopped)
@app.post("/api/workers/restart")
def restart_workers_endpoint() -> dict:
result = restart_queue_workers()
return _worker_control_payload(
"restart",
stopped=result.get("stopped", []),
started=result.get("started", []),
)
@app.get("/api/jobs/revision")
def get_jobs_revision(
response: Response,
@@ -870,6 +947,152 @@ def gtfs_harmonization_inventory_endpoint(db: Session = Depends(get_db)) -> dict
return gtfs_harmonization_inventory(db)
@app.get("/api/harmonization/gtfs/snapshot")
def gtfs_harmonized_snapshot_endpoint(db: Session = Depends(get_db)) -> dict:
return gtfs_harmonized_snapshot(db)
@app.get("/api/harmonization/gtfs/snapshot/routes")
def gtfs_harmonized_snapshot_routes_endpoint(
role: Optional[str] = None,
dataset_id: Optional[int] = None,
source_id: Optional[int] = None,
shadowed_by_dataset_id: Optional[int] = None,
mode: Optional[str] = None,
q: Optional[str] = None,
limit: int = 200,
offset: int = 0,
db: Session = Depends(get_db),
) -> dict:
if role is not None and role not in {"included", "shadowed", "excluded"}:
raise HTTPException(status_code=400, detail="role must be included, shadowed, or excluded")
return list_active_harmonized_snapshot_routes(
db,
role=role,
dataset_id=dataset_id,
source_id=source_id,
shadowed_by_dataset_id=shadowed_by_dataset_id,
mode=mode,
q=q,
limit=limit,
offset=offset,
)
@app.get("/api/harmonization/gtfs/snapshot/routes.csv")
def gtfs_harmonized_snapshot_routes_csv_endpoint(
role: Optional[str] = None,
dataset_id: Optional[int] = None,
source_id: Optional[int] = None,
shadowed_by_dataset_id: Optional[int] = None,
mode: Optional[str] = None,
q: Optional[str] = None,
limit: int = 10000,
offset: int = 0,
db: Session = Depends(get_db),
) -> Response:
if role is not None and role not in {"included", "shadowed", "excluded"}:
raise HTTPException(status_code=400, detail="role must be included, shadowed, or excluded")
payload = list_active_harmonized_snapshot_routes(
db,
role=role,
dataset_id=dataset_id,
source_id=source_id,
shadowed_by_dataset_id=shadowed_by_dataset_id,
mode=mode,
q=q,
limit=limit,
offset=offset,
max_limit=100_000,
)
output = io.StringIO()
fieldnames = [
"snapshot_id",
"gtfs_route_id",
"dataset_id",
"source_id",
"source_name",
"route_id",
"route_ref",
"route_name",
"mode",
"operator",
"overlap_key",
"role",
"reason",
"shadowed_by_gtfs_route_id",
"shadowed_by_dataset_id",
"shadowed_by_source_id",
]
writer = csv.DictWriter(output, fieldnames=fieldnames, extrasaction="ignore")
writer.writeheader()
for route in payload.get("routes", []):
writer.writerow(route)
snapshot = payload.get("snapshot") or {}
filename = f"gtfs_harmonized_snapshot_routes_{snapshot.get('id') or 'none'}.csv"
return Response(
content=output.getvalue(),
media_type="text/csv; charset=utf-8",
headers={"Content-Disposition": f'attachment; filename="{filename}"'},
)
@app.get("/api/harmonization/gtfs/diffs")
def list_gtfs_update_diffs(
source_id: Optional[int] = None,
limit: int = 20,
db: Session = Depends(get_db),
) -> dict:
return {
"summary": gtfs_update_diff_summary(db),
"diffs": [diff_run_payload(diff) for diff in latest_gtfs_update_diffs(db, source_id=source_id, limit=limit)],
}
@app.get("/api/harmonization/gtfs/diffs/{diff_id}")
def get_gtfs_update_diff(diff_id: int, item_limit: int = 200, db: Session = Depends(get_db)) -> dict:
detail = gtfs_update_diff_detail(db, diff_id, item_limit=item_limit)
if detail is None:
raise HTTPException(status_code=404, detail="GTFS update diff not found")
return detail
@app.get("/api/workbench/gtfs-workflow")
def gtfs_workflow_status_endpoint(db: Session = Depends(get_db)) -> dict:
inventory = gtfs_harmonization_inventory(db)
snapshot_diagnostics = inventory.get("snapshot_diagnostics") or gtfs_harmonized_snapshot_diagnostics(db)
snapshot = inventory["snapshot"]
snapshot_summary = snapshot.get("summary") or {}
snapshot_route_preview = (
list_active_harmonized_snapshot_routes(db, role="shadowed", limit=8)
if snapshot.get("persisted") and int(snapshot_summary.get("snapshot_route_rows") or 0) > 0
else {"snapshot": None, "summary": {}, "routes": []}
)
review = map_gtfs_review_summary(db)
diff_summary = gtfs_update_diff_summary(db)
latest_diffs = [diff_run_payload(diff) for diff in latest_gtfs_update_diffs(db, limit=5)]
jobs = latest_jobs(db, limit=8)
return {
"inventory_summary": inventory["summary"],
"snapshot": snapshot,
"snapshot_diagnostics": snapshot_diagnostics,
"snapshot_route_preview": snapshot_route_preview,
"review_summary": review,
"diff_summary": diff_summary,
"latest_diffs": latest_diffs,
"jobs": [job_payload(job) for job in jobs],
}
@app.post("/api/jobs/gtfs-harmonized-snapshot")
@write_endpoint("queue harmonized GTFS snapshot build")
def queue_gtfs_harmonized_snapshot(activate: bool = True, priority: int = 0, db: Session = Depends(get_db)) -> dict:
job = create_gtfs_harmonized_snapshot_job(db, activate=activate, priority=priority)
db.commit()
db.refresh(job)
return job_payload(job)
@app.get("/api/harmonization/gtfs/sources/{source_id}")
def gtfs_harmonization_feed_detail_endpoint(source_id: int, db: Session = Depends(get_db)) -> dict:
detail = gtfs_harmonization_feed_detail(db, source_id)
@@ -886,15 +1109,48 @@ def update_gtfs_harmonization_feed_review(source_id: int, payload: GtfsFeedRevie
raise HTTPException(status_code=404, detail="GTFS source not found")
if payload.review_status is not None and payload.review_status not in {"unreviewed", "approved", "needs_review", "blocked", "rejected"}:
raise HTTPException(status_code=400, detail="review_status must be unreviewed, approved, needs_review, blocked, or rejected")
license_flags = {
"can_import": payload.can_import,
"can_derive": payload.can_derive,
"can_redistribute": payload.can_redistribute,
"requires_attribution": payload.requires_attribution,
"commercial_restrictions": payload.commercial_restrictions,
}
for field, value in license_flags.items():
if value is not None and value not in {"unknown", "yes", "no"}:
raise HTTPException(status_code=400, detail=f"{field} must be unknown, yes, or no")
if payload.authority_level is not None and payload.authority_level not in {
"unknown",
"national_official",
"regional_authority",
"operator",
"aggregator",
"mirror",
"secondary_discovery",
}:
raise HTTPException(status_code=400, detail="authority_level is not valid")
if payload.license is not None:
source.license = _truncate(payload.license, 255)
if payload.enabled is not None:
source.enabled = bool(payload.enabled)
if payload.review_status is not None or payload.review_note is not None:
existing_review = gtfs_qa_review_payload(source.notes)
review_fields_present = (
payload.review_status is not None
or payload.review_note is not None
or any(value is not None for value in license_flags.values())
or payload.authority_level is not None
)
if review_fields_present:
source.notes = _upsert_gtfs_qa_note(
source.notes,
status=payload.review_status or "unreviewed",
note=payload.review_note or "",
status=payload.review_status or existing_review["status"],
note=payload.review_note if payload.review_note is not None else existing_review["note"],
can_import=payload.can_import or existing_review["can_import"],
can_derive=payload.can_derive or existing_review["can_derive"],
can_redistribute=payload.can_redistribute or existing_review["can_redistribute"],
requires_attribution=payload.requires_attribution or existing_review["requires_attribution"],
commercial_restrictions=payload.commercial_restrictions or existing_review["commercial_restrictions"],
authority_level=payload.authority_level or existing_review["authority_level"],
)
db.commit()
detail = gtfs_harmonization_feed_detail(db, source_id)
@@ -903,6 +1159,84 @@ def update_gtfs_harmonization_feed_review(source_id: int, payload: GtfsFeedRevie
return detail
@app.get("/api/workbench/map-gtfs/summary")
def map_gtfs_workbench_summary_endpoint(db: Session = Depends(get_db)) -> dict:
return map_gtfs_review_summary(db)
@app.get("/api/workbench/map-gtfs/review-items")
def list_map_gtfs_workbench_review_items(
queue: Optional[str] = None,
status: Optional[str] = "open",
severity: Optional[str] = None,
mode: Optional[str] = None,
q: Optional[str] = None,
source_id: Optional[int] = None,
dataset_id: Optional[int] = None,
limit: int = 100,
db: Session = Depends(get_db),
) -> dict:
items = list_map_gtfs_review_items(
db,
queue=queue,
status=status,
severity=severity,
mode=mode,
q=q,
source_id=source_id,
dataset_id=dataset_id,
limit=limit,
)
return {
"summary": map_gtfs_review_summary(db),
"filters": {
"queue": queue,
"status": status,
"severity": severity,
"mode": mode,
"q": q,
"source_id": source_id,
"dataset_id": dataset_id,
},
"items": [review_item_payload(item) for item in items],
}
@app.get("/api/workbench/map-gtfs/review-items/{item_id}")
def get_map_gtfs_workbench_review_item(item_id: int, db: Session = Depends(get_db)) -> dict:
item = db.get(MapGtfsReviewItem, item_id)
if item is None:
raise HTTPException(status_code=404, detail="review item not found")
return review_item_payload(item)
@app.post("/api/workbench/map-gtfs/review-items/generate")
@write_endpoint("generate Map/GTFS review queue")
def generate_map_gtfs_workbench_review_items(
source_id: Optional[int] = None,
dataset_id: Optional[int] = None,
limit: int = 1000,
db: Session = Depends(get_db),
) -> dict:
result = generate_map_gtfs_review_items(db, source_id=source_id, dataset_id=dataset_id, limit=limit)
db.commit()
return {"summary": map_gtfs_review_summary(db), "result": result}
@app.patch("/api/workbench/map-gtfs/review-items/{item_id}")
@write_endpoint("update Map/GTFS review item")
def update_map_gtfs_workbench_review_item(item_id: int, payload: WorkbenchReviewUpdate, db: Session = Depends(get_db)) -> dict:
try:
item = update_map_gtfs_review_item(db, item_id, status=payload.status, note=payload.note)
except ValueError as exc:
detail = str(exc)
status_code = 404 if detail == "review item not found" else 400
raise HTTPException(status_code=status_code, detail=detail) from exc
db.commit()
db.refresh(item)
return review_item_payload(item)
@app.get("/api/matches")
def list_matches(
status: Optional[str] = None,
@@ -933,6 +1267,7 @@ def accept_match(match_id: int, db: Session = Depends(get_db)) -> dict:
match.rule_source = "manual"
match.updated_at = datetime.now(timezone.utc)
_persist_match_rule(db, match, "accept_match")
record_route_match_decision(db, match, note="Accepted from route match review.")
db.commit()
return {"id": match.id, "status": match.status}
@@ -947,6 +1282,7 @@ def reject_match(match_id: int, db: Session = Depends(get_db)) -> dict:
match.rule_source = "manual"
match.updated_at = datetime.now(timezone.utc)
_persist_match_rule(db, match, "reject_match")
record_route_match_decision(db, match, note="Rejected from route match review.")
db.commit()
return {"id": match.id, "status": match.status}
@@ -1030,6 +1366,7 @@ def accept_match_candidate(match_id: int, osm_feature_id: str, db: Session = Dep
match.reasons_json = json.dumps(reasons, separators=(",", ":"))
match.updated_at = datetime.now(timezone.utc)
_persist_match_rule(db, match, "accept_match")
record_route_match_decision(db, match, note="Accepted from route candidate review.")
db.commit()
db.refresh(match)
return {"id": match.id, "status": match.status, "confidence": match.confidence, "match": match_row(match)}
@@ -1423,6 +1760,7 @@ def journey_nearest_location(
source_ids=_csv_ints(source_id, "source_id"),
limit=1,
radius_m=max(5, min(float(stop_radius_m), 120)),
grouped=False,
)
location = stops[0] if stops else None
if location is not None:
@@ -1509,7 +1847,8 @@ def journey_search(
@app.post("/api/journey/searches")
def start_progressive_journey_search(payload: JourneySearchRequest) -> dict:
mode = payload.mode if payload.mode in {"transit", "walk", "drive", "car"} else "transit"
requested_mode = str(payload.mode or "transit").lower()
mode = "bike" if requested_mode in {"bike", "bicycle", "cycle", "cycling"} else requested_mode if requested_mode in {"transit", "walk", "drive", "car"} else "transit"
ranking = payload.ranking if payload.ranking in {"recommended", "earliest_arrival", "duration", "fewest_transfers"} else "recommended"
try:
return start_journey_search(
@@ -1978,6 +2317,29 @@ def _json_object_value(value: object, key: str) -> object:
return value.get(key) if isinstance(value, dict) else None
def _worker_control_payload(action: str, **groups: list[object]) -> dict:
return {
"action": action,
"operations": {
name: [_worker_handle_payload(handle) for handle in handles]
for name, handles in groups.items()
},
"workers": queue_worker_status(),
}
def _worker_handle_payload(handle: object) -> dict:
return {
"index": getattr(handle, "index", None),
"worker_id": getattr(handle, "worker_id", None),
"pid": getattr(handle, "pid", None),
"status": getattr(handle, "status", None),
"pid_file": str(getattr(handle, "pid_file", "")),
"log_file": str(getattr(handle, "log_file", "")),
"started_by_server": bool(getattr(handle, "started_by_server", False)),
}
def _job_queue_revision_payload(
db: Session,
*,
@@ -2018,19 +2380,51 @@ def _set_etag(response: Response, revision: str) -> None:
response.headers["ETag"] = f'W/"{revision}"'
def _upsert_gtfs_qa_note(notes: str | None, *, status: str, note: str) -> str | None:
def _upsert_gtfs_qa_note(
notes: str | None,
*,
status: str,
note: str,
can_import: str | None = None,
can_derive: str | None = None,
can_redistribute: str | None = None,
requires_attribution: str | None = None,
commercial_restrictions: str | None = None,
authority_level: str | None = None,
) -> str | None:
status_text = (status or "unreviewed").strip() or "unreviewed"
note_text = " ".join((note or "").strip().split())
updated_at = datetime.now(timezone.utc).isoformat()
marker = f"{GTFS_QA_NOTE_PREFIX} status={status_text}; updated_at={updated_at}"
if note_text:
marker = f"{marker}; note={note_text}"
payload = {
"status": status_text,
"updated_at": updated_at,
"note": note_text,
"can_import": can_import or "unknown",
"can_derive": can_derive or "unknown",
"can_redistribute": can_redistribute or "unknown",
"requires_attribution": requires_attribution or "unknown",
"commercial_restrictions": commercial_restrictions or "unknown",
"authority_level": authority_level or "unknown",
}
marker = f"{GTFS_QA_NOTE_PREFIX} {json.dumps(payload, sort_keys=True, separators=(',', ':'))}"
preserved = [
line
for line in str(notes or "").splitlines()
if line.strip() and not line.startswith(GTFS_QA_NOTE_PREFIX)
]
if status_text != "unreviewed" or note_text:
has_decision = (
status_text != "unreviewed"
or note_text
or any(payload[key] != "unknown" for key in [
"can_import",
"can_derive",
"can_redistribute",
"requires_attribution",
"commercial_restrictions",
"authority_level",
])
)
if has_decision:
preserved.insert(0, marker)
return "\n".join(preserved) or None

186
app/models.py
View File

@@ -81,6 +81,115 @@ class Dataset(Base):
source: Mapped[Source] = relationship(back_populates="datasets")
class GtfsHarmonizedSnapshot(Base):
__tablename__ = "gtfs_harmonized_snapshots"
__table_args__ = (UniqueConstraint("snapshot_key", name="uq_gtfs_harmonized_snapshot_key"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
snapshot_key: Mapped[str] = mapped_column(String(128), nullable=False, index=True)
status: Mapped[str] = mapped_column(String(64), nullable=False, default="draft", index=True)
source: Mapped[str] = mapped_column(String(64), nullable=False, default="computed")
note: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
summary_json: Mapped[str] = mapped_column(Text, nullable=False)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
activated_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
datasets: Mapped[list["GtfsHarmonizedSnapshotDataset"]] = relationship(back_populates="snapshot", cascade="all, delete-orphan")
routes: Mapped[list["GtfsHarmonizedSnapshotRoute"]] = relationship(back_populates="snapshot", cascade="all, delete-orphan")
class GtfsHarmonizedSnapshotDataset(Base):
__tablename__ = "gtfs_harmonized_snapshot_datasets"
__table_args__ = (UniqueConstraint("snapshot_id", "dataset_id", name="uq_gtfs_harmonized_snapshot_dataset"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
snapshot_id: Mapped[int] = mapped_column(ForeignKey("gtfs_harmonized_snapshots.id"), nullable=False, index=True)
dataset_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
source_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
source_name: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
role: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
reason: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
shadowed_by_dataset_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
shadowed_by_source_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
authority_level: Mapped[Optional[str]] = mapped_column(String(64), nullable=True, index=True)
review_authority_level: Mapped[Optional[str]] = mapped_column(String(64), nullable=True)
route_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
route_key_count: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
route_key_overlap: Mapped[int] = mapped_column(Integer, nullable=False, default=0)
route_key_overlap_ratio: Mapped[float] = mapped_column(Float, nullable=False, default=0.0)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
snapshot: Mapped[GtfsHarmonizedSnapshot] = relationship(back_populates="datasets")
class GtfsHarmonizedSnapshotRoute(Base):
__tablename__ = "gtfs_harmonized_snapshot_routes"
__table_args__ = (UniqueConstraint("snapshot_id", "gtfs_route_id", name="uq_gtfs_harmonized_snapshot_route"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
snapshot_id: Mapped[int] = mapped_column(ForeignKey("gtfs_harmonized_snapshots.id"), nullable=False, index=True)
gtfs_route_id: Mapped[int] = mapped_column(ForeignKey("gtfs_routes.id"), nullable=False, index=True)
dataset_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
source_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
route_id: Mapped[str] = mapped_column(Text, nullable=False)
overlap_key: Mapped[Optional[str]] = mapped_column(String(255), nullable=True, index=True)
role: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
reason: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
shadowed_by_gtfs_route_id: Mapped[Optional[int]] = mapped_column(ForeignKey("gtfs_routes.id"), nullable=True, index=True)
shadowed_by_dataset_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
shadowed_by_source_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
snapshot: Mapped[GtfsHarmonizedSnapshot] = relationship(back_populates="routes")
gtfs_route: Mapped["GtfsRoute"] = relationship(foreign_keys=[gtfs_route_id])
shadowed_by_gtfs_route: Mapped[Optional["GtfsRoute"]] = relationship(foreign_keys=[shadowed_by_gtfs_route_id])
class GtfsFeedDiffRun(Base):
__tablename__ = "gtfs_feed_diff_runs"
id: Mapped[int] = mapped_column(Integer, primary_key=True)
source_id: Mapped[int] = mapped_column(ForeignKey("sources.id"), nullable=False, index=True)
previous_dataset_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
new_dataset_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
status: Mapped[str] = mapped_column(String(64), nullable=False, default="completed", index=True)
diff_key: Mapped[str] = mapped_column(String(128), nullable=False, index=True)
summary_json: Mapped[str] = mapped_column(Text, nullable=False)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
source: Mapped[Source] = relationship()
items: Mapped[list["GtfsFeedDiffItem"]] = relationship(back_populates="diff_run", cascade="all, delete-orphan")
class GtfsFeedDiffItem(Base):
__tablename__ = "gtfs_feed_diff_items"
__table_args__ = (UniqueConstraint("diff_run_id", "item_type", "object_key", "change_type", name="uq_gtfs_feed_diff_item"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
diff_run_id: Mapped[int] = mapped_column(ForeignKey("gtfs_feed_diff_runs.id"), nullable=False, index=True)
source_id: Mapped[int] = mapped_column(ForeignKey("sources.id"), nullable=False, index=True)
previous_dataset_id: Mapped[Optional[int]] = mapped_column(Integer, nullable=True, index=True)
new_dataset_id: Mapped[int] = mapped_column(Integer, nullable=False, index=True)
item_type: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
object_key: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
change_type: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
severity: Mapped[str] = mapped_column(String(32), nullable=False, default="info", index=True)
status: Mapped[str] = mapped_column(String(64), nullable=False, default="open", index=True)
title: Mapped[str] = mapped_column(Text, nullable=False)
old_payload_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
new_payload_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
diff_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
resolved_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
diff_run: Mapped[GtfsFeedDiffRun] = relationship(back_populates="items")
source: Mapped[Source] = relationship()
class SourceUpdateCheck(Base):
__tablename__ = "source_update_checks"
@@ -543,6 +652,83 @@ class MatchRule(Base):
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False)
class MapGtfsDecision(Base):
__tablename__ = "map_gtfs_decisions"
__table_args__ = (UniqueConstraint("decision_key", name="uq_map_gtfs_decision_key"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
decision_key: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
decision_type: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
status: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
active: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False, index=True)
source_id: Mapped[Optional[int]] = mapped_column(ForeignKey("sources.id"), nullable=True, index=True)
gtfs_dataset_id: Mapped[Optional[int]] = mapped_column(ForeignKey("datasets.id"), nullable=True, index=True)
gtfs_route_id: Mapped[Optional[int]] = mapped_column(ForeignKey("gtfs_routes.id"), nullable=True, index=True)
gtfs_stop_id: Mapped[Optional[int]] = mapped_column(ForeignKey("gtfs_stops.id"), nullable=True, index=True)
osm_dataset_id: Mapped[Optional[int]] = mapped_column(ForeignKey("datasets.id"), nullable=True, index=True)
osm_feature_id: Mapped[Optional[int]] = mapped_column(ForeignKey("osm_features.id"), nullable=True, index=True)
route_match_id: Mapped[Optional[int]] = mapped_column(ForeignKey("route_matches.id"), nullable=True, index=True)
canonical_stop_id: Mapped[Optional[int]] = mapped_column(ForeignKey("canonical_stops.id"), nullable=True, index=True)
canonical_stop_link_id: Mapped[Optional[int]] = mapped_column(ForeignKey("canonical_stop_links.id"), nullable=True, index=True)
confidence: Mapped[Optional[float]] = mapped_column(Float, nullable=True)
evidence_hash: Mapped[Optional[str]] = mapped_column(String(128), nullable=True, index=True)
selector_json: Mapped[str] = mapped_column(Text, nullable=False)
action_json: Mapped[str] = mapped_column(Text, nullable=False)
note: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
reviewer: Mapped[Optional[str]] = mapped_column(String(255), nullable=True)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
source: Mapped[Optional[Source]] = relationship(foreign_keys=[source_id])
gtfs_dataset: Mapped[Optional[Dataset]] = relationship(foreign_keys=[gtfs_dataset_id])
gtfs_route: Mapped[Optional[GtfsRoute]] = relationship()
gtfs_stop: Mapped[Optional[GtfsStop]] = relationship()
osm_dataset: Mapped[Optional[Dataset]] = relationship(foreign_keys=[osm_dataset_id])
osm_feature: Mapped[Optional[OsmFeature]] = relationship()
route_match: Mapped[Optional[RouteMatch]] = relationship()
canonical_stop: Mapped[Optional[CanonicalStop]] = relationship()
canonical_stop_link: Mapped[Optional[CanonicalStopLink]] = relationship()
class MapGtfsReviewItem(Base):
__tablename__ = "map_gtfs_review_items"
__table_args__ = (UniqueConstraint("fingerprint", name="uq_map_gtfs_review_item_fingerprint"),)
id: Mapped[int] = mapped_column(Integer, primary_key=True)
fingerprint: Mapped[str] = mapped_column(String(255), nullable=False, index=True)
queue: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
item_type: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
status: Mapped[str] = mapped_column(String(64), nullable=False, default="open", index=True)
severity: Mapped[str] = mapped_column(String(32), nullable=False, default="info", index=True)
title: Mapped[str] = mapped_column(Text, nullable=False)
description: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
source_id: Mapped[Optional[int]] = mapped_column(ForeignKey("sources.id"), nullable=True, index=True)
dataset_id: Mapped[Optional[int]] = mapped_column(ForeignKey("datasets.id"), nullable=True, index=True)
gtfs_route_id: Mapped[Optional[int]] = mapped_column(ForeignKey("gtfs_routes.id"), nullable=True, index=True)
gtfs_stop_id: Mapped[Optional[int]] = mapped_column(ForeignKey("gtfs_stops.id"), nullable=True, index=True)
osm_feature_id: Mapped[Optional[int]] = mapped_column(ForeignKey("osm_features.id"), nullable=True, index=True)
route_match_id: Mapped[Optional[int]] = mapped_column(ForeignKey("route_matches.id"), nullable=True, index=True)
canonical_stop_id: Mapped[Optional[int]] = mapped_column(ForeignKey("canonical_stops.id"), nullable=True, index=True)
canonical_stop_link_id: Mapped[Optional[int]] = mapped_column(ForeignKey("canonical_stop_links.id"), nullable=True, index=True)
decision_id: Mapped[Optional[int]] = mapped_column(ForeignKey("map_gtfs_decisions.id"), nullable=True, index=True)
evidence_hash: Mapped[Optional[str]] = mapped_column(String(128), nullable=True, index=True)
metadata_json: Mapped[Optional[str]] = mapped_column(Text, nullable=True)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=now_utc, nullable=False, index=True)
resolved_at: Mapped[Optional[datetime]] = mapped_column(DateTime(timezone=True), nullable=True, index=True)
source: Mapped[Optional[Source]] = relationship(foreign_keys=[source_id])
dataset: Mapped[Optional[Dataset]] = relationship(foreign_keys=[dataset_id])
gtfs_route: Mapped[Optional[GtfsRoute]] = relationship()
gtfs_stop: Mapped[Optional[GtfsStop]] = relationship()
osm_feature: Mapped[Optional[OsmFeature]] = relationship()
route_match: Mapped[Optional[RouteMatch]] = relationship()
canonical_stop: Mapped[Optional[CanonicalStop]] = relationship()
canonical_stop_link: Mapped[Optional[CanonicalStopLink]] = relationship()
decision: Mapped[Optional[MapGtfsDecision]] = relationship()
class JourneySearchCache(Base):
__tablename__ = "journey_search_cache"

7
app/osm_storage.py
View File

@@ -293,7 +293,12 @@ def query_osm_features(
dataset.id: dataset
for dataset in session.scalars(select(Dataset).where(Dataset.id.in_([int(value) for value in dataset_ids]))).all()
}
materialized_ids = _materialized_ids_by_identity(session, list(datasets)) if prefer_materialized_ids else {}
sidecar_dataset_ids = [
dataset_id
for dataset_id, dataset in datasets.items()
if features_are_sidecar(dataset)
]
materialized_ids = _materialized_ids_by_identity(session, sidecar_dataset_ids) if prefer_materialized_ids else {}
rows: list[OsmFeature] = []
main_dataset_ids = [dataset_id for dataset_id, dataset in datasets.items() if not features_are_sidecar(dataset)]
if main_dataset_ids:

137
app/pipeline/gtfs.py
View File

@@ -15,6 +15,8 @@ from sqlalchemy import func, select, text
from sqlalchemy.orm import Session
from app.config import settings
from app.decision_replay import apply_gtfs_decision_replay, capture_gtfs_decision_replay
from app.gtfs_diff import create_gtfs_update_diff_from_stage
from app.gtfs_storage import GTFS_STORAGE_MAIN, GTFS_STORAGE_METADATA_KEY, GTFS_STORAGE_SIDECAR_STOP_TIMES, effective_gtfs_timetable_storage
from app.models import (
Dataset,
@@ -64,6 +66,7 @@ GTFS_EXTENDED_MODE_RANGES = [
GTFS_IMPORTER_VERSION = "gtfs_import_v6_sidecar_stop_times"
REQUIRED_FILES = {"agency.txt", "stops.txt", "routes.txt", "trips.txt", "stop_times.txt"}
GTFS_CALENDAR_WEEKDAY_COLUMNS = ["monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday"]
GTFS_STAGE_BATCH_SIZE = 50_000
ProgressCallback = Callable[[str, str, int | None, int | None, dict[str, Any] | None], None]
@@ -82,7 +85,7 @@ def run_gtfs_source(session: Session, source: Source, progress_callback: Progres
)
.order_by(Dataset.id.desc())
)
if existing is not None and _dataset_importer_version(existing) == GTFS_IMPORTER_VERSION:
if existing is not None and _dataset_import_can_reuse(existing):
return existing
return import_gtfs_zip(session=session, source=source, zip_path=local_path, source_hash=source_hash, progress_callback=progress_callback)
@@ -395,15 +398,96 @@ def _activate_staged_gtfs(
dataset = session.get(Dataset, dataset.id) or dataset
source = session.get(Source, source.id) or source
replaced_datasets = [existing for existing in list(source.datasets) if existing.id != dataset.id and existing.kind == "gtfs"]
previous_dataset = next(
(
existing
for existing in sorted(replaced_datasets, key=lambda item: (not item.is_active, item.created_at, item.id))
if existing.status == "imported"
),
None,
)
for existing in source.datasets:
if existing.id != dataset.id:
existing.is_active = False
copy_stop_times = _copy_stop_times_to_main(summary)
heavy_index_drop = copy_stop_times and _should_drop_indexes_for_activation(stage_path)
diff_result: dict[str, Any] | None = None
decision_replay_plan: dict[str, Any] | None = None
if heavy_index_drop:
_emit_progress(progress_callback, "gtfs_activation_indexes_dropped", "Dropping heavy GTFS lookup indexes before bulk activation.", None, None, None)
_drop_gtfs_bulk_indexes(session.connection())
try:
if previous_dataset is not None:
_emit_progress(
progress_callback,
"gtfs_update_diff_started",
f"Comparing staged GTFS dataset #{dataset.id} with previous dataset #{previous_dataset.id}.",
None,
None,
{"previous_dataset_id": previous_dataset.id, "new_dataset_id": dataset.id},
)
try:
diff_result = create_gtfs_update_diff_from_stage(
session,
source=source,
previous_dataset=previous_dataset,
new_dataset=dataset,
stage_path=stage_path,
)
except Exception as exc: # noqa: BLE001 - update diff must not block activation
diff_result = None
summary["gtfs_update_diff_error"] = str(exc)
_emit_progress(
progress_callback,
"gtfs_update_diff_failed",
"GTFS update diff failed; continuing dataset activation.",
None,
None,
{"error": str(exc), "previous_dataset_id": previous_dataset.id, "new_dataset_id": dataset.id},
)
if diff_result is not None:
summary["gtfs_update_diff"] = {
"id": diff_result["id"],
"summary": diff_result["summary"],
}
_emit_progress(
progress_callback,
"gtfs_update_diff_completed",
"Stored GTFS update diff before replacing the previous dataset.",
None,
None,
diff_result,
)
try:
decision_replay_plan = capture_gtfs_decision_replay(
session,
previous_dataset,
diff_run_id=None if diff_result is None else int(diff_result["id"]),
)
summary["gtfs_decision_replay"] = {
"captured": len(decision_replay_plan.get("decisions", [])),
"diff_run_id": decision_replay_plan.get("diff_run_id"),
"previous_dataset_id": previous_dataset.id,
}
_emit_progress(
progress_callback,
"gtfs_decision_replay_captured",
"Captured existing GTFS-to-map decisions for update replay.",
None,
None,
summary["gtfs_decision_replay"],
)
except Exception as exc: # noqa: BLE001 - replay capture must not block activation
decision_replay_plan = None
summary["gtfs_decision_replay_error"] = str(exc)
_emit_progress(
progress_callback,
"gtfs_decision_replay_capture_failed",
"GTFS decision replay capture failed; continuing dataset activation.",
None,
None,
{"error": str(exc), "previous_dataset_id": previous_dataset.id, "new_dataset_id": dataset.id},
)
if replaced_datasets:
_emit_progress(
progress_callback,
@@ -511,6 +595,33 @@ def _activate_staged_gtfs(
],
progress_callback,
)
if decision_replay_plan is not None:
try:
replay_result = apply_gtfs_decision_replay(
session,
decision_replay_plan,
new_dataset=dataset,
diff_run_id=None if diff_result is None else int(diff_result["id"]),
)
summary["gtfs_decision_replay"] = replay_result
_emit_progress(
progress_callback,
"gtfs_decision_replay_completed",
"Replayed unchanged GTFS-to-map decisions and queued changed decisions for review.",
None,
None,
replay_result,
)
except Exception as exc: # noqa: BLE001 - replay must not block activation
summary["gtfs_decision_replay_error"] = str(exc)
_emit_progress(
progress_callback,
"gtfs_decision_replay_failed",
"GTFS decision replay failed; continuing dataset activation.",
None,
None,
{"error": str(exc), "new_dataset_id": dataset.id},
)
finally:
if heavy_index_drop:
_emit_progress(progress_callback, "gtfs_activation_indexes_rebuilding", "Rebuilding GTFS lookup indexes after bulk activation.", None, None, None)
@@ -557,10 +668,7 @@ def _copy_stage_table(
rows = cursor.fetchmany(GTFS_STAGE_BATCH_SIZE)
if not rows:
break
payload = [
{"dataset_id": dataset_id, **{column: row[index] for index, column in enumerate(columns)}}
for row in rows
]
payload = [_copy_stage_row_payload(table, dataset_id, columns, row) for row in rows]
session.execute(text(insert_sql), payload)
copied += len(rows)
_emit_progress(
@@ -573,6 +681,15 @@ def _copy_stage_table(
)
def _copy_stage_row_payload(table: str, dataset_id: int, columns: list[str], row: sqlite3.Row | tuple[Any, ...]) -> dict[str, Any]:
payload = {"dataset_id": dataset_id, **{column: row[index] for index, column in enumerate(columns)}}
if table == "gtfs_calendars":
for column in GTFS_CALENDAR_WEEKDAY_COLUMNS:
if column in payload:
payload[column] = _bool_flag(payload[column])
return payload
def _should_drop_indexes_for_activation(stage_path: Path) -> bool:
if settings.is_postgresql_database:
return False
@@ -1325,3 +1442,13 @@ def _dataset_importer_version(dataset: Dataset) -> str:
return str(json.loads(dataset.metadata_json or "{}").get("importer") or "")
except json.JSONDecodeError:
return ""
def _dataset_import_can_reuse(dataset: Dataset) -> bool:
try:
metadata = json.loads(dataset.metadata_json or "{}")
except json.JSONDecodeError:
return False
if str(metadata.get("importer") or "") != GTFS_IMPORTER_VERSION:
return False
return int(metadata.get("stop_times_import_limit") or 0) == int(settings.gtfs_stop_times_import_limit or 0)

172
app/pipeline/matcher.py
View File

@@ -2,13 +2,15 @@ from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import json
from typing import Callable, Optional
from typing import Callable, Optional, Sequence
from shapely.geometry import LineString, MultiLineString, Point, shape
from sqlalchemy import delete, select
from sqlalchemy.orm import Session
from app.harmonization import active_harmonized_gtfs_shadowed_route_ids
from app.config import settings
from app.models import Dataset, GtfsRoute, MatchRule, OsmFeature, RouteMatch
from app.osm_storage import ensure_main_osm_feature, osm_feature_bbox, query_osm_features
@@ -34,7 +36,7 @@ MAX_EXACT_REF_CANDIDATES = 120
OSM_SCOPE_NEAR_DISTANCE_DEG = 0.15
GEOMETRY_PROXIMITY_DEG = 0.0035
GEOMETRY_SAMPLE_POINTS = 24
MATCHER_VERSION = "matcher_v4_scope_spatial_manual_rules"
MATCHER_VERSION = "matcher_v5_spatial_candidate_gate"
ProgressCallback = Callable[[str, str, int | None, int | None, dict[str, object] | None], None]
@@ -78,6 +80,7 @@ def run_route_matching(
*,
progress_callback: ProgressCallback | None = None,
batch_size: int | None = None,
gtfs_dataset_ids: Sequence[int] | None = None,
) -> dict[str, object]:
"""Match active GTFS routes against active OSM route features."""
active_datasets = session.execute(
@@ -86,22 +89,37 @@ def run_route_matching(
if not active_datasets:
return {"routes": 0, "matches": 0, "missing": 0}
dataset_source_ids = {int(dataset_id): int(source_id) for dataset_id, _, source_id in active_datasets}
gtfs_dataset_ids = [int(dataset_id) for dataset_id, kind, _ in active_datasets if kind == "gtfs"]
active_gtfs_dataset_ids = [int(dataset_id) for dataset_id, kind, _ in active_datasets if kind == "gtfs"]
osm_dataset_ids = [int(dataset_id) for dataset_id, kind, _ in active_datasets if kind == "osm_geojson"]
if not gtfs_dataset_ids:
if gtfs_dataset_ids is None:
selected_gtfs_dataset_ids = active_gtfs_dataset_ids
else:
requested_gtfs_dataset_ids = {int(dataset_id) for dataset_id in gtfs_dataset_ids}
selected_gtfs_dataset_ids = [
dataset_id
for dataset_id in active_gtfs_dataset_ids
if dataset_id in requested_gtfs_dataset_ids
]
if not selected_gtfs_dataset_ids:
return {"routes": 0, "matches": 0, "missing": 0}
route_row_ids = session.scalars(
select(GtfsRoute.id)
.where(GtfsRoute.dataset_id.in_(gtfs_dataset_ids))
.order_by(GtfsRoute.dataset_id, GtfsRoute.route_id, GtfsRoute.id)
).all()
all_route_row_ids = [
int(route_id)
for route_id in session.scalars(
select(GtfsRoute.id)
.where(GtfsRoute.dataset_id.in_(selected_gtfs_dataset_ids))
.order_by(GtfsRoute.dataset_id, GtfsRoute.route_id, GtfsRoute.id)
).all()
]
shadowed_route_ids = active_harmonized_gtfs_shadowed_route_ids(session)
route_row_ids = [route_id for route_id in all_route_row_ids if route_id not in shadowed_route_ids]
shadowed_skipped = len(all_route_row_ids) - len(route_row_ids)
# Reconcile current match rows from auto scoring plus durable manual rules.
total_routes = len(route_row_ids)
if total_routes == 0:
return {"routes": 0, "matches": 0, "missing": 0}
return {"routes": 0, "matches": 0, "missing": 0, "shadowed_skipped": shadowed_skipped}
dependency = _route_matching_dependency(session, active_datasets)
dependency = _route_matching_dependency(session, active_datasets, shadowed_route_ids=shadowed_route_ids)
run = start_pipeline_run(
session,
stage=STAGE_MATCH_ROUTES,
@@ -117,11 +135,49 @@ def run_route_matching(
f"Matching {total_routes} GTFS routes in batches of {effective_batch_size}.",
0,
total_routes,
{"gtfs_datasets": gtfs_dataset_ids, "osm_datasets": osm_dataset_ids, "batch_size": effective_batch_size},
{
"gtfs_datasets": selected_gtfs_dataset_ids,
"osm_datasets": osm_dataset_ids,
"batch_size": effective_batch_size,
"shadowed_skipped": shadowed_skipped,
},
)
manual_rules = _manual_match_rules(session)
osm_scope_bbox = osm_feature_bbox(session, osm_dataset_ids, kinds=["route"])
counts = {"routes": total_routes, "matches": 0, "missing": 0, "manual": 0, "created": 0, "updated": 0, "unchanged": 0}
_emit_progress(
progress_callback,
"route_matching_osm_index_started",
"Loading OSM route candidates for matcher.",
0,
None,
{"osm_datasets": osm_dataset_ids},
)
osm_routes = query_osm_features(
session,
osm_dataset_ids,
kinds=["route"],
prefer_materialized_ids=False,
)
osm_route_index = _build_osm_route_index(osm_routes)
feature_profile_cache: dict[tuple[int, str, str], _GeometryProfile | None] = {}
_emit_progress(
progress_callback,
"route_matching_osm_index_completed",
f"Loaded {len(osm_routes)} OSM route candidates.",
len(osm_routes),
None,
{"osm_routes": len(osm_routes)},
)
counts = {
"routes": total_routes,
"matches": 0,
"missing": 0,
"manual": 0,
"created": 0,
"updated": 0,
"unchanged": 0,
"shadowed_skipped": shadowed_skipped,
}
scoped_counts = {"in_osm_scope": 0, "near_osm_scope": 0, "outside_osm_scope": 0, "unknown_scope": 0}
processed = 0
for chunk in _chunks_int(route_row_ids, effective_batch_size):
@@ -137,6 +193,8 @@ def run_route_matching(
dataset_source_ids=dataset_source_ids,
manual_rules=manual_rules,
osm_scope_bbox=osm_scope_bbox,
osm_route_index=osm_route_index,
feature_profile_cache=feature_profile_cache,
scoped_counts=scoped_counts,
)
counts["matches"] += batch_counts["matches"]
@@ -178,7 +236,12 @@ def run_route_matching(
return result
def _route_matching_dependency(session: Session, active_datasets) -> dict[str, object]:
def _route_matching_dependency(
session: Session,
active_datasets,
*,
shadowed_route_ids: set[int] | None = None,
) -> dict[str, object]:
datasets = [
{"id": int(dataset_id), "kind": str(kind), "source_id": int(source_id), "sha256": _dataset_sha(session, int(dataset_id))}
for dataset_id, kind, source_id in active_datasets
@@ -193,7 +256,15 @@ def _route_matching_dependency(session: Session, active_datasets) -> dict[str, o
}
for rule in session.scalars(select(MatchRule).order_by(MatchRule.id)).all()
]
return {"version": MATCHER_VERSION, "active_datasets": datasets, "manual_rules": rules}
return {
"version": MATCHER_VERSION,
"active_datasets": datasets,
"manual_rules": rules,
"shadowed_routes": {
"count": len(shadowed_route_ids or set()),
"digest": _int_set_digest(shadowed_route_ids or set()),
},
}
def _dataset_sha(session: Session, dataset_id: int) -> str | None:
@@ -201,6 +272,13 @@ def _dataset_sha(session: Session, dataset_id: int) -> str | None:
return None if dataset is None else dataset.sha256
def _int_set_digest(values: set[int]) -> str:
if not values:
return ""
payload = ",".join(str(value) for value in sorted(values))
return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:32]
def _match_route_batch(
*,
session: Session,
@@ -209,6 +287,8 @@ def _match_route_batch(
dataset_source_ids: dict[int, int],
manual_rules: list[_ManualMatchRule],
osm_scope_bbox: tuple[float | None, float | None, float | None, float | None],
osm_route_index: _OsmRouteIndex,
feature_profile_cache: dict[tuple[int, str, str], _GeometryProfile | None],
scoped_counts: dict[str, int],
) -> dict[str, int]:
matches = 0
@@ -265,10 +345,13 @@ def _match_route_batch(
best_score = 0.0
best_reasons: dict[str, object] = {}
route_geometry_profile = _geometry_profile(route.geometry_geojson)
for feature in candidate_osm_routes_for_route(session, route, osm_dataset_ids):
for feature in _candidate_osm_routes(route, osm_route_index):
if _is_rejected_pair(manual_rules, route, route_source_id, feature, dataset_source_ids.get(feature.dataset_id)):
continue
feature_geometry_profile = _geometry_profile(feature.geometry_geojson)
feature_key = _feature_identity_key(feature)
if feature_key not in feature_profile_cache:
feature_profile_cache[feature_key] = _geometry_profile(feature.geometry_geojson)
feature_geometry_profile = feature_profile_cache[feature_key]
score, reasons = score_route_pair(
route,
feature,
@@ -393,22 +476,26 @@ def _build_osm_route_index(osm_routes: list[OsmFeature]) -> _OsmRouteIndex:
def _candidate_osm_routes(route: GtfsRoute, index: _OsmRouteIndex) -> list[OsmFeature]:
selected: list[OsmFeature] = []
seen: set[int] = set()
seen: set[tuple[int, str, str]] = set()
def add(features: list[OsmFeature], *, require_compatible_mode: bool = True) -> None:
def add(features: list[OsmFeature], *, require_compatible_mode: bool = True, require_spatial: bool = False) -> None:
for feature in features:
if feature.id in seen:
key = (feature.dataset_id, feature.osm_type, feature.osm_id)
if key in seen:
continue
if require_compatible_mode and not _mode_compatible(route.mode or "", feature.mode or ""):
continue
seen.add(feature.id)
if require_spatial and not _candidate_is_spatially_plausible(route, feature):
continue
seen.add(key)
selected.append(feature)
route_ref = norm_ref(route.short_name or route.route_id)
route_has_bbox = _has_known_bbox((route.min_lon, route.min_lat, route.max_lon, route.max_lat))
if route_ref:
add(index.by_ref.get(route_ref, []))
add(index.by_ref.get(route_ref, []), require_spatial=route_has_bbox)
if route.route_key:
add(index.by_route_key.get(route.route_key, []))
add(index.by_route_key.get(route.route_key, []), require_spatial=route_has_bbox)
if selected:
return _spatially_ranked_candidates(route, selected, MAX_EXACT_REF_CANDIDATES)
@@ -431,7 +518,7 @@ def _candidate_osm_routes(route: GtfsRoute, index: _OsmRouteIndex) -> list[OsmFe
near_candidates.append((distance, feature))
fallback_limit = MAX_FALLBACK_CANDIDATES_WITH_REF if route_ref else MAX_FALLBACK_CANDIDATES_WITHOUT_REF
fallback = [feature for _, feature in sorted(near_candidates, key=lambda item: item[0])[:fallback_limit]]
if not fallback:
if not fallback and not route_has_bbox:
fallback = mode_candidates[:fallback_limit]
add(fallback)
return _spatially_ranked_candidates(route, selected, fallback_limit)
@@ -443,17 +530,22 @@ def candidate_osm_routes_for_route(session: Session, route: GtfsRoute, osm_datas
selected: list[OsmFeature] = []
seen: set[tuple[int, str, str]] = set()
def add(features: list[OsmFeature], *, require_compatible_mode: bool = True) -> None:
def add(features: list[OsmFeature], *, require_compatible_mode: bool = True, require_spatial: bool = False) -> None:
for feature in features:
key = (feature.dataset_id, feature.osm_type, feature.osm_id)
if key in seen:
continue
if require_compatible_mode and not _mode_compatible(route.mode or "", feature.mode or ""):
continue
if require_spatial and not _candidate_is_spatially_plausible(route, feature):
continue
seen.add(key)
selected.append(feature)
route_ref = norm_ref(route.short_name or route.route_id)
gtfs_bbox = (route.min_lon, route.min_lat, route.max_lon, route.max_lat)
route_has_bbox = _has_known_bbox(gtfs_bbox)
exact_ref_bbox = _expanded_bbox(gtfs_bbox, OSM_SCOPE_NEAR_DISTANCE_DEG) if route_has_bbox else None
route_keys = [key for key in [route.route_key, route_ref] if key]
for route_key in dict.fromkeys(route_keys):
add(
@@ -462,14 +554,15 @@ def candidate_osm_routes_for_route(session: Session, route: GtfsRoute, osm_datas
osm_dataset_ids,
kinds=["route"],
route_key=route_key,
)
bbox=exact_ref_bbox,
),
require_spatial=route_has_bbox,
)
if selected:
return _spatially_ranked_candidates(route, selected, MAX_EXACT_REF_CANDIDATES)
gtfs_bbox = (route.min_lon, route.min_lat, route.max_lon, route.max_lat)
compatible_modes = sorted(MODE_GROUPS.get(route.mode or "", {route.mode or ""}) - {""})
if not any(value is None for value in gtfs_bbox):
if route_has_bbox:
bbox = _expanded_bbox(gtfs_bbox, 0.10)
add(
query_osm_features(
@@ -482,7 +575,7 @@ def candidate_osm_routes_for_route(session: Session, route: GtfsRoute, osm_datas
),
require_compatible_mode=False,
)
if not selected:
if not selected and not route_has_bbox:
add(
query_osm_features(
session,
@@ -634,6 +727,27 @@ def route_match_scope(route: GtfsRoute, osm_scope_bbox: tuple[float | None, floa
return "outside_osm_scope"
def _candidate_is_spatially_plausible(route: GtfsRoute, feature: OsmFeature) -> bool:
route_bbox = (route.min_lon, route.min_lat, route.max_lon, route.max_lat)
feature_bbox = (feature.min_lon, feature.min_lat, feature.max_lon, feature.max_lat)
if not _has_known_bbox(route_bbox):
return True
if not _has_known_bbox(feature_bbox):
return False
if bbox_overlap(route_bbox, feature_bbox):
return True
distance = approx_bbox_center_distance_deg(route_bbox, feature_bbox)
return distance is not None and distance < OSM_SCOPE_NEAR_DISTANCE_DEG
def _has_known_bbox(bbox: tuple[float | None, float | None, float | None, float | None]) -> bool:
return not any(value is None for value in bbox)
def _feature_identity_key(feature: OsmFeature) -> tuple[int, str, str]:
return (int(feature.dataset_id), str(feature.osm_type), str(feature.osm_id))
def _combined_bbox(features: list[OsmFeature]) -> tuple[float | None, float | None, float | None, float | None]:
boxes = [
(feature.min_lon, feature.min_lat, feature.max_lon, feature.max_lat)

26
app/pipeline/route_layer.py
View File

@@ -11,6 +11,7 @@ from sqlalchemy.orm import Session
from app.config import settings
from app.gtfs_storage import all_scheduled_stop_ids, stop_times_by_trip as storage_stop_times_by_trip
from app.harmonization import active_harmonized_gtfs_shadowed_route_ids
from app.models import (
CanonicalStop,
CanonicalStopLink,
@@ -36,7 +37,7 @@ from app.pipeline.utils import bbox_overlap, geometry_json_and_bbox, norm_ref, n
from app.spatial import analyze_postgresql_tables, refresh_postgis_geometries, using_postgresql
ROUTE_LAYER_VERSION = "route_layer_v3_stop_alias_matching"
ROUTE_LAYER_VERSION = "route_layer_v4_harmonized_route_ownership"
GTFS_ROUTE_PATTERN_NULL_SHAPE = "__route__"
OSM_STOP_LINK_RADIUS_DEG = 0.0018
OSM_STOP_NAME_LINK_RADIUS_DEG = 0.0032
@@ -144,6 +145,7 @@ def rebuild_route_layer(
"route_patterns_updated": pattern_result.get("route_patterns_updated", 0),
"route_patterns_reused": pattern_result.get("route_patterns_reused", 0),
"route_patterns_removed": pattern_result.get("route_patterns_removed", 0),
"shadowed_routes_skipped": pattern_result.get("shadowed_routes_skipped", 0),
"route_pattern_links": pattern_result["route_pattern_links"],
"trip_pattern_links": pattern_result["trip_pattern_links"],
"route_pattern_stops": pattern_result["route_pattern_stops"],
@@ -161,6 +163,7 @@ def _route_layer_dependency(session: Session) -> dict[str, object]:
for dataset in session.scalars(select(Dataset).where(Dataset.is_active.is_(True)).order_by(Dataset.kind, Dataset.id)).all()
]
match_rows = session.execute(select(RouteMatch.id, RouteMatch.gtfs_route_id, RouteMatch.osm_feature_id, RouteMatch.status, RouteMatch.updated_at).order_by(RouteMatch.id)).all()
shadowed_route_ids = active_harmonized_gtfs_shadowed_route_ids(session)
match_signature = dependency_hash(
[
[
@@ -177,6 +180,10 @@ def _route_layer_dependency(session: Session) -> dict[str, object]:
"version": ROUTE_LAYER_VERSION,
"active_datasets": active_datasets,
"route_matches": {"count": len(match_rows), "signature": match_signature},
"shadowed_routes": {
"count": len(shadowed_route_ids),
"signature": dependency_hash(sorted(shadowed_route_ids)) if shadowed_route_ids else "",
},
}
@@ -945,14 +952,15 @@ def _build_route_patterns(
) -> dict[str, int]:
osm_candidates = _osm_route_candidates(session, progress_callback=progress_callback)
overrides = _route_layer_overrides(session)
seeds = _gtfs_pattern_seeds(session)
shadowed_route_ids = active_harmonized_gtfs_shadowed_route_ids(session)
seeds, shadowed_routes_skipped = _gtfs_pattern_seeds(session, shadowed_route_ids=shadowed_route_ids)
_emit_progress(
progress_callback,
"route_layer_pattern_seeds",
f"Loaded {len(seeds)} GTFS route-pattern seeds.",
0,
len(seeds),
{"seeds": len(seeds)},
{"seeds": len(seeds), "shadowed_routes_skipped": shadowed_routes_skipped},
)
link_count = 0
stop_count = 0
@@ -1256,6 +1264,7 @@ def _build_route_patterns(
"trip_pattern_links": trip_link_count,
"route_pattern_stops": stop_count,
"gtfs_proposed_patterns": proposed_count,
"shadowed_routes_skipped": shadowed_routes_skipped,
}
_emit_progress(
progress_callback,
@@ -1494,10 +1503,11 @@ def _route_layer_overrides(session: Session) -> _RouteLayerOverrides:
return _RouteLayerOverrides(accepted_by_gtfs_route_id=accepted, rejected_by_gtfs_route_id=rejected)
def _gtfs_pattern_seeds(session: Session) -> list[_GtfsPatternSeed]:
def _gtfs_pattern_seeds(session: Session, *, shadowed_route_ids: set[int] | None = None) -> tuple[list[_GtfsPatternSeed], int]:
active_gtfs_dataset_ids = _active_dataset_ids(session, "gtfs")
if not active_gtfs_dataset_ids:
return []
return [], 0
shadowed_route_ids = shadowed_route_ids or set()
rows = session.execute(
select(GtfsRoute, GtfsTrip.shape_id, func.min(GtfsTrip.trip_id))
.join(GtfsTrip, and_(GtfsTrip.dataset_id == GtfsRoute.dataset_id, GtfsTrip.route_id == GtfsRoute.route_id))
@@ -1525,7 +1535,11 @@ def _gtfs_pattern_seeds(session: Session) -> list[_GtfsPatternSeed]:
for dataset_id, shape_id, geometry, min_lon, min_lat, max_lon, max_lat in shape_rows
}
seeds = []
skipped = 0
for route, shape_id, trip_id in rows:
if int(route.id) in shadowed_route_ids:
skipped += 1
continue
geometry_text = None
geometry_source = "none"
bbox = (route.min_lon, route.min_lat, route.max_lon, route.max_lat)
@@ -1556,7 +1570,7 @@ def _gtfs_pattern_seeds(session: Session) -> list[_GtfsPatternSeed]:
center_point=center_point,
)
)
return seeds
return seeds, skipped
def _choose_osm_candidate(

14
app/pipeline/sample_data.py
View File

@@ -19,6 +19,11 @@ from app.models import (
GtfsAgency,
GtfsCalendar,
GtfsCalendarDate,
GtfsFeedDiffItem,
GtfsFeedDiffRun,
GtfsHarmonizedSnapshot,
GtfsHarmonizedSnapshotDataset,
GtfsHarmonizedSnapshotRoute,
GtfsRoute,
GtfsRoutePatternLink,
GtfsShape,
@@ -30,6 +35,8 @@ from app.models import (
ItineraryLeg,
Job,
JobEvent,
MapGtfsDecision,
MapGtfsReviewItem,
MatchRule,
OsmDiffState,
OsmFeature,
@@ -96,8 +103,15 @@ def clear_project_data(
ItineraryLeg,
Itinerary,
TravelRequest,
GtfsFeedDiffItem,
GtfsFeedDiffRun,
GtfsHarmonizedSnapshotRoute,
GtfsHarmonizedSnapshotDataset,
GtfsHarmonizedSnapshot,
SourceUpdateCheck,
OsmDiffState,
MapGtfsReviewItem,
MapGtfsDecision,
MatchRule,
RouteMatch,
GtfsTripRoutePatternLink,

309
app/routing.py
View File

@@ -20,12 +20,18 @@ from app.serializers import feature_collection
WALK_HEURISTIC_MPS = 1.6
BIKE_HEURISTIC_MPS = 6.0
DRIVE_HEURISTIC_MPS = 36.0
BIKE_SPEED_MPS = 4.8
DEFAULT_MAX_VISITED = 160_000
PGR_WALK_BBOX_PADDING_KM = [0.5, 1.5, 4, 10, 25]
PGR_BIKE_BBOX_PADDING_KM = [1, 3, 8, 20, 60]
PGR_DRIVE_BBOX_PADDING_KM = [2, 8, 25, 75, 200]
PGR_WALK_STATEMENT_TIMEOUT_MS = 2_500
PGR_BIKE_STATEMENT_TIMEOUT_MS = 4_000
PGR_DRIVE_STATEMENT_TIMEOUT_MS = 7_500
ROUTING_MODES = {"walk", "bike", "drive"}
ROUTING_MODE_ALIASES = {"bicycle": "bike", "cycle": "bike", "cycling": "bike", "car": "drive"}
ROUTE_CACHE_TTL_SECONDS = 15 * 60
ROUTE_CACHE_MAX_ENTRIES = 512
_route_cache_lock = threading.RLock()
@@ -123,6 +129,43 @@ def _metadata(dataset: Dataset) -> dict[str, object]:
return value if isinstance(value, dict) else {}
def _normalize_routing_mode(mode: str) -> str:
normalized = ROUTING_MODE_ALIASES.get(str(mode or "").lower(), str(mode or "").lower())
if normalized not in ROUTING_MODES:
raise ValueError("mode must be walk, bike, or drive")
return normalized
def _cost_columns(mode: str) -> tuple[str, str]:
if mode == "drive":
return "drive_cost_s", "reverse_drive_cost_s"
return "walk_cost_s", "reverse_walk_cost_s"
def _heuristic_mps(mode: str) -> float:
if mode == "walk":
return WALK_HEURISTIC_MPS
if mode == "bike":
return BIKE_HEURISTIC_MPS
return DRIVE_HEURISTIC_MPS
def _pgrouting_padding_km(mode: str) -> list[float]:
if mode == "walk":
return PGR_WALK_BBOX_PADDING_KM
if mode == "bike":
return PGR_BIKE_BBOX_PADDING_KM
return PGR_DRIVE_BBOX_PADDING_KM
def _pgrouting_timeout_ms(mode: str) -> int:
if mode == "walk":
return PGR_WALK_STATEMENT_TIMEOUT_MS
if mode == "bike":
return PGR_BIKE_STATEMENT_TIMEOUT_MS
return PGR_DRIVE_STATEMENT_TIMEOUT_MS
def route_between_points(
db: Session,
*,
@@ -133,9 +176,11 @@ def route_between_points(
mode: str = "walk",
dataset_id: int | None = None,
max_visited: int = DEFAULT_MAX_VISITED,
allow_python_fallback: bool = True,
pgrouting_timeout_ms: int | None = None,
pgrouting_padding_km: tuple[float, ...] | list[float] | None = None,
) -> dict[str, object]:
if mode not in {"walk", "drive"}:
raise ValueError("mode must be walk or drive")
mode = _normalize_routing_mode(mode)
dataset = db.get(Dataset, dataset_id) if dataset_id is not None else active_routing_dataset(db)
if dataset is None:
raise ValueError("No routing dataset is available.")
@@ -152,7 +197,8 @@ def route_between_points(
payload = _single_point_route(start, from_lon, from_lat, to_lon, to_lat, mode, dataset_id)
_route_cache_put(cache_key, payload)
return payload
if settings.is_postgresql_database and _pgrouting_installed(db):
pgrouting_available = settings.is_postgresql_database and _pgrouting_installed(db)
if pgrouting_available:
try:
payload = _route_with_pgrouting(
db,
@@ -164,15 +210,23 @@ def route_between_points(
from_lat=from_lat,
to_lon=to_lon,
to_lat=to_lat,
timeout_ms=pgrouting_timeout_ms,
padding_km=pgrouting_padding_km,
)
_route_cache_put(cache_key, payload)
return payload
except ValueError:
pass
if not allow_python_fallback:
raise
except SQLAlchemyError:
db.rollback()
if not allow_python_fallback:
raise
heuristic_mps = WALK_HEURISTIC_MPS if mode == "walk" else DRIVE_HEURISTIC_MPS
if not allow_python_fallback:
raise ValueError("pgRouting did not find a bounded route and Python fallback is disabled.")
heuristic_mps = _heuristic_mps(mode)
queue: list[tuple[float, float, int]] = []
heapq.heappush(queue, (0.0, 0.0, start.osm_node_id))
costs: dict[int, float] = {start.osm_node_id: 0.0}
@@ -226,8 +280,7 @@ def direct_route_between_points(
dataset_id: int | None = None,
reason: str | None = None,
) -> dict[str, object]:
if mode not in {"walk", "drive"}:
raise ValueError("mode must be walk or drive")
mode = _normalize_routing_mode(mode)
dataset = db.get(Dataset, dataset_id) if dataset_id is not None else active_routing_dataset(db)
payload = _direct_route_payload(
dataset_id=0 if dataset is None else int(dataset.id),
@@ -251,8 +304,7 @@ def snap_point_to_routing_graph(
dataset_id: int | None = None,
max_distance_m: float = 250,
) -> dict[str, object] | None:
if mode not in {"walk", "drive"}:
raise ValueError("mode must be walk or drive")
mode = _normalize_routing_mode(mode)
dataset = db.get(Dataset, dataset_id) if dataset_id is not None else active_routing_dataset(db)
if dataset is None:
return None
@@ -288,8 +340,8 @@ def _snap_point_to_routing_edge_postgresql(
mode: str,
max_distance_m: float,
) -> dict[str, object] | None:
cost_column = "walk_cost_s" if mode == "walk" else "drive_cost_s"
reverse_cost_column = "reverse_walk_cost_s" if mode == "walk" else "reverse_drive_cost_s"
cost_column, reverse_cost_column = _cost_columns(mode)
access_filter = f"AND {_bike_access_sql('edge')}" if mode == "bike" else ""
radius_deg = max_distance_m / 111_320
row = db.execute(
text(
@@ -319,6 +371,7 @@ def _snap_point_to_routing_edge_postgresql(
CROSS JOIN point
WHERE edge.dataset_id = :dataset_id
AND (edge.{cost_column} IS NOT NULL OR edge.{reverse_cost_column} IS NOT NULL)
{access_filter}
AND box(point(edge.max_lon, edge.max_lat), point(edge.min_lon, edge.min_lat))
&& box(
point(:lon + :radius_deg, :lat + :radius_deg),
@@ -337,13 +390,16 @@ def _snap_point_to_routing_edge_postgresql(
ST_DistanceSphere(edges.edge_geom, point.geom) AS distance_m,
CASE
WHEN edges.highway IN ('footway', 'pedestrian', 'steps') THEN 0
WHEN edges.highway IN ('path', 'cycleway', 'bridleway') THEN 1
WHEN edges.highway = 'cycleway' THEN 0
WHEN edges.highway IN ('path', 'bridleway') THEN 1
WHEN edges.highway IN ('living_street', 'residential') THEN 2
WHEN edges.highway = 'service' THEN 3
ELSE 4
END AS highway_rank,
CASE
WHEN :mode != 'walk' THEN 0
WHEN :mode NOT IN ('walk', 'bike') THEN 0
WHEN :mode = 'bike' AND edges.highway = 'cycleway' THEN -12
WHEN :mode = 'bike' AND edges.highway IN ('path', 'living_street', 'residential') THEN -4
WHEN edges.highway = 'service' THEN 20
WHEN edges.highway IN ('primary', 'primary_link', 'secondary', 'secondary_link') THEN 10
WHEN edges.highway IN ('tertiary', 'tertiary_link', 'unclassified', 'road') THEN 5
@@ -359,7 +415,9 @@ def _snap_point_to_routing_edge_postgresql(
)
ORDER BY
ST_DistanceSphere(edges.edge_geom, point.geom) + CASE
WHEN :mode != 'walk' THEN 0
WHEN :mode NOT IN ('walk', 'bike') THEN 0
WHEN :mode = 'bike' AND edges.highway = 'cycleway' THEN -12
WHEN :mode = 'bike' AND edges.highway IN ('path', 'living_street', 'residential') THEN -4
WHEN edges.highway = 'service' THEN 20
WHEN edges.highway IN ('primary', 'primary_link', 'secondary', 'secondary_link') THEN 10
WHEN edges.highway IN ('tertiary', 'tertiary_link', 'unclassified', 'road') THEN 5
@@ -455,15 +513,26 @@ def _route_with_pgrouting(
from_lat: float,
to_lon: float,
to_lat: float,
timeout_ms: int | None = None,
padding_km: tuple[float, ...] | list[float] | None = None,
) -> dict[str, object]:
cost_column = "walk_cost_s" if mode == "walk" else "drive_cost_s"
reverse_cost_column = "reverse_walk_cost_s" if mode == "walk" else "reverse_drive_cost_s"
cost_column, reverse_cost_column = _cost_columns(mode)
routing_cost = _routing_cost_expression(cost_column, mode)
reverse_routing_cost = _routing_cost_expression(reverse_cost_column, mode)
for padding_km in PGR_WALK_BBOX_PADDING_KM if mode == "walk" else PGR_DRIVE_BBOX_PADDING_KM:
reverse_routing_cost = _routing_cost_expression(reverse_cost_column, mode, reverse=True)
actual_cost_sql = "steps.cost" if mode == "bike" else f"""
CASE
WHEN steps.from_node = edge.source_osm_node_id THEN edge.{cost_column}
ELSE edge.{reverse_cost_column}
END
"""
paddings = list(padding_km or _pgrouting_padding_km(mode))
statement_timeout_ms = int(
timeout_ms if timeout_ms is not None else _pgrouting_timeout_ms(mode)
)
for padding_km in paddings:
_set_local_statement_timeout(
db,
PGR_WALK_STATEMENT_TIMEOUT_MS if mode == "walk" else PGR_DRIVE_STATEMENT_TIMEOUT_MS,
statement_timeout_ms,
)
bbox = _expanded_bbox(
min(from_lon, to_lon, start.lon, target.lon),
@@ -517,10 +586,7 @@ def _route_with_pgrouting(
edge.highway,
edge.name,
edge.geometry_geojson,
CASE
WHEN steps.from_node = edge.source_osm_node_id THEN edge.{cost_column}
ELSE edge.{reverse_cost_column}
END AS actual_cost_s
{actual_cost_sql} AS actual_cost_s
FROM steps
JOIN routing_edges AS edge ON edge.id = steps.edge
WHERE steps.edge <> -1
@@ -616,33 +682,109 @@ def _pgrouting_payload(
return payload
def _routing_cost_expression(column: str, mode: str) -> str:
if mode != "walk":
def _routing_cost_expression(column: str, mode: str, *, reverse: bool = False) -> str:
if mode == "drive":
return column
if mode == "bike":
direction_filter = f"WHEN {_bike_direction_forbidden_sql(reverse=reverse)} THEN NULL"
return f"""
CASE
WHEN {column} IS NULL THEN NULL
WHEN NOT ({_bike_access_sql()}) THEN NULL
{direction_filter}
ELSE (length_m / {BIKE_SPEED_MPS:.4f}) * CASE
WHEN highway = 'cycleway' THEN 0.82
WHEN highway = 'path' THEN 0.95
WHEN highway IN ('living_street', 'residential') THEN 1.00
WHEN highway = 'service' THEN 1.08
WHEN highway = 'track' THEN 1.10
WHEN highway IN ('unclassified', 'road', 'tertiary', 'tertiary_link') THEN 1.12
WHEN highway IN ('secondary', 'secondary_link') THEN 1.22
WHEN highway IN ('primary', 'primary_link') THEN 1.35
WHEN highway IN ('footway', 'pedestrian', 'bridleway') THEN 1.40
ELSE 1.05
END
END
"""
return f"""
CASE
WHEN {column} IS NULL THEN NULL
ELSE {column} * CASE
WHEN highway IN ('footway', 'pedestrian') THEN 0.70
WHEN highway = 'path' THEN 0.78
WHEN highway = 'steps' THEN 0.95
WHEN highway = 'cycleway' THEN 1.05
WHEN highway = 'bridleway' THEN 1.10
WHEN highway IN ('living_street', 'track') THEN 1.15
WHEN highway IN ('residential', 'service') THEN 1.35
WHEN highway IN ('unclassified', 'road') THEN 1.55
WHEN highway IN ('tertiary', 'tertiary_link') THEN 1.80
WHEN highway IN ('secondary', 'secondary_link') THEN 2.15
WHEN highway IN ('primary', 'primary_link') THEN 2.50
ELSE 1.30
WHEN highway IN ('footway', 'pedestrian') THEN 0.92
WHEN highway = 'path' THEN 0.94
WHEN highway = 'steps' THEN 1.05
WHEN highway = 'cycleway' THEN 1.02
WHEN highway = 'bridleway' THEN 1.05
WHEN highway = 'living_street' THEN 1.00
WHEN highway IN ('residential', 'service') THEN 1.05
WHEN highway = 'track' THEN 1.10
WHEN highway IN ('unclassified', 'road') THEN 1.12
WHEN highway IN ('tertiary', 'tertiary_link') THEN 1.18
WHEN highway IN ('secondary', 'secondary_link') THEN 1.28
WHEN highway IN ('primary', 'primary_link') THEN 1.40
ELSE 1.08
END
END
"""
def _routing_tags_sql(alias: str | None = None) -> str:
prefix = f"{alias}." if alias else ""
return f"COALESCE(NULLIF({prefix}tags_json, '')::jsonb, '{{}}'::jsonb)"
def _bike_access_sql(alias: str | None = None) -> str:
tags = _routing_tags_sql(alias)
highway = f"{alias}.highway" if alias else "highway"
return f"""
(
{highway} NOT IN ('motorway', 'motorway_link', 'steps', 'platform')
AND NOT (
COALESCE({tags} ->> 'bicycle', '') IN ('no', 'private')
OR (
COALESCE({tags} ->> 'access', '') IN ('no', 'private')
AND COALESCE({tags} ->> 'bicycle', '') NOT IN ('yes', 'designated', 'permissive', 'destination')
AND COALESCE({tags} ->> 'vehicle', '') NOT IN ('yes', 'designated', 'permissive', 'destination')
)
)
AND NOT (
{highway} IN ('footway', 'pedestrian', 'bridleway')
AND COALESCE({tags} ->> 'bicycle', '') NOT IN ('yes', 'designated', 'permissive', 'destination')
)
AND NOT (
{highway} IN ('trunk', 'trunk_link')
AND COALESCE({tags} ->> 'bicycle', '') NOT IN ('yes', 'designated', 'permissive', 'destination')
)
)
"""
def _bike_direction_forbidden_sql(alias: str | None = None, *, reverse: bool) -> str:
tags = _routing_tags_sql(alias)
highway = f"{alias}.highway" if alias else "highway"
bicycle_direction_key = "bicycle:backward" if reverse else "bicycle:forward"
one_way_condition = (
f"COALESCE({tags} ->> 'oneway', '') IN ('yes', 'true', '1') "
f"OR COALESCE({tags} ->> 'junction', '') = 'roundabout' "
f"OR {highway} = 'motorway'"
if reverse
else f"COALESCE({tags} ->> 'oneway', '') = '-1'"
)
return f"""
(
(
{one_way_condition}
)
AND COALESCE({tags} ->> 'oneway:bicycle', '') != 'no'
AND COALESCE({tags} ->> 'bicycle:oneway', '') != 'no'
AND COALESCE({tags} ->> '{bicycle_direction_key}', '') NOT IN ('yes', 'designated', 'permissive')
)
"""
def _nearest_node(db: Session, dataset_id: int, lon: float, lat: float, mode: str) -> _GraphNode | None:
cost_column = "walk_cost_s" if mode == "walk" else "drive_cost_s"
reverse_cost_column = "reverse_walk_cost_s" if mode == "walk" else "reverse_drive_cost_s"
cost_column, reverse_cost_column = _cost_columns(mode)
access_filter = f"AND {_bike_access_sql('edge')}" if mode == "bike" else ""
row = None
for candidate_limit in (64, 512, 4096):
row = db.execute(
@@ -663,6 +805,7 @@ def _nearest_node(db: Session, dataset_id: int, lon: float, lat: float, mode: st
SELECT 1
FROM routing_edges AS edge
WHERE edge.dataset_id = :dataset_id
{access_filter}
AND (
(edge.source_osm_node_id = nearest.osm_node_id AND edge.{cost_column} IS NOT NULL)
OR (edge.target_osm_node_id = nearest.osm_node_id AND edge.{reverse_cost_column} IS NOT NULL)
@@ -686,8 +829,7 @@ def _nearest_node(db: Session, dataset_id: int, lon: float, lat: float, mode: st
def _outgoing_edges(db: Session, dataset_id: int, node_id: int, mode: str) -> list[_Traversal]:
cost_column = "walk_cost_s" if mode == "walk" else "drive_cost_s"
reverse_cost_column = "reverse_walk_cost_s" if mode == "walk" else "reverse_drive_cost_s"
cost_column, reverse_cost_column = _cost_columns(mode)
rows = db.execute(
text(
f"""
@@ -695,6 +837,7 @@ def _outgoing_edges(db: Session, dataset_id: int, node_id: int, mode: str) -> li
id, source_osm_node_id, target_osm_node_id,
source_lon, source_lat, target_lon, target_lat,
length_m, highway, name, geometry_geojson,
tags_json,
CASE
WHEN source_osm_node_id = :node_id THEN {cost_column}
ELSE {reverse_cost_column}
@@ -713,6 +856,14 @@ def _outgoing_edges(db: Session, dataset_id: int, node_id: int, mode: str) -> li
edges = []
for row in rows:
forward = bool(row.forward)
cost_s = float(row.cost_s)
if mode == "bike":
tags = _json_object(str(row.tags_json or ""))
if not _bike_access_python(row.highway, tags):
continue
if _bike_direction_forbidden_python(row.highway, tags, reverse=not forward):
continue
cost_s = _bike_edge_seconds(float(row.length_m), row.highway, tags)
if forward:
to_node = int(row.target_osm_node_id)
from_lon, from_lat = float(row.source_lon), float(row.source_lat)
@@ -730,7 +881,7 @@ def _outgoing_edges(db: Session, dataset_id: int, node_id: int, mode: str) -> li
from_lat=from_lat,
to_lon=to_lon,
to_lat=to_lat,
cost_s=float(row.cost_s),
cost_s=cost_s,
length_m=float(row.length_m),
highway=row.highway,
name=row.name,
@@ -869,10 +1020,82 @@ def _connector_feature(kind: str, mode: str, coordinates: list[list[float]], dis
def _connector_seconds(distance_m: float, mode: str) -> float:
speed = 1.35 if mode == "walk" else 8.0
speed = 1.35 if mode == "walk" else BIKE_SPEED_MPS if mode == "bike" else 8.0
return float(distance_m) / speed
def _bike_edge_seconds(length_m: float, highway: str | None, tags: dict[str, object]) -> float:
factor = {
"cycleway": 0.82,
"path": 0.95,
"living_street": 1.0,
"residential": 1.0,
"service": 1.08,
"track": 1.10,
"unclassified": 1.12,
"road": 1.12,
"tertiary": 1.12,
"tertiary_link": 1.12,
"secondary": 1.22,
"secondary_link": 1.22,
"primary": 1.35,
"primary_link": 1.35,
"footway": 1.40,
"pedestrian": 1.40,
"bridleway": 1.40,
}.get(str(highway or ""), 1.05)
return float(length_m) / BIKE_SPEED_MPS * factor
def _bike_access_python(highway: str | None, tags: dict[str, object]) -> bool:
highway_text = str(highway or "")
bicycle = _tag_text(tags, "bicycle")
access = _tag_text(tags, "access")
vehicle = _tag_text(tags, "vehicle")
if highway_text in {"motorway", "motorway_link", "steps", "platform"}:
return False
if bicycle in {"no", "private"}:
return False
if access in {"no", "private"} and bicycle not in {"yes", "designated", "permissive", "destination"} and vehicle not in {"yes", "designated", "permissive", "destination"}:
return False
if highway_text in {"footway", "pedestrian", "bridleway"} and bicycle not in {"yes", "designated", "permissive", "destination"}:
return False
if highway_text in {"trunk", "trunk_link"} and bicycle not in {"yes", "designated", "permissive", "destination"}:
return False
return True
def _bike_direction_forbidden_python(highway: str | None, tags: dict[str, object], *, reverse: bool) -> bool:
oneway = _tag_text(tags, "oneway")
junction = _tag_text(tags, "junction")
if reverse:
one_way = oneway in {"yes", "true", "1"} or junction == "roundabout" or str(highway or "") == "motorway"
else:
one_way = oneway == "-1"
if not one_way:
return False
bicycle_direction_key = "bicycle:backward" if reverse else "bicycle:forward"
return (
_tag_text(tags, "oneway:bicycle") != "no"
and _tag_text(tags, "bicycle:oneway") != "no"
and _tag_text(tags, bicycle_direction_key) not in {"yes", "designated", "permissive"}
)
def _tag_text(tags: dict[str, object], key: str) -> str:
return str(tags.get(key) or "").strip().lower()
def _json_object(value: str | None) -> dict[str, object]:
if not value:
return {}
try:
parsed = json.loads(value)
except json.JSONDecodeError:
return {}
return parsed if isinstance(parsed, dict) else {}
def _duration_minutes_ceil(seconds: int | float | None) -> int | None:
if seconds is None:
return None

988
app/static/app.js
View File

File diff suppressed because it is too large Load Diff

244
app/static/style.css
View File

@@ -383,10 +383,16 @@ input.journey-selected-stop {
gap: 8px;
}
.harmonization-review-form label:has(textarea),
.harmonization-review-form .harmonization-license-grid,
.harmonization-review-form .source-actions,
.harmonization-review-form > .muted {
grid-column: 1 / -1;
}
.harmonization-license-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(160px, 1fr));
gap: 8px;
}
.harmonization-review-item {
display: grid;
gap: 2px;
@@ -416,6 +422,155 @@ input.journey-selected-stop {
.harmonization-review-item span {
color: #52606d;
}
.map-gtfs-review-queue {
display: grid;
gap: 8px;
max-height: 360px;
overflow-x: hidden;
overflow-y: auto;
padding-right: 4px;
min-width: 0;
}
.map-gtfs-review-summary {
display: grid;
grid-template-columns: repeat(4, minmax(0, 1fr));
gap: 5px;
}
.map-gtfs-review-summary div {
border: 1px solid #e2e8f0;
border-radius: 6px;
background: #f8fafc;
padding: 6px;
min-width: 0;
}
.map-gtfs-review-summary strong,
.map-gtfs-review-summary span {
display: block;
min-width: 0;
overflow-wrap: anywhere;
}
.map-gtfs-review-summary strong {
font-size: 13px;
color: #17212b;
}
.map-gtfs-review-summary span {
font-size: 10px;
color: #5a6875;
}
.map-gtfs-review-item {
display: grid;
gap: 5px;
border-left: 4px solid #64748b;
border-radius: 6px;
background: #f8fafc;
padding: 8px;
font-size: 12px;
min-width: 0;
}
.map-gtfs-review-item.error {
border-left-color: #c24141;
background: #fff7f7;
}
.map-gtfs-review-item.warn {
border-left-color: #c47a12;
background: #fffaf0;
}
.map-gtfs-review-item.info {
border-left-color: #2563eb;
background: #f6f9ff;
}
.map-gtfs-review-title {
display: flex;
justify-content: space-between;
gap: 8px;
min-width: 0;
}
.map-gtfs-review-title > * {
min-width: 0;
overflow-wrap: anywhere;
}
.map-gtfs-review-actions {
display: flex;
flex-wrap: wrap;
gap: 6px;
}
.gtfs-workflow,
.gtfs-workflow-steps,
.gtfs-workflow-diffs,
.gtfs-diff-item-list {
display: grid;
gap: 8px;
min-width: 0;
}
.gtfs-workflow-step {
grid-template-columns: auto minmax(0, 1fr);
}
.gtfs-workflow-step.pending {
opacity: 0.82;
}
.gtfs-workflow-actions {
display: flex;
flex-wrap: wrap;
gap: 6px;
margin-top: 6px;
}
.gtfs-workflow-diffs {
border-top: 1px solid #e2e8f0;
padding-top: 8px;
}
.gtfs-workflow-audit {
display: grid;
gap: 7px;
border-top: 1px solid #e2e8f0;
padding-top: 8px;
min-width: 0;
}
.gtfs-workflow-audit h4 {
margin: 0;
font-size: 12px;
color: #334155;
}
.snapshot-route-preview {
display: grid;
gap: 6px;
max-height: 190px;
overflow-y: auto;
padding-right: 4px;
min-width: 0;
}
.snapshot-route-row {
display: grid;
gap: 2px;
border-left: 3px solid #2563eb;
background: #f8fafc;
padding: 6px 8px;
min-width: 0;
}
.snapshot-route-row strong,
.snapshot-route-row span {
min-width: 0;
overflow-wrap: anywhere;
}
.gtfs-workflow-diffs h4 {
margin: 0;
font-size: 12px;
color: #334155;
}
.gtfs-workflow-diff {
display: grid;
gap: 2px;
width: 100%;
border: 1px solid #e2e8f0;
border-radius: 6px;
background: #f8fafc;
color: #17212b;
padding: 8px;
text-align: left;
}
.gtfs-workflow-diff span {
color: #52606d;
font-size: 11px;
}
.source, .match, .catalog-entry {
border-top: 1px solid #e1e7ee;
padding: 8px 0;
@@ -814,6 +969,14 @@ input.journey-selected-stop {
gap: 8px;
font-size: 12px;
min-width: 0;
}
.jobs-worker-status {
min-width: 0;
}
.jobs-queue {
display: grid;
gap: 8px;
min-width: 0;
max-height: 280px;
overflow-x: hidden;
overflow-y: auto;
@@ -831,6 +994,12 @@ input.journey-selected-stop {
display: grid;
gap: 4px;
}
.worker-controls {
display: flex;
flex-wrap: wrap;
gap: 6px;
min-width: 0;
}
.worker-row {
border: 1px solid #e1e7ee;
border-radius: 6px;
@@ -1202,6 +1371,8 @@ input.journey-selected-stop {
.mode-coach { background: #fef3c7; color: #a16207; }
.mode-ferry { background: #dbeafe; color: #0284c7; }
.mode-walk { background: #dcfce7; color: #16a34a; }
.mode-bike,
.mode-bicycle { background: #ccfbf1; color: #0d9488; }
.mode-drive,
.mode-car { background: #ffedd5; color: #f97316; }
.mode-monorail,
@@ -1416,6 +1587,79 @@ input.journey-selected-stop {
white-space: pre-wrap;
overflow-wrap: anywhere;
}
.review-detail {
display: grid;
gap: 14px;
font-size: 12px;
}
.review-detail section {
display: grid;
gap: 8px;
}
.review-detail h3,
.review-entity-card h4 {
margin: 0;
font-size: 13px;
}
.review-detail-title {
display: flex;
flex-wrap: wrap;
gap: 6px;
}
.review-detail-grid,
.review-comparison {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
gap: 8px;
min-width: 0;
}
.review-entity-card {
display: grid;
gap: 5px;
border: 1px solid #e1e7ee;
border-radius: 6px;
background: #fbfdff;
padding: 8px;
min-width: 0;
}
.review-entity-card div {
display: grid;
grid-template-columns: minmax(80px, 0.42fr) minmax(0, 1fr);
gap: 8px;
min-width: 0;
}
.review-entity-card span {
color: #64748b;
}
.review-entity-card strong,
.review-diff-table span,
.review-diff-table strong {
min-width: 0;
overflow-wrap: anywhere;
}
.review-diff-table {
display: grid;
gap: 5px;
}
.review-diff-table > div {
display: grid;
grid-template-columns: minmax(90px, 0.7fr) repeat(2, minmax(0, 1fr));
gap: 8px;
border: 1px solid #e2e8f0;
border-radius: 6px;
background: #f8fafc;
padding: 7px;
}
.review-detail pre {
margin: 5px 0 0;
padding: 8px;
border-radius: 6px;
background: #f7fafc;
white-space: pre-wrap;
overflow-wrap: anywhere;
max-height: 320px;
overflow-y: auto;
}
.canonical-stop-detail {
display: grid;
gap: 14px;

81
app/templates/index.html
View File

@@ -5,7 +5,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Mobility Workbench</title>
<link rel="stylesheet" href="https://unpkg.com/leaflet@1.9.4/dist/leaflet.css" crossorigin="" />
<link rel="stylesheet" href="/static/style.css?v=20260701-harmonizer-module" />
<link rel="stylesheet" href="/static/style.css?v={{ static_version }}" />
</head>
<body>
<header>
@@ -49,7 +49,7 @@
<summary><h2>GTFS Harmonization</h2></summary>
<div class="sidebar-section-body">
<details class="nested-section" data-sidebar-section="add-gtfs-source">
<summary><h3>Add GTFS source</h3></summary>
<summary><h3>Add GTFS feed</h3></summary>
<div class="nested-section-body">
<form id="sourceForm">
<input name="catalog_entry_id" type="hidden" />
@@ -58,13 +58,13 @@
<label>URL or path <input name="url" required placeholder="https://.../feed.zip or ./data/feed.zip" /></label>
<label>Country <input name="country" placeholder="DE" maxlength="8" /></label>
<label>License <input name="license" placeholder="ODbL / CC-BY / unknown" /></label>
<button type="submit">Add GTFS source</button>
<button type="submit">Add GTFS feed</button>
</form>
</div>
</details>
<details class="nested-section source-catalog-card" data-sidebar-section="source-catalog">
<summary><h3>Transit source catalog</h3></summary>
<summary><h3>Discovery catalog</h3></summary>
<div class="nested-section-body">
<div id="sourceCatalogSummary" class="muted"></div>
<div class="filter-row source-catalog-filter">
@@ -83,14 +83,24 @@
</div>
<div class="source-catalog-actions">
<button type="button" id="importSourceCatalogBtn">Import catalog</button>
<button type="button" id="importIngestableSourcesBtn">Import ingestable seeds</button>
<button type="button" id="importIngestableSourcesBtn">Import feed seeds</button>
</div>
<div id="sourceCatalog"></div>
</div>
</details>
<details class="nested-section" data-sidebar-section="gtfs-workflow" open>
<summary><h3>GTFS workflow</h3></summary>
<div class="nested-section-body">
<div class="qa-toolbar">
<button type="button" id="refreshGtfsWorkflowBtn">Refresh workflow</button>
</div>
<div id="gtfsWorkflow" class="gtfs-workflow muted">Workflow status loading...</div>
</div>
</details>
<details class="nested-section" data-sidebar-section="gtfs-feed-qa" open>
<summary><h3>Feed QA</h3></summary>
<summary><h3>GTFS feed QA</h3></summary>
<div class="nested-section-body">
<div class="qa-toolbar">
<button type="button" id="refreshGtfsHarmonizationBtn">Refresh feeds</button>
@@ -99,11 +109,57 @@
</div>
</details>
<details class="nested-section" data-sidebar-section="map-gtfs-workbench" open>
<summary><h3>Map/GTFS workbench</h3></summary>
<div class="nested-section-body">
<p class="muted">Review and persist GTFS-to-map decisions here. Run the ordered workflow from the GTFS workflow panel above.</p>
<div class="qa-toolbar">
<button type="button" id="refreshMapGtfsWorkbenchBtn">Refresh queue</button>
</div>
<div class="filter-row">
<select id="mapGtfsReviewQueueFilter">
<option value="">all queues</option>
<option value="gtfs_deduplication">GTFS deduplication</option>
<option value="gtfs_update_diff">GTFS update diff</option>
<option value="route_matching">route matching</option>
<option value="stop_matching">stop matching</option>
</select>
<select id="mapGtfsReviewStatus">
<option value="open">open</option>
<option value="in_review">in review</option>
<option value="resolved">resolved</option>
<option value="dismissed">dismissed</option>
<option value="">all statuses</option>
</select>
</div>
<div class="filter-row">
<select id="mapGtfsReviewSeverityFilter">
<option value="">all severities</option>
<option value="error">error</option>
<option value="warn">warn</option>
<option value="info">info</option>
</select>
<select id="mapGtfsReviewModeFilter">
<option value="">all modes</option>
<option value="train">train</option>
<option value="tram">tram</option>
<option value="subway">subway</option>
<option value="bus">bus</option>
<option value="ferry">ferry</option>
</select>
</div>
<div class="filter-row">
<input id="mapGtfsReviewSearch" placeholder="Search review queue" />
</div>
<div id="mapGtfsReviewQueue" class="map-gtfs-review-queue muted">No workbench review queue loaded.</div>
</div>
</details>
<details class="nested-section" data-sidebar-section="gtfs-source-management" open>
<summary><h3>GTFS source library</h3></summary>
<summary><h3>Registered GTFS feeds</h3></summary>
<div class="nested-section-body">
<div class="filter-row">
<input id="sourceSearch" placeholder="Filter GTFS sources" />
<input id="sourceSearch" placeholder="Filter registered feeds" />
</div>
<div id="sources"></div>
</div>
@@ -172,10 +228,9 @@
<summary><h3>Derivation pipeline</h3></summary>
<div class="nested-section-body">
<div class="workflow-actions">
<button id="runMatchBtn" type="button">Run matcher</button>
<button id="buildRouteLayerBtn" type="button">Build route layer</button>
<button id="loadSampleBtn" type="button">Reset sample</button>
</div>
<p class="muted">Matcher and route-layer controls live in the Map/GTFS workbench so review actions stay in one workflow.</p>
</div>
</details>
@@ -194,8 +249,9 @@
</details>
<details class="nested-section matches-card" data-sidebar-section="route-matches">
<summary><h3>Route matches</h3></summary>
<summary><h3>Raw route matches</h3></summary>
<div class="nested-section-body">
<p class="muted">Inspection/debug view of raw matcher rows. Use the Map/GTFS workbench for review decisions.</p>
<div class="filter-row">
<select id="matchStatusFilter">
<option value="">all</option>
@@ -272,6 +328,7 @@
<div class="journey-mode" role="radiogroup" aria-label="Route mode">
<label><input type="radio" name="journeyMode" value="transit" checked /> Public transport</label>
<label><input type="radio" name="journeyMode" value="walk" /> Walk</label>
<label><input type="radio" name="journeyMode" value="bike" /> Bike</label>
<label><input type="radio" name="journeyMode" value="drive" /> Car</label>
</div>
<div class="journey-options">
@@ -324,6 +381,6 @@
</div>
<script src="https://unpkg.com/leaflet@1.9.4/dist/leaflet.js" crossorigin=""></script>
<script src="/static/app.js?v=20260701-harmonizer-module"></script>
<script src="/static/app.js?v={{ static_version }}"></script>
</body>
</html>

985
app/workbench.py Normal file
View File

@@ -0,0 +1,985 @@
from __future__ import annotations
import hashlib
import json
from datetime import datetime, timezone
from typing import Any, Callable
from sqlalchemy import and_, case, exists, func, or_, select
from sqlalchemy.orm import Session, aliased, joinedload
from app.harmonization import active_harmonized_gtfs_route_shadow_map, gtfs_harmonized_snapshot, gtfs_route_overlap_key
from app.models import (
CanonicalStopLink,
Dataset,
GtfsRoute,
GtfsStop,
GtfsTrip,
MapGtfsDecision,
MapGtfsReviewItem,
OsmFeature,
RouteMatch,
Source,
)
ProgressCallback = Callable[[str, str, int | None, int | None, dict[str, Any] | None], None]
OPEN_REVIEW_STATUSES = {"open", "in_review"}
TERMINAL_REVIEW_STATUSES = {"resolved", "dismissed"}
ROUTE_REVIEW_STATUSES = {"matched", "probable", "weak", "missing"}
def generate_map_gtfs_review_items(
session: Session,
*,
source_id: int | None = None,
dataset_id: int | None = None,
limit: int = 1000,
progress_callback: ProgressCallback | None = None,
) -> dict[str, Any]:
selected_limit = max(1, min(int(limit), 50_000))
result = {
"created": 0,
"updated": 0,
"reopened": 0,
"unchanged": 0,
"resolved": 0,
"route_items": 0,
"dedup_items": 0,
"stop_items": 0,
"limit": selected_limit,
}
route_limit = max(1, selected_limit // 2)
dedup_limit = max(1, selected_limit // 4)
_emit(progress_callback, "map_gtfs_review_routes_started", "Scanning GTFS/OSM route matches.", 0, route_limit, result)
route_result = _generate_route_review_items(session, source_id=source_id, dataset_id=dataset_id, limit=route_limit)
_merge_counts(result, route_result)
result["route_items"] = route_result["items"]
remaining = max(0, selected_limit - int(route_result["items"]))
_emit(
progress_callback,
"map_gtfs_review_routes_completed",
"Scanned GTFS/OSM route matches.",
int(route_result["items"]),
route_limit,
route_result,
)
if remaining and dedup_limit:
current_dedup_limit = min(dedup_limit, remaining)
_emit(progress_callback, "map_gtfs_review_dedup_started", "Scanning shadowed GTFS routes for duplicate/conflict review.", 0, current_dedup_limit, result)
dedup_result = _generate_gtfs_route_dedup_review_items(session, source_id=source_id, dataset_id=dataset_id, limit=current_dedup_limit)
_merge_counts(result, dedup_result)
result["dedup_items"] = dedup_result["items"]
remaining = max(0, remaining - int(dedup_result["items"]))
_emit(
progress_callback,
"map_gtfs_review_dedup_completed",
"Scanned GTFS route duplicates.",
int(dedup_result["items"]),
current_dedup_limit,
dedup_result,
)
if remaining:
_emit(progress_callback, "map_gtfs_review_stops_started", "Scanning GTFS stops needing concrete map links.", 0, remaining, result)
stop_result = _generate_stop_review_items(session, source_id=source_id, dataset_id=dataset_id, limit=remaining)
_merge_counts(result, stop_result)
result["stop_items"] = stop_result["items"]
_emit(
progress_callback,
"map_gtfs_review_stops_completed",
"Scanned GTFS stops needing concrete map links.",
int(stop_result["items"]),
remaining,
stop_result,
)
session.flush()
_emit(progress_callback, "map_gtfs_review_completed", "Map/GTFS review queue generated.", None, None, result)
return result
def list_map_gtfs_review_items(
session: Session,
*,
queue: str | None = None,
status: str | None = "open",
severity: str | None = None,
mode: str | None = None,
q: str | None = None,
source_id: int | None = None,
dataset_id: int | None = None,
limit: int = 100,
) -> list[MapGtfsReviewItem]:
stmt = (
select(MapGtfsReviewItem)
.outerjoin(GtfsRoute, GtfsRoute.id == MapGtfsReviewItem.gtfs_route_id)
.outerjoin(GtfsStop, GtfsStop.id == MapGtfsReviewItem.gtfs_stop_id)
.outerjoin(OsmFeature, OsmFeature.id == MapGtfsReviewItem.osm_feature_id)
.outerjoin(Source, Source.id == MapGtfsReviewItem.source_id)
.options(
joinedload(MapGtfsReviewItem.source),
joinedload(MapGtfsReviewItem.dataset),
joinedload(MapGtfsReviewItem.gtfs_route),
joinedload(MapGtfsReviewItem.gtfs_stop),
joinedload(MapGtfsReviewItem.osm_feature),
joinedload(MapGtfsReviewItem.route_match),
joinedload(MapGtfsReviewItem.decision),
)
.order_by(
_severity_order_expression(),
MapGtfsReviewItem.updated_at.desc(),
MapGtfsReviewItem.id.desc(),
)
)
if queue:
stmt = stmt.where(MapGtfsReviewItem.queue == queue)
if status:
stmt = stmt.where(MapGtfsReviewItem.status == status)
if severity:
stmt = stmt.where(MapGtfsReviewItem.severity == severity)
mode_text = " ".join(str(mode or "").strip().split())
if mode_text:
stmt = stmt.where(or_(GtfsRoute.mode == mode_text, OsmFeature.mode == mode_text))
query_text = " ".join(str(q or "").strip().split())
if query_text:
pattern = f"%{query_text}%"
stmt = stmt.where(
or_(
MapGtfsReviewItem.fingerprint.ilike(pattern),
MapGtfsReviewItem.title.ilike(pattern),
MapGtfsReviewItem.description.ilike(pattern),
GtfsRoute.route_id.ilike(pattern),
GtfsRoute.short_name.ilike(pattern),
GtfsRoute.long_name.ilike(pattern),
GtfsRoute.operator_name.ilike(pattern),
GtfsStop.stop_id.ilike(pattern),
GtfsStop.name.ilike(pattern),
OsmFeature.ref.ilike(pattern),
OsmFeature.name.ilike(pattern),
Source.name.ilike(pattern),
)
)
if source_id is not None:
stmt = stmt.where(MapGtfsReviewItem.source_id == source_id)
if dataset_id is not None:
stmt = stmt.where(MapGtfsReviewItem.dataset_id == dataset_id)
return session.scalars(stmt.limit(max(1, min(int(limit), 1000)))).all()
def map_gtfs_review_summary(session: Session) -> dict[str, Any]:
rows = session.execute(
select(MapGtfsReviewItem.queue, MapGtfsReviewItem.status, MapGtfsReviewItem.severity, func.count())
.select_from(MapGtfsReviewItem)
.group_by(MapGtfsReviewItem.queue, MapGtfsReviewItem.status, MapGtfsReviewItem.severity)
).all()
by_queue: dict[str, dict[str, int]] = {}
by_status: dict[str, int] = {}
by_severity: dict[str, int] = {}
total = 0
for queue, status, severity, count in rows:
count = int(count or 0)
total += count
queue_counts = by_queue.setdefault(str(queue), {})
queue_counts[str(status)] = queue_counts.get(str(status), 0) + count
by_status[str(status)] = by_status.get(str(status), 0) + count
by_severity[str(severity)] = by_severity.get(str(severity), 0) + count
decisions = session.scalar(select(func.count()).select_from(MapGtfsDecision)) or 0
active_decisions = session.scalar(select(func.count()).select_from(MapGtfsDecision).where(MapGtfsDecision.active.is_(True))) or 0
return {
"total": total,
"by_queue": by_queue,
"by_status": by_status,
"by_severity": by_severity,
"decisions": int(decisions),
"active_decisions": int(active_decisions),
}
def update_map_gtfs_review_item(
session: Session,
item_id: int,
*,
status: str | None = None,
note: str | None = None,
) -> MapGtfsReviewItem:
item = session.get(MapGtfsReviewItem, item_id)
if item is None:
raise ValueError("review item not found")
now = datetime.now(timezone.utc)
if status is not None:
if status not in {"open", "in_review", "resolved", "dismissed"}:
raise ValueError("review status must be open, in_review, resolved, or dismissed")
item.status = status
item.resolved_at = now if status in TERMINAL_REVIEW_STATUSES else None
if note is not None:
metadata = _json_object(item.metadata_json)
metadata["review_note"] = note
item.metadata_json = _json_dumps(metadata)
item.updated_at = now
session.flush()
return item
def record_route_match_decision(
session: Session,
match: RouteMatch,
*,
decision_type: str = "route_osm_relation",
reviewer: str | None = "manual",
note: str | None = None,
) -> MapGtfsDecision | None:
route = session.get(GtfsRoute, match.gtfs_route_id)
if route is None:
return None
route_dataset = session.get(Dataset, route.dataset_id)
feature = session.get(OsmFeature, match.osm_feature_id) if match.osm_feature_id else None
feature_dataset = session.get(Dataset, feature.dataset_id) if feature is not None else None
selector = {
"gtfs": {
"source_id": None if route_dataset is None else route_dataset.source_id,
"dataset_id": route.dataset_id,
"route_id": route.route_id,
"route_key": route.route_key,
"ref": route.short_name,
"mode": route.mode,
},
"route_match_id": match.id,
}
action: dict[str, Any] = {"status": match.status}
if feature is not None:
action["osm"] = {
"source_id": None if feature_dataset is None else feature_dataset.source_id,
"dataset_id": feature.dataset_id,
"osm_type": feature.osm_type,
"osm_id": feature.osm_id,
"route_key": feature.route_key,
"ref": feature.ref,
"mode": feature.mode,
}
evidence = {
"selector": selector,
"action": action,
"confidence": match.confidence,
"reasons": _json_object(match.reasons_json),
}
decision_key = _stable_key(decision_type, selector, action)
evidence_hash = _hash_payload(evidence)
decision = session.scalar(select(MapGtfsDecision).where(MapGtfsDecision.decision_key == decision_key))
now = datetime.now(timezone.utc)
if decision is None:
decision = MapGtfsDecision(
decision_key=decision_key,
decision_type=decision_type,
status=match.status,
active=True,
source_id=None if route_dataset is None else route_dataset.source_id,
gtfs_dataset_id=route.dataset_id,
gtfs_route_id=route.id,
osm_dataset_id=None if feature is None else feature.dataset_id,
osm_feature_id=None if feature is None else feature.id,
route_match_id=match.id,
confidence=match.confidence,
evidence_hash=evidence_hash,
selector_json=_json_dumps(selector),
action_json=_json_dumps(action),
note=note,
reviewer=reviewer,
metadata_json=_json_dumps({"reasons": _json_object(match.reasons_json)}),
created_at=now,
updated_at=now,
)
session.add(decision)
else:
decision.status = match.status
decision.active = True
decision.osm_dataset_id = None if feature is None else feature.dataset_id
decision.osm_feature_id = None if feature is None else feature.id
decision.route_match_id = match.id
decision.confidence = match.confidence
decision.evidence_hash = evidence_hash
decision.selector_json = _json_dumps(selector)
decision.action_json = _json_dumps(action)
decision.note = note if note is not None else decision.note
decision.reviewer = reviewer if reviewer is not None else decision.reviewer
decision.metadata_json = _json_dumps({"reasons": _json_object(match.reasons_json)})
decision.updated_at = now
_resolve_review_item_by_fingerprint(session, _route_fingerprint(route), decision=decision)
session.flush()
return decision
def review_item_payload(item: MapGtfsReviewItem) -> dict[str, Any]:
route = item.gtfs_route
stop = item.gtfs_stop
osm = item.osm_feature
return {
"id": item.id,
"fingerprint": item.fingerprint,
"queue": item.queue,
"item_type": item.item_type,
"status": item.status,
"severity": item.severity,
"title": item.title,
"description": item.description,
"source": None if item.source is None else {"id": item.source.id, "name": item.source.name, "kind": item.source.kind},
"dataset": None if item.dataset is None else {"id": item.dataset.id, "kind": item.dataset.kind, "sha256": item.dataset.sha256},
"gtfs_route": None
if route is None
else {
"id": route.id,
"dataset_id": route.dataset_id,
"route_id": route.route_id,
"ref": route.short_name,
"name": route.long_name,
"mode": route.mode,
"operator": route.operator_name,
},
"gtfs_stop": None
if stop is None
else {
"id": stop.id,
"dataset_id": stop.dataset_id,
"stop_id": stop.stop_id,
"name": stop.name,
"lat": stop.lat,
"lon": stop.lon,
},
"osm_feature": None
if osm is None
else {
"id": osm.id,
"dataset_id": osm.dataset_id,
"osm_type": osm.osm_type,
"osm_id": osm.osm_id,
"kind": osm.kind,
"mode": osm.mode,
"ref": osm.ref,
"name": osm.name,
},
"route_match_id": item.route_match_id,
"canonical_stop_id": item.canonical_stop_id,
"canonical_stop_link_id": item.canonical_stop_link_id,
"decision_id": item.decision_id,
"evidence_hash": item.evidence_hash,
"metadata": _json_object(item.metadata_json),
"created_at": item.created_at.isoformat(),
"updated_at": item.updated_at.isoformat(),
"resolved_at": None if item.resolved_at is None else item.resolved_at.isoformat(),
}
def decision_payload(decision: MapGtfsDecision) -> dict[str, Any]:
return {
"id": decision.id,
"decision_key": decision.decision_key,
"decision_type": decision.decision_type,
"status": decision.status,
"active": decision.active,
"source_id": decision.source_id,
"gtfs_dataset_id": decision.gtfs_dataset_id,
"gtfs_route_id": decision.gtfs_route_id,
"gtfs_stop_id": decision.gtfs_stop_id,
"osm_dataset_id": decision.osm_dataset_id,
"osm_feature_id": decision.osm_feature_id,
"route_match_id": decision.route_match_id,
"canonical_stop_id": decision.canonical_stop_id,
"canonical_stop_link_id": decision.canonical_stop_link_id,
"confidence": decision.confidence,
"evidence_hash": decision.evidence_hash,
"selector": _json_object(decision.selector_json),
"action": _json_object(decision.action_json),
"note": decision.note,
"reviewer": decision.reviewer,
"metadata": _json_object(decision.metadata_json),
"created_at": decision.created_at.isoformat(),
"updated_at": decision.updated_at.isoformat(),
}
def _generate_route_review_items(
session: Session,
*,
source_id: int | None,
dataset_id: int | None,
limit: int,
) -> dict[str, int]:
stmt = (
select(RouteMatch)
.options(joinedload(RouteMatch.gtfs_route), joinedload(RouteMatch.osm_feature))
.join(RouteMatch.gtfs_route)
.join(Dataset, Dataset.id == GtfsRoute.dataset_id)
.where(Dataset.is_active.is_(True), Dataset.kind == "gtfs", RouteMatch.status.in_(ROUTE_REVIEW_STATUSES))
.order_by(RouteMatch.status.desc(), RouteMatch.confidence.asc(), RouteMatch.id)
)
if source_id is not None:
stmt = stmt.where(Dataset.source_id == source_id)
if dataset_id is not None:
stmt = stmt.where(GtfsRoute.dataset_id == dataset_id)
matches = session.scalars(stmt.limit(limit)).all()
counts = _empty_counts()
counts["items"] = len(matches)
for match in matches:
route = match.gtfs_route
if route is None:
continue
dataset = session.get(Dataset, route.dataset_id)
metadata = {
"route_match_status": match.status,
"confidence": match.confidence,
"reasons": _json_object(match.reasons_json),
"gtfs": {
"route_id": route.route_id,
"route_key": route.route_key,
"ref": route.short_name,
"mode": route.mode,
},
"osm_feature_id": match.osm_feature_id,
}
status_label = match.status.replace("_", " ")
item = _upsert_review_item(
session,
fingerprint=_route_fingerprint(route),
queue="route_matching",
item_type="gtfs_route_osm_route",
severity=_route_review_severity(match.status),
title=f"Review route {route.short_name or route.route_id}: {status_label}",
description=_route_review_description(route, match),
source_id=None if dataset is None else dataset.source_id,
dataset_id=route.dataset_id,
gtfs_route_id=route.id,
osm_feature_id=match.osm_feature_id,
route_match_id=match.id,
evidence_payload=metadata,
)
counts[item] += 1
return counts
def _generate_gtfs_route_dedup_review_items(
session: Session,
*,
source_id: int | None,
dataset_id: int | None,
limit: int,
) -> dict[str, int]:
snapshot = gtfs_harmonized_snapshot(session)
snapshot_items_by_dataset = {
int(item["dataset_id"]): item
for item in snapshot.get("datasets", [])
if item.get("dataset_id") is not None
}
included_dataset_ids = [int(item["dataset_id"]) for item in snapshot.get("included", []) if item.get("dataset_active", True)]
shadowed_items = {
int(item["dataset_id"]): item
for item in snapshot.get("shadowed", [])
if item.get("dataset_id") is not None
}
if source_id is not None:
shadowed_items = {dataset_id: item for dataset_id, item in shadowed_items.items() if int(item.get("source_id") or 0) == int(source_id)}
if dataset_id is not None:
shadowed_items = {int(dataset_id): item} if int(dataset_id) in shadowed_items else {}
route_shadow_map = active_harmonized_gtfs_route_shadow_map(session)
if not included_dataset_ids and not route_shadow_map:
counts = _empty_counts()
counts["items"] = 0
return counts
pairs: list[tuple[GtfsRoute, GtfsRoute, dict[str, Any]]] = []
if included_dataset_ids and shadowed_items:
included_by_key = _routes_by_overlap_key(session, included_dataset_ids)
shadowed_routes = session.scalars(
select(GtfsRoute)
.where(GtfsRoute.dataset_id.in_(list(shadowed_items)))
.order_by(GtfsRoute.dataset_id, GtfsRoute.mode, GtfsRoute.short_name, GtfsRoute.route_id)
.limit(max(1, int(limit)) * 3)
).all()
for route in shadowed_routes:
key = gtfs_route_overlap_key(route)
if not key:
continue
candidates = included_by_key.get(key, [])
if not candidates:
continue
shadowed_item = shadowed_items[int(route.dataset_id)]
preferred_dataset_id = _optional_int(shadowed_item.get("shadowed_by_dataset_id"))
candidate = _best_included_duplicate_candidate(candidates, preferred_dataset_id=preferred_dataset_id)
pairs.append((route, candidate, shadowed_item))
if len(pairs) >= max(1, int(limit)):
break
if len(pairs) < max(1, int(limit)) and route_shadow_map:
pairs.extend(
_partial_route_shadow_pairs(
session,
route_shadow_map=route_shadow_map,
snapshot_items_by_dataset=snapshot_items_by_dataset,
source_id=source_id,
dataset_id=dataset_id,
limit=max(0, int(limit) - len(pairs)),
)
)
trip_counts = _trip_counts_for_routes(session, [route for pair in pairs for route in pair[:2]])
counts = _empty_counts()
counts["items"] = len(pairs)
for shadowed_route, included_route, shadowed_item in pairs:
dataset = session.get(Dataset, shadowed_route.dataset_id)
evidence = _route_dedup_evidence(
shadowed_route,
included_route,
shadowed_item=shadowed_item,
trip_counts=trip_counts,
)
severity = "warn" if evidence["data_differs"] else "info"
item = _upsert_review_item(
session,
fingerprint=_route_dedup_fingerprint(shadowed_route, included_route),
queue="gtfs_deduplication",
item_type="gtfs_route_duplicate",
severity=severity,
title=_route_dedup_title(shadowed_route, included_route, shadowed_item),
description=_route_dedup_description(evidence),
source_id=None if dataset is None else dataset.source_id,
dataset_id=shadowed_route.dataset_id,
gtfs_route_id=shadowed_route.id,
evidence_payload=evidence,
)
counts[item] += 1
return counts
def _partial_route_shadow_pairs(
session: Session,
*,
route_shadow_map: dict[int, int],
snapshot_items_by_dataset: dict[int, dict[str, Any]],
source_id: int | None,
dataset_id: int | None,
limit: int,
) -> list[tuple[GtfsRoute, GtfsRoute, dict[str, Any]]]:
if limit <= 0 or not route_shadow_map:
return []
route_ids = sorted({*route_shadow_map.keys(), *route_shadow_map.values()})
routes = {int(route.id): route for route in session.scalars(select(GtfsRoute).where(GtfsRoute.id.in_(route_ids))).all()}
dataset_ids = sorted({int(route.dataset_id) for route in routes.values()})
datasets = {int(dataset.id): dataset for dataset in session.scalars(select(Dataset).where(Dataset.id.in_(dataset_ids))).all()}
pairs: list[tuple[GtfsRoute, GtfsRoute, dict[str, Any]]] = []
for shadowed_route_id, included_route_id in sorted(route_shadow_map.items()):
shadowed_route = routes.get(int(shadowed_route_id))
included_route = routes.get(int(included_route_id))
if shadowed_route is None or included_route is None:
continue
shadowed_dataset = datasets.get(int(shadowed_route.dataset_id))
included_dataset = datasets.get(int(included_route.dataset_id))
if source_id is not None and (shadowed_dataset is None or int(shadowed_dataset.source_id) != int(source_id)):
continue
if dataset_id is not None and int(shadowed_route.dataset_id) != int(dataset_id):
continue
shadowed_item = {
**snapshot_items_by_dataset.get(int(shadowed_route.dataset_id), {}),
"reason": "route_covered_by_higher_precedence_dataset",
"shadowed_by_dataset_id": included_route.dataset_id,
"shadowed_by_source_id": None if included_dataset is None else included_dataset.source_id,
"route_key_overlap": 1,
"route_key_overlap_ratio": 1.0,
}
pairs.append((shadowed_route, included_route, shadowed_item))
if len(pairs) >= limit:
break
return pairs
def _generate_stop_review_items(
session: Session,
*,
source_id: int | None,
dataset_id: int | None,
limit: int,
) -> dict[str, int]:
gtfs_link = aliased(CanonicalStopLink)
visual_link = aliased(CanonicalStopLink)
visual_link_exists = (
select(visual_link.id)
.where(
visual_link.canonical_stop_id == gtfs_link.canonical_stop_id,
visual_link.object_type == "osm_feature",
)
.exists()
)
stmt = (
select(GtfsStop, gtfs_link.id, gtfs_link.canonical_stop_id)
.join(Dataset, Dataset.id == GtfsStop.dataset_id)
.outerjoin(
gtfs_link,
and_(
gtfs_link.object_type == "gtfs_stop",
gtfs_link.dataset_id == GtfsStop.dataset_id,
gtfs_link.object_id == GtfsStop.id,
),
)
.where(
Dataset.is_active.is_(True),
Dataset.kind == "gtfs",
or_(gtfs_link.id.is_(None), ~visual_link_exists),
)
.order_by(GtfsStop.dataset_id, GtfsStop.name, GtfsStop.stop_id)
)
if source_id is not None:
stmt = stmt.where(Dataset.source_id == source_id)
if dataset_id is not None:
stmt = stmt.where(GtfsStop.dataset_id == dataset_id)
rows = session.execute(stmt.limit(limit)).all()
counts = _empty_counts()
counts["items"] = len(rows)
for stop, link_id, canonical_stop_id in rows:
dataset = session.get(Dataset, stop.dataset_id)
missing_canonical = link_id is None
metadata = {
"missing_canonical_stop": missing_canonical,
"missing_visual_stop": not missing_canonical,
"gtfs": {"stop_id": stop.stop_id, "name": stop.name, "lat": stop.lat, "lon": stop.lon},
"canonical_stop_id": canonical_stop_id,
}
item = _upsert_review_item(
session,
fingerprint=_stop_fingerprint(stop),
queue="stop_matching",
item_type="gtfs_stop_osm_stop",
severity="warn" if missing_canonical else "info",
title=f"Review stop {stop.name or stop.stop_id}",
description="GTFS stop has no canonical stop link." if missing_canonical else "GTFS stop has no concrete OSM stop/platform link.",
source_id=None if dataset is None else dataset.source_id,
dataset_id=stop.dataset_id,
gtfs_stop_id=stop.id,
canonical_stop_id=canonical_stop_id,
canonical_stop_link_id=link_id,
evidence_payload=metadata,
)
counts[item] += 1
return counts
def _routes_by_overlap_key(session: Session, dataset_ids: list[int]) -> dict[str, list[GtfsRoute]]:
if not dataset_ids:
return {}
rows = session.scalars(
select(GtfsRoute)
.where(GtfsRoute.dataset_id.in_(dataset_ids))
.order_by(GtfsRoute.dataset_id, GtfsRoute.mode, GtfsRoute.short_name, GtfsRoute.route_id)
).all()
routes: dict[str, list[GtfsRoute]] = {}
for route in rows:
key = gtfs_route_overlap_key(route)
if key:
routes.setdefault(key, []).append(route)
return routes
def _best_included_duplicate_candidate(routes: list[GtfsRoute], *, preferred_dataset_id: int | None) -> GtfsRoute:
return sorted(
routes,
key=lambda route: (
0 if preferred_dataset_id is not None and int(route.dataset_id) == preferred_dataset_id else 1,
route.dataset_id,
route.route_id,
),
)[0]
def _trip_counts_for_routes(session: Session, routes: list[GtfsRoute]) -> dict[tuple[int, str], int]:
keys = {(int(route.dataset_id), str(route.route_id)) for route in routes}
if not keys:
return {}
dataset_ids = sorted({dataset_id for dataset_id, _route_id in keys})
route_ids = sorted({route_id for _dataset_id, route_id in keys})
rows = session.execute(
select(GtfsTrip.dataset_id, GtfsTrip.route_id, func.count())
.where(GtfsTrip.dataset_id.in_(dataset_ids), GtfsTrip.route_id.in_(route_ids))
.group_by(GtfsTrip.dataset_id, GtfsTrip.route_id)
).all()
return {
(int(dataset_id), str(route_id)): int(count or 0)
for dataset_id, route_id, count in rows
if (int(dataset_id), str(route_id)) in keys
}
def _route_dedup_evidence(
shadowed_route: GtfsRoute,
included_route: GtfsRoute,
*,
shadowed_item: dict[str, Any],
trip_counts: dict[tuple[int, str], int],
) -> dict[str, Any]:
shadowed_trip_count = trip_counts.get((int(shadowed_route.dataset_id), str(shadowed_route.route_id)), 0)
included_trip_count = trip_counts.get((int(included_route.dataset_id), str(included_route.route_id)), 0)
differences = []
for field, left, right in [
("name", shadowed_route.long_name, included_route.long_name),
("operator", shadowed_route.operator_name, included_route.operator_name),
("route_scope", shadowed_route.route_scope, included_route.route_scope),
]:
if _dedup_text(left) != _dedup_text(right):
differences.append(field)
if shadowed_trip_count != included_trip_count:
differences.append("trip_count")
if _geometry_digest(shadowed_route.geometry_geojson) != _geometry_digest(included_route.geometry_geojson):
differences.append("geometry")
return {
"data_differs": bool(differences),
"differences": differences,
"overlap_key": gtfs_route_overlap_key(shadowed_route),
"shadowed_dataset": {
"dataset_id": shadowed_route.dataset_id,
"source_id": shadowed_item.get("source_id"),
"source_name": shadowed_item.get("source_name"),
"route_id": shadowed_route.route_id,
"ref": shadowed_route.short_name,
"name": shadowed_route.long_name,
"operator": shadowed_route.operator_name,
"mode": shadowed_route.mode,
"trip_count": shadowed_trip_count,
},
"included_dataset": {
"dataset_id": included_route.dataset_id,
"source_id": shadowed_item.get("shadowed_by_source_id"),
"route_id": included_route.route_id,
"ref": included_route.short_name,
"name": included_route.long_name,
"operator": included_route.operator_name,
"mode": included_route.mode,
"trip_count": included_trip_count,
},
"snapshot": {
"reason": shadowed_item.get("reason"),
"route_key_overlap": shadowed_item.get("route_key_overlap"),
"route_key_overlap_ratio": shadowed_item.get("route_key_overlap_ratio"),
},
}
def _route_dedup_title(shadowed_route: GtfsRoute, included_route: GtfsRoute, shadowed_item: dict[str, Any]) -> str:
ref = shadowed_route.short_name or shadowed_route.route_id
source = shadowed_item.get("source_name") or f"dataset #{shadowed_route.dataset_id}"
return f"Review duplicate GTFS route {ref}: {source} -> dataset #{included_route.dataset_id}"
def _route_dedup_description(evidence: dict[str, Any]) -> str:
differences = evidence.get("differences") or []
if differences:
return f"Route is shadowed by the active snapshot but differs in {', '.join(differences)}."
return "Route is shadowed by the active snapshot and looks equivalent on first-pass route evidence."
def _route_dedup_fingerprint(shadowed_route: GtfsRoute, included_route: GtfsRoute) -> str:
return f"gtfs-dedup:{shadowed_route.dataset_id}:{shadowed_route.route_id}:{included_route.dataset_id}:{included_route.route_id}"
def _geometry_digest(value: str | None) -> str:
if not value:
return ""
return hashlib.sha256(str(value).encode("utf-8")).hexdigest()
def _dedup_text(value: str | None) -> str:
return " ".join(str(value or "").casefold().replace("ß", "ss").split())
def _upsert_review_item(
session: Session,
*,
fingerprint: str,
queue: str,
item_type: str,
severity: str,
title: str,
description: str,
source_id: int | None = None,
dataset_id: int | None = None,
gtfs_route_id: int | None = None,
gtfs_stop_id: int | None = None,
osm_feature_id: int | None = None,
route_match_id: int | None = None,
canonical_stop_id: int | None = None,
canonical_stop_link_id: int | None = None,
decision_id: int | None = None,
evidence_payload: dict[str, Any] | None = None,
) -> str:
evidence_hash = _hash_payload(evidence_payload or {})
metadata_json = _json_dumps(evidence_payload or {})
item = session.scalar(select(MapGtfsReviewItem).where(MapGtfsReviewItem.fingerprint == fingerprint))
now = datetime.now(timezone.utc)
if item is None:
session.add(
MapGtfsReviewItem(
fingerprint=fingerprint,
queue=queue,
item_type=item_type,
status="open",
severity=severity,
title=title,
description=description,
source_id=source_id,
dataset_id=dataset_id,
gtfs_route_id=gtfs_route_id,
gtfs_stop_id=gtfs_stop_id,
osm_feature_id=osm_feature_id,
route_match_id=route_match_id,
canonical_stop_id=canonical_stop_id,
canonical_stop_link_id=canonical_stop_link_id,
decision_id=decision_id,
evidence_hash=evidence_hash,
metadata_json=metadata_json,
created_at=now,
updated_at=now,
)
)
return "created"
if item.evidence_hash == evidence_hash and item.status in TERMINAL_REVIEW_STATUSES:
return "unchanged"
changed = (
item.queue != queue
or item.item_type != item_type
or item.severity != severity
or item.title != title
or item.description != description
or item.source_id != source_id
or item.dataset_id != dataset_id
or item.gtfs_route_id != gtfs_route_id
or item.gtfs_stop_id != gtfs_stop_id
or item.osm_feature_id != osm_feature_id
or item.route_match_id != route_match_id
or item.canonical_stop_id != canonical_stop_id
or item.canonical_stop_link_id != canonical_stop_link_id
or item.decision_id != decision_id
or item.evidence_hash != evidence_hash
or item.metadata_json != metadata_json
)
was_terminal = item.status in TERMINAL_REVIEW_STATUSES
item.queue = queue
item.item_type = item_type
item.severity = severity
item.title = title
item.description = description
item.source_id = source_id
item.dataset_id = dataset_id
item.gtfs_route_id = gtfs_route_id
item.gtfs_stop_id = gtfs_stop_id
item.osm_feature_id = osm_feature_id
item.route_match_id = route_match_id
item.canonical_stop_id = canonical_stop_id
item.canonical_stop_link_id = canonical_stop_link_id
item.decision_id = decision_id
item.evidence_hash = evidence_hash
item.metadata_json = metadata_json
if was_terminal:
item.status = "open"
item.resolved_at = None
item.updated_at = now
return "reopened"
if changed:
item.updated_at = now
return "updated"
return "unchanged"
def _resolve_review_item_by_fingerprint(
session: Session,
fingerprint: str,
*,
decision: MapGtfsDecision | None = None,
) -> None:
item = session.scalar(select(MapGtfsReviewItem).where(MapGtfsReviewItem.fingerprint == fingerprint))
if item is None:
return
now = datetime.now(timezone.utc)
item.status = "resolved"
item.resolved_at = now
item.updated_at = now
if decision is not None:
item.decision = decision
def _route_review_severity(status: str) -> str:
if status == "missing":
return "error"
if status in {"weak", "probable"}:
return "warn"
return "info"
def _route_review_description(route: GtfsRoute, match: RouteMatch) -> str:
if match.status == "missing":
return "No plausible OSM route relation is linked yet."
if match.status == "weak":
return "The best OSM route candidate has weak evidence."
if match.status == "probable":
return "The OSM route candidate is plausible but should be reviewed."
return "Automatic route match is available for review."
def _route_fingerprint(route: GtfsRoute) -> str:
return f"route:{route.dataset_id}:{route.route_id}"
def _stop_fingerprint(stop: GtfsStop) -> str:
return f"stop:{stop.dataset_id}:{stop.stop_id}"
def _stable_key(decision_type: str, selector: dict[str, Any], action: dict[str, Any]) -> str:
payload = {"decision_type": decision_type, "selector": selector, "action": action}
return f"{decision_type}:{_hash_payload(payload)[:40]}"
def _hash_payload(payload: dict[str, Any]) -> str:
encoded = json.dumps(payload, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
return hashlib.sha256(encoded).hexdigest()
def _json_object(value: str | None) -> dict[str, Any]:
if not value:
return {}
try:
payload = json.loads(value)
except json.JSONDecodeError:
return {}
return payload if isinstance(payload, dict) else {}
def _json_dumps(value: dict[str, Any]) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), default=str)
def _empty_counts() -> dict[str, int]:
return {"created": 0, "updated": 0, "reopened": 0, "unchanged": 0, "resolved": 0, "items": 0}
def _merge_counts(target: dict[str, Any], source: dict[str, int]) -> None:
for key in ["created", "updated", "reopened", "unchanged", "resolved"]:
target[key] = int(target.get(key, 0)) + int(source.get(key, 0))
def _optional_int(value: object) -> int | None:
try:
return None if value is None else int(value)
except (TypeError, ValueError):
return None
def _emit(
progress_callback: ProgressCallback | None,
event_type: str,
message: str,
progress_current: int | None,
progress_total: int | None,
metadata: dict[str, Any] | None = None,
) -> None:
if progress_callback is not None:
progress_callback(event_type, message, progress_current, progress_total, metadata)
def _severity_order_expression():
return case(
(MapGtfsReviewItem.severity == "error", 0),
(MapGtfsReviewItem.severity == "warn", 1),
(MapGtfsReviewItem.severity == "info", 2),
else_=9,
)

159
app/worker_supervisor.py
View File

@@ -4,6 +4,7 @@ import os
import signal
import subprocess
import sys
import time
from dataclasses import dataclass
from pathlib import Path
@@ -24,8 +25,8 @@ class WorkerHandle:
_handles: list[WorkerHandle] = []
def start_queue_workers() -> list[WorkerHandle]:
if not settings.queue_worker_autostart:
def start_queue_workers(*, force: bool = False) -> list[WorkerHandle]:
if not force and not settings.queue_worker_autostart:
return []
worker_count = max(0, int(settings.queue_worker_count))
handles: list[WorkerHandle] = []
@@ -36,7 +37,7 @@ def start_queue_workers() -> list[WorkerHandle]:
pid_file = worker_dir / f"{worker_id}.pid"
log_file = worker_dir / f"{worker_id}.log"
existing_pid = _read_pid(pid_file)
if existing_pid is not None and _pid_running(existing_pid):
if existing_pid is not None and _worker_pid_running(existing_pid, worker_id):
handles.append(
WorkerHandle(
index=index,
@@ -69,16 +70,77 @@ def start_queue_workers() -> list[WorkerHandle]:
def stop_queue_workers() -> None:
if not settings.queue_worker_stop_on_shutdown:
return
for handle in list(_handles):
if not handle.started_by_server or handle.pid is None:
stop_configured_queue_workers()
def stop_configured_queue_workers() -> list[WorkerHandle]:
handles: list[WorkerHandle] = []
worker_dir = settings.data_dir / "workers"
worker_dir.mkdir(parents=True, exist_ok=True)
worker_count = max(0, int(settings.queue_worker_count))
for index in range(worker_count):
worker_id = f"server-worker-{index + 1}"
pid_file = worker_dir / f"{worker_id}.pid"
log_file = worker_dir / f"{worker_id}.log"
pid = _read_pid(pid_file)
if pid is None:
handles.append(
WorkerHandle(
index=index,
worker_id=worker_id,
pid=None,
status="already_stopped",
pid_file=pid_file,
log_file=log_file,
)
)
continue
_terminate_pid(handle.pid)
handle.pid_file.unlink(missing_ok=True)
if not _worker_pid_running(pid, worker_id):
pid_file.unlink(missing_ok=True)
handles.append(
WorkerHandle(
index=index,
worker_id=worker_id,
pid=pid,
status="stale_pid_removed",
pid_file=pid_file,
log_file=log_file,
)
)
continue
stopped = _terminate_pid(pid)
if stopped:
pid_file.unlink(missing_ok=True)
status = "stopped"
stored_pid = None
else:
status = "stop_requested"
stored_pid = pid
handles.append(
WorkerHandle(
index=index,
worker_id=worker_id,
pid=stored_pid,
status=status,
pid_file=pid_file,
log_file=log_file,
)
)
_handles[:] = [
handle
for handle in _handles
if handle.pid is not None and _worker_pid_running(handle.pid, handle.worker_id)
]
return handles
def restart_queue_workers() -> dict[str, list[WorkerHandle]]:
stopped = stop_configured_queue_workers()
started = start_queue_workers(force=True)
return {"stopped": stopped, "started": started}
def queue_worker_status() -> list[dict[str, object]]:
if not settings.queue_worker_autostart:
return []
worker_dir = settings.data_dir / "workers"
statuses: list[dict[str, object]] = []
configured_count = max(0, int(settings.queue_worker_count))
@@ -87,13 +149,20 @@ def queue_worker_status() -> list[dict[str, object]]:
pid_file = worker_dir / f"{worker_id}.pid"
log_file = worker_dir / f"{worker_id}.log"
pid = _read_pid(pid_file)
running = pid is not None and _pid_running(pid)
pid_alive = pid is not None and _pid_running(pid)
running = pid is not None and _worker_pid_running(pid, worker_id)
stale = pid is not None and not running
status = "running" if running else "stale" if stale else "stopped"
if pid_alive and stale:
status = "stale_pid_reused"
statuses.append(
{
"index": index,
"worker_id": worker_id,
"pid": pid,
"running": running,
"stale": stale,
"status": status,
"pid_file": str(pid_file),
"log_file": str(log_file),
}
@@ -145,11 +214,75 @@ def _pid_running(pid: int) -> bool:
return False
except PermissionError:
return True
return True
return not _pid_is_zombie(pid)
def _terminate_pid(pid: int) -> None:
def _pid_is_zombie(pid: int) -> bool:
try:
os.kill(pid, signal.SIGTERM)
for line in Path(f"/proc/{pid}/status").read_text(encoding="utf-8", errors="replace").splitlines():
if line.startswith("State:"):
return "\tZ" in line or "zombie" in line.lower()
except OSError:
return False
return False
def _worker_pid_running(pid: int, worker_id: str) -> bool:
return _pid_running(pid) and _pid_matches_worker(pid, worker_id)
def _pid_matches_worker(pid: int, worker_id: str) -> bool:
try:
raw_parts = Path(f"/proc/{pid}/cmdline").read_bytes().split(b"\0")
except (FileNotFoundError, ProcessLookupError, PermissionError, OSError):
return False
parts = [part.decode("utf-8", errors="replace") for part in raw_parts if part]
if "app.cli" not in parts or "worker" not in parts:
return False
try:
worker_id_index = parts.index("--worker-id") + 1
except ValueError:
return False
return worker_id_index < len(parts) and parts[worker_id_index] == worker_id
def _terminate_pid(pid: int, *, timeout_seconds: float = 5.0) -> bool:
_signal_pid(pid, signal.SIGTERM)
if _wait_for_exit(pid, timeout_seconds):
return True
_signal_pid(pid, signal.SIGKILL)
return _wait_for_exit(pid, 1.0)
def _signal_pid(pid: int, sig: signal.Signals) -> None:
try:
os.killpg(pid, sig)
except ProcessLookupError:
return
except OSError:
try:
os.kill(pid, sig)
except ProcessLookupError:
return
def _wait_for_exit(pid: int, timeout_seconds: float) -> bool:
deadline = time.monotonic() + max(0.0, timeout_seconds)
while time.monotonic() < deadline:
if not _pid_running(pid):
_reap_pid(pid)
return True
time.sleep(0.1)
if not _pid_running(pid):
_reap_pid(pid)
return True
return False
def _reap_pid(pid: int) -> None:
try:
os.waitpid(pid, os.WNOHANG)
except ChildProcessError:
return
except OSError:
return

202
docs/backlog.md
View File

@@ -1,202 +0,0 @@
# Product and Engineering Backlog
Last updated: 2026-07-01
This backlog reflects the current Germany-scale PostGIS prototype. The target remains a Europe-scale mobility data workbench that builds canonical stops, stations, routes, route geometry, timetable links, transfer rules, routing graph data, address search, and coverage evidence from many public sources.
OSM-derived geometry is the preferred visual authority. GTFS, NeTEx, realtime, and official APIs are timetable, validation, routing, and gap-detection inputs. GTFS shapes are still valuable evidence, especially for missing OSM relations and temporary detours.
## Current State
- PostgreSQL/PostGIS is the active development database path; SQLite remains a legacy/test fallback.
- Germany OSM and Germany GTFS/DELFI-scale imports are supported.
- OSM address indexing is available and address search is bbox-aware without being bbox-limited.
- Jobs and job events exist for imports, route matching, route-layer rebuilds, address indexing, relabeling, deletes, and maintenance.
- Job rows expose a generic details overlay with planned/current/done phases, event log, metadata, and a compact queue snapshot.
- A first QA dashboard skeleton exists for source discovery, import health, GTFS validation, canonical stop/link coverage, route matching, and publication readiness.
- The GTFS harmonization target architecture is documented in `docs/gtfs_harmonization.md`.
- GTFS source management is presented as a separate `GTFS Harmonization` UI module; OSM/map inputs are presented as a separate `Mapping Data` module.
- Journey search consumes the active harmonized transit snapshot instead of exposing a raw GTFS source selector.
- Route-layer rebuild runs through the queue, but it is still coarse-grained and can take minutes on national datasets.
- The route-layer builder links canonical GTFS stops, OSM stops, OSM route relations, GTFS route patterns, and trip-pattern links.
- Journey search is progressive and can publish intermediate results, but the underlying routing algorithm is still a prototype.
- Walk and drive routing use the OSM-derived routing layer when available.
## Current Caveats
- Journey search is not yet a full RAPTOR/CSA-style router.
- Address endpoints can multiply the search space: current behavior can use up to 4 access stops and 4 egress stops, creating up to 16 transit stop-pair searches per transfer stage.
- Progressive stages still recompute too much. Searching `up to 2 transfers` repeats direct and one-transfer work before deeper expansion.
- Walking access/egress legs are represented separately in journey output, but the search engine still needs a cleaner transfer budget model where access/egress walking never consumes public-transport transfer count.
- Route-search caches are in-process only. They do not survive server restart, do not deduplicate identical searches already running in another thread/process, and only help once a stage/search has completed.
- Route-layer rebuild currently clears/rebuilds derived tables. Until the rebuild completes, visual route-pattern link tables can be incomplete.
- Timetable reachability should not depend on visual route-pattern links. The code has been patched in this checkout, but a running server must reload before using that fix.
- Canonical stop extraction on national feeds is CPU/memory heavy and does too much Python-side grouping.
- OSM stop-linking and OSM route-candidate indexing are still large spatial/batch operations.
- GTFS detours are not classified as first-class route variants yet.
- Local-transport-only routing is not a first-class profile yet.
- Proper Alembic migrations are still missing; runtime schema maintenance should be reduced to an explicit migration/maintenance path.
- The source and job database tables are still shared between harmonization, mapping, and routing; the current split is a product/UI boundary, not a separate service or database boundary yet.
## P0: Routing Performance and Correctness
These items directly address slow or failed searches such as `Berlin, Alexanderplatz` to `Heidelberg, Blumenstrasse 36`.
- Replace the demo round-expansion router with a timetable-native algorithm.
Preferred direction: RAPTOR or CSA over preloaded arrays/tables, with rounds representing public-transport boardings rather than ad hoc SQL expansion.
- Precompute a transfer graph.
Store station-internal transfers, nearby walking transfers, platform/stop-place links, and allowed transfer times by mode/source/station.
- Separate access/egress from transfer count.
Walking from an address to the first stop, and from the last stop to an address, should never count as a vehicle transfer.
- Add a durable journey cache.
Cache normalized requests, address-to-stop candidates, stop-to-stop stage results, common station-pair results, and in-flight request deduplication in PostgreSQL.
- Add hub-aware long-distance routing.
For long-distance OD pairs, search local access to likely hubs, trunk rail/regional candidates, then local egress. Candidate hubs can be ranked by station importance, service frequency, route scope, distance, and direction.
- Add a local-transport-only profile.
Implement a Deutschlandticket-like profile that excludes long-distance route scopes and still supports regional rail, S-Bahn, subway, tram, bus, ferry, and walking transfers.
- Add admissible pruning.
Bound exploration by best known arrival, remaining distance, direction/off-course penalty, transfer budget, service frequency, and maximum tolerated detour.
- Add journey diagnostics.
Return searched stages, candidate counts, pruned reasons, access/egress stops, service date, source feeds, transfer stops, and whether no-route means no timetable path or a search limit was hit.
- Add arrive-by search.
This is important for route quality and for comparing against operator/DB route planners.
- Add route profile controls in the UI.
`fastest`, `earliest arrival`, `fewest transfers`, `local only`, `walk`, `drive`, `arrive by`, `via`, `avoid`, and transfer buffer controls.
## P0: Queue and Rebuild Robustness
- Move runtime schema maintenance out of normal app startup.
The current checkout avoids redundant PostgreSQL DDL, but explicit migrations are still needed.
- Add Alembic migrations.
Use migrations for PostGIS columns, indexes, route-layer tables, routing tables, and cache tables.
- Make route-layer rebuild use shadow tables or versioned rows.
Build replacement rows without deleting the readable active layer first; atomically promote the new version when complete.
- Make route-layer rebuild incremental.
Rebuild only affected route patterns after new matches, stop-link decisions, source updates, or OSM diffs.
- Add stale worker and stale pid reconciliation.
Worker status should never report a pid as running unless the current server can verify it.
- Improve cancellation.
Long PostgreSQL statements need cancellable phases and visible progress rather than only a queued/running state.
- Improve progress granularity and timings.
The UI can display job events now, but long PostgreSQL statements still need finer checkpoints, elapsed times, estimated remaining work, and cancellable sub-phases.
## P1: Route Layer, Detours, and Geometry Provenance
- Classify GTFS route variants.
Group trips by route, direction, shape, stop sequence, service date span, and trip frequency. Mark rare/temporary shapes as detours or temporary variants rather than replacing the canonical visual route.
- Add stop-by-stop OSM path fallback.
When an OSM route relation is missing or a GTFS shape is a detour, assemble geometry between matched consecutive stops using mode-constrained OSM paths.
- Cache stop-to-stop route geometry.
Key by mode, from canonical stop, to canonical stop, direction constraints, and graph version.
- Store geometry provenance per route pattern.
Examples: `osm_route_relation`, `gtfs_shape`, `stop_to_stop_osm_path`, `manual_override`, `detour_variant`.
- Respect directionality.
Bus/car paths need oneway handling; tram/rail paths need topology and direction evidence; reverse links must not be assumed valid.
- Add route-pattern detail inspection.
Show OSM geometry, GTFS shapes, linked trips, linked stops, direction evidence, confidence, and variant/detour status.
- Add generalized route geometries.
Store high-detail inspection geometry and simplified map geometry.
## P1: Canonical Stops, Stations, and Addresses
- Optimize canonical stop extraction.
Push more grouping/linking into SQL, avoid loading all scheduled stops into Python, batch inserts, and keep stable canonical IDs when possible.
- Build a canonical stop alias table.
Persist normalized names, multilingual names, station codes, IBNR/EVA/UIC/IFOPT, stop_area IDs, OSM IDs, and source-specific aliases.
- Improve station-complex modeling.
Separate public stop place, station complex, platforms/tracks, entrances, bus bays, and nearby stop groups.
- Add canonical stop detail overlay.
Show linked GTFS stops, linked OSM stops/stations, source names, confidence, distances, and manual overrides.
- Add manual canonical stop link/unlink decisions.
Persist stop matching decisions like route matching decisions, so source updates do not overwrite reviewed links.
- Improve address result folding.
Prefer street-level suggestions for dense house-number ranges, but preserve exact address selection when a full address is typed.
- Precompute address access candidates.
Store nearest useful public-transport stops per address/street point, with mode/source/radius metadata.
## P1: More GTFS Sources and Deduplication
- Import more GTFS feeds where they improve authority or coverage.
DB long-distance/regional feeds, state feeds, and neighboring-country feeds touching Germany are useful test cases.
- Add source priority and authority ranking.
Decide which source is more authoritative for stops, operators, routes, calendars, and geometry evidence.
- Deduplicate operators/agencies.
Merge agency/operator records with provenance and aliases instead of treating each GTFS `agency.txt` row as a separate operator.
- Turn QA summary counters into review queues.
Drill down from each bad/warn metric into concrete sources, stops, routes, links, and conflicts.
- Add GTFS feed QA reports.
Calendar coverage, stale feeds, missing shapes, impossible stop times, duplicate routes, route direction coverage, stop coordinate outliers.
- Add conflict dashboards and reusable resolution workflows.
Show canonical stops/routes with competing source claims, weak matches, missing visual geometry, authority-rule conflicts, and license blockers.
## P1: Scalable OSM and Map Outputs
- Keep OSM PBF import chunked and resumable.
Keep previous active visual datasets available while the next import builds.
- Add vector tile or PMTiles export.
Needed for Germany/Europe route layers and dense editing views.
- Add route-scope and mode-specific map generalization.
Different zooms should use different detail levels and route classes.
- Improve OSM route candidate indexing.
Use stronger SQL/PostGIS filtering before loading route geometry into Python.
- Add OSM diffs later.
Minutely/hourly/daily diffs can update route and address layers without full country rebuilds.
## P2: Data Platform Hardening
- Add explicit read/write transaction boundaries for all long requests and jobs.
- Add API pagination for large result sets.
- Add import logs and source-run history.
- Add database maintenance commands: analyze, vacuum, reindex, orphan cleanup.
- Add test fixtures that do not mutate the live development database.
- Add observability: query timings, job timings, row counts, cache hit rates, and per-stage routing metrics.
## P2: Better Map and Editing Workflows
- Add canonical stop and route detail side panels.
- Add candidate map preview for stop matching, not only route matching.
- Add unmatched/matched/weak/proposed visual layers with source filters.
- Keep calculated journey geometry and stop markers always on top.
- Add editable match queues for stops, station complexes, routes, and operators.
- Add route-layer diff view after rebuilds.
## P3: Additional Formats and Live Data
- Add NeTEx import.
- Add GTFS-Realtime ingestion for service alerts and trip updates.
- Add SIRI profile support where national APIs expose it.
- Add GBFS/shared mobility only after core public transport data is stable.
- Model temporary closures and disruptions as validity-windowed events, not modifications to base route geometry.
## Open Optimization List
Not yet implemented, or only partially implemented:
- RAPTOR/CSA routing core.
- Precomputed public-transport transfer graph.
- Durable PostgreSQL route-search cache.
- In-flight identical search coalescing.
- Hub-aware long-distance routing.
- Local-transport-only routing profile.
- Access/egress legs excluded from transfer budget at the search-state level.
- Better pruning for off-course exploration and dominated labels.
- SQL/array-based canonical stop extraction.
- Incremental route-layer rebuild.
- Route-layer shadow tables/versioned activation.
- Stop-to-stop OSM route fallback for missing routes and detours.
- Detour/temporary variant classification.
- PostGIS-first OSM route candidate filtering.
- Vector tiles or PMTiles for large route layers.
- Alembic migrations.
- Persistent query/stage timing diagnostics.
## Recommended Next Sprint
1. Finish the route-layer rebuild currently in progress and verify route-pattern/trip-pattern link counts.
2. Restart/reload the server so it picks up the current checkout fixes.
3. Add route-search diagnostics and timing instrumentation around address access, direct, one-transfer, and round-search stages.
4. Implement transfer graph precomputation and exclude access/egress walking from transfer count.
5. Add a hub-aware city-to-city search path for long-distance requests.
6. Add a local-only routing profile using route scopes.
7. Convert route-layer rebuild to shadow/versioned tables or incremental updates.
8. Add Alembic migrations and stop doing routine schema checks during normal app/worker startup.

15
docs/generated/gtfs_discovery_report.json
View File

@@ -1,5 +1,5 @@
{
"generated_at": "2026-07-01T13:22:51.928761+00:00",
"generated_at": "2026-07-02T10:38:44.832065+00:00",
"countries": [
"DE",
"AT",
@@ -18,27 +18,30 @@
"sources": {
"mobility_database": "https://files.mobilitydatabase.org/feeds_v2.csv",
"mobility_acceptance_test_list": "https://raw.githubusercontent.com/MobilityData/gtfs-validator/master/scripts/mobility-database-harvester/acceptance_test_feed_list.csv",
"ptna": "https://ptna.openstreetmap.de/gtfs/index.html"
"ptna": "https://ptna.openstreetmap.de/gtfs/index.html",
"opendata_oepnv": "https://www.opendata-oepnv.de/ht/de/datensaetze"
},
"counts": {
"candidates": 1090,
"ingestable": 990,
"candidates": 1131,
"ingestable": 998,
"test_run": 24,
"by_source": {
"curated_seed": 9,
"curated_seed; mobility_database": 2,
"curated_seed; mobility_database; mobility_validator_acceptance": 2,
"mobility_database": 805,
"mobility_database; mobility_validator_acceptance": 149,
"mobility_database; mobility_validator_acceptance": 148,
"mobility_database; mobility_validator_acceptance; opendata_oepnv": 1,
"mobility_database; mobility_validator_acceptance; ptna": 1,
"mobility_validator_acceptance": 3,
"opendata_oepnv": 41,
"ptna": 119
},
"ingestable_by_country": {
"AT": 6,
"BE": 15,
"CH": 10,
"DE": 68,
"DE": 76,
"DK": 1,
"EU": 1,
"FI": 31,

2223
docs/generated/gtfs_feed_candidates.csv
View File

File diff suppressed because it is too large Load Diff

1988
docs/generated/gtfs_ingestable_sources.csv
View File

File diff suppressed because it is too large Load Diff

48
docs/generated/gtfs_test_run_sources.csv
View File

@@ -1,25 +1,25 @@
name,kind,url,country,license,mode_scope,source_basis,priority,notes
CH Swiss national GTFS,gtfs,https://gtfs.geops.ch/dl/gtfs_complete.zip,CH,verify at opentransportdata.swiss,"rail,tram,metro,bus,ferry",European transport feeds / official Swiss OTD derivative,P0,geOps feed is derived from official Swiss Open Transport Data; verify production terms.
NL OpenOV national GTFS,gtfs,http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip,NL,verify OpenOV/NDOV terms,"rail,tram,metro,bus,ferry",European transport feeds / OpenOV,P0,Use NDOV/OVapi for production and realtime.
NL OVapi GTFS,gtfs,http://gtfs.ovapi.nl/gtfs-nl.zip,NL,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip
DK Rejseplanen GTFS,gtfs,https://www.rejseplanen.info/labs/GTFS.zip,DK,verify Rejseplanen Labs terms,"rail,bus",Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip
FI Helsingin seudun liikenne (HSL) GTFS,gtfs,http://dev.hsl.fi/gtfs/hsl.zip,FI,see http://developer.reittiopas.fi/pages/en/home.php,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip
NO Entur national aggregated GTFS,gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip,NO,verify Entur terms/NLOD,"rail,tram,metro,bus,ferry",Entur; Mobility Database feed catalog,P0,GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip
IE Aircoach GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip
"IE BK & Sons, JJ/Bernard Kavanagh GTFS",gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip,IE,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip
IE Bus Éireann GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip
GB BODS national GTFS,gtfs,https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/,GB,OGL/verify BODS terms,"rail,bus",BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip
DE DB Long-distance Rail GTFS.DE,gtfs,https://download.gtfs.de/germany/fv_free/latest.zip,DE,Creative Commons 4.0,rail,GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog,P1,Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip
DE Public Transport Germany GTFS,gtfs,https://download.gtfs.de/germany/nv_free/latest.zip,DE,see https://www.nvbw.de/open-data/lizenz,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1090/latest.zip
DE Regional Rail Transport Germany GTFS,gtfs,https://download.gtfs.de/germany/rv_free/latest.zip,DE,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1089/latest.zip
DE Verkehrsverbund Berlin-Brandenburg (VBB) GTFS,gtfs,http://vbb.de/vbbgtfs,DE,see http://vbb.de/vbbgtfs,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-782/latest.zip
DE Verkehrsverbund Rhein-Neckar GTFS,gtfs,https://geoportal.vrn.de/services/sharing/rest/content/items/4ec4b1d131eb46a6bb8e216ce9b90eff/data,DE,see https://www.vrn.de/opendata/datasets/soll-fahrplandaten-gtfs-aktuell,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1173/latest.zip
"AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS",gtfs,http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip
NL OVapi GTFS,gtfs,https://gtfs.ovapi.nl/nl/gtfs-nl.zip,NL,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi,P2,PTNA candidate; use original publisher URL where available.
FI Helsingin seudun liikenne GTFS,gtfs,https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip,FI,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL,P2,PTNA candidate; use original publisher URL where available.
CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip
CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip
AT Bean Shuttle GTFS,gtfs,https://www.beanshuttle.com/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip
AT Optima Express GTFS,gtfs,https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip,AT,,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip
"FI Haarasilta Toivo Samuli, Järvisen Liikenne Oy, Koiviston Auto Oy, Lehtimäen Liikenne Oy, Bus Travel Oy Reissu Ruoti, Tilausliikenne Kuisma Ky GTFS",gtfs,https://tvv.fra1.digitaloceanspaces.com/223.zip,FI,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1129/latest.zip
"NO Vestfold Kollektivtrafikk, Bastø Fosen GTFS",gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_vkt-aggregated-gtfs.zip,NO,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-1048/latest.zip
CH Swiss national GTFS,gtfs,https://gtfs.geops.ch/dl/gtfs_complete.zip,CH,verify at opentransportdata.swiss,"rail,tram,metro,bus,ferry",European transport feeds / official Swiss OTD derivative,P0,geOps feed is derived from official Swiss Open Transport Data; verify production terms.; Import policy: canonical_candidate; overlap role: canonical_baseline.
DE Rhein-Neckar-Verkehr GTFS,gtfs,https://gtfs-sandbox-dds.rnv-online.de/latest/gtfs.zip,DE,Creative Commons,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list; OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-rnv&cHash=6b4f1a9d1a4c981c2703f4446d234027; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details,P0,Selected Mobility Database latest.zip mirror because the catalog direct URL is known to be stale.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Selected non-mirror publisher URL from merged source evidence.; Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-777/latest.zip; Import policy: qa_or_override_not_additive; overlap role: operator_detail_candidate.
NL OpenOV national GTFS,gtfs,http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip,NL,verify OpenOV/NDOV terms,"rail,tram,metro,bus,ferry",European transport feeds / OpenOV,P0,Use NDOV/OVapi for production and realtime.; Import policy: canonical_candidate; overlap role: canonical_baseline.
NL OVapi GTFS,gtfs,http://gtfs.ovapi.nl/gtfs-nl.zip,NL,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
DK Rejseplanen GTFS,gtfs,https://www.rejseplanen.info/labs/GTFS.zip,DK,verify Rejseplanen Labs terms,"rail,bus",Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
FI Helsingin seudun liikenne (HSL) GTFS,gtfs,http://dev.hsl.fi/gtfs/hsl.zip,FI,see http://developer.reittiopas.fi/pages/en/home.php,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
NO Entur national aggregated GTFS,gtfs,https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip,NO,verify Entur terms/NLOD,"rail,tram,metro,bus,ferry",Entur; Mobility Database feed catalog,P0,GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
IE Aircoach GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
"IE BK & Sons, JJ/Bernard Kavanagh GTFS",gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip,IE,,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
IE Bus Éireann GTFS,gtfs,https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip,IE,see https://www.transportforireland.ie/transitData/PT_Data.html,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
GB BODS national GTFS,gtfs,https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/,GB,OGL/verify BODS terms,"rail,bus",BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
DE Aachener Verkehrsverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/dede57c7-9601-43cd-9fb6-8ab4c27412d2/resource/94744341-7116-402a-b0d5-9517d6c2060f/download/avv_gtfs_masten_mit_spnv_january_2020.zip,DE,Creative Commons Attribution,bus,OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-avv&cHash=b604ba50698ef0a26cd2a2685f168041; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
DE DB Long-distance Rail GTFS.DE,gtfs,https://download.gtfs.de/germany/fv_free/latest.zip,DE,Creative Commons 4.0,rail,GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog,P1,Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
DE Hamburger Verkehrsverbund GTFS,gtfs,https://opendata-oepnv.de/fileadmin/datasets/hvv/10-08-2017_Soll-Fahrplandaten.zip,DE,Creative Commons,"rail,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-hvv&cHash=559eb3ae637273e6cc2da67b34bc3c04; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
DE Münchner Verkehrs- und Tarifverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/885dfcc6-571e-47e4-85dc-e95d900fb257/resource/8f4addd8-ba55-4dde-8937-90c880fc6da0/download/250922_mvvcombi_ohneshape_google_transit.zip,DE,Creative Commons Attribution,"rail,tram,metro,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_gesamt&cHash=b3e6bda4cc9e5656172f0b0d0c120a98; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahr,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
DE Münchner Verkehrs- und Tarifverbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/17065229-c3fd-46d7-84a9-aae55aadbf40/resource/879dd863-1abc-4f64-8b27-a8b88994572d/download/mvv_gtfs.zip,DE,Creative Commons Attribution,bus,OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_reg&cHash=f25f41feaa97b6e02cb9dfcc260dd4c7; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrpla,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
DE Nordhessischer VerkehrsVerbund GTFS,gtfs,https://www.opendata-oepnv.de/dataset/4c35703c-62d2-463a-8f8d-014594ab6e30/resource/21c5b646-6f3f-461a-9493-4603357850eb/download/gtfs-20250704.zip,DE,Creative Commons,"rail,tram,bus",OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten_nvv&cHash=1fa9e9f4ab59d86496e5f03b934b6316; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat,P1,Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
"AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS",gtfs,http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P1,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
NL OVapi GTFS,gtfs,https://gtfs.ovapi.nl/nl/gtfs-nl.zip,NL,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi,P2,PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
FI Helsingin seudun liikenne GTFS,gtfs,https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip,FI,,bus,PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL,P2,PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS,gtfs,https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4,CH,see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0,bus,Mobility Database feed catalog,P0,Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
AT Bean Shuttle GTFS,gtfs,https://www.beanshuttle.com/gtfs.zip,AT,,bus,Mobility Database feed catalog; MobilityData validator acceptance-test feed list,P0,Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
AT Optima Express GTFS,gtfs,https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip,AT,,bus,Mobility Database feed catalog,P1,Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
1 name kind url country license mode_scope source_basis priority notes
2 CH Swiss national GTFS gtfs https://gtfs.geops.ch/dl/gtfs_complete.zip CH verify at opentransportdata.swiss rail,tram,metro,bus,ferry European transport feeds / official Swiss OTD derivative P0 geOps feed is derived from official Swiss Open Transport Data; verify production terms. geOps feed is derived from official Swiss Open Transport Data; verify production terms.; Import policy: canonical_candidate; overlap role: canonical_baseline.
3 NL OpenOV national GTFS DE Rhein-Neckar-Verkehr GTFS gtfs http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip https://gtfs-sandbox-dds.rnv-online.de/latest/gtfs.zip NL DE verify OpenOV/NDOV terms Creative Commons rail,tram,metro,bus,ferry bus European transport feeds / OpenOV Mobility Database feed catalog; MobilityData validator acceptance-test feed list; OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-rnv&cHash=6b4f1a9d1a4c981c2703f4446d234027; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details P0 Use NDOV/OVapi for production and realtime. Selected Mobility Database latest.zip mirror because the catalog direct URL is known to be stale.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Selected non-mirror publisher URL from merged source evidence.; Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-777/latest.zip; Import policy: qa_or_override_not_additive; overlap role: operator_detail_candidate.
4 NL OVapi GTFS NL OpenOV national GTFS gtfs http://gtfs.ovapi.nl/gtfs-nl.zip http://gtfs.openov.nl/gtfs-rt/gtfs-openov-nl.zip NL verify OpenOV/NDOV terms bus rail,tram,metro,bus,ferry Mobility Database feed catalog; MobilityData validator acceptance-test feed list European transport feeds / OpenOV P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip Use NDOV/OVapi for production and realtime.; Import policy: canonical_candidate; overlap role: canonical_baseline.
5 DK Rejseplanen GTFS NL OVapi GTFS gtfs https://www.rejseplanen.info/labs/GTFS.zip http://gtfs.ovapi.nl/gtfs-nl.zip DK NL verify Rejseplanen Labs terms rail,bus bus Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1077/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
6 FI Helsingin seudun liikenne (HSL) GTFS DK Rejseplanen GTFS gtfs http://dev.hsl.fi/gtfs/hsl.zip https://www.rejseplanen.info/labs/GTFS.zip FI DK see http://developer.reittiopas.fi/pages/en/home.php verify Rejseplanen Labs terms bus rail,bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list Rejseplanen Labs / European transport feeds; Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip May require account/terms review for production.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1292/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
7 NO Entur national aggregated GTFS FI Helsingin seudun liikenne (HSL) GTFS gtfs https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip http://dev.hsl.fi/gtfs/hsl.zip NO FI verify Entur terms/NLOD see http://developer.reittiopas.fi/pages/en/home.php rail,tram,metro,bus,ferry bus Entur; Mobility Database feed catalog Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-865/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
8 IE Aircoach GTFS NO Entur national aggregated GTFS gtfs https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_norway-aggregated-gtfs.zip IE NO see https://www.transportforireland.ie/transitData/PT_Data.html verify Entur terms/NLOD bus rail,tram,metro,bus,ferry Mobility Database feed catalog; MobilityData validator acceptance-test feed list Entur; Mobility Database feed catalog P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip GTFS is a subset; NeTEx is official/most complete.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1078/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
9 IE BK & Sons, JJ/Bernard Kavanagh GTFS IE Aircoach GTFS gtfs https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip https://www.transportforireland.ie/transitData/Data/GTFS_Aircoach.zip IE see https://www.transportforireland.ie/transitData/PT_Data.html bus Mobility Database feed catalog Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2640/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
10 IE Bus Éireann GTFS IE BK & Sons, JJ/Bernard Kavanagh GTFS gtfs https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip https://www.transportforireland.ie/transitData/Data/GTFS_Bernard_Kavanagh.zip IE see https://www.transportforireland.ie/transitData/PT_Data.html bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list Mobility Database feed catalog P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip Mobility Database mirror: https://files.mobilitydatabase.org/tld-592/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
11 GB BODS national GTFS IE Bus Éireann GTFS gtfs https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/ https://www.transportforireland.ie/transitData/Data/GTFS_Bus_Eireann.zip GB IE OGL/verify BODS terms see https://www.transportforireland.ie/transitData/PT_Data.html rail,bus bus BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2636/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
12 DE DB Long-distance Rail GTFS.DE GB BODS national GTFS gtfs https://download.gtfs.de/germany/fv_free/latest.zip https://data.bus-data.dft.gov.uk/timetable/download/gtfs-file/all/ DE GB Creative Commons 4.0 OGL/verify BODS terms rail rail,bus GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog BODS / Mobility Database; Mobility Database feed catalog; MobilityData validator acceptance-test feed list P1 P0 Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip England/GB bus focus; heavy rail separate.; Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2014/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
13 DE Public Transport Germany GTFS DE Aachener Verkehrsverbund GTFS gtfs https://download.gtfs.de/germany/nv_free/latest.zip https://www.opendata-oepnv.de/dataset/dede57c7-9601-43cd-9fb6-8ab4c27412d2/resource/94744341-7116-402a-b0d5-9517d6c2060f/download/avv_gtfs_masten_mit_spnv_january_2020.zip DE see https://www.nvbw.de/open-data/lizenz Creative Commons Attribution bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-avv&cHash=b604ba50698ef0a26cd2a2685f168041; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/avv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat P1 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1090/latest.zip Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
14 DE Regional Rail Transport Germany GTFS DE DB Long-distance Rail GTFS.DE gtfs https://download.gtfs.de/germany/rv_free/latest.zip https://download.gtfs.de/germany/fv_free/latest.zip DE Creative Commons 4.0 bus rail Mobility Database feed catalog; MobilityData validator acceptance-test feed list GTFS.DE / Deutsche Bahn long-distance rail; Mobility Database feed catalog P1 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1089/latest.zip Use as the first focused German rail feed for cross-source station deduplication with VBB and FlixTrain.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-768/latest.zip; Import policy: canonical_candidate; overlap role: canonical_baseline.
15 DE Verkehrsverbund Berlin-Brandenburg (VBB) GTFS DE Hamburger Verkehrsverbund GTFS gtfs http://vbb.de/vbbgtfs https://opendata-oepnv.de/fileadmin/datasets/hvv/10-08-2017_Soll-Fahrplandaten.zip DE see http://vbb.de/vbbgtfs Creative Commons bus rail,bus Mobility Database feed catalog OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-hvv&cHash=559eb3ae637273e6cc2da67b34bc3c04; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/hvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat P1 Mobility Database mirror: https://files.mobilitydatabase.org/mdb-782/latest.zip Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
16 DE Verkehrsverbund Rhein-Neckar GTFS DE Münchner Verkehrs- und Tarifverbund GTFS gtfs https://geoportal.vrn.de/services/sharing/rest/content/items/4ec4b1d131eb46a6bb8e216ce9b90eff/data https://www.opendata-oepnv.de/dataset/885dfcc6-571e-47e4-85dc-e95d900fb257/resource/8f4addd8-ba55-4dde-8937-90c880fc6da0/download/250922_mvvcombi_ohneshape_google_transit.zip DE see https://www.vrn.de/opendata/datasets/soll-fahrplandaten-gtfs-aktuell Creative Commons Attribution bus rail,tram,metro,bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_gesamt&cHash=b3e6bda4cc9e5656172f0b0d0c120a98; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahr P1 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1173/latest.zip Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
17 AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS DE Münchner Verkehrs- und Tarifverbund GTFS gtfs http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip https://www.opendata-oepnv.de/dataset/17065229-c3fd-46d7-84a9-aae55aadbf40/resource/879dd863-1abc-4f64-8b27-a8b88994572d/download/mvv_gtfs.zip AT DE Creative Commons Attribution bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrplandaten_mvv_reg&cHash=f25f41feaa97b6e02cb9dfcc260dd4c7; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/mvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll_fahrpla P1 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
18 NL OVapi GTFS DE Nordhessischer VerkehrsVerbund GTFS gtfs https://gtfs.ovapi.nl/nl/gtfs-nl.zip https://www.opendata-oepnv.de/dataset/4c35703c-62d2-463a-8f8d-014594ab6e30/resource/21c5b646-6f3f-461a-9493-4603357850eb/download/gtfs-20250704.zip NL DE Creative Commons bus rail,tram,bus PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi OpenData ÖPNV dataset catalog; details: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten_nvv&cHash=1fa9e9f4ab59d86496e5f03b934b6316; release: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsverbuende/nvv/startseite?tx_vrrkit_view%5Baction%5D=details&tx_vrrkit_view%5Bcontroller%5D=View&tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandat P2 P1 PTNA candidate; use original publisher URL where available. Discovered from OpenData ÖPNV; review per-dataset license and authority before publication.; Selected the first current GTFS download URL exposed on the detail page.; Import policy: qa_or_override_not_additive; overlap role: regional_override_candidate.
19 FI Helsingin seudun liikenne GTFS AT Wiener Lokalbahnen (WLB), Wiener Linien GTFS gtfs https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip http://www.wienerlinien.at/ogd_realtime/doku/ogd/gtfs/gtfs.zip FI AT bus PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL Mobility Database feed catalog; MobilityData validator acceptance-test feed list P2 P1 PTNA candidate; use original publisher URL where available. Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-648/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
20 CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS NL OVapi GTFS gtfs https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb https://gtfs.ovapi.nl/nl/gtfs-nl.zip CH NL see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0 bus Mobility Database feed catalog PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=NL-OVApi P0 P2 Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
21 CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS FI Helsingin seudun liikenne GTFS gtfs https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4 https://infopalvelut.storage.hsldev.com/gtfs/hsl.zip CH FI see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0 bus Mobility Database feed catalog PTNA GTFS analysis; details: https://ptna.openstreetmap.de/en/gtfs-details.php?feed=FI-18-HSL P0 P2 Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip PTNA candidate; use original publisher URL where available.; Import policy: metadata_only; overlap role: discovery_evidence.
22 AT Bean Shuttle GTFS CH Communauté d'Agglomération Annemasse - les Voirons Agglomération GTFS gtfs https://www.beanshuttle.com/gtfs.zip https://www.data.gouv.fr/api/1/datasets/r/373e19e2-af0a-4939-9f33-3f1268d1e0bb AT CH see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0 bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list Mobility Database feed catalog P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip Mobility Database mirror: https://files.mobilitydatabase.org/tdg-76779/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
23 AT Optima Express GTFS CH Communauté de communes pays d'Evian - vallée d'Abondance GTFS gtfs https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip https://www.data.gouv.fr/api/1/datasets/r/429c8587-676a-4ed3-8279-e67403bc36f4 AT CH see https://www.data.gouv.fr/pages/legal/licences/etalab-2.0 bus Mobility Database feed catalog P1 P0 Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip Mobility Database mirror: https://files.mobilitydatabase.org/tdg-80973/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
24 FI Haarasilta Toivo Samuli, Järvisen Liikenne Oy, Koiviston Auto Oy, Lehtimäen Liikenne Oy, Bus Travel Oy Reissu Ruoti, Tilausliikenne Kuisma Ky GTFS AT Bean Shuttle GTFS gtfs https://tvv.fra1.digitaloceanspaces.com/223.zip https://www.beanshuttle.com/gtfs.zip FI AT bus Mobility Database feed catalog; MobilityData validator acceptance-test feed list P0 Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-1129/latest.zip Useful smoke-test feed list; prefer Mobility Database feeds_v2 metadata for production source review.; Mobility Database mirror: https://files.mobilitydatabase.org/mdb-2036/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.
25 NO Vestfold Kollektivtrafikk, Bastø Fosen GTFS AT Optima Express GTFS gtfs https://storage.googleapis.com/marduk-production/outbound/gtfs/rb_vkt-aggregated-gtfs.zip https://github.com/jonaes/gtfs/raw/refs/heads/main/output/optima_gtfs.zip NO AT bus Mobility Database feed catalog P0 P1 Mobility Database mirror: https://files.mobilitydatabase.org/tld-1048/latest.zip Mobility Database mirror: https://files.mobilitydatabase.org/mdb-3123/latest.zip; Import policy: metadata_only; overlap role: catalog_metadata.

85
docs/gtfs_harmonization.md
View File

@@ -1,6 +1,6 @@
# GTFS Harmonization and QA Concept
Last updated: 2026-07-01
Last updated: 2026-07-02
## Decision
@@ -41,6 +41,24 @@ The pan-European output should be a canonical mobility dataset first, not one gi
## Core Concepts
### Layered Source Model
The target model has four distinct layers:
1. **Source directories / discovery indexes**: places where sources are found, such as Mobility Database, PTNA, OpenData ÖPNV, NAP registries, Transitland, or manually curated seed files.
2. **Sources / publishers / authorities**: the organization or authority that owns or publishes data, such as DELFI, VBB, RNV, a regional transport authority, or an operator.
3. **Feeds**: concrete machine-readable products from a source, such as a static GTFS ZIP, NeTEx package, GTFS-RT TripUpdates endpoint, GTFS-RT ServiceAlerts endpoint, or operator-specific API endpoint.
4. **Feed snapshots / datasets**: immutable downloaded/imported versions of a feed with hash, timestamp, parser version, validation report, activation status, and imported rows.
The current database does not fully separate these layers yet:
- `source_catalog_entries` is a lightweight discovery/backlog catalog.
- `sources` currently represents a registered importable feed, even though the name is broader.
- `datasets` represents imported feed snapshots.
- generated discovery files under `docs/generated/` contain many more feed candidates than are currently promoted into the database review workflow.
Because of this, the discovery catalog should be much larger than GTFS Feed QA. Feed QA should only show registered GTFS feeds that have been selected for review/import, not every source directory or every discovered candidate row.
### Source Registry
Track every identified source, including feeds not yet imported:
@@ -56,6 +74,8 @@ Mobility Database can be used as a broad discovery connector. Prefer the full `f
PTNA can be used as a GTFS/OSM QA and crosswalk connector. Its country pages expose feed IDs, provider names, release dates, validity windows, route-analysis links, detail pages, and original release-page links. Detail pages can add license text, OSM permission notes, `network:guid`, and route matching hints. PTNA should not become the canonical publisher for a feed; the harmonizer should follow the original provider URL where possible and keep PTNA as evidence.
OpenData ÖPNV is a Germany-focused source-registry connector. It lists DELFI national GTFS/NeTEx/ZHV datasets and many regional/operator datasets. Some downloads, including the Germany-wide DELFI GTFS on the portal, require registration, so those rows must stay in review until credentials and license decisions are recorded.
The generated discovery files live under `docs/generated/`:
- `gtfs_feed_candidates.csv` keeps every discovered feed/evidence row.
@@ -70,6 +90,23 @@ Required license flags before publication:
- `requires_attribution`
- `commercial_restrictions`
### Source Hierarchy
The harmonizer should not import every overlapping GTFS source into one active canonical snapshot. Each discovered source gets hierarchy metadata:
- `national_official` + `national` -> canonical baseline candidate.
- `regional_authority` -> QA or override candidate for a bounded region, not additive by default when a national baseline already covers the same services.
- `operator_feed` -> operator detail candidate, useful for QA, freshest local data, or replacing a bounded subset.
- `mirror` -> bootstrap only unless the original source is unavailable and license review allows use.
- `secondary_discovery` -> evidence metadata only.
Default import policy:
- Use one baseline per coverage universe.
- Add regional/operator feeds only as QA inputs or explicit replacements for a defined subset.
- Keep mirrors and aggregators out of canonical publication unless they are the only legally usable source and the provenance is explicit.
- Record every override decision so later imports can replay the same source precedence.
### Raw Snapshots
Every update should preserve immutable raw input:
@@ -84,6 +121,23 @@ Every update should preserve immutable raw input:
This keeps deduplication and conflict decisions reproducible.
### Update and Diff Policy
Source updates should not reset reviewed work. A new feed or map import should create a new raw snapshot, compute entity-level diffs, replay existing matching rules, and open review items only for changed, new, conflicting, or no-longer-valid evidence.
Initial diff units:
- feed metadata, hash, freshness, and service horizon;
- agencies/operators/networks;
- stops/platforms/station complexes;
- routes/lines;
- route patterns and stop sequences;
- trips, calendars, and stop times;
- route geometry evidence from GTFS shapes and OSM relations;
- OSM stop/platform/route relation candidates.
Unchanged raw rows may still be stored in the immutable snapshot tables for auditability. The expensive derived work should be incremental: canonical links, route geometry, route-layer rows, and QA review queues should rebuild only where the diff says something changed.
### Canonical Entities
Stable meubility IDs should be the internal truth. Source IDs remain aliases.
@@ -99,6 +153,26 @@ Initial canonical entity families:
- shapes/geometries;
- fares/ticketing references later.
### GTFS to Map Matching Workbench
The workbench should turn raw source evidence into reviewed, reusable map/timetable links:
- match GTFS stops to concrete OSM stop/platform/bay nodes or ways, preserving side-of-street and platform detail;
- link nearby but distinct stops with explicit transfer edges instead of merging them into one physical stop;
- match GTFS routes and route patterns to OSM route relations where the evidence is strong;
- when OSM route relations are missing or weak, propose stop-to-stop paths over mode-compatible OSM infrastructure;
- keep GTFS shapes as geometry evidence, especially for detours, temporary routes, and missing OSM relations;
- allow reviewer decisions such as accept OSM relation, use GTFS shape, use generated stop-to-stop map path, ignore visual geometry, split variant, mark detour, or create manual override;
- persist every decision as a reusable rule keyed by canonical entity, source scope, upstream IDs, normalized route keys, geometry hashes, and validity window.
Candidate generation must be constrained before expensive comparison:
- compare only OSM route features for route matching;
- filter by mode, network/operator/ref/name evidence, and source authority;
- use PostGIS bbox and distance filters before loading geometry into Python;
- prefer route-pattern and stop-sequence signatures over whole-feed scans;
- rerun candidate generation only for new or changed routes, stops, or map features.
### Authority Ranking
Conflict resolution needs explicit source authority:
@@ -123,6 +197,8 @@ The QA dashboard should expose review queues for:
- canonical stop groups with large spatial disagreement;
- routes with missing, weak, or conflicting OSM links;
- routes with missing shapes or route-pattern geometry;
- map-matching proposals that need reviewer approval;
- update diffs that invalidate a previous manual match;
- stale calendars and short service horizons;
- license/redistribution blockers.
@@ -149,7 +225,7 @@ Each export needs:
## Current Implementation Step
The first implementation is a lightweight harmonization boundary:
The current implementation is a lightweight harmonization boundary:
- `/api/qa/summary`;
- source discovery metrics;
@@ -159,7 +235,10 @@ The first implementation is a lightweight harmonization boundary:
- route matching and geometry counters;
- publication-readiness warnings.
- GTFS source add/import/review controls live in the `GTFS Harmonization` sidebar module.
- Feed review decisions store review status, note, source authority ranking, and explicit publication flags:
can import, can derive, can redistribute, attribution required, and commercial restrictions.
- OSM/route-layer source controls live in the `Mapping Data` sidebar module.
- The journey panel displays the active harmonized transit snapshot instead of a GTFS source picker.
- The current active harmonized snapshot is computed from active GTFS datasets, review authority, source priority, and route-key overlap; lower-precedence feeds that are mostly covered by a higher-precedence feed are shadowed for default routing while remaining available for QA/source-specific inspection.
This is intentionally a skeleton. The next step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions.
This is intentionally still a skeleton. The next implementation step is to turn non-zero warning/bad counters into review queues with drill-down lists and persistent resolution actions, then run the generated 24-source test set through review, import, validation, and deduplication.

88
docs/product_vision.md Normal file
View File

@@ -0,0 +1,88 @@
# Meubility Product Vision
Last updated: 2026-07-06
Meubility should become an EU-wide mobility platform that combines harmonized mobility data, clear visual representation, routing, comparison, and analysis tooling.
The platform should support both traveller-facing use cases and institutional planning use cases. The shared foundation is a canonical, auditable mobility dataset: sources are discovered, licensed, imported, validated, harmonized, matched to map evidence, versioned, and exposed through stable APIs and visual tools.
## Product Goal
The long-term product goal is to answer:
```text
How can a person or organization get from A to B, what options exist, how do those options compare, how reliable are they, what do they cost, what emissions do they produce, and what can be booked or reserved?
```
The answer should be explainable. Every route, price, emission estimate, reliability prognosis, and availability signal needs provenance and confidence metadata.
## Target Audiences
- General public travellers who need understandable route options across public transport, walking, biking, driving, and shared mobility.
- Public institutions, municipalities, states, and governments analyzing network quality, coverage gaps, accessibility, emissions, car dependency, and improvement potential.
- Companies planning employee travel, business travel, mobility budgets, or commuting alternatives.
- Municipalities, venues, and event agencies offering "car-free travel to destination" guidance.
- Data teams and transport planners who need a workbench for source review, harmonization, QA, and publication.
- Third-party developers and researchers consuming harmonized mobility data through APIs.
## Mobility Coverage
The canonical data model should eventually cover all relevant mobility options:
- Public transport: rail, regional rail, subway, tram, bus, coach, ferry, cableways, and demand-responsive services where available.
- Active mobility: walking and biking.
- Shared mobility: rental bikes, e-scooters, car sharing, station-based rental cars, free-floating vehicles, and similar services.
- Commercial individual transport: taxi, ride-hailing, chauffeur, rental car, and transfer services.
- Private car comparison: routing, duration, cost, parking assumptions, and emissions.
Booking and reservation support should be treated as a separate layer from route plausibility. The platform can show a plausible itinerary before it can guarantee or sell the full connection. Where booking exists, it should be clear whether the result is a deep link, a partner booking, multiple independent bookings, or a protected through-ticket.
## Traveller Comparison
Routing results should compare options by:
- Duration and arrival/departure time.
- Number of changes and transfer complexity.
- Walking/biking distance and accessibility constraints.
- Price or estimated cost.
- Emissions.
- Reliability prognosis, including connection risk and disruption sensitivity.
- Booking or reservation availability.
- Comfort and policy flags where available, such as local-transport-only, bike carriage, wheelchair accessibility, or company travel rules.
Missed-connection simulation should be a first-class analysis capability: if a transfer fails, the platform should estimate the likely delay, fallback options, booking consequences, and whether the route remains acceptable.
## Institutional Analytics
The same harmonized dataset should support analysis for public bodies and organizations:
- Coverage and accessibility by region, municipality, station, stop, venue, or event site.
- Route and timetable gaps.
- Missing map or timetable evidence.
- Reliability weak points and fragile transfers.
- Emissions and modal-shift potential.
- Car-free access plans for events, public buildings, tourism, campuses, and large employers.
- Source-quality reporting: freshness, license confidence, validation errors, route matching confidence, stop matching confidence, and coverage by authority level.
## Platform Architecture Direction
The product should be split into clear layers:
1. Source discovery and licensing.
2. Raw source snapshots and update history.
3. Validation and QA reports.
4. Harmonization and deduplication.
5. Map/GTFS matching workbench and persistent reviewer decisions.
6. Active canonical mobility snapshots for routing and analysis.
7. Routing, comparison, reliability, pricing, emissions, and booking-readiness services.
8. Public UI, institutional dashboard, and third-party API surfaces.
GTFS harmonization remains the next critical sprint because high-quality routing, comparison, and analytics depend on stable canonical timetable data. Shared mobility, pricing, emissions, reliability, and booking layers should build on that canonical base rather than bypassing it.
## Near-Term Product Implications
- Keep prioritizing the GTFS harmonization and Map/GTFS workbench tracks.
- Treat routing improvements as a parallel technical track, especially RAPTOR/CSA routing, durable caches, transfer graph precomputation, and route diagnostics.
- Add cost, emissions, reliability, and booking-readiness as explicit future data layers in the roadmap.
- Keep API readiness in view: harmonized GTFS and canonical mobility snapshots should become useful outputs even before the public traveller app is complete.
- Keep institutional analytics in scope from the start, because the workbench already stores the provenance, confidence, and quality evidence that public bodies and planners need.

11
docs/source_acquisition.md
View File

@@ -4,7 +4,7 @@ This repository now contains two seed catalogues:
- `docs/source_catalog_seed.csv` — broad discovery catalogue for official NAPs, feed registries, route-geometry evidence, realtime/disruption sources, rail/air registries and country notes.
- `docs/ingestable_sources_seed.csv` — direct static feeds that the current prototype can import immediately.
- `docs/generated/gtfs_feed_candidates.csv` — generated GTFS discovery manifest from Mobility Database, PTNA, the validator acceptance list, and curated local seeds.
- `docs/generated/gtfs_feed_candidates.csv` — generated GTFS discovery manifest from Mobility Database, PTNA, OpenData ÖPNV, the validator acceptance list, and curated local seeds.
- `docs/generated/gtfs_ingestable_sources.csv` — generated direct GTFS source rows suitable for source-registry import after license/source review.
- `docs/generated/gtfs_test_run_sources.csv` — generated focused feed set for the first multi-source harmonization/deduplication run.
@@ -14,7 +14,14 @@ Regenerate the GTFS discovery manifests:
python -m app.cli discover-gtfs-sources --max-ptna-details 0 --test-limit 24
```
Use `--countries ALL` for the broad global Mobility Database/acceptance-list pass. Use a positive `--max-ptna-details` when you want PTNA license and OSM crosswalk fields; the country-table scrape is fast, while detail pages can be slow.
Use `--countries ALL` for the broad global Mobility Database/acceptance-list pass. Use a positive `--max-ptna-details` when you want PTNA license and OSM crosswalk fields; the country-table scrape is fast, while detail pages can be slow. OpenData ÖPNV detail pages are fetched for Germany by default and can be skipped with `--no-opendata-oepnv`.
The candidate manifest includes source hierarchy fields:
- `authority_level` ranks the publisher/evidence source, for example `national_official`, `regional_authority`, `operator_feed`, `mirror`, or `secondary_discovery`.
- `coverage_scope` marks whether a feed is national, regional, operator-specific, multi-country, or unknown.
- `overlap_role` explains how to treat overlaps, for example `canonical_baseline`, `regional_override_candidate`, `operator_detail_candidate`, `bootstrap_mirror`, or `discovery_evidence`.
- `import_policy` is the practical guardrail. `qa_or_override_not_additive` means do not import the feed into the active canonical snapshot together with a broader baseline unless it is replacing a defined subset or being used for QA.
Import the direct feed seed list into the source registry:

1
docs/source_catalog_seed.csv
View File

@@ -28,6 +28,7 @@ Finland,FI,"rail, bus, tram, metro, ferry, bike",Fintraffic FINAP / national GTF
France,FR,"rail, bus, tram, metro, ferry, coach, air metadata",transport.data.gouv.fr,National NAP/catalog,"GTFS, NeTEx, GTFS-RT, SIRI; SSIM reference for air",public catalog; per-feed terms/auth,"Large official catalog for public transport, road, shared vehicles, carpooling, etc.; public transit datasets describe networks, stops, routes and times.",GTFS/NeTEx shapes/stops supersede OSM; consolidated stop datasets help stop registry.,GTFS-RT and SIRI feeds including SNCF service alerts/trip updates where available.,Operators/publishers from dataset metadata and agencies.,Licence per dataset.,P0,https://transport.data.gouv.fr/,https://transport.data.gouv.fr/datasets?type=public-transit&locale=en,Crawl catalog API/search; import SNCF and regional feeds; add RT after static.
France,FR,national rail,SNCF Open Data,Operator feed,"GTFS, NeTEx, SIRI Lite, GTFS-RT TripUpdates/ServiceAlerts",public; terms per dataset,SNCF publishes static and realtime passenger information datasets.,SNCF GTFS/NeTEx route and stop data supersede OSM for timetable layer.,GTFS-RT TripUpdates and ServiceAlerts; SIRI ET/SX Lite.,SNCF agency/operator records.,Terms per data.gouv dataset.,P0,https://ressources.data.sncf.com/,https://transport.data.gouv.fr/datasets?organization=sncf&locale=en,Create SNCF connector; add stale-feed and preview-window checks.
Germany,DE,"rail, bus, tram, metro, ferry if in ÖPNV",DELFI / Mobilithek national NeTEx + GTFS.de,National feed / derived GTFS,"NeTEx, GTFS; GTFS-RT aggregate",public; official static via NAP; GTFS derived,National static timetable data published via Mobilithek/DELFI; GTFS.de offers daily GTFS covering DB long-distance/regional and local/urban transit.,Official NeTEx / GTFS shapes supersede OSM for planned services.,GTFS.de RT stream aggregates realtime where open/licensed: TripUpdates/ServiceAlerts.,Agencies/operators from feed; NAP publishers.,Static open; RT may depend on open licences/special agreements.,P0,https://gtfs.de/en/,https://gtfs.de/en/,"Use GTFS.de for bootstrap, plan NeTEx ingestion for higher fidelity."
Germany,DE,"rail, bus, tram, metro, ferry if in ÖPNV",OpenData ÖPNV Deutschland,National/regional open-data portal and source registry,"GTFS, NeTEx, GTFS-RT, CSV, GeoJSON, XML, JSON, APIs",public catalog; some downloads require registration,"Portal lists DELFI national datasets plus regional/operator datasets such as VBB, VRR, MVV, HVV, VVS, RNV, AVV, RMV, NVV and others.",GTFS/NeTEx/stops can supersede OSM for planned services only after source hierarchy and license review.,GTFS-RT/TRIAS/API datasets are listed for selected publishers; access varies by dataset.,Publishers and organizations can seed authority/operator review.,Per-dataset license; common cases include CC BY and Datenlizenz Deutschland; explicit import/derivation/redistribution review required.,P0,https://www.opendata-oepnv.de/ht/de/datensaetze,https://www.opendata-oepnv.de/ht/de/datensaetze,"Use discovery connector; treat DELFI/national feeds as baseline, regional/operator feeds as QA or override candidates to avoid duplicate imports."
Germany / Berlin-Brandenburg,DE,"rail, bus, tram, metro, ferry",VBB Berlin-Brandenburg GTFS,Regional authority feed / official GTFS,GTFS,public; updated twice weekly,Official VBB timetable feed for Berlin and Brandenburg bus and rail services; useful regional bootstrap/demo source before national-scale ingestion.,GTFS stops/shapes/timetables supersede OSM for planned service verification where present.,Static timetable only in this row; VBB GTFS-RT is a separate realtime source.,Agencies/operators from agency.txt and VBB feed metadata.,CC-BY; attribution required: VBB Verkehrsverbund Berlin-Brandenburg GmbH.,P5,https://www.vbb.de/vbbgtfs,https://daten.berlin.de/datensaetze/vbb-fahrplandaten-via-gtfs,Use for Berlin model bootstrap; pair with Geofabrik Berlin OSM PBF and validate route geometry coverage.
Germany,DE,rail disruptions,DB Baustellen / construction works,Operator disruption/planned works,web/API where available,public web; structured access variable,Major long-distance construction works and timetable changes; useful for planned closure enrichment.,"Can supersede OSM for temporary rail service restrictions, not geometry.","Planned works, timetable changes.",DB as operator/infrastructure-related source.,Terms to verify.,P2,https://bauinfos.deutschebahn.com/,https://bauinfos.deutschebahn.com/,Research structured endpoints; otherwise link as non-ingested evidence.
Greece,GR,"metro, tram, bus, coach, ferry",Greece NAP + OASA telematics,NAP / operator portals,web/API; GTFS unknown/fragmented,public web; machine access uncertain,"Official NAP exists; Athens OASA publishes line schedules/stops and telematics app information, but open machine-readable coverage needs verification.",Official feeds if found supersede OSM; otherwise OSM remains strong existence layer.,Telematics app includes real-time arrivals/locations; open reuse unclear.,Operators from NAP/operator pages and OSM.,Likely access/licence work required.,P3,https://www.nap.gov.gr/,https://telematics.oasa.gr/,Investigate NAP catalog API and OASA endpoints/licence.
1 Geography Country code Mode scope Source name Source category Formats / APIs Availability Coverage notes Supersedes OSM for Disruptions / closures Operator-list use Access / licence notes Priority Source URL Evidence URL Next pipeline action
28 France FR rail, bus, tram, metro, ferry, coach, air metadata transport.data.gouv.fr National NAP/catalog GTFS, NeTEx, GTFS-RT, SIRI; SSIM reference for air public catalog; per-feed terms/auth Large official catalog for public transport, road, shared vehicles, carpooling, etc.; public transit datasets describe networks, stops, routes and times. GTFS/NeTEx shapes/stops supersede OSM; consolidated stop datasets help stop registry. GTFS-RT and SIRI feeds including SNCF service alerts/trip updates where available. Operators/publishers from dataset metadata and agencies. Licence per dataset. P0 https://transport.data.gouv.fr/ https://transport.data.gouv.fr/datasets?type=public-transit&locale=en Crawl catalog API/search; import SNCF and regional feeds; add RT after static.
29 France FR national rail SNCF Open Data Operator feed GTFS, NeTEx, SIRI Lite, GTFS-RT TripUpdates/ServiceAlerts public; terms per dataset SNCF publishes static and realtime passenger information datasets. SNCF GTFS/NeTEx route and stop data supersede OSM for timetable layer. GTFS-RT TripUpdates and ServiceAlerts; SIRI ET/SX Lite. SNCF agency/operator records. Terms per data.gouv dataset. P0 https://ressources.data.sncf.com/ https://transport.data.gouv.fr/datasets?organization=sncf&locale=en Create SNCF connector; add stale-feed and preview-window checks.
30 Germany DE rail, bus, tram, metro, ferry if in ÖPNV DELFI / Mobilithek national NeTEx + GTFS.de National feed / derived GTFS NeTEx, GTFS; GTFS-RT aggregate public; official static via NAP; GTFS derived National static timetable data published via Mobilithek/DELFI; GTFS.de offers daily GTFS covering DB long-distance/regional and local/urban transit. Official NeTEx / GTFS shapes supersede OSM for planned services. GTFS.de RT stream aggregates realtime where open/licensed: TripUpdates/ServiceAlerts. Agencies/operators from feed; NAP publishers. Static open; RT may depend on open licences/special agreements. P0 https://gtfs.de/en/ https://gtfs.de/en/ Use GTFS.de for bootstrap, plan NeTEx ingestion for higher fidelity.
31 Germany DE rail, bus, tram, metro, ferry if in ÖPNV OpenData ÖPNV Deutschland National/regional open-data portal and source registry GTFS, NeTEx, GTFS-RT, CSV, GeoJSON, XML, JSON, APIs public catalog; some downloads require registration Portal lists DELFI national datasets plus regional/operator datasets such as VBB, VRR, MVV, HVV, VVS, RNV, AVV, RMV, NVV and others. GTFS/NeTEx/stops can supersede OSM for planned services only after source hierarchy and license review. GTFS-RT/TRIAS/API datasets are listed for selected publishers; access varies by dataset. Publishers and organizations can seed authority/operator review. Per-dataset license; common cases include CC BY and Datenlizenz Deutschland; explicit import/derivation/redistribution review required. P0 https://www.opendata-oepnv.de/ht/de/datensaetze https://www.opendata-oepnv.de/ht/de/datensaetze Use discovery connector; treat DELFI/national feeds as baseline, regional/operator feeds as QA or override candidates to avoid duplicate imports.
32 Germany / Berlin-Brandenburg DE rail, bus, tram, metro, ferry VBB Berlin-Brandenburg GTFS Regional authority feed / official GTFS GTFS public; updated twice weekly Official VBB timetable feed for Berlin and Brandenburg bus and rail services; useful regional bootstrap/demo source before national-scale ingestion. GTFS stops/shapes/timetables supersede OSM for planned service verification where present. Static timetable only in this row; VBB GTFS-RT is a separate realtime source. Agencies/operators from agency.txt and VBB feed metadata. CC-BY; attribution required: VBB Verkehrsverbund Berlin-Brandenburg GmbH. P5 https://www.vbb.de/vbbgtfs https://daten.berlin.de/datensaetze/vbb-fahrplandaten-via-gtfs Use for Berlin model bootstrap; pair with Geofabrik Berlin OSM PBF and validate route geometry coverage.
33 Germany DE rail disruptions DB Baustellen / construction works Operator disruption/planned works web/API where available public web; structured access variable Major long-distance construction works and timetable changes; useful for planned closure enrichment. Can supersede OSM for temporary rail service restrictions, not geometry. Planned works, timetable changes. DB as operator/infrastructure-related source. Terms to verify. P2 https://bauinfos.deutschebahn.com/ https://bauinfos.deutschebahn.com/ Research structured endpoints; otherwise link as non-ingested evidence.
34 Greece GR metro, tram, bus, coach, ferry Greece NAP + OASA telematics NAP / operator portals web/API; GTFS unknown/fragmented public web; machine access uncertain Official NAP exists; Athens OASA publishes line schedules/stops and telematics app information, but open machine-readable coverage needs verification. Official feeds if found supersede OSM; otherwise OSM remains strong existence layer. Telematics app includes real-time arrivals/locations; open reuse unclear. Operators from NAP/operator pages and OSM. Likely access/licence work required. P3 https://www.nap.gov.gr/ https://telematics.oasa.gr/ Investigate NAP catalog API and OASA endpoints/licence.

6
scripts/discover_gtfs_sources.py
View File

@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Build GTFS source discovery manifests from Mobility Database, PTNA, and local seeds."""
"""Build GTFS source discovery manifests from Mobility Database, PTNA, OpenData ÖPNV, and local seeds."""
from __future__ import annotations
import argparse
@@ -25,7 +25,9 @@ def main() -> None:
parser.add_argument("--no-mobility-database", action="store_true", help="Skip Mobility Database feeds_v2.csv")
parser.add_argument("--no-acceptance-test-list", action="store_true", help="Skip MobilityData validator acceptance-test feed list")
parser.add_argument("--no-ptna", action="store_true", help="Skip PTNA GTFS analysis pages")
parser.add_argument("--no-opendata-oepnv", action="store_true", help="Skip OpenData ÖPNV Germany dataset catalog")
parser.add_argument("--max-ptna-details", type=int, default=80, help="Maximum PTNA detail pages to fetch")
parser.add_argument("--max-opendata-oepnv-details", type=int, default=80, help="Maximum OpenData ÖPNV detail pages to fetch")
parser.add_argument("--test-limit", type=int, default=24, help="Rows written to the focused test-run CSV")
parser.add_argument("--check-urls", action="store_true", help="Run HEAD/range checks for ingestable feed URLs")
args = parser.parse_args()
@@ -36,7 +38,9 @@ def main() -> None:
include_mobility_database=not args.no_mobility_database,
include_acceptance_test_list=not args.no_acceptance_test_list,
include_ptna=not args.no_ptna,
include_opendata_oepnv=not args.no_opendata_oepnv,
max_ptna_details=args.max_ptna_details,
max_opendata_oepnv_details=args.max_opendata_oepnv_details,
test_limit=args.test_limit,
check_urls=args.check_urls,
)

430
tests/test_api.py
View File

@@ -1,19 +1,24 @@
from __future__ import annotations
import json
from pathlib import Path
from fastapi.testclient import TestClient
from sqlalchemy import select
from sqlalchemy import delete, select
import app.jobs as jobs_module
import app.main as main_module
from app.config import settings
from app.db import init_db, session_scope
from app.db_lock import DatabaseWriteBusy, database_write_lock
from app.jobs import run_worker_once
from app.harmonization import active_harmonized_gtfs_dataset_ids, active_harmonized_gtfs_shadowed_route_ids
from app.jobs import add_job_event, run_worker_once
from app.main import app
from app.models import Dataset, GtfsRoute, Job, Source
from app.models import Dataset, GtfsHarmonizedSnapshotRoute, GtfsRoute, Job, RouteMatch, Source
from app.pipeline.matcher import run_route_matching
from app.pipeline.route_layer import rebuild_route_layer
from app.source_catalog import import_ingestable_sources
from app.worker_supervisor import WorkerHandle
def test_api_sample_and_geojson():
@@ -127,6 +132,28 @@ def test_route_matching_job_endpoint_completes():
assert event_types[-1] == "completed"
def test_job_events_endpoint_returns_latest_window_in_chronological_order():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
with session_scope() as session:
job = Job(kind="maintenance", status="running", description="event window test")
session.add(job)
session.flush()
for index in range(5):
add_job_event(
session,
job,
event_type=f"event_{index + 1}",
message=f"message {index + 1}",
progress_current=index + 1,
progress_total=5,
)
job_id = job.id
events = client.get(f"/api/jobs/{job_id}/events?limit=3").json()["events"]
assert [event["event_type"] for event in events] == ["event_3", "event_4", "event_5"]
def test_qa_summary_endpoint_exposes_harmonization_sections():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
@@ -168,14 +195,342 @@ def test_gtfs_harmonization_inventory_and_detail():
reviewed = client.patch(
f"/api/harmonization/gtfs/sources/{feed['source']['id']}/review",
json={"license": "CC-BY-4.0", "review_status": "approved", "review_note": "Operator publication allowed.", "enabled": True},
json={
"license": "CC-BY-4.0",
"review_status": "approved",
"review_note": "Operator publication allowed.",
"can_import": "yes",
"can_derive": "yes",
"can_redistribute": "yes",
"requires_attribution": "yes",
"commercial_restrictions": "no",
"authority_level": "regional_authority",
"enabled": True,
},
).json()
assert reviewed["source"]["license"] == "CC-BY-4.0"
assert reviewed["source"]["qa_review"]["status"] == "approved"
assert reviewed["source"]["qa_review"]["note"] == "Operator publication allowed."
assert reviewed["source"]["qa_review"]["can_redistribute"] == "yes"
assert reviewed["source"]["qa_review"]["authority_level"] == "regional_authority"
assert reviewed["license"]["redistribution_status"] == "allowed"
assert reviewed["source"]["enabled"] is True
def test_gtfs_harmonized_snapshot_shadows_lower_authority_duplicate_feed():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
with session_scope() as session:
primary_source = session.scalar(select(Source).where(Source.kind == "gtfs"))
primary_dataset = next(dataset for dataset in primary_source.datasets if dataset.kind == "gtfs" and dataset.is_active)
primary_source.priority = "P0"
primary_source.notes = _gtfs_review_note("national_official")
duplicate_source = Source(
name="Sample Berlin regional duplicate",
kind="gtfs",
url="https://example.invalid/regional-duplicate.zip",
country="DE",
license="sample",
priority="P1",
notes=_gtfs_review_note("regional_authority"),
)
session.add(duplicate_source)
session.flush()
duplicate_dataset = Dataset(
source_id=duplicate_source.id,
kind="gtfs",
local_path="data/sample/regional-duplicate.gtfs.zip",
sha256="d" * 64,
is_active=True,
status="imported",
)
session.add(duplicate_dataset)
session.flush()
routes = session.scalars(select(GtfsRoute).where(GtfsRoute.dataset_id == primary_dataset.id)).all()
for route in routes:
session.add(
GtfsRoute(
dataset_id=duplicate_dataset.id,
route_id=f"dup_{route.route_id}",
agency_id=route.agency_id,
short_name=route.short_name,
long_name=route.long_name,
route_type=route.route_type,
mode=route.mode,
route_scope=route.route_scope,
operator_name=route.operator_name,
route_key=route.route_key,
operator_key=route.operator_key,
min_lon=route.min_lon,
min_lat=route.min_lat,
max_lon=route.max_lon,
max_lat=route.max_lat,
)
)
primary_dataset_id = primary_dataset.id
duplicate_dataset_id = duplicate_dataset.id
snapshot = client.get("/api/harmonization/gtfs/snapshot").json()
assert snapshot["summary"]["raw_active_datasets"] == 2
assert snapshot["summary"]["included_datasets"] == 1
assert snapshot["summary"]["shadowed_datasets"] == 1
assert snapshot["dataset_ids"] == [primary_dataset_id]
shadowed = next(item for item in snapshot["datasets"] if item["dataset_id"] == duplicate_dataset_id)
assert shadowed["role"] == "shadowed"
assert shadowed["shadowed_by_dataset_id"] == primary_dataset_id
assert shadowed["route_key_overlap_ratio"] >= 0.8
with session_scope() as session:
assert active_harmonized_gtfs_dataset_ids(session) == [primary_dataset_id]
queued = client.post("/api/jobs/gtfs-harmonized-snapshot?activate=true").json()
assert queued["kind"] == "gtfs_harmonized_snapshot"
assert run_worker_once(worker_id="test-worker")["processed"] == 1
job = client.get(f"/api/jobs/{queued['id']}").json()
assert job["status"] == "completed"
assert job["result"]["snapshot"]["persisted"] is True
assert job["result"]["snapshot"]["dataset_ids"] == [primary_dataset_id]
persisted_snapshot = client.get("/api/harmonization/gtfs/snapshot").json()
assert persisted_snapshot["persisted"] is True
assert persisted_snapshot["dataset_ids"] == [primary_dataset_id]
workflow = client.get("/api/workbench/gtfs-workflow").json()
assert workflow["snapshot"]["persisted"] is True
assert workflow["inventory_summary"]["snapshot_shadowed_datasets"] == 1
diffs = client.get("/api/harmonization/gtfs/diffs").json()
assert diffs["summary"]["runs"] == 0
dedup_job = client.post("/api/jobs/map-gtfs-review?limit=50").json()
assert run_worker_once(worker_id="test-worker")["processed"] == 1
dedup_result = client.get(f"/api/jobs/{dedup_job['id']}").json()["result"]["review_result"]
assert dedup_result["dedup_items"] > 0
dedup_items = client.get("/api/workbench/map-gtfs/review-items?queue=gtfs_deduplication&status=open&limit=20").json()["items"]
assert dedup_items
assert dedup_items[0]["metadata"]["data_differs"] is True
def test_gtfs_harmonized_snapshot_partially_shadows_aggregator_routes_by_bbox():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
with session_scope() as session:
aggregator_source = session.scalar(select(Source).where(Source.kind == "gtfs"))
aggregator_source.notes = _gtfs_review_note("aggregator")
aggregator_dataset = next(dataset for dataset in aggregator_source.datasets if dataset.kind == "gtfs" and dataset.is_active)
local_route = session.scalar(select(GtfsRoute).where(GtfsRoute.dataset_id == aggregator_dataset.id).order_by(GtfsRoute.id))
assert local_route is not None
local_route.min_lon = 13.35
local_route.min_lat = 52.48
local_route.max_lon = 13.45
local_route.max_lat = 52.55
far_route = GtfsRoute(
dataset_id=aggregator_dataset.id,
route_id="far-same-key",
agency_id=local_route.agency_id,
short_name=local_route.short_name,
long_name=local_route.long_name,
route_type=local_route.route_type,
mode=local_route.mode,
route_scope=local_route.route_scope,
operator_name=local_route.operator_name,
route_key=local_route.route_key,
operator_key=local_route.operator_key,
min_lon=8.35,
min_lat=49.35,
max_lon=8.45,
max_lat=49.45,
)
session.add(far_route)
regional_source = Source(
name="Sample regional operator overlay",
kind="gtfs",
url="https://example.invalid/regional-overlay.zip",
country="DE",
license="sample",
priority="P1",
notes=_gtfs_review_note("operator"),
)
session.add(regional_source)
session.flush()
regional_dataset = Dataset(
source_id=regional_source.id,
kind="gtfs",
local_path="data/sample/regional-overlay.gtfs.zip",
sha256="e" * 64,
is_active=True,
status="imported",
)
session.add(regional_dataset)
session.flush()
session.add(
GtfsRoute(
dataset_id=regional_dataset.id,
route_id="regional-same-key",
agency_id=local_route.agency_id,
short_name=local_route.short_name,
long_name=local_route.long_name,
route_type=local_route.route_type,
mode=local_route.mode,
route_scope=local_route.route_scope,
operator_name="Regional Operator",
route_key=local_route.route_key,
operator_key="regional operator",
min_lon=13.36,
min_lat=52.49,
max_lon=13.44,
max_lat=52.54,
)
)
session.flush()
local_route_id = local_route.id
far_route_id = far_route.id
aggregator_dataset_id = aggregator_dataset.id
regional_dataset_id = regional_dataset.id
snapshot = client.get("/api/harmonization/gtfs/snapshot").json()
assert snapshot["summary"]["included_datasets"] == 2
assert snapshot["summary"]["shadowed_datasets"] == 0
assert snapshot["summary"]["route_shadowed_routes"] == 1
aggregator_item = next(item for item in snapshot["datasets"] if item["dataset_id"] == aggregator_dataset_id)
assert aggregator_item["role"] == "included"
assert aggregator_item["route_shadow_count"] == 1
assert aggregator_item["route_shadowed_by_dataset_ids"] == [regional_dataset_id]
workflow = client.get("/api/workbench/gtfs-workflow").json()
assert workflow["snapshot_diagnostics"]["needs_rebuild"] is True
assert workflow["snapshot_diagnostics"]["computed"]["route_shadowed_routes"] == 1
with session_scope() as session:
shadowed_route_ids = active_harmonized_gtfs_shadowed_route_ids(session)
assert local_route_id in shadowed_route_ids
assert far_route_id not in shadowed_route_ids
snapshot_job = client.post("/api/jobs/gtfs-harmonized-snapshot?activate=true").json()
assert run_worker_once(worker_id="test-worker")["processed"] == 1
snapshot_result = client.get(f"/api/jobs/{snapshot_job['id']}").json()["result"]["snapshot"]
assert snapshot_result["persisted"] is True
assert snapshot_result["summary"]["snapshot_route_rows"] >= 1
assert snapshot_result["summary"]["snapshot_route_included_rows"] >= 1
assert snapshot_result["summary"]["snapshot_route_shadowed_rows"] == 1
assert snapshot_result["summary"]["route_shadowed_routes"] == 1
workflow = client.get("/api/workbench/gtfs-workflow").json()
assert workflow["snapshot_diagnostics"]["needs_rebuild"] is False
assert workflow["snapshot_diagnostics"]["active"]["route_rows"] >= 1
assert workflow["snapshot_route_preview"]["summary"]["filtered_total"] == 1
assert workflow["snapshot_route_preview"]["routes"][0]["gtfs_route_id"] == local_route_id
with session_scope() as session:
session.execute(delete(GtfsHarmonizedSnapshotRoute).where(GtfsHarmonizedSnapshotRoute.snapshot_id == snapshot_result["id"]))
legacy_workflow = client.get("/api/workbench/gtfs-workflow").json()
assert legacy_workflow["snapshot_diagnostics"]["needs_rebuild"] is True
assert "route_ownership_not_materialized" in legacy_workflow["snapshot_diagnostics"]["reasons"]
assert legacy_workflow["snapshot_route_preview"]["routes"] == []
backfill_job = client.post("/api/jobs/gtfs-harmonized-snapshot?activate=true").json()
assert run_worker_once(worker_id="test-worker")["processed"] == 1
backfilled_snapshot = client.get(f"/api/jobs/{backfill_job['id']}").json()["result"]["snapshot"]
assert backfilled_snapshot["id"] == snapshot_result["id"]
assert backfilled_snapshot["summary"]["snapshot_route_rows"] >= 1
assert backfilled_snapshot["summary"]["route_shadowed_routes"] == 1
backfilled_workflow = client.get("/api/workbench/gtfs-workflow").json()
assert backfilled_workflow["snapshot_route_preview"]["summary"]["filtered_total"] == 1
assert backfilled_workflow["snapshot_route_preview"]["routes"][0]["gtfs_route_id"] == local_route_id
route_rows = client.get("/api/harmonization/gtfs/snapshot/routes?role=shadowed&limit=20").json()
assert route_rows["summary"]["filtered_total"] == 1
assert route_rows["routes"][0]["gtfs_route_id"] == local_route_id
assert route_rows["routes"][0]["shadowed_by_dataset_id"] == regional_dataset_id
filtered_route_rows = client.get(
"/api/harmonization/gtfs/snapshot/routes",
params={"role": "shadowed", "mode": route_rows["routes"][0]["mode"], "q": route_rows["routes"][0]["route_ref"], "limit": 20},
).json()
assert filtered_route_rows["summary"]["filtered_total"] == 1
assert filtered_route_rows["routes"][0]["gtfs_route_id"] == local_route_id
csv_response = client.get("/api/harmonization/gtfs/snapshot/routes.csv?role=shadowed&limit=20")
assert csv_response.status_code == 200
assert "text/csv" in csv_response.headers["content-type"]
assert "gtfs_route_id" in csv_response.text.splitlines()[0]
assert str(local_route_id) in csv_response.text
with session_scope() as session:
persisted_shadowed_route_ids = active_harmonized_gtfs_shadowed_route_ids(session)
assert local_route_id in persisted_shadowed_route_ids
assert far_route_id not in persisted_shadowed_route_ids
session.execute(delete(RouteMatch).where(RouteMatch.gtfs_route_id == local_route_id))
match_result = run_route_matching(session, batch_size=50)
assert match_result["shadowed_skipped"] == 1
assert session.scalar(select(RouteMatch).where(RouteMatch.gtfs_route_id == local_route_id)) is None
route_layer_result = rebuild_route_layer(session, commit_between_steps=False)
assert route_layer_result["shadowed_routes_skipped"] >= 1
dedup_job = client.post("/api/jobs/map-gtfs-review?limit=100").json()
assert run_worker_once(worker_id="test-worker")["processed"] == 1
dedup_result = client.get(f"/api/jobs/{dedup_job['id']}").json()["result"]["review_result"]
assert dedup_result["dedup_items"] > 0
dedup_items = client.get("/api/workbench/map-gtfs/review-items?queue=gtfs_deduplication&status=open&limit=100").json()["items"]
assert any(
item["gtfs_route"]["id"] == local_route_id
and item["metadata"]["snapshot"]["reason"] == "route_covered_by_higher_precedence_dataset"
for item in dedup_items
)
dedup_detail_id = next(item["id"] for item in dedup_items if item["gtfs_route"]["id"] == local_route_id)
dedup_detail = client.get(f"/api/workbench/map-gtfs/review-items/{dedup_detail_id}").json()
assert dedup_detail["metadata"]["shadowed_dataset"]["dataset_id"] == aggregator_dataset_id
assert dedup_detail["metadata"]["included_dataset"]["dataset_id"] == regional_dataset_id
filtered_dedup = client.get(
"/api/workbench/map-gtfs/review-items",
params={
"queue": "gtfs_deduplication",
"status": "open",
"severity": "warn",
"mode": route_rows["routes"][0]["mode"],
"q": route_rows["routes"][0]["route_ref"],
"limit": 20,
},
).json()
assert any(item["gtfs_route"]["id"] == local_route_id for item in filtered_dedup["items"])
def test_map_gtfs_workbench_review_queue_and_decision_replay_foundation():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
queued = client.post("/api/jobs/map-gtfs-review?limit=20").json()
assert queued["kind"] == "map_gtfs_review"
assert run_worker_once(worker_id="test-worker")["processed"] == 1
job = client.get(f"/api/jobs/{queued['id']}").json()
assert job["status"] == "completed"
assert job["result"]["review_result"]["route_items"] > 0
listed = client.get("/api/workbench/map-gtfs/review-items?queue=route_matching&status=open&limit=20").json()
assert listed["items"]
route_item = next(item for item in listed["items"] if item["route_match_id"])
assert route_item["queue"] == "route_matching"
assert route_item["gtfs_route"]
accepted = client.post(f"/api/matches/{route_item['route_match_id']}/accept").json()
assert accepted["status"] == "accepted"
summary = client.get("/api/workbench/map-gtfs/summary").json()
assert summary["active_decisions"] >= 1
resolved = client.get("/api/workbench/map-gtfs/review-items?queue=route_matching&status=resolved&limit=20").json()
assert any(item["fingerprint"] == route_item["fingerprint"] and item["decision_id"] for item in resolved["items"])
def _gtfs_review_note(authority_level: str) -> str:
return "[GTFS QA] " + json.dumps(
{
"status": "approved",
"note": "",
"can_import": "yes",
"can_derive": "yes",
"can_redistribute": "yes",
"requires_attribution": "yes",
"commercial_restrictions": "no",
"authority_level": authority_level,
},
sort_keys=True,
separators=(",", ":"),
)
def test_terminal_jobs_can_be_dismissed_from_default_view():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200
@@ -276,6 +631,73 @@ def test_worker_once_returns_idle_when_claim_is_busy(monkeypatch):
assert jobs_module.run_worker_once(worker_id="test-worker") == {"worker_id": "test-worker", "processed": 0}
def test_worker_control_endpoints(monkeypatch):
calls = []
handle = WorkerHandle(
index=0,
worker_id="server-worker-1",
pid=1234,
status="started",
pid_file=Path("data/workers/server-worker-1.pid"),
log_file=Path("data/workers/server-worker-1.log"),
started_by_server=True,
)
stopped = WorkerHandle(
index=0,
worker_id="server-worker-1",
pid=None,
status="stopped",
pid_file=Path("data/workers/server-worker-1.pid"),
log_file=Path("data/workers/server-worker-1.log"),
)
def fake_start_queue_workers(*, force=False):
calls.append(("start", force))
return [handle]
def fake_stop_configured_queue_workers():
calls.append(("stop", None))
return [stopped]
def fake_restart_queue_workers():
calls.append(("restart", None))
return {"stopped": [stopped], "started": [handle]}
monkeypatch.setattr(main_module, "start_queue_workers", fake_start_queue_workers)
monkeypatch.setattr(main_module, "stop_configured_queue_workers", fake_stop_configured_queue_workers)
monkeypatch.setattr(main_module, "restart_queue_workers", fake_restart_queue_workers)
monkeypatch.setattr(
main_module,
"queue_worker_status",
lambda: [
{
"index": 0,
"worker_id": "server-worker-1",
"pid": 1234,
"running": True,
"stale": False,
"status": "running",
"pid_file": "data/workers/server-worker-1.pid",
"log_file": "data/workers/server-worker-1.log",
}
],
)
client = TestClient(app)
started = client.post("/api/workers/start").json()
assert ("start", True) in calls
assert started["operations"]["started"][0]["status"] == "started"
assert started["workers"][0]["running"] is True
stopped_response = client.post("/api/workers/stop").json()
assert ("stop", None) in calls
assert stopped_response["operations"]["stopped"][0]["status"] == "stopped"
restarted = client.post("/api/workers/restart").json()
assert ("restart", None) in calls
assert restarted["operations"]["started"][0]["worker_id"] == "server-worker-1"
def test_running_job_can_be_stopped_while_write_lock_is_held():
client = TestClient(app)
assert client.post("/api/sample/reset").status_code == 200

81
tests/test_feed_discovery.py
View File

@@ -5,8 +5,11 @@ import csv
from app import feed_discovery
from app.feed_discovery import (
FeedCandidate,
apply_source_hierarchy,
build_gtfs_discovery_manifests,
enrich_ptna_candidate_from_details,
parse_opendata_oepnv_dataset_links,
parse_opendata_oepnv_detail_page,
parse_ptna_country_page,
parse_ptna_detail_fields,
select_test_run_candidates,
@@ -105,6 +108,7 @@ def test_build_gtfs_discovery_manifests_from_stubbed_sources(tmp_path, monkeypat
monkeypatch.setattr(feed_discovery, "fetch_mobility_database_candidates", lambda **_: mobility)
monkeypatch.setattr(feed_discovery, "fetch_mobility_acceptance_candidates", lambda **_: [])
monkeypatch.setattr(feed_discovery, "fetch_ptna_candidates", lambda **_: ptna)
monkeypatch.setattr(feed_discovery, "fetch_opendata_oepnv_candidates", lambda **_: [])
monkeypatch.setattr(feed_discovery, "load_curated_ingestable_seed", lambda **_: curated)
report = build_gtfs_discovery_manifests(output_dir=tmp_path, countries=["DE", "CH"], test_limit=10)
@@ -116,6 +120,83 @@ def test_build_gtfs_discovery_manifests_from_stubbed_sources(tmp_path, monkeypat
assert "ptna" in next(row for row in ingestable_rows if row["url"] == "https://example.test/rnv.zip")["source_basis"]
def test_parse_opendata_oepnv_catalog_and_details():
catalog_html = """
<a href="/ht/de/organisation/verkehrsunternehmen/rnv/startseite?tx_vrrkit_view%5Baction%5D=details&amp;tx_vrrkit_view%5Bcontroller%5D=View&amp;tx_vrrkit_view%5Bdataset_name%5D=soll-fahrplandaten-rnv">
<span itemprop="headline">Soll-Fahrplandaten (GTFS - General Transit Feed Specification)</span>
</a>
"""
links = parse_opendata_oepnv_dataset_links(catalog_html, "https://www.opendata-oepnv.de/ht/de/datensaetze")
assert len(links) == 1
assert links[0].slug == "soll-fahrplandaten-rnv"
assert links[0].title.startswith("Soll-Fahrplandaten")
detail_html = """
<div class="element-header">Soll-Fahrplandaten (GTFS - General Transit Feed Specification)</div>
<p>In dieser Ressource hinterlegen wir stets den aktuellsten GTFS-Datensatz.</p>
<a href="https://gtfs-sandbox-dds.rnv-online.de/latest/gtfs.zip">Download [ZIP]</a>
<td id="dodp_license"><a href="/?id=20">Creative Commons</a></td>
"""
candidate = parse_opendata_oepnv_detail_page(
detail_html,
detail_url=links[0].url,
slug=links[0].slug,
title_hint=links[0].title,
)
assert candidate.discovery_source == "opendata_oepnv"
assert candidate.provider == "Rhein-Neckar-Verkehr"
assert candidate.selected_url == "https://gtfs-sandbox-dds.rnv-online.de/latest/gtfs.zip"
assert candidate.data_type == "gtfs"
assert candidate.authority_level == "operator_feed"
assert candidate.import_policy == "qa_or_override_not_additive"
def test_opendata_oepnv_registration_required_is_not_direct_ingestable():
detail_html = """
<div class="element-header">Deutschlandweite Sollfahrplandaten (GTFS)</div>
<p>This Download is only available for registered Users.</p>
<td id="dodp_license"><a href="http://www.opendefinition.org/licenses/cc-by">Creative Commons Attribution</a></td>
"""
candidate = parse_opendata_oepnv_detail_page(
detail_html,
detail_url="https://www.opendata-oepnv.de/ht/de/organisation/delfi/startseite?tx_vrrkit_view%5Bdataset_name%5D=deutschlandweite-sollfahrplandaten-gtfs",
slug="deutschlandweite-sollfahrplandaten-gtfs",
)
assert candidate.status == "registration_required"
assert candidate.selected_url == ""
assert candidate.authority_level == "national_official"
assert candidate.overlap_role == "review_baseline"
assert candidate.import_policy == "review_before_import"
def test_source_hierarchy_marks_mirrors_and_regional_feeds():
mirror = FeedCandidate(
discovery_source="curated_seed",
country="DE",
provider="DE generated national",
selected_url="https://scraped.data.public-transport.earth/de/gtfs.zip",
source_basis="European transport feeds mirror",
)
regional = FeedCandidate(
discovery_source="mobility_database",
country="DE",
provider="Verkehrsverbund Berlin-Brandenburg",
selected_url="https://www.vbb.de/vbbgtfs",
is_official="True",
)
apply_source_hierarchy(mirror)
apply_source_hierarchy(regional)
assert mirror.authority_level == "mirror"
assert mirror.import_policy == "bootstrap_only"
assert regional.authority_level == "regional_authority"
assert regional.import_policy == "qa_or_override_not_additive"
def test_select_test_run_candidates_keeps_overlapping_german_feeds():
candidates = [
FeedCandidate(

329
tests/test_gtfs_import.py
View File

@@ -5,10 +5,24 @@ import zipfile
from sqlalchemy import func, select
from app.data_management import delete_dataset
from app.db import reset_db, session_scope
from app.gtfs_storage import sidecar_path, stop_time_count, stop_times_by_trip
from app.journey import find_journeys, search_scheduled_stops
from app.models import Dataset, GtfsCalendar, Source
from app.models import (
Dataset,
GtfsCalendar,
GtfsFeedDiffItem,
GtfsFeedDiffRun,
GtfsRoute,
GtfsRoutePatternLink,
GtfsTripRoutePatternLink,
MapGtfsDecision,
MapGtfsReviewItem,
RoutePattern,
Source,
)
from app.pipeline.gtfs import _copy_stage_row_payload
from app.pipeline.run import run_source
@@ -70,3 +84,316 @@ def test_gtfs_import_uses_staging_bulk_loader_and_reports_chunks(tmp_path, monke
assert "gtfs_file_chunk" in event_types
assert "gtfs_activation_sidecar_stop_times" in event_types
assert "gtfs_activation_completed" in event_types
def test_gtfs_calendar_stage_payload_casts_weekday_flags_to_bool():
columns = ["service_id", "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday", "start_date", "end_date"]
payload = _copy_stage_row_payload(
"gtfs_calendars",
5,
columns,
("Special", 0, 1, "0", "1", False, True, "", 20260401, 20261211),
)
assert payload == {
"dataset_id": 5,
"service_id": "Special",
"monday": False,
"tuesday": True,
"wednesday": False,
"thursday": True,
"friday": False,
"saturday": True,
"sunday": False,
"start_date": 20260401,
"end_date": 20261211,
}
def test_gtfs_reimports_same_hash_when_stop_time_limit_changes(tmp_path, monkeypatch):
reset_db()
gtfs_path = tmp_path / "limited.gtfs.zip"
with zipfile.ZipFile(gtfs_path, "w") as zf:
zf.writestr("agency.txt", "agency_id,agency_name,agency_url,agency_timezone\nA,Agency,https://example.invalid,Europe/Berlin\n")
zf.writestr("stops.txt", "stop_id,stop_name,stop_lat,stop_lon\nA,Alpha,52.0,13.0\nB,Beta,52.1,13.1\nC,Gamma,52.2,13.2\n")
zf.writestr("routes.txt", "route_id,agency_id,route_short_name,route_long_name,route_type\nR,A,R1,Alpha - Gamma,3\n")
zf.writestr("trips.txt", "route_id,service_id,trip_id\nR,daily,t1\n")
zf.writestr("calendar.txt", "service_id,monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date\ndaily,1,1,1,1,1,1,1,20260101,20261231\n")
zf.writestr(
"stop_times.txt",
"\n".join(
[
"trip_id,arrival_time,departure_time,stop_id,stop_sequence",
"t1,08:00:00,08:00:00,A,1",
"t1,08:05:00,08:05:00,B,2",
"t1,08:10:00,08:10:00,C,3",
]
)
+ "\n",
)
with session_scope() as session:
source = Source(name="Limited GTFS", kind="gtfs", url=str(gtfs_path))
session.add(source)
session.flush()
monkeypatch.setattr("app.pipeline.gtfs.settings.gtfs_stop_times_import_limit", 2)
first_dataset = run_source(session, source)
first_id = first_dataset.id
assert stop_time_count(session, first_id) == 2
monkeypatch.setattr("app.pipeline.gtfs.settings.gtfs_stop_times_import_limit", 0)
second_dataset = run_source(session, source)
assert second_dataset.id != first_id
assert stop_time_count(session, second_dataset.id) == 3
def test_gtfs_reimport_stores_update_diff_before_replacing_old_dataset(tmp_path):
reset_db()
gtfs_path = tmp_path / "updating.gtfs.zip"
_write_update_diff_gtfs(gtfs_path, route_long_name="Alpha - Beta", beta_name="Beta")
with session_scope() as session:
source = Source(name="Updating GTFS", kind="gtfs", url=str(gtfs_path))
session.add(source)
session.flush()
first_dataset = run_source(session, source)
first_id = first_dataset.id
_write_update_diff_gtfs(
gtfs_path,
route_long_name="Alpha - Beta revised",
beta_name="Beta moved",
include_extra_stop=True,
include_extra_route=True,
)
with session_scope() as session:
source = session.scalar(select(Source).where(Source.name == "Updating GTFS"))
second_dataset = run_source(session, source)
second_id = second_dataset.id
diff_run = session.scalar(select(GtfsFeedDiffRun).where(GtfsFeedDiffRun.new_dataset_id == second_id))
assert diff_run is not None
assert diff_run.previous_dataset_id == first_id
assert session.get(Dataset, first_id) is None
summary = json.loads(diff_run.summary_json)
assert summary["routes"]["changed"] == 1
assert summary["routes"]["added"] == 1
assert summary["stops"]["changed"] == 1
assert summary["stops"]["added"] == 1
items = {
(item.item_type, item.object_key, item.change_type)
for item in session.scalars(select(GtfsFeedDiffItem).where(GtfsFeedDiffItem.diff_run_id == diff_run.id))
}
assert ("route", "R1", "changed") in items
assert ("route", "R2", "added") in items
assert ("stop", "B", "changed") in items
assert ("stop", "C", "added") in items
def test_gtfs_reimport_replays_unchanged_decisions_and_queues_changed_items(tmp_path):
reset_db()
gtfs_path = tmp_path / "decision-replay.gtfs.zip"
_write_update_diff_gtfs(gtfs_path, route_long_name="Alpha - Beta", beta_name="Beta", include_stable_route=True)
with session_scope() as session:
source = Source(name="Decision replay GTFS", kind="gtfs", url=str(gtfs_path))
session.add(source)
session.flush()
first_dataset = run_source(session, source)
first_id = first_dataset.id
stable_route = session.scalar(
select(GtfsRoute).where(GtfsRoute.dataset_id == first_id, GtfsRoute.route_id == "R_KEEP")
)
assert stable_route is not None
selector = {
"gtfs": {
"source_id": source.id,
"dataset_id": first_id,
"route_id": stable_route.route_id,
"route_key": stable_route.route_key,
"ref": stable_route.short_name,
"mode": stable_route.mode,
}
}
session.add(
MapGtfsDecision(
decision_key="test-stable-route-decision",
decision_type="route_osm_relation",
status="accepted",
active=True,
source_id=source.id,
gtfs_dataset_id=first_id,
gtfs_route_id=stable_route.id,
confidence=0.91,
selector_json=json.dumps(selector, sort_keys=True),
action_json=json.dumps({"status": "accepted", "osm": {"osm_id": 12345}}, sort_keys=True),
note="keep stable route",
reviewer="test",
metadata_json=json.dumps({"manual": True}, sort_keys=True),
)
)
_write_update_diff_gtfs(
gtfs_path,
route_long_name="Alpha - Beta revised",
beta_name="Beta moved",
include_extra_stop=True,
include_extra_route=True,
include_stable_route=True,
)
with session_scope() as session:
source = session.scalar(select(Source).where(Source.name == "Decision replay GTFS"))
second_dataset = run_source(session, source)
second_id = second_dataset.id
stable_route = session.scalar(
select(GtfsRoute).where(GtfsRoute.dataset_id == second_id, GtfsRoute.route_id == "R_KEEP")
)
assert stable_route is not None
replayed = session.scalar(
select(MapGtfsDecision).where(MapGtfsDecision.gtfs_dataset_id == second_id, MapGtfsDecision.gtfs_route_id == stable_route.id)
)
assert replayed is not None
assert replayed.reviewer == "decision_replay"
assert replayed.confidence == 0.91
replay_metadata = json.loads(replayed.metadata_json or "{}")
assert replay_metadata["manual"] is True
assert replay_metadata["replay"]["previous_dataset_id"] == first_id
assert replay_metadata["replay"]["new_dataset_id"] == second_id
assert session.scalar(select(func.count()).select_from(MapGtfsDecision).where(MapGtfsDecision.gtfs_dataset_id == first_id)) == 0
review_items = session.scalars(select(MapGtfsReviewItem).where(MapGtfsReviewItem.queue == "gtfs_update_diff")).all()
review_keys = {
(item.item_type, (json.loads(item.metadata_json or "{}")).get("object_key"), (json.loads(item.metadata_json or "{}")).get("change_type"))
for item in review_items
}
assert ("gtfs_route_update", "R1", "changed") in review_keys
assert ("gtfs_stop_update", "B", "changed") in review_keys
def test_gtfs_dataset_delete_removes_route_pattern_links_by_pattern_id():
reset_db()
with session_scope() as session:
source = Source(name="Delete GTFS", kind="gtfs", url="./delete.zip")
other_source = Source(name="Other GTFS", kind="gtfs", url="./other.zip")
session.add_all([source, other_source])
session.flush()
dataset = Dataset(source_id=source.id, kind="gtfs", local_path="./delete.zip", sha256="a" * 64, is_active=False, status="imported")
other_dataset = Dataset(source_id=other_source.id, kind="gtfs", local_path="./other.zip", sha256="b" * 64, is_active=True, status="imported")
session.add_all([dataset, other_dataset])
session.flush()
route = GtfsRoute(
dataset_id=dataset.id,
route_id="R1",
short_name="R1",
long_name="Delete route",
route_type=3,
mode="bus",
)
session.add(route)
session.flush()
pattern = RoutePattern(
pattern_key="gtfs:delete:R1:S1",
route_ref="R1",
route_name="Delete route",
mode="bus",
source_kind="gtfs_proposed",
status="active",
gtfs_route_id=route.id,
gtfs_shape_id="S1",
geometry_geojson='{"type":"LineString","coordinates":[[13.0,52.0],[13.1,52.1]]}',
)
session.add(pattern)
session.flush()
session.add(
GtfsRoutePatternLink(
dataset_id=other_dataset.id,
gtfs_route_id=route.id,
route_id="R1",
shape_id="S1",
route_pattern_id=pattern.id,
confidence=1,
status="linked",
source_kind="gtfs_proposed",
)
)
session.add(
GtfsTripRoutePatternLink(
dataset_id=other_dataset.id,
trip_id="T1",
route_id="R1",
shape_id="S1",
route_pattern_id=pattern.id,
source_kind="gtfs_proposed",
confidence=1,
status="linked",
)
)
session.flush()
result = delete_dataset(session, dataset.id)
assert result["deleted"] is True
assert session.scalar(select(func.count()).select_from(RoutePattern).where(RoutePattern.id == pattern.id)) == 0
assert session.scalar(select(func.count()).select_from(GtfsRoutePatternLink)) == 0
assert session.scalar(select(func.count()).select_from(GtfsTripRoutePatternLink)) == 0
def _write_update_diff_gtfs(
gtfs_path,
*,
route_long_name: str,
beta_name: str,
include_extra_stop: bool = False,
include_extra_route: bool = False,
include_stable_route: bool = False,
) -> None:
stops = [
"A,Alpha,52.0,13.0",
f"B,{beta_name},52.1,13.1",
]
if include_extra_stop:
stops.append("C,Gamma,52.2,13.2")
routes = [
f"R1,A,R1,{route_long_name},3",
]
if include_extra_route:
routes.append("R2,A,R2,Alpha - Gamma,3")
if include_stable_route:
routes.append("R_KEEP,A,RK,Stable route,3")
trips = [
"R1,daily,t1",
]
if include_extra_route:
trips.append("R2,daily,t2")
if include_stable_route:
trips.append("R_KEEP,daily,t_keep")
stop_times = [
"t1,08:00:00,08:00:00,A,1",
"t1,08:10:00,08:10:00,B,2",
]
if include_extra_route:
stop_times.extend(
[
"t2,09:00:00,09:00:00,A,1",
"t2,09:15:00,09:15:00,C,2",
]
)
if include_stable_route:
stop_times.extend(
[
"t_keep,10:00:00,10:00:00,A,1",
"t_keep,10:12:00,10:12:00,B,2",
]
)
with zipfile.ZipFile(gtfs_path, "w") as zf:
zf.writestr("agency.txt", "agency_id,agency_name,agency_url,agency_timezone\nA,Agency,https://example.invalid,Europe/Berlin\n")
zf.writestr("stops.txt", "stop_id,stop_name,stop_lat,stop_lon\n" + "\n".join(stops) + "\n")
zf.writestr("routes.txt", "route_id,agency_id,route_short_name,route_long_name,route_type\n" + "\n".join(routes) + "\n")
zf.writestr("trips.txt", "route_id,service_id,trip_id\n" + "\n".join(trips) + "\n")
zf.writestr("calendar.txt", "service_id,monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date\ndaily,1,1,1,1,1,1,1,20260101,20261231\n")
zf.writestr("stop_times.txt", "trip_id,arrival_time,departure_time,stop_id,stop_sequence\n" + "\n".join(stop_times) + "\n")

147
tests/test_journey_search.py Normal file
View File

@@ -0,0 +1,147 @@
from __future__ import annotations
from app.journey import _filter_dominated_journeys
from app.journey_search import _rank_journeys, _select_diverse_journeys
def test_progressive_selection_keeps_fast_and_low_walk_transit_options():
fast_with_walk = _journey(
"fast",
arrival=30_000,
transfers=0,
legs=[
_leg("tram", "22", "A", "B"),
_leg("walk", "walk", "B", "Z", distance_m=1200),
],
)
slower_low_walk = _journey(
"low-walk",
arrival=31_200,
transfers=1,
legs=[
_leg("tram", "22", "A", "Hbf"),
_leg("bus", "34", "Hbf", "Y"),
_leg("walk", "walk", "Y", "Z", distance_m=120),
],
)
walk_only = _journey(
"walk-only",
arrival=30_900,
transfers=0,
legs=[_leg("walk", "walk", "A", "Z", distance_m=1800)],
)
ranked = _rank_journeys([slower_low_walk, walk_only, fast_with_walk], "recommended")
selected = _select_diverse_journeys(ranked, limit=3)
assert len(selected) <= 3
assert {journey["id"] for journey in selected} == {"fast", "low-walk", "walk-only"}
def test_progressive_selection_collapses_same_journey_from_duplicate_gtfs_feeds():
rnv = _journey(
"rnv",
arrival=29_820,
transfers=0,
legs=[
_leg("walk", "walk", "Jakob-Neu-Straße 2", "Eppelheim Rathaus", dataset_id=10, source_name=None),
_leg("tram", "RNV 22", "Eppelheim Rathaus", "Heidelberg Hbf", dataset_id=10, source_name="DE Verkehrsverbund Rhein-Neckar GTFS"),
],
)
delfi = _journey(
"delfi",
arrival=29_820,
transfers=0,
legs=[
_leg("walk", "walk", "Jakob-Neu-Straße 2", "Eppelheim Rathaus", dataset_id=4, source_name=None),
_leg("tram", "RNV 22", "Eppelheim Rathaus", "Heidelberg Hbf", dataset_id=4, source_name="DELFI / GTFS.de Germany", stop_count=8),
],
)
ranked = _rank_journeys([rnv, delfi], "recommended")
selected = _select_diverse_journeys(ranked, limit=3)
assert [journey["id"] for journey in selected] == ["delfi"]
def test_dominance_removes_only_slower_more_transfer_same_endpoint_options():
direct = _journey(
"direct",
arrival=30_000,
transfers=0,
legs=[
_leg("tram", "22", "A", "platform-a", canonical_to=500),
_leg("walk", "walk", "platform-a", "Z", distance_m=800),
],
)
faster_transfer = _journey(
"faster-transfer",
arrival=29_500,
transfers=1,
legs=[
_leg("bus", "34", "A", "X"),
_leg("tram", "5", "X", "platform-b", canonical_to=500),
_leg("walk", "walk", "platform-b", "Z", distance_m=800),
],
)
slower_transfer = _journey(
"slower-transfer",
arrival=31_000,
transfers=1,
legs=[
_leg("bus", "713", "A", "Y"),
_leg("tram", "22", "Y", "platform-c", canonical_to=500),
_leg("walk", "walk", "platform-c", "Z", distance_m=800),
],
)
filtered = _filter_dominated_journeys([slower_transfer, faster_transfer, direct])
assert {journey["id"] for journey in filtered} == {"direct", "faster-transfer"}
def _journey(journey_id: str, *, arrival: int, transfers: int, legs: list[dict]) -> dict:
departure = 28_800
return {
"id": journey_id,
"departure_seconds": departure,
"arrival_seconds": arrival,
"duration_seconds": arrival - departure,
"duration_minutes": (arrival - departure + 59) // 60,
"transfers": transfers,
"legs": legs,
}
def _leg(
mode: str,
route: str,
from_stop: str,
to_stop: str,
*,
distance_m: float = 0,
canonical_to: int | None = None,
dataset_id: int = 1,
source_name: str | None = "Test GTFS",
stop_count: int = 2,
) -> dict:
to_payload = {"stop_id": to_stop}
stops = [{"stop_id": from_stop}, {"stop_id": to_stop}]
if canonical_to is not None:
stops[-1]["canonical_stop"] = {"id": canonical_to}
return {
"dataset_id": dataset_id,
"source_name": source_name,
"mode": mode,
"route_id": route,
"route_ref": route,
"trip_id": f"{route}-trip" if mode != "walk" else None,
"from": {"stop_id": from_stop},
"to": to_payload,
"departure_time": "08:00:00",
"arrival_time": "08:10:00",
"distance_m": distance_m,
"stop_count": stop_count if mode != "walk" else None,
"intermediate_stop_count": max(0, stop_count - 2) if mode != "walk" else None,
"stops": stops,
}

74
tests/test_pipeline.py
View File

@@ -6,7 +6,7 @@ from shapely.geometry import LineString
from sqlalchemy import select
from app.db import init_db, session_scope
from app.models import GtfsRoute, OsmFeature, RouteMatch, RoutePattern
from app.models import Dataset, GtfsRoute, OsmFeature, RouteMatch, RoutePattern, Source
from app.osm_classification import infer_osm_route_scope
from app.pipeline.gtfs import _gtfs_mode
from app.pipeline.matcher import _build_osm_route_index, _candidate_osm_routes, route_match_scope, run_route_matching, score_route_pair
@@ -53,6 +53,47 @@ def test_route_matching_preserves_unchanged_match_rows():
assert after == before
def test_route_matching_can_target_single_gtfs_dataset():
init_db()
with session_scope() as session:
load_sample_project(session)
source = Source(
name="Second feed",
kind="gtfs",
url="./second-feed.zip",
country="DE",
)
session.add(source)
session.flush()
dataset = Dataset(
source_id=source.id,
kind="gtfs",
local_path="second-feed.zip",
sha256="second-feed",
is_active=True,
status="imported",
)
session.add(dataset)
session.flush()
route = GtfsRoute(
dataset_id=dataset.id,
route_id="second-only",
short_name="ZZ",
mode="bus",
min_lon=13.0,
min_lat=52.0,
max_lon=13.1,
max_lat=52.1,
)
session.add(route)
session.flush()
result = run_route_matching(session, gtfs_dataset_ids=[dataset.id])
assert result["routes"] == 1
assert session.scalar(select(RouteMatch).where(RouteMatch.gtfs_route_id == route.id)) is not None
def test_route_layer_reuses_unchanged_route_patterns():
init_db()
with session_scope() as session:
@@ -241,6 +282,37 @@ def test_common_short_ref_candidates_are_spatially_ranked():
candidates = _candidate_osm_routes(route, _build_osm_route_index([far, near]))
assert candidates[0].osm_id == "near"
assert "far" not in {candidate.osm_id for candidate in candidates}
def test_known_bbox_route_does_not_fallback_to_global_mode_candidates():
route = GtfsRoute(
route_id="bus-x-berlin",
short_name="X",
mode="bus",
min_lon=13.30,
min_lat=52.40,
max_lon=13.40,
max_lat=52.50,
route_key="x",
)
far = OsmFeature(
id=1,
osm_type="relation",
osm_id="far-bus",
kind="route",
mode="bus",
ref="9",
min_lon=7.0,
min_lat=50.0,
max_lon=7.1,
max_lat=50.1,
route_key="9",
)
candidates = _candidate_osm_routes(route, _build_osm_route_index([far]))
assert candidates == []
def test_exact_ref_far_away_is_not_promoted_without_spatial_or_geometry_evidence():

415
tests/test_route_layer.py
View File

@@ -6,8 +6,9 @@ from shapely.geometry import LineString, Point, shape
from sqlalchemy import select
from app.db import reset_db, session_scope
from app.journey import find_journeys, search_scheduled_stops
from app.journey import find_journeys, nearest_scheduled_stops, search_scheduled_stops
from app.models import (
CanonicalStop,
CanonicalStopLink,
Dataset,
GtfsCalendar,
@@ -33,6 +34,7 @@ from app.pipeline.route_layer import (
rebuild_route_layer,
)
from app.pipeline.utils import geometry_json_and_bbox, norm_ref
from app.routing import direct_route_between_points
def test_directional_candidate_selection_prefers_matching_osm_geometry_orientation():
@@ -245,6 +247,318 @@ def test_opposite_gtfs_shapes_share_osm_visual_route_and_reverse_journey_segment
assert tuple(coords[-1]) == (0.0, 0.0)
def test_exact_stop_token_does_not_expand_to_merged_canonical_stop():
reset_db()
with session_scope() as session:
source = Source(name="Merged Station GTFS", kind="gtfs", url="./merged.zip")
session.add(source)
session.flush()
dataset = Dataset(
source_id=source.id,
kind="gtfs",
local_path="./merged.zip",
sha256="merged",
is_active=True,
status="imported",
)
session.add(dataset)
session.flush()
session.add_all(
[
GtfsStop(dataset_id=dataset.id, stop_id="station_bus", name="Example Hbf", lat=52.0, lon=13.0000),
GtfsStop(dataset_id=dataset.id, stop_id="station_train", name="Example Hbf", lat=52.0001, lon=13.0001),
GtfsStop(dataset_id=dataset.id, stop_id="target", name="Target", lat=52.01, lon=13.01),
GtfsRoute(
dataset_id=dataset.id,
route_id="bus",
short_name="B",
route_type=3,
mode="bus",
route_key=norm_ref("B"),
),
GtfsRoute(
dataset_id=dataset.id,
route_id="train",
short_name="T",
route_type=2,
mode="train",
route_key=norm_ref("T"),
),
GtfsTrip(dataset_id=dataset.id, route_id="bus", trip_id="bus_trip", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="train", trip_id="train_trip", service_id="daily"),
_stop_time(dataset.id, "train_trip", "station_train", 1, "08:00:00", 28800),
_stop_time(dataset.id, "train_trip", "target", 2, "08:05:00", 29100),
_stop_time(dataset.id, "bus_trip", "station_bus", 1, "08:20:00", 30000),
_stop_time(dataset.id, "bus_trip", "target", 2, "08:35:00", 30900),
]
)
session.flush()
rebuild_route_layer(session)
grouped = search_scheduled_stops(session, "Example Hbf")
assert grouped
assert grouped[0]["grouped"] is True
exact_hits = nearest_scheduled_stops(session, lat=52.0, lon=13.0, radius_m=30, grouped=False)
assert exact_hits[0]["id"] == f"stop:{dataset.id}:station_bus"
assert exact_hits[0]["grouped"] is False
journey = find_journeys(
session,
from_stop_id=f"stop:{dataset.id}:station_bus",
to_stop_id="target",
departure="07:55",
max_transfers=0,
limit=1,
)
assert journey["journeys"]
legs = journey["journeys"][0]["legs"]
assert legs[0]["from"]["stop_id"] == "station_bus"
assert legs[0]["mode"] in {"walk", "bus"}
if legs[0]["mode"] == "walk":
assert legs[0]["to"]["stop_id"] == "station_train"
assert legs[1]["trip_id"] == "train_trip"
else:
assert legs[0]["trip_id"] == "bus_trip"
def test_exact_stop_can_walk_to_adjacent_concrete_stop_for_transit():
reset_db()
with session_scope() as session:
source = Source(name="Adjacent Stops GTFS", kind="gtfs", url="./adjacent.zip")
session.add(source)
session.flush()
dataset = Dataset(
source_id=source.id,
kind="gtfs",
local_path="./adjacent.zip",
sha256="adjacent",
is_active=True,
status="imported",
)
session.add(dataset)
session.flush()
session.add_all(
[
GtfsStop(dataset_id=dataset.id, stop_id="station_bus", name="Example Hbf", lat=52.0, lon=13.0000),
GtfsStop(dataset_id=dataset.id, stop_id="station_train", name="Example Hbf", lat=52.0001, lon=13.0001),
GtfsStop(dataset_id=dataset.id, stop_id="target", name="Target", lat=52.01, lon=13.01),
GtfsStop(dataset_id=dataset.id, stop_id="elsewhere", name="Elsewhere", lat=52.02, lon=13.02),
GtfsRoute(
dataset_id=dataset.id,
route_id="bus",
short_name="B",
route_type=3,
mode="bus",
route_key=norm_ref("B"),
),
GtfsRoute(
dataset_id=dataset.id,
route_id="train",
short_name="T",
route_type=2,
mode="train",
route_key=norm_ref("T"),
),
GtfsTrip(dataset_id=dataset.id, route_id="bus", trip_id="bus_unrelated", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="train", trip_id="train_trip", service_id="daily"),
_stop_time(dataset.id, "bus_unrelated", "station_bus", 1, "09:00:00", 32400),
_stop_time(dataset.id, "bus_unrelated", "elsewhere", 2, "09:20:00", 33600),
_stop_time(dataset.id, "train_trip", "station_train", 1, "08:05:00", 29100),
_stop_time(dataset.id, "train_trip", "target", 2, "08:15:00", 29700),
]
)
session.flush()
rebuild_route_layer(session)
journey = find_journeys(
session,
from_stop_id=f"stop:{dataset.id}:station_bus",
to_stop_id="target",
departure="08:00",
max_transfers=0,
limit=2,
)
assert journey["journeys"]
legs = journey["journeys"][0]["legs"]
assert [leg["mode"] for leg in legs] == ["walk", "train"]
assert legs[0]["from"]["stop_id"] == "station_bus"
assert legs[0]["to"]["stop_id"] == "station_train"
assert legs[0]["arrival_time"] == legs[1]["departure_time"]
assert legs[0]["departure_time"] != "08:00:00"
assert legs[1]["trip_id"] == "train_trip"
walk_points = [
feature
for feature in journey["journeys"][0]["features"]["features"]
if feature["geometry"]["type"] == "Point" and feature["properties"].get("mode") == "walk"
]
assert [point["properties"]["stop_id"] for point in walk_points] == ["station_bus", "station_train"]
def test_walk_only_journey_reuses_routed_geometry(monkeypatch):
reset_db()
with session_scope() as session:
source = Source(name="Walk Geometry GTFS", kind="gtfs", url="./walk.zip")
session.add(source)
session.flush()
dataset = Dataset(
source_id=source.id,
kind="gtfs",
local_path="./walk.zip",
sha256="walk",
is_active=True,
status="imported",
)
session.add(dataset)
session.flush()
origin = GtfsStop(dataset_id=dataset.id, stop_id="origin", name="Origin", lat=52.0, lon=13.0)
target = GtfsStop(dataset_id=dataset.id, stop_id="target", name="Target", lat=52.001, lon=13.002)
session.add_all(
[
origin,
target,
GtfsRoute(dataset_id=dataset.id, route_id="marker", short_name="M", route_type=3, mode="bus"),
GtfsTrip(dataset_id=dataset.id, route_id="marker", trip_id="origin_marker", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="marker", trip_id="target_marker", service_id="daily"),
]
)
session.flush()
session.add_all(
[
_stop_time(dataset.id, "origin_marker", "origin", 1, "07:00:00", 25200),
_stop_time(dataset.id, "target_marker", "target", 1, "07:00:00", 25200),
]
)
origin_place = CanonicalStop(
stop_key=f"gtfs:{dataset.id}:origin",
name="Origin",
normalized_name="origin",
lat=52.0,
lon=13.0,
mode="bus",
)
target_place = CanonicalStop(
stop_key=f"gtfs:{dataset.id}:target",
name="Target",
normalized_name="target",
lat=52.001,
lon=13.002,
mode="bus",
)
session.add_all([origin_place, target_place])
session.flush()
session.add_all(
[
CanonicalStopLink(
canonical_stop_id=origin_place.id,
layer="timetable",
object_type="gtfs_stop",
dataset_id=dataset.id,
object_id=origin.id,
external_id="origin",
role="primary",
confidence=1.0,
),
CanonicalStopLink(
canonical_stop_id=target_place.id,
layer="timetable",
object_type="gtfs_stop",
dataset_id=dataset.id,
object_id=target.id,
external_id="target",
role="primary",
confidence=1.0,
),
]
)
session.flush()
routed_coordinates = [[13.0, 52.0], [13.001, 52.0007], [13.002, 52.001]]
def fake_route_between_points(*args, **kwargs):
assert kwargs["mode"] == "walk"
return {
"distance_m": 240.0,
"duration_seconds": 180.2,
"features": {
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"geometry": {"type": "LineString", "coordinates": routed_coordinates},
"properties": {"mode": "walk"},
}
],
},
}
monkeypatch.setattr("app.journey.route_between_points", fake_route_between_points)
result = find_journeys(
session,
from_stop_id="origin",
to_stop_id="target",
departure="08:00",
max_transfers=0,
limit=3,
)
walk_journey = next(
journey
for journey in result["journeys"]
if [leg["mode"] for leg in journey["legs"]] == ["walk"]
)
walk_feature = next(
feature
for feature in walk_journey["features"]["features"]
if feature["geometry"]["type"] == "LineString"
)
assert walk_feature["geometry"]["coordinates"] == routed_coordinates
assert walk_feature["properties"]["geometry_source"] == "routing_layer:walk"
assert walk_journey["legs"][0]["distance_m"] == 240.0
assert walk_journey["legs"][0]["duration_seconds"] == 181
def test_direct_point_routing_supports_bike_mode_and_aliases():
reset_db()
with session_scope() as session:
walk = direct_route_between_points(
session,
from_lon=13.0,
from_lat=52.0,
to_lon=13.002,
to_lat=52.001,
mode="walk",
)
bike = direct_route_between_points(
session,
from_lon=13.0,
from_lat=52.0,
to_lon=13.002,
to_lat=52.001,
mode="bike",
)
bicycle = direct_route_between_points(
session,
from_lon=13.0,
from_lat=52.0,
to_lon=13.002,
to_lat=52.001,
mode="bicycle",
)
assert bike["mode"] == "bike"
assert bicycle["mode"] == "bike"
assert bike["duration_seconds"] < walk["duration_seconds"]
bike_feature = bike["features"]["features"][0]
assert bike_feature["properties"]["mode"] == "bike"
def test_journey_geometry_rejects_remote_route_pattern_and_uses_trip_shape():
reset_db()
with session_scope() as session:
@@ -959,6 +1273,105 @@ def test_journey_service_date_filters_duplicate_clock_time_trips():
assert [journey["legs"][0]["trip_id"] for journey in monday["journeys"]] == ["weekday_trip"]
def test_direct_nearby_destination_walk_dominates_same_endpoint_transfer():
reset_db()
with session_scope() as session:
source = Source(name="Nearby Destination GTFS", kind="gtfs", url="./nearby.zip")
session.add(source)
session.flush()
dataset = Dataset(
source_id=source.id,
kind="gtfs",
local_path="./nearby.zip",
sha256="nearby",
is_active=True,
status="imported",
)
session.add(dataset)
session.flush()
session.add_all(
[
GtfsStop(dataset_id=dataset.id, stop_id="origin", name="Eppelheim Rathaus", lat=49.4010, lon=8.6330),
GtfsStop(dataset_id=dataset.id, stop_id="market", name="Pfaffengrund Marktstrasse", lat=49.4050, lon=8.6410),
GtfsStop(dataset_id=dataset.id, stop_id="henkel", name="Pfaffengrund Henkel-Teroson", lat=49.4090, lon=8.6500),
GtfsStop(dataset_id=dataset.id, stop_id="hbf", name="Heidelberg Hbf", lat=49.4030, lon=8.6550),
GtfsStop(dataset_id=dataset.id, stop_id="target", name="Heidelberg Ochsenkopf", lat=49.4108, lon=8.6520),
GtfsStop(dataset_id=dataset.id, stop_id="marker", name="Target Marker", lat=49.4120, lon=8.6540),
GtfsRoute(
dataset_id=dataset.id,
route_id="rnv22",
short_name="RNV 22",
route_type=0,
mode="tram",
route_key=norm_ref("RNV 22"),
),
GtfsRoute(
dataset_id=dataset.id,
route_id="marker",
short_name="M",
route_type=3,
mode="bus",
route_key=norm_ref("M"),
),
GtfsRoute(
dataset_id=dataset.id,
route_id="rnv5",
short_name="RNV 5",
route_type=0,
mode="tram",
route_key=norm_ref("RNV 5"),
),
GtfsTrip(dataset_id=dataset.id, route_id="rnv22", trip_id="through_tram", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="rnv22", trip_id="later_tram", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="marker", trip_id="target_marker", service_id="daily"),
GtfsTrip(dataset_id=dataset.id, route_id="rnv5", trip_id="exact_destination_tram", service_id="daily"),
_stop_time(dataset.id, "through_tram", "origin", 1, "08:04:00", 29040),
_stop_time(dataset.id, "through_tram", "market", 2, "08:08:00", 29280),
_stop_time(dataset.id, "through_tram", "henkel", 3, "08:09:00", 29340),
_stop_time(dataset.id, "through_tram", "hbf", 4, "08:17:00", 29820),
_stop_time(dataset.id, "later_tram", "market", 1, "08:18:00", 29880),
_stop_time(dataset.id, "later_tram", "henkel", 2, "08:19:00", 29940),
_stop_time(dataset.id, "target_marker", "marker", 1, "07:00:00", 25200),
_stop_time(dataset.id, "target_marker", "target", 2, "07:01:00", 25260),
_stop_time(dataset.id, "exact_destination_tram", "hbf", 1, "08:28:00", 30480),
_stop_time(dataset.id, "exact_destination_tram", "target", 2, "08:31:00", 30660),
]
)
session.flush()
rebuild_route_layer(session)
result = find_journeys(
session,
from_stop_id="origin",
to_stop_id="target",
departure="08:00",
max_transfers=1,
limit=5,
)
assert result["journeys"]
direct_with_final_walk = [
journey
for journey in result["journeys"]
if [leg.get("trip_id") for leg in journey["legs"] if leg.get("mode") != "walk"] == ["through_tram"]
and journey["legs"][-1]["mode"] == "walk"
and str(journey["legs"][-1]["from"]["stop_id"]).startswith("canonical:")
and journey["legs"][-1]["from"]["name"] == "Pfaffengrund Henkel-Teroson"
and journey["transfers"] == 0
]
assert direct_with_final_walk
assert all(
[leg.get("trip_id") for leg in journey["legs"] if leg.get("mode") != "walk"]
!= ["through_tram", "later_tram"]
for journey in result["journeys"]
)
assert any(
[leg.get("trip_id") for leg in journey["legs"] if leg.get("mode") != "walk"]
== ["through_tram", "exact_destination_tram"]
for journey in result["journeys"]
)
def _route_candidate(
feature_id: int,
osm_id: str,