add-ideas/pikebacker
1
0
Fork 0
Code Issues 16 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
pikebacker/services/data-pipeline/README.md
Albrecht Degering abed21be21 Alpha stage commit
2026-07-02 21:04:05 +02:00

1.1 KiB
Raw Permalink Blame History

Data Pipeline

Purpose: ingest and normalize map, POI, elevation, and rule data.

MVP pipeline steps:

  1. Load OSM fixture or small regional extract.
  2. Extract bikepacking POIs.
  3. Normalize categories.
  4. Store source, source_id, tags, and confidence.
  5. Prepare route-corridor query indexes.

Production steps:

  1. Download Geofabrik/planet extracts.
  2. Build routing graph for selected engine.
  3. Import POIs into PostGIS/search index.
  4. Enrich segments with elevation.
  5. Import protected areas and rules.
  6. Generate offline pack assets.

Do not use public Overpass instances for production-scale queries.

Implemented skeleton

The package includes a small OSM-like fixture and a normalizer:

npm -w @pikebacker/data-pipeline run normalize:sample

The normalizer maps OSM tags into bikepacking POI categories (water, food, sleep, repair, bailout, charging, emergency) and preserves source, tags, opening_hours, confidence, and OpenStreetMap attribution hooks. Production ingestion should write the normalized records into PostGIS using schemas/database.sql.