33 lines
1.1 KiB
Markdown
33 lines
1.1 KiB
Markdown
# Data Pipeline
|
|
|
|
Purpose: ingest and normalize map, POI, elevation, and rule data.
|
|
|
|
MVP pipeline steps:
|
|
|
|
1. Load OSM fixture or small regional extract.
|
|
2. Extract bikepacking POIs.
|
|
3. Normalize categories.
|
|
4. Store source, source_id, tags, and confidence.
|
|
5. Prepare route-corridor query indexes.
|
|
|
|
Production steps:
|
|
|
|
1. Download Geofabrik/planet extracts.
|
|
2. Build routing graph for selected engine.
|
|
3. Import POIs into PostGIS/search index.
|
|
4. Enrich segments with elevation.
|
|
5. Import protected areas and rules.
|
|
6. Generate offline pack assets.
|
|
|
|
Do not use public Overpass instances for production-scale queries.
|
|
|
|
## Implemented skeleton
|
|
|
|
The package includes a small OSM-like fixture and a normalizer:
|
|
|
|
```bash
|
|
npm -w @pikebacker/data-pipeline run normalize:sample
|
|
```
|
|
|
|
The normalizer maps OSM tags into bikepacking POI categories (`water`, `food`, `sleep`, `repair`, `bailout`, `charging`, `emergency`) and preserves `source`, `tags`, `opening_hours`, confidence, and OpenStreetMap attribution hooks. Production ingestion should write the normalized records into PostGIS using `schemas/database.sql`.
|