Alpha stage commit
This commit is contained in:
32
services/data-pipeline/README.md
Normal file
32
services/data-pipeline/README.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# Data Pipeline
|
||||
|
||||
Purpose: ingest and normalize map, POI, elevation, and rule data.
|
||||
|
||||
MVP pipeline steps:
|
||||
|
||||
1. Load OSM fixture or small regional extract.
|
||||
2. Extract bikepacking POIs.
|
||||
3. Normalize categories.
|
||||
4. Store source, source_id, tags, and confidence.
|
||||
5. Prepare route-corridor query indexes.
|
||||
|
||||
Production steps:
|
||||
|
||||
1. Download Geofabrik/planet extracts.
|
||||
2. Build routing graph for selected engine.
|
||||
3. Import POIs into PostGIS/search index.
|
||||
4. Enrich segments with elevation.
|
||||
5. Import protected areas and rules.
|
||||
6. Generate offline pack assets.
|
||||
|
||||
Do not use public Overpass instances for production-scale queries.
|
||||
|
||||
## Implemented skeleton
|
||||
|
||||
The package includes a small OSM-like fixture and a normalizer:
|
||||
|
||||
```bash
|
||||
npm -w @pikebacker/data-pipeline run normalize:sample
|
||||
```
|
||||
|
||||
The normalizer maps OSM tags into bikepacking POI categories (`water`, `food`, `sleep`, `repair`, `bailout`, `charging`, `emergency`) and preserves `source`, `tags`, `opening_hours`, confidence, and OpenStreetMap attribution hooks. Production ingestion should write the normalized records into PostGIS using `schemas/database.sql`.
|
||||
Reference in New Issue
Block a user