ekke
da6f968725
Excel export (from TWG feedback) - src/analysis/xlsx.js: a dependency-free XLSX writer — an .xlsx is a ZIP of XML, so this packs the required parts with a small stored-ZIP writer. Avoids SheetJS (stale npm package with advisories) and ExcelJS (heavy for an offline-first field app), and does not rely on the JSZip that only reaches us transitively via shp-write. Lazy-loaded as a ~5.6 kB chunk. - After a zonal run the Analysis panel offers "Export table (Excel)", writing a two-sheet workbook: Results (figures as real numbers) and Parameters (zone and input layers with feature counts, the "Apply to" scope, membership rule, statistics, numeric field and export time) — so a table can be verified or reproduced later rather than being an unattributed set of numbers. The button is hidden for overlay runs and cleared when the mode changes. - Verified against two independent readers: openpyxl loads it with zero warnings and correct numeric types, and LibreOffice Calc opens it as a spreadsheet. Also exercised end-to-end through the real zonal pipeline. COG raster entry point - Add External Layer gains a COG type alongside WMS/WFS/XYZ, with a URL pre-flight check that distinguishes a web page, a 404, a CORS block and a server without byte-range support — geotiff.js otherwise reports these only as an opaque "AggregateError: Request failed". Digital Earth Africa ETL - etl/deafrica_dem_to_minio.py exports a DE Africa DEM for a district as a COG and uploads it to the LUSPA MinIO bucket raster-objects, then verifies the object is anonymously readable and range-capable. Credentials come from the environment; the existing PHP integration hardcodes them and an earlier key pair reached Gitea. NOTE: not yet run against the Sandbox — start with --list-products to confirm the DEM product name. Documents - Concept note: Digital Earth Africa added as a raster source (§6.3), separating the live WMS route from the batch Sandbox export; corrected the in-house contour table's provenance to OpenTopography (gdal_contour over an SRTM 30 m / Copernicus 30 m DEM), and added the ToR §2.4.2 / FS §2 alignment chapter. - TWG presentation, architecture and two integration workflow charts (SVG sources kept in the repo so they stay editable), and a user guide for the Analyse tools. Service worker v13 -> v14. .gitignore: exclude Python bytecode from etl/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
84 lines
3.0 KiB
Markdown
84 lines
3.0 KiB
Markdown
# LUPMIS2 raster ETL
|
|
|
|
One-off batch jobs that prepare raster data for LUPMIS2 and publish it to the
|
|
LUSPA MinIO object store, from where the PWA streams it as a Cloud-Optimized
|
|
GeoTIFF (COG).
|
|
|
|
This mirrors the existing OpenTopography contour workflow: a **preparation
|
|
step run occasionally**, not a live connection from the app.
|
|
|
|
```
|
|
DE Africa (Sandbox) → write_cog() → MinIO raster-objects → LUPMIS2 map
|
|
```
|
|
|
|
## Target infrastructure
|
|
|
|
| | |
|
|
|---|---|
|
|
| S3 API endpoint | `https://minioapi.lupmis4luspa.org` (path-style) |
|
|
| Bucket | `raster-objects` (anonymous read enabled) |
|
|
| Console | `https://minio.lupmis4luspa.org` — admin UI only, **not** the API |
|
|
|
|
Verified working for COG streaming: anonymous unsigned `GET` returns 200,
|
|
range requests return `206 Partial Content`, and CORS allows the PWA origin
|
|
with `Range` on preflight and `Content-Range` exposed.
|
|
|
|
## Credentials
|
|
|
|
Supplied via environment variables — **never** hardcode them:
|
|
|
|
```bash
|
|
export MINIO_KEY=...
|
|
export MINIO_SECRET=...
|
|
```
|
|
|
|
The PHP integration under `FromKwesi/minio-uploads` embeds its keys in source,
|
|
and an earlier key pair was committed to Gitea. Keep these out of git.
|
|
|
|
## `deafrica_dem_to_minio.py`
|
|
|
|
Exports a Digital Earth Africa DEM for an area of interest and uploads it.
|
|
|
|
Run it in the **DE Africa Sandbox** — the Open Data Cube is configured there.
|
|
Locally, DE Africa's docs note that `dc.load` / `load_ard` need additional
|
|
configuration (a database), so the Sandbox is the lower-maintenance choice.
|
|
|
|
```bash
|
|
# 1. confirm which DEM products exist before the first real run
|
|
python deafrica_dem_to_minio.py --list-products
|
|
|
|
# 2. export + upload a pilot district
|
|
python deafrica_dem_to_minio.py --district koforidua
|
|
|
|
# preferred for production: clip to the authoritative boundary
|
|
python deafrica_dem_to_minio.py --geojson koforidua.geojson --name koforidua
|
|
|
|
# inspect the output without uploading
|
|
python deafrica_dem_to_minio.py --district tamale --dry-run
|
|
```
|
|
|
|
The script writes `<name>_dem.tif`, uploads it to `dem/<name>_dem.tif`, then
|
|
verifies the public URL really is anonymously readable and range-capable. It
|
|
also emits `<name>_dem.manifest.json` recording the URL, product, bbox, CRS and
|
|
resolution.
|
|
|
|
### Loading the result in LUPMIS2
|
|
|
|
**Add External Layer → COG →** paste the object URL, e.g.
|
|
|
|
```
|
|
https://minioapi.lupmis4luspa.org/raster-objects/dem/koforidua_dem.tif
|
|
```
|
|
|
|
### Notes and caveats
|
|
|
|
- **Preset district bounding boxes are approximate.** Use `--geojson` with the
|
|
real district boundary from PostGIS for anything beyond a first test.
|
|
- **Output CRS defaults to EPSG:3857** so the browser does no reprojection.
|
|
For Stage 3 hydrology and slope, re-export in a metric CRS (UTM 30N =
|
|
`EPSG:32630` west, 31N = `EPSG:32631` east) — Web Mercator distorts distance
|
|
with latitude and will bias slope and flow-accumulation results.
|
|
- **Verify the product name** with `--list-products` first. The default is
|
|
`dem_srtm`; DE Africa also publishes derivatives (`dem_srtm_deriv`), which is
|
|
the source of the slope WMS layer LUPMIS2 already consumes.
|