ekke da6f968725 Zonal-statistics Excel export, COG layer entry point, DE Africa ETL

Excel export (from TWG feedback)
- src/analysis/xlsx.js: a dependency-free XLSX writer — an .xlsx is a ZIP of
  XML, so this packs the required parts with a small stored-ZIP writer. Avoids
  SheetJS (stale npm package with advisories) and ExcelJS (heavy for an
  offline-first field app), and does not rely on the JSZip that only reaches us
  transitively via shp-write. Lazy-loaded as a ~5.6 kB chunk.
- After a zonal run the Analysis panel offers "Export table (Excel)", writing a
  two-sheet workbook: Results (figures as real numbers) and Parameters (zone
  and input layers with feature counts, the "Apply to" scope, membership rule,
  statistics, numeric field and export time) — so a table can be verified or
  reproduced later rather than being an unattributed set of numbers.
  The button is hidden for overlay runs and cleared when the mode changes.
- Verified against two independent readers: openpyxl loads it with zero
  warnings and correct numeric types, and LibreOffice Calc opens it as a
  spreadsheet. Also exercised end-to-end through the real zonal pipeline.

COG raster entry point
- Add External Layer gains a COG type alongside WMS/WFS/XYZ, with a URL
  pre-flight check that distinguishes a web page, a 404, a CORS block and a
  server without byte-range support — geotiff.js otherwise reports these only
  as an opaque "AggregateError: Request failed".

Digital Earth Africa ETL
- etl/deafrica_dem_to_minio.py exports a DE Africa DEM for a district as a COG
  and uploads it to the LUSPA MinIO bucket raster-objects, then verifies the
  object is anonymously readable and range-capable. Credentials come from the
  environment; the existing PHP integration hardcodes them and an earlier key
  pair reached Gitea. NOTE: not yet run against the Sandbox — start with
  --list-products to confirm the DEM product name.

Documents
- Concept note: Digital Earth Africa added as a raster source (§6.3), separating
  the live WMS route from the batch Sandbox export; corrected the in-house
  contour table's provenance to OpenTopography (gdal_contour over an SRTM 30 m /
  Copernicus 30 m DEM), and added the ToR §2.4.2 / FS §2 alignment chapter.
- TWG presentation, architecture and two integration workflow charts (SVG
  sources kept in the repo so they stay editable), and a user guide for the
  Analyse tools.

Service worker v13 -> v14. .gitignore: exclude Python bytecode from etl/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-23 16:28:45 +02:00

3.0 KiB

Raw Blame History

LUPMIS2 raster ETL

One-off batch jobs that prepare raster data for LUPMIS2 and publish it to the LUSPA MinIO object store, from where the PWA streams it as a Cloud-Optimized GeoTIFF (COG).

This mirrors the existing OpenTopography contour workflow: a preparation step run occasionally, not a live connection from the app.

DE Africa (Sandbox)  →  write_cog()  →  MinIO raster-objects  →  LUPMIS2 map

Target infrastructure


S3 API endpoint	`https://minioapi.lupmis4luspa.org` (path-style)
Bucket	`raster-objects` (anonymous read enabled)
Console	`https://minio.lupmis4luspa.org` — admin UI only, not the API

Verified working for COG streaming: anonymous unsigned GET returns 200, range requests return 206 Partial Content, and CORS allows the PWA origin with Range on preflight and Content-Range exposed.

Credentials

Supplied via environment variables — never hardcode them:

export MINIO_KEY=...
export MINIO_SECRET=...

The PHP integration under FromKwesi/minio-uploads embeds its keys in source, and an earlier key pair was committed to Gitea. Keep these out of git.

`deafrica_dem_to_minio.py`

Exports a Digital Earth Africa DEM for an area of interest and uploads it.

Run it in the DE Africa Sandbox — the Open Data Cube is configured there. Locally, DE Africa's docs note that dc.load / load_ard need additional configuration (a database), so the Sandbox is the lower-maintenance choice.

# 1. confirm which DEM products exist before the first real run
python deafrica_dem_to_minio.py --list-products

# 2. export + upload a pilot district
python deafrica_dem_to_minio.py --district koforidua

# preferred for production: clip to the authoritative boundary
python deafrica_dem_to_minio.py --geojson koforidua.geojson --name koforidua

# inspect the output without uploading
python deafrica_dem_to_minio.py --district tamale --dry-run

The script writes <name>_dem.tif, uploads it to dem/<name>_dem.tif, then verifies the public URL really is anonymously readable and range-capable. It also emits <name>_dem.manifest.json recording the URL, product, bbox, CRS and resolution.

Loading the result in LUPMIS2

Add External Layer → COG → paste the object URL, e.g.

https://minioapi.lupmis4luspa.org/raster-objects/dem/koforidua_dem.tif

Notes and caveats

Preset district bounding boxes are approximate. Use --geojson with the real district boundary from PostGIS for anything beyond a first test.
Output CRS defaults to EPSG:3857 so the browser does no reprojection. For Stage 3 hydrology and slope, re-export in a metric CRS (UTM 30N = EPSG:32630 west, 31N = EPSG:32631 east) — Web Mercator distorts distance with latitude and will bias slope and flow-accumulation results.
Verify the product name with --list-products first. The default is dem_srtm; DE Africa also publishes derivatives (dem_srtm_deriv), which is the source of the slope WMS layer LUPMIS2 already consumes.

3.0 KiB Raw Blame History