ekke da6f968725 Zonal-statistics Excel export, COG layer entry point, DE Africa ETL
Excel export (from TWG feedback)
- src/analysis/xlsx.js: a dependency-free XLSX writer — an .xlsx is a ZIP of
  XML, so this packs the required parts with a small stored-ZIP writer. Avoids
  SheetJS (stale npm package with advisories) and ExcelJS (heavy for an
  offline-first field app), and does not rely on the JSZip that only reaches us
  transitively via shp-write. Lazy-loaded as a ~5.6 kB chunk.
- After a zonal run the Analysis panel offers "Export table (Excel)", writing a
  two-sheet workbook: Results (figures as real numbers) and Parameters (zone
  and input layers with feature counts, the "Apply to" scope, membership rule,
  statistics, numeric field and export time) — so a table can be verified or
  reproduced later rather than being an unattributed set of numbers.
  The button is hidden for overlay runs and cleared when the mode changes.
- Verified against two independent readers: openpyxl loads it with zero
  warnings and correct numeric types, and LibreOffice Calc opens it as a
  spreadsheet. Also exercised end-to-end through the real zonal pipeline.

COG raster entry point
- Add External Layer gains a COG type alongside WMS/WFS/XYZ, with a URL
  pre-flight check that distinguishes a web page, a 404, a CORS block and a
  server without byte-range support — geotiff.js otherwise reports these only
  as an opaque "AggregateError: Request failed".

Digital Earth Africa ETL
- etl/deafrica_dem_to_minio.py exports a DE Africa DEM for a district as a COG
  and uploads it to the LUSPA MinIO bucket raster-objects, then verifies the
  object is anonymously readable and range-capable. Credentials come from the
  environment; the existing PHP integration hardcodes them and an earlier key
  pair reached Gitea. NOTE: not yet run against the Sandbox — start with
  --list-products to confirm the DEM product name.

Documents
- Concept note: Digital Earth Africa added as a raster source (§6.3), separating
  the live WMS route from the batch Sandbox export; corrected the in-house
  contour table's provenance to OpenTopography (gdal_contour over an SRTM 30 m /
  Copernicus 30 m DEM), and added the ToR §2.4.2 / FS §2 alignment chapter.
- TWG presentation, architecture and two integration workflow charts (SVG
  sources kept in the repo so they stay editable), and a user guide for the
  Analyse tools.

Service worker v13 -> v14. .gitignore: exclude Python bytecode from etl/.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 16:28:45 +02:00

643 lines
23 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

/**
* Service Worker
*
* Handles caching of:
* - App shell (HTML, CSS, JS)
* - Map tiles (passive runtime caching, per-host buckets)
* - API responses (network-first)
*
* Note: Database operations are handled by the SharedWorker (shared-db-worker.js),
* NOT by this service worker. They serve different purposes:
* - Service Worker: Caching, offline asset serving, push notifications
* - SharedWorker: Shared database connection across tabs
*/
// v3: lower per-cache limits (5000 → 1500) and counter-based eviction to
// prevent Safari memory-pressure reloads.
// v4: raise OSM and Topographic limits to 8000 to support active offline
// downloads (Phase 2). Other providers stay at 1500.
// v5: switch LayerSwitcher icon path to base-URL-aware; force shell refresh.
// v6: rename /icons/ → /app-icons/ to dodge Apache's default mod_alias
// mapping (Alias /icons/ /usr/share/apache2/icons/) which intercepts
// the path server-side. Force shell refresh so deployed clients
// re-fetch the new HTML/manifest with the new path.
// v7: HTML pages now use network-first (was cache-first) so new deploys
// are picked up immediately without needing another SW version bump.
// Hashed JS / CSS / WASM stay cache-first (they're immutable per build).
// v8: GPS trail recording feature (reusable src/geotracker/ engine, expandable
// My Location control, navbar GPS readout, gps_trails SQLocal tables) plus
// mobile drawing-toolbar wrap, base-map "None" option, and the Safari
// 100svh dock fix. New hashed bundle + updated shell — bump to evict the
// old module/shell caches.
// v9: Iframe embed endpoint (/embed via public/embed.php) + postMessage bridge
// for the permitting integration; lu_parcels structural refactor in the
// local DB; production access guard + no-district overlay; LayerSwitcher
// ordering fix. New shell + hashed bundle.
// v10: UPN-grid layer (cache + click popup + dashed-violet style + 1:25,000
// label threshold); external-dataset import → staging → upload pipeline
// (external_imports/_features tables, mapping modal, LayerSwitcher chip,
// staged-upload payload with user_id_upload from SSO); GIS export from
// Area/Circle Analysis popups (GeoJSON / SHP via shp-write / KML, with
// field-rename modal). New hashed bundle + shp-write chunk.
// v11: Permit-iframe hardening (in-iframe auth-required card replacing the
// 302 redirect, X-Frame-Options strip at both .htaccess and PHP layers);
// import-flow UX refinements (parse spinner, per-feature client_uuid
// tagging, geometry-edit + delete persistence to the staging tables,
// sample values + Unicode-bold field names in the mapping dropdown).
// New hashed bundle + updated embed.php shell.
// v12: Session/district correctness — a real /?logout=1 endpoint that destroys
// the PWA's own PHP session and then hands off to the portal's
// /user-logout for a FULL SSO logout (token invalidated + sso_auth_token
// cleared by the portal). Previously logout left the PHP session intact,
// pinning a stale district_id. SSO is validated once per session at login;
// a transfer is picked up on the next logout→login.
// Client-side stale-district guard (wipes district-scoped caches when the
// session district changes) + district-keyed boundary cache; GPS read-out
// UTM coordinate-format setting; ol-ext touch-cursor gated to touch-only
// devices. New hashed bundle + updated index.php shell.
// v13: Analytical tools Stage 1 — client-side spatial analysis. New "Analyse"
// dock button opens an Analysis panel offering vector overlays
// (intersect / clip / difference / union, via Turf.js) and vector-in-vector
// zonal statistics (count / sum / mean / min / max / area per zone); results
// are added as layers in a new "Analysis" group and work fully offline.
// Analysis can be scoped via "Apply to": whole layer, current map view,
// selected features, or the catch of a drawn Circle/Area. The Circle/Area
// analysis popup now has one "Export" button (PDF folded into the export
// modal as a fourth format) plus an "Analyse" button that opens the panel
// pre-scoped to the intersecting features.
// Raster display gains Cloud-Optimized GeoTIFF support (MapView.addCOGLayer).
// Turf, the analysis panel, geotiff.js and pako are all code-split and
// fetched on first use, so the eager bundle is unchanged.
// v14: Zonal statistics results can be exported as a two-sheet Excel workbook
// (Results + the Parameters that produced them) via a dependency-free
// XLSX writer in src/analysis/xlsx.js; Add External Layer gains a COG
// option with a URL pre-flight check.
const CACHE_VERSION = 'v14';
const SHELL_CACHE = `shell-${CACHE_VERSION}`;
const MODULES_CACHE = `modules-${CACHE_VERSION}`;
const API_CACHE = `api-${CACHE_VERSION}`;
// ----------------------------------------------------------------------------
// Tile caches — one per provider so users can clear them independently.
// Limits are per-cache (not global). 5 000 tiles ≈ ~150 MB at ~30 KB/tile,
// which covers a Ghana district at zoom 1015 (typical field-work range).
// ----------------------------------------------------------------------------
const TILES_OSM = `tiles-osm-${CACHE_VERSION}`;
const TILES_TOPO = `tiles-topo-${CACHE_VERSION}`;
const TILES_SATELLITE = `tiles-satellite-${CACHE_VERSION}`;
const TILES_CARTO_LIGHT = `tiles-carto-light-${CACHE_VERSION}`;
const TILES_CARTO_DARK = `tiles-carto-dark-${CACHE_VERSION}`;
// Per-provider tile limits.
// • OSM and Topographic are the providers offered for active offline
// download (Phase 2 dialog), so they get a higher cap (~240 MB each at
// ~30 KB/tile) — enough for a typical Ghana district at zoom 1015.
// • The other providers serve passive caching only (whatever the user has
// already viewed), so 1 500 tiles ≈ 45 MB is plenty.
//
// Total max ≈ 5 × ~150 MB = ~750 MB on disk in the worst case, but only the
// two downloadable buckets are likely to fill. Eviction sweeps run every 100
// inserts (see EVICTION_CHECK_INTERVAL) so memory pressure stays bounded.
const TILE_LIMITS = {
[TILES_OSM]: 8000,
[TILES_TOPO]: 8000,
[TILES_SATELLITE]: 1500,
[TILES_CARTO_LIGHT]: 1500,
[TILES_CARTO_DARK]: 1500,
};
// Per-cache running insert counter, in memory. Avoids calling cache.keys()
// (which materialises every Request object in the cache) on every put — that
// was the cause of the Safari "reloaded due to memory pressure" failures.
//
// We only run a real eviction sweep every EVICTION_CHECK_INTERVAL inserts.
const _tileInsertCounters = new Map(); // cacheName → number of inserts since last eviction
const EVICTION_CHECK_INTERVAL = 100;
// Friendly name shown in the UI (matches Settings card labels)
const TILE_CACHE_LABELS = {
[TILES_OSM]: 'OpenStreetMap',
[TILES_TOPO]: 'Topographic',
[TILES_SATELLITE]: 'Satellite',
[TILES_CARTO_LIGHT]: 'Carto Light',
[TILES_CARTO_DARK]: 'Carto Dark',
};
const ALL_TILE_CACHES = Object.keys(TILE_LIMITS);
// Approximate average tile size — used for storage estimation.
// Real measurements: PNG tiles range 580 KB; 30 KB is a good middle ground.
const AVG_TILE_BYTES = 30 * 1024;
// ----------------------------------------------------------------------------
// App shell assets — precached on install.
// ----------------------------------------------------------------------------
const SHELL_ASSETS = [
'/',
'/index.html',
'/offline.html',
'/manifest.json'
];
// ============================================================================
// INSTALL EVENT
// ============================================================================
self.addEventListener('install', (event) => {
console.log('[SW] Installing...');
event.waitUntil(
caches.open(SHELL_CACHE)
.then((cache) => {
console.log('[SW] Precaching app shell');
return cache.addAll(SHELL_ASSETS);
})
.then(() => self.skipWaiting())
);
});
// ============================================================================
// ACTIVATE EVENT
// ============================================================================
self.addEventListener('activate', (event) => {
console.log('[SW] Activating...');
event.waitUntil(
caches.keys()
.then((cacheNames) => {
// Build the set of caches that should remain
const keep = new Set([SHELL_CACHE, MODULES_CACHE, API_CACHE, ...ALL_TILE_CACHES]);
return Promise.all(
cacheNames
// Delete anything that:
// • belongs to one of our managed cache prefixes (shell-, tiles-, modules-, api-)
// • but is NOT in the current keep set
// This includes the legacy "tiles-v1" single bucket.
.filter((name) => {
const isOurs =
name.startsWith('shell-') ||
name.startsWith('tiles-') ||
name.startsWith('modules-') ||
name.startsWith('api-');
return isOurs && !keep.has(name);
})
.map((name) => {
console.log('[SW] Deleting old cache:', name);
return caches.delete(name);
})
);
})
.then(() => self.clients.claim())
);
});
// ============================================================================
// FETCH EVENT
// ============================================================================
self.addEventListener('fetch', (event) => {
const request = event.request;
const url = new URL(request.url);
// Only handle GET requests
if (request.method !== 'GET') return;
// Skip chrome-extension and other non-http(s) requests
if (!url.protocol.startsWith('http')) return;
// Skip worker files and Vite dev-server node_modules requests —
// intercepting these breaks module workers (e.g. SQLocal/SQLite).
if (url.pathname.includes('node_modules') ||
url.search.includes('worker_file') ||
request.destination === 'worker') return;
// ----- TILE REQUESTS — passive cache-then-network (per-host bucket) -----
const tileCache = getTileCacheName(url);
if (tileCache) {
event.respondWith(tileCacheThenNetwork(request, tileCache));
return;
}
// ----- OTHER ROUTES -----
if (isApiRequest(url)) {
event.respondWith(networkFirst(request, API_CACHE));
} else if (isModuleAsset(url)) {
event.respondWith(staleWhileRevalidate(request, MODULES_CACHE));
} else if (isHtmlAsset(url)) {
// HTML uses network-first so a fresh deploy is picked up immediately.
// Falls back to the cached copy when offline (so the app still loads).
event.respondWith(networkFirst(request, SHELL_CACHE));
} else if (isAppAsset(url)) {
// Hashed JS / CSS / WASM / icons are immutable per build — cache-first
// is the right strategy here.
event.respondWith(cacheFirst(request, SHELL_CACHE));
}
// Let other requests pass through to network
});
// ============================================================================
// URL CLASSIFICATION
// ============================================================================
/**
* Classify a URL into the appropriate tile cache.
* Returns `null` for non-tile requests, or for tile providers we deliberately
* do NOT cache (e.g. Google — caching is forbidden by their ToS).
*/
function getTileCacheName(url) {
const host = url.hostname;
// OpenStreetMap — tile.openstreetmap.org and a/b/c subdomains
if (host.endsWith('tile.openstreetmap.org')) return TILES_OSM;
// OpenTopoMap — a/b/c.tile.opentopomap.org
if (host.endsWith('tile.opentopomap.org') || host.endsWith('opentopomap.org')) return TILES_TOPO;
// Carto Basemaps — light_all / dark_all distinguished by path
if (host.endsWith('basemaps.cartocdn.com')) {
if (url.pathname.includes('/light_all/')) return TILES_CARTO_LIGHT;
if (url.pathname.includes('/dark_all/')) return TILES_CARTO_DARK;
return null; // unknown Carto style
}
// Esri — server.arcgisonline.com
if (host.endsWith('arcgisonline.com')) return TILES_SATELLITE;
// Google — caching forbidden by ToS, do not store
if (host.endsWith('google.com') || host.endsWith('googleapis.com')) return null;
// Other tile providers (WMS endpoints, OWS, custom) — not cached at this layer
// (the user's "online only" toast handles those).
return null;
}
function isApiRequest(url) {
return url.pathname.startsWith('/api/') ||
url.pathname.endsWith('.php');
}
function isModuleAsset(url) {
return url.pathname.startsWith('/modules/');
}
/**
* HTML pages (and the bare PWA root) — fetched network-first so new deploys
* roll out immediately. We never want a stale shell pointing at hashed
* asset URLs that no longer exist on the server.
*/
function isHtmlAsset(url) {
if (url.origin !== self.location.origin) return false;
if (url.pathname.endsWith('.html')) return true;
// Treat the bare site root and any trailing-slash path as HTML too,
// since they map to index.html / index.php server-side.
if (url.pathname === '/' || url.pathname.endsWith('/')) return true;
return false;
}
/**
* Immutable per-build static assets (hashed JS / CSS / WASM / images) —
* safe to cache-first; each new build produces new URLs so there's no
* stale-content risk.
*/
function isAppAsset(url) {
if (url.origin !== self.location.origin) return false;
if (isHtmlAsset(url)) return false; // HTML handled separately
return (
url.pathname.endsWith('.css') ||
url.pathname.endsWith('.js') ||
url.pathname.endsWith('.wasm') ||
url.pathname.endsWith('.json') ||
!!url.pathname.match(/\.(png|jpg|jpeg|gif|svg|ico|webp|woff2?|ttf|otf)$/)
);
}
// ============================================================================
// CACHING STRATEGIES
// ============================================================================
/**
* Cache First — Use cache, fallback to network.
* Best for: App shell, static assets.
*/
async function cacheFirst(request, cacheName) {
const cached = await caches.match(request);
if (cached) return cached;
try {
const response = await fetch(request);
if (response.ok) {
const cache = await caches.open(cacheName);
cache.put(request, response.clone());
}
return response;
} catch (error) {
if (request.mode === 'navigate') {
return caches.match('/offline.html');
}
throw error;
}
}
/**
* Network First — Try network, fallback to cache.
* Best for: API requests, dynamic content.
*/
async function networkFirst(request, cacheName) {
try {
const response = await fetch(request);
if (response.ok) {
const cache = await caches.open(cacheName);
cache.put(request, response.clone());
}
return response;
} catch (error) {
const cached = await caches.match(request);
if (cached) return cached;
throw error;
}
}
/**
* Stale While Revalidate — Return cache immediately, update in background.
* Best for: Module assets, frequently updated content.
*/
async function staleWhileRevalidate(request, cacheName) {
const cache = await caches.open(cacheName);
const cached = await cache.match(request);
const fetchPromise = fetch(request).then((response) => {
if (response.ok) {
cache.put(request, response.clone());
}
return response;
}).catch(() => cached);
return cached || fetchPromise;
}
/**
* Tile Cache then Network — Per-host bucket with size limit.
* Cache first; on miss, fetch from network and store.
*
* Memory-conservative eviction:
* • Increments an in-memory counter on every successful insert
* • Only calls cache.keys() (which materialises all Request objects) every
* EVICTION_CHECK_INTERVAL inserts — so the cost is amortised
* • Eviction drops the oldest 10 % when over the per-host limit
*
* On network failure (offline), serves a 408 so the map renders a blank tile
* rather than throwing.
*/
async function tileCacheThenNetwork(request, cacheName) {
const cache = await caches.open(cacheName);
const cached = await cache.match(request);
if (cached) return cached;
try {
const response = await fetch(request);
if (response.ok) {
// Bump the counter; periodically run a real eviction sweep
const count = (_tileInsertCounters.get(cacheName) || 0) + 1;
_tileInsertCounters.set(cacheName, count);
if (count % EVICTION_CHECK_INTERVAL === 0) {
// Reset the counter — next sweep is another EVICTION_CHECK_INTERVAL away
_tileInsertCounters.set(cacheName, 0);
await maybeEvict(cache, cacheName);
}
// Don't await put() — it can run after we return the response, keeping
// the fetch hot path lightweight.
cache.put(request, response.clone()).catch((err) => {
// QuotaExceededError → run an immediate eviction sweep and retry once
if (err && err.name === 'QuotaExceededError') {
maybeEvict(cache, cacheName, /* force */ true).catch(() => {});
}
});
}
return response;
} catch (error) {
// Offline — let the map renderer show a blank tile
return new Response('', { status: 408, statusText: 'Offline' });
}
}
/**
* Run an eviction sweep on a cache, dropping the oldest 10 % of entries
* when over the per-cache limit. Heavy: only call periodically.
*/
async function maybeEvict(cache, cacheName, force = false) {
try {
const limit = TILE_LIMITS[cacheName] || 1500;
const keys = await cache.keys();
if (force || keys.length >= limit) {
const drop = Math.max(1, Math.ceil(limit * 0.1));
const toDelete = keys.slice(0, drop);
await Promise.all(toDelete.map((k) => cache.delete(k)));
}
} catch (err) {
console.warn('[SW] eviction sweep failed for', cacheName, err);
}
}
// ============================================================================
// MESSAGE HANDLING
// ============================================================================
/**
* Reply to a message, preferring the transferred MessageChannel port (the
* window's pwa.js sends a port for request/response correlation), and
* falling back to the originating WindowClient if no port was supplied.
*/
function replyTo(event, message) {
if (event.ports && event.ports[0]) {
try { event.ports[0].postMessage(message); return; } catch {}
}
if (event.source && typeof event.source.postMessage === 'function') {
event.source.postMessage(message);
}
}
self.addEventListener('message', (event) => {
const { type, payload, cacheName } = event.data || {};
switch (type) {
case 'SKIP_WAITING':
self.skipWaiting();
break;
case 'CACHE_MODULES':
cacheModules(payload.modules);
break;
case 'CLEAR_USER_CACHE':
clearUserCaches();
break;
case 'GET_CACHE_STATUS':
getCacheStatus().then((status) => replyTo(event, { type: 'CACHE_STATUS', status }));
break;
// ----- Tile-cache management (Phase 1 offline maps) -----
case 'GET_TILE_STATS':
getTileStats().then((stats) => replyTo(event, { type: 'TILE_STATS', stats }));
break;
case 'CLEAR_TILE_CACHES':
clearTileCaches().then(() => replyTo(event, { type: 'TILE_CACHES_CLEARED' }));
break;
// Clear a single provider's tile cache (Phase 3 — per-provider Clear).
// Validates the requested name against the known ALL_TILE_CACHES list so
// a misbehaving caller can't drop unrelated caches.
case 'CLEAR_TILE_CACHE':
if (typeof cacheName === 'string' && ALL_TILE_CACHES.includes(cacheName)) {
caches.delete(cacheName).then((deleted) => {
_tileInsertCounters.delete(cacheName);
_cachedStats = null; _cachedStatsAt = 0; // invalidate stats cache
replyTo(event, { type: 'TILE_CACHE_CLEARED', cacheName, deleted });
});
} else {
replyTo(event, { type: 'TILE_CACHE_CLEARED', cacheName, deleted: false, error: 'Unknown or invalid cache name' });
}
break;
}
});
// ============================================================================
// HELPER FUNCTIONS
// ============================================================================
/**
* Cache specific modules on demand.
*/
async function cacheModules(moduleNames) {
const cache = await caches.open(MODULES_CACHE);
for (const moduleName of moduleNames) {
try {
const moduleAssets = [
`/modules/${moduleName}/index.js`,
`/modules/${moduleName}/index.css`,
`/modules/${moduleName}/index.html`
];
await cache.addAll(moduleAssets.filter(async (url) => {
try {
const response = await fetch(url, { method: 'HEAD' });
return response.ok;
} catch {
return false;
}
}));
console.log('[SW] Cached module:', moduleName);
} catch (error) {
console.warn('[SW] Failed to cache module:', moduleName, error);
}
}
}
/**
* Clear user-specific caches (call on logout).
* Tile caches are NOT cleared here — those belong to the device, not the user.
*/
async function clearUserCaches() {
await caches.delete(API_CACHE);
await caches.delete(MODULES_CACHE);
console.log('[SW] Cleared user caches');
}
/**
* Get summary status of all caches (count of entries in each).
*/
async function getCacheStatus() {
const cacheNames = await caches.keys();
const status = {};
for (const name of cacheNames) {
const cache = await caches.open(name);
const keys = await cache.keys();
status[name] = keys.length;
}
return status;
}
/**
* Get per-provider tile cache statistics.
*
* Returns shape:
* {
* totals: { count, estBytes },
* byProvider: [{ key, label, count, limit, estBytes }, ...]
* }
*
* estBytes is an approximation (count × AVG_TILE_BYTES). For an exact size,
* the caller can use navigator.storage.estimate() on the page side.
*
* Result is cached for STATS_TTL_MS so rapid re-queries (e.g. multiple
* Settings opens) don't re-enumerate every cache.
*/
const STATS_TTL_MS = 10 * 1000;
let _cachedStats = null;
let _cachedStatsAt = 0;
async function getTileStats({ force = false } = {}) {
const now = Date.now();
if (!force && _cachedStats && (now - _cachedStatsAt) < STATS_TTL_MS) {
return _cachedStats;
}
const byProvider = [];
let totalCount = 0;
for (const cacheName of ALL_TILE_CACHES) {
let count = 0;
if (await caches.has(cacheName)) {
const cache = await caches.open(cacheName);
// matchAll returns a smaller payload than keys() on Safari, but neither
// is free. Done at most once per STATS_TTL_MS thanks to the cache above.
const keys = await cache.keys();
count = keys.length;
}
byProvider.push({
key: cacheName,
label: TILE_CACHE_LABELS[cacheName] || cacheName,
count,
limit: TILE_LIMITS[cacheName] || 0,
estBytes: count * AVG_TILE_BYTES,
});
totalCount += count;
}
_cachedStats = {
totals: {
count: totalCount,
estBytes: totalCount * AVG_TILE_BYTES,
},
byProvider,
};
_cachedStatsAt = now;
return _cachedStats;
}
/**
* Delete every tile cache. Frees the device storage used by cached map tiles.
* Does not affect app-shell, modules, or API caches.
*/
async function clearTileCaches() {
const results = await Promise.all(
ALL_TILE_CACHES.map((name) => caches.delete(name))
);
// Reset counters and invalidate stats cache
_tileInsertCounters.clear();
_cachedStats = null;
_cachedStatsAt = 0;
console.log('[SW] Cleared tile caches:', ALL_TILE_CACHES.filter((_, i) => results[i]));
}