diff --git a/docs/LUPMIS-OSM-Import-Runbook.md b/docs/LUPMIS-OSM-Import-Runbook.md new file mode 100644 index 0000000..325dc2b --- /dev/null +++ b/docs/LUPMIS-OSM-Import-Runbook.md @@ -0,0 +1,634 @@ +# LUPMIS2 — Re-importing the OSM data so it can be kept current + +**For:** whoever runs the import (LUSPA database team) +**Date:** 11 August 2026 + +> **What has and has not been verified.** +> +> **Executed** — against osm2pgsql 2.2.0 and a scratch PostGIS database, using a +> hand-made extract covering every shape this config has to handle: an amenity +> node, a `healthcare`-only node, a `highway` way, a `waterway` way, an amenity +> mapped as a closed way, and one mapped as a multipolygon relation. +> The Lua config in §3 imported all four tables cleanly; `updatable` and +> `attributes` both came out `true`; all four amenity shapes reached +> `pi_osm_points`; geometries are EPSG:4326. A subsequent `--append` applied a +> diff correctly, and — the claim §5.2 depends on — left `districtid` **NULL on +> the changed row while untouched rows kept theirs**. +> +> **Not verified** — anything that needs the LUPMIS database, which is not +> reachable from where this was written: the current state of the import, the +> real name of the district boundary table, and the row counts. §1 is the set of +> checks to run before trusting those parts. + +--- + +## Why this is needed + +The LUPMIS roads endpoint returns, for district 1, 5,238 rows with these +columns: + +``` +geom, districtid, osm_id, surface, oneway, name +``` + +`highway` is not among them. The road class — trunk, primary, residential, +track — was discarded when the data was loaded. That is the normal outcome of +importing with osm2pgsql's `default.style`, which keeps a fixed list of tags +and silently drops everything else; the same mechanism discards `phone`, +`opening_hours` and `website`. The consequences are already visible in the +application: + +- Roads cannot be styled or filtered by class. Every road is drawn identically, + so a trunk road and a footpath look the same at every zoom. +- `src/import-detect.js` lists `highway` as an expected field for the + `osm_roads` target, so the mapping UI offers a column the data does not have. + +A discarded tag cannot be recovered by querying; it was never written. The only +way to get `highway` is to import again — and if the import is going to be +redone, it is worth redoing on terms that allow it to be kept current +afterwards, rather than repeating the same exercise in a year. + +Whether the rest of the diagnosis applies — no middle tables, no replication +set up, a stale `import_timestamp` — is what §1 determines. + +--- + +## What this import decides + +Three decisions determine whether this import can be maintained or has to be +repeated by hand every time. + +**The flex output**, configured by the Lua file in §3. The alternative — the +classic pgsql output that produces `planet_osm_point`, `_line`, `_polygon` and +`_roads` — is deprecated in osm2pgsql 2.x and warns on every run, and its tag +selection is governed by a style file that drops whatever it does not know. +Flex puts that choice in a configuration you own: the columns are the ones you +list, and everything else can be kept in a `jsonb` column instead of being +thrown away. It also writes one table per purpose rather than four fixed ones, +which is what allows an amenity mapped as a building outline to be served as a +point. + +**`--slim`, without `--drop`.** This keeps the middle tables — osm2pgsql's own +record of every node, way and relation it has seen. They cost disk, and they +are the entire difference between an import that can accept a daily diff and +one that can only ever be replaced wholesale. An import run without them +records `updatable=false`, and no later flag will change that; the only repair +is a full re-import. `--drop` discards them, which is why it must not be used +here. + +**`--extra-attributes`.** Records each object's OSM version and timestamp. That +makes it possible to say when a feature was last edited, and to answer "what +changed" at all. + +Two further decisions are specific to LUPMIS. + +**A separate `osm` schema — this one is not negotiable.** osm2pgsql `--create` +drops and recreates every table it owns in its schema. The `spatial` schema +holds `lu_parcels`, the hand-edited planning data this whole system exists to +manage. Pointing osm2pgsql at `spatial` puts a `--create` run one name +collision away from destroying it. Import into `osm`; expose to the API through +views in `spatial` (§7). + +**Districts are stamped after import, not during.** osm2pgsql imports the +national Ghana extract and has no concept of a district. Every LUPMIS table and +endpoint is district-scoped, so a `districtid` column is filled by a spatial +join once the import finishes (§5). This has to run after every daily update as +well, and §5.2 explains why that turns out to be cheap. + +--- + +## 1. Find out what you actually have + +Run these before anything else. They decide whether this is a re-import or a +first proper import, and none of the answers can be guessed from outside. + +```sql +-- Does an osm2pgsql import exist at all, and on what terms? +SELECT property, value FROM osm2pgsql_properties + ORDER BY property; +-- Look for: updatable, attributes, output, style, +-- import_timestamp, replication_base_url, replication_sequence_number +-- If this table does not exist, pi_osm_roads was loaded some other way +-- (ogr2ogr, a hand-written script, a one-off SQL dump). + +-- Where does the current roads data live, and what is in it? +SELECT table_schema, table_name + FROM information_schema.tables + WHERE table_name LIKE '%osm%' OR table_name LIKE 'planet_osm%' + ORDER BY 1, 2; + +SELECT column_name, data_type + FROM information_schema.columns + WHERE table_name = 'pi_osm_roads' + ORDER BY ordinal_position; + +-- Middle tables present? Their absence is what makes --append impossible. +SELECT to_regclass('osm.planet_osm_nodes') AS nodes, + to_regclass('osm.planet_osm_ways') AS ways, + to_regclass('osm.planet_osm_rels') AS rels; +``` + +**If `osm2pgsql_properties` does not exist**, there is nothing to preserve or +migrate: skip §2 and import fresh. That is the more likely case here, because a +`pi_`-prefixed table in `spatial` with exactly six columns looks like a +purpose-built extract rather than anything osm2pgsql produced. + +**If it does exist**, note `replication_base_url` and +`replication_sequence_number` before touching anything — even a non-updatable +import records them, and they tell a fresh import where the data left off. + +--- + +## 2. Clear out only what is safe to clear + +Nothing in this section touches `spatial`. The live application keeps reading +`spatial.pi_osm_roads` until §7 switches it over. + +```sql +CREATE SCHEMA IF NOT EXISTS osm; +``` + +If a previous osm2pgsql run left pgsql-output tables in `osm`, drop these four +and **no others**: + +```sql +DROP TABLE IF EXISTS osm.planet_osm_point, + osm.planet_osm_line, + osm.planet_osm_polygon, + osm.planet_osm_roads; +``` + +`planet_osm_nodes`, `planet_osm_ways`, `planet_osm_rels` and +`planet_osm_users` look like the same family and are not — they are the middle +tables that `--slim` exists to keep, and flex uses them exactly as the pgsql +output did. Dropping them costs you `--append`, which is the point of this +exercise. + +Leave `osm2pgsql_properties` alone; `--create` rewrites it. + +--- + +## 3. The flex configuration + +Save as `sql/lupmis-osm.lua` next to wherever the import is run. + +Four tables, matching the four datasets requested. Every geometry is EPSG:4326, +so the API serves what the table stores and nothing transforms on read. Each +table carries a `districtid` column that the import leaves empty and §5 fills. + +```lua +-- lupmis-osm.lua — osm2pgsql flex config for LUPMIS2 +-- +-- Four tables in the `osm` schema: +-- pi_osm_roads routable `highway` ways (LineString) +-- pi_osm_lines other linear features (LineString) +-- pi_osm_points amenities and healthcare (Point) +-- pi_osm_polygons buildings, land use, areas (MultiPolygon) +-- +-- districtid is declared here but never written by the import. It is filled by +-- the spatial join in §5 of the runbook. Because osm2pgsql re-inserts a row +-- whenever its object changes, an updated row comes back with districtid NULL — +-- which is what makes the incremental re-stamp in §5.2 both simple and correct. + +local srid = 4326 + +local roads = osm2pgsql.define_table({ + name = 'pi_osm_roads', schema = 'osm', + ids = { type = 'way', id_column = 'osm_id' }, + columns = { + { column = 'highway', type = 'text' }, -- the column that was missing + { column = 'name', type = 'text' }, + { column = 'ref', type = 'text' }, + { column = 'surface', type = 'text' }, + { column = 'oneway', type = 'text' }, + { column = 'bridge', type = 'text' }, + { column = 'tunnel', type = 'text' }, + { column = 'layer', type = 'text' }, + { column = 'districtid', type = 'int' }, + { column = 'tags', type = 'jsonb' }, + { column = 'geom', type = 'linestring', projection = srid, not_null = true }, + } +}) + +local lines = osm2pgsql.define_table({ + name = 'pi_osm_lines', schema = 'osm', + ids = { type = 'way', id_column = 'osm_id' }, + columns = { + { column = 'waterway', type = 'text' }, + { column = 'railway', type = 'text' }, + { column = 'power', type = 'text' }, + { column = 'barrier', type = 'text' }, + { column = 'name', type = 'text' }, + { column = 'districtid', type = 'int' }, + { column = 'tags', type = 'jsonb' }, + { column = 'geom', type = 'linestring', projection = srid, not_null = true }, + } +}) + +-- Points accept nodes, ways and relations, so an amenity mapped as a building +-- outline or a multipolygon appears here as a point like any other. That is the +-- single biggest gain over the classic layout, where areas lived in +-- planet_osm_polygon and the endpoint never looked there. +local points = osm2pgsql.define_table({ + name = 'pi_osm_points', schema = 'osm', + ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' }, + columns = { + { column = 'amenity', type = 'text' }, + { column = 'healthcare', type = 'text' }, + { column = 'name', type = 'text' }, + { column = 'districtid', type = 'int' }, + { column = 'geom', type = 'point', projection = srid, not_null = true }, + } +}) + +local polygons = osm2pgsql.define_table({ + name = 'pi_osm_polygons', schema = 'osm', + ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' }, + columns = { + { column = 'building', type = 'text' }, + { column = 'landuse', type = 'text' }, + { column = 'amenity', type = 'text' }, + { column = 'leisure', type = 'text' }, + { column = 'natural', type = 'text' }, + { column = 'name', type = 'text' }, + { column = 'districtid', type = 'int' }, + { column = 'tags', type = 'jsonb' }, + { column = 'geom', type = 'multipolygon', projection = srid, not_null = true }, + } +}) + +-- Tags that say nothing once the typed columns exist. +local uninteresting = { + 'created_by', 'source', 'source:date', 'note', 'comment', + 'fixme', 'FIXME', 'attribution', +} + +local function clean(tags) + for _, k in ipairs(uninteresting) do tags[k] = nil end +end + +local function is_area(tags) + return tags.area == 'yes' + or tags.building or tags.landuse or tags.leisure or tags.natural +end + +function osm2pgsql.process_node(object) + local t = object.tags + if t.amenity or t.healthcare then + points:insert({ + amenity = t.amenity, + healthcare = t.healthcare, + name = t.name, + geom = object:as_point(), + }) + end +end + +function osm2pgsql.process_way(object) + local t = object.tags + clean(t) + + if object.is_closed and is_area(t) then + polygons:insert({ + building = t.building, landuse = t.landuse, amenity = t.amenity, + leisure = t.leisure, natural = t['natural'], name = t.name, + tags = t, geom = object:as_polygon(), + }) + -- An amenity mapped as an area is also a point, so it is findable + -- alongside amenities mapped as nodes. + if t.amenity or t.healthcare then + points:insert({ + amenity = t.amenity, healthcare = t.healthcare, name = t.name, + geom = object:as_polygon():centroid(), + }) + end + return + end + + if t.highway then + roads:insert({ + highway = t.highway, name = t.name, ref = t.ref, + surface = t.surface, oneway = t.oneway, + bridge = t.bridge, tunnel = t.tunnel, layer = t.layer, + tags = t, geom = object:as_linestring(), + }) + elseif t.waterway or t.railway or t.power or t.barrier then + lines:insert({ + waterway = t.waterway, railway = t.railway, + power = t.power, barrier = t.barrier, name = t.name, + tags = t, geom = object:as_linestring(), + }) + end +end + +function osm2pgsql.process_relation(object) + local t = object.tags + clean(t) + if t.type ~= 'multipolygon' and t.type ~= 'boundary' then return end + + if is_area(t) or t.amenity or t.healthcare then + polygons:insert({ + building = t.building, landuse = t.landuse, amenity = t.amenity, + leisure = t.leisure, natural = t['natural'], name = t.name, + tags = t, geom = object:as_multipolygon(), + }) + if t.amenity or t.healthcare then + points:insert({ + amenity = t.amenity, healthcare = t.healthcare, name = t.name, + geom = object:as_multipolygon():centroid(), + }) + end + end +end +``` + +--- + +## 4. Import + +```bash +curl -O https://download.geofabrik.de/africa/ghana-latest.osm.pbf + +osm2pgsql --create --slim --output=flex \ + --style sql/lupmis-osm.lua --extra-attributes \ + --database lupmis --schema osm --middle-schema osm \ + --cache 2000 \ + ghana-latest.osm.pbf +``` + +`--slim` **without** `--drop`. `--drop` discards the middle tables and is what +makes an import permanently un-updatable. + +Substitute the real database name for `lupmis`. + +--- + +## 5. Stamp the districts + +osm2pgsql knows nothing about districts, so this step has no equivalent in the +standard OSM tooling. It is what makes a national import usable by an +application where every table and every endpoint is district-scoped. + +### 5.1 First pass, after the initial import + +Confirm the boundary table's real name and column first — the application only +ever sees it through `get_district_boundary.php`: + +```sql +SELECT table_schema, table_name FROM information_schema.tables + WHERE table_name ILIKE '%district%'; +``` + +Then, for each of the four tables (shown here for roads): + +```sql +UPDATE osm.pi_osm_roads r + SET districtid = d.districtid + FROM spatial.districts d -- confirm this name first + WHERE r.districtid IS NULL + AND ST_Intersects(d.geom, r.geom); +``` + +A linear feature crossing a district boundary matches more than one district. +`UPDATE` takes an arbitrary one of them, which is wrong for a road that spans a +boundary. If roads must appear in every district they touch, use a join table +instead of a column: + +```sql +CREATE TABLE osm.pi_osm_roads_district AS +SELECT r.osm_id, d.districtid + FROM osm.pi_osm_roads r + JOIN spatial.districts d ON ST_Intersects(d.geom, r.geom); +CREATE INDEX ON osm.pi_osm_roads_district (districtid); +``` + +Points never have this problem. For polygons, decide whether a district should +own an area by intersection or by where its centroid falls — +`ST_Intersects(d.geom, ST_Centroid(p.geom))` gives one district per polygon. + +### 5.2 After every update + +The same `UPDATE` again — `WHERE districtid IS NULL` is doing real work here, +not just guarding against repetition. + +When an object changes, osm2pgsql deletes its row and re-inserts it from the +new data. The Lua config never writes `districtid`, so the re-inserted row +comes back **NULL**. `WHERE districtid IS NULL` therefore selects exactly the +objects that are new or were edited since the last stamp — including any that +moved across a boundary — and nothing else. A daily re-stamp touches a few +hundred rows rather than millions. + +Put it in the same cron entry as the update, immediately after it. If the two +ever get separated, the symptom is new roads that no district can see. + +--- + +## 6. Indexes + +osm2pgsql creates the geometry and id indexes. Add what the endpoints filter on: + +```sql +CREATE INDEX ON osm.pi_osm_roads (districtid); +CREATE INDEX ON osm.pi_osm_lines (districtid); +CREATE INDEX ON osm.pi_osm_points (districtid); +CREATE INDEX ON osm.pi_osm_polygons (districtid); + +CREATE INDEX ON osm.pi_osm_roads (highway) WHERE highway IS NOT NULL; +CREATE INDEX ON osm.pi_osm_points (amenity) WHERE amenity IS NOT NULL; +CREATE INDEX ON osm.pi_osm_points (healthcare) WHERE healthcare IS NOT NULL; +CREATE INDEX ON osm.pi_osm_roads USING gin (tags); +CREATE INDEX ON osm.pi_osm_polygons USING gin (tags); +``` + +--- + +## 7. Expose it to the API + +Keep osm2pgsql's tables in `osm`, and give the API views in `spatial` under the +existing `pi_` naming. The application then reads the names it already expects, +and no import can ever write into `spatial`. + +The current `spatial.pi_osm_roads` is a table, and a view cannot replace a table +of the same name. Rename it rather than dropping it, so there is a way back +until the new endpoints have been verified: + +```sql +ALTER TABLE spatial.pi_osm_roads RENAME TO pi_osm_roads_pre_osm2pgsql; + +CREATE VIEW spatial.pi_osm_roads AS + SELECT osm_id, highway, name, ref, surface, oneway, districtid, geom + FROM osm.pi_osm_roads; + +CREATE VIEW spatial.pi_osm_points AS + SELECT osm_type, osm_id, amenity, healthcare, name, districtid, geom + FROM osm.pi_osm_points; + +CREATE VIEW spatial.pi_osm_lines AS + SELECT osm_id, waterway, railway, power, barrier, name, districtid, geom + FROM osm.pi_osm_lines; + +CREATE VIEW spatial.pi_osm_polygons AS + SELECT osm_type, osm_id, building, landuse, amenity, leisure, name, districtid, geom + FROM osm.pi_osm_polygons; +``` + +The existing `get_osm_roads.php` keeps working unchanged and gains `highway`, +because it reads `spatial.pi_osm_roads` and the view now supplies that column. + +Three new endpoints are needed, in the shape of the existing one — same +`{ api_token, district_id }` request, same `{ success, data: [...] }` response, +geometry as WKT in `geom`: + +| Endpoint | Serves | Application layer | +|---|---|---| +| `get_osm_points.php` | `spatial.pi_osm_points` | OSM Points | +| `get_osm_lines.php` | `spatial.pi_osm_lines` | OSM Lines | +| `get_osm_polygons.php` | `spatial.pi_osm_polygons` | OSM Polygons | + +All four belong to layer group **5, Physical Infrastructures**, which is where +`OSM_roads` already sits. + +**Volumes are the thing to watch.** Roads alone are 5,238 rows for district 1 +under the current extract, and the endpoint returns every row as WKT in one +response. Points and polygons will be larger. Before wiring the new layers into +the application, decide whether these endpoints should take a bounding box, or +a `highway`/`amenity` filter, rather than returning a whole district. That +decision belongs with the endpoints, not the application. + +--- + +## 8. Keep it current + +This is the part the import exists for. Everything above only pays off if the +daily update actually runs. + +### 8.1 osm2pgsql-replication needs a Python that has its libraries + +`osm2pgsql-replication` ships with osm2pgsql but is a **Python** script, and +package managers do not install what it imports. Its first run says: + +``` +Missing required Python libraries psycopg2 osmium. +To install them via pip run: pip install psycopg2 osmium +``` + +Following that advice often changes nothing, and it is worth understanding why +before chasing it. The script's shebang is `#!/usr/bin/env python3`, and **`env` +does not see shell aliases**. If `python3` in your shell is aliased — a MAMP +installation does exactly this, and MAMP is present on at least one machine in +this project — then `pip3 install` puts the libraries where the script will +never look. Check before doing anything: + +```bash +env python3 -c "import sys; print(sys.executable)" +env python3 -c "import psycopg2, osmium; print('both present')" +``` + +A Homebrew Python is also likely to be PEP 668 "externally managed", which +refuses the install outright. + +**On the Linux server, use distribution packages** — no virtualenv, no pip: + +```bash +apt install python3-psycopg2 python3-pyosmium # Debian/Ubuntu +``` + +**On a workstation**, give the script an interpreter that has both, once: + +```bash +python3.12 -m venv ~/.venvs/osm2pgsql +~/.venvs/osm2pgsql/bin/pip install psycopg2-binary osmium +``` + +`psycopg2-binary` rather than `psycopg2`: it ships as a wheel and needs no +`pg_config` or compiler. Then invoke the tool with that interpreter: + +```bash +~/.venvs/osm2pgsql/bin/python $(brew --prefix)/bin/osm2pgsql-replication … +``` + +You know it is working when the message changes from the missing-library error +to `Updates not set up correctly. Run 'osm2pgsql-replication init' first.` — +that is the tool running properly and telling you about §8.2, not about Python. + +### 8.2 Initialise, then run daily + +```bash +osm2pgsql-replication init \ + --database lupmis --schema osm --osm-file ghana-latest.osm.pbf +``` + +That reads the replication URL and sequence number out of the extract's header, +so updates start exactly where the downloaded file left off. + +Then daily, with the district stamp in the same job: + +```bash +osm2pgsql-replication update \ + --database lupmis --schema osm \ + --diff-file /var/osm/diffs/$(date +%F).osc.gz \ + -- --slim --output=flex --style sql/lupmis-osm.lua --extra-attributes \ + --schema osm --middle-schema osm \ +&& psql -d lupmis -f sql/stamp-districts.sql +``` + +The append updates the tables in place; there is no refresh step. `&&` rather +than `;` so a failed update does not leave the stamp running against half-applied +data. + +In cron, spell out the full path to every binary — `osm2pgsql-replication`, +`psql`, and the Python interpreter if §8.1 required one. A cron job does not +inherit your shell's `PATH`, and this is the most common reason a daily update +works when run by hand and silently never runs from cron. + +`--diff-file` keeps each day's changes. It is what lets you answer "what +changed near this parcel last week", which nothing else in this pipeline +records. + +--- + +## 9. Check it worked + +```sql +-- The column that started all this. +SELECT highway, count(*) FROM osm.pi_osm_roads + WHERE highway IS NOT NULL GROUP BY 1 ORDER BY 2 DESC LIMIT 12; + +-- Amenities mapped as areas — these could not appear in the old layout. +SELECT osm_type, count(*) FROM osm.pi_osm_points + WHERE amenity IS NOT NULL GROUP BY 1; + +-- Clinics carrying healthcare and no amenity, invisible until now. +SELECT count(*) FROM osm.pi_osm_points + WHERE amenity IS NULL AND healthcare IS NOT NULL; + +-- Nothing left unstamped. A non-zero count is usually genuine — offshore +-- features, or gaps between district polygons — but check before assuming. +SELECT count(*) FROM osm.pi_osm_roads WHERE districtid IS NULL; + +-- Comparable with the endpoint's 5,238 for district 1. +SELECT count(*) FROM osm.pi_osm_roads WHERE districtid = 1; + +-- The doors are open this time. +SELECT property, value FROM osm.osm2pgsql_properties + WHERE property IN ('updatable','attributes','output','current_timestamp'); +-- updatable and attributes must both be true. +``` + +Report freshness from `current_timestamp`, which moves with every append, not +`import_timestamp`, which is fixed at the original import. + +Once the endpoints serve from the views and the application is verified, the +renamed original can go: + +```sql +DROP TABLE spatial.pi_osm_roads_pre_osm2pgsql; +``` + +--- + +## 10. When it goes wrong + +Fetch a fresh extract, re-run §4 with `--create`, re-run §5.1 and +`osm2pgsql-replication init`. Nothing in `spatial` is touched by any of it, +which is the entire reason the import lives in its own schema. diff --git a/docs/external-layers-table.sql b/docs/external-layers-table.sql new file mode 100644 index 0000000..4d55798 --- /dev/null +++ b/docs/external-layers-table.sql @@ -0,0 +1,446 @@ +-- ============================================================================ +-- LUPMIS2 — external layer definitions +-- +-- Stores the layers a user adds through the map's "Add External Layer" dialog, +-- so that they persist, can be shared within a district, and survive a +-- re-install of the application. +-- +-- Two tables: +-- +-- spatial.hlp_layer_types the kinds of layer that can be added +-- (wms, wfs, xyz, cog, and whatever comes next) +-- spatial.es_external_layers one row per layer a user has added +-- +-- The four fields the dialog collects are layer_type, url, layer_name and +-- title. Everything else in es_external_layers exists for one of three reasons, +-- marked in the column comments: +-- +-- [dialog] captured directly from the Add External Layer form +-- [parity] needed to describe the external layers the application already +-- hard-codes; without these, moving those layers into the database +-- would lose information the map currently relies on +-- [admin] ownership, scoping, ordering and audit +-- +-- Target: PostgreSQL 10+ (uses GENERATED ALWAYS AS IDENTITY). +-- Schema: `spatial` is used for consistency with the rest of LUPMIS, although +-- neither table holds geometry — see the notes at the end. +-- +-- NOTE ON SEQUENCING. The application still hard-codes the four layer types in +-- the Add External Layer dialog and will keep doing so until the endpoint in +-- note 4 exists. Until then hlp_layer_types is the database's own record, and +-- the two must be kept in step by hand: adding a row here does not make the +-- type appear in the dialog, and adding a radio button there without a row +-- here will be rejected by the foreign key. +-- ============================================================================ + + +-- ============================================================================ +-- 1. Helper table — the available layer types +-- +-- This exists so a new kind of layer (WMTS, ArcGIS REST, vector tiles, a +-- GeoPackage URL) can be introduced by inserting a row, with no change to the +-- schema, no deployment and no code release on the database side. +-- +-- `hlp_` marks it as a lookup/helper table rather than data. +-- ============================================================================ + +CREATE TABLE spatial.hlp_layer_types ( + code text PRIMARY KEY, + -- the value stored in es_external_layers.layer_type and + -- sent in the API payload: lowercase, no spaces + label text NOT NULL, + -- what the dialog shows on the button, e.g. 'WMS' + description text, + -- one line explaining the type, for a tooltip or docs + + -- ---- rules the client and the database both need ----------------------- + requires_layer_name boolean NOT NULL DEFAULT false, + -- true for WMS/WFS, where a service publishes many + -- layers and one must be named. This is the rule that + -- used to be a hard-coded CHECK constraint. + layer_name_label text, + -- what to call that field for this type, e.g. + -- 'WMS LAYERS parameter' vs 'WFS typename' + url_placeholder text, + -- example URL shown in the empty form field + hint text, + -- extra guidance shown when the type is selected + + -- ---- housekeeping ------------------------------------------------------ + sort_order integer NOT NULL DEFAULT 0, + is_active boolean NOT NULL DEFAULT true, + -- false retires a type: no new layers may use it, but + -- existing layers of that type keep working + created_at timestamptz NOT NULL DEFAULT now(), + updated_at timestamptz NOT NULL DEFAULT now(), + + -- The code travels through URLs, JSON payloads and CSS class names, so it + -- is restricted to a plain lowercase identifier. + CONSTRAINT hlp_layer_types_code_chk + CHECK (code ~ '^[a-z][a-z0-9_]*$') +); + +COMMENT ON TABLE spatial.hlp_layer_types IS + 'Lookup of layer types available in the Add External Layer dialog. A new ' + 'type is added by inserting a row — no schema change required.'; +COMMENT ON COLUMN spatial.hlp_layer_types.requires_layer_name IS + 'Whether es_external_layers.layer_name is mandatory for this type. ' + 'Enforced by the trigger on that table, because a CHECK constraint cannot ' + 'read another table.'; +COMMENT ON COLUMN spatial.hlp_layer_types.is_active IS + 'false stops new layers being created with this type without invalidating ' + 'the ones that already exist.'; + +INSERT INTO spatial.hlp_layer_types + (code, label, description, + requires_layer_name, layer_name_label, url_placeholder, hint, sort_order) +VALUES + ('wms', 'WMS', 'OGC Web Map Service — server-rendered map images.', + true, 'WMS LAYERS parameter (e.g. workspace:layer)', + 'https://example.com/wms', NULL, 10), + + ('wfs', 'WFS', 'OGC Web Feature Service — vector features as GeoJSON.', + true, 'WFS typename (e.g. workspace:layer)', + 'https://example.com/wfs', NULL, 20), + + ('xyz', 'XYZ', 'Pre-rendered tile pyramid addressed by z/x/y.', + false, NULL, + 'https://example.com/tiles/{z}/{x}/{y}.png', NULL, 30), + + ('cog', 'COG', 'Cloud-Optimized GeoTIFF streamed by byte-range request.', + false, NULL, + 'https://example.com/data/elevation_cog.tif', + 'The server must allow CORS and byte-range requests.', 40); + + +-- ============================================================================ +-- 2. The layers themselves +-- ============================================================================ + +CREATE TABLE spatial.es_external_layers ( + id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + + -- ---- from the Add External Layer dialog ------------------------------- + layer_type text NOT NULL + REFERENCES spatial.hlp_layer_types (code) + ON UPDATE CASCADE ON DELETE RESTRICT, + -- [dialog] see hlp_layer_types + url text NOT NULL, -- [dialog] service or file URL + layer_name text, -- [dialog] required when the type + -- says so; see the trigger + title text NOT NULL, -- [dialog] name in the layer switcher + + -- ---- scoping and ownership ------------------------------------------- + districtid integer, -- [admin] NULL = every district + userid integer NOT NULL, -- [admin] who added it + is_shared boolean NOT NULL DEFAULT false, + -- [admin] false = creator only; + -- true = the district + + -- ---- rendering ------------------------------------------------------- + layer_group_id integer NOT NULL DEFAULT 8, + -- [admin] 8 = "External Sources" + style_name text, -- [parity] WMS STYLES parameter + server_type text, -- [parity] geoserver | mapserver | qgis | NULL + opacity numeric(3,2) NOT NULL DEFAULT 1.00, -- [parity] + z_index integer, -- [parity] negative sits behind overlays + visible_by_default boolean NOT NULL DEFAULT false, -- [parity] + online_only boolean NOT NULL DEFAULT false, + -- [parity] warn when toggled on offline + attribution text, -- [parity] credit/licence; often required + legend_url text, -- [parity] legend shown while visible + min_zoom smallint, -- [parity] + max_zoom smallint, -- [parity] + + -- ---- lifecycle and audit --------------------------------------------- + description text, -- [admin] free note about the source + is_active boolean NOT NULL DEFAULT true, -- [admin] soft delete + sort_order integer NOT NULL DEFAULT 0, -- [admin] + created_at timestamptz NOT NULL DEFAULT now(), -- [admin] + updated_at timestamptz NOT NULL DEFAULT now(), -- [admin] + updated_by integer, -- [admin] + + -- ---- constraints ------------------------------------------------------ + -- layer_type is validated by the foreign key above, and the conditional + -- layer_name rule by the trigger below, because a CHECK constraint cannot + -- read another table. + + -- The PWA is served over HTTPS, so a browser blocks any http:// sub-resource + -- as mixed content. An http URL cannot render, it just fails silently — so + -- it is rejected here rather than stored and puzzled over later. + CONSTRAINT es_external_layers_https_chk + CHECK (url ~* '^https://'), + + CONSTRAINT es_external_layers_opacity_chk + CHECK (opacity >= 0 AND opacity <= 1), + + CONSTRAINT es_external_layers_zoom_chk + CHECK (min_zoom IS NULL OR max_zoom IS NULL OR min_zoom <= max_zoom) +); + + +-- --------------------------------------------------------------------------- +-- The rule that moved out of the schema and into the data +-- +-- `requires_layer_name` now lives in hlp_layer_types, so it has to be enforced +-- somewhere that can read it. A CHECK constraint cannot; a trigger can. +-- +-- It raises check_violation deliberately, so any API error handling that +-- already distinguishes constraint failures keeps working unchanged. +-- +-- The is_active test applies to INSERT only: retiring a type should stop new +-- layers being created with it without freezing the ones that already exist. +-- --------------------------------------------------------------------------- +CREATE OR REPLACE FUNCTION spatial.es_external_layers_validate() +RETURNS trigger AS $$ +DECLARE + t spatial.hlp_layer_types%ROWTYPE; +BEGIN + SELECT * INTO t + FROM spatial.hlp_layer_types + WHERE code = NEW.layer_type; + -- The foreign key guarantees a row exists, so no NOT FOUND branch here. + + IF TG_OP = 'INSERT' AND NOT t.is_active THEN + RAISE EXCEPTION 'Layer type "%" is retired and cannot be used for new layers', + NEW.layer_type + USING ERRCODE = 'check_violation'; + END IF; + + IF t.requires_layer_name + AND (NEW.layer_name IS NULL OR btrim(NEW.layer_name) = '') THEN + RAISE EXCEPTION 'Layer type "%" requires layer_name (%)', + NEW.layer_type, COALESCE(t.layer_name_label, 'layer name') + USING ERRCODE = 'check_violation'; + END IF; + + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER es_external_layers_validate_trg + BEFORE INSERT OR UPDATE ON spatial.es_external_layers + FOR EACH ROW EXECUTE FUNCTION spatial.es_external_layers_validate(); + + +-- --------------------------------------------------------------------------- +-- Foreign keys to districts and users +-- +-- Left commented because the referenced table and column names were not +-- verifiable from the application side — it only ever sees districts and users +-- through the API. Confirm them, then enable: +-- +-- SELECT table_schema, table_name FROM information_schema.tables +-- WHERE table_name ILIKE '%district%' OR table_name ILIKE '%user%'; +-- +-- ON DELETE RESTRICT for the district: removing a district that still has +-- layers attached should be a decision, not a side effect. +-- --------------------------------------------------------------------------- +-- ALTER TABLE spatial.es_external_layers +-- ADD CONSTRAINT es_external_layers_district_fk +-- FOREIGN KEY (districtid) REFERENCES spatial.districts (districtid) +-- ON DELETE RESTRICT; +-- +-- ALTER TABLE spatial.es_external_layers +-- ADD CONSTRAINT es_external_layers_user_fk +-- FOREIGN KEY (userid) REFERENCES public.users (id) +-- ON DELETE RESTRICT; + + +-- --------------------------------------------------------------------------- +-- Stop the same layer being added twice +-- +-- COALESCE is needed because NULL never equals NULL: without it a duplicate +-- would be allowed whenever districtid or layer_name is NULL, which is exactly +-- the XYZ and COG case. +-- --------------------------------------------------------------------------- +CREATE UNIQUE INDEX es_external_layers_unique_idx + ON spatial.es_external_layers ( + COALESCE(districtid, -1), + layer_type, + url, + COALESCE(layer_name, '') + ) + WHERE is_active; + + +-- --------------------------------------------------------------------------- +-- Indexes for the read path +-- --------------------------------------------------------------------------- +-- The endpoint's main query: everything this district may see, which is the +-- district's own layers plus the global ones (districtid IS NULL). +CREATE INDEX es_external_layers_district_idx + ON spatial.es_external_layers (districtid, sort_order) + WHERE is_active; + +CREATE INDEX es_external_layers_user_idx + ON spatial.es_external_layers (userid) + WHERE is_active; + +-- Supports the ON DELETE RESTRICT check on hlp_layer_types, and "how many +-- layers use this type" questions. +CREATE INDEX es_external_layers_type_idx + ON spatial.es_external_layers (layer_type); + + +-- --------------------------------------------------------------------------- +-- Keep updated_at honest, on both tables +-- --------------------------------------------------------------------------- +CREATE OR REPLACE FUNCTION spatial.touch_updated_at() +RETURNS trigger AS $$ +BEGIN + NEW.updated_at := now(); + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +CREATE TRIGGER es_external_layers_touch_trg + BEFORE UPDATE ON spatial.es_external_layers + FOR EACH ROW EXECUTE FUNCTION spatial.touch_updated_at(); + +CREATE TRIGGER hlp_layer_types_touch_trg + BEFORE UPDATE ON spatial.hlp_layer_types + FOR EACH ROW EXECUTE FUNCTION spatial.touch_updated_at(); + + +-- --------------------------------------------------------------------------- +-- Documentation that travels with the schema +-- --------------------------------------------------------------------------- +COMMENT ON TABLE spatial.es_external_layers IS + 'External map layers added by users through the Add External Layer dialog. ' + 'One row per layer. Holds no geometry — the data stays on the remote server ' + 'and is fetched by the client at display time.'; + +COMMENT ON COLUMN spatial.es_external_layers.layer_type IS + 'References spatial.hlp_layer_types.code. Determines how the client builds ' + 'the request and which other columns apply.'; +COMMENT ON COLUMN spatial.es_external_layers.url IS + 'WMS/WFS service endpoint, XYZ template with {z}/{x}/{y}, or the URL of a ' + 'Cloud-Optimized GeoTIFF. Must be https — see the mixed-content constraint.'; +COMMENT ON COLUMN spatial.es_external_layers.layer_name IS + 'WMS LAYERS parameter or WFS typename, e.g. workspace:layer. Required only ' + 'for types whose hlp_layer_types.requires_layer_name is true.'; +COMMENT ON COLUMN spatial.es_external_layers.districtid IS + 'NULL means the layer is offered in every district. A value restricts it ' + 'to that district.'; +COMMENT ON COLUMN spatial.es_external_layers.is_shared IS + 'false: visible only to the user who created it. true: visible to everyone ' + 'within its district scope.'; +COMMENT ON COLUMN spatial.es_external_layers.is_active IS + 'Soft delete. The dialog marks user-added layers as removable, and a hard ' + 'delete would silently break the map for colleagues who rely on a shared ' + 'layer, with no way to tell what disappeared.'; +COMMENT ON COLUMN spatial.es_external_layers.online_only IS + 'The layer needs connectivity. The client uses this to explain, rather ' + 'than show an empty layer, when it is switched on in the field.'; +COMMENT ON COLUMN spatial.es_external_layers.z_index IS + 'Render order. Negative values push a layer behind the standard overlays — ' + 'used for background imagery such as slope or hillshade.'; + + +-- --------------------------------------------------------------------------- +-- Examples +-- +-- The first is a layer the application currently hard-codes. It is the reason +-- for the [parity] columns: without style_name, opacity, z_index, attribution, +-- legend_url and online_only, this layer could not be represented here. +-- --------------------------------------------------------------------------- +INSERT INTO spatial.es_external_layers + (layer_type, url, layer_name, title, districtid, userid, is_shared, + style_name, server_type, opacity, z_index, visible_by_default, + online_only, attribution, legend_url, description) +VALUES + ('wms', + 'https://ows.digitalearth.africa/wms', + 'srtm_deriv', + 'DEAfrica Slope (SRTM 30m)', + NULL, -- every district + 1, + true, + 'style_slope', + NULL, -- not a GeoServer; serverType must stay unset + 0.50, + -50, -- behind the standard overlays + false, + true, + '© Digital Earth Africa — SRTM-derived Slope', + 'https://ows.digitalearth.africa/legend/srtm_deriv/style_slope/legend.png', + 'Terrain slope derived from SRTM, served by Digital Earth Africa.'); + +-- A raster published to the LUSPA object store: no layer name, single file. +INSERT INTO spatial.es_external_layers + (layer_type, url, title, districtid, userid, is_shared, description) +VALUES + ('cog', + 'https://minioapi.lupmis4luspa.org/raster-objects/dem/koforidua_dem.tif', + 'Koforidua DEM', + 1, + 1, + true, + 'Digital elevation model for the district, published as a Cloud-Optimized GeoTIFF.'); + + +-- ============================================================================ +-- Adding a layer type later +-- +-- This is the whole point of the helper table. No schema change, no migration: +-- +-- INSERT INTO spatial.hlp_layer_types +-- (code, label, description, requires_layer_name, layer_name_label, +-- url_placeholder, sort_order) +-- VALUES +-- ('wmts', 'WMTS', 'OGC Web Map Tile Service — pre-rendered tiles.', +-- true, 'WMTS layer identifier', +-- 'https://example.com/wmts', 50); +-- +-- Retiring one, without invalidating the layers already using it: +-- +-- UPDATE spatial.hlp_layer_types SET is_active = false WHERE code = 'wfs'; +-- +-- Deleting a type that is still in use is refused by the foreign key, which is +-- the intended behaviour — retire it instead. +-- ============================================================================ + + +-- ============================================================================ +-- Notes for the database team +-- +-- 1. Schema choice. `spatial` matches the rest of LUPMIS, but neither table +-- holds geometry — they are configuration. If there is a schema for +-- application settings they belong there instead; only the qualifier changes. +-- +-- 2. Prefixes. `hlp_` marks a helper/lookup table, as requested. `es_` follows +-- the existing convention of naming a table after its layer group (`be_` +-- biophysical, `lu_` land use, `pi_` physical infrastructure); group 8 is +-- "External Sources". +-- +-- 3. The client still hard-codes the four types. Until the endpoint below +-- exists, the dialog in src/components/MapView.js and the rows in +-- hlp_layer_types must be changed together. Adding a row here alone will not +-- surface the type in the application; adding a radio button there alone +-- will fail the foreign key on save. +-- +-- 4. Suggested endpoints, in the shape of the existing spatial_planning ones +-- ({ api_token, district_id } in, { success, data: [...] } out): +-- +-- get_layer_types.php rows where is_active, ordered by sort_order +-- — this is what lets the dialog stop +-- hard-coding the list +-- get_external_layers.php rows where is_active +-- and (districtid = :district OR districtid IS NULL) +-- and (is_shared OR userid = :user) +-- ordered by sort_order, title +-- save_external_layer.php insert or update one row +-- delete_external_layer.php sets is_active = false, never DELETE +-- +-- 5. Authorisation is the part worth deciding before this ships. The read query +-- above assumes a user may see their own layers plus shared ones in their +-- district. Who may edit or deactivate a *shared* layer — only its creator, +-- or any user in the district — is a policy question this table can support +-- either way, through userid. +-- +-- 6. The url column holds a value the browser will fetch. Two consequences: +-- the https constraint is functional, not cosmetic; and the endpoint should +-- validate the URL server-side as well, since a stored URL is effectively an +-- instruction to every client that loads the layer. +-- ============================================================================