ekke/pwaLUPMIS2
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs for the database team: OSM re-import, external layer tables

Browse Source 
Two documents, both for work the database team will carry out. No application
code changes.

OSM re-import runbook
---------------------
The roads endpoint returns geom, districtid, osm_id, surface, oneway and name
for district 1 — no `highway`. The road class was discarded at import, which is
what osm2pgsql's default.style does to any tag not on its list, so roads cannot
be styled or filtered by class. A discarded tag cannot be recovered by querying;
the only fix is to import again, and if that is happening anyway it is worth
doing on terms that allow the data to be kept current afterwards.

The runbook covers a flex import with --slim and --extra-attributes, producing
four tables — roads, lines, points, polygons — plus daily replication. Two
decisions are specific to LUPMIS:

  * The import goes into its own `osm` schema. osm2pgsql --create drops and
    recreates every table it owns, and `spatial` holds lu_parcels; the API is
    given views in `spatial` instead, so the existing get_osm_roads.php keeps
    working and simply gains `highway`.
  * districtid is stamped by a spatial join after each import, since osm2pgsql
    has no concept of a district.

Executed against osm2pgsql 2.2.0 and a scratch PostGIS database with a hand-made
extract covering every shape the config handles. All four tables imported
cleanly; updatable and attributes both true; amenities mapped as node, closed
way and multipolygon relation all reached the points table, as did a
healthcare-tagged node with no amenity — three cases the current layout cannot
serve. An --append then applied a diff and left districtid NULL on the changed
row while untouched rows kept theirs, which is the behaviour the incremental
re-stamp in section 5.2 depends on.

Not verified: anything needing the LUPMIS database itself — the current import
state, the district boundary table's real name, row counts. Section 1 is the
set of checks to run first.

External layer tables
---------------------
DDL for storing layers added through the Add External Layer dialog:

  spatial.hlp_layer_types     available layer types (wms, wfs, xyz, cog)
  spatial.es_external_layers  one row per layer a user has added

Layer types live in a table rather than a CHECK constraint so a new kind can be
introduced by inserting a row. The rule that WMS and WFS require a layer name
moved there too as requires_layer_name — left in the schema it would have meant
a migration for every new type anyway. Validity is enforced by a foreign key and
the conditional rule by a trigger, since a CHECK cannot read another table; the
trigger raises check_violation so existing API error handling still applies.

Beyond the dialog's four fields the table carries districtid (NULL = every
district), userid and is_shared, plus the columns needed to represent the
external layers the application already hard-codes — style, opacity, z_index,
attribution, legend_url, online_only. Without those, moving the existing
DEAfrica slope layer into the database would lose information the map relies on;
it is included as a seed row to prove the schema can hold it.

Executed against PostgreSQL 16. Nine cases pass, including: a new type added as
data with no DDL and usable immediately, that type enforcing its own
requires_layer_name, a retired type blocked for new layers while existing ones
stay editable, and deletion of a type still in use being refused.

The application still hard-codes the four types and will until
get_layer_types.php exists, so the dialog and hlp_layer_types must be changed
together until then — noted in the file, as the two fail in opposite directions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
ekke 2026-08-20 10:42:16 +02:00
parent 664c242980
commit 8f48abc858
2 changed files with 1080 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

634
docs/LUPMIS-OSM-Import-Runbook.md Normal file
View File

@ -0,0 +1,634 @@
# LUPMIS2 — Re-importing the OSM data so it can be kept current
**For:** whoever runs the import (LUSPA database team)
**Date:** 11 August 2026
> **What has and has not been verified.**
>
> **Executed** — against osm2pgsql 2.2.0 and a scratch PostGIS database, using a
> hand-made extract covering every shape this config has to handle: an amenity
> node, a `healthcare`-only node, a `highway` way, a `waterway` way, an amenity
> mapped as a closed way, and one mapped as a multipolygon relation.
> The Lua config in §3 imported all four tables cleanly; `updatable` and
> `attributes` both came out `true`; all four amenity shapes reached
> `pi_osm_points`; geometries are EPSG:4326. A subsequent `--append` applied a
> diff correctly, and — the claim §5.2 depends on — left `districtid` **NULL on
> the changed row while untouched rows kept theirs**.
>
> **Not verified** — anything that needs the LUPMIS database, which is not
> reachable from where this was written: the current state of the import, the
> real name of the district boundary table, and the row counts. §1 is the set of
> checks to run before trusting those parts.
---
## Why this is needed
The LUPMIS roads endpoint returns, for district 1, 5,238 rows with these
columns:
```
geom, districtid, osm_id, surface, oneway, name
```
`highway` is not among them. The road class — trunk, primary, residential,
track — was discarded when the data was loaded. That is the normal outcome of
importing with osm2pgsql's `default.style`, which keeps a fixed list of tags
and silently drops everything else; the same mechanism discards `phone`,
`opening_hours` and `website`. The consequences are already visible in the
application:
- Roads cannot be styled or filtered by class. Every road is drawn identically,
so a trunk road and a footpath look the same at every zoom.
- `src/import-detect.js` lists `highway` as an expected field for the
`osm_roads` target, so the mapping UI offers a column the data does not have.
A discarded tag cannot be recovered by querying; it was never written. The only
way to get `highway` is to import again — and if the import is going to be
redone, it is worth redoing on terms that allow it to be kept current
afterwards, rather than repeating the same exercise in a year.
Whether the rest of the diagnosis applies — no middle tables, no replication
set up, a stale `import_timestamp` — is what §1 determines.
---
## What this import decides
Three decisions determine whether this import can be maintained or has to be
repeated by hand every time.
**The flex output**, configured by the Lua file in §3. The alternative — the
classic pgsql output that produces `planet_osm_point`, `_line`, `_polygon` and
`_roads` — is deprecated in osm2pgsql 2.x and warns on every run, and its tag
selection is governed by a style file that drops whatever it does not know.
Flex puts that choice in a configuration you own: the columns are the ones you
list, and everything else can be kept in a `jsonb` column instead of being
thrown away. It also writes one table per purpose rather than four fixed ones,
which is what allows an amenity mapped as a building outline to be served as a
point.
**`--slim`, without `--drop`.** This keeps the middle tables — osm2pgsql's own
record of every node, way and relation it has seen. They cost disk, and they
are the entire difference between an import that can accept a daily diff and
one that can only ever be replaced wholesale. An import run without them
records `updatable=false`, and no later flag will change that; the only repair
is a full re-import. `--drop` discards them, which is why it must not be used
here.
**`--extra-attributes`.** Records each object's OSM version and timestamp. That
makes it possible to say when a feature was last edited, and to answer "what
changed" at all.
Two further decisions are specific to LUPMIS.
**A separate `osm` schema — this one is not negotiable.** osm2pgsql `--create`
drops and recreates every table it owns in its schema. The `spatial` schema
holds `lu_parcels`, the hand-edited planning data this whole system exists to
manage. Pointing osm2pgsql at `spatial` puts a `--create` run one name
collision away from destroying it. Import into `osm`; expose to the API through
views in `spatial` (§7).
**Districts are stamped after import, not during.** osm2pgsql imports the
national Ghana extract and has no concept of a district. Every LUPMIS table and
endpoint is district-scoped, so a `districtid` column is filled by a spatial
join once the import finishes (§5). This has to run after every daily update as
well, and §5.2 explains why that turns out to be cheap.
---
## 1. Find out what you actually have
Run these before anything else. They decide whether this is a re-import or a
first proper import, and none of the answers can be guessed from outside.
```sql
-- Does an osm2pgsql import exist at all, and on what terms?
SELECT property, value FROM osm2pgsql_properties
ORDER BY property;
-- Look for: updatable, attributes, output, style,
-- import_timestamp, replication_base_url, replication_sequence_number
-- If this table does not exist, pi_osm_roads was loaded some other way
-- (ogr2ogr, a hand-written script, a one-off SQL dump).
-- Where does the current roads data live, and what is in it?
SELECT table_schema, table_name
FROM information_schema.tables
WHERE table_name LIKE '%osm%' OR table_name LIKE 'planet_osm%'
ORDER BY 1, 2;
SELECT column_name, data_type
FROM information_schema.columns
WHERE table_name = 'pi_osm_roads'
ORDER BY ordinal_position;
-- Middle tables present? Their absence is what makes --append impossible.
SELECT to_regclass('osm.planet_osm_nodes') AS nodes,
to_regclass('osm.planet_osm_ways') AS ways,
to_regclass('osm.planet_osm_rels') AS rels;
```
**If `osm2pgsql_properties` does not exist**, there is nothing to preserve or
migrate: skip §2 and import fresh. That is the more likely case here, because a
`pi_`-prefixed table in `spatial` with exactly six columns looks like a
purpose-built extract rather than anything osm2pgsql produced.
**If it does exist**, note `replication_base_url` and
`replication_sequence_number` before touching anything — even a non-updatable
import records them, and they tell a fresh import where the data left off.
---
## 2. Clear out only what is safe to clear
Nothing in this section touches `spatial`. The live application keeps reading
`spatial.pi_osm_roads` until §7 switches it over.
```sql
CREATE SCHEMA IF NOT EXISTS osm;
```
If a previous osm2pgsql run left pgsql-output tables in `osm`, drop these four
and **no others**:
```sql
DROP TABLE IF EXISTS osm.planet_osm_point,
osm.planet_osm_line,
osm.planet_osm_polygon,
osm.planet_osm_roads;
```
`planet_osm_nodes`, `planet_osm_ways`, `planet_osm_rels` and
`planet_osm_users` look like the same family and are not — they are the middle
tables that `--slim` exists to keep, and flex uses them exactly as the pgsql
output did. Dropping them costs you `--append`, which is the point of this
exercise.
Leave `osm2pgsql_properties` alone; `--create` rewrites it.
---
## 3. The flex configuration
Save as `sql/lupmis-osm.lua` next to wherever the import is run.
Four tables, matching the four datasets requested. Every geometry is EPSG:4326,
so the API serves what the table stores and nothing transforms on read. Each
table carries a `districtid` column that the import leaves empty and §5 fills.
```lua
-- lupmis-osm.lua — osm2pgsql flex config for LUPMIS2
--
-- Four tables in the `osm` schema:
-- pi_osm_roads routable `highway` ways (LineString)
-- pi_osm_lines other linear features (LineString)
-- pi_osm_points amenities and healthcare (Point)
-- pi_osm_polygons buildings, land use, areas (MultiPolygon)
--
-- districtid is declared here but never written by the import. It is filled by
-- the spatial join in §5 of the runbook. Because osm2pgsql re-inserts a row
-- whenever its object changes, an updated row comes back with districtid NULL —
-- which is what makes the incremental re-stamp in §5.2 both simple and correct.
local srid = 4326
local roads = osm2pgsql.define_table({
name = 'pi_osm_roads', schema = 'osm',
ids = { type = 'way', id_column = 'osm_id' },
columns = {
{ column = 'highway', type = 'text' }, -- the column that was missing
{ column = 'name', type = 'text' },
{ column = 'ref', type = 'text' },
{ column = 'surface', type = 'text' },
{ column = 'oneway', type = 'text' },
{ column = 'bridge', type = 'text' },
{ column = 'tunnel', type = 'text' },
{ column = 'layer', type = 'text' },
{ column = 'districtid', type = 'int' },
{ column = 'tags', type = 'jsonb' },
{ column = 'geom', type = 'linestring', projection = srid, not_null = true },
}
})
local lines = osm2pgsql.define_table({
name = 'pi_osm_lines', schema = 'osm',
ids = { type = 'way', id_column = 'osm_id' },
columns = {
{ column = 'waterway', type = 'text' },
{ column = 'railway', type = 'text' },
{ column = 'power', type = 'text' },
{ column = 'barrier', type = 'text' },
{ column = 'name', type = 'text' },
{ column = 'districtid', type = 'int' },
{ column = 'tags', type = 'jsonb' },
{ column = 'geom', type = 'linestring', projection = srid, not_null = true },
}
})
-- Points accept nodes, ways and relations, so an amenity mapped as a building
-- outline or a multipolygon appears here as a point like any other. That is the
-- single biggest gain over the classic layout, where areas lived in
-- planet_osm_polygon and the endpoint never looked there.
local points = osm2pgsql.define_table({
name = 'pi_osm_points', schema = 'osm',
ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
columns = {
{ column = 'amenity', type = 'text' },
{ column = 'healthcare', type = 'text' },
{ column = 'name', type = 'text' },
{ column = 'districtid', type = 'int' },
{ column = 'geom', type = 'point', projection = srid, not_null = true },
}
})
local polygons = osm2pgsql.define_table({
name = 'pi_osm_polygons', schema = 'osm',
ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
columns = {
{ column = 'building', type = 'text' },
{ column = 'landuse', type = 'text' },
{ column = 'amenity', type = 'text' },
{ column = 'leisure', type = 'text' },
{ column = 'natural', type = 'text' },
{ column = 'name', type = 'text' },
{ column = 'districtid', type = 'int' },
{ column = 'tags', type = 'jsonb' },
{ column = 'geom', type = 'multipolygon', projection = srid, not_null = true },
}
})
-- Tags that say nothing once the typed columns exist.
local uninteresting = {
'created_by', 'source', 'source:date', 'note', 'comment',
'fixme', 'FIXME', 'attribution',
}
local function clean(tags)
for _, k in ipairs(uninteresting) do tags[k] = nil end
end
local function is_area(tags)
return tags.area == 'yes'
or tags.building or tags.landuse or tags.leisure or tags.natural
end
function osm2pgsql.process_node(object)
local t = object.tags
if t.amenity or t.healthcare then
points:insert({
amenity = t.amenity,
healthcare = t.healthcare,
name = t.name,
geom = object:as_point(),
})
end
end
function osm2pgsql.process_way(object)
local t = object.tags
clean(t)
if object.is_closed and is_area(t) then
polygons:insert({
building = t.building, landuse = t.landuse, amenity = t.amenity,
leisure = t.leisure, natural = t['natural'], name = t.name,
tags = t, geom = object:as_polygon(),
})
-- An amenity mapped as an area is also a point, so it is findable
-- alongside amenities mapped as nodes.
if t.amenity or t.healthcare then
points:insert({
amenity = t.amenity, healthcare = t.healthcare, name = t.name,
geom = object:as_polygon():centroid(),
})
end
return
end
if t.highway then
roads:insert({
highway = t.highway, name = t.name, ref = t.ref,
surface = t.surface, oneway = t.oneway,
bridge = t.bridge, tunnel = t.tunnel, layer = t.layer,
tags = t, geom = object:as_linestring(),
})
elseif t.waterway or t.railway or t.power or t.barrier then
lines:insert({
waterway = t.waterway, railway = t.railway,
power = t.power, barrier = t.barrier, name = t.name,
tags = t, geom = object:as_linestring(),
})
end
end
function osm2pgsql.process_relation(object)
local t = object.tags
clean(t)
if t.type ~= 'multipolygon' and t.type ~= 'boundary' then return end
if is_area(t) or t.amenity or t.healthcare then
polygons:insert({
building = t.building, landuse = t.landuse, amenity = t.amenity,
leisure = t.leisure, natural = t['natural'], name = t.name,
tags = t, geom = object:as_multipolygon(),
})
if t.amenity or t.healthcare then
points:insert({
amenity = t.amenity, healthcare = t.healthcare, name = t.name,
geom = object:as_multipolygon():centroid(),
})
end
end
end
```
---
## 4. Import
```bash
curl -O https://download.geofabrik.de/africa/ghana-latest.osm.pbf
osm2pgsql --create --slim --output=flex \
--style sql/lupmis-osm.lua --extra-attributes \
--database lupmis --schema osm --middle-schema osm \
--cache 2000 \
ghana-latest.osm.pbf
```
`--slim` **without** `--drop`. `--drop` discards the middle tables and is what
makes an import permanently un-updatable.
Substitute the real database name for `lupmis`.
---
## 5. Stamp the districts
osm2pgsql knows nothing about districts, so this step has no equivalent in the
standard OSM tooling. It is what makes a national import usable by an
application where every table and every endpoint is district-scoped.
### 5.1 First pass, after the initial import
Confirm the boundary table's real name and column first — the application only
ever sees it through `get_district_boundary.php`:
```sql
SELECT table_schema, table_name FROM information_schema.tables
WHERE table_name ILIKE '%district%';
```
Then, for each of the four tables (shown here for roads):
```sql
UPDATE osm.pi_osm_roads r
SET districtid = d.districtid
FROM spatial.districts d -- confirm this name first
WHERE r.districtid IS NULL
AND ST_Intersects(d.geom, r.geom);
```
A linear feature crossing a district boundary matches more than one district.
`UPDATE` takes an arbitrary one of them, which is wrong for a road that spans a
boundary. If roads must appear in every district they touch, use a join table
instead of a column:
```sql
CREATE TABLE osm.pi_osm_roads_district AS
SELECT r.osm_id, d.districtid
FROM osm.pi_osm_roads r
JOIN spatial.districts d ON ST_Intersects(d.geom, r.geom);
CREATE INDEX ON osm.pi_osm_roads_district (districtid);
```
Points never have this problem. For polygons, decide whether a district should
own an area by intersection or by where its centroid falls —
`ST_Intersects(d.geom, ST_Centroid(p.geom))` gives one district per polygon.
### 5.2 After every update
The same `UPDATE` again — `WHERE districtid IS NULL` is doing real work here,
not just guarding against repetition.
When an object changes, osm2pgsql deletes its row and re-inserts it from the
new data. The Lua config never writes `districtid`, so the re-inserted row
comes back **NULL**. `WHERE districtid IS NULL` therefore selects exactly the
objects that are new or were edited since the last stamp — including any that
moved across a boundary — and nothing else. A daily re-stamp touches a few
hundred rows rather than millions.
Put it in the same cron entry as the update, immediately after it. If the two
ever get separated, the symptom is new roads that no district can see.
---
## 6. Indexes
osm2pgsql creates the geometry and id indexes. Add what the endpoints filter on:
```sql
CREATE INDEX ON osm.pi_osm_roads (districtid);
CREATE INDEX ON osm.pi_osm_lines (districtid);
CREATE INDEX ON osm.pi_osm_points (districtid);
CREATE INDEX ON osm.pi_osm_polygons (districtid);
CREATE INDEX ON osm.pi_osm_roads (highway) WHERE highway IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (amenity) WHERE amenity IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (healthcare) WHERE healthcare IS NOT NULL;
CREATE INDEX ON osm.pi_osm_roads USING gin (tags);
CREATE INDEX ON osm.pi_osm_polygons USING gin (tags);
```
---
## 7. Expose it to the API
Keep osm2pgsql's tables in `osm`, and give the API views in `spatial` under the
existing `pi_` naming. The application then reads the names it already expects,
and no import can ever write into `spatial`.
The current `spatial.pi_osm_roads` is a table, and a view cannot replace a table
of the same name. Rename it rather than dropping it, so there is a way back
until the new endpoints have been verified:
```sql
ALTER TABLE spatial.pi_osm_roads RENAME TO pi_osm_roads_pre_osm2pgsql;
CREATE VIEW spatial.pi_osm_roads AS
SELECT osm_id, highway, name, ref, surface, oneway, districtid, geom
FROM osm.pi_osm_roads;
CREATE VIEW spatial.pi_osm_points AS
SELECT osm_type, osm_id, amenity, healthcare, name, districtid, geom
FROM osm.pi_osm_points;
CREATE VIEW spatial.pi_osm_lines AS
SELECT osm_id, waterway, railway, power, barrier, name, districtid, geom
FROM osm.pi_osm_lines;
CREATE VIEW spatial.pi_osm_polygons AS
SELECT osm_type, osm_id, building, landuse, amenity, leisure, name, districtid, geom
FROM osm.pi_osm_polygons;
```
The existing `get_osm_roads.php` keeps working unchanged and gains `highway`,
because it reads `spatial.pi_osm_roads` and the view now supplies that column.
Three new endpoints are needed, in the shape of the existing one — same
`{ api_token, district_id }` request, same `{ success, data: [...] }` response,
geometry as WKT in `geom`:
| Endpoint | Serves | Application layer |
|---|---|---|
| `get_osm_points.php` | `spatial.pi_osm_points` | OSM Points |
| `get_osm_lines.php` | `spatial.pi_osm_lines` | OSM Lines |
| `get_osm_polygons.php` | `spatial.pi_osm_polygons` | OSM Polygons |
All four belong to layer group **5, Physical Infrastructures**, which is where
`OSM_roads` already sits.
**Volumes are the thing to watch.** Roads alone are 5,238 rows for district 1
under the current extract, and the endpoint returns every row as WKT in one
response. Points and polygons will be larger. Before wiring the new layers into
the application, decide whether these endpoints should take a bounding box, or
a `highway`/`amenity` filter, rather than returning a whole district. That
decision belongs with the endpoints, not the application.
---
## 8. Keep it current
This is the part the import exists for. Everything above only pays off if the
daily update actually runs.
### 8.1 osm2pgsql-replication needs a Python that has its libraries
`osm2pgsql-replication` ships with osm2pgsql but is a **Python** script, and
package managers do not install what it imports. Its first run says:
```
Missing required Python libraries psycopg2 osmium.
To install them via pip run: pip install psycopg2 osmium
```
Following that advice often changes nothing, and it is worth understanding why
before chasing it. The script's shebang is `#!/usr/bin/env python3`, and **`env`
does not see shell aliases**. If `python3` in your shell is aliased — a MAMP
installation does exactly this, and MAMP is present on at least one machine in
this project — then `pip3 install` puts the libraries where the script will
never look. Check before doing anything:
```bash
env python3 -c "import sys; print(sys.executable)"
env python3 -c "import psycopg2, osmium; print('both present')"
```
A Homebrew Python is also likely to be PEP 668 "externally managed", which
refuses the install outright.
**On the Linux server, use distribution packages** — no virtualenv, no pip:
```bash
apt install python3-psycopg2 python3-pyosmium # Debian/Ubuntu
```
**On a workstation**, give the script an interpreter that has both, once:
```bash
python3.12 -m venv ~/.venvs/osm2pgsql
~/.venvs/osm2pgsql/bin/pip install psycopg2-binary osmium
```
`psycopg2-binary` rather than `psycopg2`: it ships as a wheel and needs no
`pg_config` or compiler. Then invoke the tool with that interpreter:
```bash
~/.venvs/osm2pgsql/bin/python $(brew --prefix)/bin/osm2pgsql-replication …
```
You know it is working when the message changes from the missing-library error
to `Updates not set up correctly. Run 'osm2pgsql-replication init' first.` —
that is the tool running properly and telling you about §8.2, not about Python.
### 8.2 Initialise, then run daily
```bash
osm2pgsql-replication init \
--database lupmis --schema osm --osm-file ghana-latest.osm.pbf
```
That reads the replication URL and sequence number out of the extract's header,
so updates start exactly where the downloaded file left off.
Then daily, with the district stamp in the same job:
```bash
osm2pgsql-replication update \
--database lupmis --schema osm \
--diff-file /var/osm/diffs/$(date +%F).osc.gz \
-- --slim --output=flex --style sql/lupmis-osm.lua --extra-attributes \
--schema osm --middle-schema osm \
&& psql -d lupmis -f sql/stamp-districts.sql
```
The append updates the tables in place; there is no refresh step. `&&` rather
than `;` so a failed update does not leave the stamp running against half-applied
data.
In cron, spell out the full path to every binary — `osm2pgsql-replication`,
`psql`, and the Python interpreter if §8.1 required one. A cron job does not
inherit your shell's `PATH`, and this is the most common reason a daily update
works when run by hand and silently never runs from cron.
`--diff-file` keeps each day's changes. It is what lets you answer "what
changed near this parcel last week", which nothing else in this pipeline
records.
---
## 9. Check it worked
```sql
-- The column that started all this.
SELECT highway, count(*) FROM osm.pi_osm_roads
WHERE highway IS NOT NULL GROUP BY 1 ORDER BY 2 DESC LIMIT 12;
-- Amenities mapped as areas — these could not appear in the old layout.
SELECT osm_type, count(*) FROM osm.pi_osm_points
WHERE amenity IS NOT NULL GROUP BY 1;
-- Clinics carrying healthcare and no amenity, invisible until now.
SELECT count(*) FROM osm.pi_osm_points
WHERE amenity IS NULL AND healthcare IS NOT NULL;
-- Nothing left unstamped. A non-zero count is usually genuine — offshore
-- features, or gaps between district polygons — but check before assuming.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid IS NULL;
-- Comparable with the endpoint's 5,238 for district 1.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid = 1;
-- The doors are open this time.
SELECT property, value FROM osm.osm2pgsql_properties
WHERE property IN ('updatable','attributes','output','current_timestamp');
-- updatable and attributes must both be true.
```
Report freshness from `current_timestamp`, which moves with every append, not
`import_timestamp`, which is fixed at the original import.
Once the endpoints serve from the views and the application is verified, the
renamed original can go:
```sql
DROP TABLE spatial.pi_osm_roads_pre_osm2pgsql;
```
---
## 10. When it goes wrong
Fetch a fresh extract, re-run §4 with `--create`, re-run §5.1 and
`osm2pgsql-replication init`. Nothing in `spatial` is touched by any of it,
which is the entire reason the import lives in its own schema.

446
docs/external-layers-table.sql Normal file
View File

@ -0,0 +1,446 @@
-- ============================================================================
-- LUPMIS2 — external layer definitions
--
-- Stores the layers a user adds through the map's "Add External Layer" dialog,
-- so that they persist, can be shared within a district, and survive a
-- re-install of the application.
--
-- Two tables:
--
-- spatial.hlp_layer_types the kinds of layer that can be added
-- (wms, wfs, xyz, cog, and whatever comes next)
-- spatial.es_external_layers one row per layer a user has added
--
-- The four fields the dialog collects are layer_type, url, layer_name and
-- title. Everything else in es_external_layers exists for one of three reasons,
-- marked in the column comments:
--
-- [dialog] captured directly from the Add External Layer form
-- [parity] needed to describe the external layers the application already
-- hard-codes; without these, moving those layers into the database
-- would lose information the map currently relies on
-- [admin] ownership, scoping, ordering and audit
--
-- Target: PostgreSQL 10+ (uses GENERATED ALWAYS AS IDENTITY).
-- Schema: `spatial` is used for consistency with the rest of LUPMIS, although
-- neither table holds geometry — see the notes at the end.
--
-- NOTE ON SEQUENCING. The application still hard-codes the four layer types in
-- the Add External Layer dialog and will keep doing so until the endpoint in
-- note 4 exists. Until then hlp_layer_types is the database's own record, and
-- the two must be kept in step by hand: adding a row here does not make the
-- type appear in the dialog, and adding a radio button there without a row
-- here will be rejected by the foreign key.
-- ============================================================================
-- ============================================================================
-- 1. Helper table — the available layer types
--
-- This exists so a new kind of layer (WMTS, ArcGIS REST, vector tiles, a
-- GeoPackage URL) can be introduced by inserting a row, with no change to the
-- schema, no deployment and no code release on the database side.
--
-- `hlp_` marks it as a lookup/helper table rather than data.
-- ============================================================================
CREATE TABLE spatial.hlp_layer_types (
code text PRIMARY KEY,
-- the value stored in es_external_layers.layer_type and
-- sent in the API payload: lowercase, no spaces
label text NOT NULL,
-- what the dialog shows on the button, e.g. 'WMS'
description text,
-- one line explaining the type, for a tooltip or docs
-- ---- rules the client and the database both need -----------------------
requires_layer_name boolean NOT NULL DEFAULT false,
-- true for WMS/WFS, where a service publishes many
-- layers and one must be named. This is the rule that
-- used to be a hard-coded CHECK constraint.
layer_name_label text,
-- what to call that field for this type, e.g.
-- 'WMS LAYERS parameter' vs 'WFS typename'
url_placeholder text,
-- example URL shown in the empty form field
hint text,
-- extra guidance shown when the type is selected
-- ---- housekeeping ------------------------------------------------------
sort_order integer NOT NULL DEFAULT 0,
is_active boolean NOT NULL DEFAULT true,
-- false retires a type: no new layers may use it, but
-- existing layers of that type keep working
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
-- The code travels through URLs, JSON payloads and CSS class names, so it
-- is restricted to a plain lowercase identifier.
CONSTRAINT hlp_layer_types_code_chk
CHECK (code ~ '^[a-z][a-z0-9_]*$')
);
COMMENT ON TABLE spatial.hlp_layer_types IS
'Lookup of layer types available in the Add External Layer dialog. A new '
'type is added by inserting a row — no schema change required.';
COMMENT ON COLUMN spatial.hlp_layer_types.requires_layer_name IS
'Whether es_external_layers.layer_name is mandatory for this type. '
'Enforced by the trigger on that table, because a CHECK constraint cannot '
'read another table.';
COMMENT ON COLUMN spatial.hlp_layer_types.is_active IS
'false stops new layers being created with this type without invalidating '
'the ones that already exist.';
INSERT INTO spatial.hlp_layer_types
(code, label, description,
requires_layer_name, layer_name_label, url_placeholder, hint, sort_order)
VALUES
('wms', 'WMS', 'OGC Web Map Service — server-rendered map images.',
true, 'WMS LAYERS parameter (e.g. workspace:layer)',
'https://example.com/wms', NULL, 10),
('wfs', 'WFS', 'OGC Web Feature Service — vector features as GeoJSON.',
true, 'WFS typename (e.g. workspace:layer)',
'https://example.com/wfs', NULL, 20),
('xyz', 'XYZ', 'Pre-rendered tile pyramid addressed by z/x/y.',
false, NULL,
'https://example.com/tiles/{z}/{x}/{y}.png', NULL, 30),
('cog', 'COG', 'Cloud-Optimized GeoTIFF streamed by byte-range request.',
false, NULL,
'https://example.com/data/elevation_cog.tif',
'The server must allow CORS and byte-range requests.', 40);
-- ============================================================================
-- 2. The layers themselves
-- ============================================================================
CREATE TABLE spatial.es_external_layers (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
-- ---- from the Add External Layer dialog -------------------------------
layer_type text NOT NULL
REFERENCES spatial.hlp_layer_types (code)
ON UPDATE CASCADE ON DELETE RESTRICT,
-- [dialog] see hlp_layer_types
url text NOT NULL, -- [dialog] service or file URL
layer_name text, -- [dialog] required when the type
-- says so; see the trigger
title text NOT NULL, -- [dialog] name in the layer switcher
-- ---- scoping and ownership -------------------------------------------
districtid integer, -- [admin] NULL = every district
userid integer NOT NULL, -- [admin] who added it
is_shared boolean NOT NULL DEFAULT false,
-- [admin] false = creator only;
-- true = the district
-- ---- rendering -------------------------------------------------------
layer_group_id integer NOT NULL DEFAULT 8,
-- [admin] 8 = "External Sources"
style_name text, -- [parity] WMS STYLES parameter
server_type text, -- [parity] geoserver | mapserver | qgis | NULL
opacity numeric(3,2) NOT NULL DEFAULT 1.00, -- [parity]
z_index integer, -- [parity] negative sits behind overlays
visible_by_default boolean NOT NULL DEFAULT false, -- [parity]
online_only boolean NOT NULL DEFAULT false,
-- [parity] warn when toggled on offline
attribution text, -- [parity] credit/licence; often required
legend_url text, -- [parity] legend shown while visible
min_zoom smallint, -- [parity]
max_zoom smallint, -- [parity]
-- ---- lifecycle and audit ---------------------------------------------
description text, -- [admin] free note about the source
is_active boolean NOT NULL DEFAULT true, -- [admin] soft delete
sort_order integer NOT NULL DEFAULT 0, -- [admin]
created_at timestamptz NOT NULL DEFAULT now(), -- [admin]
updated_at timestamptz NOT NULL DEFAULT now(), -- [admin]
updated_by integer, -- [admin]
-- ---- constraints ------------------------------------------------------
-- layer_type is validated by the foreign key above, and the conditional
-- layer_name rule by the trigger below, because a CHECK constraint cannot
-- read another table.
-- The PWA is served over HTTPS, so a browser blocks any http:// sub-resource
-- as mixed content. An http URL cannot render, it just fails silently — so
-- it is rejected here rather than stored and puzzled over later.
CONSTRAINT es_external_layers_https_chk
CHECK (url ~* '^https://'),
CONSTRAINT es_external_layers_opacity_chk
CHECK (opacity >= 0 AND opacity <= 1),
CONSTRAINT es_external_layers_zoom_chk
CHECK (min_zoom IS NULL OR max_zoom IS NULL OR min_zoom <= max_zoom)
);
-- ---------------------------------------------------------------------------
-- The rule that moved out of the schema and into the data
--
-- `requires_layer_name` now lives in hlp_layer_types, so it has to be enforced
-- somewhere that can read it. A CHECK constraint cannot; a trigger can.
--
-- It raises check_violation deliberately, so any API error handling that
-- already distinguishes constraint failures keeps working unchanged.
--
-- The is_active test applies to INSERT only: retiring a type should stop new
-- layers being created with it without freezing the ones that already exist.
-- ---------------------------------------------------------------------------
CREATE OR REPLACE FUNCTION spatial.es_external_layers_validate()
RETURNS trigger AS $$
DECLARE
t spatial.hlp_layer_types%ROWTYPE;
BEGIN
SELECT * INTO t
FROM spatial.hlp_layer_types
WHERE code = NEW.layer_type;
-- The foreign key guarantees a row exists, so no NOT FOUND branch here.
IF TG_OP = 'INSERT' AND NOT t.is_active THEN
RAISE EXCEPTION 'Layer type "%" is retired and cannot be used for new layers',
NEW.layer_type
USING ERRCODE = 'check_violation';
END IF;
IF t.requires_layer_name
AND (NEW.layer_name IS NULL OR btrim(NEW.layer_name) = '') THEN
RAISE EXCEPTION 'Layer type "%" requires layer_name (%)',
NEW.layer_type, COALESCE(t.layer_name_label, 'layer name')
USING ERRCODE = 'check_violation';
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER es_external_layers_validate_trg
BEFORE INSERT OR UPDATE ON spatial.es_external_layers
FOR EACH ROW EXECUTE FUNCTION spatial.es_external_layers_validate();
-- ---------------------------------------------------------------------------
-- Foreign keys to districts and users
--
-- Left commented because the referenced table and column names were not
-- verifiable from the application side — it only ever sees districts and users
-- through the API. Confirm them, then enable:
--
-- SELECT table_schema, table_name FROM information_schema.tables
-- WHERE table_name ILIKE '%district%' OR table_name ILIKE '%user%';
--
-- ON DELETE RESTRICT for the district: removing a district that still has
-- layers attached should be a decision, not a side effect.
-- ---------------------------------------------------------------------------
-- ALTER TABLE spatial.es_external_layers
-- ADD CONSTRAINT es_external_layers_district_fk
-- FOREIGN KEY (districtid) REFERENCES spatial.districts (districtid)
-- ON DELETE RESTRICT;
--
-- ALTER TABLE spatial.es_external_layers
-- ADD CONSTRAINT es_external_layers_user_fk
-- FOREIGN KEY (userid) REFERENCES public.users (id)
-- ON DELETE RESTRICT;
-- ---------------------------------------------------------------------------
-- Stop the same layer being added twice
--
-- COALESCE is needed because NULL never equals NULL: without it a duplicate
-- would be allowed whenever districtid or layer_name is NULL, which is exactly
-- the XYZ and COG case.
-- ---------------------------------------------------------------------------
CREATE UNIQUE INDEX es_external_layers_unique_idx
ON spatial.es_external_layers (
COALESCE(districtid, -1),
layer_type,
url,
COALESCE(layer_name, '')
)
WHERE is_active;
-- ---------------------------------------------------------------------------
-- Indexes for the read path
-- ---------------------------------------------------------------------------
-- The endpoint's main query: everything this district may see, which is the
-- district's own layers plus the global ones (districtid IS NULL).
CREATE INDEX es_external_layers_district_idx
ON spatial.es_external_layers (districtid, sort_order)
WHERE is_active;
CREATE INDEX es_external_layers_user_idx
ON spatial.es_external_layers (userid)
WHERE is_active;
-- Supports the ON DELETE RESTRICT check on hlp_layer_types, and "how many
-- layers use this type" questions.
CREATE INDEX es_external_layers_type_idx
ON spatial.es_external_layers (layer_type);
-- ---------------------------------------------------------------------------
-- Keep updated_at honest, on both tables
-- ---------------------------------------------------------------------------
CREATE OR REPLACE FUNCTION spatial.touch_updated_at()
RETURNS trigger AS $$
BEGIN
NEW.updated_at := now();
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER es_external_layers_touch_trg
BEFORE UPDATE ON spatial.es_external_layers
FOR EACH ROW EXECUTE FUNCTION spatial.touch_updated_at();
CREATE TRIGGER hlp_layer_types_touch_trg
BEFORE UPDATE ON spatial.hlp_layer_types
FOR EACH ROW EXECUTE FUNCTION spatial.touch_updated_at();
-- ---------------------------------------------------------------------------
-- Documentation that travels with the schema
-- ---------------------------------------------------------------------------
COMMENT ON TABLE spatial.es_external_layers IS
'External map layers added by users through the Add External Layer dialog. '
'One row per layer. Holds no geometry — the data stays on the remote server '
'and is fetched by the client at display time.';
COMMENT ON COLUMN spatial.es_external_layers.layer_type IS
'References spatial.hlp_layer_types.code. Determines how the client builds '
'the request and which other columns apply.';
COMMENT ON COLUMN spatial.es_external_layers.url IS
'WMS/WFS service endpoint, XYZ template with {z}/{x}/{y}, or the URL of a '
'Cloud-Optimized GeoTIFF. Must be https — see the mixed-content constraint.';
COMMENT ON COLUMN spatial.es_external_layers.layer_name IS
'WMS LAYERS parameter or WFS typename, e.g. workspace:layer. Required only '
'for types whose hlp_layer_types.requires_layer_name is true.';
COMMENT ON COLUMN spatial.es_external_layers.districtid IS
'NULL means the layer is offered in every district. A value restricts it '
'to that district.';
COMMENT ON COLUMN spatial.es_external_layers.is_shared IS
'false: visible only to the user who created it. true: visible to everyone '
'within its district scope.';
COMMENT ON COLUMN spatial.es_external_layers.is_active IS
'Soft delete. The dialog marks user-added layers as removable, and a hard '
'delete would silently break the map for colleagues who rely on a shared '
'layer, with no way to tell what disappeared.';
COMMENT ON COLUMN spatial.es_external_layers.online_only IS
'The layer needs connectivity. The client uses this to explain, rather '
'than show an empty layer, when it is switched on in the field.';
COMMENT ON COLUMN spatial.es_external_layers.z_index IS
'Render order. Negative values push a layer behind the standard overlays — '
'used for background imagery such as slope or hillshade.';
-- ---------------------------------------------------------------------------
-- Examples
--
-- The first is a layer the application currently hard-codes. It is the reason
-- for the [parity] columns: without style_name, opacity, z_index, attribution,
-- legend_url and online_only, this layer could not be represented here.
-- ---------------------------------------------------------------------------
INSERT INTO spatial.es_external_layers
(layer_type, url, layer_name, title, districtid, userid, is_shared,
style_name, server_type, opacity, z_index, visible_by_default,
online_only, attribution, legend_url, description)
VALUES
('wms',
'https://ows.digitalearth.africa/wms',
'srtm_deriv',
'DEAfrica Slope (SRTM 30m)',
NULL, -- every district
1,
true,
'style_slope',
NULL, -- not a GeoServer; serverType must stay unset
0.50,
-50, -- behind the standard overlays
false,
true,
'&copy; <a href="https://www.digitalearthafrica.org/">Digital Earth Africa</a> — SRTM-derived Slope',
'https://ows.digitalearth.africa/legend/srtm_deriv/style_slope/legend.png',
'Terrain slope derived from SRTM, served by Digital Earth Africa.');
-- A raster published to the LUSPA object store: no layer name, single file.
INSERT INTO spatial.es_external_layers
(layer_type, url, title, districtid, userid, is_shared, description)
VALUES
('cog',
'https://minioapi.lupmis4luspa.org/raster-objects/dem/koforidua_dem.tif',
'Koforidua DEM',
1,
1,
true,
'Digital elevation model for the district, published as a Cloud-Optimized GeoTIFF.');
-- ============================================================================
-- Adding a layer type later
--
-- This is the whole point of the helper table. No schema change, no migration:
--
-- INSERT INTO spatial.hlp_layer_types
-- (code, label, description, requires_layer_name, layer_name_label,
-- url_placeholder, sort_order)
-- VALUES
-- ('wmts', 'WMTS', 'OGC Web Map Tile Service — pre-rendered tiles.',
-- true, 'WMTS layer identifier',
-- 'https://example.com/wmts', 50);
--
-- Retiring one, without invalidating the layers already using it:
--
-- UPDATE spatial.hlp_layer_types SET is_active = false WHERE code = 'wfs';
--
-- Deleting a type that is still in use is refused by the foreign key, which is
-- the intended behaviour — retire it instead.
-- ============================================================================
-- ============================================================================
-- Notes for the database team
--
-- 1. Schema choice. `spatial` matches the rest of LUPMIS, but neither table
-- holds geometry — they are configuration. If there is a schema for
-- application settings they belong there instead; only the qualifier changes.
--
-- 2. Prefixes. `hlp_` marks a helper/lookup table, as requested. `es_` follows
-- the existing convention of naming a table after its layer group (`be_`
-- biophysical, `lu_` land use, `pi_` physical infrastructure); group 8 is
-- "External Sources".
--
-- 3. The client still hard-codes the four types. Until the endpoint below
-- exists, the dialog in src/components/MapView.js and the rows in
-- hlp_layer_types must be changed together. Adding a row here alone will not
-- surface the type in the application; adding a radio button there alone
-- will fail the foreign key on save.
--
-- 4. Suggested endpoints, in the shape of the existing spatial_planning ones
-- ({ api_token, district_id } in, { success, data: [...] } out):
--
-- get_layer_types.php rows where is_active, ordered by sort_order
-- — this is what lets the dialog stop
-- hard-coding the list
-- get_external_layers.php rows where is_active
-- and (districtid = :district OR districtid IS NULL)
-- and (is_shared OR userid = :user)
-- ordered by sort_order, title
-- save_external_layer.php insert or update one row
-- delete_external_layer.php sets is_active = false, never DELETE
--
-- 5. Authorisation is the part worth deciding before this ships. The read query
-- above assumes a user may see their own layers plus shared ones in their
-- district. Who may edit or deactivate a *shared* layer — only its creator,
-- or any user in the district — is a policy question this table can support
-- either way, through userid.
--
-- 6. The url column holds a value the browser will fetch. Two consequences:
-- the https constraint is functional, not cosmetic; and the endpoint should
-- validate the URL server-side as well, since a stored URL is effectively an
-- instruction to every client that loads the layer.
-- ============================================================================