# LUPMIS2 — Re-importing the OSM data so it can be kept current **For:** whoever runs the import (LUSPA database team) **Date:** 11 August 2026 > **What has and has not been verified.** > > **Executed** — against osm2pgsql 2.2.0 and a scratch PostGIS database, using a > hand-made extract covering every shape this config has to handle: an amenity > node, a `healthcare`-only node, a `highway` way, a `waterway` way, an amenity > mapped as a closed way, and one mapped as a multipolygon relation. > The Lua config in §3 imported all four tables cleanly; `updatable` and > `attributes` both came out `true`; all four amenity shapes reached > `pi_osm_points`; geometries are EPSG:4326. A subsequent `--append` applied a > diff correctly, and — the claim §5.2 depends on — left `districtid` **NULL on > the changed row while untouched rows kept theirs**. > > **Not verified** — anything that needs the LUPMIS database, which is not > reachable from where this was written: the current state of the import, the > real name of the district boundary table, and the row counts. §1 is the set of > checks to run before trusting those parts. --- ## Why this is needed The LUPMIS roads endpoint returns, for district 1, 5,238 rows with these columns: ``` geom, districtid, osm_id, surface, oneway, name ``` `highway` is not among them. The road class — trunk, primary, residential, track — was discarded when the data was loaded. That is the normal outcome of importing with osm2pgsql's `default.style`, which keeps a fixed list of tags and silently drops everything else; the same mechanism discards `phone`, `opening_hours` and `website`. The consequences are already visible in the application: - Roads cannot be styled or filtered by class. Every road is drawn identically, so a trunk road and a footpath look the same at every zoom. - `src/import-detect.js` lists `highway` as an expected field for the `osm_roads` target, so the mapping UI offers a column the data does not have. A discarded tag cannot be recovered by querying; it was never written. The only way to get `highway` is to import again — and if the import is going to be redone, it is worth redoing on terms that allow it to be kept current afterwards, rather than repeating the same exercise in a year. Whether the rest of the diagnosis applies — no middle tables, no replication set up, a stale `import_timestamp` — is what §1 determines. --- ## What this import decides Three decisions determine whether this import can be maintained or has to be repeated by hand every time. **The flex output**, configured by the Lua file in §3. The alternative — the classic pgsql output that produces `planet_osm_point`, `_line`, `_polygon` and `_roads` — is deprecated in osm2pgsql 2.x and warns on every run, and its tag selection is governed by a style file that drops whatever it does not know. Flex puts that choice in a configuration you own: the columns are the ones you list, and everything else can be kept in a `jsonb` column instead of being thrown away. It also writes one table per purpose rather than four fixed ones, which is what allows an amenity mapped as a building outline to be served as a point. **`--slim`, without `--drop`.** This keeps the middle tables — osm2pgsql's own record of every node, way and relation it has seen. They cost disk, and they are the entire difference between an import that can accept a daily diff and one that can only ever be replaced wholesale. An import run without them records `updatable=false`, and no later flag will change that; the only repair is a full re-import. `--drop` discards them, which is why it must not be used here. **`--extra-attributes`.** Records each object's OSM version and timestamp. That makes it possible to say when a feature was last edited, and to answer "what changed" at all. Two further decisions are specific to LUPMIS. **A separate `osm` schema — this one is not negotiable.** osm2pgsql `--create` drops and recreates every table it owns in its schema. The `spatial` schema holds `lu_parcels`, the hand-edited planning data this whole system exists to manage. Pointing osm2pgsql at `spatial` puts a `--create` run one name collision away from destroying it. Import into `osm`; expose to the API through views in `spatial` (§7). **Districts are stamped after import, not during.** osm2pgsql imports the national Ghana extract and has no concept of a district. Every LUPMIS table and endpoint is district-scoped, so a `districtid` column is filled by a spatial join once the import finishes (§5). This has to run after every daily update as well, and §5.2 explains why that turns out to be cheap. --- ## 1. Find out what you actually have Run these before anything else. They decide whether this is a re-import or a first proper import, and none of the answers can be guessed from outside. ```sql -- Does an osm2pgsql import exist at all, and on what terms? SELECT property, value FROM osm2pgsql_properties ORDER BY property; -- Look for: updatable, attributes, output, style, -- import_timestamp, replication_base_url, replication_sequence_number -- If this table does not exist, pi_osm_roads was loaded some other way -- (ogr2ogr, a hand-written script, a one-off SQL dump). -- Where does the current roads data live, and what is in it? SELECT table_schema, table_name FROM information_schema.tables WHERE table_name LIKE '%osm%' OR table_name LIKE 'planet_osm%' ORDER BY 1, 2; SELECT column_name, data_type FROM information_schema.columns WHERE table_name = 'pi_osm_roads' ORDER BY ordinal_position; -- Middle tables present? Their absence is what makes --append impossible. SELECT to_regclass('osm.planet_osm_nodes') AS nodes, to_regclass('osm.planet_osm_ways') AS ways, to_regclass('osm.planet_osm_rels') AS rels; ``` **If `osm2pgsql_properties` does not exist**, there is nothing to preserve or migrate: skip §2 and import fresh. That is the more likely case here, because a `pi_`-prefixed table in `spatial` with exactly six columns looks like a purpose-built extract rather than anything osm2pgsql produced. **If it does exist**, note `replication_base_url` and `replication_sequence_number` before touching anything — even a non-updatable import records them, and they tell a fresh import where the data left off. --- ## 2. Clear out only what is safe to clear Nothing in this section touches `spatial`. The live application keeps reading `spatial.pi_osm_roads` until §7 switches it over. ```sql CREATE SCHEMA IF NOT EXISTS osm; ``` If a previous osm2pgsql run left pgsql-output tables in `osm`, drop these four and **no others**: ```sql DROP TABLE IF EXISTS osm.planet_osm_point, osm.planet_osm_line, osm.planet_osm_polygon, osm.planet_osm_roads; ``` `planet_osm_nodes`, `planet_osm_ways`, `planet_osm_rels` and `planet_osm_users` look like the same family and are not — they are the middle tables that `--slim` exists to keep, and flex uses them exactly as the pgsql output did. Dropping them costs you `--append`, which is the point of this exercise. Leave `osm2pgsql_properties` alone; `--create` rewrites it. --- ## 3. The flex configuration Save as `sql/lupmis-osm.lua` next to wherever the import is run. Four tables, matching the four datasets requested. Every geometry is EPSG:4326, so the API serves what the table stores and nothing transforms on read. Each table carries a `districtid` column that the import leaves empty and §5 fills. ```lua -- lupmis-osm.lua — osm2pgsql flex config for LUPMIS2 -- -- Four tables in the `osm` schema: -- pi_osm_roads routable `highway` ways (LineString) -- pi_osm_lines other linear features (LineString) -- pi_osm_points amenities and healthcare (Point) -- pi_osm_polygons buildings, land use, areas (MultiPolygon) -- -- districtid is declared here but never written by the import. It is filled by -- the spatial join in §5 of the runbook. Because osm2pgsql re-inserts a row -- whenever its object changes, an updated row comes back with districtid NULL — -- which is what makes the incremental re-stamp in §5.2 both simple and correct. local srid = 4326 local roads = osm2pgsql.define_table({ name = 'pi_osm_roads', schema = 'osm', ids = { type = 'way', id_column = 'osm_id' }, columns = { { column = 'highway', type = 'text' }, -- the column that was missing { column = 'name', type = 'text' }, { column = 'ref', type = 'text' }, { column = 'surface', type = 'text' }, { column = 'oneway', type = 'text' }, { column = 'bridge', type = 'text' }, { column = 'tunnel', type = 'text' }, { column = 'layer', type = 'text' }, { column = 'districtid', type = 'int' }, { column = 'tags', type = 'jsonb' }, { column = 'geom', type = 'linestring', projection = srid, not_null = true }, } }) local lines = osm2pgsql.define_table({ name = 'pi_osm_lines', schema = 'osm', ids = { type = 'way', id_column = 'osm_id' }, columns = { { column = 'waterway', type = 'text' }, { column = 'railway', type = 'text' }, { column = 'power', type = 'text' }, { column = 'barrier', type = 'text' }, { column = 'name', type = 'text' }, { column = 'districtid', type = 'int' }, { column = 'tags', type = 'jsonb' }, { column = 'geom', type = 'linestring', projection = srid, not_null = true }, } }) -- Points accept nodes, ways and relations, so an amenity mapped as a building -- outline or a multipolygon appears here as a point like any other. That is the -- single biggest gain over the classic layout, where areas lived in -- planet_osm_polygon and the endpoint never looked there. local points = osm2pgsql.define_table({ name = 'pi_osm_points', schema = 'osm', ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' }, columns = { { column = 'amenity', type = 'text' }, { column = 'healthcare', type = 'text' }, { column = 'name', type = 'text' }, { column = 'districtid', type = 'int' }, { column = 'geom', type = 'point', projection = srid, not_null = true }, } }) local polygons = osm2pgsql.define_table({ name = 'pi_osm_polygons', schema = 'osm', ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' }, columns = { { column = 'building', type = 'text' }, { column = 'landuse', type = 'text' }, { column = 'amenity', type = 'text' }, { column = 'leisure', type = 'text' }, { column = 'natural', type = 'text' }, { column = 'name', type = 'text' }, { column = 'districtid', type = 'int' }, { column = 'tags', type = 'jsonb' }, { column = 'geom', type = 'multipolygon', projection = srid, not_null = true }, } }) -- Tags that say nothing once the typed columns exist. local uninteresting = { 'created_by', 'source', 'source:date', 'note', 'comment', 'fixme', 'FIXME', 'attribution', } local function clean(tags) for _, k in ipairs(uninteresting) do tags[k] = nil end end local function is_area(tags) return tags.area == 'yes' or tags.building or tags.landuse or tags.leisure or tags.natural end function osm2pgsql.process_node(object) local t = object.tags if t.amenity or t.healthcare then points:insert({ amenity = t.amenity, healthcare = t.healthcare, name = t.name, geom = object:as_point(), }) end end function osm2pgsql.process_way(object) local t = object.tags clean(t) if object.is_closed and is_area(t) then polygons:insert({ building = t.building, landuse = t.landuse, amenity = t.amenity, leisure = t.leisure, natural = t['natural'], name = t.name, tags = t, geom = object:as_polygon(), }) -- An amenity mapped as an area is also a point, so it is findable -- alongside amenities mapped as nodes. if t.amenity or t.healthcare then points:insert({ amenity = t.amenity, healthcare = t.healthcare, name = t.name, geom = object:as_polygon():centroid(), }) end return end if t.highway then roads:insert({ highway = t.highway, name = t.name, ref = t.ref, surface = t.surface, oneway = t.oneway, bridge = t.bridge, tunnel = t.tunnel, layer = t.layer, tags = t, geom = object:as_linestring(), }) elseif t.waterway or t.railway or t.power or t.barrier then lines:insert({ waterway = t.waterway, railway = t.railway, power = t.power, barrier = t.barrier, name = t.name, tags = t, geom = object:as_linestring(), }) end end function osm2pgsql.process_relation(object) local t = object.tags clean(t) if t.type ~= 'multipolygon' and t.type ~= 'boundary' then return end if is_area(t) or t.amenity or t.healthcare then polygons:insert({ building = t.building, landuse = t.landuse, amenity = t.amenity, leisure = t.leisure, natural = t['natural'], name = t.name, tags = t, geom = object:as_multipolygon(), }) if t.amenity or t.healthcare then points:insert({ amenity = t.amenity, healthcare = t.healthcare, name = t.name, geom = object:as_multipolygon():centroid(), }) end end end ``` --- ## 4. Import ```bash curl -O https://download.geofabrik.de/africa/ghana-latest.osm.pbf osm2pgsql --create --slim --output=flex \ --style sql/lupmis-osm.lua --extra-attributes \ --database lupmis --schema osm --middle-schema osm \ --cache 2000 \ ghana-latest.osm.pbf ``` `--slim` **without** `--drop`. `--drop` discards the middle tables and is what makes an import permanently un-updatable. Substitute the real database name for `lupmis`. --- ## 5. Stamp the districts osm2pgsql knows nothing about districts, so this step has no equivalent in the standard OSM tooling. It is what makes a national import usable by an application where every table and every endpoint is district-scoped. ### 5.1 First pass, after the initial import Confirm the boundary table's real name and column first — the application only ever sees it through `get_district_boundary.php`: ```sql SELECT table_schema, table_name FROM information_schema.tables WHERE table_name ILIKE '%district%'; ``` Then, for each of the four tables (shown here for roads): ```sql UPDATE osm.pi_osm_roads r SET districtid = d.districtid FROM spatial.districts d -- confirm this name first WHERE r.districtid IS NULL AND ST_Intersects(d.geom, r.geom); ``` A linear feature crossing a district boundary matches more than one district. `UPDATE` takes an arbitrary one of them, which is wrong for a road that spans a boundary. If roads must appear in every district they touch, use a join table instead of a column: ```sql CREATE TABLE osm.pi_osm_roads_district AS SELECT r.osm_id, d.districtid FROM osm.pi_osm_roads r JOIN spatial.districts d ON ST_Intersects(d.geom, r.geom); CREATE INDEX ON osm.pi_osm_roads_district (districtid); ``` Points never have this problem. For polygons, decide whether a district should own an area by intersection or by where its centroid falls — `ST_Intersects(d.geom, ST_Centroid(p.geom))` gives one district per polygon. ### 5.2 After every update The same `UPDATE` again — `WHERE districtid IS NULL` is doing real work here, not just guarding against repetition. When an object changes, osm2pgsql deletes its row and re-inserts it from the new data. The Lua config never writes `districtid`, so the re-inserted row comes back **NULL**. `WHERE districtid IS NULL` therefore selects exactly the objects that are new or were edited since the last stamp — including any that moved across a boundary — and nothing else. A daily re-stamp touches a few hundred rows rather than millions. Put it in the same cron entry as the update, immediately after it. If the two ever get separated, the symptom is new roads that no district can see. --- ## 6. Indexes osm2pgsql creates the geometry and id indexes. Add what the endpoints filter on: ```sql CREATE INDEX ON osm.pi_osm_roads (districtid); CREATE INDEX ON osm.pi_osm_lines (districtid); CREATE INDEX ON osm.pi_osm_points (districtid); CREATE INDEX ON osm.pi_osm_polygons (districtid); CREATE INDEX ON osm.pi_osm_roads (highway) WHERE highway IS NOT NULL; CREATE INDEX ON osm.pi_osm_points (amenity) WHERE amenity IS NOT NULL; CREATE INDEX ON osm.pi_osm_points (healthcare) WHERE healthcare IS NOT NULL; CREATE INDEX ON osm.pi_osm_roads USING gin (tags); CREATE INDEX ON osm.pi_osm_polygons USING gin (tags); ``` --- ## 7. Expose it to the API Keep osm2pgsql's tables in `osm`, and give the API views in `spatial` under the existing `pi_` naming. The application then reads the names it already expects, and no import can ever write into `spatial`. The current `spatial.pi_osm_roads` is a table, and a view cannot replace a table of the same name. Rename it rather than dropping it, so there is a way back until the new endpoints have been verified: ```sql ALTER TABLE spatial.pi_osm_roads RENAME TO pi_osm_roads_pre_osm2pgsql; CREATE VIEW spatial.pi_osm_roads AS SELECT osm_id, highway, name, ref, surface, oneway, districtid, geom FROM osm.pi_osm_roads; CREATE VIEW spatial.pi_osm_points AS SELECT osm_type, osm_id, amenity, healthcare, name, districtid, geom FROM osm.pi_osm_points; CREATE VIEW spatial.pi_osm_lines AS SELECT osm_id, waterway, railway, power, barrier, name, districtid, geom FROM osm.pi_osm_lines; CREATE VIEW spatial.pi_osm_polygons AS SELECT osm_type, osm_id, building, landuse, amenity, leisure, name, districtid, geom FROM osm.pi_osm_polygons; ``` The existing `get_osm_roads.php` keeps working unchanged and gains `highway`, because it reads `spatial.pi_osm_roads` and the view now supplies that column. Three new endpoints are needed, in the shape of the existing one — same `{ api_token, district_id }` request, same `{ success, data: [...] }` response, geometry as WKT in `geom`: | Endpoint | Serves | Application layer | |---|---|---| | `get_osm_points.php` | `spatial.pi_osm_points` | OSM Points | | `get_osm_lines.php` | `spatial.pi_osm_lines` | OSM Lines | | `get_osm_polygons.php` | `spatial.pi_osm_polygons` | OSM Polygons | All four belong to layer group **5, Physical Infrastructures**, which is where `OSM_roads` already sits. **Volumes are the thing to watch.** Roads alone are 5,238 rows for district 1 under the current extract, and the endpoint returns every row as WKT in one response. Points and polygons will be larger. Before wiring the new layers into the application, decide whether these endpoints should take a bounding box, or a `highway`/`amenity` filter, rather than returning a whole district. That decision belongs with the endpoints, not the application. --- ## 8. Keep it current This is the part the import exists for. Everything above only pays off if the daily update actually runs. ### 8.1 osm2pgsql-replication needs a Python that has its libraries `osm2pgsql-replication` ships with osm2pgsql but is a **Python** script, and package managers do not install what it imports. Its first run says: ``` Missing required Python libraries psycopg2 osmium. To install them via pip run: pip install psycopg2 osmium ``` Following that advice often changes nothing, and it is worth understanding why before chasing it. The script's shebang is `#!/usr/bin/env python3`, and **`env` does not see shell aliases**. If `python3` in your shell is aliased — a MAMP installation does exactly this, and MAMP is present on at least one machine in this project — then `pip3 install` puts the libraries where the script will never look. Check before doing anything: ```bash env python3 -c "import sys; print(sys.executable)" env python3 -c "import psycopg2, osmium; print('both present')" ``` A Homebrew Python is also likely to be PEP 668 "externally managed", which refuses the install outright. **On the Linux server, use distribution packages** — no virtualenv, no pip: ```bash apt install python3-psycopg2 python3-pyosmium # Debian/Ubuntu ``` **On a workstation**, give the script an interpreter that has both, once: ```bash python3.12 -m venv ~/.venvs/osm2pgsql ~/.venvs/osm2pgsql/bin/pip install psycopg2-binary osmium ``` `psycopg2-binary` rather than `psycopg2`: it ships as a wheel and needs no `pg_config` or compiler. Then invoke the tool with that interpreter: ```bash ~/.venvs/osm2pgsql/bin/python $(brew --prefix)/bin/osm2pgsql-replication … ``` You know it is working when the message changes from the missing-library error to `Updates not set up correctly. Run 'osm2pgsql-replication init' first.` — that is the tool running properly and telling you about §8.2, not about Python. ### 8.2 Initialise, then run daily ```bash osm2pgsql-replication init \ --database lupmis --schema osm --osm-file ghana-latest.osm.pbf ``` That reads the replication URL and sequence number out of the extract's header, so updates start exactly where the downloaded file left off. Then daily, with the district stamp in the same job: ```bash osm2pgsql-replication update \ --database lupmis --schema osm \ --diff-file /var/osm/diffs/$(date +%F).osc.gz \ -- --slim --output=flex --style sql/lupmis-osm.lua --extra-attributes \ --schema osm --middle-schema osm \ && psql -d lupmis -f sql/stamp-districts.sql ``` The append updates the tables in place; there is no refresh step. `&&` rather than `;` so a failed update does not leave the stamp running against half-applied data. In cron, spell out the full path to every binary — `osm2pgsql-replication`, `psql`, and the Python interpreter if §8.1 required one. A cron job does not inherit your shell's `PATH`, and this is the most common reason a daily update works when run by hand and silently never runs from cron. `--diff-file` keeps each day's changes. It is what lets you answer "what changed near this parcel last week", which nothing else in this pipeline records. --- ## 9. Check it worked ```sql -- The column that started all this. SELECT highway, count(*) FROM osm.pi_osm_roads WHERE highway IS NOT NULL GROUP BY 1 ORDER BY 2 DESC LIMIT 12; -- Amenities mapped as areas — these could not appear in the old layout. SELECT osm_type, count(*) FROM osm.pi_osm_points WHERE amenity IS NOT NULL GROUP BY 1; -- Clinics carrying healthcare and no amenity, invisible until now. SELECT count(*) FROM osm.pi_osm_points WHERE amenity IS NULL AND healthcare IS NOT NULL; -- Nothing left unstamped. A non-zero count is usually genuine — offshore -- features, or gaps between district polygons — but check before assuming. SELECT count(*) FROM osm.pi_osm_roads WHERE districtid IS NULL; -- Comparable with the endpoint's 5,238 for district 1. SELECT count(*) FROM osm.pi_osm_roads WHERE districtid = 1; -- The doors are open this time. SELECT property, value FROM osm.osm2pgsql_properties WHERE property IN ('updatable','attributes','output','current_timestamp'); -- updatable and attributes must both be true. ``` Report freshness from `current_timestamp`, which moves with every append, not `import_timestamp`, which is fixed at the original import. Once the endpoints serve from the views and the application is verified, the renamed original can go: ```sql DROP TABLE spatial.pi_osm_roads_pre_osm2pgsql; ``` --- ## 10. When it goes wrong Fetch a fresh extract, re-run §4 with `--create`, re-run §5.1 and `osm2pgsql-replication init`. Nothing in `spatial` is touched by any of it, which is the entire reason the import lives in its own schema.