pwaLUPMIS2/docs/LUPMIS-OSM-Import-Runbook.md

LUPMIS2 — Re-importing the OSM data so it can be kept current

For: whoever runs the import (LUSPA database team) Date: 11 August 2026

What has and has not been verified.

Executed — against osm2pgsql 2.2.0 and a scratch PostGIS database, using a hand-made extract covering every shape this config has to handle: an amenity node, a healthcare-only node, a highway way, a waterway way, an amenity mapped as a closed way, and one mapped as a multipolygon relation. The Lua config in §3 imported all four tables cleanly; updatable and attributes both came out true; all four amenity shapes reached pi_osm_points; geometries are EPSG:4326. A subsequent --append applied a diff correctly, and — the claim §5.2 depends on — left districtid NULL on the changed row while untouched rows kept theirs.

Not verified — anything that needs the LUPMIS database, which is not reachable from where this was written: the current state of the import, the real name of the district boundary table, and the row counts. §1 is the set of checks to run before trusting those parts.

---

Why this is needed

The LUPMIS roads endpoint returns, for district 1, 5,238 rows with these columns:

geom, districtid, osm_id, surface, oneway, name

highway is not among them. The road class — trunk, primary, residential, track — was discarded when the data was loaded. That is the normal outcome of importing with osm2pgsql's default.style, which keeps a fixed list of tags and silently drops everything else; the same mechanism discards phone, opening_hours and website. The consequences are already visible in the application:

• Roads cannot be styled or filtered by class. Every road is drawn identically, so a trunk road and a footpath look the same at every zoom.
• src/import-detect.js lists highway as an expected field for the osm_roads target, so the mapping UI offers a column the data does not have.

A discarded tag cannot be recovered by querying; it was never written. The only way to get highway is to import again — and if the import is going to be redone, it is worth redoing on terms that allow it to be kept current afterwards, rather than repeating the same exercise in a year.

Whether the rest of the diagnosis applies — no middle tables, no replication set up, a stale import_timestamp — is what §1 determines.

---

What this import decides

Three decisions determine whether this import can be maintained or has to be repeated by hand every time.

The flex output, configured by the Lua file in §3. The alternative — the classic pgsql output that produces planet_osm_point, _line, _polygon and _roads — is deprecated in osm2pgsql 2.x and warns on every run, and its tag selection is governed by a style file that drops whatever it does not know. Flex puts that choice in a configuration you own: the columns are the ones you list, and everything else can be kept in a jsonb column instead of being thrown away. It also writes one table per purpose rather than four fixed ones, which is what allows an amenity mapped as a building outline to be served as a point.

--slim, without --drop. This keeps the middle tables — osm2pgsql's own record of every node, way and relation it has seen. They cost disk, and they are the entire difference between an import that can accept a daily diff and one that can only ever be replaced wholesale. An import run without them records updatable=false, and no later flag will change that; the only repair is a full re-import. --drop discards them, which is why it must not be used here.

--extra-attributes. Records each object's OSM version and timestamp. That makes it possible to say when a feature was last edited, and to answer "what changed" at all.

Two further decisions are specific to LUPMIS.

A separate osm schema — this one is not negotiable. osm2pgsql --create drops and recreates every table it owns in its schema. The spatial schema holds lu_parcels, the hand-edited planning data this whole system exists to manage. Pointing osm2pgsql at spatial puts a --create run one name collision away from destroying it. Import into osm; expose to the API through views in spatial (§7).

Districts are stamped after import, not during. osm2pgsql imports the national Ghana extract and has no concept of a district. Every LUPMIS table and endpoint is district-scoped, so a districtid column is filled by a spatial join once the import finishes (§5). This has to run after every daily update as well, and §5.2 explains why that turns out to be cheap.

---

1. Find out what you actually have

Run these before anything else. They decide whether this is a re-import or a first proper import, and none of the answers can be guessed from outside.

-- Does an osm2pgsql import exist at all, and on what terms?
SELECT property, value FROM osm2pgsql_properties
 ORDER BY property;
-- Look for: updatable, attributes, output, style,
--           import_timestamp, replication_base_url, replication_sequence_number
-- If this table does not exist, pi_osm_roads was loaded some other way
-- (ogr2ogr, a hand-written script, a one-off SQL dump).

-- Where does the current roads data live, and what is in it?
SELECT table_schema, table_name
  FROM information_schema.tables
 WHERE table_name LIKE '%osm%' OR table_name LIKE 'planet_osm%'
 ORDER BY 1, 2;

SELECT column_name, data_type
  FROM information_schema.columns
 WHERE table_name = 'pi_osm_roads'
 ORDER BY ordinal_position;

-- Middle tables present? Their absence is what makes --append impossible.
SELECT to_regclass('osm.planet_osm_nodes') AS nodes,
       to_regclass('osm.planet_osm_ways')  AS ways,
       to_regclass('osm.planet_osm_rels')  AS rels;

If osm2pgsql_properties does not exist, there is nothing to preserve or migrate: skip §2 and import fresh. That is the more likely case here, because a pi_-prefixed table in spatial with exactly six columns looks like a purpose-built extract rather than anything osm2pgsql produced.

If it does exist, note replication_base_url and replication_sequence_number before touching anything — even a non-updatable import records them, and they tell a fresh import where the data left off.

---

2. Clear out only what is safe to clear

Nothing in this section touches spatial. The live application keeps reading spatial.pi_osm_roads until §7 switches it over.

CREATE SCHEMA IF NOT EXISTS osm;

If a previous osm2pgsql run left pgsql-output tables in osm, drop these four and no others:

DROP TABLE IF EXISTS osm.planet_osm_point,
                     osm.planet_osm_line,
                     osm.planet_osm_polygon,
                     osm.planet_osm_roads;

planet_osm_nodes, planet_osm_ways, planet_osm_rels and planet_osm_users look like the same family and are not — they are the middle tables that --slim exists to keep, and flex uses them exactly as the pgsql output did. Dropping them costs you --append, which is the point of this exercise.

Leave osm2pgsql_properties alone; --create rewrites it.

---

3. The flex configuration

Save as sql/lupmis-osm.lua next to wherever the import is run.

Four tables, matching the four datasets requested. Every geometry is EPSG:4326, so the API serves what the table stores and nothing transforms on read. Each table carries a districtid column that the import leaves empty and §5 fills.

-- lupmis-osm.lua — osm2pgsql flex config for LUPMIS2
--
-- Four tables in the `osm` schema:
--   pi_osm_roads     routable `highway` ways      (LineString)
--   pi_osm_lines     other linear features        (LineString)
--   pi_osm_points    amenities and healthcare     (Point)
--   pi_osm_polygons  buildings, land use, areas   (MultiPolygon)
--
-- districtid is declared here but never written by the import. It is filled by
-- the spatial join in §5 of the runbook. Because osm2pgsql re-inserts a row
-- whenever its object changes, an updated row comes back with districtid NULL —
-- which is what makes the incremental re-stamp in §5.2 both simple and correct.

local srid = 4326

local roads = osm2pgsql.define_table({
    name = 'pi_osm_roads', schema = 'osm',
    ids = { type = 'way', id_column = 'osm_id' },
    columns = {
        { column = 'highway',    type = 'text' },   -- the column that was missing
        { column = 'name',       type = 'text' },
        { column = 'ref',        type = 'text' },
        { column = 'surface',    type = 'text' },
        { column = 'oneway',     type = 'text' },
        { column = 'bridge',     type = 'text' },
        { column = 'tunnel',     type = 'text' },
        { column = 'layer',      type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'linestring', projection = srid, not_null = true },
    }
})

local lines = osm2pgsql.define_table({
    name = 'pi_osm_lines', schema = 'osm',
    ids = { type = 'way', id_column = 'osm_id' },
    columns = {
        { column = 'waterway',   type = 'text' },
        { column = 'railway',    type = 'text' },
        { column = 'power',      type = 'text' },
        { column = 'barrier',    type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'linestring', projection = srid, not_null = true },
    }
})

-- Points accept nodes, ways and relations, so an amenity mapped as a building
-- outline or a multipolygon appears here as a point like any other. That is the
-- single biggest gain over the classic layout, where areas lived in
-- planet_osm_polygon and the endpoint never looked there.
local points = osm2pgsql.define_table({
    name = 'pi_osm_points', schema = 'osm',
    ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
    columns = {
        { column = 'amenity',    type = 'text' },
        { column = 'healthcare', type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'geom',       type = 'point', projection = srid, not_null = true },
    }
})

local polygons = osm2pgsql.define_table({
    name = 'pi_osm_polygons', schema = 'osm',
    ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
    columns = {
        { column = 'building',   type = 'text' },
        { column = 'landuse',    type = 'text' },
        { column = 'amenity',    type = 'text' },
        { column = 'leisure',    type = 'text' },
        { column = 'natural',    type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'multipolygon', projection = srid, not_null = true },
    }
})

-- Tags that say nothing once the typed columns exist.
local uninteresting = {
    'created_by', 'source', 'source:date', 'note', 'comment',
    'fixme', 'FIXME', 'attribution',
}

local function clean(tags)
    for _, k in ipairs(uninteresting) do tags[k] = nil end
end

local function is_area(tags)
    return tags.area == 'yes'
        or tags.building or tags.landuse or tags.leisure or tags.natural
end

function osm2pgsql.process_node(object)
    local t = object.tags
    if t.amenity or t.healthcare then
        points:insert({
            amenity    = t.amenity,
            healthcare = t.healthcare,
            name       = t.name,
            geom       = object:as_point(),
        })
    end
end

function osm2pgsql.process_way(object)
    local t = object.tags
    clean(t)

    if object.is_closed and is_area(t) then
        polygons:insert({
            building = t.building, landuse = t.landuse, amenity = t.amenity,
            leisure  = t.leisure,  natural = t['natural'], name = t.name,
            tags = t, geom = object:as_polygon(),
        })
        -- An amenity mapped as an area is also a point, so it is findable
        -- alongside amenities mapped as nodes.
        if t.amenity or t.healthcare then
            points:insert({
                amenity = t.amenity, healthcare = t.healthcare, name = t.name,
                geom = object:as_polygon():centroid(),
            })
        end
        return
    end

    if t.highway then
        roads:insert({
            highway = t.highway, name = t.name, ref = t.ref,
            surface = t.surface, oneway = t.oneway,
            bridge  = t.bridge,  tunnel = t.tunnel, layer = t.layer,
            tags = t, geom = object:as_linestring(),
        })
    elseif t.waterway or t.railway or t.power or t.barrier then
        lines:insert({
            waterway = t.waterway, railway = t.railway,
            power    = t.power,    barrier = t.barrier, name = t.name,
            tags = t, geom = object:as_linestring(),
        })
    end
end

function osm2pgsql.process_relation(object)
    local t = object.tags
    clean(t)
    if t.type ~= 'multipolygon' and t.type ~= 'boundary' then return end

    if is_area(t) or t.amenity or t.healthcare then
        polygons:insert({
            building = t.building, landuse = t.landuse, amenity = t.amenity,
            leisure  = t.leisure,  natural = t['natural'], name = t.name,
            tags = t, geom = object:as_multipolygon(),
        })
        if t.amenity or t.healthcare then
            points:insert({
                amenity = t.amenity, healthcare = t.healthcare, name = t.name,
                geom = object:as_multipolygon():centroid(),
            })
        end
    end
end

---

4. Import

curl -O https://download.geofabrik.de/africa/ghana-latest.osm.pbf

osm2pgsql --create --slim --output=flex \
  --style sql/lupmis-osm.lua --extra-attributes \
  --database lupmis --schema osm --middle-schema osm \
  --cache 2000 \
  ghana-latest.osm.pbf

--slim without --drop. --drop discards the middle tables and is what makes an import permanently un-updatable.

Substitute the real database name for lupmis.

---

5. Stamp the districts

osm2pgsql knows nothing about districts, so this step has no equivalent in the standard OSM tooling. It is what makes a national import usable by an application where every table and every endpoint is district-scoped.

5.1 First pass, after the initial import

Confirm the boundary table's real name and column first — the application only ever sees it through get_district_boundary.php:

SELECT table_schema, table_name FROM information_schema.tables
 WHERE table_name ILIKE '%district%';

Then, for each of the four tables (shown here for roads):

UPDATE osm.pi_osm_roads r
   SET districtid = d.districtid
  FROM spatial.districts d              -- confirm this name first
 WHERE r.districtid IS NULL
   AND ST_Intersects(d.geom, r.geom);

A linear feature crossing a district boundary matches more than one district. UPDATE takes an arbitrary one of them, which is wrong for a road that spans a boundary. If roads must appear in every district they touch, use a join table instead of a column:

CREATE TABLE osm.pi_osm_roads_district AS
SELECT r.osm_id, d.districtid
  FROM osm.pi_osm_roads r
  JOIN spatial.districts d ON ST_Intersects(d.geom, r.geom);
CREATE INDEX ON osm.pi_osm_roads_district (districtid);

Points never have this problem. For polygons, decide whether a district should own an area by intersection or by where its centroid falls — ST_Intersects(d.geom, ST_Centroid(p.geom)) gives one district per polygon.

5.2 After every update

The same UPDATE again — WHERE districtid IS NULL is doing real work here, not just guarding against repetition.

When an object changes, osm2pgsql deletes its row and re-inserts it from the new data. The Lua config never writes districtid, so the re-inserted row comes back NULL. WHERE districtid IS NULL therefore selects exactly the objects that are new or were edited since the last stamp — including any that moved across a boundary — and nothing else. A daily re-stamp touches a few hundred rows rather than millions.

Put it in the same cron entry as the update, immediately after it. If the two ever get separated, the symptom is new roads that no district can see.

---

6. Indexes

osm2pgsql creates the geometry and id indexes. Add what the endpoints filter on:

CREATE INDEX ON osm.pi_osm_roads    (districtid);
CREATE INDEX ON osm.pi_osm_lines    (districtid);
CREATE INDEX ON osm.pi_osm_points   (districtid);
CREATE INDEX ON osm.pi_osm_polygons (districtid);

CREATE INDEX ON osm.pi_osm_roads  (highway)               WHERE highway   IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (amenity)               WHERE amenity    IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (healthcare)            WHERE healthcare IS NOT NULL;
CREATE INDEX ON osm.pi_osm_roads    USING gin (tags);
CREATE INDEX ON osm.pi_osm_polygons USING gin (tags);

---

7. Expose it to the API

Keep osm2pgsql's tables in osm, and give the API views in spatial under the existing pi_ naming. The application then reads the names it already expects, and no import can ever write into spatial.

The current spatial.pi_osm_roads is a table, and a view cannot replace a table of the same name. Rename it rather than dropping it, so there is a way back until the new endpoints have been verified:

ALTER TABLE spatial.pi_osm_roads RENAME TO pi_osm_roads_pre_osm2pgsql;

CREATE VIEW spatial.pi_osm_roads AS
  SELECT osm_id, highway, name, ref, surface, oneway, districtid, geom
    FROM osm.pi_osm_roads;

CREATE VIEW spatial.pi_osm_points AS
  SELECT osm_type, osm_id, amenity, healthcare, name, districtid, geom
    FROM osm.pi_osm_points;

CREATE VIEW spatial.pi_osm_lines AS
  SELECT osm_id, waterway, railway, power, barrier, name, districtid, geom
    FROM osm.pi_osm_lines;

CREATE VIEW spatial.pi_osm_polygons AS
  SELECT osm_type, osm_id, building, landuse, amenity, leisure, name, districtid, geom
    FROM osm.pi_osm_polygons;

The existing get_osm_roads.php keeps working unchanged and gains highway, because it reads spatial.pi_osm_roads and the view now supplies that column.

Three new endpoints are needed, in the shape of the existing one — same { api_token, district_id } request, same { success, data: [...] } response, geometry as WKT in geom:

Endpoint	Serves	Application layer
get_osm_points.php	spatial.pi_osm_points	OSM Points
get_osm_lines.php	spatial.pi_osm_lines	OSM Lines
get_osm_polygons.php	spatial.pi_osm_polygons	OSM Polygons

All four belong to layer group 5, Physical Infrastructures, which is where OSM_roads already sits.

Volumes are the thing to watch. Roads alone are 5,238 rows for district 1 under the current extract, and the endpoint returns every row as WKT in one response. Points and polygons will be larger. Before wiring the new layers into the application, decide whether these endpoints should take a bounding box, or a highway/amenity filter, rather than returning a whole district. That decision belongs with the endpoints, not the application.

---

8. Keep it current

This is the part the import exists for. Everything above only pays off if the daily update actually runs.

8.1 osm2pgsql-replication needs a Python that has its libraries

osm2pgsql-replication ships with osm2pgsql but is a Python script, and package managers do not install what it imports. Its first run says:

Missing required Python libraries psycopg2 osmium.
To install them via pip run: pip install psycopg2 osmium

Following that advice often changes nothing, and it is worth understanding why before chasing it. The script's shebang is #!/usr/bin/env python3, and env does not see shell aliases. If python3 in your shell is aliased — a MAMP installation does exactly this, and MAMP is present on at least one machine in this project — then pip3 install puts the libraries where the script will never look. Check before doing anything:

env python3 -c "import sys; print(sys.executable)"
env python3 -c "import psycopg2, osmium; print('both present')"

A Homebrew Python is also likely to be PEP 668 "externally managed", which refuses the install outright.

On the Linux server, use distribution packages — no virtualenv, no pip:

apt install python3-psycopg2 python3-pyosmium      # Debian/Ubuntu

On a workstation, give the script an interpreter that has both, once:

python3.12 -m venv ~/.venvs/osm2pgsql
~/.venvs/osm2pgsql/bin/pip install psycopg2-binary osmium

psycopg2-binary rather than psycopg2: it ships as a wheel and needs no pg_config or compiler. Then invoke the tool with that interpreter:

~/.venvs/osm2pgsql/bin/python $(brew --prefix)/bin/osm2pgsql-replication …

You know it is working when the message changes from the missing-library error to Updates not set up correctly. Run 'osm2pgsql-replication init' first. — that is the tool running properly and telling you about §8.2, not about Python.

8.2 Initialise, then run daily

osm2pgsql-replication init \
  --database lupmis --schema osm --osm-file ghana-latest.osm.pbf

That reads the replication URL and sequence number out of the extract's header, so updates start exactly where the downloaded file left off.

Then daily, with the district stamp in the same job:

osm2pgsql-replication update \
  --database lupmis --schema osm \
  --diff-file /var/osm/diffs/$(date +%F).osc.gz \
  -- --slim --output=flex --style sql/lupmis-osm.lua --extra-attributes \
     --schema osm --middle-schema osm \
&& psql -d lupmis -f sql/stamp-districts.sql

The append updates the tables in place; there is no refresh step. && rather than ; so a failed update does not leave the stamp running against half-applied data.

In cron, spell out the full path to every binary — osm2pgsql-replication, psql, and the Python interpreter if §8.1 required one. A cron job does not inherit your shell's PATH, and this is the most common reason a daily update works when run by hand and silently never runs from cron.

--diff-file keeps each day's changes. It is what lets you answer "what changed near this parcel last week", which nothing else in this pipeline records.

---

9. Check it worked

-- The column that started all this.
SELECT highway, count(*) FROM osm.pi_osm_roads
 WHERE highway IS NOT NULL GROUP BY 1 ORDER BY 2 DESC LIMIT 12;

-- Amenities mapped as areas — these could not appear in the old layout.
SELECT osm_type, count(*) FROM osm.pi_osm_points
 WHERE amenity IS NOT NULL GROUP BY 1;

-- Clinics carrying healthcare and no amenity, invisible until now.
SELECT count(*) FROM osm.pi_osm_points
 WHERE amenity IS NULL AND healthcare IS NOT NULL;

-- Nothing left unstamped. A non-zero count is usually genuine — offshore
-- features, or gaps between district polygons — but check before assuming.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid IS NULL;

-- Comparable with the endpoint's 5,238 for district 1.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid = 1;

-- The doors are open this time.
SELECT property, value FROM osm.osm2pgsql_properties
 WHERE property IN ('updatable','attributes','output','current_timestamp');
-- updatable and attributes must both be true.

Report freshness from current_timestamp, which moves with every append, not import_timestamp, which is fixed at the original import.

Once the endpoints serve from the views and the application is verified, the renamed original can go:

DROP TABLE spatial.pi_osm_roads_pre_osm2pgsql;

---

10. When it goes wrong

Fetch a fresh extract, re-run §4 with --create, re-run §5.1 and osm2pgsql-replication init. Nothing in spatial is touched by any of it, which is the entire reason the import lives in its own schema.