pwaLUPMIS2/docs/LUPMIS-OSM-Import-Runbook.md

# LUPMIS2 — Re-importing the OSM data so it can be kept current

**For:** whoever runs the import (LUSPA database team)
**Date:** 11 August 2026

> **What has and has not been verified.**
>
> **Executed** — against osm2pgsql 2.2.0 and a scratch PostGIS database, using a
> hand-made extract covering every shape this config has to handle: an amenity
> node, a `healthcare`-only node, a `highway` way, a `waterway` way, an amenity
> mapped as a closed way, and one mapped as a multipolygon relation.
> The Lua config in §3 imported all four tables cleanly; `updatable` and
> `attributes` both came out `true`; all four amenity shapes reached
> `pi_osm_points`; geometries are EPSG:4326. A subsequent `--append` applied a
> diff correctly, and — the claim §5.2 depends on — left `districtid` **NULL on
> the changed row while untouched rows kept theirs**.
>
> **Not verified** — anything that needs the LUPMIS database, which is not
> reachable from where this was written: the current state of the import, the
> real name of the district boundary table, and the row counts. §1 is the set of
> checks to run before trusting those parts.

---

## Why this is needed

The LUPMIS roads endpoint returns, for district 1, 5,238 rows with these
columns:

```
geom, districtid, osm_id, surface, oneway, name
```

`highway` is not among them. The road class — trunk, primary, residential,
track — was discarded when the data was loaded. That is the normal outcome of
importing with osm2pgsql's `default.style`, which keeps a fixed list of tags
and silently drops everything else; the same mechanism discards `phone`,
`opening_hours` and `website`. The consequences are already visible in the
application:

- Roads cannot be styled or filtered by class. Every road is drawn identically,
  so a trunk road and a footpath look the same at every zoom.
- `src/import-detect.js` lists `highway` as an expected field for the
  `osm_roads` target, so the mapping UI offers a column the data does not have.

A discarded tag cannot be recovered by querying; it was never written. The only
way to get `highway` is to import again — and if the import is going to be
redone, it is worth redoing on terms that allow it to be kept current
afterwards, rather than repeating the same exercise in a year.

Whether the rest of the diagnosis applies — no middle tables, no replication
set up, a stale `import_timestamp` — is what §1 determines.

---

## What this import decides

Three decisions determine whether this import can be maintained or has to be
repeated by hand every time.

**The flex output**, configured by the Lua file in §3. The alternative — the
classic pgsql output that produces `planet_osm_point`, `_line`, `_polygon` and
`_roads` — is deprecated in osm2pgsql 2.x and warns on every run, and its tag
selection is governed by a style file that drops whatever it does not know.
Flex puts that choice in a configuration you own: the columns are the ones you
list, and everything else can be kept in a `jsonb` column instead of being
thrown away. It also writes one table per purpose rather than four fixed ones,
which is what allows an amenity mapped as a building outline to be served as a
point.

**`--slim`, without `--drop`.** This keeps the middle tables — osm2pgsql's own
record of every node, way and relation it has seen. They cost disk, and they
are the entire difference between an import that can accept a daily diff and
one that can only ever be replaced wholesale. An import run without them
records `updatable=false`, and no later flag will change that; the only repair
is a full re-import. `--drop` discards them, which is why it must not be used
here.

**`--extra-attributes`.** Records each object's OSM version and timestamp. That
makes it possible to say when a feature was last edited, and to answer "what
changed" at all.

Two further decisions are specific to LUPMIS.

**A separate `osm` schema — this one is not negotiable.** osm2pgsql `--create`
drops and recreates every table it owns in its schema. The `spatial` schema
holds `lu_parcels`, the hand-edited planning data this whole system exists to
manage. Pointing osm2pgsql at `spatial` puts a `--create` run one name
collision away from destroying it. Import into `osm`; expose to the API through
views in `spatial` (§7).

**Districts are stamped after import, not during.** osm2pgsql imports the
national Ghana extract and has no concept of a district. Every LUPMIS table and
endpoint is district-scoped, so a `districtid` column is filled by a spatial
join once the import finishes (§5). This has to run after every daily update as
well, and §5.2 explains why that turns out to be cheap.

---

## 1. Find out what you actually have

Run these before anything else. They decide whether this is a re-import or a
first proper import, and none of the answers can be guessed from outside.

```sql
-- Does an osm2pgsql import exist at all, and on what terms?
SELECT property, value FROM osm2pgsql_properties
 ORDER BY property;
-- Look for: updatable, attributes, output, style,
--           import_timestamp, replication_base_url, replication_sequence_number
-- If this table does not exist, pi_osm_roads was loaded some other way
-- (ogr2ogr, a hand-written script, a one-off SQL dump).

-- Where does the current roads data live, and what is in it?
SELECT table_schema, table_name
  FROM information_schema.tables
 WHERE table_name LIKE '%osm%' OR table_name LIKE 'planet_osm%'
 ORDER BY 1, 2;

SELECT column_name, data_type
  FROM information_schema.columns
 WHERE table_name = 'pi_osm_roads'
 ORDER BY ordinal_position;

-- Middle tables present? Their absence is what makes --append impossible.
SELECT to_regclass('osm.planet_osm_nodes') AS nodes,
       to_regclass('osm.planet_osm_ways')  AS ways,
       to_regclass('osm.planet_osm_rels')  AS rels;
```

**If `osm2pgsql_properties` does not exist**, there is nothing to preserve or
migrate: skip §2 and import fresh. That is the more likely case here, because a
`pi_`-prefixed table in `spatial` with exactly six columns looks like a
purpose-built extract rather than anything osm2pgsql produced.

**If it does exist**, note `replication_base_url` and
`replication_sequence_number` before touching anything — even a non-updatable
import records them, and they tell a fresh import where the data left off.

---

## 2. Clear out only what is safe to clear

Nothing in this section touches `spatial`. The live application keeps reading
`spatial.pi_osm_roads` until §7 switches it over.

```sql
CREATE SCHEMA IF NOT EXISTS osm;
```

If a previous osm2pgsql run left pgsql-output tables in `osm`, drop these four
and **no others**:

```sql
DROP TABLE IF EXISTS osm.planet_osm_point,
                     osm.planet_osm_line,
                     osm.planet_osm_polygon,
                     osm.planet_osm_roads;
```

`planet_osm_nodes`, `planet_osm_ways`, `planet_osm_rels` and
`planet_osm_users` look like the same family and are not — they are the middle
tables that `--slim` exists to keep, and flex uses them exactly as the pgsql
output did. Dropping them costs you `--append`, which is the point of this
exercise.

Leave `osm2pgsql_properties` alone; `--create` rewrites it.

---

## 3. The flex configuration

Save as `sql/lupmis-osm.lua` next to wherever the import is run.

Four tables, matching the four datasets requested. Every geometry is EPSG:4326,
so the API serves what the table stores and nothing transforms on read. Each
table carries a `districtid` column that the import leaves empty and §5 fills.

```lua
-- lupmis-osm.lua — osm2pgsql flex config for LUPMIS2
--
-- Four tables in the `osm` schema:
--   pi_osm_roads     routable `highway` ways      (LineString)
--   pi_osm_lines     other linear features        (LineString)
--   pi_osm_points    amenities and healthcare     (Point)
--   pi_osm_polygons  buildings, land use, areas   (MultiPolygon)
--
-- districtid is declared here but never written by the import. It is filled by
-- the spatial join in §5 of the runbook. Because osm2pgsql re-inserts a row
-- whenever its object changes, an updated row comes back with districtid NULL —
-- which is what makes the incremental re-stamp in §5.2 both simple and correct.

local srid = 4326

local roads = osm2pgsql.define_table({
    name = 'pi_osm_roads', schema = 'osm',
    ids = { type = 'way', id_column = 'osm_id' },
    columns = {
        { column = 'highway',    type = 'text' },   -- the column that was missing
        { column = 'name',       type = 'text' },
        { column = 'ref',        type = 'text' },
        { column = 'surface',    type = 'text' },
        { column = 'oneway',     type = 'text' },
        { column = 'bridge',     type = 'text' },
        { column = 'tunnel',     type = 'text' },
        { column = 'layer',      type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'linestring', projection = srid, not_null = true },
    }
})

local lines = osm2pgsql.define_table({
    name = 'pi_osm_lines', schema = 'osm',
    ids = { type = 'way', id_column = 'osm_id' },
    columns = {
        { column = 'waterway',   type = 'text' },
        { column = 'railway',    type = 'text' },
        { column = 'power',      type = 'text' },
        { column = 'barrier',    type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'linestring', projection = srid, not_null = true },
    }
})

-- Points accept nodes, ways and relations, so an amenity mapped as a building
-- outline or a multipolygon appears here as a point like any other. That is the
-- single biggest gain over the classic layout, where areas lived in
-- planet_osm_polygon and the endpoint never looked there.
local points = osm2pgsql.define_table({
    name = 'pi_osm_points', schema = 'osm',
    ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
    columns = {
        { column = 'amenity',    type = 'text' },
        { column = 'healthcare', type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'geom',       type = 'point', projection = srid, not_null = true },
    }
})

local polygons = osm2pgsql.define_table({
    name = 'pi_osm_polygons', schema = 'osm',
    ids = { type = 'any', type_column = 'osm_type', id_column = 'osm_id' },
    columns = {
        { column = 'building',   type = 'text' },
        { column = 'landuse',    type = 'text' },
        { column = 'amenity',    type = 'text' },
        { column = 'leisure',    type = 'text' },
        { column = 'natural',    type = 'text' },
        { column = 'name',       type = 'text' },
        { column = 'districtid', type = 'int'  },
        { column = 'tags',       type = 'jsonb' },
        { column = 'geom',       type = 'multipolygon', projection = srid, not_null = true },
    }
})

-- Tags that say nothing once the typed columns exist.
local uninteresting = {
    'created_by', 'source', 'source:date', 'note', 'comment',
    'fixme', 'FIXME', 'attribution',
}

local function clean(tags)
    for _, k in ipairs(uninteresting) do tags[k] = nil end
end

local function is_area(tags)
    return tags.area == 'yes'
        or tags.building or tags.landuse or tags.leisure or tags.natural
end

function osm2pgsql.process_node(object)
    local t = object.tags
    if t.amenity or t.healthcare then
        points:insert({
            amenity    = t.amenity,
            healthcare = t.healthcare,
            name       = t.name,
            geom       = object:as_point(),
        })
    end
end

function osm2pgsql.process_way(object)
    local t = object.tags
    clean(t)

    if object.is_closed and is_area(t) then
        polygons:insert({
            building = t.building, landuse = t.landuse, amenity = t.amenity,
            leisure  = t.leisure,  natural = t['natural'], name = t.name,
            tags = t, geom = object:as_polygon(),
        })
        -- An amenity mapped as an area is also a point, so it is findable
        -- alongside amenities mapped as nodes.
        if t.amenity or t.healthcare then
            points:insert({
                amenity = t.amenity, healthcare = t.healthcare, name = t.name,
                geom = object:as_polygon():centroid(),
            })
        end
        return
    end

    if t.highway then
        roads:insert({
            highway = t.highway, name = t.name, ref = t.ref,
            surface = t.surface, oneway = t.oneway,
            bridge  = t.bridge,  tunnel = t.tunnel, layer = t.layer,
            tags = t, geom = object:as_linestring(),
        })
    elseif t.waterway or t.railway or t.power or t.barrier then
        lines:insert({
            waterway = t.waterway, railway = t.railway,
            power    = t.power,    barrier = t.barrier, name = t.name,
            tags = t, geom = object:as_linestring(),
        })
    end
end

function osm2pgsql.process_relation(object)
    local t = object.tags
    clean(t)
    if t.type ~= 'multipolygon' and t.type ~= 'boundary' then return end

    if is_area(t) or t.amenity or t.healthcare then
        polygons:insert({
            building = t.building, landuse = t.landuse, amenity = t.amenity,
            leisure  = t.leisure,  natural = t['natural'], name = t.name,
            tags = t, geom = object:as_multipolygon(),
        })
        if t.amenity or t.healthcare then
            points:insert({
                amenity = t.amenity, healthcare = t.healthcare, name = t.name,
                geom = object:as_multipolygon():centroid(),
            })
        end
    end
end
```

---

## 4. Import

```bash
curl -O https://download.geofabrik.de/africa/ghana-latest.osm.pbf

osm2pgsql --create --slim --output=flex \
  --style sql/lupmis-osm.lua --extra-attributes \
  --database lupmis --schema osm --middle-schema osm \
  --cache 2000 \
  ghana-latest.osm.pbf
```

`--slim` **without** `--drop`. `--drop` discards the middle tables and is what
makes an import permanently un-updatable.

Substitute the real database name for `lupmis`.

---

## 5. Stamp the districts

osm2pgsql knows nothing about districts, so this step has no equivalent in the
standard OSM tooling. It is what makes a national import usable by an
application where every table and every endpoint is district-scoped.

### 5.1 First pass, after the initial import

Confirm the boundary table's real name and column first — the application only
ever sees it through `get_district_boundary.php`:

```sql
SELECT table_schema, table_name FROM information_schema.tables
 WHERE table_name ILIKE '%district%';
```

Then, for each of the four tables (shown here for roads):

```sql
UPDATE osm.pi_osm_roads r
   SET districtid = d.districtid
  FROM spatial.districts d              -- confirm this name first
 WHERE r.districtid IS NULL
   AND ST_Intersects(d.geom, r.geom);
```

A linear feature crossing a district boundary matches more than one district.
`UPDATE` takes an arbitrary one of them, which is wrong for a road that spans a
boundary. If roads must appear in every district they touch, use a join table
instead of a column:

```sql
CREATE TABLE osm.pi_osm_roads_district AS
SELECT r.osm_id, d.districtid
  FROM osm.pi_osm_roads r
  JOIN spatial.districts d ON ST_Intersects(d.geom, r.geom);
CREATE INDEX ON osm.pi_osm_roads_district (districtid);
```

Points never have this problem. For polygons, decide whether a district should
own an area by intersection or by where its centroid falls —
`ST_Intersects(d.geom, ST_Centroid(p.geom))` gives one district per polygon.

### 5.2 After every update

The same `UPDATE` again — `WHERE districtid IS NULL` is doing real work here,
not just guarding against repetition.

When an object changes, osm2pgsql deletes its row and re-inserts it from the
new data. The Lua config never writes `districtid`, so the re-inserted row
comes back **NULL**. `WHERE districtid IS NULL` therefore selects exactly the
objects that are new or were edited since the last stamp — including any that
moved across a boundary — and nothing else. A daily re-stamp touches a few
hundred rows rather than millions.

Put it in the same cron entry as the update, immediately after it. If the two
ever get separated, the symptom is new roads that no district can see.

---

## 6. Indexes

osm2pgsql creates the geometry and id indexes. Add what the endpoints filter on:

```sql
CREATE INDEX ON osm.pi_osm_roads    (districtid);
CREATE INDEX ON osm.pi_osm_lines    (districtid);
CREATE INDEX ON osm.pi_osm_points   (districtid);
CREATE INDEX ON osm.pi_osm_polygons (districtid);

CREATE INDEX ON osm.pi_osm_roads  (highway)               WHERE highway   IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (amenity)               WHERE amenity    IS NOT NULL;
CREATE INDEX ON osm.pi_osm_points (healthcare)            WHERE healthcare IS NOT NULL;
CREATE INDEX ON osm.pi_osm_roads    USING gin (tags);
CREATE INDEX ON osm.pi_osm_polygons USING gin (tags);
```

---

## 7. Expose it to the API

Keep osm2pgsql's tables in `osm`, and give the API views in `spatial` under the
existing `pi_` naming. The application then reads the names it already expects,
and no import can ever write into `spatial`.

The current `spatial.pi_osm_roads` is a table, and a view cannot replace a table
of the same name. Rename it rather than dropping it, so there is a way back
until the new endpoints have been verified:

```sql
ALTER TABLE spatial.pi_osm_roads RENAME TO pi_osm_roads_pre_osm2pgsql;

CREATE VIEW spatial.pi_osm_roads AS
  SELECT osm_id, highway, name, ref, surface, oneway, districtid, geom
    FROM osm.pi_osm_roads;

CREATE VIEW spatial.pi_osm_points AS
  SELECT osm_type, osm_id, amenity, healthcare, name, districtid, geom
    FROM osm.pi_osm_points;

CREATE VIEW spatial.pi_osm_lines AS
  SELECT osm_id, waterway, railway, power, barrier, name, districtid, geom
    FROM osm.pi_osm_lines;

CREATE VIEW spatial.pi_osm_polygons AS
  SELECT osm_type, osm_id, building, landuse, amenity, leisure, name, districtid, geom
    FROM osm.pi_osm_polygons;
```

The existing `get_osm_roads.php` keeps working unchanged and gains `highway`,
because it reads `spatial.pi_osm_roads` and the view now supplies that column.

Three new endpoints are needed, in the shape of the existing one — same
`{ api_token, district_id }` request, same `{ success, data: [...] }` response,
geometry as WKT in `geom`:

| Endpoint | Serves | Application layer |
|---|---|---|
| `get_osm_points.php` | `spatial.pi_osm_points` | OSM Points |
| `get_osm_lines.php` | `spatial.pi_osm_lines` | OSM Lines |
| `get_osm_polygons.php` | `spatial.pi_osm_polygons` | OSM Polygons |

All four belong to layer group **5, Physical Infrastructures**, which is where
`OSM_roads` already sits.

**Volumes are the thing to watch.** Roads alone are 5,238 rows for district 1
under the current extract, and the endpoint returns every row as WKT in one
response. Points and polygons will be larger. Before wiring the new layers into
the application, decide whether these endpoints should take a bounding box, or
a `highway`/`amenity` filter, rather than returning a whole district. That
decision belongs with the endpoints, not the application.

---

## 8. Keep it current

This is the part the import exists for. Everything above only pays off if the
daily update actually runs.

### 8.1 osm2pgsql-replication needs a Python that has its libraries

`osm2pgsql-replication` ships with osm2pgsql but is a **Python** script, and
package managers do not install what it imports. Its first run says:

```
Missing required Python libraries psycopg2 osmium.
To install them via pip run: pip install psycopg2 osmium
```

Following that advice often changes nothing, and it is worth understanding why
before chasing it. The script's shebang is `#!/usr/bin/env python3`, and **`env`
does not see shell aliases**. If `python3` in your shell is aliased — a MAMP
installation does exactly this, and MAMP is present on at least one machine in
this project — then `pip3 install` puts the libraries where the script will
never look. Check before doing anything:

```bash
env python3 -c "import sys; print(sys.executable)"
env python3 -c "import psycopg2, osmium; print('both present')"
```

A Homebrew Python is also likely to be PEP 668 "externally managed", which
refuses the install outright.

**On the Linux server, use distribution packages** — no virtualenv, no pip:

```bash
apt install python3-psycopg2 python3-pyosmium      # Debian/Ubuntu
```

**On a workstation**, give the script an interpreter that has both, once:

```bash
python3.12 -m venv ~/.venvs/osm2pgsql
~/.venvs/osm2pgsql/bin/pip install psycopg2-binary osmium
```

`psycopg2-binary` rather than `psycopg2`: it ships as a wheel and needs no
`pg_config` or compiler. Then invoke the tool with that interpreter:

```bash
~/.venvs/osm2pgsql/bin/python $(brew --prefix)/bin/osm2pgsql-replication …
```

You know it is working when the message changes from the missing-library error
to `Updates not set up correctly. Run 'osm2pgsql-replication init' first.` —
that is the tool running properly and telling you about §8.2, not about Python.

### 8.2 Initialise, then run daily

```bash
osm2pgsql-replication init \
  --database lupmis --schema osm --osm-file ghana-latest.osm.pbf
```

That reads the replication URL and sequence number out of the extract's header,
so updates start exactly where the downloaded file left off.

Then daily, with the district stamp in the same job:

```bash
osm2pgsql-replication update \
  --database lupmis --schema osm \
  --diff-file /var/osm/diffs/$(date +%F).osc.gz \
  -- --slim --output=flex --style sql/lupmis-osm.lua --extra-attributes \
     --schema osm --middle-schema osm \
&& psql -d lupmis -f sql/stamp-districts.sql
```

The append updates the tables in place; there is no refresh step. `&&` rather
than `;` so a failed update does not leave the stamp running against half-applied
data.

In cron, spell out the full path to every binary — `osm2pgsql-replication`,
`psql`, and the Python interpreter if §8.1 required one. A cron job does not
inherit your shell's `PATH`, and this is the most common reason a daily update
works when run by hand and silently never runs from cron.

`--diff-file` keeps each day's changes. It is what lets you answer "what
changed near this parcel last week", which nothing else in this pipeline
records.

---

## 9. Check it worked

```sql
-- The column that started all this.
SELECT highway, count(*) FROM osm.pi_osm_roads
 WHERE highway IS NOT NULL GROUP BY 1 ORDER BY 2 DESC LIMIT 12;

-- Amenities mapped as areas — these could not appear in the old layout.
SELECT osm_type, count(*) FROM osm.pi_osm_points
 WHERE amenity IS NOT NULL GROUP BY 1;

-- Clinics carrying healthcare and no amenity, invisible until now.
SELECT count(*) FROM osm.pi_osm_points
 WHERE amenity IS NULL AND healthcare IS NOT NULL;

-- Nothing left unstamped. A non-zero count is usually genuine — offshore
-- features, or gaps between district polygons — but check before assuming.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid IS NULL;

-- Comparable with the endpoint's 5,238 for district 1.
SELECT count(*) FROM osm.pi_osm_roads WHERE districtid = 1;

-- The doors are open this time.
SELECT property, value FROM osm.osm2pgsql_properties
 WHERE property IN ('updatable','attributes','output','current_timestamp');
-- updatable and attributes must both be true.
```

Report freshness from `current_timestamp`, which moves with every append, not
`import_timestamp`, which is fixed at the original import.

Once the endpoints serve from the views and the application is verified, the
renamed original can go:

```sql
DROP TABLE spatial.pi_osm_roads_pre_osm2pgsql;
```

---

## 10. When it goes wrong

Fetch a fresh extract, re-run §4 with `--create`, re-run §5.1 and
`osm2pgsql-replication init`. Nothing in `spatial` is touched by any of it,
which is the entire reason the import lives in its own schema.