Health Sync — Deployment Runbook

This runbook walks through exactly how a Health Sync deployment is done, command by command, so an engineer can reproduce it end to end. Everything here is the actual procedure used at Rwanda’s nine live facility sites; only credentials, hostnames, and IPs have been replaced with placeholders. For what the system is and why it is designed this way, start with the Health Sync overview.

Conventions used throughout:

The deployment has two halves:

• Central side (done once, plus a preload per site): PostgreSQL + API stack, site token generation, historical data preload.
• Facility side (done per site): server assessment, safe MySQL/Docker preparation, source-database indexing, runner installation.

Part A — Central server setup (once)

A.1 Compose stack: PostgreSQL + API

The central stack is a server-local compose file (deliberately not committed to the repository — site runners self-update from the repo, and the central API must never restart just because a runner fix was pushed).

docker-compose.central.yml:

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: healthsync
      POSTGRES_PASSWORD: <strong-password>     # avoid @ : / # — they break connection URLs
      POSTGRES_DB: healthsync
    volumes:
      - pg-data:/var/lib/postgresql/data
 
  api:
    build: { context: ./api }
    restart: unless-stopped
    depends_on: [db]
    deploy:
      replicas: 4          # declarative — survives a plain `up -d`
    environment:
      DATABASE_URL: postgres://healthsync:<strong-password>@db:5432/healthsync
      TOKEN_SIGNING_SECRET: <64-hex-chars>     # see below; min 32 chars, API refuses to start without it
      PORT: 3000
 
volumes:
  pg-data:

Generate the token-signing secret (this is the master secret — every site token is an HMAC keyed by it; rotating it invalidates all site tokens at once):

node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

Bring the stack up and verify:

docker compose -f docker-compose.central.yml up -d --build
curl -s http://localhost:3000/status | python3 -m json.tool

Hard-won rule: always scope compose commands to the service you are deploying (up -d --build api, not a bare up -d). On a shared server, an unscoped up -d recreates every service in the project — we once restarted an unrelated team’s containers this way.

A.2 HTTPS reverse proxy

Runners talk to the API over HTTPS. On the central host, nginx terminates TLS (Let’s Encrypt via certbot) and load-balances across the API replicas:

sudo apt update
sudo apt install -y --no-install-recommends certbot python3-certbot-nginx
sudo certbot --nginx -d <central-host>

server {
    listen 443 ssl;                      # or an alternate port if 443 is taken — check first!
    server_name <central-host>;
 
    ssl_certificate     /etc/letsencrypt/live/<central-host>/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/<central-host>/privkey.pem;
 
    client_max_body_size 50m;            # sync batches can be large — avoid silent 413s
 
    location / {
        resolver 127.0.0.11 valid=10s;   # Docker's embedded DNS — resolves replicas at request time
        set $upstream http://api:3000;   # variable proxy_pass = deferred resolution
        proxy_pass $upstream;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

sudo nginx -t && sudo systemctl reload nginx
curl -s https://<central-host>/status

War story: before assuming network-level TLS interception when a port serves the wrong certificate, check who owns the port locally — sudo ss -tlnp | grep ':443'. In our case the “interceptor” was another team’s container already bound to 443 on the shared server. We moved to an alternate port in twenty minutes after spending hours on interception theories.

A.3 PostgreSQL tuning for a shared host

On a shared government server, tune PostgreSQL to coexist with other tenants. Two gotchas: ALTER SYSTEM cannot run inside a transaction (use a heredoc, not psql -c "a; b;"), and shared_buffers needs a full restart, not a reload.

docker compose -f docker-compose.central.yml exec -T db \
  psql -U healthsync -d healthsync <<'SQL'
ALTER SYSTEM SET shared_buffers = '3GB';
ALTER SYSTEM SET effective_cache_size = '8GB';
ALTER SYSTEM SET work_mem = '32MB';
ALTER SYSTEM SET maintenance_work_mem = '512MB';
ALTER SYSTEM SET random_page_cost = 1.1;
ALTER SYSTEM SET wal_compression = on;
ALTER SYSTEM SET checkpoint_timeout = '15min';
ALTER SYSTEM SET max_wal_size = '4GB';
ALTER SYSTEM SET autovacuum_naptime = '30s';
ALTER SYSTEM SET autovacuum_vacuum_scale_factor = 0.05;
SELECT pg_reload_conf();
SQL
 
docker compose -f docker-compose.central.yml restart db   # shared_buffers needs restart

Scale numbers to your host — these are for a 15 GB machine shared with ~25 other containers.

Part B — Site token generation (per site, on the central server)

Each site authenticates with a random 32-byte token. The raw token is printed once; only its HMAC hash and a lookup prefix are stored. Site identity is derived from the token at request time — never from the request body — so a misconfigured runner cannot write into another site’s schema.

node scripts/gen-token.js --site <site_id>
# → prints the raw token ONCE. Paste it into the runner wizard at the facility. It is never stored.

If the central DB is only reachable from inside the API container (recommended), run the generator inside it:

docker compose -f docker-compose.central.yml exec \
  -e NODE_PATH=/app/node_modules api \
  node scripts/gen-token.js --site <site_id>

Verify (no secrets are revealed by this query):

SELECT site_id, token_prefix, revoked, created_at FROM health_sync.site_tokens;

Part C — Historical preload (per site, on the central server)

Principle: preload history offline; sync only the delta. Letting the runner backfill years of history would hammer both the facility server and the central API. Instead, a database dump is loaded centrally, and the runner starts from a watermark just before the dump cutoff.

C.1 The one-command way

Everything below is automated in a single resumable script:

# Fresh site — full preload from a dump
./scripts/preload-site.sh --site <site_id> --dump ~/dumps/<site_id>/<dump>.sql.gz
 
# Options you will use:
#   --prune-before 2020-01-01   drop obs/encounter rows older than a date
#   --no-prune                  keep full history
#   --dry-run                   show what each phase would do, change nothing
#   --status                    show which of the 14 phases are complete
#   --reset                     forget state, start from phase 1
#   --only t1,t2,...            add tables to an ALREADY-live site (see C.4)

The script encodes every lesson from the manual runs: a hard 4 GB memory cap on the throwaway MySQL container (an uncapped one once starved the whole shared server), memory pre-flight checks with auto-abort, per-phase state files kept in the home directory (not /tmp — a reboot once wiped hours of progress), and per-table verification counts.

Always run preloads inside tmux. A dropped SSH session once cost an hour of import progress.

tmux new -s preload
./scripts/preload-site.sh --site <site_id> --dump <dump>
# detach: Ctrl-b d · reattach: tmux attach -t preload

C.2 What the script does under the hood

Understanding the phases matters more than memorizing the script — this is the part to adapt to your own EMR.

Phase 1 — Slim the dump. Multi-GB EMR dumps contain ~500 tables; we sync 18. A streaming Python filter extracts only the target tables (60–80% smaller, and it never loads the file into memory):

import re
TABLES = {'person', 'patient', 'person_name', 'patient_identifier',
          'encounter', 'encounter_type', 'concept_name', 'obs', ...}  # your table set
in_target = False
with open(dump, 'r', errors='replace') as src, open(out, 'w') as dst:  # errors='replace': real dumps contain non-UTF-8 bytes
    for line in src:
        m = re.match(r"-- Table structure for table `([^`]+)`", line)
        if m: in_target = m.group(1) in TABLES
        if in_target: dst.write(line)

Phase 2 — Throwaway, capped, speed-tuned MySQL container. Durability features are pointless for a disposable import target — disabling them makes the import 2–4× faster and safely cancellable. The memory cap is non-negotiable on a shared host:

docker run -d --name mysql-tmp \
  --memory=4g --memory-swap=4g \
  -e MYSQL_ROOT_PASSWORD=<strong-password> \
  -e MYSQL_DATABASE=openmrs \
  mysql:8.0 \
  --innodb-buffer-pool-size=2G \
  --innodb-flush-log-at-trx-commit=2 \
  --innodb-doublewrite=OFF \
  --skip-log-bin \
  --max-allowed-packet=1G
 
until docker exec mysql-tmp mysqladmin ping -uroot -p<strong-password> --silent 2>/dev/null; do sleep 1; done

Phase 3 — Import the slim dump with speed flags wrapped around it:

{
  echo "SET sql_mode='';"                                  # MySQL 8 strict mode rejects legacy zero-dates
  echo "SET autocommit=0; SET unique_checks=0; SET foreign_key_checks=0;"
  echo "ALTER INSTANCE DISABLE INNODB REDO_LOG;"
  cat <site>-slim.sql
  echo "COMMIT;"
  echo "ALTER INSTANCE ENABLE INNODB REDO_LOG;"
} | docker exec -i mysql-tmp mysql -uroot -p<strong-password> --max-allowed-packet=1G openmrs

Verify with instant estimates, not COUNT(*):

SELECT TABLE_NAME, TABLE_ROWS FROM information_schema.TABLES
 WHERE TABLE_SCHEMA='openmrs' ORDER BY TABLE_ROWS DESC;

Phase 4 (optional) — Prune old history. Only obs and encounter are pruned (other tables stay complete for referential integrity). Index the timestamp column first, then delete in primary-key ranges — never one giant DELETE:

START=<min_pk>; END=<max_pk>; STEP=100000
while [ $START -le $END ]; do
  docker exec mysql-tmp mysql -uroot -p<strong-password> -e "
    SET autocommit=1; SET foreign_key_checks=0;
    DELETE FROM openmrs.obs
     WHERE obs_id BETWEEN $START AND $((START+STEP-1))
       AND date_created < '2020-01-01';"
  START=$((START+STEP))
done

(foreign_key_checks=0 must be set inside the same statement — each docker exec mysql -e is a fresh session. obs has self-referential foreign keys that silently reject deletes otherwise.)

Phase 5 — Transfer into PostgreSQL. MySQL → PostgreSQL streaming, renaming the source schema to <site_id>, casting zero-dates to NULL, loading data only (no indexes, no foreign keys — added after). The project ships a purpose-built Python streaming loader (pymysql → PostgreSQL COPY) that creates tables from the runner’s declared column types, after the off-the-shelf migration tool proved unreliable at scale (silent hangs, a false-success mode, MySQL 8 auth incompatibility).

Phase 6 — Post-load fixes (required for the runner to work). The API’s upsert is INSERT … ON CONFLICT (<pk>) DO UPDATE — every table must have its primary key, and the API stamps _synced_at on every row:

-- every table in the site schema gets _synced_at
DO $$
DECLARE r record;
BEGIN
  FOR r IN SELECT tablename FROM pg_tables WHERE schemaname='<site_id>' LOOP
    EXECUTE format(
      'ALTER TABLE <site_id>.%I ADD COLUMN IF NOT EXISTS _synced_at TIMESTAMPTZ DEFAULT now()',
      r.tablename);
  END LOOP;
END$$;
 
-- primary keys (verify the real PK name per table — conventions are not universal!)
ALTER TABLE <site_id>.person ADD PRIMARY KEY (person_id);
ALTER TABLE <site_id>.obs    ADD PRIMARY KEY (obs_id);
-- ... etc. Example of a convention-breaker we hit:
ALTER TABLE <site_id>.moh_bill_insurance_policy ADD PRIMARY KEY (insurance_policy_id);

Phase 7 — Analytics indexes, built CONCURRENTLY (no table locks), smallest tables first, the giant obs indexes one at a time with validity checks between:

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_obs_concept_dt
  ON <site_id>.obs (concept_id, obs_datetime);
-- verify before starting the next one:
SELECT ic.relname, i.indisvalid
  FROM pg_index i JOIN pg_class c ON c.oid=i.indrelid
  JOIN pg_class ic ON ic.oid=i.indexrelid
  JOIN pg_namespace n ON n.oid=c.relnamespace
 WHERE n.nspname='<site_id>' AND c.relname='obs';
-- indisvalid = f → DROP INDEX CONCURRENTLY and retry

Phase 8 — Teardown and watermark note. docker rm -f mysql-tmp frees the memory, and the script prints the recommended start_from per table: max loaded timestamp minus one day. The overlap is deliberately generous — duplicates are free (upserts), gaps lose data.

C.3 Verifying a preload

docker compose -f docker-compose.central.yml exec db psql -U healthsync -d healthsync

\dn                                   -- site schemas
\dt <site_id>.*                       -- tables in the site schema
SELECT count(*) FROM <site_id>.obs;   -- compare against source counts

C.4 Adding tables to a live site later

Never re-run a greenfield preload against a live schema. The incremental mode is surgical: its own state namespace, per-table verification, prune/index phases skipped, and — critically — it never regenerates the site token (which would break the live runner):

./scripts/preload-site.sh --site <site_id> --dump <fresh-dump> \
  --only orders,concept_class,moh_bill_global_bill

Part D — Facility server preparation (per site)

D.1 Reconnaissance first — touch nothing until you know the machine

# OS + resources
lsb_release -a && uname -a && uptime && free -h && df -h / && nproc
 
# What's already installed and listening
dpkg -l | grep -iE "mysql|mariadb|docker"
sudo ss -tlnp
 
# Automatic-update policy — critical on a production EMR server
systemctl is-enabled unattended-upgrades
cat /etc/apt/apt.conf.d/20auto-upgrades
apt-mark showhold
tail -50 /var/log/apt/history.log

Record the answers. They decide everything that follows — and they are your evidence if anything on the machine changes later.

D.2 Stability hardening — do no harm to the EMR

# Prevent accidental OS release upgrades
sudo sed -i 's/^Prompt=.*/Prompt=never/' /etc/update-manager/release-upgrades

Anything you install gets version-pinned so unattended upgrades can never restart the EMR’s database out from under it:

sudo apt-mark hold docker-ce docker-ce-cli containerd.io docker-compose-plugin

Issues you find that belong to the facility (e.g., MySQL bound to all interfaces, missing package holds on the EMR’s own MySQL) are flagged to the facility’s administrator in writing — not changed unilaterally. You inherit a working clinical system; keep it working.

D.3 Read-only database user for the runner

The runner needs SELECT and nothing else. The '%' host matters — the runner connects from a container IP, not localhost:

CREATE USER 'syncer'@'%' IDENTIFIED BY '<strong-password>';
GRANT SELECT ON openmrs.* TO 'syncer'@'%';
FLUSH PRIVILEGES;

If MySQL is bound to 127.0.0.1 only, the container cannot reach it. Bind to the Docker bridge (the read-only user + host firewall keep it safe), and use the bridge gateway 172.17.0.1 as db.host in the runner config — never localhost, which inside a container means the container itself.

D.4 Index the timestamp columns — the single biggest performance factor

Every facility EMR database we met was missing indexes on date_created. At one site this made each sync cycle a 23-second full scan of a 16.7M-row table; the index build took under a minute and eliminated it.

-- check what exists
SHOW INDEX FROM obs;
SELECT COUNT(*) FROM obs;   -- know the size before you build
 
-- MySQL 8 online DDL: no table lock, safe on a live EMR, safe to cancel
-- NOTE: options separated by SPACES, not commas (a comma is a syntax error — verified on 8.0.42)
CREATE INDEX idx_obs_date_created ON obs (date_created) ALGORITHM=INPLACE LOCK=NONE;

Build one index at a time, smallest table first, largest (obs) last. If you must cancel, it is safe — MySQL rolls the partial index back (allow roughly the build time again for the rollback).

If the EMR’s sql_mode rejects legacy zero-date defaults during a build, scope the workaround to your session only: SET SESSION sql_mode='';

D.5 Docker install and the build-network pre-check

Install Docker with minimal footprint (--no-install-recommends), hold the packages (D.2). Then — before running the installer — verify the Docker build network works, because the host’s network being fine does not mean the build bridge is:

docker run --rm curlimages/curl -sI --max-time 10 https://github.com
docker run --rm curlimages/curl -sI --max-time 10 https://objects.githubusercontent.com

Both must return an HTTP status line. If they time out (we hit this twice — broken IPv6 preference, wrong MTU), fix it permanently in /etc/docker/daemon.json:

{ "ipv6": false, "dns": ["8.8.8.8", "1.1.1.1"], "mtu": 1450 }

sudo systemctl restart docker

On networks where apt itself fails over IPv6: echo 'Acquire::ForceIPv4 "true";' | sudo tee /etc/apt/apt.conf.d/99force-ipv4 (at one site this took apt update from 71 seconds of failures to 4.7 seconds).

Part E — Runner installation (per site)

E.1 The one-liner

export GITHUB_TOKEN=<github-pat>
bash <(curl -fsSL -H "Authorization: token $GITHUB_TOKEN" \
  https://raw.githubusercontent.com/<org>/health-sync/main/scripts/site-install.sh)

Flags if you need them: --api-token <token> (pre-fill the wizard), --config <path> (reuse an existing config.json), --install-dir <path> (default ~/health-sync).

The installer checks Docker, clones the repository, builds the runner image, walks an interactive wizard, and starts the container.

E.2 Wizard cheat sheet

start_from is the safety-critical answer. Too early merely re-upserts preloaded rows (free). Unset or too late and you either re-pull years of history or silently skip rows. When in doubt, go earlier.

For deployment #2 onward, skip most of the wizard: copy a previous site’s config.json, change the five site-specific fields (site_id, api_token, DB password, start_from values), and pass it with --config.

Add tables to a running site later:

docker compose exec runner node scripts/add-table.js

E.3 The config that comes out (shape)

{
  "site_id": "<site_id>",
  "api_endpoint": "https://<central-host>",
  "api_token": "<raw-token>",
  "db": { "type": "mysql", "host": "172.17.0.1", "port": 3306,
          "user": "syncer", "password": "<strong-password>", "database": "openmrs" },
  "tables": [
    { "name": "obs", "primary_key": "obs_id", "timestamp_col": "date_created",
      "start_from": "2026-06-29 00:00:00",
      "columns": { "obs_id": "integer", "person_id": "integer",
                   "date_created": "timestamp", "...": "..." } }
  ],
  "sync_interval_seconds": 60,
  "update_check_interval_seconds": 300,
  "heartbeat_interval_seconds": 30,
  "github": { "repo": "<org>/health-sync", "branch": "main", "token": "<github-pat>" }
}

Notes: columns maps names → PostgreSQL types (integer, bigint, text, boolean, date, timestamp, numeric, jsonb) and is an object, not an array — the central API creates the tables from it. Watch source types: we once nearly declared a decimal column as integer. config.json is git-ignored — it holds secrets and is never committed.

Two pitfalls at this exact step, hit at multiple sites:

1. Never start the container before config.json exists — Docker silently creates the missing bind-mount path as a directory, and every later start fails with a confusing error. Fix: docker compose down && rm -rf config.json, create the real file, start again.
2. If the facility can’t reach Docker Hub at all, configure a registry mirror in daemon.json before building.

E.4 Verify the deployment

# 1. Runner logs — look for sync cycles and zero errors
docker compose logs -f runner
 
# 2. Heartbeat visible centrally?
curl -s https://<central-host>/status | python3 -m json.tool
#    → site should be "online", queue_depth near 0, last_sync recent
 
# 3. Rows actually flowing? (on the central server)
#    run twice a few minutes apart — the count should grow with clinic activity
SELECT count(*) FROM <site_id>.obs WHERE _synced_at > now() - interval '1 hour';

A site is done when: heartbeats are green, queue depth returns to ~0 after each cycle, central row counts advance, and one full self-update cycle has been observed.

Part F — Day-2 operations

F.1 Fleet monitoring

curl -s https://<central-host>/status | python3 -m json.tool   # online / stale / offline per site

The dashboard (same host, /dashboard) shows hourly ingestion bar charts per site and table — a stalled site appears as a visible gap, which non-technical staff notice far faster than a stale timestamp.

F.2 Shipping a fix to all sites

Push to the main branch. Every runner polls, pulls, and restarts itself within update_check_interval_seconds — no site visits. Queued data survives restarts (durable SQLite state). The central API deliberately does not auto-update; deploy it with a scoped rebuild:

git pull && docker compose -f docker-compose.central.yml up -d --build api

F.3 The on-runner debug console

Every runner can expose a loopback-only console (never reachable from the network):

DEV_MODE=1 docker compose up -d runner
docker compose exec runner nc localhost 9100

status                                  # queue depth, watermarks, MySQL alive/dead, code version
sync obs                                # trigger one table's sync cycle now
queue 10                                # peek at queued items
set watermark obs 2026-06-29T00:00:00Z  # force re-sync from a point in time (safe: upserts dedupe)
ping db / ping api                      # connectivity checks
pause / resume                          # halt all loops without stopping the container

The set watermark command is also how the day-granularity tables (created_date of type date, e.g. several moh_bill_* tables) get their periodic backfill — a strict > watermark on a date column can miss same-day late writes, so the watermark is periodically nudged back one day; the re-sent rows dedupe for free.

F.4 Troubleshooting cheat sheet

F.5 Anti-patterns we learned to avoid

The safety rules, condensed

If you take one page home, take this one:

1. Never uncapped, never unpinned, never unlogged. Memory caps on every batch container; apt-mark hold on everything you install; every error path logs.
2. Read-only at the source. The sync system must be physically unable to damage clinical data.
3. Duplicates are free; gaps lose data. Overlap watermarks, upsert everything, re-send without fear.
4. Resumable or it doesn’t run. Phase state files (outside /tmp), tmux, idempotent re-runs.
5. Measure, then touch. Row counts, SHOW INDEX, load averages, and timed queries before any change on a live hospital server.
6. Flag what you don’t own. Facility-owned issues go to the facility’s administrator in writing.
7. Verify per table, per site. Primary-key names, timestamp column names and types, dump formats, and character encodings all vary — even across facilities running the same EMR product.

All commands verified in production across nine facility deployments, March–July 2026. Placeholders replace real hostnames, credentials, and tokens throughout — generate your own.