Skip to Content
The HIC Learning Exchange begins July 13, 2026. View the agenda →
ReadingItem 5 of 22 · 20 min

Operations, Troubleshooting & Glossary

The platform is live — this page covers the day-two work: routine maintenance, a cross-layer troubleshooting table, and a glossary of the terms used throughout the module.

Part 4 of 4 — Overview · Lab 1: Provision & Configure · Lab 2: Deploy & Release

What you'll learn
  • Rotate the warehouse database password via GitHub secrets, without SSH or manual .env edits
  • Confirm Let's Encrypt TLS renewal is automated via certbot's cron
  • Take dated VM snapshots before risky changes
  • Diagnose cross-layer issues (host, VM, Ansible, CI/CD, app) using the troubleshooting table
  • Look up unfamiliar terms in the glossary

Phase 6 — Operations & Maintenance

Rotating the database password

A plaintext DB password was previously committed and remains in git history. Rotate it on the DB server itself, then update the secret and redeploy:

ALTER USER <DB_USER> WITH PASSWORD 'new-strong-password-here';
gh secret set DB_PASSWORD --body 'new-strong-password-here'

A push to main (or re-running the workflow) regenerates the VM’s .env and restarts the containers with the new value. General rule: rotating any credential = gh secret set NAME --body '…' + redeploy. No SSH or manual .env edit required.

Renewing TLS certificates

If you used Ansible playbook 04, certbot installed a daily renewal cron (certbot renew --quiet && systemctl reload nginx). Let’s Encrypt certs last 90 days; the cron keeps them fresh automatically.

Snapshots around risky changes

Take a fresh VM snapshot before any significant change (OS update, big deploy, config edit):

sudo virsh snapshot-create-as --domain <VM_NAME> \
  --name "pre-<change>-$(date +%Y%m%d)" --description "before <change>" --atomic

Name snapshots with a date + reason so their purpose is clear at revert time.

Phase 7 — Troubleshooting (cross-layer)

SymptomLayerFix
Port-forwarding rules gone after rebootHostsudo netfilter-persistent save (rules are volatile)
VM got a different IP after rebootVMNetplan not saved / netplan apply not run — redo Step 1.3
nc -zv <PUBLIC_IP> hangs from inside the VMVMExpected — hairpin NAT unsupported; test from an external machine
Port 443 refused despite nginx runningVMCheck ss -tlnp | grep 443; if empty, the sites-enabled symlink is missing — re-create it
nginx -t syntax errorVMUsually a missing ; or brace — sudo nginx -t 2>&1 shows the line
SSH asks to accept a new host keyanyVM was rebuilt — ssh-keygen -R <IP_ADDRESS> on your client
ansible all -m ping failsAnsibleCheck inventory ansible_user/key path and that port-forward/SSH is reachable
GHCR pull 401/403 on first deployCI/CDGive the repo Write access under each package’s Manage Actions access (Step 3.4)
Superset lost all dashboards / using SQLiteAppThe ./.superset:/app/pythonpath volume mount is missing — restore it
Dagster schedule never firesAppConfirm the schedule’s default_status=RUNNING; Dagster schedules default to stopped

Glossary

The full glossary of terms now lives on its own page: Glossary.