Rabbi Gerzi State and Database Recovery

This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the current application state directory and PostgreSQL database with the selected archive.

The backup contains:

/var/lib/rabbi-gerzi
/var/backup/rabbi-gerzi/rabbi-gerzi.dump

Requirements

BorgBase repository URL: ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo
Decrypted Borg SSH private key and repository passphrase
The original Rabbi Gerzi production environment, including APP_KEY
A target server already switched to the Rabbi Gerzi NixOS module
A shell with borg, openssh, postgresql, and rsync

On NixOS or another machine with Nix:

nix-shell -p borgbackup openssh postgresql rsync

The commands below assume the default state directory, database, user, and PHP-FPM pool names. Substitute the configured values if the NixOS module uses non-default names.

Prepare Borg Access

Copy the Borg SSH key into a local recovery directory:

mkdir -p ~/borg-recovery/rabbi-gerzi
cp /path/to/borg-private-key \
  ~/borg-recovery/rabbi-gerzi/borg-private-key
chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key

Set the Borg connection environment:

export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo'
export BORG_RSH='ssh'\
' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\
' -o IdentitiesOnly=yes'\
' -o StrictHostKeyChecking=accept-new'

Read the Borg passphrase without displaying it:

read -rsp 'Borg passphrase: ' BORG_PASSPHRASE
export BORG_PASSPHRASE
echo

Extract an Archive

List the available archives:

borg list

Choose an archive, then extract it into a temporary working directory. Do not extract directly into /.

export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST'

mkdir -p ~/borg-recovery/rabbi-gerzi/extract
cd ~/borg-recovery/rabbi-gerzi/extract

borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \
  var/backup/rabbi-gerzi/rabbi-gerzi.dump

Verify Without Replacing Production

Restore the extracted dump into a temporary database and inspect its core table counts:

sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify
sudo -u postgres createdb \
  --owner=rabbi-gerzi rabbi-gerzi-backup-verify
sudo -u postgres pg_restore \
  --exit-on-error \
  --no-owner \
  --role=rabbi-gerzi \
  --dbname=rabbi-gerzi-backup-verify \
  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL'
SELECT 'users' AS table_name, count(*) FROM users
UNION ALL
SELECT 'sets', count(*) FROM sets
UNION ALL
SELECT 'elements', count(*) FROM elements;
SQL

List the extracted managed files and confirm expected uploads are present:

find ./var/lib/rabbi-gerzi/storage/app/public -type f -print

Remove only the temporary verification database when the checks pass:

sudo -u postgres dropdb rabbi-gerzi-backup-verify

Restore Production

Run these commands from ~/borg-recovery/rabbi-gerzi/extract on the target server. Capture absolute tool paths so they remain available through sudo:

RSYNC="$(command -v rsync)"
PSQL="$(command -v psql)"
DROPDB="$(command -v dropdb)"
CREATEDB="$(command -v createdb)"
PG_RESTORE="$(command -v pg_restore)"

Stop the API and ensure PostgreSQL is running:

sudo systemctl stop phpfpm-rabbi-gerzi.service
sudo systemctl start postgresql.service

Replace the application state and repair ownership:

sudo "$RSYNC" -a --delete \
  ./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/
sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi

Replace the PostgreSQL database:

sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL'
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'rabbi-gerzi'
  AND pid <> pg_backend_pid();
SQL

sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi
sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi
sudo -u postgres "$PG_RESTORE" \
  --exit-on-error \
  --no-owner \
  --role=rabbi-gerzi \
  --dbname=rabbi-gerzi \
  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump

Run current migrations and restart the API:

set -e
sudo systemctl restart rabbi-gerzi-setup.service
sudo systemctl start phpfpm-rabbi-gerzi.service

Verify Production

Check service state and recent logs:

sudo systemctl status rabbi-gerzi-setup.service --no-pager
sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager
sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100
sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100

Open the public site and confirm that sets, elements, uploaded images, PDFs, and administrator login all work.

4.6 KiB Raw Blame History