document borg backup recovery

RECOVERY.md

@@ -0,0 +1,185 @@
+# Rabbi Gerzi State and Database Recovery
+
+This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the
+current application state directory and PostgreSQL database with the selected
+archive.
+
+The backup contains:
+
+- `/var/lib/rabbi-gerzi`
+- `/var/backup/rabbi-gerzi/rabbi-gerzi.dump`
+
+## Requirements
+
+- BorgBase repository URL:
+  `ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo`
+- Decrypted Borg SSH private key and repository passphrase
+- The original Rabbi Gerzi production environment, including `APP_KEY`
+- A target server already switched to the Rabbi Gerzi NixOS module
+- A shell with `borg`, `openssh`, `postgresql`, and `rsync`
+
+On NixOS or another machine with Nix:
+
+```sh
+nix-shell -p borgbackup openssh postgresql rsync
+```
+
+The commands below assume the default state directory, database, user, and
+PHP-FPM pool names. Substitute the configured values if the NixOS module uses
+non-default names.
+
+## Prepare Borg Access
+
+Copy the Borg SSH key into a local recovery directory:
+
+```sh
+mkdir -p ~/borg-recovery/rabbi-gerzi
+cp /path/to/borg-private-key \
+  ~/borg-recovery/rabbi-gerzi/borg-private-key
+chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key
+```
+
+Set the Borg connection environment:
+
+```sh
+export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo'
+export BORG_RSH='ssh'\
+' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\
+' -o IdentitiesOnly=yes'\
+' -o StrictHostKeyChecking=accept-new'
+```
+
+Read the Borg passphrase without displaying it:
+
+```sh
+read -rsp 'Borg passphrase: ' BORG_PASSPHRASE
+export BORG_PASSPHRASE
+echo
+```
+
+## Extract an Archive
+
+List the available archives:
+
+```sh
+borg list
+```
+
+Choose an archive, then extract it into a temporary working directory. Do not
+extract directly into `/`.
+
+```sh
+export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST'
+
+mkdir -p ~/borg-recovery/rabbi-gerzi/extract
+cd ~/borg-recovery/rabbi-gerzi/extract
+
+borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \
+  var/backup/rabbi-gerzi/rabbi-gerzi.dump
+```
+
+## Verify Without Replacing Production
+
+Restore the extracted dump into a temporary database and inspect its core
+table counts:
+
+```sh
+sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify
+sudo -u postgres createdb \
+  --owner=rabbi-gerzi rabbi-gerzi-backup-verify
+sudo -u postgres pg_restore \
+  --exit-on-error \
+  --no-owner \
+  --role=rabbi-gerzi \
+  --dbname=rabbi-gerzi-backup-verify \
+  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
+sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL'
+SELECT 'users' AS table_name, count(*) FROM users
+UNION ALL
+SELECT 'sets', count(*) FROM sets
+UNION ALL
+SELECT 'elements', count(*) FROM elements;
+SQL
+```
+
+List the extracted managed files and confirm expected uploads are present:
+
+```sh
+find ./var/lib/rabbi-gerzi/storage/app/public -type f -print
+```
+
+Remove only the temporary verification database when the checks pass:
+
+```sh
+sudo -u postgres dropdb rabbi-gerzi-backup-verify
+```
+
+## Restore Production
+
+Run these commands from `~/borg-recovery/rabbi-gerzi/extract` on the target
+server. Capture absolute tool paths so they remain available through `sudo`:
+
+```sh
+RSYNC="$(command -v rsync)"
+PSQL="$(command -v psql)"
+DROPDB="$(command -v dropdb)"
+CREATEDB="$(command -v createdb)"
+PG_RESTORE="$(command -v pg_restore)"
+```
+
+Stop the API and ensure PostgreSQL is running:
+
+```sh
+sudo systemctl stop phpfpm-rabbi-gerzi.service
+sudo systemctl start postgresql.service
+```
+
+Replace the application state and repair ownership:
+
+```sh
+sudo "$RSYNC" -a --delete \
+  ./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/
+sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi
+```
+
+Replace the PostgreSQL database:
+
+```sh
+sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL'
+SELECT pg_terminate_backend(pid)
+FROM pg_stat_activity
+WHERE datname = 'rabbi-gerzi'
+  AND pid <> pg_backend_pid();
+SQL
+
+sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi
+sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi
+sudo -u postgres "$PG_RESTORE" \
+  --exit-on-error \
+  --no-owner \
+  --role=rabbi-gerzi \
+  --dbname=rabbi-gerzi \
+  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
+```
+
+Run current migrations and restart the API:
+
+```sh
+set -e
+sudo systemctl restart rabbi-gerzi-setup.service
+sudo systemctl start phpfpm-rabbi-gerzi.service
+```
+
+## Verify Production
+
+Check service state and recent logs:
+
+```sh
+sudo systemctl status rabbi-gerzi-setup.service --no-pager
+sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager
+sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100
+sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100
+```
+
+Open the public site and confirm that sets, elements, uploaded images, PDFs,
+and administrator login all work.