From cee2d18ab2fbdbd373f9d9a051791e4c9cfef51f Mon Sep 17 00:00:00 2001 From: Yisroel Baum Date: Mon, 17 Aug 2026 20:52:30 +0300 Subject: [PATCH] document borg backup recovery --- README.md | 46 +++++++++++++ RECOVERY.md | 185 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 231 insertions(+) create mode 100644 RECOVERY.md diff --git a/README.md b/README.md index cfc6dad..2360599 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,10 @@ RABBI_GERZI_INITIAL_ADMIN_PASSWORD=replace-with-a-long-random-password - `RABBI_GERZI_INITIAL_ADMIN_PASSWORD` is used only to create that admin if it does not already exist. +The module also requires paths to a Borg repository passphrase file and SSH +private key through `services.rabbi-gerzi.secretFiles`. Keep both files +outside the Nix store and readable only by root. + Generate an app key inside the dev shell: ```sh @@ -148,6 +152,34 @@ cd frontend/rabbi_gerzi printf '%s\n' 'VITE_API_BASE_URL=http://127.0.0.1:8000' > .env.local ``` +## Production Backups + +The NixOS module creates an encrypted Borg backup in: + +```text +ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo +``` + +Each archive contains the complete application state directory and a +custom-format PostgreSQL dump. This covers database records, users, sessions, +and every uploaded icon and PDF. The Nix store, runtime secrets, and +`/var/cache/rabbi-gerzi` are reproducible or recovered separately and are not +archived. + +The persistent timer runs daily at 05:15 in the server's local timezone and +keeps 7 daily, 4 weekly, and 6 monthly archives. PHP-FPM drains active requests +and remains stopped while the dump and archive are created, so database rows +and uploaded files come from the same write-free window. The static frontend +remains available during that window. + +Run an immediate backup with: + +```sh +sudo systemctl start borgbackup-job-rabbi-gerzi.service +``` + +See [RECOVERY.md](./RECOVERY.md) for archive verification and full restoration. + ## Local Development Enter the Nix dev shell before running local development commands: @@ -364,6 +396,8 @@ Example host configuration: "app-key" = { }; "admin-email" = { }; "admin-password" = { }; + "borg-passphrase" = { }; + "borg-private-key" = { }; }; templates."rabbi-gerzi.env".content = '' @@ -376,6 +410,13 @@ Example host configuration: services.rabbi-gerzi = { enable = true; + secretFiles = { + borgPassphrase = + config.sops.secrets."borg-passphrase".path; + borgPrivateKey = + config.sops.secrets."borg-private-key".path; + }; + frontend.hostName = "rabbigerzi.com"; backend = { @@ -412,6 +453,11 @@ Add these values in the editor opened by `sops`: app-key: base64:replace-with-laravel-app-key admin-email: admin@example.com admin-password: replace-with-a-long-random-password +borg-passphrase: replace-with-the-repository-passphrase +borg-private-key: | + -----BEGIN OPENSSH PRIVATE KEY----- + replace-with-the-private-key + -----END OPENSSH PRIVATE KEY----- ``` Commit only the encrypted file written by `sops`. At activation time, diff --git a/RECOVERY.md b/RECOVERY.md new file mode 100644 index 0000000..84903b8 --- /dev/null +++ b/RECOVERY.md @@ -0,0 +1,185 @@ +# Rabbi Gerzi State and Database Recovery + +This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the +current application state directory and PostgreSQL database with the selected +archive. + +The backup contains: + +- `/var/lib/rabbi-gerzi` +- `/var/backup/rabbi-gerzi/rabbi-gerzi.dump` + +## Requirements + +- BorgBase repository URL: + `ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo` +- Decrypted Borg SSH private key and repository passphrase +- The original Rabbi Gerzi production environment, including `APP_KEY` +- A target server already switched to the Rabbi Gerzi NixOS module +- A shell with `borg`, `openssh`, `postgresql`, and `rsync` + +On NixOS or another machine with Nix: + +```sh +nix-shell -p borgbackup openssh postgresql rsync +``` + +The commands below assume the default state directory, database, user, and +PHP-FPM pool names. Substitute the configured values if the NixOS module uses +non-default names. + +## Prepare Borg Access + +Copy the Borg SSH key into a local recovery directory: + +```sh +mkdir -p ~/borg-recovery/rabbi-gerzi +cp /path/to/borg-private-key \ + ~/borg-recovery/rabbi-gerzi/borg-private-key +chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key +``` + +Set the Borg connection environment: + +```sh +export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo' +export BORG_RSH='ssh'\ +' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\ +' -o IdentitiesOnly=yes'\ +' -o StrictHostKeyChecking=accept-new' +``` + +Read the Borg passphrase without displaying it: + +```sh +read -rsp 'Borg passphrase: ' BORG_PASSPHRASE +export BORG_PASSPHRASE +echo +``` + +## Extract an Archive + +List the available archives: + +```sh +borg list +``` + +Choose an archive, then extract it into a temporary working directory. Do not +extract directly into `/`. + +```sh +export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST' + +mkdir -p ~/borg-recovery/rabbi-gerzi/extract +cd ~/borg-recovery/rabbi-gerzi/extract + +borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \ + var/backup/rabbi-gerzi/rabbi-gerzi.dump +``` + +## Verify Without Replacing Production + +Restore the extracted dump into a temporary database and inspect its core +table counts: + +```sh +sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify +sudo -u postgres createdb \ + --owner=rabbi-gerzi rabbi-gerzi-backup-verify +sudo -u postgres pg_restore \ + --exit-on-error \ + --no-owner \ + --role=rabbi-gerzi \ + --dbname=rabbi-gerzi-backup-verify \ + ./var/backup/rabbi-gerzi/rabbi-gerzi.dump +sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL' +SELECT 'users' AS table_name, count(*) FROM users +UNION ALL +SELECT 'sets', count(*) FROM sets +UNION ALL +SELECT 'elements', count(*) FROM elements; +SQL +``` + +List the extracted managed files and confirm expected uploads are present: + +```sh +find ./var/lib/rabbi-gerzi/storage/app/public -type f -print +``` + +Remove only the temporary verification database when the checks pass: + +```sh +sudo -u postgres dropdb rabbi-gerzi-backup-verify +``` + +## Restore Production + +Run these commands from `~/borg-recovery/rabbi-gerzi/extract` on the target +server. Capture absolute tool paths so they remain available through `sudo`: + +```sh +RSYNC="$(command -v rsync)" +PSQL="$(command -v psql)" +DROPDB="$(command -v dropdb)" +CREATEDB="$(command -v createdb)" +PG_RESTORE="$(command -v pg_restore)" +``` + +Stop the API and ensure PostgreSQL is running: + +```sh +sudo systemctl stop phpfpm-rabbi-gerzi.service +sudo systemctl start postgresql.service +``` + +Replace the application state and repair ownership: + +```sh +sudo "$RSYNC" -a --delete \ + ./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/ +sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi +``` + +Replace the PostgreSQL database: + +```sh +sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL' +SELECT pg_terminate_backend(pid) +FROM pg_stat_activity +WHERE datname = 'rabbi-gerzi' + AND pid <> pg_backend_pid(); +SQL + +sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi +sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi +sudo -u postgres "$PG_RESTORE" \ + --exit-on-error \ + --no-owner \ + --role=rabbi-gerzi \ + --dbname=rabbi-gerzi \ + ./var/backup/rabbi-gerzi/rabbi-gerzi.dump +``` + +Run current migrations and restart the API: + +```sh +set -e +sudo systemctl restart rabbi-gerzi-setup.service +sudo systemctl start phpfpm-rabbi-gerzi.service +``` + +## Verify Production + +Check service state and recent logs: + +```sh +sudo systemctl status rabbi-gerzi-setup.service --no-pager +sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager +sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100 +sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100 +``` + +Open the public site and confirm that sets, elements, uploaded images, PDFs, +and administrator login all work.