document borg backup recovery

README.md

@@ -82,6 +82,10 @@ RABBI_GERZI_INITIAL_ADMIN_PASSWORD=replace-with-a-long-random-password
 - `RABBI_GERZI_INITIAL_ADMIN_PASSWORD` is used only to create that admin if
   it does not already exist.
 
+The module also requires paths to a Borg repository passphrase file and SSH
+private key through `services.rabbi-gerzi.secretFiles`. Keep both files
+outside the Nix store and readable only by root.
+
 Generate an app key inside the dev shell:
 
 ```sh
@@ -148,6 +152,34 @@ cd frontend/rabbi_gerzi
 printf '%s\n' 'VITE_API_BASE_URL=http://127.0.0.1:8000' > .env.local
 ```
 
+## Production Backups
+
+The NixOS module creates an encrypted Borg backup in:
+
+```text
+ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo
+```
+
+Each archive contains the complete application state directory and a
+custom-format PostgreSQL dump. This covers database records, users, sessions,
+and every uploaded icon and PDF. The Nix store, runtime secrets, and
+`/var/cache/rabbi-gerzi` are reproducible or recovered separately and are not
+archived.
+
+The persistent timer runs daily at 05:15 in the server's local timezone and
+keeps 7 daily, 4 weekly, and 6 monthly archives. PHP-FPM drains active requests
+and remains stopped while the dump and archive are created, so database rows
+and uploaded files come from the same write-free window. The static frontend
+remains available during that window.
+
+Run an immediate backup with:
+
+```sh
+sudo systemctl start borgbackup-job-rabbi-gerzi.service
+```
+
+See [RECOVERY.md](./RECOVERY.md) for archive verification and full restoration.
+
 ## Local Development
 
 Enter the Nix dev shell before running local development commands:
@@ -364,6 +396,8 @@ Example host configuration:
                   "app-key" = { };
                   "admin-email" = { };
                   "admin-password" = { };
+                  "borg-passphrase" = { };
+                  "borg-private-key" = { };
                 };
 
                 templates."rabbi-gerzi.env".content = ''
@@ -376,6 +410,13 @@ Example host configuration:
               services.rabbi-gerzi = {
                 enable = true;
 
+                secretFiles = {
+                  borgPassphrase =
+                    config.sops.secrets."borg-passphrase".path;
+                  borgPrivateKey =
+                    config.sops.secrets."borg-private-key".path;
+                };
+
                 frontend.hostName = "rabbigerzi.com";
 
                 backend = {
@@ -412,6 +453,11 @@ Add these values in the editor opened by `sops`:
 app-key: base64:replace-with-laravel-app-key
 admin-email: admin@example.com
 admin-password: replace-with-a-long-random-password
+borg-passphrase: replace-with-the-repository-passphrase
+borg-private-key: |
+  -----BEGIN OPENSSH PRIVATE KEY-----
+  replace-with-the-private-key
+  -----END OPENSSH PRIVATE KEY-----
 ```
 
 Commit only the encrypted file written by `sops`. At activation time,

RECOVERY.md

@@ -0,0 +1,185 @@
+# Rabbi Gerzi State and Database Recovery
+
+This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the
+current application state directory and PostgreSQL database with the selected
+archive.
+
+The backup contains:
+
+- `/var/lib/rabbi-gerzi`
+- `/var/backup/rabbi-gerzi/rabbi-gerzi.dump`
+
+## Requirements
+
+- BorgBase repository URL:
+  `ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo`
+- Decrypted Borg SSH private key and repository passphrase
+- The original Rabbi Gerzi production environment, including `APP_KEY`
+- A target server already switched to the Rabbi Gerzi NixOS module
+- A shell with `borg`, `openssh`, `postgresql`, and `rsync`
+
+On NixOS or another machine with Nix:
+
+```sh
+nix-shell -p borgbackup openssh postgresql rsync
+```
+
+The commands below assume the default state directory, database, user, and
+PHP-FPM pool names. Substitute the configured values if the NixOS module uses
+non-default names.
+
+## Prepare Borg Access
+
+Copy the Borg SSH key into a local recovery directory:
+
+```sh
+mkdir -p ~/borg-recovery/rabbi-gerzi
+cp /path/to/borg-private-key \
+  ~/borg-recovery/rabbi-gerzi/borg-private-key
+chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key
+```
+
+Set the Borg connection environment:
+
+```sh
+export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo'
+export BORG_RSH='ssh'\
+' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\
+' -o IdentitiesOnly=yes'\
+' -o StrictHostKeyChecking=accept-new'
+```
+
+Read the Borg passphrase without displaying it:
+
+```sh
+read -rsp 'Borg passphrase: ' BORG_PASSPHRASE
+export BORG_PASSPHRASE
+echo
+```
+
+## Extract an Archive
+
+List the available archives:
+
+```sh
+borg list
+```
+
+Choose an archive, then extract it into a temporary working directory. Do not
+extract directly into `/`.
+
+```sh
+export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST'
+
+mkdir -p ~/borg-recovery/rabbi-gerzi/extract
+cd ~/borg-recovery/rabbi-gerzi/extract
+
+borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \
+  var/backup/rabbi-gerzi/rabbi-gerzi.dump
+```
+
+## Verify Without Replacing Production
+
+Restore the extracted dump into a temporary database and inspect its core
+table counts:
+
+```sh
+sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify
+sudo -u postgres createdb \
+  --owner=rabbi-gerzi rabbi-gerzi-backup-verify
+sudo -u postgres pg_restore \
+  --exit-on-error \
+  --no-owner \
+  --role=rabbi-gerzi \
+  --dbname=rabbi-gerzi-backup-verify \
+  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
+sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL'
+SELECT 'users' AS table_name, count(*) FROM users
+UNION ALL
+SELECT 'sets', count(*) FROM sets
+UNION ALL
+SELECT 'elements', count(*) FROM elements;
+SQL
+```
+
+List the extracted managed files and confirm expected uploads are present:
+
+```sh
+find ./var/lib/rabbi-gerzi/storage/app/public -type f -print
+```
+
+Remove only the temporary verification database when the checks pass:
+
+```sh
+sudo -u postgres dropdb rabbi-gerzi-backup-verify
+```
+
+## Restore Production
+
+Run these commands from `~/borg-recovery/rabbi-gerzi/extract` on the target
+server. Capture absolute tool paths so they remain available through `sudo`:
+
+```sh
+RSYNC="$(command -v rsync)"
+PSQL="$(command -v psql)"
+DROPDB="$(command -v dropdb)"
+CREATEDB="$(command -v createdb)"
+PG_RESTORE="$(command -v pg_restore)"
+```
+
+Stop the API and ensure PostgreSQL is running:
+
+```sh
+sudo systemctl stop phpfpm-rabbi-gerzi.service
+sudo systemctl start postgresql.service
+```
+
+Replace the application state and repair ownership:
+
+```sh
+sudo "$RSYNC" -a --delete \
+  ./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/
+sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi
+```
+
+Replace the PostgreSQL database:
+
+```sh
+sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL'
+SELECT pg_terminate_backend(pid)
+FROM pg_stat_activity
+WHERE datname = 'rabbi-gerzi'
+  AND pid <> pg_backend_pid();
+SQL
+
+sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi
+sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi
+sudo -u postgres "$PG_RESTORE" \
+  --exit-on-error \
+  --no-owner \
+  --role=rabbi-gerzi \
+  --dbname=rabbi-gerzi \
+  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
+```
+
+Run current migrations and restart the API:
+
+```sh
+set -e
+sudo systemctl restart rabbi-gerzi-setup.service
+sudo systemctl start phpfpm-rabbi-gerzi.service
+```
+
+## Verify Production
+
+Check service state and recent logs:
+
+```sh
+sudo systemctl status rabbi-gerzi-setup.service --no-pager
+sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager
+sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100
+sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100
+```
+
+Open the public site and confirm that sets, elements, uploaded images, PDFs,
+and administrator login all work.