# Rabbi Gerzi State and Database Recovery

This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the
current application state directory and PostgreSQL database with the selected
archive.

The backup contains:

- `/var/lib/rabbi-gerzi`
- `/var/backup/rabbi-gerzi/rabbi-gerzi.dump`

## Requirements

- BorgBase repository URL:
  `ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo`
- Decrypted Borg SSH private key and repository passphrase
- The original Rabbi Gerzi production environment, including `APP_KEY`
- A target server already switched to the Rabbi Gerzi NixOS module
- A shell with `borg`, `openssh`, `postgresql`, and `rsync`

On NixOS or another machine with Nix:

```sh
nix-shell -p borgbackup openssh postgresql rsync
```

The commands below assume the default state directory, database, user, and
PHP-FPM pool names. Substitute the configured values if the NixOS module uses
non-default names.

## Prepare Borg Access

Copy the Borg SSH key into a local recovery directory:

```sh
mkdir -p ~/borg-recovery/rabbi-gerzi
cp /path/to/borg-private-key \
  ~/borg-recovery/rabbi-gerzi/borg-private-key
chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key
```

Set the Borg connection environment:

```sh
export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo'
export BORG_RSH='ssh'\
' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\
' -o IdentitiesOnly=yes'\
' -o StrictHostKeyChecking=accept-new'
```

Read the Borg passphrase without displaying it:

```sh
read -rsp 'Borg passphrase: ' BORG_PASSPHRASE
export BORG_PASSPHRASE
echo
```

## Extract an Archive

List the available archives:

```sh
borg list
```

Choose an archive, then extract it into a temporary working directory. Do not
extract directly into `/`.

```sh
export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST'

mkdir -p ~/borg-recovery/rabbi-gerzi/extract
cd ~/borg-recovery/rabbi-gerzi/extract

borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \
  var/backup/rabbi-gerzi/rabbi-gerzi.dump
```

## Verify Without Replacing Production

Restore the extracted dump into a temporary database and inspect its core
table counts:

```sh
sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify
sudo -u postgres createdb \
  --owner=rabbi-gerzi rabbi-gerzi-backup-verify
sudo -u postgres pg_restore \
  --exit-on-error \
  --no-owner \
  --role=rabbi-gerzi \
  --dbname=rabbi-gerzi-backup-verify \
  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL'
SELECT 'users' AS table_name, count(*) FROM users
UNION ALL
SELECT 'sets', count(*) FROM sets
UNION ALL
SELECT 'elements', count(*) FROM elements;
SQL
```

List the extracted managed files and confirm expected uploads are present:

```sh
find ./var/lib/rabbi-gerzi/storage/app/public -type f -print
```

Remove only the temporary verification database when the checks pass:

```sh
sudo -u postgres dropdb rabbi-gerzi-backup-verify
```

## Restore Production

Run these commands from `~/borg-recovery/rabbi-gerzi/extract` on the target
server. Capture absolute tool paths so they remain available through `sudo`:

```sh
RSYNC="$(command -v rsync)"
PSQL="$(command -v psql)"
DROPDB="$(command -v dropdb)"
CREATEDB="$(command -v createdb)"
PG_RESTORE="$(command -v pg_restore)"
```

Stop the API and ensure PostgreSQL is running:

```sh
sudo systemctl stop phpfpm-rabbi-gerzi.service
sudo systemctl start postgresql.service
```

Replace the application state and repair ownership:

```sh
sudo "$RSYNC" -a --delete \
  ./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/
sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi
```

Replace the PostgreSQL database:

```sh
sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL'
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'rabbi-gerzi'
  AND pid <> pg_backend_pid();
SQL

sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi
sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi
sudo -u postgres "$PG_RESTORE" \
  --exit-on-error \
  --no-owner \
  --role=rabbi-gerzi \
  --dbname=rabbi-gerzi \
  ./var/backup/rabbi-gerzi/rabbi-gerzi.dump
```

Run current migrations and restart the API:

```sh
set -e
sudo systemctl restart rabbi-gerzi-setup.service
sudo systemctl start phpfpm-rabbi-gerzi.service
```

## Verify Production

Check service state and recent logs:

```sh
sudo systemctl status rabbi-gerzi-setup.service --no-pager
sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager
sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100
sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100
```

Open the public site and confirm that sets, elements, uploaded images, PDFs,
and administrator login all work.