document borg backup recovery
This commit is contained in:
parent
9cb10a96fd
commit
cee2d18ab2
GPG key ID:
0FA60884F75520A9
2 changed files with 231 additions and 0 deletions
46
README.md
46
README.md
|
|
@ -82,6 +82,10 @@ RABBI_GERZI_INITIAL_ADMIN_PASSWORD=replace-with-a-long-random-password
|
|||
- `RABBI_GERZI_INITIAL_ADMIN_PASSWORD` is used only to create that admin if
|
||||
it does not already exist.
|
||||
|
||||
The module also requires paths to a Borg repository passphrase file and SSH
|
||||
private key through `services.rabbi-gerzi.secretFiles`. Keep both files
|
||||
outside the Nix store and readable only by root.
|
||||
|
||||
Generate an app key inside the dev shell:
|
||||
|
||||
```sh
|
||||
|
|
@ -148,6 +152,34 @@ cd frontend/rabbi_gerzi
|
|||
printf '%s\n' 'VITE_API_BASE_URL=http://127.0.0.1:8000' > .env.local
|
||||
```
|
||||
|
||||
## Production Backups
|
||||
|
||||
The NixOS module creates an encrypted Borg backup in:
|
||||
|
||||
```text
|
||||
ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo
|
||||
```
|
||||
|
||||
Each archive contains the complete application state directory and a
|
||||
custom-format PostgreSQL dump. This covers database records, users, sessions,
|
||||
and every uploaded icon and PDF. The Nix store, runtime secrets, and
|
||||
`/var/cache/rabbi-gerzi` are reproducible or recovered separately and are not
|
||||
archived.
|
||||
|
||||
The persistent timer runs daily at 05:15 in the server's local timezone and
|
||||
keeps 7 daily, 4 weekly, and 6 monthly archives. PHP-FPM drains active requests
|
||||
and remains stopped while the dump and archive are created, so database rows
|
||||
and uploaded files come from the same write-free window. The static frontend
|
||||
remains available during that window.
|
||||
|
||||
Run an immediate backup with:
|
||||
|
||||
```sh
|
||||
sudo systemctl start borgbackup-job-rabbi-gerzi.service
|
||||
```
|
||||
|
||||
See [RECOVERY.md](./RECOVERY.md) for archive verification and full restoration.
|
||||
|
||||
## Local Development
|
||||
|
||||
Enter the Nix dev shell before running local development commands:
|
||||
|
|
@ -364,6 +396,8 @@ Example host configuration:
|
|||
"app-key" = { };
|
||||
"admin-email" = { };
|
||||
"admin-password" = { };
|
||||
"borg-passphrase" = { };
|
||||
"borg-private-key" = { };
|
||||
};
|
||||
|
||||
templates."rabbi-gerzi.env".content = ''
|
||||
|
|
@ -376,6 +410,13 @@ Example host configuration:
|
|||
services.rabbi-gerzi = {
|
||||
enable = true;
|
||||
|
||||
secretFiles = {
|
||||
borgPassphrase =
|
||||
config.sops.secrets."borg-passphrase".path;
|
||||
borgPrivateKey =
|
||||
config.sops.secrets."borg-private-key".path;
|
||||
};
|
||||
|
||||
frontend.hostName = "rabbigerzi.com";
|
||||
|
||||
backend = {
|
||||
|
|
@ -412,6 +453,11 @@ Add these values in the editor opened by `sops`:
|
|||
app-key: base64:replace-with-laravel-app-key
|
||||
admin-email: admin@example.com
|
||||
admin-password: replace-with-a-long-random-password
|
||||
borg-passphrase: replace-with-the-repository-passphrase
|
||||
borg-private-key: |
|
||||
-----BEGIN OPENSSH PRIVATE KEY-----
|
||||
replace-with-the-private-key
|
||||
-----END OPENSSH PRIVATE KEY-----
|
||||
```
|
||||
|
||||
Commit only the encrypted file written by `sops`. At activation time,
|
||||
|
|
|
|||
185
RECOVERY.md
Normal file
185
RECOVERY.md
Normal file
|
|
@ -0,0 +1,185 @@
|
|||
# Rabbi Gerzi State and Database Recovery
|
||||
|
||||
This procedure restores Rabbi Gerzi from its BorgBase backup. It replaces the
|
||||
current application state directory and PostgreSQL database with the selected
|
||||
archive.
|
||||
|
||||
The backup contains:
|
||||
|
||||
- `/var/lib/rabbi-gerzi`
|
||||
- `/var/backup/rabbi-gerzi/rabbi-gerzi.dump`
|
||||
|
||||
## Requirements
|
||||
|
||||
- BorgBase repository URL:
|
||||
`ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo`
|
||||
- Decrypted Borg SSH private key and repository passphrase
|
||||
- The original Rabbi Gerzi production environment, including `APP_KEY`
|
||||
- A target server already switched to the Rabbi Gerzi NixOS module
|
||||
- A shell with `borg`, `openssh`, `postgresql`, and `rsync`
|
||||
|
||||
On NixOS or another machine with Nix:
|
||||
|
||||
```sh
|
||||
nix-shell -p borgbackup openssh postgresql rsync
|
||||
```
|
||||
|
||||
The commands below assume the default state directory, database, user, and
|
||||
PHP-FPM pool names. Substitute the configured values if the NixOS module uses
|
||||
non-default names.
|
||||
|
||||
## Prepare Borg Access
|
||||
|
||||
Copy the Borg SSH key into a local recovery directory:
|
||||
|
||||
```sh
|
||||
mkdir -p ~/borg-recovery/rabbi-gerzi
|
||||
cp /path/to/borg-private-key \
|
||||
~/borg-recovery/rabbi-gerzi/borg-private-key
|
||||
chmod 600 ~/borg-recovery/rabbi-gerzi/borg-private-key
|
||||
```
|
||||
|
||||
Set the Borg connection environment:
|
||||
|
||||
```sh
|
||||
export BORG_REPO='ssh://mgjjruz9@mgjjruz9.repo.borgbase.com/./repo'
|
||||
export BORG_RSH='ssh'\
|
||||
' -i ~/borg-recovery/rabbi-gerzi/borg-private-key'\
|
||||
' -o IdentitiesOnly=yes'\
|
||||
' -o StrictHostKeyChecking=accept-new'
|
||||
```
|
||||
|
||||
Read the Borg passphrase without displaying it:
|
||||
|
||||
```sh
|
||||
read -rsp 'Borg passphrase: ' BORG_PASSPHRASE
|
||||
export BORG_PASSPHRASE
|
||||
echo
|
||||
```
|
||||
|
||||
## Extract an Archive
|
||||
|
||||
List the available archives:
|
||||
|
||||
```sh
|
||||
borg list
|
||||
```
|
||||
|
||||
Choose an archive, then extract it into a temporary working directory. Do not
|
||||
extract directly into `/`.
|
||||
|
||||
```sh
|
||||
export ARCHIVE='ARCHIVE_NAME_FROM_BORG_LIST'
|
||||
|
||||
mkdir -p ~/borg-recovery/rabbi-gerzi/extract
|
||||
cd ~/borg-recovery/rabbi-gerzi/extract
|
||||
|
||||
borg extract ::"$ARCHIVE" var/lib/rabbi-gerzi \
|
||||
var/backup/rabbi-gerzi/rabbi-gerzi.dump
|
||||
```
|
||||
|
||||
## Verify Without Replacing Production
|
||||
|
||||
Restore the extracted dump into a temporary database and inspect its core
|
||||
table counts:
|
||||
|
||||
```sh
|
||||
sudo -u postgres dropdb --if-exists rabbi-gerzi-backup-verify
|
||||
sudo -u postgres createdb \
|
||||
--owner=rabbi-gerzi rabbi-gerzi-backup-verify
|
||||
sudo -u postgres pg_restore \
|
||||
--exit-on-error \
|
||||
--no-owner \
|
||||
--role=rabbi-gerzi \
|
||||
--dbname=rabbi-gerzi-backup-verify \
|
||||
./var/backup/rabbi-gerzi/rabbi-gerzi.dump
|
||||
sudo -u postgres psql -d rabbi-gerzi-backup-verify <<'SQL'
|
||||
SELECT 'users' AS table_name, count(*) FROM users
|
||||
UNION ALL
|
||||
SELECT 'sets', count(*) FROM sets
|
||||
UNION ALL
|
||||
SELECT 'elements', count(*) FROM elements;
|
||||
SQL
|
||||
```
|
||||
|
||||
List the extracted managed files and confirm expected uploads are present:
|
||||
|
||||
```sh
|
||||
find ./var/lib/rabbi-gerzi/storage/app/public -type f -print
|
||||
```
|
||||
|
||||
Remove only the temporary verification database when the checks pass:
|
||||
|
||||
```sh
|
||||
sudo -u postgres dropdb rabbi-gerzi-backup-verify
|
||||
```
|
||||
|
||||
## Restore Production
|
||||
|
||||
Run these commands from `~/borg-recovery/rabbi-gerzi/extract` on the target
|
||||
server. Capture absolute tool paths so they remain available through `sudo`:
|
||||
|
||||
```sh
|
||||
RSYNC="$(command -v rsync)"
|
||||
PSQL="$(command -v psql)"
|
||||
DROPDB="$(command -v dropdb)"
|
||||
CREATEDB="$(command -v createdb)"
|
||||
PG_RESTORE="$(command -v pg_restore)"
|
||||
```
|
||||
|
||||
Stop the API and ensure PostgreSQL is running:
|
||||
|
||||
```sh
|
||||
sudo systemctl stop phpfpm-rabbi-gerzi.service
|
||||
sudo systemctl start postgresql.service
|
||||
```
|
||||
|
||||
Replace the application state and repair ownership:
|
||||
|
||||
```sh
|
||||
sudo "$RSYNC" -a --delete \
|
||||
./var/lib/rabbi-gerzi/ /var/lib/rabbi-gerzi/
|
||||
sudo chown -R rabbi-gerzi:rabbi-gerzi /var/lib/rabbi-gerzi
|
||||
```
|
||||
|
||||
Replace the PostgreSQL database:
|
||||
|
||||
```sh
|
||||
sudo -u postgres "$PSQL" -d postgres -v ON_ERROR_STOP=1 <<'SQL'
|
||||
SELECT pg_terminate_backend(pid)
|
||||
FROM pg_stat_activity
|
||||
WHERE datname = 'rabbi-gerzi'
|
||||
AND pid <> pg_backend_pid();
|
||||
SQL
|
||||
|
||||
sudo -u postgres "$DROPDB" --if-exists rabbi-gerzi
|
||||
sudo -u postgres "$CREATEDB" --owner=rabbi-gerzi rabbi-gerzi
|
||||
sudo -u postgres "$PG_RESTORE" \
|
||||
--exit-on-error \
|
||||
--no-owner \
|
||||
--role=rabbi-gerzi \
|
||||
--dbname=rabbi-gerzi \
|
||||
./var/backup/rabbi-gerzi/rabbi-gerzi.dump
|
||||
```
|
||||
|
||||
Run current migrations and restart the API:
|
||||
|
||||
```sh
|
||||
set -e
|
||||
sudo systemctl restart rabbi-gerzi-setup.service
|
||||
sudo systemctl start phpfpm-rabbi-gerzi.service
|
||||
```
|
||||
|
||||
## Verify Production
|
||||
|
||||
Check service state and recent logs:
|
||||
|
||||
```sh
|
||||
sudo systemctl status rabbi-gerzi-setup.service --no-pager
|
||||
sudo systemctl status phpfpm-rabbi-gerzi.service --no-pager
|
||||
sudo journalctl -u rabbi-gerzi-setup.service -b --no-pager -n 100
|
||||
sudo journalctl -u phpfpm-rabbi-gerzi.service -b --no-pager -n 100
|
||||
```
|
||||
|
||||
Open the public site and confirm that sets, elements, uploaded images, PDFs,
|
||||
and administrator login all work.
|
||||
Loading…
Add table
Add a link
Reference in a new issue