Update guide

1 files changed, 231 insertions, 3 deletions

diff --git a/op_guide/stolon/manual_backup.md b/op_guide/stolon/manual_backup.md
index b952174..4cbc055 100644
--- a/op_guide/stolon/manual_backup.md
+++ b/op_guide/stolon/manual_backup.md
@@ -27,25 +27,253 @@ First setup a SSH tunnel on your machine that bind postgresql, eg:
 ssh -L 5432:psql-proxy.service.2.cluster.deuxfleurs.fr:5432 ...
 ```
 
+*Later, we will use `/tmp/sql` as our working directory. Depending on your distribution, this
+folder may be a `tmpfs` and thus mounted on RAM. If it is the case, choose another folder, that is not a `tmpfs`, otherwise you will fill your RAM
+and fail your backup. I am using NixOS and the `/tmp` folder is a regular folder, persisted on disk, which explain why I am using it.*
+
 Then export your password in `PGPASSWORD` and launch the backup:
 
 ```bash
 export PGPASSWORD=xxx
 
+mkdir -p /tmp/sql
+cd /tmp/sql
+
 pg_basebackup \
   --host=127.0.0.1 \
   --username=replicator \
   --pgdata=/tmp/sql \
   --format=tar \
-  --wal-method=none \
+  --wal-method=stream \
   --gzip \
   --compress=6 \
   --progress \
-  --max-rate=2M
+  --max-rate=5M
 ```
 
+*Something you should now: while it seems optional, fetching the WAL is mandatory. At first, I thought it was a way to have a "more recent backup".
+But after some reading, it appears that the base backup is corrupted because it is not a snapshot at all, but a copy of the postgres folder with no specific state.
+The whole point of the WAL is, in fact, to fix this corrupted archive...*
+
 *Take a cup of coffe, it will take some times...*
 
+The result I get (the important file is `base.tar.gz`, `41921.tar.gz` will probably be missing as it is a secondary tablespace I will deactivate soon):
+
+```
+[nix-shell:/tmp/sql]$ ls
+41921.tar.gz  backup_manifest  base.tar.gz  pg_wal.tar.gz
+```
+
+From now, disconnect from the production to continue your work.
+You don't need it anymore and it will prevent some disaster if you fail a command.
+
+
 ## Importing the backup
 
-## Dump SQL
+> The backup taken with `pg_basebckup` is an exact copy of your data directory so, all you need to do to restore from that backup is to point postgres at that directory and start it up.
+
+```bash
+mkdir -p /tmp/sql/pg_data && cd /tmp/sql/pg_data
+tar xzfv ../base.tar.gz
+```
+
+Now you should have something like that:
+
+```
+[nix-shell:/tmp/sql/pg_data]$ ls
+backup_label      base    pg_commit_ts  pg_hba.conf    pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_wal   postgresql.conf              tablespace_map
+backup_label.old  global  pg_dynshmem   pg_ident.conf  pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   pg_xact  stolon-temp-postgresql.conf
+```
+
+Now we will extract the WAL:
+
+```bash
+mkdir -p /tmp/sql/wal && cd /tmp/sql/wal
+tar xzfv ../pg_wal.tar.gz
+```
+
+You should have something like that:
+
+```
+[nix-shell:/tmp/sql/wal]$ ls
+00000003000014AF000000C9  00000003000014AF000000CA  00000003.history  archive_status
+```
+
+Now, We must edit/read some files before launching our ephemeral server:
+  - Set `listen_addresses = '0.0.0.0'` in `postgresql.conf` 
+  - Add `restore_command = 'cp /mnt/wal/%f %p' ` in `postgresql.conf` 
+  - Check `port` in `postgresql.conf`, in our case it is `5433`.
+  - Create an empty file named `recovery.signal`
+
+*Do not create a `recovery.conf` file, it might be written on the internet but this is a deprecated method and your postgres daemon will refuse to boot if it finds one.*
+
+*Currently, we use port 5433 in oour postgresql configuration despite 5432 being the default port. Indeed, in production, clients access the cluster transparently through the Stolon Proxy that listens on port 5432 and redirect the requests to the correct PostgreSQL instance, listening secretly on port 5433! To export our binary backup in text, we will directly query our postgres instance without passing through the proxy, which is why you must note this port.*
+
+Now we will start our postgres container on our machine.
+
+At the time of writing the live version is `superboum/amd64_postgres:v9`.
+We must start by getting `postgres` user id. Our container are run by default with this user, so you only need to run:
+
+```bash
+docker run --rm -it superboum/amd64_postgres:v9 id
+```
+
+And we get:
+
+```
+uid=999(postgres) gid=999(postgres) groups=999(postgres),101(ssl-cert)
+```
+
+Now `chown` your `pg_data`:
+
+```bash
+chown 999:999 -R /tmp/sql/{pg_data,wal}
+chmod 700 -R /tmp/sql/{pg_data,wal}
+```
+
+And finally:
+
+```
+docker run \
+  --rm \
+  -it \
+  -p 5433:5433 \
+  -v /tmp/sql/:/mnt/ \
+  superboum/amd64_postgres:v9 \
+  postgres -D /mnt/pg_data
+```
+
+I have the following output:
+
+```
+2022-01-28 14:46:39.750 GMT [1] LOG:  skipping missing configuration file "/mnt/pg_data/postgresql.auto.conf"
+2022-01-28 14:46:39.763 UTC [1] LOG:  starting PostgreSQL 13.3 (Debian 13.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
+2022-01-28 14:46:39.764 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5433
+2022-01-28 14:46:39.767 UTC [1] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5433"
+2022-01-28 14:46:39.773 UTC [7] LOG:  database system was interrupted; last known up at 2022-01-28 14:33:13 UTC
+cp: cannot stat '/mnt/wal/00000004.history': No such file or directory
+2022-01-28 14:46:40.318 UTC [7] LOG:  starting archive recovery
+2022-01-28 14:46:40.321 UTC [7] LOG:  restored log file "00000003.history" from archive
+2022-01-28 14:46:40.336 UTC [7] LOG:  restored log file "00000003000014AF000000C9" from archive
+2022-01-28 14:46:41.426 UTC [7] LOG:  could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
+2022-01-28 14:46:41.445 UTC [7] LOG:  could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
+2022-01-28 14:46:41.457 UTC [7] LOG:  redo starts at 14AF/C9000028
+2022-01-28 14:46:41.500 UTC [7] LOG:  restored log file "00000003000014AF000000CA" from archive
+2022-01-28 14:46:42.461 UTC [7] LOG:  consistent recovery state reached at 14AF/CA369AB0
+2022-01-28 14:46:42.461 UTC [1] LOG:  database system is ready to accept read only connections
+cp: cannot stat '/mnt/wal/00000003000014AF000000CB': No such file or directory
+2022-01-28 14:46:42.463 UTC [7] LOG:  redo done at 14AF/CA369AB0
+2022-01-28 14:46:42.463 UTC [7] LOG:  last completed transaction was at log time 2022-01-28 14:35:04.698438+00
+2022-01-28 14:46:42.480 UTC [7] LOG:  could not open directory "pg_tblspc/41921/PG_13_202007201": No such file or directory
+2022-01-28 14:46:42.493 UTC [7] LOG:  restored log file "00000003000014AF000000CA" from archive
+cp: cannot stat '/mnt/wal/00000004.history': No such file or directory
+2022-01-28 14:46:43.462 UTC [7] LOG:  selected new timeline ID: 4
+2022-01-28 14:46:44.441 UTC [7] LOG:  archive recovery complete
+2022-01-28 14:46:44.444 UTC [7] LOG:  restored log file "00000003.history" from archive
+2022-01-28 14:46:45.614 UTC [1] LOG:  database system is ready to accept connections
+```
+
+*Notes: the missing tablespace is a legacy tablesplace used in the past to debug Matrix. It will be removed soon, we can safely ignore it. Other errors on cp seems to be intended as postgres might want to know how far it can rewind with the WAL but I a not 100% sure.*
+
+Your ephemeral instance should work:
+
+```bash
+export PGPASSWORD=xxx # your postgres (admin) account password
+
+psql -h 127.0.0.1 -p 5433 -U postgres postgres
+```
+
+And your databases should appear:
+
+```
+[nix-shell:~/Documents/dev/infrastructure]$ psql -h 127.0.0.1 -p 5433 -U postgres postgres
+psql (13.5, server 13.3 (Debian 13.3-1.pgdg100+1))
+Type "help" for help.
+
+postgres=# \l
+                                 List of databases
+   Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges
+-----------+----------+----------+------------+------------+-----------------------
+ xxxxx     | xxxxx    | UTF8     | en_US.utf8 | en_US.utf8 |
+ xxxxx     | xxxxx    | UTF8     | en_US.utf8 | en_US.utf8 |
+ xxxxx     | xxxxx    | UTF8     | en_US.utf8 | en_US.utf8 |
+ postgres  | postgres | UTF8     | en_US.utf8 | en_US.utf8 |
+ xxxx      | xxxxx    | UTF8     | en_US.utf8 | en_US.utf8 |
+ xxxx      | xxxxx    | UTF8     | en_US.utf8 | en_US.utf8 |
+ template0 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+ template1 | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +
+           |          |          |            |            | postgres=CTc/postgres
+(8 rows)
+```
+
+## Dump your ephemeral database as SQL
+
+Now we can do a SQL export of our ephemeral database.
+We use zstd to automatically compress the outputed file.
+We use multiple parameters:
+  - `-vv` gives use some idea on the progress
+  - `-9` is a quite high value and should compress efficiently. Decrease it if your machine is low powered
+  - `-T0` asks zstd to use all your cores. By default, zstd uses only one core.
+
+```bash
+pg_dumpall -h 127.0.0.1 -p 5433 -U postgres \
+  | zstd -vv -9 -T0 --format=zstd > dump-`date --rfc-3339=seconds | sed 's/ /T/'`.sql.zstd
+```
+
+I get the following result:
+
+```
+[nix-shell:/tmp/sql]$ ls -lah dump*
+-rw-r--r-- 1 quentin users 749M janv. 28 16:07 dump-2022-01-28T16:06:29+01:00.sql.zstd
+```
+
+Now you can stop your ephemeral server.
+
+## Restore your SQL file
+
+First, start a blank server:
+
+```bash
+docker run \
+  --rm -it \
+  --name postgres \
+  -p 5433:5432 \
+  superboum/amd64_postgres:v9 \
+  bash -c '
+    set -ex
+    mkdir /tmp/psql
+    initdb -D /tmp/psql --no-locale --encoding=UTF8
+    echo "host all postgres 0.0.0.0/0 md5" >> /tmp/psql/pg_hba.conf
+    postgres -D /tmp/psql
+  '
+```
+
+Then set the same password as your prod for the `posgtgres` user (it will be required as part of the restore):
+
+```bash
+docker exec -ti postgres bash -c "echo \"ALTER USER postgres WITH PASSWORD '$PGPASSWORD';\" | psql"
+echo '\l' | psql -h 127.0.0.1 -p 5433 -U postgres postgres
+# the database should have no entry (except `posgtres`, `template0` and `template1`) otherwise ABORT EVERYTHING, YOU ARE ON THE WRONG DB
+```
+
+And finally, restore your SQL backup:
+
+```bash
+zstdcat -vv dump-* | \
+  grep -P -v '^(CREATE|DROP) ROLE postgres;' | \
+  psql -h 127.0.0.1 -p 5433 -U postgres --set ON_ERROR_STOP=on  postgres
+```
+
+*Note: we must skip CREATE/DROP ROLE postgres during the restore as it aready exists and would generate an error.
+Because we want to be extra careful, we specifically asked to crash on every error and do not want to change this behavior.
+So, instead, we simply remove any entry that contains the specific regex stated in the previous command.*
+
+
+## Ref
+
+https://philipmcclarence.com/backing-up-and-restoring-postgres-using-pg_basebackup/
+https://www.cybertec-postgresql.com/en/pg_basebackup-creating-self-sufficient-backups/
+https://www.postgresql.org/docs/14/continuous-archiving.html
+https://www.postgresql.org/docs/14/backup-dump.html#BACKUP-DUMP-RESTORE
+https://dba.stackexchange.com/questions/75033/how-to-restore-everything-including-postgres-role-from-pg-dumpall-backup