Document DB snapshot restore

docs/developer-guide/100-system-administration.md

@@ -268,6 +268,50 @@ mv -v data/old/* $PGDATA
 rm -r data/new data/old
 ```
 
+### Attaching DB snapshot to an EC2 instance
+For data recovery or testing purposes, it may be desirable to attach a DB snapshot to an existing instance. The following steps explain how to do this using an AWS EC2 EBS snapshot of the Docker volumes.
+
+#### New instance
+A volume based on a snapshot can be easily created and attached to a new EC2 instance using the `Provision Host` GitHub action or the `provision_host.sh` bash script directly.
+
+#### Rollback an existing instance using AWS SSM
+When using the SSM documents provided in `cloudformation-create-ssm-documents.yml` the rollback process can be completed automatically in just a few minutes. This reduces the number of manual steps required and is therefore less error-prone.
+To start the rollback process, select the `replace_volume` SSM document from the AWS SSM console. The document requires only a few parameters to be filled in:
+
+- `SnapshotId` (The ID of the snapshot to restore from)
+- `VolumeId` (The ID of the EBS volume to replace with the snapshot)
+- `DeleteVolume` (Specifies whether the existing EBS volume should be deleted after replacement. Set to `true` to delete the volume or `false` to keep it.)
+
+When executing the automation, AWS will perform several actions under the hood.
+
+1. Retrieve the instance details for the specified `VolumeId`
+2. Create a new EBS volume from the snapshot specified in `SnapshotId`
+3. Detach the current EBS volume and attach the newly created volume.
+4. Execute the `attach_volume` SSM automation. This automation runs several commands directly on the EC2 instance to mount the volume and update the `fstab` file. The `tag` attached to the EBS volume is also updated to ensure that AWS Data Lifecycle Manager (DLM) targets the volume when creating daily and weekly backups.
+5. Based on the `DeleteVolume` parameter, the original EBS volume is either deleted or retained as backup.
+
+#### Manually rollback an existing instance
+1. Create a new EBS volume from the EBS snapshot
+2. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command
+3. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps)
+4. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot`
+5. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data`
+6. Start a temporary PostgreSQL instance to connect to the snapshot DB. Locate the PostgreSQL data volume shown by `ls /mnt/snapshot` (generally called `or_postgresql-data`) and use that name in the `-v /mnt/snapshot/<YOUR_POSTGRES_VOLUME_NAME>/_data:/var/lib/postgresql/data` mount below.
+    ```shell
+    sudo docker run -d \
+      --rm \
+      --name temp_recovery_db \
+      -p 5433:5432 \
+      -e POSTGRES_PASSWORD=postgres \
+      -v /mnt/snapshot/<YOUR_POSTGRES_VOLUME_NAME>/_data:/var/lib/postgresql/data \
+      -v /tmp:/export \
+      openremote/postgresql:latest
+    ```
+7. Wait for the DB to be ready. When starting from a large snapshot, this can take a while; check the Docker container logs for the DB-ready message.
+8. Extract/Analyse the data as required either by:
+   * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote`
+   * Modifying the Manager and Keycloak DB settings to connect to the temporary DB on port 5433.
+
 ### Useful resources
 - [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB)
 - [Index maintenance](https://wiki.postgresql.org/wiki/Index_Maintenance)