From 69ff8c5ab9d6861a4752f84521563573637602da Mon Sep 17 00:00:00 2001 From: Rich Turner <7072278+richturner@users.noreply.github.com> Date: Mon, 22 Jun 2026 13:06:01 +0100 Subject: [PATCH 1/9] Document DB snapshot restore --- docs/developer-guide/system-administration.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/developer-guide/system-administration.md b/docs/developer-guide/system-administration.md index faaf0d05..efbf436e 100644 --- a/docs/developer-guide/system-administration.md +++ b/docs/developer-guide/system-administration.md @@ -272,6 +272,31 @@ mv -v data/old/* $PGDATA rm -r data/new data/old ``` +### Attaching DB snapshot to an EC2 instance +For data recovery or testing purposes it may be desirable to attach a DB snapshot to an existing instance. the following steps explain how to do this using an AWS EC2 EBS snapshot of the docker volumes. + +#### New instance +A snapshot can be easily attached to a new EC2 instance using the `Provision Host` github action or the `provision_host.sh` bash script directly. + +#### Existing instance +1. Create a new EBS volume from the EBS snapshot +1. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command +1. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) +1. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` +2. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. + ```shell + sudo docker run -d \ + --rm \ + --name temp_recovery_db \ + -p 5433:5432 \ + -e POSTGRES_PASSWORD=postgres \ + -v /mnt/snapshot//_data:/var/lib/postgresql/data \ + -v /tmp:/export \ + openremote/postgresql:latest + ``` +1. Wait for the DB to be ready; it can take a while when starting from a large snapshot, look at the docker container logs for DB ready message +1. Exec into the container and run DB queries, exports, etc. `docker exec -it temp_recovery_db psql -U postgres -d openremote` + ### Useful resources - [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB) - [Index maintenance](https://wiki.postgresql.org/wiki/Index_Maintenance) From 67bf73e420e9b8783cbcc2240be73e995e8517d5 Mon Sep 17 00:00:00 2001 From: Rich Turner <7072278+richturner@users.noreply.github.com> Date: Mon, 22 Jun 2026 13:10:52 +0100 Subject: [PATCH 2/9] WIP --- docs/developer-guide/system-administration.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/developer-guide/system-administration.md b/docs/developer-guide/system-administration.md index efbf436e..7a75a2d3 100644 --- a/docs/developer-guide/system-administration.md +++ b/docs/developer-guide/system-administration.md @@ -295,7 +295,10 @@ A snapshot can be easily attached to a new EC2 instance using the `Provision Hos openremote/postgresql:latest ``` 1. Wait for the DB to be ready; it can take a while when starting from a large snapshot, look at the docker container logs for DB ready message -1. Exec into the container and run DB queries, exports, etc. `docker exec -it temp_recovery_db psql -U postgres -d openremote` +1. Extract/Analyse the data as required either by: + * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote` + or + * Modify the manager and keycloak DB settings to connect to the temporary DB on port 5433 ### Useful resources - [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB) From a9f4a70e903daff7e9dd9dc97aca394f03771f8d Mon Sep 17 00:00:00 2001 From: Rich Turner <7072278+richturner@users.noreply.github.com> Date: Mon, 22 Jun 2026 14:16:20 +0100 Subject: [PATCH 3/9] WIP --- docs/developer-guide/system-administration.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/developer-guide/system-administration.md b/docs/developer-guide/system-administration.md index 7a75a2d3..f157ee87 100644 --- a/docs/developer-guide/system-administration.md +++ b/docs/developer-guide/system-administration.md @@ -277,13 +277,15 @@ For data recovery or testing purposes it may be desirable to attach a DB snapsho #### New instance A snapshot can be easily attached to a new EC2 instance using the `Provision Host` github action or the `provision_host.sh` bash script directly. +1. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /var/lib/docker/volumes/or_postgresql-data/_data` #### Existing instance 1. Create a new EBS volume from the EBS snapshot 1. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command 1. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) 1. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` -2. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. +1. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data` +1. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. ```shell sudo docker run -d \ --rm \ From 54d7997de2be439fb3a6d3b1ed4b49d73bcc9d4a Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:15:48 +0200 Subject: [PATCH 4/9] add docs for AWS SSM --- docs/developer-guide/system-administration.md | 35 +++++++++++++------ 1 file changed, 25 insertions(+), 10 deletions(-) diff --git a/docs/developer-guide/system-administration.md b/docs/developer-guide/system-administration.md index f157ee87..b3591a7b 100644 --- a/docs/developer-guide/system-administration.md +++ b/docs/developer-guide/system-administration.md @@ -276,16 +276,31 @@ rm -r data/new data/old For data recovery or testing purposes it may be desirable to attach a DB snapshot to an existing instance. the following steps explain how to do this using an AWS EC2 EBS snapshot of the docker volumes. #### New instance -A snapshot can be easily attached to a new EC2 instance using the `Provision Host` github action or the `provision_host.sh` bash script directly. -1. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /var/lib/docker/volumes/or_postgresql-data/_data` +A volume based on a snapshot can be easily created and attached to a new EC2 instance using the `Provision Host` GitHub action or the `provision_host.sh` bash script directly. -#### Existing instance +#### Rollback an existing instance using AWS SSM +When using the SSM documents provided in `cloudformation-create-ssm-documents.yml` the rollback process can be completed automatically in just a few minutes. This reduces the number of manual steps required and is therefore less error-prone. +To start the rollback process, select the `replace_volume` SSM document from the AWS SSM console. The document requires only a few parameters to be filled in: + +- `SnapshotId` (The ID of the snapshot to restore from) +- `VolumeId` (The ID of the EBS volume to replace with the snapshot) +- `DeleteVolume` (Specifies whether the existing EBS volume should be deleted after replacement. Set to `true` to delete the volume or `false` to keep it.) + +When executing the automation, AWS will perform several actions under the hood. + +1. Retrieve the instance details for the specified `VolumeId` +2. Create a new EBS volume from the snapshot specified in `SnapshotId` +3. Detach the current EBS volume and attach the newly created volume. +4. Execute the `attach_volume` SSM automation. This automation runs several commands directly on the EC2 instance to mount the volume and update the `fstab` file. The `tag` attached to the EBS volume is also updated to ensure that AWS Data Lifecycle Manager (DLM) targets the volume when creating daily and weekly backups. +5. Based on the `DeleteVolume` parameter, the original EBS volume is either deleted or retained as backup. + +#### Manually rollback an existing instance 1. Create a new EBS volume from the EBS snapshot -1. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command -1. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) -1. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` -1. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data` -1. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. +2. Attach the new EBS volume to the existing EC2 instance using the AWS console or the `aws ec2 attach-volume` command +3. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) +4. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` +5. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data` +6. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. ```shell sudo docker run -d \ --rm \ @@ -296,8 +311,8 @@ A snapshot can be easily attached to a new EC2 instance using the `Provision Hos -v /tmp:/export \ openremote/postgresql:latest ``` -1. Wait for the DB to be ready; it can take a while when starting from a large snapshot, look at the docker container logs for DB ready message -1. Extract/Analyse the data as required either by: +7. Wait for the DB to be ready; it can take a while when starting from a large snapshot, look at the docker container logs for DB ready message +8. Extract/Analyse the data as required either by: * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote` or * Modify the manager and keycloak DB settings to connect to the temporary DB on port 5433 From 17aba22a39c09eac8d4f7efd7e31796ee0159116 Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:44:43 +0200 Subject: [PATCH 5/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/developer-guide/100-system-administration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-guide/100-system-administration.md b/docs/developer-guide/100-system-administration.md index 49d20838..9915900a 100644 --- a/docs/developer-guide/100-system-administration.md +++ b/docs/developer-guide/100-system-administration.md @@ -269,7 +269,7 @@ rm -r data/new data/old ``` ### Attaching DB snapshot to an EC2 instance -For data recovery or testing purposes it may be desirable to attach a DB snapshot to an existing instance. the following steps explain how to do this using an AWS EC2 EBS snapshot of the docker volumes. +For data recovery or testing purposes, it may be desirable to attach a DB snapshot to an existing instance. The following steps explain how to do this using an AWS EC2 EBS snapshot of the Docker volumes. #### New instance A volume based on a snapshot can be easily created and attached to a new EC2 instance using the `Provision Host` GitHub action or the `provision_host.sh` bash script directly. From bff2827f3dc346a9e226e75ef2349f9ec4ef43a8 Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:45:13 +0200 Subject: [PATCH 6/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/developer-guide/100-system-administration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-guide/100-system-administration.md b/docs/developer-guide/100-system-administration.md index 9915900a..d2e36992 100644 --- a/docs/developer-guide/100-system-administration.md +++ b/docs/developer-guide/100-system-administration.md @@ -307,7 +307,7 @@ When executing the automation, AWS will perform several actions under the hood. -v /tmp:/export \ openremote/postgresql:latest ``` -7. Wait for the DB to be ready; it can take a while when starting from a large snapshot, look at the docker container logs for DB ready message +7. Wait for the DB to be ready. When starting from a large snapshot, this can take a while; check the Docker container logs for the DB-ready message. 8. Extract/Analyse the data as required either by: * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote` or From e9393bc401619c1bc62f458ca30dcb3b1bcfa6ef Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:45:55 +0200 Subject: [PATCH 7/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/developer-guide/100-system-administration.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/developer-guide/100-system-administration.md b/docs/developer-guide/100-system-administration.md index d2e36992..eacd79ec 100644 --- a/docs/developer-guide/100-system-administration.md +++ b/docs/developer-guide/100-system-administration.md @@ -310,8 +310,7 @@ When executing the automation, AWS will perform several actions under the hood. 7. Wait for the DB to be ready. When starting from a large snapshot, this can take a while; check the Docker container logs for the DB-ready message. 8. Extract/Analyse the data as required either by: * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote` - or - * Modify the manager and keycloak DB settings to connect to the temporary DB on port 5433 + * Modifying the Manager and Keycloak DB settings to connect to the temporary DB on port 5433 ### Useful resources - [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB) From 7328f2071c085d2015583deae7d851b8e118026a Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:47:17 +0200 Subject: [PATCH 8/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/developer-guide/100-system-administration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/developer-guide/100-system-administration.md b/docs/developer-guide/100-system-administration.md index eacd79ec..8962ba6f 100644 --- a/docs/developer-guide/100-system-administration.md +++ b/docs/developer-guide/100-system-administration.md @@ -296,7 +296,7 @@ When executing the automation, AWS will perform several actions under the hood. 3. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) 4. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` 5. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data` -6. Start a temporary PostgreSQL instance to connect to the snapshot DB:Locate the PostgreSQL data volume in `ls /mnt/snapshot` generally called `or_postgresql-data and set the `PGDATA` environment variable to the mount point. +6. Start a temporary PostgreSQL instance to connect to the snapshot DB: locate the PostgreSQL data volume shown by `ls /mnt/snapshot` (generally called `or_postgresql-data`) and use that name in the `-v /mnt/snapshot//_data:/var/lib/postgresql/data` mount below. ```shell sudo docker run -d \ --rm \ From 260d5657954f78cf125ef0330172d31a7ceddb83 Mon Sep 17 00:00:00 2001 From: Dennis Kuijs Date: Fri, 10 Jul 2026 15:50:59 +0200 Subject: [PATCH 9/9] wip --- docs/developer-guide/100-system-administration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/developer-guide/100-system-administration.md b/docs/developer-guide/100-system-administration.md index 8962ba6f..06e3f3fd 100644 --- a/docs/developer-guide/100-system-administration.md +++ b/docs/developer-guide/100-system-administration.md @@ -296,7 +296,7 @@ When executing the automation, AWS will perform several actions under the hood. 3. Check the EBS volume is attached as a disk using `lsblk` (assuming it is attached as `/dev/nvme2n1` for remaining steps) 4. Mount the EBS volume `sudo mount -t xfs -o nouuid /dev/nvme2n1 /mnt/snapshot` 5. Change PostgreSQL data volume permissions `sudo chown -R 70:70 /mnt/snapshot/or_postgresql-data/_data` -6. Start a temporary PostgreSQL instance to connect to the snapshot DB: locate the PostgreSQL data volume shown by `ls /mnt/snapshot` (generally called `or_postgresql-data`) and use that name in the `-v /mnt/snapshot//_data:/var/lib/postgresql/data` mount below. +6. Start a temporary PostgreSQL instance to connect to the snapshot DB. Locate the PostgreSQL data volume shown by `ls /mnt/snapshot` (generally called `or_postgresql-data`) and use that name in the `-v /mnt/snapshot//_data:/var/lib/postgresql/data` mount below. ```shell sudo docker run -d \ --rm \ @@ -310,7 +310,7 @@ When executing the automation, AWS will perform several actions under the hood. 7. Wait for the DB to be ready. When starting from a large snapshot, this can take a while; check the Docker container logs for the DB-ready message. 8. Extract/Analyse the data as required either by: * Exec'ing into the container `docker exec -it temp_recovery_db psql -U postgres -d openremote` - * Modifying the Manager and Keycloak DB settings to connect to the temporary DB on port 5433 + * Modifying the Manager and Keycloak DB settings to connect to the temporary DB on port 5433. ### Useful resources - [Shared memory](https://www.instaclustr.com/blog/postgresql-docker-and-shared-memory/#:~:text=Docker%20and%20SHM%2DSize&text=This%20means%20that%20instead%20of,default%2C%20this%20limit%20is%2064MB)