Post

ENG | Backup and disaster recovery plan

Guide to backup and disaster recovery for self-hosted services like Gitea and Nextcloud. Examples of data restoration and service recovery using Podman containers.

Posted Updated
By Pavel Peřina
12 min read
ENG | Backup and disaster recovery plan

Motivation: The Harsh Reality

Creating images using DALL·E on serious topic could be fun.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

# smartctl -x /dev/sda

smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-23-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     Verbatim Vi550 S3
User Capacity:    256,060,514,304 bytes [256 GB]
Rotation Rate:    Solid State Device
Form Factor:      2.5 inches
Local Time is:    Sun May 28 13:37:27 2023 EDT

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
No failed Attributes found.

SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     -O--CK   100   100   050    -    0
  5 Reallocated_Sector_Ct   -O--CK   100   100   050    -    0
  9 Power_On_Hours          -O--CK   100   100   050    -    1861
 12 Power_Cycle_Count       -O--CK   100   100   050    -    30

Yep. SSD dead and gone after 80 days. It reads corrupted data, disconnects intermittently, and rejects any attempts to write new data – even new SMART values or self-test results. Hopefully this harddrive contained Linux hosting this server and data were moved to NVMe drive just a few weeks before failure, but was still used for some experiments.

Have you ever envisioned your meticulously crafted home server succumbing to disaster, leaving your precious data and services inaccessible? It’s a chilling thought, but one worth contemplating in the pursuit of digital resilience. This article serves as your practical guide to navigating the storm by focusing on recovering Podman containers and rebuilding your server foundation.

While I’ll delve into the technical steps with specific examples, it’s crucial to acknowledge that data backup and recovery extend far beyond mere technicalities. This journey serves as a gentle nudge to consider the vast spectrum of data you value, from irreplaceable documents to cherished photo memories. Remember, recovery success hinges not only on knowing how to restore, but also on prioritizing what to safeguard.

However, I’d like to note that topic of data backup and recovery is broader. It nudges readers to consider the importance of prioritizing their backup strategy, noting that while critical documents may take up minimal space, the volume of photos can be substantially larger. This mention serves as a reminder that, alongside specific restoration techniques, understanding what data to safeguard is crucial in any comprehensive data protection plan.

MTBF (Mean Time Between Failures) is discussed in the article Advanced Use of the Logarithmic Slide Rule: Solved Word Problems - one million hours sounds great, but is it?

Setting Realistic Expectations

This article assumes you’ve already implemented a data backup strategy outlined in previous resources (like file collection). While comprehensive backups are ideal, let’s face it, they are time consuming.

Here’s the current state of our backups:

  • Backup of docker compose files in ~/docker directory
  • Docker (actually podman) volumes in ~/.local/share/container/storage/volumes directory
  • Jekyll source files in ~/dev-blog directory.
  • Bin directory scripts (recommended to back up)

Backups in git repository (Gitea) do not count, because they are not useful without running server and it’s not self-sufficient. Having this post how to restore Gitea server inside it would be quite unfortunate.

Server configuration is backed up in the form of maybe 10 blog posts from May to July 2023.

As of 2023-07-04 I have these services:

  • Nextcloud & DB
  • Gitea & DB
  • Nginx (Jekyll blog)
  • Cloudflare DDNS (no volumes)
  • Cloudflare Tunnels (no volumes)
  • .. and a few scripts

We’ll prioritize reconstructing essential services like Gitea, Nextcloud and Jekyll blog while acknowledging the limitations of incomplete server configuration backups. Cloudflare DDNS and Tunnels, lacking podman volumes, require minimal recovery efforts. Throughout the process, we’ll emphasize the importance of adaptability and resourcefulness as you navigate the unique challenges of your specific disaster scenario.

Remember, this is a learning experience, and even partial recovery is a victory. So, buckle up, and let’s embark on this journey together towards a more resilient digital future!

Restoring directories

First things to restore are ~/docker and ~/dev-blog (tarballs of directories). From there we can move to restoration of podman volumes. Note that ~/docker/.env has certain settings specific for machine, at this point Cloudflare tunnel token, but there can be more in the future.

Restoring docker podman volumes.

Technically, we don’t need to restore Jekyll blog. Running deployment script is enough. Cloudflare does not have any other data than tokens in ~/docker/.env file.

We are left with six volumes containing NextCloud/Gitea data, configs and databases. Question is, if we want to backup and restore databases as SQL dump or as a volume. Gitea is better candidate for experiments, as it’s significantly smaller.

We somehow need to create volumes first and then run podman volume import.

Let’s try something …

Restoring Gitea (on virtual machine)

Create backup files for experiment

  • Backup backups to a single tar file and copy it to web server root directory.
1
2
3
4
5
6
7
8
9
10
11

[pavel@marten -=- /home/pavel]$ tar cfv gitea.tar backup/gitea-* docker/.env docker/docker-compose-gitea.yml
backup/gitea-config-2023-07-04.tar.zst
backup/gitea-data-2023-07-04.tar.zst
backup/gitea-db-2023-07-04.tar.zst
docker/.env
docker/docker-compose-gitea.yml

[pavel@marten -=- /home/pavel]$ ls -la gitea.tar 
-rw-r--r--. 1 pavel pavel 6174720 Jul  4 11:07 gitea.tar

[pavel@marten -=- /home/pavel]$ podman cp gitea.tar nginx:/config/www

Download backup

1
2
3
4
5
6
7
8
9
10
11

pavel@debian-vbox:~$ wget https://www.pavelp.cz/gitea.tar
--2023-07-04 11:14:05--  https://www.pavelp.cz/gitea.tar
Resolving www.pavelp.cz (www.pavelp.cz)... 172.67.185.80, 104.21.32.91, 2606:4700:3035::6815:205b, ...
Connecting to www.pavelp.cz (www.pavelp.cz)|172.67.185.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6174720 (5.9M) [application/octet-stream]
Saving to: ‘gitea.tar’

gitea.tar                     100%[=================================================>]   5.89M   449KB/s    in 16s

2023-07-04 11:14:22 (373 KB/s) - ‘gitea.tar’ saved [6174720/6174720]

Restore backup files

This will overwrite existing files, such as ~/docker/.env

1
2
3
4
5
6

pavel@debian-vbox:~$ tar xfv gitea.tar
backup/gitea-config-2023-07-04.tar.zst
backup/gitea-data-2023-07-04.tar.zst
backup/gitea-db-2023-07-04.tar.zst
docker/.env
docker/docker-compose-gitea.yml

Prepare podman containers and volumes

Download images (can take some time) and create volumes. Do not start containers yet.

Note: there was Nextcloud already running from previous experiments, so MariaDB database is not downloaded. Also I lost .env file with cloudflare token specific to Debian.

Because I do’nt have .env file(s) specified in docker-compose files (which I should), always run podman-compose from docker directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

pavel@debian-vbox:~/docker$ podman-compose -f ~/docker/docker-compose-gitea.yml up --no-start
['podman', '--version', '']
using podman version: 4.3.1
** excluding:  set()
podman volume inspect docker_gitea-db || podman volume create docker_gitea-db
['podman', 'volume', 'inspect', 'docker_gitea-db']
Error: inspecting object: no such volume docker_gitea-db
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=docker', '--label', 'com.docker.compose.project=docker', 'docker_gitea-db']
['podman', 'volume', 'inspect', 'docker_gitea-db']
['podman', 'network', 'exists', 'docker_default']
podman create --name=gitea-mariadb --label io.podman.compose.config-hash=123 --label io.podman.compose.project=docker --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=docker --label com.docker.compose.project.working_dir=/home/pavel/docker --label com.docker.compose.project.config_files=/home/pavel/docker/docker-compose-gitea.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=gitea-db -e PUID=1000 -e TZ=Europe/Prague -e MYSQL_DATABASE=gitea -e MYSQL_USER=gitea -e MYSQL_PASSWORD=gitea -e MYSQL_ROOT_PASSWORD=gitea -v docker_gitea-db:/config --net docker_default --network-alias gitea-db --restart unless-stopped lscr.io/linuxserver/mariadb:latest
d2c691e82688ee2976d464e574ba774af88babd2b4901ee0b47bc95dc961d522
exit code: 0
podman volume inspect docker_gitea-data || podman volume create docker_gitea-data
['podman', 'volume', 'inspect', 'docker_gitea-data']
Error: inspecting object: no such volume docker_gitea-data
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=docker', '--label', 'com.docker.compose.project=docker', 'docker_gitea-data']
['podman', 'volume', 'inspect', 'docker_gitea-data']
podman volume inspect docker_gitea-config || podman volume create docker_gitea-config
['podman', 'volume', 'inspect', 'docker_gitea-config']
Error: inspecting object: no such volume docker_gitea-config
['podman', 'volume', 'create', '--label', 'io.podman.compose.project=docker', '--label', 'com.docker.compose.project=docker', 'docker_gitea-config']
['podman', 'volume', 'inspect', 'docker_gitea-config']
['podman', 'network', 'exists', 'docker_default']
podman create --name=gitea --label io.podman.compose.config-hash=123 --label io.podman.compose.project=docker --label io.podman.compose.version=0.0.1 --label com.docker.compose.project=docker --label com.docker.compose.project.working_dir=/home/pavel/docker --label com.docker.compose.project.config_files=/home/pavel/docker/docker-compose-gitea.yml --label com.docker.compose.container-number=1 --label com.docker.compose.service=gitea-server -e GITEA__database__DB_TYPE=mysql -e GITEA__database__HOST=gitea-db:3306 -e GITEA__database__NAME=gitea -e GITEA__database__USER=gitea -e GITEA__database__PASSWD=gitea -v /usr/share/zoneinfo/Europe/Prague:/etc/localtime:ro -v docker_gitea-data:/var/lib/gitea -v docker_gitea-config:/etc/gitea --net docker_default --network-alias gitea-server -p 8082:3000 -p 2222:2222 --restart always docker.io/gitea/gitea:1.19-rootless
Trying to pull docker.io/gitea/gitea:1.19-rootless...
Getting image source signatures
Copying blob f56be85fc22e done
Copying blob 45a481aabd09 done
Copying blob 6c9554cfb042 done
Copying blob 156fbfd332ab done
Copying blob dc4fb7e2930a done
Copying blob 6b2185564ca4 done
Copying blob 846082bed60e done
Copying blob 16e931dd20bb done
Copying blob b92911e7328f done
Copying blob c4842721387a done
Copying blob 4d40cbbbd47e done
Copying blob 4f4fb700ef54 done
Copying config 7829ff7c21 done
Writing manifest to image destination
Storing signatures
65f273ab7c25e255fe4e244e85cf18e3b33d34404c33ba149d1d595a7c68e63d
exit code: 0

List volumes and check their sizes

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

pavel@debian-vbox:~/docker$ podman volume ls
DRIVER      VOLUME NAME
local       docker_gitea-config
local       docker_gitea-data
local       docker_gitea-db
local       docker_nextcloud-config
local       docker_nextcloud-data
local       docker_nextcloud-database

root@debian-vbox:/home/pavel/.local/share/containers/storage/volumes# du -sh *
8.0K    docker_gitea-config
8.0K    docker_gitea-data
8.0K    docker_gitea-db
287M    docker_nextcloud-config
3.5M    docker_nextcloud-data
207M    docker_nextcloud-database

Restore podman volumes from backup files

Experiment a bit 🙂

1
2

pavel@debian-vbox:~/backup$ zstd -d < gitea-config-2023-07-04.tar.zst  | tar xv
app.ini

Restore volumes. Uncompress backup to tar, send content to podman volume import via pipe (syntax is podman volume import <volume> <tarball> where - means read tarball from standart input/pipe )

1
2
3

pavel@debian-vbox:~/backup$ zstd -d < gitea-config-2023-07-04.tar.zst  | podman volume import docker_gitea-config -
pavel@debian-vbox:~/backup$ zstd -d < gitea-data-2023-07-04.tar.zst  | podman volume import docker_gitea-data -
pavel@debian-vbox:~/backup$ zstd -d < gitea-db-2023-07-04.tar.zst  | podman volume import docker_gitea-db -

Check it

1
2
3
4
5
6
7

root@debian-vbox:/home/pavel/.local/share/containers/storage/volumes# du -sh *
12K     docker_gitea-config
5.8M    docker_gitea-data
190M    docker_gitea-db
287M    docker_nextcloud-config
3.5M    docker_nextcloud-data
207M    docker_nextcloud-database

Start service(s)

1

pavel@debian-vbox:~/docker$ podman-compose -f docker-compose-gitea.yml up

Check service(s)

Now check connection. Get IP and port and try connection from a browser (note: it assumes that VirtualBox network is in bridge mode)

1
2
3
4
5
6
7

pavel@debian-vbox:~$ hostname -I
192.168.1.116

pavel@debian-vbox:~$ grep -A 2 ports ~/docker/docker-compose-gitea.yml
    ports:
      - "8082:3000"
      - "2222:2222"

Restoring Nextcloud (after update failure)

Introduction and reason

This section was added on September 1st 2023. My last backup was from July 14th. In end of July I tried to update Nextcloud, sadly without doing backup. It complained about wrong PHP version when I updated podman containers to latest and i tried to do upgrade from console somehow. I ended by incompatible versions of NextCloud, PHP and database which was always too new or too old when I reverted to older image.

At this point I moved nextcloud to separate compose file.

Remove Nextcloud

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

[pavel@marten -=- /home/pavel/docker]$ podman-compose -f docker-compose-nextcloud.yml down
⋮
[pavel@marten -=- /home/pavel/docker]$ podman volume ls
DRIVER      VOLUME NAME
⋮
local       docker_nextcloud-app-config
local       docker_nextcloud-app-data
local       docker_nextcloud-db
⋮
[pavel@marten -=- /home/pavel/docker]$ podman volume rm docker_nextcloud-app-data
docker_nextcloud-app-data
[pavel@marten -=- /home/pavel/docker]$ podman volume rm docker_nextcloud-app-config
docker_nextcloud-app-config
[pavel@marten -=- /home/pavel/docker]$ podman volume rm docker_nextcloud-db
docker_nextcloud-db
[pavel@marten -=- /home/pavel/docker]$ podman volume ls
DRIVER      VOLUME NAME
local       docker_gitea-config
local       docker_gitea-data
local       docker_gitea-db
local       docker_nginx-config

Initialize Nextcloud containers and volumes

1
2
3
4
5
6
7
8
9
10
11

[pavel@marten -=- /home/pavel/backup]$ podman-compose -f ~/docker/docker-compose-nextcloud.yml up --no-start
⋮
[pavel@marten -=- /home/pavel/backup]$ podman volume ls
DRIVER      VOLUME NAME
local       docker_gitea-config
local       docker_gitea-data
local       docker_gitea-db
local       docker_nextcloud-app-config
local       docker_nextcloud-app-data
local       docker_nextcloud-db
local       docker_nginx-config

Restore data

This take roughly one minute for 20GB

1
2
3

[pavel@marten -=- /home/pavel/backup]$ zstd -dc nextcloud-app-data-2023-07-14.tar.zst | podman volume import docker_nextcloud-app-data -
[pavel@marten -=- /home/pavel/backup]$ zstd -dc nextcloud-app-config-2023-07-14.tar.zst | podman volume import docker_nextcloud-app-config -

Restore database

First start nextcloud-mariadb container.

1
2
3
4
5
6
7
8
9
10
11

[pavel@marten -=- /home/pavel/backup]$ podman container ps -a
CONTAINER ID  IMAGE                                       COMMAND               CREATED         STATUS         PORTS                                           NAMES
f93e76ba9398  lscr.io/linuxserver/mariadb:latest                                5 weeks ago     Created                                                        gitea-mariadb
f1f5e84d828b  docker.io/gitea/gitea:1.20-rootless                               5 weeks ago     Created        0.0.0.0:2222->2222/tcp, 0.0.0.0:8082->3000/tcp  gitea
1eceb67372ce  docker.io/cloudflare/cloudflared:latest     tunnel --no-autou...  50 minutes ago  Up 50 minutes                                                  cloudflared
e69fe89d6662  docker.io/favonia/cloudflare-ddns:latest                          50 minutes ago  Up 50 minutes                                                  cloudflare-ddns
33609db7e05b  lscr.io/linuxserver/nginx:latest                                  50 minutes ago  Up 50 minutes  0.0.0.0:8080->80/tcp                            nginx
2e47e7f9211e  lscr.io/linuxserver/mariadb:latest                                2 minutes ago   Created                                                        nextcloud-mariadb
a55866c2b2d9  lscr.io/linuxserver/nextcloud:26.0.2-ls246                        2 minutes ago   Created        0.0.0.0:8081->80/tcp                            nextcloud-app
[pavel@marten -=- /home/pavel/backup]$ podman start nextcloud-mariadb
nextcloud-mariadb

Then unpack backup file and copy it to a container volume

1
2
3
4
5

[pavel@marten -=- /home/pavel/backup]$ zstd -d nextcloud-db-2023-07-14.sql.zst
nextcloud-db-2023-07-14.sql.zst: 8442414 bytes
[pavel@marten -=- /home/pavel/backup]$ ls -la *.sql
-rw-r--r--. 1 pavel pavel 8442414 Jul 14 08:16 nextcloud-db-2023-07-14.sql
[pavel@marten -=- /home/pavel/backup]$ podman cp nextcloud-db-2023-07-14.sql nextcloud-mariadb:/config

Proceed by import of SQL database.

Here I have no idea why volume is actually accessible from container, because i haven’t specified anything and neither started it by podman-compose. Maybe configuration is saved by podman-compose start. Username, password and database are manually copied from ~/docker/.env file.

1
2
3
4
5
6
7
8
9

[pavel@marten -=- /home/pavel/backup]$ podman exec -it nextcloud-mariadb /bin/bash
root@2e47e7f9211e:/# cd co
command/ config/
root@2e47e7f9211e:/# cd config/
root@2e47e7f9211e:/config# ls *sql
nextcloud-db-2023-07-14.sql
root@2e47e7f9211e:/config# mysql --user=nextcloud --password=C********u nextcloud < nextcloud-db-2023-07-14.sql
root@2e47e7f9211e:/config/databases# exit
exit

Stop nextcloud-mariadb and …

Start Nextcloud

1
2
3
4
5
6
7
8
9

[pavel@marten -=- /home/pavel/backup]$ podman stop nextcloud-mariadb
nextcloud-mariadb
[pavel@marten -=- /home/pavel/backup]$ cd ../docker
[pavel@marten -=- /home/pavel/docker]$ podman-compose -f ~/docker/docker-compose-nextcloud.yml up -d
podman-compose version: 1.0.6
⋮
[pavel@marten -=- /home/pavel/docker]$ podman exec -it nextcloud-app /bin/bash
root@c757c325b0c5:/# occ maintenance:mode --off
Maintenance mode disabled

Update Nextcloud

This might be dependent on NextCloud version and used 3rd party Docker image.

Now log in as admin (my password is C*******u) go to Administrative Settings and proceed to updater Click on … and then click on Disable maintenance mode and continue in the web based updater. Proceed with Start upgrade

  • This updated nextcloud from 25.0.6 to 25.0.10
  • Repeat with 25.0.10 to 26.0.5
  • Repeat with 26.0.5 to 27.0.2

Wait for mobile app to synchronize

Do backup

Celebrate

Image with notebook and beer

Summary

This guide offers a roadmap for recovering from a server disaster and rebuilding containerized services like Gitea and Nextcloud. It walks you through key stages like assessing damage, restoring data, and rebuilding services, providing a practical approach informed by real-world experience.

While specific examples focus on Gitea, the core principles are adaptable to various Podman and service recovery scenarios, empowering you to navigate your unique situation with confidence. Remember, adapting the steps to your specific needs and disaster is crucial for a successful recovery.

However, this guide is meant to inspire, not replace, your own server recovery plan. Consider your unique setup and adapt the steps accordingly for a smooth and successful recovery.

Changelog

  • September 2023
    • Added NextCloud troubleshooting
  • February 2024
    • New intro and summary (Gemini AI test)
    • Link MTBF example
English
containers self hosting
This post is licensed under CC BY 4.0 by the author.