Browse Source

Added new post.

dev
Franco Masotti 3 years ago
parent
commit
d242402277
  1. 346
      _posts/2019-02-23-my-backup-system.md

346
_posts/2019-02-23-my-backup-system.md

@ -0,0 +1,346 @@
---
title: My backup system
tags: [bash, shell, backup, rsync, LUKS, cryptsetup, sync]
updated: 2019-02-23 17:24
description: A detailed exaplanation of my backup system which is both encrypted and unencrypted for different purposes.
---
## Introduction
[Rsync](https://rsync.samba.org/) is a very useful and flexible tool to do
incremental backups and I have been using it for years without any problems.
Recently I discovered the power of Rsync and [Cryptsetup with
LUKS](https://gitlab.com/cryptsetup/cryptsetup/) to do encrypted and
incremental backups as well.
### Disclaimer
*Please note that the scripts (and the steps) presented in this post are not
exactly the same I am using right now (and are not the same steps I did). I
intend, however, to use these new scripts in the near future. Run them in a
test environment first and in the meantime if you find bugs or other problems
just leave a comment below.*
## Partitioning scheme
I have chosen a partitioning scheme that separates *bigger and rarely used
files* (on HDDs) from *smaller and commonly used files* (on SSDs). For this
reason I need two different hard disks for backup:
| Disk name | Disk type | Size (order of magnitude) | Backup of mountpoint |
|-----------|-----------|---------------------------|----------------------|
| backup root [1,2] | HDD | Gigabyte | `/` |
| backup data [1,2] | HDD | Terabyte | `/data` |
## Two different types of backups
I also keep two different types of incremental backups. It's not exactly the *3 2 1 rule* but it's fine for my use case.
| Backup name | Online | Offsite | Automatic | Encrypted | Frequency | Reason |
|-------------|--------|---------|-----------|-----------|-----------|--------|
| backup | x | | x | | daily | to recover files deleted by mistake |
| backup enc | | x | | x | monthly (at best) | in case some disaster occurs |
## Totals
| Disk name | Backup name | Device |
|-----------|-------------|--------|
| backup root 1 | backup | `/dev/sdx` |
| backup data 1 | backup | `/dev/sdy` |
| backup root 2 | backup enc | `/dev/sdw` |
| backup data 2 | backup enc | `/dev/sdz` |
## Preliminary actions
### Notes
- A table of the notation used from this point on
| Mountpoint | Partition | Notes |
|------------|-----------|-------|
| `/mnt/backup_root` | `/dev/sdx1` | unencrypted root partition backup |
| `/mnt/backup_data` | `/dev/sdy1` | unencrypted data partition backup |
| `/mnt/backup_root_enc` | `/dev/sdw1` | encrypted root partition backup |
| `/mnt/backup_data_enc` | `/dev/sdz1` | encrypted data partition backup |
- [*Bash wildcards*](http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm)
will be used in this section.
### Steps
1. Install [Rsync](https://rsync.samba.org/),
[Cryptsetup](https://gitlab.com/cryptsetup/cryptsetup/),
[msmtp](https://marlam.de/msmtp/),
[S-nail](https://www.sdaoden.eu/code.html#s-nail),
[GNU Bash](http://www.gnu.org/software/bash/bash.html) and
[GNU awk](http://www.gnu.org/software/gawk/)
2. Make the mountpoints and change the permissions. Make them accessible only
to the `root` user and group, just to be on the safe side. The first thing
to do is to unmount everything under `/mnt`
# umount -R /mnt
# pushd /mnt
# mkdir -p backup_{root,data}{_enc,}
# chown 700 backup_{root,data}{_enc,}
# popd
3. Partition, format and mount the unencrypted hard disks. You should use
fstab instead of mounting the filesystems manually
# fdisk /dev/sd{x,y}
# mkfs.ext4 /dev/sd{x,y}1
# mount /dev/sd{x,y}1 /mnt/backup_{root,data}
4. Partition and format the encrypted backups as written in the [Arch Wiki page](https://wiki.archlinux.org/index.php?title=Dm-crypt/Device_encryption&oldid=557648#Encryption_options_with_dm-crypt).
We will use some non-default options to make the encryption stronger
# cryptsetup -v --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-urandom --verify-passphrase luksFormat device /dev/sd{w,z}1
5. We will now
[mount and create a filesystem on the encrypted partitions](https://wiki.archlinux.org/index.php?title=Dm-crypt/Encrypting_a_non-root_file_system&oldid=495610#Manual_mounting_and_unmounting).
You will be prompted for a password which will then be used every time you
call cryptsetup
# cryptsetup open /dev/sd{w,z}1 {root,data}_enc
# mkfs.ext4 /dev/mapper/{root,data}_enc
# mount /dev/mapper/root_enc /mnt/backup_{root,data}_enc
6. Unmount
# umount /mnt/backup_{root,data}_enc
# cryptsetup close /dev/mapper/{root,data}_enc
7. [Backup the LUKS headers](https://wiki.archlinux.org/index.php?title=Dm-crypt/Device_encryption&oldid=557648#Backup_using_cryptsetup)
of the encrypted drives in case they need to be recovered in the future.
# pushd "${a_safe_place_for_backups}"
# cryptsetup luksHeaderBackup /dev/sd{w,z}1 --header-backup-file backup_{root,data}_enc_luks_header.img
# popd
## The scripts
### Incremental backup script
This script, called `backup.sh`, was inspired by an
[Arch Wiki](https://wiki.archlinux.org/index.php?title=Rsync&oldid=566635#Full_system_backup)
page dating back some years ago, so I have to use the same license. The
following code is licensed under the [GNU Free Documentation License
1.3](https://www.gnu.org/copyleft/fdl.html) or later.
```shell
#!/bin/bash
#
# backup.sh
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
declare -A BACKUP_ALIAS
. ./backup.conf
SRC="${1}"
DST="${2}"
{ command -V rsync && command -V bash && command -V mail \
&& command -V printf && command -V sync; } \
|| exit 1
{ [ -z "${SRC}" ] || [ -z "${DST}" ]; } \
&& printf "%s\n" 'check inputs' 2>&1 && exit 1
[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
if [ "${SRC}" = '/*' ]; then
# Use the exclude option to avoid infinite loops due to symlinks and to
# avoid backing up the /data mountpoint.
rsync \
--archive --acls --xattrs --hard-link --delete \
/* "${DST}" \
--exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/data}
elif [ "${SRC}" = '/data/*' ]; then
rsync \
--archive --acls --xattrs --hard-link --delete \
/data/* "${DST}"
else
rsync \
--archive --acls --xattrs --hard-link --delete \
"${SRC}" "${DST}"
fi
ret_rsync=${?}
# Flush data.
sync
if [ ${ret_rsync} -eq 0 ]; then
FINISH=$(date +%s)
if [ "${LOG_TO_FILE}" = 'true' ]; then
printf "%s\n" "[$(date '+%Y-%m-%d, %T') OK backup_ext_disks] "${DST}" ; \
$((${FINISH}-${START}))s" >> "${LOG_FILE}"
fi
if [ "${LOG_TO_EMAIL}" = 'true' ]; then
printf "%s\n" "Backup on "${BACKUP_ALIAS["${DST}"]}" took $((${FINISH}-${START}))s" \
| mail -r "${EMAIL_SENDER}" -s 'Backup complete' "${EMAIL_RECIPIENT}"
fi
else
if [ "${LOG_TO_FILE}" = 'true' ]; then
printf "%s\n" "[$(date '+%Y-%m-%d, %T') FAILED backup_ext_disks\
ret_rsync=${ret_rsync}] "${DST}"" \
>> /var/log/rsync.log
fi
if [ "${LOG_TO_EMAIL}" = 'true' ]; then
printf "%s\n" "Backup error on "${BACKUP_ALIAS["${DST}"]}". Error code ${ret_rsync}" \
| mail -r "${EMAIL_SENDER}" -s 'Backup error' "${EMAIL_RECIPIENT}"
fi
fi
```
#### Configuration file
```shell
#
# backup.conf
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
# Friendlier names used in logs and emails.
BACKUP_ALIAS['/mnt/backup_root']='backup-root'
BACKUP_ALIAS['/mnt/backup_root_enc']='backup-root-encrypted'
BACKUP_ALIAS['/mnt/backup_data']='backup-data'
BACKUP_ALIAS['/mnt/backup_data_enc']='backup-data-encrypted'
LOG_TO_EMAIL='true'
EMAIL_SENDER='Computer <your.email@address.com>'
# You can use aliases if your mailing system is set up correctly.
EMAIL_RECIPIENT='all'
LOG_TO_FILE='true'
LOG_FILE='/var/log/rsync.log'
```
### Encrypted backup script
To do our encrypted backups we will use LUKS and the previous script as a
base. The idea is quite simple:
1. mount the encrypted partition.
2. do the backup using the previous script with the appropriate parameters.
3. unmount the partition.
This new script is called `backup_enc.sh` and was inspired by
[this Arch Wiki](https://wiki.archlinux.org/index.php?title=Dm-crypt/Encrypting_a_non-root_file_system&oldid=495610#Manual_mounting_and_unmounting)
page.
```shell
#!/bin/bash
#
# backup_enc.sh
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
#
# This backup is intended to be run manually.
#
declare -A BACKUP_ALIAS_ROOT
declare -A BACKUP_ALIAS_DATA
. ./backup_enc.conf
WHAT="${1}"
{ command -V cryptsetup && command -V bash && command -V printf \
command -V mount && command -V umount && command -V awk; } \
|| exit 1
[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
if [ "${WHAT}" = 'root' ]; then
SRC=BACKUP_ALIAS_ROOT['SRC']
DST=BACKUP_ALIAS_ROOT['DST']
UUID=BACKUP_ALIAS_ROOT['UUID']
NAME=BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']
elif [ "${WHAT}" = 'data' ]; then
SRC=BACKUP_ALIAS_DATA['SRC']
DST=BACKUP_ALIAS_DATA['DST']
UUID=BACKUP_ALIAS_DATA['UUID']
NAME=BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']
else
printf "%s\n" 'error: configuration not set' 2>&1 && exit 1
fi
# Find partition via UUID.
dev="/dev/"$(lsblk -x name -i -o name,uuid | grep "${UUID}" | awk '{print $1}')""
[ -z "${dev}" ] && exit 1
cryptsetup open "${dev}" "${NAME}"
mount /dev/mapper/"${NAME}" "${DST}"
./backup.sh "${SRC}" "${DST}"
umount "${DST}"
cryptsetup close "${NAME}"
```
#### Configuration file
You must put the correct UUIDs of the partitions in the configuration file. To
get these run:
$ lsblk -o name,uuid
and copy the appropriate ones.
```shell
#
# backup_enc.conf
#
# Copyright (C) 2019 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
BACKUP_ALIAS_ROOT['SRC']='/*'
BACKUP_ALIAS_ROOT['DST']='/mnt/backup_root_enc'
BACKUP_ALIAS_ROOT['UUID']='<put the UUID here>'
BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']='root_enc'
BACKUP_ALIAS_DATA['SRC']='/data/*'
BACKUP_ALIAS_DATA['DST']='/mnt/backup_data_enc'
BACKUP_ALIAS_DATA['UUID']='<put the UUID here>'
BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']='data_enc'
```
## First backups
Once you have everything in place, including the configuration files,
you may start backups manually along with Rsync's `--dry-run` option in the
script.
# ./backup.sh '/*' /mnt/backup_root
# ./backup.sh '/data/*' /mnt/backup_data
If everything works as expected, remove that option.
You can use Cron for the automatic backups. In my case I use
[Cronie](https://github.com/cronie-crond/cronie/). Just remember to
call the backup scripts from their directory otherwise the configuration files
won't be sourced.
~
Till next time!
Loading…
Cancel
Save