Added new post.

3 years ago · d242402277
1 changed files with 346 additions and 0 deletions
--- a/_posts/2019-02-23-my-backup-system.md
+++ b/_posts/2019-02-23-my-backup-system.md
@ -0,0 +1,346 @@
+---
+title: My backup system
+tags: [bash, shell, backup, rsync, LUKS, cryptsetup, sync]
+updated: 2019-02-23 17:24
+description: A detailed exaplanation of my backup system which is both encrypted and unencrypted for different purposes.
+---
+
+## Introduction
+
+[Rsync](https://rsync.samba.org/) is a very useful and flexible tool to do 
+incremental backups and I have been using it for years without any problems.
+Recently I discovered the power of Rsync and [Cryptsetup with 
+LUKS](https://gitlab.com/cryptsetup/cryptsetup/) to do encrypted and 
+incremental backups as well.
+
+### Disclaimer
+
+*Please note that the scripts (and the steps) presented in this post are not 
+exactly the same I am using right now (and are not the same steps I did). I 
+intend, however, to use these new scripts in the near future. Run them in a 
+test environment first and in the meantime if you find bugs or other problems 
+just leave a comment below.*
+
+## Partitioning scheme
+
+I have chosen a partitioning scheme that separates *bigger and rarely used 
+files* (on HDDs) from *smaller and commonly used files* (on SSDs). For this 
+reason I need two different hard disks for backup:
+
+| Disk name | Disk type | Size (order of magnitude) | Backup of mountpoint |
+|-----------|-----------|---------------------------|----------------------|
+| backup root [1,2] | HDD | Gigabyte | `/` |
+| backup data [1,2] | HDD | Terabyte | `/data` |
+
+## Two different types of backups
+
+I also keep two different types of incremental backups. It's not exactly the *3 2 1 rule* but it's fine for my use case.
+
+| Backup name | Online | Offsite | Automatic | Encrypted | Frequency | Reason |
+|-------------|--------|---------|-----------|-----------|-----------|--------|
+| backup | x | | x | | daily |  to recover files deleted by mistake |
+| backup enc | | x | | x | monthly (at best) | in case some disaster occurs |
+
+## Totals
+
+| Disk name | Backup name | Device |
+|-----------|-------------|--------|
+| backup root 1 | backup | `/dev/sdx` |
+| backup data 1 | backup | `/dev/sdy` |
+| backup root 2 | backup enc | `/dev/sdw` |
+| backup data 2 | backup enc | `/dev/sdz` |
+
+## Preliminary actions
+
+### Notes
+
+- A table of the notation used from this point on
+
+  | Mountpoint | Partition | Notes |
+  |------------|-----------|-------|
+  | `/mnt/backup_root` | `/dev/sdx1` | unencrypted root partition backup |
+  | `/mnt/backup_data` | `/dev/sdy1` | unencrypted data partition backup |
+  | `/mnt/backup_root_enc` | `/dev/sdw1` | encrypted root partition backup |
+  | `/mnt/backup_data_enc` | `/dev/sdz1` | encrypted data partition backup |
+
+- [*Bash wildcards*](http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/x11655.htm) 
+  will be used in this section.
+
+### Steps
+
+1. Install [Rsync](https://rsync.samba.org/), 
+[Cryptsetup](https://gitlab.com/cryptsetup/cryptsetup/), 
+[msmtp](https://marlam.de/msmtp/),
+[S-nail](https://www.sdaoden.eu/code.html#s-nail),
+[GNU Bash](http://www.gnu.org/software/bash/bash.html) and
+[GNU awk](http://www.gnu.org/software/gawk/)
+
+2. Make the mountpoints and change the permissions. Make them accessible only 
+   to the `root` user and group, just to be on the safe side. The first thing
+   to do is to unmount everything under `/mnt`
+
+       # umount -R /mnt
+       # pushd /mnt
+       # mkdir -p backup_{root,data}{_enc,}
+       # chown 700 backup_{root,data}{_enc,}
+       # popd
+
+3. Partition, format and mount the unencrypted hard disks. You should use 
+   fstab instead of mounting the filesystems manually
+
+       # fdisk /dev/sd{x,y}
+       # mkfs.ext4 /dev/sd{x,y}1
+       # mount /dev/sd{x,y}1 /mnt/backup_{root,data}
+
+4. Partition and format the encrypted backups as written in the [Arch Wiki page](https://wiki.archlinux.org/index.php?title=Dm-crypt/Device_encryption&oldid=557648#Encryption_options_with_dm-crypt).
+   We will use some non-default options to make the encryption stronger
+
+       # cryptsetup -v --type luks2 --cipher aes-xts-plain64 --key-size 512 --hash sha512 --iter-time 5000 --use-urandom --verify-passphrase luksFormat device /dev/sd{w,z}1
+
+5. We will now
+   [mount and create a filesystem on the encrypted partitions](https://wiki.archlinux.org/index.php?title=Dm-crypt/Encrypting_a_non-root_file_system&oldid=495610#Manual_mounting_and_unmounting). 
+   You will be prompted for a password which will then be used every time you 
+   call cryptsetup
+
+       # cryptsetup open /dev/sd{w,z}1 {root,data}_enc
+       # mkfs.ext4 /dev/mapper/{root,data}_enc
+       # mount /dev/mapper/root_enc /mnt/backup_{root,data}_enc
+
+6. Unmount
+
+       # umount /mnt/backup_{root,data}_enc
+       # cryptsetup close /dev/mapper/{root,data}_enc
+
+7. [Backup the LUKS headers](https://wiki.archlinux.org/index.php?title=Dm-crypt/Device_encryption&oldid=557648#Backup_using_cryptsetup)
+   of the encrypted drives in case they need to be recovered in the future.
+
+       # pushd "${a_safe_place_for_backups}"
+       # cryptsetup luksHeaderBackup /dev/sd{w,z}1 --header-backup-file backup_{root,data}_enc_luks_header.img
+       # popd
+
+## The scripts
+
+### Incremental backup script
+
+This script, called `backup.sh`, was inspired by an 
+[Arch Wiki](https://wiki.archlinux.org/index.php?title=Rsync&oldid=566635#Full_system_backup) 
+page dating back some years ago, so I have to use the same license. The 
+following code is licensed under the [GNU Free Documentation License 
+1.3](https://www.gnu.org/copyleft/fdl.html) or later.
+
+```shell
+#!/bin/bash
+#
+# backup.sh
+#
+# Copyright (C)  2019  Franco Masotti <franco.masotti@live.com>.
+# Permission is granted to copy, distribute and/or modify this document
+# under the terms of the GNU Free Documentation License, Version 1.3
+# or any later version published by the Free Software Foundation;
+# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+# A copy of the license is included in the section entitled "GNU
+# Free Documentation License".
+
+declare -A BACKUP_ALIAS
+. ./backup.conf
+
+SRC="${1}"
+DST="${2}"
+
+{ command -V rsync && command -V bash && command -V mail \
+    && command -V printf && command -V sync; } \
+    || exit 1
+{ [ -z "${SRC}" ] || [ -z "${DST}" ]; } \
+    && printf "%s\n" 'check inputs' 2>&1 && exit 1
+[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
+
+if [ "${SRC}" = '/*' ]; then
+    # Use the exclude option to avoid infinite loops due to symlinks and to 
+    # avoid backing up the /data mountpoint.
+     rsync \
+         --archive --acls --xattrs --hard-link --delete \
+         /* "${DST}" \
+        --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found,/data}
+elif [ "${SRC}" = '/data/*' ]; then
+    rsync \ 
+        --archive --acls --xattrs --hard-link --delete \
+        /data/* "${DST}"
+else
+    rsync \ 
+        --archive --acls --xattrs --hard-link --delete \
+        "${SRC}" "${DST}"
+fi
+
+ret_rsync=${?}
+# Flush data.
+sync
+
+if [ ${ret_rsync} -eq 0 ]; then
+    FINISH=$(date +%s)
+    if [ "${LOG_TO_FILE}" = 'true' ]; then
+        printf "%s\n" "[$(date '+%Y-%m-%d, %T') OK backup_ext_disks] "${DST}" ; \
+$((${FINISH}-${START}))s" >> "${LOG_FILE}"
+    fi
+    if [ "${LOG_TO_EMAIL}" = 'true' ]; then
+        printf "%s\n" "Backup on "${BACKUP_ALIAS["${DST}"]}" took $((${FINISH}-${START}))s" \
+            | mail -r "${EMAIL_SENDER}" -s 'Backup complete' "${EMAIL_RECIPIENT}"
+    fi
+else
+    if [ "${LOG_TO_FILE}" = 'true' ]; then
+        printf "%s\n" "[$(date '+%Y-%m-%d, %T') FAILED backup_ext_disks\
+ret_rsync=${ret_rsync}] "${DST}"" \
+        >> /var/log/rsync.log
+    fi
+    if [ "${LOG_TO_EMAIL}" = 'true' ]; then
+        printf "%s\n" "Backup error on "${BACKUP_ALIAS["${DST}"]}". Error code ${ret_rsync}" \
+            | mail -r "${EMAIL_SENDER}" -s 'Backup error' "${EMAIL_RECIPIENT}"
+    fi
+fi
+```
+
+#### Configuration file
+
+```shell
+#
+# backup.conf
+#
+# Copyright (C)  2019  Franco Masotti <franco.masotti@live.com>.
+# Permission is granted to copy, distribute and/or modify this document
+# under the terms of the GNU Free Documentation License, Version 1.3
+# or any later version published by the Free Software Foundation;
+# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+# A copy of the license is included in the section entitled "GNU
+# Free Documentation License".
+
+# Friendlier names used in logs and emails.
+BACKUP_ALIAS['/mnt/backup_root']='backup-root'
+BACKUP_ALIAS['/mnt/backup_root_enc']='backup-root-encrypted'
+BACKUP_ALIAS['/mnt/backup_data']='backup-data'
+BACKUP_ALIAS['/mnt/backup_data_enc']='backup-data-encrypted'
+
+LOG_TO_EMAIL='true'
+EMAIL_SENDER='Computer <your.email@address.com>'
+# You can use aliases if your mailing system is set up correctly.
+EMAIL_RECIPIENT='all'
+
+LOG_TO_FILE='true'
+LOG_FILE='/var/log/rsync.log'
+```
+
+### Encrypted backup script
+
+To do our encrypted backups we will use LUKS and the previous script as a 
+base. The idea is quite simple:
+1. mount the encrypted partition.
+2. do the backup using the previous script with the appropriate parameters.
+3. unmount the partition.
+
+This new script is called `backup_enc.sh` and was inspired by 
+[this Arch Wiki](https://wiki.archlinux.org/index.php?title=Dm-crypt/Encrypting_a_non-root_file_system&oldid=495610#Manual_mounting_and_unmounting)
+page.
+
+```shell
+#!/bin/bash
+#
+# backup_enc.sh
+#
+# Copyright (C)  2019  Franco Masotti <franco.masotti@live.com>.
+# Permission is granted to copy, distribute and/or modify this document
+# under the terms of the GNU Free Documentation License, Version 1.3
+# or any later version published by the Free Software Foundation;
+# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+# A copy of the license is included in the section entitled "GNU
+# Free Documentation License".
+
+#
+# This backup is intended to be run manually.
+#
+
+declare -A BACKUP_ALIAS_ROOT
+declare -A BACKUP_ALIAS_DATA
+
+. ./backup_enc.conf
+
+WHAT="${1}"
+
+{ command -V cryptsetup && command -V bash && command -V printf \
+    command -V mount && command -V umount && command -V awk; } \
+    || exit 1
+[ ${UID} -ne 0 ] && printf "%s\n" 'user must be root' 2>&1 && exit 1
+
+if [ "${WHAT}" = 'root' ]; then
+    SRC=BACKUP_ALIAS_ROOT['SRC']
+    DST=BACKUP_ALIAS_ROOT['DST']
+    UUID=BACKUP_ALIAS_ROOT['UUID']
+    NAME=BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']
+elif [ "${WHAT}" = 'data' ]; then
+    SRC=BACKUP_ALIAS_DATA['SRC']
+    DST=BACKUP_ALIAS_DATA['DST']
+    UUID=BACKUP_ALIAS_DATA['UUID']
+    NAME=BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']
+else
+    printf "%s\n" 'error: configuration not set' 2>&1 && exit 1
+fi
+
+# Find partition via UUID.
+dev="/dev/"$(lsblk -x name -i -o name,uuid | grep "${UUID}" | awk '{print $1}')""
+[ -z "${dev}" ] && exit 1
+
+cryptsetup open "${dev}" "${NAME}"
+mount /dev/mapper/"${NAME}" "${DST}"
+./backup.sh "${SRC}" "${DST}"
+umount "${DST}"
+cryptsetup close "${NAME}"
+```
+
+#### Configuration file
+
+You must put the correct UUIDs of the partitions in the configuration file. To 
+get these run:
+
+    $ lsblk -o name,uuid
+
+and copy the appropriate ones.
+
+```shell
+#
+# backup_enc.conf
+#
+# Copyright (C)  2019  Franco Masotti <franco.masotti@live.com>.
+# Permission is granted to copy, distribute and/or modify this document
+# under the terms of the GNU Free Documentation License, Version 1.3
+# or any later version published by the Free Software Foundation;
+# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
+# A copy of the license is included in the section entitled "GNU
+# Free Documentation License".
+
+BACKUP_ALIAS_ROOT['SRC']='/*'
+BACKUP_ALIAS_ROOT['DST']='/mnt/backup_root_enc'
+BACKUP_ALIAS_ROOT['UUID']='<put the UUID here>'
+BACKUP_ALIAS_ROOT['DEVICE_MAPPER_NAME']='root_enc'
+
+BACKUP_ALIAS_DATA['SRC']='/data/*'
+BACKUP_ALIAS_DATA['DST']='/mnt/backup_data_enc'
+BACKUP_ALIAS_DATA['UUID']='<put the UUID here>'
+BACKUP_ALIAS_DATA['DEVICE_MAPPER_NAME']='data_enc'
+```
+
+## First backups
+
+Once you have everything in place, including the configuration files,
+you may start backups manually along with Rsync's `--dry-run` option in the 
+script.
+
+    # ./backup.sh '/*' /mnt/backup_root
+    # ./backup.sh '/data/*' /mnt/backup_data
+
+If everything works as expected, remove that option.
+
+You can use Cron for the automatic backups. In my case I use 
+[Cronie](https://github.com/cronie-crond/cronie/). Just remember to
+call the backup scripts from their directory otherwise the configuration files 
+won't be sourced.
+
+~
+
+Till next time!