Added new post. Several fixes.

dev
Franco Masotti 4 years ago
parent d9dc9e57c5
commit 2a0689c301
  1. 2
      _config.yml
  2. 13
      _pages/about.md
  3. 407
      _posts/2018-10-02-automatic-removable-media-syncronization.md

@ -4,7 +4,7 @@ description: "A blog about libre software experiences and everything else
# Author Info
author_name: "Franco Masotti"
author_email: franco.masotti@student.unife.it
author_email: volvopolar731@gmail.com
# Site Settings
baseurl: "" # Base URL must end WITHOUT a slash, default: ""

@ -26,8 +26,8 @@ user running a very lightweight desktop environment like
settled for a plain Lineage OS because of the amount of problems that
surfaced up.
I hope that this turns out to be a useful blog for anyone that encouters
similar problems I have solved.
I hope this turns out to be a useful blog for anyone who encounters
similar computer science problems.
Enjoy!
@ -52,14 +52,15 @@ verbatim:
## Contacts
You will find me on [GitHub](https://github.com/frnmst),
[GitLab](https://gitlab.com/frnmst), [GNU Social](https://quitter.no/frnmst),
[mail one](mailto://franco.masotti@student.unife.it),
[mail two](mailto://franco.masotti@live.com),
[GitLab](https://gitlab.com/frnmst),
[mail one](mailto://franco.masotti@live.com),
[mail two](mailto://volvpolar731@gmail.com),
[Tox](tox:9D855839E4BB0ADBF4F49063BF2ABC1479A7728011F20B563EA104B2EE10FF19DC8C255D8F3D),
Ring,
[XMPP](xmpp://franco.masotti@xabber.de) and other
less relevant web locations.
You won't find me on Facebook, Whatsapp, Twitter, and similar.
You won't find me on Facebook, Whatsapp, Twitter, Telegram, Youtube, etc...
If you want to see a similar blog to which I contribute as a co-author have a
look at [Linux Difficile](https://linuxdifficile.wordpress.com/).

@ -0,0 +1,407 @@
---
title: Automatic synchronization of media files from SD cards
tags: [sync, rsync, media, files, automatic]
updated: 2018-10-02 12:00
description: Automatic syncronization of media files from a removable block device such as an SD card
---
# Backups, yet again
Continuing to talk about [backups](android-backup-with-rsync.html) I want to
share a script I have written for the sole purpose of copying the content of
mass storage devices that contain media files. This applies for example to SD
cards used in digital cameras, but it can be used for any removable block
device.
# Automation
## Use of metadata
The script that I called `auto_media_backup.sh` works on a simple principle: it
uses the metadata of the original file to *compute* the destination directory
of the copied file, like this:
- get the base destination directory from the configuration file (variable `DST_PATH`)
- get the device's UUID using `$ lsblk -o name,uuid` (variable `uuid`)
- get year and month of the media file either using the
[EXIF data](https://en.wikipedia.org/wiki/Exif), if available, or retrieving
some of the [filesystems' metadata](https://en.wikipedia.org/wiki/Comparison_of_file_systems#Metadata)
related to the last access or last modification (variables `year`,
`month` and `filename`)
I opted for the `${DST_PATH}/${uuid}/${year}/${month}/${filename}` scheme.
## Block device UUID monitoring
Another important aspect is that when I insert the card in the reader the
synchronization starts automatically:
```shell
[...] udevadm monitor --udev -s block [...]
```
To avoid synchronizing every removable block device I opted for a whitelisting
method based on the UUID we computed earlier. A code extract should clarify
the situation:
```shell
[...]
for uuid in ${WHITELIST_UUID}; do
if [ "${uuid}" = "$(get_uuid "${devname}")" ]; then
sync_started_message "${uuid}"
[...]
```
## Logging
There are four possible ways to know if the operation started and
how it finished. Each logging system is independent from the other so it
possibile to activate or deactivate them singularly. These systems are:
- logging to `stdout`
- logging to a text file
- emailing using the program `mail`
- beeping using the program `beep`
If you use the mail facility make sure for it to be already configured and
working. In my case I used the
[`s-nail`](https://wiki.archlinux.org/index.php/S-nail) package along with
[`msmtp`](https://wiki.archlinux.org/index.php/Msmtp) as the
SMTP client.
Listening for beeps is a very immediate action and it does not require to
be in front of the computer's monitor or checking the email.
Think about how microwaves, fridges, ovens, washing machines and other domestic
appliances work.
The script logs when a synchronization starts, finishes or
fails and does not know about its progress.
## Loop
The script will continue to monitor for new devices when in idle and does not
quit even if the previous synchronization has failed. If you need to edit the
configuration file be sure to reload the script.
## Synchronization
[Rsync](https://rsync.samba.org/) was an obvious choice for this case because
it provides fast, incremental and secure file copying just with a bunch of
options.
# The script
I saved the following code listing as `auto_media_backup.sh`. Due to the fact
that [a small part of the script was took and modified from the Arch
Wiki](https://wiki.archlinux.org/index.php/Udisks#udevadm_monitor),
this code is licensed under the [GNU Free Documentation License
1.3](https://www.gnu.org/copyleft/fdl.html) or later.
Other small parts were took from a [github
gist](https://gist.github.com/jvhaarst/2343281) which was put
in the public domain.
```shell
#!/bin/sh
# auto_media_backup .sh
#
# Copyright (C) 2018 Franco Masotti <franco.masotti@live.com>.
# Permission is granted to copy, distribute and/or modify this document
# under the terms of the GNU Free Documentation License, Version 1.3
# or any later version published by the Free Software Foundation;
# with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
# A copy of the license is included in the section entitled "GNU
# Free Documentation License".
. ./auto_media_backup.conf
# Check that all the software is installed.
check_tools()
{
which \
mail \
lsblk \
grep \
awk \
udevadm \
udisksctl \
stat \
stdbuf \
rsync \
rm \
beep \
exiftool
} 1>&2
get_uuid()
{
local kernel_device_name="${1}"
lsblk -o name,uuid | grep "${kernel_device_name/\/dev\//}" \
| awk '{print $2}'
}
pathtoname()
{
local kernel_device_name="${1}"
udevadm info -p /sys/"${kernel_device_name}" \
| awk -v FS== '/DEVNAME/ {print $2}'
}
get_mountpoint()
{
local kernel_device_name="${1}"
udisksctl info --block-device "${kernel_device_name}" \
| grep "MountPoints" | awk '{$1=""; print $0}' | sed -e 's/^[ \t]*//'
}
log()
{
local message="${1}"
local log_file_preamble="${2}"
local log_email_subject="${3}"
local log_beep_args="${4}"
if [ "${LOG_TO_BEEP}" = "true" ]; then
beep $log_beep_args
fi
if [ "${LOG_TO_STDOUT}" = "true" ]; then
echo -e "${message}"
fi
if [ "${LOG_TO_FILE}" = "true" ]; then
echo -e "${log_file_preamble} ${message}" >> "${LOG_FILE}"
fi
if [ "${LOG_TO_EMAIL}" = "true" ]; then
echo "${message}" | mail -r "${EMAIL_SENDER}" -s "${log_email_subject}" "${EMAIL_RECIPIENT}"
fi
}
sync_started_message()
{
local uuid="${1}"
message="Starting sync for "${uuid}""
log "${message}" \
"[$(date '+%Y-%m-%d, %T') START auto_media_backup]" \
"Media transfer starting" \
"-f 1000 -l 400 -D 20"
}
sync_completed_message()
{
local uuid="${1}"
local start="${2}"
local end="${3}"
local number_of_files="${4}"
message="Sync successful for "${uuid}", in $(( ($end-$start) ))s. $synced_files synced files"
log "${message}" \
"[$(date '+%Y-%m-%d, %T') OK auto_media_backup]" \
"Media transfer complete" \
"-r 3 -f 2000 -l 400 -D 20"
}
sync_failed_message()
{
local uuid="${1}"
local retval="${2}"
message="Sync for "${uuid}" failed with ${retval}"
log "${message}" \
"[$(date '+%Y-%m-%d, %T') FAILED auto_media_backup]" \
"Media transfer failed" \
"-f 500 -r 4 -d 200"
}
# Since we are going to use the shoot date to organize the directories
# it needs to be computed using EXIF data.
# If the file does not have EXIF data with coherent date field. Use
# filesystem timestamps instead.
# See https://en.wikipedia.org/wiki/Comparison_of_file_systems#Metadata
# for a list of supported timestamps for the most common filesystems.
# 0. Use EXIF data DateTimeOriginal
# 1. Use EXIF data MediaCreateDate
# 2. Use last modification time
# 3. Use last access time
# 4. Bail out
compute_media_date()
{
local media="${1}"
computed_date="$(exiftool -quiet -s -s -s -tab -dateformat "%Y-%m-%d" \
-DateTimeOriginal "${media}")"
if [ -z "${computed_date}" ]; then
computed_date="$(exiftool -quiet -s -s -s -tab -dateformat "%Y-%m-%d" \
-MediaCreateDate "${media}")"
fi
if [ -z "${computed_date}" ]; then
computed_date="$(stat -c%y "${media}" | awk '{print $1}')"
fi
if [ -z "${computed_date}" ]; then
computed_date="$(stat -c%x "${media}" | awk '{print $1}')"
fi
if [ -z "${computed_date}" ]; then
return 1
fi
echo "${computed_date}"
}
compute_media_dst_top()
{
local computed_date="${1}"
local dst_base_path="${2}"
local month="$(date -d ${computed_date} "+%m")"
local year="$(date -d ${computed_date} "+%Y")"
echo ""${dst_base_path}"/"${year}"/"${month}""
}
compute_media_dst()
{
local media_base_dst="${1}"
echo "${media_base_dst}/"$(basename "${media}")""
}
rsync_media()
{
local src="${1}"
local dst="${2}"
mkdir -p "${dst_top}"
rsync -ab --backup-dir="${dst_top}"_backup "${src}" "${dst}"
}
# See https://en.wikipedia.org/wiki/Design_rule_for_Camera_File_system
get_media()
{
local src="${1}"
local dst_base_path="${2}"
# See https://gist.github.com/jvhaarst/2343281
# which was put in public domain.
num_of_files=0
for media in $(find "${src}" -not -wholename "*._*" -iname "*.JPG" \
-or -iname "*.JPEG" -or -iname "*.CRW" -or -iname "*.THM" -or -iname "*.RW2" \
-or -iname '*.ARW' -or -iname "*AVI" -or -iname "*MOV" -or -iname "*MP4" \
-or -iname "*MTS" -or -iname "*PNG"); do
dst_top="$(compute_media_dst_top "$(compute_media_date "${media}")" "${dst_base_path}")"
dst="$(compute_media_dst "${dst_top}" "${media}")"
rsync_media "${media}" "${dst}" || return 1
num_of_files=$(($num_of_files+1))
done
echo "${num_of_files}"
}
# DANGEROUS. Disabled by default.
delete_media()
{
local src="${1}"
echo "NOOP. Source file deletion is disabled because the author does not \
want to be liable for any data loss"
echo "Edit the \"delete_media\" function to enable it"
# rm -rf "${src}"/*
}
loop()
{
local event="${1}"
local devpath="${2}"
if [ "${event}" = add ]; then
devname=$(pathtoname "${devpath}")
for uuid in ${WHITELIST_UUID}; do
if [ "${uuid}" = "$(get_uuid "${devname}")" ]; then
sync_started_message "${uuid}"
start=$(date +%s)
udisksctl mount --block-device "${devname}" --no-user-interaction \
|| { sync_failed_message "${uuid}" "$?"; return 1; }
mountpoint="$(get_mountpoint "${devname}")"
final_dst_path=""${DST_PATH}"/"${uuid}""
synced_files=$(get_media "${mountpoint}" "${final_dst_path}") \
|| { sync_failed_message "${uuid}" "$?"; return 1; }
# Just in case.
sync
if [ "${DELETE_SRC_MEDIA_ON_SYNC_SUCCESS}" = "true" ]; then
delete_media "${mountpoint}"
fi
udisksctl unmount --block-device "${devname}" --no-user-interaction
end=$(date +%s)
sync_completed_message "${uuid}" "${start}" "${end}" "${synced_files}"
fi
done
fi
}
# Original source for the automount part:
# https://wiki.archlinux.org/index.php/Udisks#udevadm_monitor
check_tools || exit 1
printf "%s\n" "Monitoring for these UUIDs:"
printf "%s\n" "==========================="
for uuid in ${WHITELIST_UUID}; do
printf "%s\n" "${uuid}"
done
# Note that this is a loop.
stdbuf -oL -- udevadm monitor --udev -s block | while read -r -- _ _ event devpath _; do
loop "${event}" "${devpath}"
done
```
# The configuration file
You must save and edit the following as `auto_media_backup.conf`.
This file must be placed in the same directory as the script.
Variables should be self explanatory.
Please note that the script does not contain any parsing concerning
these variables. You, as the user, must take care of this.
```shell
# Put your UUIDs separated by a white space character
WHITELIST_UUID="0000-0001 0000-0002"
DST_PATH="/home/user/media_backup"
LOG_TO_STDOUT="true"
LOG_TO_FILE="true"
LOG_FILE="/var/log/auto_media_backup.log"
LOG_TO_EMAIL="true"
EMAIL_SENDER="Computer <your.email@address.com>"
# You can use aliases if your mailing system was set up correctly.
EMAIL_RECIPIENT="all"
LOG_TO_BEEP="true"
# If you want to "recycle" your devices set the following to true.
# See the script for a disclaimer and how to really enable it.
DELETE_SRC_MEDIA_ON_SYNC_SUCCESS="false"
```
# Running
To run the script in the background simply do:
$ chmod +x auto_media_backup.sh
$ ./auto_media_backup.sh &
# Future steps
- More photo compression and/or scaling to save precious space.
- Same for videos.
~
Till next time.
Loading…
Cancel
Save