This version includes the following changes
- Uses bind mounts instead of symlinks
- Flock is used to handle the multiple syncs happening together
- Backups are compressed using zstd
- The backup process is optimized and uses link-dest flag to decrease the size of backups
This will avoid symlinking issues and race conditions which leave the
directory in inconsistent state. The change was inspired by the
implementation of @ThibaultLemaire with some minor tweaks to keep the
backup functionality intact. Directories with hardlinks aren't supported
anymore to keep the backup functionality intact
Co-authored-by: Thibault Lemaire <thibault.lemaire@zoho.eu>
This change won't preserve the hardlinks anymore right now, but will
allow the backups to be much faster and space efficient.
Also, this change maintains two states now, one directory refers to the
directory before running asd and the other directory keeps track of the
directory after running asd
If a proper unsync happens. The new directory will be moved atomically
to the place where the original directory existed, if not, the previous
directory will be backed up in tar.zstd format and the updated directory
will be moved back to the place of original directory ( since this
directory might have suffered syncing issues, backup is formed from the
old directory)
This can help save the storage space as well as decrease the writes to
HDD ( the main goal of asd ) incase of some inconsistent state
It adds a dependency of zstd as it is very efficient in terms of
performance as well as compression ratio which shouldn't give much
overhead while compression
instead of locking all instances, we will lock the seperate asd
instances based on ASDNAME. This will allow parallel rsync in cases
where asd don't interact with each other
flock will now be a dependency of asd, asd will automatically lock
itself whenever run so that at a point of time, only one instance of asd
can be run. This can avoid race conditions that causes the rsyncs to
lockup due to multiple parallel syncs happening
- During unsync, it was trying to remove all the directories in all the
directories instead of just the directory which had been unsynced
- In rare scenarios when some race condition occurs, the directory as
well as the backup directory both of them existed which caused the move
to fail. So, during ungraceful state check, since we have a backup, We
can delete the directory so that the move is successful
NOTE: This might still fail if in the short period of time, any other
process/OS creates our sync directory again before the move is done
( This race can only be fixed by bind mounting suggestions I think )
running asd with `DEBUG=1` environment variable will give additional
insights on the working of asd and what it is doing, it can helpful for
debugging issues in future
* Document ASDCONF and use it for initial check of config file existence
* Ensure no collisions between different instances
* review
* fix double hyphen when no ASDCONF
asd.conf is needed by the main script in order to run, so it
would be better to install them togheter, improving installation
for users who are not using systemd nor upstart.
Update bail filter to allow using Zram devices, which are not tmpfs. Also, better filtering of tmpfs, to be sure the grep does not match for path like /path/to/fake/tmpfs mounted.
Manorit Chawdhry