I've used unison for a long while for keeping things like my music in sync between machines. But it's never felt entirely safe, or right. (Or fast!) Using a VCS would be better, but would consume a lot more space.
Well, space still matters on laptops, with their smallish SSDs, but I have terabytes of disk on my file servers, so VCS space overhead there is no longer of much concern for files smaller than videos. So, here's a way I've been experimenting with to get rid of unison in this situation.
Set up some sort of networked filesystem connection to the file server. I hate to admit I'm still using NFS.
Log into the file server, init a git repo, and check all your music (or whatever) into it.
When checking out on each client, use
git clone --shared. This avoids including any objects in the client's local
git clone --shared /mnt/fileserver/stuff.git stuff
- Now you can just use git as usual, to add/remove stuff, commit, update, etc.
git addis not very fast. Reading, checksumming, and writing out gig after gig of data can be slow. Think hours. Maybe days. (OTOH, I ran that on an Thecus.)
- Overall, I'm happy with the speed, after the initial setup. Git pushes data around faster than unison, despite not really being intended to be used this way.
- Note that use of
git clone --shared, and read the caveats about this mode in
git repackis not recommended on clients because it would read and write the whole git repo over NFS.
- Make sure your NFS server has large file support. (The userspace one doesn't; kernel one does.) You don't just need it for enormous pack files. The failure mode I saw was git failing in amusing ways that involved creating empty files.
- Git doesn't deal very well with a bit flipping somewhere
in the middle of a 32 gigabyte pack file. And since this
method avoids duplicating the data in
.git, the clones are not available as backups if something goes wrong. So if regenerating your entire repo doesn't appeal, keep a backup of it.
(Thanks to Ted T'so for the hint about using --shared, which makes this work significantly better, and simpler.)