So I have a nearly full 4 TB hard drive in my server that I want to make an offline backup of. However, the only spare hard drives I have are a few 500 GB and 1 TB ones, so the entire contents will not fit all at once, but I do have enough total space for it. I also only have one USB hard drive dock so I can only plug in one hard drive at a time, and in any case I don’t want to do any sort of RAID 0 or striping because the hard drives are old and I don’t want a single one of them failing to make the entire backup unrecoverable.
I could just play digital Tetris and just manually copy over individual directories to each smaller drive until they fill up while mentally keeping track of which directories still need to be copied when I change drives, but I’m hoping for a more automatic and less error prone way. Ideally, I’d want something that can automatically begin copying the entire contents of a given drive or directory to a drive that isn’t big enough to fit everything, automatically round down to the last file that will fit in its entirety (I don’t want to split files between drives), and then wait for me to unplug the first drive and plug in another drive and specify a new mount point before continuing to copy the remaining files, using as many drives as necessary to copy everything.
Does anyone know of something that can accomplish all of this on a Linux system?
This will happen in any case unless you had enough capacity for redundancy.
What is in this 4TB drive? A Linux installation? A bunch of user data? Both? What kind of data?
The first step to this is to separate your concerns. If you had e.g. a 20GiB Linux install, 10GiB of loose home files, 1TiB of Movies, 500GiB of photos, 1TiB of games and 500GiB of Music for example, you could back each of those up separately onto separate drives.
Now, it’s likely that you’d still have more data of one category than what fits on your largest external drive (movies are a likely candidate).
For this purpose, I use https://git-annex.branchable.com/. It’s a beast to get into and set up properly with plenty of footguns attached but it was designed to solve issues like this elegantly.
One of the most important things it does is separate file content from file metadata; making metadata available in all locations (“repos”) while data can be present in only a subset, thereby achieving distributed storage. I.e. you could have 4TiB of file contents distributed over a bunch of 500GiB drives but in each one of those repos you’d have the full file tree available (metadata of all files + content of present files) allowing you to manage your files in any place without having all the contents present (or even any). It’s quite magical.
Once configured properly, you can simply attach a drive, clone the git repo onto it and then run a
git annex sync --contentand it’ll fill that drive up with as much content as it can or until each “file”'snumcopiesor other configured constraints are reached.