I have a library of movies, TV shows and music for local hosting. Right now I have a script that uses cp to copy newly added media files to a second, identical drive in another room as a backup. there has to be a better way to do this.
ignoring cloud storage and RAID, what are some command line tools that could ensure that the integrity of the files remains intact? assuming the drive continues to function properly.
i think rsync does what you need
Definitely look into rsync. cp is not sufficient to rely on for this.
Unless I am horribly mistaken, this is what file hasing is for. You could use a speedy hash to compare the copies as an addendum to your script.
I would probably just pick something from the giant pile of back up software and enable any error checking as simpler solution though.
Right now I have a script that uses cp to copy newly added media files to a second, identical drive in another room as a backup. there has to be a better way to do this.
I would use
rsync
. It can be told exactly what kind of updates are needed from many different criteria, including modified time and file hashing. This is likely what you want.I sometimes use
git
as a form of backup, particularly my writing, that I push to remote locations. Its great for text, but not so much for already compressed media.Finally, I also use
syncthing
(which is a full program, not a CLI tool) to keep several machines synced up. There are many file versioning options and encryption that can be set independently on each machine.backups will depend lots on your setup and how far you want to go with it. if you use filesystems like btrfs or zfs you can copy snapshots (across different computers, multiple disks in the same computer, across the web). there are tools dedicated to backup like borg that tries to simplify the whole process. if you want to diy, you can use rsync to do checksumming and copying only new or different files. you can even do snapshot-like backups with a bit of effort.
regardless of the method you use, keep in mind backups are for when things go wrong. ensure they are accesible to you whenever you need them (physically accesible, you have the keys if you chose to encrypt them, the hardware is in good state, things like that). people tend to recommend to keep at least 3 copies in two different formats and one in a different physical place (also known as the 3-2-1 rule) in case you loose two copies at the same time (in case of natural disaster for example).
As an aside, RAID is more of a fault tolerance measure than a backup. I would still backup data stored in a RAID1. (Not that this applies to OP’s question)
You could put them in the same machine and set the file system to zfs. But that’s enterprise grade and can be a pain to work with imo. I switched over to and UnRAID server with double parity, should do the trick.