So I’m not really a selfhoster per say - but I run 2 NAS at home and I am working my way toward it. I don’t really need my stuff open to the internet so I just use it on my LAN.
However I do have a lot of data, and I’m constantly backing things up. My question is - I have the following setup:
- Computer hard drives
- These backup to my NAS
- I have a separate HDD in an enclosure I plug into the NAS directly and copy my data onto every few months to put in my safe.
- Some cloud storage for very important smaller stuff (pictures)
My main question is - what is the best way to copy off new data from my NAS (Synology) to my “cold storage” drive, without recopying everything every time? is there a way to detect the files that exist on both and only copy new ones? I’ve always had this question when doing backups and it seems to always be overly complex.
You guys are very knowledgeable so I’m sure someone has dealt with this!
For backking up files I can recommend restic. It encrypts and compresses your backup. At the same time it does deduplication. So if you backup a file twice, it takes only the space of one. If you are lucky it will even do that for oaets of a file. Although it only has a cli.
And it’s incremental with snapshots. There are options with Web UI for it like backrest but I didn’t use them.
I use an
rsyncjob to do it. Rsync by default uses the files metadata to determine if the file has changed or not and updates based on that. You can opt to use checksums instead if you’d rather. IIRC, you can do it with a Synology task, or just do it yourself on the command line. Ive got a Jenkins setup to run it so i can gather the logs and not have to remember the command all the time (and i use it for other periodic jobs as well), but its pretty straightforward on its own.Rsync is the correct solution. It does exactly what you want and nothing more. A script that uses rsync is future-proof. Other backup solutions depend on the maintenance of the software, which could be abandoned, go up in price, or have vulnerabilities.
I’d recommend a solid backup client. This isn’t something you want to find broken when you need it.
Kopia is what I use, and it supports local (LAN) targets, as well as cloud storage if you want 3:2:1 for some or all of your data. Good luck!
Agreed. Rsync will happily sync a deletion you didn’t mean to do.
Rsync can rename/move target changed or gone from the source.
I use those options so that I get a separate ‘archive’ dir next to my backup target, with old versions of files.
It’s useful for loose collections such as photos.
For photos I’d look at running immich as it handles duplicates well, and it also plays nicely for you to categorise your photos and videos. For general backups, as said, I would personally look at a solution. I personally use duplicati, others exist, you can have a look into what you need. rsync can do the job, but I found duplicati quicker and easier to setup
I do use duplicati for my desktop and its great.
This is more and issue of having matching, linux, and 1 windows (soon to be linux) pc. So mixing and matching is hard. I have 1 nas for movies and music and my other nas is for music production backup. That one auto backs up from my Mac.
So then, every few months I copy that nas onto another drive and put it in the safe. I guess I want a better way to do that other than plugging the drive into the nas and manually copying everytbing over.
What are you trying to guard against with backups? It sounds like your greatest concern is data loss from hardware failure.
The 3-2-1 approach exists because it addresses the different concerns about data loss: hardware failures, accidental deletion, physical disaster.
That drive in your safe isn’t a good backup - drives fail just as often when offline as online (I believe they fail more often when powered off, but I don’t have data to support that). That safe isn’t waterproof, and it’s fire resistance is designed to protect paper, not hard drives.
If this data is important enough to back up, then it’s worth having an off site copy of your backup. Backblaze is one way, but there are a number of cloud based storages that will work (Hetznet, etc).
As to your Windows/Linux concern, just have a consistent data storage location, treat that location as authoritative, and perform backups from there. For example - I have a server, a NAS, and an always-on external drive as part of my data duplication. The server is authoritative, laptops and phones continuously sync to it via Syncthing or Resilio Sync, and it duplicates to the NAS and external drives on a schedule. I never touch the NAS or external drives. The server also has a cloud backup.
There are some useful options which can be found in the rsync manual although knowing which are redundant or conflicting can be tricky. If you run with the ‘n’ option first it will dry run, then you can remove the n to do the needful.
rsync -navhP --no-compress --ignore-existing --mkpath source dest -n dry-run, no changes are made -a is for archive, which recursively preserves ownership, permissions etc. -v is for verbose, so I can see what's happening (optional) -h is for human-readable, so the transfer rate and file sizes are easier to read (optional) -P show progress --no-compress as there's no lack of bandwidth between local devices --ignore-existing will skip updating files that exist on receiver -r is recursive, included in -a --mkpath will create the destination's path component
