So far all my setups have had root on SSD mirror with separate hard disk storage pool for all the data. Years ago I used to keep the app config, databases and docker files on the root filesystem, while the app data resided on the storage pool. That was cumbersome for backups and storage size. Eventually I moved all app data to the storage pool. Essentially the apps can be started on any machine with a Linux OS that has docker installed. Database access is slower but it’s a decent compromise for having trivial all-in-one snapshots and backup. Now I’m setting up a new NAS for a friend and I’m wondering whether it’s worth keeping the root filesystem separate from the storage pool. If I put it on the disks, I’d get trivial full system snapshots and backups. I’d have the same hardware reliability as the storage pool. There wouldn’t be issues with root filling up. The caveat is that the OS would be slower. Has anyone reasoned and/or tried this? Should I go for it?
E: I recently put my laptop’s root on ZFS and the ability to do full backups while the system is running is pretty great. The full system can be pretty trivialy restored to a new drive with zfs send / recv during setup.
You generally keep OS and storage separate for functionality, not necessarily because one is more safe than the other these days with more advanced journaling filesystems that can self-heal and keep things pretty safe and sound.
The main drawbacks to having them combined is all surrounding flexibility. If one fucks up, everything is fucked up. You won’t be able to perform rescue operations on either without impacting both at the same, you can’t change the layout of one without affecting both…etc.
Performance is obviously another one, but if you’re not running critical operations for a business or whatever, it probably doesn’t matter.
Right, the flexibility angle makes sense if using a typical root fs like Ext4 with or without LVM. That’s a reason I’ve always kept the OS separate. But with ZFS there’s unlimited flexibility. Separate datasets or volumes within the same storage pool are trivial. I could do root on ZFS on separate SSDs and get those benefits but it’s more complicated that slapping it all in a single pool. Then maybe use the SSDs for cache. :D