Server NAS

My main computer clocks in with something like 6 hard drives totalling over 11 terabytes of space. That sounds like quite the flex until you realize that 90% of that is 5 year old disk based hard drives that have moved from computer to computer like a hermit crab changing its shell. Less than 3 of those terabytes are SSD.

I've done some level of backups within that mesh of drives, backing things up from drive A to drive R, etc. But all of those backups have been manual, and all within the same tower. One lightning bolt could take it all out.

For a long time I've known that I need to do some kind of backup for all those videos that I filmed in high school when I was learning to edit, or all the family photos. Growing up with terrible internet, I've become something of a digital hoarder. Never knowing when internet may go down for days developed within me a real distrust of any data not on local disk. I've never really considered backup services for that reason as well as for the cost, and as I discussed in going open source I'm not keen to pump my personal files up to an off premise service.

So when I finally decided to do a backup solution, I came up with two main goals. First, the backup must be local, and accessible over LAN / Wifi. Second, the backup must be durable and resistant to data corruption. In my mind, this meant a RAID (Redundant Array of Inexpensive Disks) NAS (Network Attached Storage). Raid basically means storing data on two or more hard drivers, where basically the data can be lost or deleted from one disk, and still read from another one, so even if a drive fails you don't lose data.

I asked around at my company and got a number of recommendations for different ways to do RAID as well as good NAS units to buy. I was surprised at how expensive most NAS options were. Most focused on the streaming aspect and had large amounts of ram; it seems the main use case is for music and video streaming. This use case is adjacent to mine, where most media I consume is purchased and local to my used device; I was looking more for a library to pull copies of media from, and where I could have a second resting place for documents etc.

And then a server admin coworker gave me a really strange suggestion: why not buy a whole server?

I had never considered buying an actual server, but my coworker made a great argument: they're dirt cheap, business grade, and much more powerful than a comparable NAS. The server I bought was slightly more expensive than the NAS I was considering, but it had significantly more hard drive space, something like 8x the ram and a much better processor. For most consumers it would have been way overkill, and would require a server admin, but for me it would prove to be an ideal learning experience. As someone who has spent much of my career pushing software to "other people's servers", having my own server mean I could dip my toes into the server admin role without risking bringing a real production down.

So far I've stumbled through setting up RAID for the drives and installed a GUIless Ubuntu, which means the only way to interact with the server is through a remote terminal and using shell commands. It's been a learning experience but it's been great to shell in from my windows desktop or linux laptop to tweak something. I've also setup mounts so that both computers can browse files from the server, and copied over all the files that should be backed up. I've installed (and then disabled) Pi Hole and set up a minecraft server that runs all my custom mods as well as runs as a service with a huge amount of ram. (As a side note I created a utility app inspired by git that looks at the updated date of every file in a minecraft world and saves only those that change to different 'version' folders. It lets me take regular backups at 1/10th the normal size, but then roll back to whatever version if something goes wrong. It's in a private repo at the moment but if I break it out I'll update here with a link). I also set up a git server for private repos that I wouldn't want on github (like for pass).

Finally, I recently created a Basic Backup app that let's me recursively grab all files in a folder (and its children) and copy any new or updated files over to a destination, erasing the old version. It's not sophisticated at all, but it works over a network mounted folder and let's me idempotently back up any 'new pictures' etc with a single command. It has no versioning, but as anything that logically should be versioned is in various git repos, it fits my use case nicely.

While not at all what I expected would be the outcome when I started looking into backup solutions, I couldn't be more pleased with running a local server. It's way overkill for my use case, but it's been a great learning experience and seems like a shell that I could continue growing into for years to come.