From time to time we would like to share information of how the solutions we put in place actually prove to be beneficial to our clients. These aren't intended to be overly technical - if there is demand for that, then we will definitely consider doing something though. The intention here is to simply highlight a solution and how we use it to make our clients more productive.
In this first case, we would like to highlight a feature of one our favourite storage solutions - Freenas. Specifically, snapshots.
Freenas you say?
Freenas is an open source network attached storage (NAS) solution based on FreeBSD and is developed by ixsystems. Freenas has many integrated plugins that can extend functionality beyond traditional storage use cases, however we will not be discussing those in this post. Because FreeNAS is based on FreeBSD it also integrates ZFS, and ZFS is awesome.
Apple made huge shifts around the time of Leopard (10.5) to integrate ZFS into the Mac OS, the rumor at that time was that Apple was going to shift entirely to ZFS for the Mac. Unfortunately, that did not occur. Apple persisted with HFS+ and have only recently shipped their APFS file system in iOS devices (which is why it takes so damn long to update to iOS 10.3!). Many of the concepts of APFS stem from Apple's flirtation with ZFS - specifically, for the case of this post, live snapshots.
Backups, backups, backups....
When disaster happens, all administrators rely on backups to restore data. The problem here is that the data is often hours/days old and is more often than not, untested. In the case of cloud based backup solutions, an internet connection is required which makes this a major bottleneck in restoring. A snapshot by no means takes the place of traditional backup solutions, rather, snapshots supplement traditional backup solutions.
Background and fix
We support a SMB client that uses a highly availably Proxmox based KVM solution for virtualising 10 servers, this setup comprises 3 compute nodes and 2 storage nodes (both Freenas).
We received a call for assistance with one of the VM's as it was unresponsive, upon checking, yes it was indeed unresponsive. All repair options were presented to the customer and the contact opted to restore from the ZFS snapshot.
As freenas had been configured for snapshots every 30 mins we were able to simply shut down the affected virtual machine, mark it as damaged, browse into the ZFS snapshot directory and simply copy the VM to the correct folder in the live directory and start the machine back up in the Proxmox console.
This process took a grand total of 7 minutes.
ZFS snapshots provide a fast reliable way to restore to a point in time without having to jump onto a backup server or online service which could house both older and untested data. In this use case, the customer made a decision to get a production system back up and running in the shortest period of time available, with minimal data loss. This decision resulted in staff production being restored quickly and also resulted in reduced technical support costs as the time required to implement this was drastically less than the time required to restore and test a backup.