Hyperconverged virtualisation clusters have always intrigued us and with the likes of windows storage spaces and vmware vsan gaining traction, the days of separate storage clusters are fast becoming numbered. Our goal was to get a running distributed filesystem in place and then attempt to break it.
Ceph vs GlusterFS vs LizardFS?
Significant time was spent researching Ceph, Gluster and LizardFS. We have deployed GlusterFS in production and love it for file storage and it has matured considerably since we last tried it. LizardFS is very similar to GlusterFS and still appears to be in it’s infancy. We will keep an eye on this as the project has potential. Ultimately, because our preferred hypervisor supports it out of the box we decided to go with:
- Each server loaded with Proxmox VE 5.3 and configured with static addresses.
- Edit hosts file on each server to ensure that all servers can ping each other via IP and hostname
- Form a cluster using the gui tools – we typically do this using the included pvecm add command but decided to give the GUI tools a whirl this time round.
Using spinning hard drives and a traditional storage design powered by FreeNas 11.2, we are able to achieve reasonable performance in a Windows Server 2016 VM.
Spining hard drives are not a recommended configuration for Ceph and this is confirmed with our results. Using ceph, the best sequential figures we could get is 400MB/s. We were impressed with the resiliency of the system. Drives we pulled at random and all data was successfully re-synced across the pool when the drive was re-added. We even went as far as formatting various disks – Ceph dealt with this like a champ and moved data to compensate. It is a very robust filesystem and in the right environment, we recommend it highly.
Why should I consider the open source solution?
Proxmox VE is based on the same technology that powers some of the worlds biggest clouds and comes at a fraction of the cost of the alternatives. When hyperconvergence is factored in, the cost of the paid solutions from Microsoft and VMWare can very quickly grow into thousands in licensing fees alone. Ceph is available at no cost and is a widely used and trusted platform for storage of virtual machines across many clouds and businesses. The low cost of entry and zero licensing fees allows businesses to save on unnecessary IT costs or invest in additional resources.