Distributed filesystems on Proxmox


Hyperconverged virtualisation clusters have always intrigued us and with the likes of windows storage spaces and vmware vsan gaining traction, the days of separate storage clusters are fast becoming numbered. Our goal was to get a running distributed filesystem in place and then attempt to break it.

hyperconvergence results in traditionally separate networking, storage and compute resources being delivered via a single host or cluster.

Ceph vs GlusterFS vs LizardFS?

Significant time was spent researching Ceph, Gluster and LizardFS. We have deployed GlusterFS in production and love it for file storage and it has matured considerably since we last tried it. LizardFS is very similar to GlusterFS and still appears to be in it’s infancy. We will keep an eye on this as the project has potential. Ultimately, because our preferred hypervisor supports it out of the box we decided to go with:

The hardware:

1x supermicro 5029p-wtr 64GB RAM, 1x 10 core xeon, 256GB NVME, 10Gbps fibre card and 8x 4TB 7200RPM seagate constellation
2x supermicro 6029-wtr with 96GB RAM, 2x 8 core Xeon, 256GB nvme, 10Gbps fibre card and 12x 4TB 7200RPM seagate constellation
Allied Telesis XS916MXT

Base Installation:

  • Each server loaded with Proxmox VE 5.3 and configured with static addresses.
  • Edit hosts file on each server to ensure that all servers can ping each other via IP and hostname
  • Form a cluster using the gui tools – we typically do this using the included pvecm add command but decided to give the GUI tools a whirl this time round.

Ceph Installation:

Excellent “how to” from Proxmox on how to install ceph. These instructions are a little dated, ceph is updated to Luminous in 5.3 and this video references an older release.


Using spinning hard drives and a traditional storage design powered by FreeNas 11.2, we are able to achieve reasonable performance in a Windows Server 2016 VM.

Freenas 6x mirrored vdevs using the same 4TB drives – NFS mount

Ceph Results:

Spining hard drives are not a recommended configuration for Ceph and this is confirmed with our results. Using ceph, the best sequential figures we could get is 400MB/s. We were impressed with the resiliency of the system. Drives we pulled at random and all data was successfully re-synced across the pool when the drive was re-added. We even went as far as formatting various disks – Ceph dealt with this like a champ and moved data to compensate. It is a very robust filesystem and in the right environment, we recommend it highly.

Why should I consider the open source solution?

Proxmox VE is based on the same technology that powers some of the worlds biggest clouds and comes at a fraction of the cost of the alternatives. When hyperconvergence is factored in, the cost of the paid solutions from Microsoft and VMWare can very quickly grow into thousands in licensing fees alone. Ceph is available at no cost and is a widely used and trusted platform for storage of virtual machines across many clouds and businesses. The low cost of entry and zero licensing fees allows businesses to save on unnecessary IT costs or invest in additional resources.