FreeNAS on HP Microserver N54L - Shared Storage for your ESXi

The HP Microserver N54L and its predecessors N40L and N36L are widely spread home servers. I've written about using it as ESXi host in my Homelab about 2 years ago. Today it is still a great system to be used with ESXi but there are better alternatives, especially due to its limited memory and CPU power. That was the reason why I rededicated my N40L to be a central storage and file server with FreeNAS. This post describes what you need to use the N40L as resilient storage and how to configure it properly for ESXi usage. I will talk about NFS sync writes or performance issues with NFS vs. iSCSI and how to work around them.

Hardware Guide

• HP Microserver: If you do not already have a Microserver, go for the N54L.
• Memory: FreeNAS requires 8GB of RAM to run properly: Kingston KVR1333D3E9SK2 or Corsair CT2KIT102464BA1339.
• Hard Drives: I recommend the WD Red drives (2TB, 3TB or 4TB). The Red series was specially made for NAS usage. You can also use the WD Velociraptor model which is a little bit faster, but much more expensive. The system has 4 slots, but you can add a 5th drive to the cd/dvd slot.
• NIC: An additional NIC is recommended for performance and redundancy: Intel Gigabit PCI-E Network Adapter EXPI9301CTBLK
• USB Flash Drive: I recommend installing Freenas to a USB Flash Drive. The recommended boot device size for FreeNAS is 8GB: SanDisk Cruzer Fit 8GB
• HP MicroServer Remote Access Card: For out-of-band management.

Installation & Configuration

Installation
Installation is very simple. You can use the internal USB port for the USB Flash Drive. If you want to use a 5th hard disk, you can fit it into the cd/dvd slot.

1. Downlad the latest Release of FreeNAS
2. Boot the installer (Burn it to a CD or mount it with the Remote Access Card)
3. Select the 8GB flash drive as target
4. Enter your root password
5. Remove the CD and reboot

That's it. You FreeNAS will now reboot and come up with DHCP enabled. The configuration can be done with the web interface.

Initial Configuration

1. Open the Web Interface
2. Login as root with the password you set during the installation. This will start the initial configuration wizard.
3. Set your language and exit the wizard at the second step
4. If you want to set a static IP address you have to add an interface in Network > Interface 

Create a Volume
With 4 disks, you can build a RAID10. This will give you the best performance. If you have 3 or a 5th disk you can create a RaidZ. This will give you the most capacity.

1. Open Storage > Volumes > Volumes Manager
2. Enter a Name
3. Configure your volume layout. RaidZ (3 or 5 Disks required):
   
   or Raid10 (4 disks required):
   
4. Press Add Volume

FreeNAS running on your N54L is now ready to be used. From that point you can create your shares to use it as Fileserver or shared storage for ESXi.

Create a NFS Share for ESXi

1. Open the Storage Tab
2. Select the Volume
3. Press the Create Dataset icon
   
4. Enter a Dataset Name and press Add Dataset
5. Navigate to Sharing > UNIX (NFS) and press Add Unix (NFS) Share
   
6. Browse for the Path you want to share (/mnt/[volume]/[dataset])
7. Press Advanced Mode
8. Change Maproot User: to root
9. Press OK
10. When it asks to enable services, press Yes

You can mount the share on your ESXi host with  Configuration > Hardware > Storage > Add Storage > Network File System

Create a iSCSI Share for ESXi

1. Navigate to Sharing > Block (iSCSI) > Portals and click Add Portal
2. Enter a Comment and press OK
3. Navigate to Initiators and click Add Initiator
4. Press OK
5. Navigate to Targets and click Add Target
6. Enter Target Name, Target Alias and set the Port/Initiator Group ID to 1
7. Navigate to Extends and click Add Extend
8. An extend is a file that simulates the Block device. You have to set a fixed size. Select a Extent Name (same as Target Name possible), a Path and the Extent Size
   
   
9. Navigate to Associated Targets and click Add Target / Extent
10. Map Target and Extent and press OK
11. Open Services Tab and enable iSCSI

iSCSI vs NFS Performance

Without doing any further optimization like jumbo frames or multipathing, my N40L with 5 WD Red 3TB drives in RAIDZ could achieve the following results.

iSCSI
Write: 72 MB/s - 145 IOPS (I/O Analyzer 512k 100%Read 0%Random)
Read: 98 MB/s - 200 IOPS (I/O Analyzer 512k 0%Read 0%Random)

NFS
Write: 5 MB/s - 20 IOPS (I/O Analyzer 512k 100%Read 0%Random)
Read: 52 MB/s - 120 IOPS (I/O Analyzer 512k 0%Read 0%Random)

That's what I expected. But why is NFS performance so bad? ESXi does synchronous writes operations. The host requests an acknowledgement that the data is actually written to the physical disk before it starts another write. A sync write causes the ZFS filesystem to flush it's ZIL (ZFS Intent Log) to the disk. The ZIL is a log of sync writes that ZFS uses to write data more efficiently to the disk.

So, what can you do to speed NFS up?

1. Write the ZIL on a separtate SSD. This will speed up NFS a little bit. I could achieve around 30 MB/s with a SLOG (Separate Intent Log).
2. If you do not care about the integrity of your VM, you can disable sync writes. If you do so, be aware that if your storage crashes while data is cached but not written to the disk you will end up with corrupt data.
   Disable Sync Writes:

   # zfs set sync=disabled [POOL]/[DATASET]

   Enable Sync Writes:

3. # zfs set sync=standard [POOL]/[DATASET]
   

   You can verify the sync status with

4. # zfs get sync