Sunday Windows gardening - Windows Server Technical blog

Testing Windows Server 2016 Storage Spaces Direct with Dell PowerEdge servers

Ariel Delgado. Posted 10/13/2016. Last Updated 10/16/2016.

All the cool kids are doing it. So below is a summary of my hands-on testing of Storage Spaces Direct (a.k.a. S2D) with my free trial copy of Windows Server 2016 Technical Preview 5. I was able to build a Hyper-Converged Hyper-V cluster using this exciting new Windows Server technology and test for decent performance once built.

On future blogs I will post my experiences when enabling Storage Spaces Direct utilizing just three or even two servers. But for now, let's abide by the requirements posted by Microsoft here.

This test environment consists of four PowerEdge 13th Generation servers named 1 thru 4 and joined to a very creative "contoso.com" domain as illustrated below:

Specs (CPU/Model/RAM)

The servers I will be using are not the same model and their hardware (CPU's, amount of RAM) differs slightly across the four nodes:

Note: My test servers don't have RDMA capable NICs so I had to leave the configuration of RDMA section out of this proof-of-concept lab. I'm using a single 1GB NIC per server for management traffic and a single 10GB NIC per server for CSV/Heartbeat/Live Migration traffic.

Disk Subsystem

Host Bus Adapters (HBA's)

Each server has a plain SAS HBA (Non-RAID capable). These devices don't have to be the same model/brand but it is important to note that they can't abstract the hardware layer in any form (a.k.a. no RAID). A special note, if you are trying to use a RAID controller in "HBA Mode or disks in "Non-RAID-mode"" it won't work as the disks will still be presented to the OS as connected to a RAID controller and won't be eligible for Storage Spaces Direct. You could check the available SAS controller devices in Device Manager or via PowerShell with the command below. I prefer tables and rows and columns.

Get-WmiObject -Class Win32_PnPSignedDriver | WHERE {$_.DeviceClass -like "*SCSIADAPTER*"} | ft -AutoSize PSComputerName, DeviceName, Description, DriverProviderName, Manufacturer

In this test environment some of the SAS HBA adapters connect directly to the local servers disks via the internal backplane and some others connect to external hard drives on a JBOD enclosure.

Disks

The following PowerShell command lists the available disks attached to each server:

Get-PhysicalDisk | ft -AutoSize FriendlyName, SerialNumber, Usage, @{N="Size (GB)";E={ "{0:N0}" -f ($_.Size/1000/1000/1000) }}, MediaType, BusType

Notice that, in addition to the disk (or disks) where the OS is installed, each server has two Solid State drives, two SAS 600GB hard drives and two SAS 1TB hard drives. So we are cool with the requirement specified by Microsoft here:

"Each server must have at least 2 solid-state drives, and at least 4 additional drives. The SATA and SAS devices should be behind a host-bus adapter (HBA) and SAS expander."

As seen below, the disks (except the OS disk/disks of course) are also automatically picked up by the Storage Spaces Controller and visible in Server Manager:

Create a Cluster

Add Roles and Features

This is a Hyper-Converged configuration where VM's will run directly on the servers providing the storage. Therefore, make sure the Hyper-V role and the Failover clustering feature have been added to all the cluster nodes:

Install-WindowsFeature -Name Failover-Clustering -IncludeManagementTools Install-WindowsFeature -Name Hyper-V -IncludeManagementTools -Restart

Prepare disks for S2D

According to this Microsoft Technet article the disks intended to be used for Storage Spaces Direct need to be empty and without partitions or other data.

Note: You can verify the status of your S2D disks examining the cluster validation report that will be generated later.

Run the below script (from the link above) on each server to ensure the applicable disks are ready for Storage Spaces Direct.

Update-StorageProviderCache Get-StoragePool |? IsPrimordial -eq $false | Set-StoragePool -IsReadOnly:$false -ErrorAction SilentlyContinue Get-StoragePool |? IsPrimordial -eq $false | Get-VirtualDisk | Remove-VirtualDisk -Confirm:$false -ErrorAction SilentlyContinue Get-StoragePool |? IsPrimordial -eq $false | Remove-StoragePool -Confirm:$false -ErrorAction SilentlyContinue Get-PhysicalDisk | Reset-PhysicalDisk -ErrorAction SilentlyContinue Get-Disk |? Number -ne $null |? IsBoot -ne $true |? IsSystem -ne $true |? PartitionStyle -ne RAW |% { $_ | Set-Disk -isoffline:$false $_ | Set-Disk -isreadonly:$false $_ | Clear-Disk -RemoveData -RemoveOEM -Confirm:$false $_ | Set-Disk -isreadonly:$true $_ | Set-Disk -isoffline:$true } Get-Disk |? Number -ne $null |? IsBoot -ne $true |? IsSystem -ne $true |? PartitionStyle -eq RAW | Group -NoElement -Property FriendlyName

The output of this command summarizes the empty disks installed on the server (model and count). Note that these lists may include non-Storage-Spaces-Direct-Claimable disks (i.e. RAID bus type)

Validation Report

Get a validation report before creating the cluster. Ensure the Storage Spaces Direct test is included.

Do it via PowerShell:

Test-Cluster -Node Node1,Node2,Node3,Node4 -Include "Hyper-V Configuration", "Storage Spaces Direct","Inventory","Network","System Configuration"

Or using the cluadmin.msc GUI:

Examine the Validation Report

Notice that the cluster validation test now includes a test for Storage Spaces Direct:

Have lots of fun examining the results for each of the Storage Spaces Direct tests:

Create the Cluster / Configure a Witness

At this point you are ready to create the cluster. Use the GUI or run the below command in PowerShell:

New-Cluster -Name "Cluster100" -Node "Node1", "Node2", "Node3", "Node4" -NoStorage -StaticAddress Your.IP.Address.Here

Make sure you configure your quorum settings (preferably configure a file share witness or a cloud witness considering we are using an even number of nodes).

Things to note at this point

Microsoft ClusPort HBA

In the Device Manager of each cluster node, there is now a "Microsoft ClusPort HBA". This new device was installed when the Failover Clustering feature was enabled.

Read this Microsoft blog for details on this new device.

Quoting directly: "The Software Storage Bus (SSB) is a virtual storage bus spanning all the servers that make up the cluster. SSB essentially makes it possible for each server to see all disks across all servers in the cluster providing full mesh connectivity. SSB consists of two components on each server in the cluster; ClusPort and ClusBlft. ClusPort implements a virtual HBA that allows the node to connect to disk devices in all the other servers in the cluster. ClusBlft implements virtualization of the disk devices and enclosures in each server for ClusPort in other servers to connect to."

Get-PhysicalDisk cmdlet

Running Get-PhysicalDisk on any node in the cluster now displays the pool of all disks available to each cluster node:

Cluster Pools

Notice that at this point, there are no cluster pools created automatically:

Cluster Enclosures

If some of your SAS/SATA/PCIe disks are presented to the server via a JBOD device, it should appear in Failover Cluster Manager:

Enable Storage Spaces Direct

Now to the fun stuff. With your cluster created and your S2D disks ready, run the below command:

Enable-ClusterS2D -CimSession Cluster100.contoso.com -PoolFriendlyName myS2DPool

This command takes a while to complete. I've seen it take from 10 minutes to around an hour depending on how many hosts are members of the cluster. Be patient ;)
According to the advertising, this command does the following automatically:
1) Pool: Creates a single large pool (in the above command this pool is named "myS2dPool")
2) Cache: Configures the Storage Spaces Direct caches (If there is more than one media (drive) type available for Storage Spaces Direct use, it enables the fastest as cache devices (read and write in most cases)
3) Tiers: Creates 2 tiers as default tiers. One is called "Capacity" and the other called "Performance". The cmdlet analyzes the devices and configures each tier with the mix of device types and resiliency.
I have not been able to locate a method to run the above tasks manually in three different steps. So:
1) Pool: To control which disks are to be used for S2D, I partition and format the ones I do not want to use for S2D
2) Cache: To prevent the "Enable-ClusterS2D" command from creating a cache, it is possible to use the -CacheMode Disabled parameter.
3) Tiers: In my experience, if you don't have at least 2 Solid State Hard Drives per cluster node, the "Enable-ClusterS2D" command will not create the "Performance" tier. Only the "Capacity" tier will be created (If you have 1 SSD per server, these SSD's will also be added to the "Capacity"" tier along with regular spindles hard drives).

-Verbose

If you add the -Verbose parameter at the end of the Enable-ClusterS2D cmdlet, you can examine the individual steps that this cmdlet executes before Storage Spaces Direct is completely enabled. It's worth noting the step where disks types are detected and sorted

Things to note at this point

Enable-ClusterS2D Report

The output of the Enable-ClusterS2D command will notify of the creation of a report after the enable is complete.

This report (located in the same directory as cluster validation reports) will specify the disks that were claimed by S2D and which disks were automatically selected for cache (more on this below):

Cluster Pools

Notice that the "Enable-ClusterS2D" command automatically created a pool and presented it to the cluster. This pool is automatically named "Cluster Pool 1". This name is one of the things I would have liked to have control over when enabling Storage Spaces Direct instead of just trusting the "Enable-ClusterS2D" command. But I can rename the pool later, so I won't cry out loud.

A physical disk number is assigned to each disk. Cluster Manager also displays the pool details (size, status and individual disks)

This pool is also visible in Server Manager for all nodes:

Cache

Notice that two of the disks were automatically selected for cache (Journal). The cache disks are the only two disks that will show 100% used space at this stage of starting to work with S2D

Tiers

Run the Get-StorageTier command in any cluster member and notice a few things after enabling S2D:

1) The Capacity tier was created automatically.
2) The Performance tier was created automatically (remember you need two SSD devices per server for this tier to be created. If you don't have two SSD's' per server, this tier won't be created automatically)
3) The Resiliency Setting for the "Capacity" tier is set to Parity. More information on this later below.
4) The Resiliency Setting for the "Performance" tier is set to Mirror. Again, more information on this later below.
5) The Capacity Tier is using the HDD media type disks (regular spindles drives)
6) The Performance Tier is using the SSD media type disks (flash media)
7) The PhysicalDiskRedundancy setting of the Performance tier is 2, which means that the resiliency mode configured automatically was a Three-Copy-Mirror which is the default when enabling S2D on a 4-node cluster (if this PhysicalDiskRedundancy number was 1, then it would mean that a two-copy mirror was in use which would be the case if working with less than 4 servers in the cluster.)
8) The PhysicalDiskRedundancy setting of the Capacity tier is 2, which means that the resiliency mode configured automatically was a Dual Parity (Reed-Solomon code / RAID 6 theory). 4 servers is the minimum required to enable parity resiliency. More details and diagrams below.

Using the Get-StorageTierSupportedSize cmdlet, We can see the two tiers that have been created and the sizes for each:

Get-StorageTierSupportedSize -FriendlyName Performance,Capacity | ft -AutoSize @{N="TierSizeMin (GB)";E={ "{0:N0}" -f ($_.TierSizeMin/1024/1024/1024) }},@{N="TierSizeMax (GB)";E={ "{0:N0}" -f ($_.TierSizeMax/1024/1024/1024) }}

Create Virtual Disks (Storage Spaces) on the S2D Pool

Use the GUI or PowerShell to create virtual disks on your newly created Pool

GUI

In Failover Cluster Manager, right click the S2D Pool and click on "New Virtual Disk"

Click the S2D pool and then OK

Click Next

Enter a name and description. Click Next

Notice a few things before clicking Next:

1) The available free space in the storage pool is the addition of the "Performance" tier and the "Capacity" tier.
2) The available free space on the Performance tier is 33.3% of the total usable capacity of all Solid State Drives because Mirror resiliency generates 3 block copies of the same data and stores them on drives in different servers to provide resiliency in the event of a single disk failure or a single server failure. (This is the default behavior when using 4 servers). See below for understanding this graphically in this environment and consider the disks sizes table from above:

3) The available free space on the "Capacity" tier is 50% of the total capacity of all the spindles disks because the Capacity tier is using dual Parity resiliency and because we are using 4 servers for this proof-of-concept environment. See figure below for a complete picture of this environment and the explanation for the available capacity to create virtual disks:

4) The Resiliency of the Performance tier is 3-copy mirror.
5) The Resiliency of the Capacity tier is Dual Parity.

Click Create:

Make sure the "Create Volume..."checkbox is selected and Click Close

In the New Volume wizard, Click Next at the Welcome screen. At the Select Server and disk screen, Ensure the newly created Cluster Storage Space is selected and click Next

Notice the fact that you cant change the size of the virtual disk due to its tiered nature. Click Next

We will be using this Virtual Disk as a Cluster Shared Volume, so there is no need to assign a letter to it. Click Next

Assign a label to the volume. Notice that you can't change the file system. ReFS is required because of its awareness of hybrid (part mirror, part parity) volumes. Click Next

Click Create and let it do its magic. When done, click Close.

In Failover Cluster Manager, under Storage > Disks, there is now the virtual disk that was just created represented along with some of its worth noting settings

Using the Get-Disk cmdlet shows that this newly created virtual disk is online and accessible to one of the cluster nodes. You can also check this in Disk Management

Get-Disk -FriendlyName myVeryFirstS2DVirtualDisk | fl FriendlyName, DiskNumber, PartitionStyle, OperationalStatus, HealthStatus, BusType, NumberOfPartitions, @{N="Size (GB)";E={ "{0:N0}" -f ($_.Size/1024/1024/1024) }}, @{N="Allocated Size (GB)";E={ "{0:N0}" -f ($_.AllocatedSize/1024/1024/1024) }}

Assign the disk to Cluster Shared Volumes

In Failover Cluster Manager, right click this newly created virtual disk and configure it as Cluster Shared Volume (we plan to hosts VMs here right?)

PowerShell

You can use the New-Volume cmdlet to create a virtual disk (storage space) on the S2D pool instead of the GUI method illustrated above. I will do this now to create a 550GB hybrid virtual disk that has 50GB in the Performance tier and 500 in the Capacity tier.

This command also automatically adds the new volume to Cluster Shared Volumes.

Create Virtual Machines

So there you have it. The cluster and its wonderful storage are ready to start hosting Virtual Machines. Let's start icing this yummy cake.
Create the VM's manually or use VMFleet (link below) or use a script like the one below:
Downnload VMFleet: https://github.com/Microsoft/diskspd

New-VM -Name ContosoSRV1 -MemoryStartupBytes 1GB -Generation 1 -NewVHDPath C:\ClusterStorage\Volume2\ContosoSRV1\VHDs\Disk1.vhdx -NewVHDSizeBytes 20GB -Path C:\ClusterStorage\Volume2\ContosoSRV1\VHDs\ New-VM -Name ContosoSRV2 -MemoryStartupBytes 1GB -Generation 1 -NewVHDPath C:\ClusterStorage\Volume2\ContosoSRV2\VHDs\Disk1.vhdx -NewVHDSizeBytes 20GB -Path C:\ClusterStorage\Volume2\ContosoSRV2\VHDs\ New-VM -Name ContosoSRV3 -MemoryStartupBytes 1GB -Generation 1 -NewVHDPath C:\ClusterStorage\Volume2\ContosoSRV3\VHDs\Disk1.vhdx -NewVHDSizeBytes 20GB -Path C:\ClusterStorage\Volume2\ContosoSRV3\VHDs\ New-VM -Name ContosoSRV4 -MemoryStartupBytes 1GB -Generation 1 -NewVHDPath C:\ClusterStorage\Volume2\ContosoSRV4\VHDs\Disk1.vhdx -NewVHDSizeBytes 20GB -Path C:\ClusterStorage\Volume2\ContosoSRV4\VHDs\ New-VM -Name ContosoSRV5 -MemoryStartupBytes 1GB -Generation 1 -NewVHDPath C:\ClusterStorage\Volume2\ContosoSRV5\VHDs\Disk1.vhdx -NewVHDSizeBytes 20GB -Path C:\ClusterStorage\Volume2\ContosoSRV5\VHDs\

Install the OS on your VMs if applicable. Distribute them to taste across the cluster nodes and power them ON. Now would be a good time to configure more mundane settings such as VMQ, Live Migration network and Jumbo Frames.

Examine Cluster-S2D behavior

Live Migration

Notice that your virtual machines can be live migrated amongst cluster nodes. A SAN is not needed, a Storage migration is not needed. The VM's always stay on the same simple SAS disks directly attached to the servers. What can you possibly not like about this ?

Tiers

Remember the Virtual Disk on which these virtual machines sit is a hybrid volume (part Mirror on the Performance tier and part Parity on the Capacity tier). Let's examine usage of these two tiers.
In Performance Monitor, there is now a "Storage Spaces Tier" Performance counter. Each Virtual Disk you create using the New-Volume cmdlet or GUI will be represented as an object and each tier that is created on these virtual disks will be represented by a separate instance. In this case, Windows automatically created for us a "Performance" tier and a "Capacity" tier. Both tiers are monitor-able in Performance Monitor as seen below. You can monitor queue length, disk transfers (iops), etc as with a regular disk.

You may infer that the Performance tier is used first whereas the Capacity tier only begins to be used as the faster tier's available space is exhausted and data is rotated.

Cache

The cache is also monitor-able via performance monitor. Have fun looking at the values for the objects monitored under the "Storage Spaces Write Cache" performance counter.

So at this point, we have a 4-node Hyper-V cluster that uses the local disks on each server for storage of VMs. It works great. Thanks for reading and stay tuned for further posts.

home | email me | 2016