@MDM
Ok, thanks... i will try again later, then will have enough time... if i don't forget

Is there Intel NVMe v6.2.0.1234 driver now available to replace v5.5.0.1360 ?
Dumo application do suggest it but I don't see him somewhere yet.

I picked up several Toshiba XG6 NVMe drives at a good price. These are OEM drives. I'm using them on Supermicro AOC-SLG3-2M2 PCIe adapters that each hold 2 M.2 drives.

My motherboard is a Supermicro X10-SRL-F that supports bifurcation. While benchmarks for a single drive gives the expected performance, running 2,4 or 8 of these XG6 in a striped volume, the performance is no where near 2,4 or 8 times that of a single unit.

So I'm thinking it is a driver issue. I'm running Windows 10 Pro and the Standard NVM Express Controller is dated 20-Jun-2006 (version 10.0.18362.693).

I downloaded the OCX/Toshiba NVMe driver from the first page in this thread (v1.2.126.843 WHQL), but when I attempt to update the drivers from Device Manager, it tells that that the best driver is already installed.

Any suggestions on how I might be able to update my ancient drivers to a newer version?

Thanks!

@crazydane: To follow what @Fernando told, you would need a dedicated RAID Card for that...

@Fernando

First a little more background. The NVMe units I'm using are all PCIe Gen 3.0 4x units, so each one has 4 PCIe 3.0 lanes directly to the CPU (E5-2683 v3 14 core). I was able to accomplish this by enabling bifurcation in the BIOS, which basically sets ports 2 and 3 on the CPU to use x4x4x4x4 instead of x8x8 or x16. So the chipset is not involved at all as I have a directly 4x PCIe path to the CPU for each M.2 device.

Each NVMe has a dedicated entry in device manager under Storage controllers. The hardware Ids are the same for all 4 controllers, and they are:

PCI\VEN_1179&DEV_011A&SUBSYS_00011179&REV_00
PCI\VEN_1179&DEV_011A&SUBSYS_00011179
PCI\VEN_1179&DEV_011A&CC_010802
PCI\VEN_1179&DEV_011A&CC_0108

If I look in HWiNFO, I see the following for each NVMe unit:

PCIe x4 Bus #3 - Toshiba AIS, Device ID: 011A
PCIe x4 Bus #4 - Toshiba AIS, Device ID: 011A
PCIe x4 Bus #5 - Toshiba AIS, Device ID: 011A
PCIe x4 Bus #6 - Toshiba AIS, Device ID: 011A

HWiNFO also confirms that each NVMe controller has a 4x link width with a bandwidth of 8.0 GT/s each (PCIe 3.0).

I created my test arrays simply by setting up a striped volume using Disk Manager with 2, 4 or 8 NVMe devices.

The dual NVMe to PCIe adapters I'm using do have an option to change the SMBus address for each card. So let me try setting them all to different addresses to see if that will address the issue.

ASRock makes a Quad M.2 Card 16-lane NVMe to PCIe adapter that based on reviews I have been, it is definitely possible to get close to 4x the performance compared to a single NVMe. One review I saw, was getting reads of 12,031 MB/s, which is 4x the performance of a single NVMe. So far the best I've been able to get is 6,3543 MB/s on reads and 4,853 on writes, so I'm leaving a lot on the table and I'm trying to determine root cause.

I'll try changing the hardware id (SMBus address?) on one of the adapter cards to see what happens and report back.

Zitat von crazydane im Beitrag #1838

---

I created my test arrays simply by setting up a striped volume using Disk Manager with 2, 4 or 8 NVMe devices.

---

But in a striped array (RAID0) there is no parity calculation, so I don't believe it would put any load on the CPU at all. If I was running RAID5 or 6, I would agree that a dedicated RAID card would be the way to go.
I in fact do that for my media server, where I run an Areca 1882 card connected to a SAS2 backplane expander with 24 drives on it configured as a RAID60 (2x12 RAID6 arrays striped together).

Based on reviews I have read, a windows striped volume is just as fast as a RAID0 setup in BIOS using Intel's RSTe.

This is all just for testing and getting any hardware issues I have sorted out. My goal is to deploy several of these as nodes in a Storage Spaces Direct cluster that will be all NVMe.

Wrong, is has to split that data onto all of the disks, and reverse for reeding no? So there is at least a bit of work...
I am not sure, but bifurcation do not work on all slots, and maybe just on one? And you have several cards...

I agree that there is definitely some overhead for splitting the data for writing and reading, but I don't think it would be significant, but I could certainly be wrong.

My motherboard has 4 slots with dedicated 8x PCIe lanes directly to the CPU, for a total of 32 lanes. 2 slots share one 16 channel port, and the other 2 slots share the 2nd 16 channel port. Bifurcation has been enabled on both 16 channel ports, feeding the 4 slots. The remaining 8 lanes are split between 2 more slots and various other I/O functions of the mobo.

I posted up more details over on another forum. Not sure if I'm allowed to link to it, but here goes:

https://hardforum.com/threads/pcie-bifur...post-1044652089

It shows the Crystal benchmarks I'm getting for the various combinations, the block diagram of my mobo, bifurcation settings, etc.

This one really got me stumped...

I stand corrected. I side graded from a E5-2683 v3 (14 cores @ 2Ghz) to a E5-2680 v3 (12 cores @ 2.5GHz) and I'm now seeing 10GB/s writes and just shy of 7GB/s on writes. Still odd that CPU utilization during benchmarks barely registered anything. Anyway, sorry for going off topic since this clearly is not driver related.

Told you, it is CPU bound when software RAID...
And it is normal for low utilization, that is not the issue here; but explaining would be quite long a heavily off-topic ;)

One final note I'll make is that when I'm running the benchmarks (be it ATTO or Crystal) those are single threaded apps and on my 12 core/24 thread CPU, only a single thread is maxed out with the remaining 23 threads are idle. So that explains why I was only seeing 4% utilization when looking at the overall load. I suspect the crux of the issue is that synthetic benchmarks like these are single threaded in nature and are not able to utilize multiple threads. The good news is that these servers, as nodes in a storage spaces direct cluster, will not exhibit this single threaded restriction VMs are accessing the data store that will live on the NVMe storage.