Stephen,
Just a thought and i believe I will attempt some testing in the near future to see if I am right. If you are using a Supermicro server, you most likely have some sort of SAS backplane (sata cables from motherboard probably attach to the backplane). Could the issue be with the backplane? Although the 2nd chipset of your motherboard has a single cable that plugs into the backplane, the first should have regular sata connecions. Have you tried bypassing the backplane? With my setup I am going to attempt to bypass the 2 sata 6gbs ports (0 and 1) and plug the directly into 2 ssds that I have for the OS. I am able to reliably corrupt the raid 1 mirror by running Burn In Test (Passmark) on 60-70 percent testing for a period of 3 days.

The two drive bays on the front of your server are behind this backplane
BPN-SAS-813LT-O-P

Oh wow,
So you have got corruption using normal HDDs, not even SSDs now?
Let me know the results of your backplane test and if you do not get a failure without the backplane then I will repeat your test as proof.

Hi Billy,
Could you also detail the server model and all RSTe driver and firmware versions, so we can see what is common between our systems?
Best regards
Steve

No No No,
Sorry for any confusion, Hdds seem to be okay. Only raid 1 using ssds seems to have a problem. The 4 hdd volume that has been in the system since I built it has never had a problem. On my spare server I am currently testing 2 hdds in place of the 2 ssds in a raid 1 volume for the OS. I currently have it running a 72 hour stress test. Upon completion I will run a parity check and I will report the results back.

Hi Stephen,
What is the same about your system and mine is that they are both Supermicro boards based on the Intel C602 chipset for sata. My board and system is a bit higher end than yours as mine is a 4u and yours is a 1u, also mine has twice the memory slots (supporting up to 1tb). What is common (and where I believe the problem probably lies) is that both are Intel Mboards based on C602 chipset that connect to drives after a backplane of some sort.
I have reported the issue to Supermicro and their recommendation was to try bypassing the backplane and connect directly to the sata drives to rule out a backplane incompatibility. On your system (hopefully it is not already installed in a rack yet), remove the top cover and connect 2 sata cables directly to ports 0 and 1 (the white sata ports) and run them to the 2 ssds without using the front drive bays. Once you have done that, try testing the system again, if it still becomes corrupt, we know that the problem lies in the chipset itself or the drivers, if the problem is resolved, it's an incompatibility with the backplane.
I can reliably cause the mirror to become corrupt by running a software called Burn In Test from Passmark. If I set it to 65-70 % (remember to set the test configuration to include all the partitions on the ssd raid volume) for 72 hours, it will read thousands of parity errors on a verify and repair from IRSTe.
I will also be trying this sometime in the next few weeks, it's just that my server is operational and this is the business's busiest time of the year, I will probably need to try this after the first.

Just wanted to report back,
First testing results back today.

2 WD HDDs in raid 1 stress tested for 72 hrs with Passmark Burn In Test, Testing showed no disk errors with literally millions of writes

Hdds are installed in Backplane Cartridges

Follow up Intel Rapid Storage Tech Enterprise software drivers version 4.1.0.1047, Parity verification and repair showed no errors



Volume OS_Volume: Verification and repair complete.

System Report

System Information
OS name: Microsoft Windows Small Business Server 2011 Essentials
OS version: 6.1.7601 Service Pack 1 7601
System name: BOXOFFICE
System manufacturer: Supermicro
System model: X9DAi
Processor: GenuineIntel Intel64 Family 6 Model 62 Stepping 4 2.601 GHz
Processor: GenuineIntel Intel64 Family 6 Model 62 Stepping 4 2.601 GHz
BIOS: American Megatrends Inc., 3.0a

Intel® Rapid Storage Technology enterprise Information
User interface version: 4.1.0.1046
Language: English (United States)
Intel controller: SATA (AHCI)
Number of SATA ports: 6
Intel controller: SAS
Number of phys: 4
RAID option ROM version: 3.8.0.1029
Driver version: 4.1.0.1046
ISDI version: 4.1.0.1046

Storage System Information
RAID Configuration

Array Name: SATA_Array_0000
Size: 1,907,737 MB
Available space: 95,383 MB
Number of volumes: 1
Volume member: OS_Volume
Number of array disks: 2
Array disk: WD-WCAW37390407
Array disk: WD-WCAW37357488
Disk data cache: Enabled

Array Name: SAS_Array_0000
Size: 3,815,474 MB
Available space: 190,725 MB
Number of volumes: 1
Volume member: PVM_Volume
Number of array disks: 4
Array disk: WD-WCAW37330401
Array disk: WD-WCAW37377253
Array disk: WD-WCAW37351879
Array disk: WD-WCAW37354503
Disk data cache: Enabled

Volume name: OS_Volume
Status: Normal
Type: RAID 1
Size: 906,176 MB
System volume: Yes
Data stripe size: 64 KB
Write-back cache: Disabled
Initialized: Yes
Parity errors: 0
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Volume name: PVM_Volume
Status: Normal
Type: RAID 10
Size: 1,812,374 MB
System volume: No
Data stripe size: 64 KB
Write-back cache: Disabled
Initialized: Yes
Parity errors: 0
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Hardware Information

Controller name: Intel(R) C600+/C220+ series chipset SATA RAID Controller
Type: SATA
Mode: RAID
Number of volumes: 1
Volume: OS_Volume
Number of spares: 0
Number of available disks: 1
Rebuild on Hot Insert: Enabled
Manufacturer: 8086
Model number: 2826
Product revision: 6
Direct attached disk: WD-WCAW37390407
Direct attached disk: WD-WCAW37357488
Direct attached disk: WD-WMC1P0306600

Controller name: Intel(R) C600 series chipset SAS RAID (SATA mode) Controller
Type: SAS
Mode: RAID
Number of enclosures: 0
Number of volumes: 1
Volume: PVM_Volume
Number of spares: 0
Number of available disks: 0
Read patrol: Disabled
Rebuild on Hot Insert: Disabled
Manufacturer: 8086
Model number: 1D6B
Product revision: 6
Direct attached disk: WD-WCAW37330401
Direct attached disk: WD-WCAW37377253
Direct attached disk: WD-WCAW37351879
Direct attached disk: WD-WCAW37354503

Disk on Controller 0, Port 0
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-010FB0
Serial number: WD-WCAW37390407
SCSI device ID: 0
Firmware: 01.01V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 1
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-010FB0
Serial number: WD-WCAW37357488
SCSI device ID: 1
Firmware: 01.01V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 2
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Available
Size: 1,863 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD2000FYYZ-01UL1B1
Serial number: WD-WMC1P0306600
SCSI device ID: 2
Firmware: 01.01K02
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 0
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37330401
SCSI device ID: 0
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 1
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37377253
SCSI device ID: 1
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 2
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37351879
SCSI device ID: 2
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 3
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37354503
SCSI device ID: 3
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Empty port
Port: 3
Controller SATA (AHCI)
Port location: Internal

Empty port
Port: 5
Controller SATA (AHCI)
Port location: Internal

I will try a second test just to confirm but it seems that hdds do not suffer from this bug

We are testing with a different type of ssd. The tests are running now.
Will keep you posted.
Best regards
Steve

As per today, Supermicro is still offering the 4 series drivers for use with their c602 chipset motherboards

Description: Intel PCH Driver(SATA)
Version: 4.1.0.1047
Link: Download
Description: Intel PCH Driver(SCU)
Version: 4.1.0.1047
Link: Download

And as I was at the server earlier tonight and ran a verification on the Main Production Server that has been running with a heavy load since early Saturday Morning, here is the results of the check




Volume OS_Volume: Verification and repair complete.

System Report

System Information
OS name: Microsoft Windows Small Business Server 2011 Essentials
OS version: 6.1.7601 Service Pack 1 7601
System name: BOXOFFICE
System manufacturer: Supermicro
System model: X9DAi
Processor: GenuineIntel Intel64 Family 6 Model 62 Stepping 4 2.601 GHz
Processor: GenuineIntel Intel64 Family 6 Model 62 Stepping 4 2.601 GHz
BIOS: American Megatrends Inc., 3.0

Intel® Rapid Storage Technology enterprise Information
User interface version: 4.1.0.1046
Language: English (United States)
Intel controller: SATA (AHCI)
Number of SATA ports: 6
Intel controller: SAS
Number of phys: 4
RAID option ROM version: 3.7.0.1049
Driver version: 4.1.0.1046
ISDI version: 4.1.0.1046

Storage System Information
RAID Configuration

Array Name: SATA_Array_0000
Size: 1,907,737 MB
Available space: 95,468 MB
Number of volumes: 1
Volume member: OS_Volume
Number of array disks: 2
Array disk: WD-WCAW37392705
Array disk: WD-WCAW37390907
Disk data cache: Enabled

Array Name: SAS_Array_0000
Size: 3,815,474 MB
Available space: 190,725 MB
Number of volumes: 1
Volume member: PVM_Volume
Number of array disks: 4
Array disk: WD-WCAW37333570
Array disk: WD-WCAW37352992
Array disk: WD-WCAW37362600
Array disk: WD-WCAW37341244
Disk data cache: Enabled

Volume name: OS_Volume
Status: Normal
Type: RAID 1
Size: 906,134 MB
System volume: Yes
Data stripe size: 64 KB
Write-back cache: Disabled
Initialized: Yes
Parity errors: 0
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Volume name: PVM_Volume
Status: Normal
Type: RAID 10
Size: 1,812,374 MB
System volume: No
Data stripe size: 64 KB
Write-back cache: Disabled
Initialized: Yes
Parity errors: 0
Blocks with media errors: 0
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Hardware Information

Controller name: Intel(R) C600+/C220+ series chipset SATA RAID Controller
Type: SATA
Mode: RAID
Number of volumes: 1
Volume: OS_Volume
Number of spares: 0
Number of available disks: 1
Rebuild on Hot Insert: Enabled
Manufacturer: 8086
Model number: 2826
Product revision: 6
Direct attached disk: WD-WCAW37392705
Direct attached disk: WD-WCAW37390907
Direct attached disk: WD-WMC1P0312819

Controller name: Intel(R) C600 series chipset SAS RAID (SATA mode) Controller
Type: SAS
Mode: RAID
Number of enclosures: 0
Number of volumes: 1
Volume: PVM_Volume
Number of spares: 0
Number of available disks: 0
Read patrol: Disabled
Rebuild on Hot Insert: Disabled
Manufacturer: 8086
Model number: 1D6B
Product revision: 6
Direct attached disk: WD-WCAW37333570
Direct attached disk: WD-WCAW37352992
Direct attached disk: WD-WCAW37362600
Direct attached disk: WD-WCAW37341244

Disk on Controller 0, Port 0
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-010FB0
Serial number: WD-WCAW37392705
SCSI device ID: 0
Firmware: 01.01V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 1
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-010FB0
Serial number: WD-WCAW37390907
SCSI device ID: 1
Firmware: 01.01V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 0, Port 2
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Available
Size: 1,863 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD2000FYYZ-01UL1B1
Serial number: WD-WMC1P0312819
SCSI device ID: 2
Firmware: 01.01K02
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 0
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37333570
SCSI device ID: 0
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 1
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37352992
SCSI device ID: 1
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 2
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37362600
SCSI device ID: 2
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Disk on Controller 1, Phy 3
Status: Normal
Type: SATA disk
Location type: Internal
Usage: Array disk
Size: 932 GB
System disk: No
Disk data cache: Enabled
Command queuing: NCQ
Model: WDC WD1003FBYZ-0
Serial number: WD-WCAW37341244
SCSI device ID: 3
Firmware: 1V03
Physical sector size: 512 Bytes
Logical sector size: 512 Bytes

Empty port
Port: 3
Controller SATA (AHCI)
Port location: Internal

Empty port
Port: 5
Controller SATA (AHCI)
Port location: Internal


Although I believe there are issues with the 4 series drivers for x99 chipsets (and the newer c600 series that are pure sata 6gbs), they seem to be rock solid so far on X79, as long as you aren't using ssds in raid 1.

Just wanted to report on the 4.1.0.1046 drivers. Running two Samsung 1TB Evo's in RAID 0 on my Intel SCU controller with the 4.1.0.1046 driver - no issues at all and TRIM seems to be working just fine. This is on a Supermicro X9DAi motherboard (C602 Chipset).

I will report any issues, but none so far.

I have a couple of machines (X79 chipset) running ssds in raid 0 and seems just fine. I think the issues are with mirroring

Hi Stephen,
I'm just wondering if you ever got anywhere with this issue. I see that on the ssd list from supermicro, the intel dc3500 series are listed as compatible but doesn't seem to mean anything. I was going to try with different ssds but I think that it might be a waste of time

Hi Billy,

We tested with HP/Intel 3700 drives and Intel 3500 drives.

All Intel drivers we were supplied that were newer than the 3.6 version are screwed if you run a two drive SSD mirror. The mirror will corrupt within hours or days. We never had an array that lasted a week.

Our servers our now live and running perfectly with these very old drivers.

We wasted thousands in testing, wasted time and worn down SSDs. Not to mention replacing all the drives with another version, just to prove a point.

We unequivocally proved the presence of the bugs with exhaustive testing. We never once got more of a reply than 'try this version - it might work'. No version we were supplied fixed the fault. No details were supplied with the drivers. No contact with Intel was ever forthcoming.

Frankly, I think the situation is shameful. Bugs are to be expected - that's life. But for Intel to hide behind the OEMs, not talk to the customer and not supply any release notes with their drivers to acknowledge known existing and fixed bugs with their drivers is unacceptable and in my mind criminal. These are server products, not home PC parts that people fiddle with at home for fun. Known bugs should be published, not hidden to maximise both sales and consequently hacked off customers.

I write this in the hope that someone at Intel who actually cares about the quality of their products and upholding a reputation of integrity reads this and thinks that a policy change is in order.