PrimoCache Server in combination with a Intel 900P Optane card

[Axel Mertes]

@support
That would be super cool!
I am currently selecting components for the new server and any hints I can get, the better...

[jussssx1]

Hello.. i have tested 900p extensively with dual xeon configuration (and primocache 3.0.2). Primocache significantly speeds certain DB operations even with optane (4k-64kb transfers with higher queue depth). There are couple of issues though: A) it seems that there is a windows glitch that (only happens with dual socket systems) that cause racing condition on file io. The end result is that after prolonged period of usage, windows will report IO errors from volumes cached by Supercache. I have not witnessed this with any other system than with dual socket systems.. and the issue goes away if second socket is disabled. The second issue is related to spectre/meltdown patch which causes OS level slowdown for all low-latency IOs. This slowdown makes even primocache/900p volumes perform as slow as vintage SSDs. Our solution has been this far avoidance of spectre/meltdown patch and waiting for microsoft to fix dual socket racing IO issue.

[Axel Mertes]

There are couple of issues though:

A) it seems that there is a windows glitch that (only happens with dual socket systems) that cause racing condition on file io. The end result is that after prolonged period of usage, windows will report IO errors from volumes cached by Supercache. I have not witnessed this with any other system than with dual socket systems.. and the issue goes away if second socket is disabled.

The second issue is related to spectre/meltdown patch which causes OS level slowdown for all low-latency IOs. This slowdown makes even primocache/900p volumes perform as slow as vintage SSDs.

Our solution has been this far avoidance of spectre/meltdown patch and waiting for microsoft to fix dual socket racing IO issue.

Mmh, is that race condition existant already in 2.4.0 of the server edition?
I am running PrimoCache Server 2.4.0 since years now and it performs fluently. For primary volumes I only use read caching right now, while for mirror drives we use read/write caching. The largest volume is 48 TByte in one piece. Due to enabling PrimoCache we can defragment all volumes daily, so we rarely see more than 1-2 fragmented files at all, making read operations perform at maximum speed possible.

That race condition you mention makes me nervous. Do you use PrimoCache Server?

I wonder how to figure out if that happens on our system too.

A solution might be to build a single CPU server, but that might limit the PCIe lanes we might need/want.

We have experienced that sometimes folders don't show up, which have been created seconds before. Or you can still see files that are already deleted. Unmounting the network share and remounting it usually fixes the problem. I think we have seen that before without PrimoCache IIRC. So I never blamed that to the caching. However, issues like that are sometimes very tricky to track down.

The spectre / meltdown patch issue is a bummer. So we need to decide if the data we have there is at risk, I understand that the main problem with the spectre / meltdown problem is that a third party might gain access to data it should not see, especially in systems running multiple VMs. Most of the data I have there is advertising images (sooner or later they are public) and we don't run any VMs on the file server. So in an isolated view this system should be safe as long as we have no virus / malware running on it (which is something we of course take care of, as much as we can).

Things to consider...

On the other hand, no other tech will get faster than the P900 I would assume. So we should trust to let Intel & Microsoft take care of making these devices the beasts these are meant to be.

Can you share any benchmarks like ATTO DiskBench or Blackmagic Design DiskSpeed on the single CPU system along without the Spectre / Meltdown fix? That'll be super cool to see/have/know.

Cheers,
Axel

[Support]

We finally published the testing results on SSD 900P and PrimoCache. Please see
viewtopic.php?f=33&t=4531
Thanks.

[jussssx1]

wow!! nice speeds & high quality article! Just one question: did you have meltdown & spectre patches applied?

[Support]

Tests in the article were made on Server 2012. We also have the test results on Server 2016 which are similar to that on server 2012.
Both systems have not applied the meltdown & spectre patches.

[jussssx1]

ok, thank you for the information! It would be nice to see some comparison results with full spectre&meltdown patches applied. Those seem to take huge toll on 4k IO

[luckz]

How does one get authorized to view the test result thread?

[minhgi]

Seem Linus Tech Tips did a minor bundle review of Primocache and a 32GB optane memory at the end of the video. Looks great when competing with Intel and AMD solution. I definitely think @support should reach out to LinusTechTips do a full review to get more exposure.

https://www.youtube.com/watch?v=rWXBo0bb_dU&t=8s

[Support]

@minhgi, thank you for the information!