Samsung 980 Pro IOPS almost halved with Primo L1 cache - what am I doing wrong?

[Dog]

Greetings,

I am interested in using PrimoCache to increase the serial read performance of my NVMe drives (Samsung 980 Pros on PCIe 4). I've tested with 2GB, 4GB, & 8GB L1 RAM caches (4K) and get the same results each time - serial read performance increases dramatically, but IOPS are almost halved. Without Primo I get almost 1 million IOPS, but with Primo it's just over 500K IOPS. I tried increasing the block size all the way up to 64K and saw no increase in IOPS (the 980 is formatted with 4K clusters if I'm not mistaken) but did see a large decrease in serial read performance. My understanding was that an L1 RAM cache should increase IOPS, not decrease them, so the drop in IOPS is definitely not what I expected to see.

The only thing that occured to me, based on what I was seeing, was that the DRAM module on the 980 is almost twice as fast as my system RAM. I looked that up and the DRAM module on the 1TB & 2TB 980 Pro is LPDDR4, rated up to 1 million IOPS; the system RAM is Corsair DDR4 (not low power), rated at 3,600MHz via DOCP with CAS latency of 16 - I didn't find an IOPS rating for the Corsair RAM or a speed or CL rating for the Samsung DRAM module. Considering the 980 is using low power DDR and my system RAM is not low power, I find it difficult to believe that the Samsung DRAM is capable of nearly twice as many IOPS as my system RAM when my system RAM is responsible for so many things beyond storage I/O...many of them 'simultaneously'. Is my intuition regarding this off-base? If not, what am I missing here? Is this due to unrealistic expectations on my part, or lack of knowledge on how to properly configure the L1 cache in Primo?

I've searched the Romex forums for the term IOPS and Googled for various sets of terms centered around the primary terms "primocache", "limited", and "iops" and read pretty much every thread in the first several pages of results, from both sites, that seemed to have even the slightest connection to encountering IOPS limitations with PrimoCache - as well as several others that seemed like they might at least provide some insight. Unfortunately, I was not able to find anything to explain what I'm seeing - lots of stuff about trying to use SATA SSD & M.2 NVMe L2 caching schemes causing IOPS losses, but nothing specifically about encountering this issue using only L1 cache. I even checked the quick start guide...no help there either.

Additional data point: I do not have a UPS and we do occasionally (although rarely) have power outages, so delayed write is not really a good option for me.

Here are the general specs for my box that I *think* apply to this situation:

Ryzen 9 5950x / 64GB (4x16GB) DDR4 @ 3,600MHz using DOCP profile in dual channel mode / X570 based motherboard using AGESA 1.2.0.6b / Samsung 980 Pro 1TB C: & 2TB D: attached via M.2 slots providing 4 lanes of PCIe 4 to each drive.

The CPU is not overclocked - the RAM's DOCP configuration and the video card's factory overclock are the only overclocks on the system.

What am I missing here? I can't imagine there is any kind of hard limit on IOPS coded into PrimoCache (in fact one thread I read even had a statement from support clearly saying that there was no limit), so I'm convinced I'm missing some crucial piece of information that explains the results I'm seeing. Would someone be willing to set me straight here?

:Edited for clarification:

[Support]

Could you upload screenshots of test results of native Samsung 980 and L1 cache? And a screenshot of the cache configuration as guided in the below link?
https://kb.romexsoftware.com/en-us/2-pr ... leshooting
Thank you.

[Dog]

I've included the screenshots you requested and would like to add more information. In reviewing the CDM results, I can now see that I was viewing the results with some obvious bias. IOPS are up for reads in all tests except RND4K/Q32T16 in which they are almost halved. The other test that shows a drop I am curious about is the RND4K/Q1T1 write test - if my quick math is correct, PrimoCache is producing results a just shy of 10% less in write IOPS than the native drive, which seems like an amount that would be more than just random noise.

Thank you for your help.

CDM - native 980 Pro IOPS


CDM - PrimoCache IOPS


PrimoCache - Drive C: only - R&W L1 only

[Support]

Could you check the CPU utilization when running the test for RND4K Q32T16. I wonder that the performance is throttled by the CPU utilization. PrimoCache takes additional CPU time to process IOs. This could have an impact on heavy workload.

[Dog]

Read uses around 22% (dipping to 16% and peaking at 26%).

Write uses around 33% (dipping to 29% and peaking at 36%)

[Support]

We will do further study on this case. It seems that PrimoCache hasn't fully utilize CPUs for RND4K(Q32T16).
Thank you for the feedback!

[Dog]

I eagerly wait to see what you find! Thank you for your help.