Benchmarking PrimoCache

[vintagedon]

As a Cloud Administrator, one of the things I try to do is inch the most performance out of any iSCSI or SAN mount that I have. In that quest, I ran across FancyCache about nine months ago, and started using it my ESXi lab, caching the RAID volume that I would use for my iSCSI mounts. I saw drastic performance increases in the lab, but had never taken the time to benchmark those results.

Having built a new SAN this week for a secondary project, I benchmarked the SAN with and without PrimoCache activated (allowing PrimoCache 16GB of cache). I thought those here might be interested in the results, have some input, and ideas for my next round of testing. You can find the benchmark here:

http://thehomeserverblog.com/esxi/esxi- ... ram-cache/

As Always,
VintageDon

[InquiringMind]

Vintagedon,

Good to see a well-written article on PrimoCache (though I would suggest any technique involving lots of RAM could hardly be called "cheap"). Your suggestion for cache sizing might benefit from more clarification (do you mean 80% of free memory or 80% of total memory?) and I would suggest finding peak memory usage first (running the most demanding applications and seeing the peak usage reported by Task Manager) and then using, say, 80% of remaining memory.

"Unmanaged memory" (PrimoCache's ability to use memory above the 3.25GB limit on 32-bit systems and outside the licenced limits on 64-bit systems - e.g. 16GB on Win7 Home Premium) is a feature well worth a mention, even though it doesn't apply in your case.

Finally, the performance figures you obtain with PrimoCache seem a little low. On my system (WinXP SP3, Core i7 930 overclocked to 3.34GHz, 18GB RAM, 6xSamsung SSDs in RAID-0) the uncached SSD RAID performance reported by CrystalDiskMark is:

[InquiringMind]

Given that 6 (128GB) SSDs are involved, the results there aren't too great (using the motherboard's Intel ICH10 controller - the drives individually can do around 230MB/s sustained read/write but the controller can only handle 3 before performance starts to degrade).

Add PrimoCache to that array though (7GB allocated using unmanaged memory, 64KB blocksize, read/write with write deferring, LFU-R) and I get:

[InquiringMind]

The sequential results I see look to be almost 4 times what your results report, which seems a little odd. I presume I have more memory bandwidth available (1600Mhz DDR3, triple channel) but that shouldn't account for a more than 50% improvement. Maybe 64-bit system overheads are an issue here?

[vintagedon]

IM:

Thanks, and I'll clarify that in the article. My suggestion was 80% of free RAM after determining a baseline for operating load. The install that I have currently is a fresh Server 2008 x64 installation, so I'm actually using 80% of my total RAM, but in any environment, I'd recommend 80% of free memory.

The performance difference is interesting. Would you mind sharing a memory bandwidth benchmark with me for your system, and I'll share with whatever program you pick to do so (so we can get a more apples-to-apples comparison)? I think it bears more investigation, and I'd love to see if it's just the triple channel, x64 overhead, or something else.

We're running the same speed of memory, so the only difference seems to be the CPU and, of course, the triple channel, which like you, I wouldn't seem to think would cause such a disparity in the results. I do have a triple channel system sitting around (although I normally design my SANs/ESXi labs around AMD architecture simply for price/performance dollar values). Although I didn't note it in the article, I did track CPU usage, and didn't see appreciable usage except during random IOMeter tests, when it was really punishing the system.

[minhgi]

nice write up.

[InquiringMind]

Vintagedon,

Spent some time with vintage versions of Sandra comparing their memory benchmark results, before getting the latest free/lite version (which sadly, seems to have gone downhill IMHO) from http://www.sisoftware.net//?d=dload&f=s ... _3264&l=en - results given below.

Another suggestion - how about a test comparing PrimoCache with Windows' own file cache? Since virtually all disk benchmarks bypass the Windows cache, the best bet may be to write your own script (with AutoHotKey perhaps?) timing repeated file loads. That sort of comparison would probably be the most relevant for those considering third party caching software.

[Support]

@Vintagedon, @InquiringMind,

I think the performance difference is probably caused by different Defer-Write latency in your cache settings. If PrimoCache happens to flush deferred write data to the disk during the testing, obviously the test results will be low.

When PrimoCache has enough cache space and all read/write requests are done in the cache without accessing the underlying disk, the test results will be the best.

And thank Vintagedon very much for the article!

[vintagedon]

InquiringMind:

Thanks, and thanks to support for the possible recommendation on write deferring. I'll get the bandwidth tests done today. IM, can you verify what your defer settings are? For the tests, I left mine at the standard 10 seconds. I'll try running some tests again tonight with the refer higher, as I have plenty of actual memory.

[InquiringMind]

When I tested, I left the write defer setting to the default (10 seconds). I did increase it to 20 seconds for subsequent use, but this has had a marginal effect on benchmark results, shown below (I also tried with 1,000 seconds and that gave slightly poorer results). So I don't think write defer settings are the cause.