Accelerating PrimoRAMDisk with PrimoCache by 100%!

[Axel Mertes]

Hi Romex Software Team!

I have refreshed my server recently and played a bit with caching a storage space consisting of 4 * RAID6 (each 8 * 2 TByte drives), each connected via 4 GBit Fibre Channel. I get around 1500 MByte/s writes and 1410 MByte/s reads on it natively, which is very close to the physical limit of 4 * 4 GBit Fibre Channel (thereoretically 1600 MByte/s, reality is more like 1520 MByte/s).

Now I activated PrimoCache read and write with a 300 second defer write setting enabled (honestly risky, but while testing I don't care). The I/Os increased over 400% with PrimoCache. The transfer values increased to 8000 to 10000 MByte/s when using PrimoCache, stable for read and write.

I used ATTO Disk Benchmark 3.05 for testing, default settings. Increasing target size from 256 MByte to 2 GByte or Queue Depth from 4 to 10 only increases overall transfer speed even further, with peaks up to 15000 MByte/s. I employ a fairly old server mainboard with dual hexacore 2.67 GHz X5650 Intel Xeon CPUs and 144 GByte DDR3 RAM 533 MHz. If we compare that to an overclocked 3200 MHz DDR4 we clearly see differences... But who cares, that all is very well beyond the 40 GBit ethernet connection the server has...

What really wondered me was that with PrimoRAMDisk, formatted as Direct IO I get only like 3800 MByte/s writes and 4300 MByte/s reads (maximum), formatted as SCSI disk emulation I get like 3700 MByte/s writes and 4000 MByte/s reads (though starting at only about 30% I/Os with small block sizes, compared to Direct IO). But then I (playfully) enabled PrimoCache for the PrimoRAMDisk (SCSI emulation) and got an amazing increase in I/Os of around 80-100% and peak performance of 9600 MByte/s writes and 8300 MByte/s reads - thats about twice as fast as PrimoRAMDisk alone...!!!

But in turn, the normal disk drive is actually as fast with PrimoCache as the RAM Disk gets with PrimoCache. And in any case, PrimoCache is about twice as fast as the RAM Disk alone. The only small difference is that the Direct IO RAM Disk has a bit higher I/Os for small block sizes, while loosing dramatially with the bigger block sizes. In fact, PrimoCache REPLACES PrimoRAMDisk completely IMHO. Instead of having a usually fairly small RAM Disk with a corresponding disk file for recovery, you can simply use a drive/partition and use PrimoCache instead. It gives you better performance and the ability to scale well beyond your RAM size if needed. If you want it to be RAM only, have a small disk or small partition...

Honestly I would have expected the both apps to be similar in speed, with the RAM Disk potentially being faster as to not needing to worry about looking up cached blocks etc. However, it turns out the opposite is true. In my opinion PrimoCache completely replaces PrimoRAMDisk.

What do you think?

Can you see similar results?

[Axel Mertes]

ATTO305 PrimoRAMDIsk DirectIO

ATTO305 PrimoRAMDIsk DirectIO.jpg (96.28 KiB) Viewed 7908 times

ATTO305 PrimoRAMDIsk SCSI

ATTO305 PrimoRAMDIsk SCSI.jpg (93.76 KiB) Viewed 7908 times

ATTO305 PrimoRAMDIsk SCSI with PrimoCache Read Write Defer 300s

ATTO305 PrimoRAMDIsk SCSI with PrimoCache Read Write Defer 300s.jpg (90.47 KiB) Viewed 7908 times

[Axel Mertes]

ATTO305 Quad 4 GBit FC RAID6 Storage Space

ATTO305 Quad 4 GBit FC RAID6 Storage Space.jpg (92.63 KiB) Viewed 7905 times

ATTO305 Quad 4 GBit FC RAID6 Storage Space with PrimoCache Read Write Defer 300s

ATTO305 Quad 4 GBit FC RAID6 Storage Space with PrimoCache Read Write Defer 300s.jpg (94.85 KiB) Viewed 7908 times

ATTO305 PrimoRAMDIsk SCSI with PrimoCache Read Write Defer 300s - Queue Depth 10

ATTO305 PrimoRAMDIsk SCSI with PrimoCache Read Write Defer 300s - Queue Depth 10.jpg (92.99 KiB) Viewed 7908 times

[Adz]

I'm using my PrimoRAMDIsk ramdisk SCSI for L2 cache on 215mb is that any good? I have pretty low ram (3gb)
what shall i use for L1 cache? direct ram or...? right i'm using direct RAM for L1 cache

Does increasing write defer latency increase performance? i have mine set to 5 seconds.
Block size 16kb with 315MB L1 cache and 215 RaMDISK L2 cache.

Also what shall i keep for gathering interval for L2 cache? Mine is set as default, should i set it as fastest? i think direct ram is faster then ramdisk right?

Need some tips.

Cheers.

[Axel Mertes]

If you have such low RAM, I think its a total waste to run a SCSI RAM Disk plus PrimoCache at the same time, especially if PrimoCache is aimed to cache the RAM Disk. This is total nonsense in my opinion to cache a RAM Disk using PrimoCache. I know, the whole thread is based on the crazy idea (because you waste twice the expensive RAM, once for the disk, once for the cache), but it was just a "try" from my side with totally unexpected results.

If I were you I'd try the following:

- Disable your RAM disk completely.
- Enable PrimoCache with an affordable size of RAM, say your 315 MB + 215 MB, say 512 MByte (1/2 GByte) as L1 cache.
- Make sure to set the same block size for the cache as you use for the partition. Bigger blocks are faster and need less memory overhead. I currently use 64 KB block size, as this is the default in Windows 10 / 2012 / 2016 for storage spaces. If you have tons of smaller than 64 KB files, then you might better use e.g. 4 KB blocks instead.
- Don't use L2 cache at all in this scenario.
- Cache a small local partition or disk.

If your small partition or disk is the same size as the cache, then the disk will be like a RAM disk, but it might turn out that its now faster...

The good points of this approach are IMHO:
- You have an instant copy of your "RAM Disk" on the partition for which you enable the cache. You will not loose anything (except in a crash whith write cache enabled).
- You can increase the size of that partition and PrimoCache will take care of moving data in/out transparently from the disk partition.
- Its only one program running, so saving memory.

Let us know how it turns out!

However, 3 GByte is really small and the effect of caching isn't very big in your scenario - depending on what you do. If you are writing software and compiling files, then it might turn out to be hell fast. If you are playing games or video, it may not really help. If you use office tools and often open/close apps, you might find it accellerating your operations strongly. It all depends on what you do...

[Adz]

Thanks for the awesome reply. I'll check out my settings.

[Axel Mertes]

To Support:

Are my findings above correct?
Is PrimoCache really faster than PrimoRAMDisk? So fast I can actually increase a PrimoRAMDisk speed by 100% making it twice as fast?

Anyone else tried this / seeing this?

Or did I something wrong in my measurements?

[Support]

@Axel Mertes, could you try other testing tools like AS SSD? I'm afraid that results in the ATTO tool got overflowed.

[Axel Mertes]

I can cross check my results in AS SSD asap.

[Axel Mertes]

These are the results of the Primo RAM Disk SCSI with NO caching enabled:

as-ssd-bench Romex RAMDISK SCSI no caching MBs 19.01.2017 11-35-03.png (29.56 KiB) Viewed 7780 times

as-ssd-bench Romex RAMDISK SCSI no caching IOPS 19.01.2017 11-35-03.png (29.14 KiB) Viewed 7780 times