Accelerating PrimoRAMDisk with PrimoCache by 100%!
Posted: Sat Jan 14, 2017 5:41 pm
Hi Romex Software Team!
I have refreshed my server recently and played a bit with caching a storage space consisting of 4 * RAID6 (each 8 * 2 TByte drives), each connected via 4 GBit Fibre Channel. I get around 1500 MByte/s writes and 1410 MByte/s reads on it natively, which is very close to the physical limit of 4 * 4 GBit Fibre Channel (thereoretically 1600 MByte/s, reality is more like 1520 MByte/s).
Now I activated PrimoCache read and write with a 300 second defer write setting enabled (honestly risky, but while testing I don't care). The I/Os increased over 400% with PrimoCache. The transfer values increased to 8000 to 10000 MByte/s when using PrimoCache, stable for read and write.
I used ATTO Disk Benchmark 3.05 for testing, default settings. Increasing target size from 256 MByte to 2 GByte or Queue Depth from 4 to 10 only increases overall transfer speed even further, with peaks up to 15000 MByte/s. I employ a fairly old server mainboard with dual hexacore 2.67 GHz X5650 Intel Xeon CPUs and 144 GByte DDR3 RAM 533 MHz. If we compare that to an overclocked 3200 MHz DDR4 we clearly see differences... But who cares, that all is very well beyond the 40 GBit ethernet connection the server has...
What really wondered me was that with PrimoRAMDisk, formatted as Direct IO I get only like 3800 MByte/s writes and 4300 MByte/s reads (maximum), formatted as SCSI disk emulation I get like 3700 MByte/s writes and 4000 MByte/s reads (though starting at only about 30% I/Os with small block sizes, compared to Direct IO). But then I (playfully) enabled PrimoCache for the PrimoRAMDisk (SCSI emulation) and got an amazing increase in I/Os of around 80-100% and peak performance of 9600 MByte/s writes and 8300 MByte/s reads - thats about twice as fast as PrimoRAMDisk alone...!!!
But in turn, the normal disk drive is actually as fast with PrimoCache as the RAM Disk gets with PrimoCache. And in any case, PrimoCache is about twice as fast as the RAM Disk alone. The only small difference is that the Direct IO RAM Disk has a bit higher I/Os for small block sizes, while loosing dramatially with the bigger block sizes. In fact, PrimoCache REPLACES PrimoRAMDisk completely IMHO. Instead of having a usually fairly small RAM Disk with a corresponding disk file for recovery, you can simply use a drive/partition and use PrimoCache instead. It gives you better performance and the ability to scale well beyond your RAM size if needed. If you want it to be RAM only, have a small disk or small partition...
Honestly I would have expected the both apps to be similar in speed, with the RAM Disk potentially being faster as to not needing to worry about looking up cached blocks etc. However, it turns out the opposite is true. In my opinion PrimoCache completely replaces PrimoRAMDisk.
What do you think?
Can you see similar results?
I have refreshed my server recently and played a bit with caching a storage space consisting of 4 * RAID6 (each 8 * 2 TByte drives), each connected via 4 GBit Fibre Channel. I get around 1500 MByte/s writes and 1410 MByte/s reads on it natively, which is very close to the physical limit of 4 * 4 GBit Fibre Channel (thereoretically 1600 MByte/s, reality is more like 1520 MByte/s).
Now I activated PrimoCache read and write with a 300 second defer write setting enabled (honestly risky, but while testing I don't care). The I/Os increased over 400% with PrimoCache. The transfer values increased to 8000 to 10000 MByte/s when using PrimoCache, stable for read and write.
I used ATTO Disk Benchmark 3.05 for testing, default settings. Increasing target size from 256 MByte to 2 GByte or Queue Depth from 4 to 10 only increases overall transfer speed even further, with peaks up to 15000 MByte/s. I employ a fairly old server mainboard with dual hexacore 2.67 GHz X5650 Intel Xeon CPUs and 144 GByte DDR3 RAM 533 MHz. If we compare that to an overclocked 3200 MHz DDR4 we clearly see differences... But who cares, that all is very well beyond the 40 GBit ethernet connection the server has...
What really wondered me was that with PrimoRAMDisk, formatted as Direct IO I get only like 3800 MByte/s writes and 4300 MByte/s reads (maximum), formatted as SCSI disk emulation I get like 3700 MByte/s writes and 4000 MByte/s reads (though starting at only about 30% I/Os with small block sizes, compared to Direct IO). But then I (playfully) enabled PrimoCache for the PrimoRAMDisk (SCSI emulation) and got an amazing increase in I/Os of around 80-100% and peak performance of 9600 MByte/s writes and 8300 MByte/s reads - thats about twice as fast as PrimoRAMDisk alone...!!!
But in turn, the normal disk drive is actually as fast with PrimoCache as the RAM Disk gets with PrimoCache. And in any case, PrimoCache is about twice as fast as the RAM Disk alone. The only small difference is that the Direct IO RAM Disk has a bit higher I/Os for small block sizes, while loosing dramatially with the bigger block sizes. In fact, PrimoCache REPLACES PrimoRAMDisk completely IMHO. Instead of having a usually fairly small RAM Disk with a corresponding disk file for recovery, you can simply use a drive/partition and use PrimoCache instead. It gives you better performance and the ability to scale well beyond your RAM size if needed. If you want it to be RAM only, have a small disk or small partition...
Honestly I would have expected the both apps to be similar in speed, with the RAM Disk potentially being faster as to not needing to worry about looking up cached blocks etc. However, it turns out the opposite is true. In my opinion PrimoCache completely replaces PrimoRAMDisk.
What do you think?
Can you see similar results?