Feature Request: Support accelerating writes with L2 Storage

Suggestions around PrimoCache
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Axel Mertes »

Have you ever heard of the HP Z Turbo Drive Pro Quad?
There is a similar device from DELL and I believe others will follow soon.

Its a very clever product. Its a base PCIe board that allows to connect 4 M.2 SSDs onto it. It provides necessary cooling and apparently has some RAM with high capacity capacitors as power buffer for save writes. So its a really clever design enhancing SSD exactly where it needs to be.

Best of all: With 4 current M.2 Samsung 950 SSDs 512 GByte you get up to 2 TByte in total with an aggregated write speed around 6 GByte/s and read speed around 9 GByte/s. The already announced successor SSDs from Samsung (which will fit into the HP board too) will allow for 4 TByte in total with write speeds around 9 GByte/s and read speeds around 12 GByte/s. The base board including 2 * 256 GByte Samsung SSDs is around 850 Euros. A 512 GByte SSD is around 300 Euros, so a full 2 TByte drive will cost around 1650 Euro I suspect.

Unfortunately these boards are BIOS locked to HPs zx40 workstation (z840, z640, z440). DELL seems to do the same. But it shows what kind of product we can use/expect soon.

There are also more expensive PCIe SSDs which are not locked to specific machines/mainboards, which get at least somewhere close. As this is around RAM speed of older systems its basically what you are looking for in the long run.


I hope Romex Software is soon implementing the often asked for write caching on L2 and allows also double deffered writes from L1 to L2 to HDD. The latter can be done as I suggested earlier by setting the defer write time from L1 to HDD and L1 to L2 and from L2 to HDD.

Example:
L1->HDD = 1 Day
L1->L2 = 10 seconds
L2->HDD = 1 hour

Your data will be written to L1 RAM cache first, after 10 seconds it will be transferred to L2 SSD cache, where it will stay for 1 hour until it gets written to HDD.


Important:
Even after writing to HDD, the cache is not invalidated/free'ed up - because there is no reason to do this. The cache is read cache at the same time, so subsequent reads of the same data will be pulled from either L1 or L2, and only if the data is no longer there, its going to be pulled from HDD.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Support »

@Axel Mertes, thank you! Yes, we're developing this feature.
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Axel Mertes »

@Support:

I am looking forward to it. Can't wait to use it. Contact me if you want me to beta-test it. I do beta testing since more than 2 decades for companies like Adobe, Blackmagic Design, Matrox and many others.

Having this feature alongside the new battery or capacitor backed up M.2 SSD boards takes reliability of SSD caching approaches to a completely new level.
minhgi
Level 10
Level 10
Posts: 255
Joined: Tue May 17, 2011 3:52 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by minhgi »

Bump. I wonder how i this feature coming along.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Support »

It will come out with 3.0 version with other new features. Just the next version after v2.5 if everything works well. :)
Thanks!
InquiringMind
Level SS
Level SS
Posts: 477
Joined: Wed Oct 06, 2010 11:10 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by InquiringMind »

As an aside, Addonics has a PCI-E card providing 4x mSATA connections with a claimed 10Gb/s speed in RAID0 mode. Not quite as impressive as the HP card, but cheaper (US$55) and with no hardware tie-in.
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Axel Mertes »

@InquiringMind:

The Addonics card has only support for mSATA, while the HP card is NVMe. The difference is that with mSATA, each M.2 mSATA card is limited to SATA 3 speeds, ie. 600 MByte/s maximum per card. Current M.2 NVMe cards such as the Samsung 950 run at 2.6 GByte/s transfer speeds, being about 5 times faster than M.2 mSATA drives.

Further, the HP product has functionality to buffer the M.2 NVMe cards with power in case of a power loss plus apparently some sort of RAM buffer between the cards and the PCIe host, making them extremely fault tolerant.

I understand that the Addonics card is not tied to a specific computer, but it can only do like 10 GBit/s while the HP can be as fast as 10-12 GByte/s, which is a completely different beast.

However, for most users and uses the Addonics will work nicely, but it is already outdated tech. If it would support M.2 NVMe, it would be a far better deal.

@support:

Any time frames as to when we can expect Version 3 with the new features?
zootie
Level 1
Level 1
Posts: 1
Joined: Sat Mar 04, 2017 7:16 pm

Re: Start thinking storage tiers

Post by zootie »

Another vote that this would be a welcome feature, even a game changer with PCIe NVMe and Optane storage on the horizon.

With this feature, we would be thinking more of tiered storage rather than caching. And indeed, the implementation and recovery possibilities can present some challenges and opportunities. Some thoughts on this.

For volumes with L2 write cache enabled, because it is tiered storage, writing should be (optionally?) done to L1 and L2 in parallel (not wait for it to fall out of L1 or wait for a timer). Furthermore, implementing some form of L2 mirroring would be a nice feature addition too (ie, write to L1 and 2x L2 volumes in parallel), to be able to implement a somewhat redundant tiered storage regardless of HW support (ie, not all (any?) motherboards allow to mirror NVMe disks, and that would imply another likely fakeraid driver that would add latency, it’d be better if PrimoCache implements a simple mirrored L2 by allowing you to select 2 disks).

Storage Spaces is a Windows direct attached solution that supports storage tiers (https://blog.cdemi.io/caching-and-stora ... es-direct/). But it can be confusing and it has many requirements that are automatic and immutable that assume enterprise grade hardware (and if you don’t have it, performance is lackluster, which is what seems to happen with most people experimenting with it). For example, in essence it requires PLP on the performance tier (Don’t do it: consumer-grade solid-state drives (SSD) in Storage Spaces Direct - https://blogs.technet.microsoft.com/fil ... sumer-ssd/), which means shelling out for a pair of $300+ disks (there might be some cheaper disks that do provide PLP to Storage Spaces satisfaction, just a bit difficult to figure out which). While it might be good that MS is not letting uninformed users make potentially risky decisions, this can bring up the price tag to unattainable levels for enthusiasts and SMBs. PrimoCache would leave it up to the user to make these decisions and compensate to their acceptable level of risk vs performance (ie, ignore or have a good enough UPS and offline backup procedure in place), positioning itself as a more flexible and powerful alternative tailored to sysadmins.

With L2 write “cache”, PrimoCache would be simple, inexpensive, and flexible solution capable of achieving performance levels reserved to Storage Spaces (with Enterprise Level M.2 or SSD) or LSI CacheCade (https://www.broadcom.com/products/stora ... o-software), at a fraction of the cost.

If it is too complicated, there might be a restriction in which L2 write caching wouldn’t be available on the system partition. This could avoid racing conditions during boot (and/or simplify driver development). For example, partitions with L2 write enabled could be of an application specific type -with MBR disks, you could hide a partition by setting its type to another OS, is something like this possible in newer GPT disks? - that prevents mounting them w/o the PrimoCache driver. That way, even if you boot another operating system, or move the disks to another computer, the volumes would not be mounted (and wouldn’t risk corrupting the file system because some blocks are on L2).

If the partitions are hidden this way (to ensure the PrimoCache driver and L2 tier are part of the equation), there should be a way to flush the L2 cache and reset the partition type back to a normal one, so the file system can be accessed w/o PrimoCache at that moment. It would also be helpful to be able to bypass it (ie, break the L2 to target partition relationship) if the L2 tier was irredeemable lost, and you’re just trying to recover data from the target partition.

This should also be possible to be done on a parallel OS installation or on a different computer, so all the configuration/relationship/block status should be written to both L2 and the target partition (probably as a file in its file system, to make it simple and avoid disrupting it). This data might need to store version specific information – if in a dual boot situation, the PrimoCache driver would know if it is able to handle it or not (if the meta-data version was upgraded by a parallel installation), if not, log an event and don’t mount it, leave it to the user to upgrade).

Really looking forward to v3, also volunteering for testing.
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Feature Request: Support accelerating writes with L2 Storage

Post by Axel Mertes »

In my opinion the biggest difference between using PrimoCache L2 SSD cache and building a storage tier like in Storage Spaces is that the SSD cache is taking only a copy of the data that is living on the HDD storage or storage array which we are caching. In Storage Spaces, the data is potentially moved between the two storage tiers, ie. depending on your redundancy setting there is only one copy existing. If then the SSD fails, that data is lost and potentially a big part of the Storage Spaces tier can't be recovered. Using the SSD as a "copy only" cache makes it much less prone to failure. It may be a "waste" of strorage space in the end, but there are a lot of benefits to it.

The idea of preventing a recoverable SSD L2 cache from being accessed accidentially by an OS makes sense. I have similar functionality with Fibre Channel drives being part of a SAN. As soon as I bind them to the SAN you are completely unable to access them, not even for formatting. You have to remove them from the SAN to make them available/useable again. Thats a nice feature.
Post Reply