Deferred Writes - Power Loss + this is awesome.

[xrstokes]

I would like to state that I'm not connected to or affiliated with ROMEX software, I have extensive IT experience, and generally know most of what I'm talking about.
Ever since I heard of caching a few years ago I've been on a mission to improve my old spinning raid 10 performance, for 3 months I used FreeNAS in a VM and presented my data over iSCSI. The performance was great at first but over time degraded. I could also only muster an SSD cache of 20GB because ZFS requires a lot of ram to hold the meta data for the L2 cache, it does have an awesome feature though called an SLOG, it's like an SSD write cache. You use mirrored drives to store deferred writes, and if the power goes out, your completely safe. Apart from the fact the L2 cache was small, It was a pretty neat setup. Always looking for a thrill though, I wanted more.
So, I used StarWind Virtual SAN to set up an LSFS drive and likewise presented the data back to my PC as an iscsi target. This let me use my full 120GB SSD as cache and was more stable, although it has no deferred writes, or the type that won’t be lost in a power out anyway, meaning my writes were a little slower. LSFS addresses this by being log structured, which really helps your writes even though there just going onto spinning disks, but your data gets really, really fragmented over time. Then you have pitiful sequential reads, I mean single digit bad. This could still be ok though if I’d gone with a flat file system instead of going with LSFS. But on the down side it flushed the L2 cache on a reboot, which is bad for me as I don’t leave my lab powered overnight.
Then I tried VELOSSD and it was a disaster, you see, it did do deferred writes by default. It was also not that stable and I suffered occasional BSOD's. After the second one it caused. My raid file system was damaged beyond repair and would only show up as RAW.
Then I tried Diskeeper[express cache] for a while, but after an update got BSOD's on boot up, whatever I did to the system drive to cache it, windows 10 didn't like anymore, uninstalling it from safe mode fixed it for me, and it least it didn’t cost me data like VELO. I don’t mean to run these other software solutions down, they just didn’t do it for me.
Primo was next on my list to try, and can i say after week, I'm super impressed, super stable and no other PC issues. I usually have running about 6 VM's and I've experienced very little data churn, at least since my L2 cache finally did a large gather. Seeing I do a lot of random writes with that many VM’s, I’m actually using an ReFS setup using storage spaces. The thing I’ve been really scared to turn on though is deferred writes, after the horror of being burnt in the past, I’m terrified to even try it.
So, my question is this. Considering I’d only try it on mirrored SSD’s, what are my possible failure outcomes. SSD drive failure would be taken of the table. Power loss is slim to none, depending on UPS failure. What about the one in a million BSOD? On FreeNAS it would get the missing data from the SLOG, but what does primo do? Would it try and save that data in the deferred write cache, or just dump it? Because I snapshot often, so the odd damaged virtual drive if no big deal. But if the whole FS is stuffed, that would be pretty scary.
Also what has people’s real world experience been using deferred writes?

Thanks if you actually read this whole post. I hope this whole endeavour works out, sick of the down time between trying different caches.

[Jaga]

I've been using Primocache since it was called Fancycache (so around 5-6 years now). And the last ~3 years of that I've had a UPS on affected hardware, and had the L1 turned on in Primo and configured with long write timers. Never had an issue of any type, even with BSODs. The only problem I had years back that I reported as a bug, was a problem with the Intel RST configuration software, which Romex fixed rather quickly after they had the info.

Since then I've never had a situation where enabling the Primocache L1 write-cache has negatively affected me in any way. UPS has covered the hardware well, and system Sleeps/Hibernates and even the occasional bluescreen haven't harmed the drive data. The only time I was ever affected was during one of the Windows 10 rollouts, where they re-enumerated drives and for some reason it didn't play well with the *Beta* version of Primocache I was testing at the time.

There are occasions where you might lose the L1/L2 contents, and have to start re-populating the caches with real data (they flush for one reason or another). But that won't affect the write buffer.

All other times, rock solid. Turn on the L1 if you're covered by a UPS and don't have overclocking issues.

[xrstokes]

I've noticed that the SSD can get over burdened when the L2 write cache is flushing at the same time the read cache is be read. I bought another SSD so I could have a separate write cache on other SSD, mirrored of course. But alas I can't have a separate write cache disk in the configuration options. Am I missing something? Or is this not possible. If there was a server version that was a bit dearer but enable this feature, would be great.

[Support]

SSD write caching option is available in the 3.x versions. By default PrimoCache set L2 for read-only purpose when you create a new cache task. However, you can change it by the following steps.
1. Click the icon button "Advanced L2 Cache Options",
2. Uncheck the option "Individual Read/Write Cache Space" if you want the whole cache space shared for both reading and writing. Otherwise, move the slider to specify a ratio of writing cache space.
When L2 write cache space is set, and defer-write option is checked, L2 has Write-Back enabled.

Regarding the defer-write, please see
http://www.romexsoftware.com/en-us/prim ... on.html#dw
I have to say that the results are unknown in the case of a sudden power-loss, system crash or bsod. It is recommended that you may only enable Defer-Write on volumes where temporary, unimportant or reproducible data is to be stored.

[xrstokes]

"I have to say that the results are unknown in the case of a sudden power-loss, system crash or bsod."

What does the software do o reboot? does it look in the L2 cache for unwritten deffered blocks and run them to the storage?

[xrstokes]

I'd just like to share some of my results. All these results were taken from inside a VM running on the cached storage.
The first pic is from an refs petition running on a freenas backend. Write-back disabled and relying on the ZIL[L2 write acceleration] to speed up the writes.
While it is a "free" solution. it's a lot of work to setup and has a limited L2 size of 3 times the ram cache. in my case, that's only 24GB.

refs on zfs.PNG (51.64 KiB) Viewed 6268 times

The next PIC is from my trials with starwind LSFS, it is free for home and non production use. much easier to set up and manage then freenas. It has unlimited L2 cache size. But always writes to the underlying storage in write through mode and has no L2 write acceleration. Only long term problem with LSFS is over time she gets really fragged up and sequential reads end up dying.

ntfs on lsfs.PNG (58.39 KiB) Viewed 6268 times

The next pic is from my current PrimoCache set up. As you can see, it's right up there with these other 2 systems, but is so much simpler to use. It has L2 write acceleration and doesn't rely on iscsi to present the storage to the host. this means higher burst speeds are possible then the other two, where in practice, my VM's never seem to get it. The one thing I really need to get to the bottom of though is if these deffered blocks at the L2 level are lost after a crash or reboot. starwind don't even bother to include a cache accelerator, freenas will check the ZIL if there is a reboot and run the changes to the storage. But really need to know what PrimoCache will do. Maybe the trade off for the performance is the risk of destroying your file system. In which case I would defiantly need to get a better UPS, but would I still use it? I don't know? that performance is pretty sweet.

refs on primocache.PNG (59.35 KiB) Viewed 6268 times

[Support]

During shutdown or reboot, PrimoCache will flush all deferred write-data from cache space to underlying disks. No data loss will happen.

[xrstokes]

But if there is power loss there is no flush, does it look at boot up in the L2 for potentially unwritten deferred blocks? Would adding this feature be something that you would consider adding in the future?
Because if you did you'd have a server level feature that you could easily charge more for. It would be worth at least $200 to me. lightning SSD fast storage writes. knowing that if there is a crash, after boot primo will find them and write them to the storage. BTW, I'd never consider primo for OS storage, just use an SSD for that. This is for large VM storage. Because from what I'm experiencing, your trading blow for blow with these other guys for a fraction of the price.

[Support]

But if there is power loss there is no flush, does it look at boot up in the L2 for potentially unwritten deferred blocks?

No.

Would adding this feature be something that you would consider adding in the future?

It's kind of difficult to implement this feature. We'll think about it.

[xrstokes]

Thanks for the clarification, I'm so happy with the rest of the product, I can live with the risk of data loss when the storage is busy. And I can just turn of deferred data when I can.