Power lost, L2 contents lost

FAQ, getting help, user experience about PrimoCache
ml70
Level 2
Level 2
Posts: 9
Joined: Sat Dec 07, 2019 6:31 am

Re: Power lost, L2 contents lost

Post by ml70 »

Don't you think it should be using the Defered Write timeout instead? It's a bit of a discrepancy when user intends no writes to hdd for 10801 seconds, but some secret rule overrides it to 120.

I'd have nothing against a default of 120 seconds if it was included in the UI and easily permanently overridden. But hardcoded secret defaults should be a no-no.

Combined with a user specified average write speed limit the results could be great. Now the issue with Average mode seems to be that the writes are still very bursty, upto 1 GB at a time. Whereas on a heavily used disk one would rather specify something like an average of 32 MB/s.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Power lost, L2 contents lost

Post by Support »

I'd like to know your incoming write-data pattern from Windows/Applications. Is it bursty or sustained? And peak data amount of write requests per second?
ml70
Level 2
Level 2
Posts: 9
Joined: Sat Dec 07, 2019 6:31 am

Re: Power lost, L2 contents lost

Post by ml70 »

Writing is very bursty, anything upto 1 gb at a time, but also sustained for long periods of time, in windows task manager terms cpu load 30-80% and same for disk usage at least 6 days a week. I'm afraid i don't know how to obtain the number of write reqs per second, it varies very much anyway so could be a meaningless metric.

That's why i had hoped that the Average mode would so some real averaging, as sudden large write bursts kill random read performance, which is the main workload. It could be a huge help if Average mode could use the set defer write period also as the averaging period (instead of a fixed 120s), and/or there was a possibility to specify the top average write speed and # writes per second (it's understandable that if L2 becomes full then writes must be performed immediately, sizing the period and speed is operator responsibility anyway).

I've estimated that in this case to keep up random read performance the average write speed to hdd should be around 32 mb/s, max (in sequential terms, does not equal 32 pcs of 1 MB writes). It doesn't matter if it lengthens out the total write time, my time scope for this is minimum weekly, anyway.

The above actually touches on the issues noted:

-Sudden crippling peaks of read/write activity (ref image posted earlier) which halt the server so bad even rdp login is not possible, lasting a few (maybe avg 5?) minutes at a time. Reads and writes are done by System(4). This only happens after a few days, but even before L2 cache is full. Could this have something to do with Primocache expecting the server to be idle sometimes, and if there's no idle then ultimately this happens? Could the definition of idle be defined somewhere, for me idle means <50% cpu and disk use. If the software expects <10% it's just not going to happen except perhaps once a week.

-After L2 is full, if the writer makes huge (upto 1 GB) writes, things slow down for a moment. Can this be caused because L2 only has a default of 32 MB free space, and there is no place for the new data to land? If there was an option to set the size of the landing zone maybe this type of slowdown could be avoided? I'd set it to 4 GB just to be sure, its only 1% of the L2 size anyway.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Power lost, L2 contents lost

Post by Support »

When the average mode is chosen, we assume system's writing workload is heavy. So the goal is to fully utilize the writing capacity of both cache device and target device, and to receive and process incoming write-data as quick as possible. This mode was designed for heavy-write-IO pattern. Mixed read-write pattern was not considered.
If the latency is too long, accumulated deferred write-data in the cache might get more and more. Finally when the cache is full of deferred write-data, it will trigger "urgent" write which will degrade the performance very much.
You may check the statistics "Deferred Blocks", if it goes to a high percentage value like 85% or more, usually it means that the latency is too long.
ml70
Level 2
Level 2
Posts: 9
Joined: Sat Dec 07, 2019 6:31 am

Re: Power lost, L2 contents lost

Post by ml70 »

First I need to apologize here, because it turned out that the crippling write spikes shown in the picture are not the Cause, but the Effect of something totally else. A rare configuration issue in an upstream router sometimes randomly starved the writer of data, and due to that idleness the cache perceived an opportunity to do some housekeeping, which results in the write activity. If anything I need to thank PrimoCache here because without that write activity spike I'd never have found this bug, it served as an indicator.

I hope you could consider the case for random-read heavy system main workloads as PrimoCache could be of significant help here. The Average mode works well enough, I have a feeling that simply being able to specify the free landing zone size (let's call it LZ here) on L2 as a user configurable value instead of fixed 32 MB could help already. Because it is understandable that if there's only 32 MB free on L2 and a sudden 1 GB write, problems will naturally follow.

Of course it needs a little logic to help, so that when free space on LZ is less than the user set value (there have been incoming writes), the data which needs to be written to hdd (whether from LZ or elsewhere in L2 cache, that's up to PrimoCache's logic) will be written in a controlled averaged fashion and not as emergency writes.

Only if free space on LZ equals 0 should there be an emergency write, but setting LZ large enough should be operator responsibility, although it wouldn't take much to make this function self learning in PrimoCache, so that if there's regular writes of size X incoming while the L2 cache is full, PrimoCache would learn to keep at least X bytes free on LZ (possible issue with too small L2 cache devices).
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Power lost, L2 contents lost

Post by Support »

I'm happy that you find the cause of the problem and solved it. That's great!
The threshold of 32MB just means that the cache is full of data. It doesn't mean the cache is full of deferred write-data. The amount of deferred write-data is indicated by "Deferred Blocks". If deferred write-data is not close to full, emergency writes will not be triggered.
Usually you may just tune the latency to avoid the cache cumulates too much deferred write-data. But yes, we'll consider the mixed write-read mode and may provide a better algorithm. Thanks.
Post Reply