L2 Cache Not Used When Needed

FAQ, getting help, user experience about PrimoCache
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: L2 Cache Not Used When Needed

Post by mell111 »

The Urgent writes definitely increased substantially, but I can't say that it was steady (as opposed to bursts) since I wasn't monitoring the whole time.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: L2 Cache Not Used When Needed

Post by Support »

Not sure if this is the cause, but if possible, you may try a smaller latency like 10s, 30s or 60s to check if the problem still happens or not.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: L2 Cache Not Used When Needed

Post by mell111 »

I tried that - it didn't help. The longer latency appears to have reduced the incidence of this (though it could be a coincidence), but once the thrashing begins, the only thing that works is to stop app activity to the affected disk and allow PrimoCache to catch up. I find it puzzling that even with this stopped activity, PrimoCache still takes a few minutes to write about 2GB, but I presume its because of internal activity of the drive.

Can you tell me what to look for if I encounter this again? Also, as it stands now, should PrimoCache reduce its writing activity when the underlying disk is very slow and it has cache space available? If not, what do you think about that? Thanks.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: L2 Cache Not Used When Needed

Post by Support »

We'll introduce a new writing option to coordinate the work of L1 and L2 write cache. It might be helpful to this problem.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: L2 Cache Not Used When Needed

Post by mell111 »

I have an additional data point that illustrates the problem with more clarity. The screenshot below shows no writing to L2 cache and plenty of available L1 cache and hardly any urgent writes while the underlying drive was thrashing and the writing app was slowed blocking on writes (indicating, I think, that PrimoCache is passing along the writes directly to the drive, rather than caching). When the deferred blocks reached around 20,000 I suspended the app and PrimoCache slowly recovered as before. This whole episode lasted around an hour with the thrashing probably lasting around 10 minutes. Is this expected behavior under the circumstances? Can you say a few words about the new writing option and how it might help? Thanks.
PrimoCache-20190115.png
PrimoCache-20190115.png (16.92 KiB) Viewed 4055 times
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: L2 Cache Not Used When Needed

Post by Jaga »

Just an observation based on that screenshot - your Urgent Writes are a little over 100MB, while total writes are over 86GB. Do you ever get into a scenario where urgent writes are a significant portion (i.e. Primocache truly "spills over" and hammers the drive directly with urgents)?
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: L2 Cache Not Used When Needed

Post by mell111 »

Typically in my usage urgent writes are less than 3% or %4. It was a coincidence that the scenario I described in this thread came up shortly after I started a session, so the data in the screenshot is not mixed with a lot of "normal" data (although there was no thrashing for approximately 30 minutes from the start). The odd thing to me is that PrimoCache is not caching incoming data as it struggles with slow writes to a thrashing drive (under normal circumstances it does continue to cache as it write to the disk at normal speed.) If it did, this should allow the drive to "catch up" and recover, but being hammered by both PrimoCache and the app doesn't allow this. Also, on two occasions my system crashed almost certainly in this scenario (I can't be 100% sure since it didn't happen while I was observing, but my system has been rock solid for a couple of years now.)
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: L2 Cache Not Used When Needed

Post by Support »

@mell111, yes, the data in the screenshot is kind of unexpected as I see Total Write on L2 is 0. This indicates that L2 didn't cache incoming write-data when L1 is full. Does your writing app bypass volumes and directly write data to the underlying disk, like write-through mode?
The new write option we plan to add is to remove L1 urgent flushing when L2 has plenty of write space. That is, during the latency time, L2 will continue to cache incoming write-data when L1 cache is full, and L1 will not start flushing its deferred data to disk.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: L2 Cache Not Used When Needed

Post by mell111 »

No - my writing app does nothing unusual, it just does plain writes. I can confirm from many observations with Windows Task Manager that under normal circumstances (no thrashing of the underlying disk) when the given volume is cached, only PrimoCache writes to it at the approrpriate interval.

Also, it isn't just L2 cache that isn't getting written to - there is plenty of available space in L1 (about 75% by my calculation) and it isn't utilized either.

Thanks for clarifying the new write option. Given that in the above screenshot there is hardly any urgent writing, what do you think would happen? If the normal writing will continue (at the latency interval) then the disk will continue to thrash. Do you think it's reasonable to have an option to check write perormance at the latency interval and continue to defer writes when there is plenty of cache space available and disk write performance is very bad (in this case I'm getting much less than 1 MBps with occasional spikes to 2-3 MBps.) Even very slow disks don't perform that badly unless they're thrashing.

Thanks.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: L2 Cache Not Used When Needed

Post by Support »

Did you check if other apps will cause such problem? For eg, when L1 is flushing, you copy files to the volume cached by L1.
If this problem only happens with that writing app, is it possible that you send us this app for our analysis?
Post Reply