PrimoCache flush write cache taking forever?

FAQ, getting help, user experience about PrimoCache
Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

PrimoCache flush write cache taking forever?

Post by Axel Mertes » Mon Apr 29, 2019 11:38 am

Hi PrimoCache support,

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Mon Apr 29, 2019 11:58 am

Hi PrimoCache Support,

Windows 2012 Server Datacenter R2 x64. Dual Intel Xeon E5-2620 v2 @ 2.1 GHz, 128 GB RAM.

I am experiencing a problem here with PrimoCache server 3.0.9 flushing write cache, taking felt forever.
I have a 64 TByte RAID6 volume, 48 TByte netto, single NTFS 64 KByte blocks partition.
For caching I use an Intel P900 Optane card with 480 GByte as Read/Write cache with 60s for deferred writes.
Today I found the server getting very unresponsive. Diverse tasks that usually run very smoothly began to be unresponsive, hang up etc.
I checked the RAID and its reportedly fine.
Then I decided to prepare for a reboot and selected flush write cache. That takes now felt forever. I can see like 15-25 MByte/s writes on that volume, but that is soooooooooo slow compared to what it should be, I can't believe what I am seeing. In the past I saw like 2 GByte/s reads & writes from the P900, about 1.1 GByte/s towards the RAID itself (8 SAS disk, 8 TByte each).

In that context an important feature wish - I think I requested this already in the past:

Please do not only show a cylcic bar while flushing the cache, but a true percentage (at least as growing number), even with fine values changing like 12.345% so you see it finely growing.

Right now I can only see a cyclic bar telling me "Precessing, please wait..." - but how long?
Does it work or is it stuck?

This feature would make me feel much better if I can estimate it'll take an hour and I am ready to go. Now it runs for already an hour and I see no end... In that context, you may add a remaining time, time take info too.

Any Idee what can cause such behavior?

Fragmentation is at 1.87% according to O&O Defrag.

Hope it turns out fine, 44 TBytes in there...

Cheers
Axel
Last edited by Axel Mertes on Mon Apr 29, 2019 12:05 pm, edited 1 time in total.

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Mon Apr 29, 2019 12:04 pm

See here:
PrimoCacheFlushDiskProblem2019-04-29.png
PrimoCacheFlushDiskProblem2019-04-29.png (53.8 KiB) Viewed 1518 times

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Mon Apr 29, 2019 12:24 pm

Status:

After about 1.5 hours, PrimoCache finished flush to disk. Changing back to 10s for deferred writes and making a reboot. Will be interesting if performance will be up again or not.

User avatar
support
Posts: 2425
Joined: Sun Dec 21, 2008 2:42 am

Re: PrimoCache flush write cache taking forever?

Post by support » Mon May 06, 2019 5:12 pm

I'm sorry for the late reply!
Glad that PrimoCache finally finished the flush. It seems that disks (L2 disk or the raid) had problems in writing.
Regarding your suggestion, we do want to show the process percentage. However, because of the cache design, it's not easy to get the percentage information when doing the flush. We'll try to work it out.
Primo Ramdisk | PrimoCache
Romex Software Support

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Tue May 07, 2019 6:01 am

I understand that the cache is potentially being used/filled, while flushing it, right?
That explains why a percentage is a bit problematic.
However, depending on the decision which blocks you flush, you know the final amout of blocks at some point.
Even if that amount is growing over time, you know you have flushed n out of m blocks. The percentage is n/m percent, regardless of the growing size of the cache...

I an worsed case I might see the percentage flush shrinking... :-) But at least I know its working.

Or simply show "n out of m blocks flushed" - that would do the job IMHO.

User avatar
support
Posts: 2425
Joined: Sun Dec 21, 2008 2:42 am

Re: PrimoCache flush write cache taking forever?

Post by support » Wed May 08, 2019 3:25 am

Axel Mertes wrote:
Tue May 07, 2019 6:01 am
I understand that the cache is potentially being used/filled, while flushing it, right?
Yes, this is one of the reasons. During the flushing, new writing data may still come in.

Anyway, we'll try to work out a rough indicator, at least telling the process is not hanging.
Primo Ramdisk | PrimoCache
Romex Software Support

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Wed May 08, 2019 5:45 am

As I wrote, "flushed n out of m blocks to disk" would be all we need. If n and m is growing, we'll see and understand that.

ferrari
Posts: 25
Joined: Fri May 17, 2019 3:40 am

Re: PrimoCache flush write cache taking forever?

Post by ferrari » Thu May 23, 2019 1:32 am

Do you have read/write enabled on the l2 cache? Maybe the L1 cache was getting full and instead of spilling over onto L2 cache it was going straight to HD.

Axel Mertes
Posts: 157
Joined: Thu Feb 03, 2011 3:22 pm

Re: PrimoCache flush write cache taking forever?

Post by Axel Mertes » Thu May 23, 2019 7:27 am

ferrari wrote:
Thu May 23, 2019 1:32 am
Do you have read/write enabled on the l2 cache? Maybe the L1 cache was getting full and instead of spilling over onto L2 cache it was going straight to HD.
Yes, I do have read/write enabled on the l2 cache.

The particular cache is 96 GByte of RAM and 480 GByte SSD of Intel Optane P900 memory with 64 KB block size on a 48 TByte NTFS volume with 64 KByte block size. The target volume is a RAID6 with 8 * 8 TByte SAS 1200 drives, performing at about 1.4 to 1.8 GBytes/s on sustained transfers for sequential reads and writes, obviously less on small file transfers. The volume is 100% defragmented. So writing back the cache should be a pretty sequential process in most cases.

While performing the flush to disk operation, nothing else was accessing the volume.

Writing back 96 GByte RAM would not take that long. Writing back the entire SSD might take a moment.

@Support:

This brings up one question:
At which stage in the flush to disk process or general write back to HDD process is an optimization taking place to write back in the most sequential order possible?

Or are the blocks written in the order they have been cached, which might not be optimal in that context?

Or is that something left over to the controllers?

The reason I am asking this is that sorting the write back order in a flush to disk operation makes IMHO total sense, so its as streamlined as it could be and as fast as anyhow possible. Writing back in the potentially fragmented timely order the blocks have been cached would only slow down transfers and create a way bigger load to the drives, creating many extra head seeks and waiting cycles for the drives to spin around to show up with the right sector. Writing back in the most sequential order possible will absolutely minimize that process.

I would believe that this is already implemented - but better ask to be sure.

Post Reply