L2 is filled from HDD instead of L1 cache

[npelov]

Hi,

I have a problem. I'm trying to speed up a 500 GB HDD. I gave it a 66 gb NVME L2 cache (INSTANT) and 1 GB L1 cache (block size 8k). The HDD is being accessed at relatively low speeds. The average is between 600 and 900 kb/s. The ssd has 20-40k iops and reaches about 168MB/s write speed at 4k, queue depth 1. It should be able to keep up with writing from L1 to L2 cache, yet the NVME barely shows any usage. when the stats showed 18 GB read from the hdd, only 1 gb was written to L2. The write was about 2.5 GB behind. When I stopped the software so nothing accesses the HDD anymore task manager showed that L2 was being populated from the HDD. That was normal because L1 didn't have the 2.5 GB capacity for all the data that needed to be written to L2. However all that data at some point has passed through L1 cache.
Because HDD is at 90-100% activity it barely has any free operations to read from it when L2 has to be populated. So instead of being speed up it's slowed down because the data has to be read again to copy to L2.
My question is is data being copied to L2 from HDD all the time (even when it's present in the L1 cache)? If yes isn't L1 being used to store data to l2 faster? If no - then why not?

In fewer words what's happening is (I think):
HDD->L1
HDD->L2

why not:
HDD->L1
L1->L2

and if the latter is really happening why is it so slow.

[Support]

When PrimoCache populates data into L2, if data is cached in L1, PrimoCache will copy data from L1 to L2, otherwise PrimoCache will read data from the source disk.

Please note that in order not to affect other applications' tasks, PrimoCache is set by default to fill L2 data when the source disk is idle. Even with INSTANT gather level, maximum L2 fill rate is limited to 40MB/s.

[Audionut]

Read-through L2 would seem nice. So many times I frequently access data within the cache task volume, and if that data hasn't been assigned to the L2 yet, it's painful having it constantly read from the volume.

Perhaps some sort of tiered approach, where existing read cache works as previous, but where fresh data is also always read through L2, and a determination is made after X time and/or XX% full L2, as to what data should be flushed from L2 read cache to make room.

Also, unless I'm seeing some unwanted behavior, or making a silly mistake, I'd like to see some validation of the L2 cache. L2 is often (always?) an SSD/NVME and the data it contains should always represent a baseline cache state on boot, IMO.

[npelov]

Please note that in order not to affect other applications' tasks, PrimoCache is set by default to fill L2 data when the source disk is idle. Even with INSTANT gather level, maximum L2 fill rate is limited to 40MB/s.

We are talking about 0.5-1 MB/s. The L2 disk was not used for anything else. Yes, the OS is on it but the computer is idle. The task manager shows almost no activity (busy 1-2% of the time).

I don't know why you would limit the speed to 40MB/s. Maybe make this adjustable, or at least do a short random read/write speed test on cache initialization to determine the speed of the disk and set speeds accordingly.

Also are you sure you detect idle state correctly. You should do more research on that - at different levels of business of the L2 disk.

[Nick7]

When PrimoCache populates data into L2, if data is cached in L1, PrimoCache will copy data from L1 to L2, otherwise PrimoCache will read data from the source disk.

Please note that in order not to affect other applications' tasks, PrimoCache is set by default to fill L2 data when the source disk is idle. Even with INSTANT gather level, maximum L2 fill rate is limited to 40MB/s.

Main issue is when data is pushed out of L1 it's not written to L2 at that time, but rather Primocache will later (when IDLE) read ALL again from HDD.
You should verify how busy is drive for L2 cache and if possible (and in 99% of cases it should be true) - write data from L1 to L2, and then push it out of L1 cache.
This avoids hammering HDD and double reads.
Also, currently - if you have (as an example): 1GB L2 - if you read 100GB from HDD, what Primocache will do is - reas again ALL 100GB, and write it to L2 (or should I say overwrite over and over, as capacity is 1GB). Instead of verifying how much data is pending for L2 write, and check that if it exceeds L2 cache size, just read last X size to fit L2 cache.

[Support]

I don't know why you would limit the speed to 40MB/s. Maybe make this adjustable, or at least do a short random read/write speed test on cache initialization to determine the speed of the disk and set speeds accordingly.

Sorry I forgot to mention that this is the maximum speed that PrimoCache will limit when it detects that the source drive is busy.

PS. We will allow users to set the idle detection threshold in the next version. Also L2 filling issue will be improved.

[npelov]

Maybe give the user option to set idle speed as well as idle detection. I'm confident that my ssd can handle a lot more than that. especially if queue depth is bigger - as in this case -you should write groups of blocks at a time, not a single block. all disks benefit from knowing what's next to write so they can plan ahead.
also I don't know you limit the speed to 40 mb/s, but it isn't very accurate. in my case it couldn't reach 1MB/s.
I hope you figure this out. Thanks!

[Support]

Maybe give the user option to set idle speed as well as idle detection. I'm confident that my ssd can handle a lot more than that. especially if queue depth is bigger - as in this case -you should write groups of blocks at a time, not a single block. all disks benefit from knowing what's next to write so they can plan ahead.
also I don't know you limit the speed to 40 mb/s, but it isn't very accurate. in my case it couldn't reach 1MB/s.

If PrimoCache detects that the system is idle, it will not throttle the speed. Therefore, adding the idle detection option is sufficient to this problem.

[tverweij]

If PrimoCache detects that the system is idle ...

Please, for the Server version, NEVER assume that the system is idle.