I need assistance to get the best configuration for a more "real-time file-access" cache.
Specifically, I'm wondering how to speed up the Cached Read rate which is incredibly slow at 3 MB\second. But I will detail my config and scenario because they are probably not like a typical user.
I have 18TB HDD with mixed use data, to be cached (data only, no OS). The L1 cache is 8 GB RAM, and the L2 Cache is 45 GB volume on a NVMe SSD (gen3).
The files accessed are random and are usually different after every windows reboot, so I have prefetch options disabled. (If there is a reason this should be enabled anyway, I would love to know that, but I believe it doesn't benefit my scenario if I am opening new files after every reboot?)
Instead of prioritizing recent or common files from previous Windows sessions, I want to setup PrimoCache so that once a file is opened or in-use, that is when it goes directly to PrimoCache for full cache benefits.
However, the 'Cached Read' rate for open file is incredibly slow- 3 MB\second. The problem with this slow rate is that files I want to cache can be very large (up to 10 GB). I also do a lot of file write actions in windows for those files. But I notice that a file write action only gets full benefit if the file is active for an hour (this is how long it would take for the whole file to be in cached read.) Also, I notice that it seems slower than expected on the first write, but if i do the same write action a second time, then I see an even greater speed.
So I'm wondering how to get the fast write action sooner?
Settings:
Write-defer enabled, latency at 300 (but in my testing, this value does not affect anything on the write action.)
Write Mode is Native, with Flush L1 to L2 enabled.
Block size: I have tried all, this has not made any difference that i can see, during the write action.
For the read\write cache spaces- neither L1 or L2 has this enabled (using default shared setting).
As far as I can tell, the thing that makes a huge difference is just waiting a super long time for the active file to be fully loaded in Cached Read. But the slow rate for this to happen does not make sense to me. There is no bottleneck in the system (CPU or RAM), and there is no bottleneck on any of the drives. Both L2 cache volume and the HDD are at 1% activity or less while primocache is running and loading cache read.
Am I missing something in settings? I get that the program was probably not designed for this scenario, so maybe I am sol, but I will keep testing if there is a way to improve the process for real-time (newly opened) files.
I should note that I'm still on the trial period- could there be a limit on cache read rate during trial period?
Appreciate any advice on this, thank you in advance.
Slow Cached Read rate
Re: Slow Cached Read rate
Cached Read or Cache Hit Rate is used to reflect the read cache performance. The 53GB cache is too small for an 18TB HDD to improve the read performance, so you see that their values are very low and there is almost no read acceleration effect. I suggest you use the cache entirely for write acceleration, i.e. set both L1 and L2 to "100% Write", and ignore read caching. Write Latency can be 10~30s, your other write settings are good.
Re: Slow Cached Read rate
Thank you for the suggestion- I tried just using write cache but unfortunately it didn't speed up the overall writes compared to before. When using both read\write cache space (the default setting), I got a small bump in performance due to the slow rate that the file was going to cached read. As apposed to no read cache at all. If there is a way to allow an open file to move to cached read more quickly, that would be huge for my workflow. I get that I do not have the same workflow\requirements as other users (and an 18TB mixed use data drive simply can't be cached easily anyhow). But, if there was an option to allow the system to speed up the rate that an open file can move to Cached Read more quickly- this would awesome. (essentially prioritizing real-time open files instead of a history of frequently used files.) I think i saw somewhere in forum comments that this might be an option in a future release? If so, I want to add my name to the list of users requesting that option- it would be a huge help in my workflow. thank you again
Re: Slow Cached Read rate
I think you don't understand something, writing on the HDD is as fast as the speed of that HDD, the SSD can only act as a buffer - the bigger the SSD, the bigger the buffer and it gives you more time until it gets filled
the only thing I can think of to accelerate writes on HDDs is to have a RAID10 type array
the only thing I can think of to accelerate writes on HDDs is to have a RAID10 type array
Re: Slow Cached Read rate
I use write-defer option, so actually there is a performance benefit on the writes to HDD as well. However, in my testing, that performance gain is maximized when the file is mostly or completely in Cached Read already. (I have seen vastly improved write times on HDD files when using defer-write, but again, mostly when the file is already in Cache.) Therefor, I'm hoping to find a way to force open files to go to Read Cache more quickly than they do now. (the current rate is only 0.5MB up to 3.0 MB /second, depending on the size of the file.) At this rate, it takes a very long time for large files to go to read cache. So again, if there was a way to allow open\active files to move Read Cache more quickly, that would be a game changer for me and I'm sure for other people as well.
It would be nice to have this as an optional feature, unless there is a way to already do it in the current version?
It would be nice to have this as an optional feature, unless there is a way to already do it in the current version?
Re: Slow Cached Read rate
The performance of Defer-Write has nothing to do with whether the file is cached. If you mean that in your application scenario, the overall performance is best after files are cached, then I'm sorry, currently PrimoCache does not have such a function to pre-read the entire file.
Re: Slow Cached Read rate
Copy and paste your file, then remove the copy.
The original will now be cached as it has been read completely for the copy.
The original will now be cached as it has been read completely for the copy.