How to set custom, very large block size?

FAQ, getting help, user experience about PrimoCache
Post Reply
Biffo
Level 1
Level 1
Posts: 3
Joined: Sat Mar 11, 2023 4:53 pm

How to set custom, very large block size?

Post by Biffo »

I want to set a block size of 10MB. how would I do this? The max I see is 512KB.

I'm trying to solve a very specific problem by adding PrimoCache to the equation. I have an external USB dock with 18TB hard drive that contains lots of media files. Typical file is maybe 500mb. I can copy this file to internal disk in a few seconds, but if I play the file with VLC, then the hard drive will constantly seek as I watch the video even though the file is contiguous (not fragmented) on the drive.

So as I'm playing a 50 minute 500mb file, every 2-3 seconds, the drive will go "thud" as it seeks. There is nothing wrong with the drive. I think perhaps the dock is poorly designed not to cache large blocks from the drive but I'm not sure.

Anyway, with 512KB blocks, that means it will read a new block every 3 seconds, so with PrimoCache in place, I'm still getting a seek thud every 3 seconds. So instead, I want to cache big 10MB blocks. Then I will only get a seek noise once per minute.

Anyway, I didn't have to explain why I wanted such large blocks, but I assumed I'd just get pushback about inefficiency, etc. unless I explained. i don't care about efficiency, I just want to solve this one problem. Right now, with PrimoCache, I can solve the issue by running md5sum on the file before playing it, since that will force cache it in a few seconds, but I'd like things to work without having to do that annoying extra step.
Nick7
Level 5
Level 5
Posts: 46
Joined: Sun Jun 25, 2017 7:50 am

Re: How to set custom, very large block size?

Post by Nick7 »

For PrimoCache to work, you need to pre-cache file. For that, it doesn't matter if block size is 10MB or 4k.

You need to setup some batch file to read&copy file you want to play to some local SSD drive, and play from there. PrimoCache is not answer for your use case.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: How to set custom, very large block size?

Post by Support »

Yes, as Nick7 said, cache block size doesn't help your problem.
Biffo
Level 1
Level 1
Posts: 3
Joined: Sat Mar 11, 2023 4:53 pm

Re: How to set custom, very large block size?

Post by Biffo »

I'm not following at all. PrimoCache is exactly what I need, except the block size is too small. Why would i need to pre-cache the file? Caching is PrimoCache's job. VLC will demand the file blocks off the hard drive, and PrimoCache will load the 10mb blocks on demand into RAM (caching them). Once a block is cached, there will be no disk access until the next block is needed.

As I said, I already tested whether PrimoCache would work by running md5sum on the file, forcing it into PrimoCache's RAM cache. It works as expected, and VLC reads from the RAM cache. So all I need is larger blocks so there is not constant disk activity.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: How to set custom, very large block size?

Post by Support »

It has nothing to do with cache block size. PrimoCache doesn't do the read-ahead or pre-fill the whole cache block. It caches data based on access, that is, if VLC or md5sum (in your case) read 1mb data, then PrimoCache will cache 1mb data. Even the block size is 10mb, PrimoCache still caches 1mb data. (This applies to L1 cache, L2 cache is another case.) That's why Nick7 said that you need to pre-cache file which currently is done by md5sum in your case.
Biffo
Level 1
Level 1
Posts: 3
Joined: Sat Mar 11, 2023 4:53 pm

Re: How to set custom, very large block size?

Post by Biffo »

I was asking ChatGPT what software would do what I want, and it directed me to PrimoCache, even though the previous responses I got in this thread indicated it doesn't do what's needed. ChatGPT doesn't have knowledge beyond 2021, so it was referring to PrimoCache 3.x. It told me:
In PrimoCache, the maximum amount of data you can read ahead depends on the "Read Ahead Block Size" setting, which is configurable when enabling the "Read Ahead" feature in the "Advanced Intelligent Cache" settings.
This definitely sounds like what I need, but apparently the block sizes were way too small anyway. Nothing as large as 10mb. Was the read ahead feature removed from PrimoCache 4.x, or was it just badly named and wasn't actually doing disk read-ahead?
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: How to set custom, very large block size?

Post by Support »

Sometimes ChatGPT makes up messages on its own, we never have the Read-Ahead feature.
SnowReborn
Level 4
Level 4
Posts: 34
Joined: Sun Dec 02, 2012 3:13 am

Re: How to set custom, very large block size?

Post by SnowReborn »

Biffo wrote: Mon Mar 13, 2023 4:27 am I'm not following at all. PrimoCache is exactly what I need, except the block size is too small. Why would i need to pre-cache the file? Caching is PrimoCache's job. VLC will demand the file blocks off the hard drive, and PrimoCache will load the 10mb blocks on demand into RAM (caching them). Once a block is cached, there will be no disk access until the next block is needed.

As I said, I already tested whether PrimoCache would work by running md5sum on the file, forcing it into PrimoCache's RAM cache. It works as expected, and VLC reads from the RAM cache. So all I need is larger blocks so there is not constant disk activity.
As far as I am aware, windows is file level caching, if you have enough free ram windows should cache the file by default, do you not have success with that? If not, I think you can still use primocache to achieve what you wanted, with a combination of a simple python script or program to read your desired file in chunks, in this case, like you mentioned 10MB, either timed, or tracking disk acitivity. It's definitely not as straight forward but a workaround, but it doesn't sound hard to achieve, but checksum reading entire file all at once is a easier approach imo.
Post Reply