Romex Software Forum

I would like to know if there's any way whatsoever to pre-cache a specific directory?

I know primo cache work with block storage and not the file system, but would there be a way to have a tool or something that would read all the files in a directory so they are then cache or something in that nature?

It would be great for my use case.

Thank you

Thanks fou your suggestions。This question is already on our list。

Sounds like Romex might be getting close to merging some of the functionality of Primo Ramdisk with PrimoCache. If you could tell PrimoCache to cache a directory using a specified amount of memory, and to reserve that memory from thereon out, you'd effectively have a Ram Disk, as long as PrimoCache prefetched it on boot-up.

But along with demanding more of their time for development and support, that would cut into their already modest profits, even if such a feature was limited. Might make for a nice advertisement of Primo Ramdisk for PrimoCache users though, and given how more and more people are getting more and more memory for their builds, I can see some PrimoCache users wanting a full blown Ram Disk.

It's both good and bad that AAA video games more and more require a huge install size. Better image quality can be had by larger textures, but it can be problematic to keep all of ones games on a SSD, let alone completely cached in memory. Spinning platter hard drives remain far from dead due to these humongous games, and humongous sized videos, and because their prices have gotten more reasonable over time.

I now keep my games and other data installed on a high quality 8TB HGST (owned by Western Digital) drive, and use 312 GB of an older 512 GB NVMe drive as an L2 cache for it, along with a generous amount of ram as an L1 cache. My new PC has two slots for NVMe drives, so I use the one closest to the CPU for my newer/faster NVMe drive, which has Windows installed to it. I haven't decided yet on how to use the free space on that 1TB drive. I might partition it, use the second one as a game partition for games that I'd keep in use for a long time, and which might show some benefit from it's extremely fast read speeds. I could always later move those games to my spinning platter drive.

But even with video games, as long as you have a healthy sized L1 cache, reloading levels is a fast proposition.

Here is a file you can download called readfile.exe. It just reads files and is used for testing disk speeds. The original build of this program from http://www.winimage.com/readfile.htm has a bug recursing subdirectories. I had a programmer correct this for me since it is open source.

You can get my modified build from here: https://www.dropbox.com/s/ysopra7ria9qp ... e.zip?dl=0

Just put it in the folder you want to precache and run readfile.exe *.* /s. I use it all the time to precache certain folders.

This is perfect for what I was looking to do. Enables me to pre-cache a game before running it.

I hacked together a quick batch file to make it easier to use on different games on my system.

All I need to do is copy the file path, leaving quotation marks in place into the location variable and click run.

Sharing as others will probably be able to improve on it, as I have no real experience of batch.

Or you can use it to hack together your own.

darryl14 wrote: ↑Sat Mar 20, 2021 11:29 am This is perfect for what I was looking to do. Enables me to pre-cache a game before running it.

I hacked together a quick batch file to make it easier to use on different games on my system.

All I need to do is copy the file path, leaving quotation marks in place into the location variable and click run.

Sharing as others will probably be able to improve on it, as I have no real experience of batch.

Or you can use it to hack together your own.

Code: Select all

@echo off

rem ##location of readfile.exe
set "readfile="C:\PATHTOFILE\readfile\readfile.exe""

rem ##location of directory to precache
set "location="D:\SteamLibrary\steamapps\common\PATHTOGAMEDIRECTORY""

rem ##copies readfile to location of directory to precache
copy %readfile% %location%

rem ## changes directory and executes readfile
cd /d %location%
readfile.exe *.* /s

echo FINSIHED
PAUSE

Maybe I can try writing a plugin for Playnite that does this. Not too familiar with Playnite yet (just started using it a couple weeks ago) but I think it is very extensible. Of course, the downside is you don't always want the ENTIRE GAME cached (for example, if you never play Multiplayer you probably don't want Multiplayer maps cached, or FMVs or other large files that are easily streamed even on HDD), so a naturally generated block-by-block cache would be better.
the ultimate feature for me would be a "File Offset" precache in Primocache. Basically, a mode to 'watch' the system, where you then run the game/application and play/use it normally, it inspects the game/app to see where IO slowdowns are occurring and generates a list of files/offsets that should be cached to accelerate the game and saves it as a profile. Then before playing you would "Apply" the profile, and it would precache those files/offsets.

Or.... just install the latest 7-zip and use the context menu entry marked "CRC SHA" and use the '*' option. It will perform a series of tests on the folder, files in it, and all subfolders which forces a read on the entire set of data. I do this and it manages to ask Windows to access all the data - whether that is placing it in the Windows cache, or Primocache.

I think we (I know I did) asked for this feature a number of years ago, so it's not anything new which is why Romex already has it on their list. But there's a short-term workaround using CRC SHA.

for the 7zip method i just run this in shell:common startup

@echo off
"C:\Program Files\7-Zip\7z.exe" h "G:\World of Warcraft\_classic_"

defo runs much faster the second time.

Even though I like to cache specific directories into L2 cache, I do not feel that Romex should change the nature of the program too much to support specific directories or programs.

The program works best because it is agnostic to software that is running and based purely on caching blocks based upon usage. I believe it would start to complicate the program and the programming and start to move away from the ultimate core nature of the program and that is that it should be agnostic to specific data and file structures except maybe boot files.

There are ways for us to cache directories such as the readfile and 7-zip tips below. Ultimately, when you do pre cache an entire folder, you are wasting resources because even a major complex game or program with tens of thousands of files really only uses a small percentage of those files. So you might be doing a lot of reading of all the files of a large program for nothing.

E.g. My windows folder has 39 Gb and 127,000 files. If I precached all that, it would be a big waste of L2 SSD writes. Your C:\Prefetch\layout.ini will show you approximately which Windows files are used the most and you will see that might only be around 3,000 files and we can see from boot up that usually PrimoCache has only read at most around 1 to 1.2 Gb by the time a system it is fully booted. There is no need to precache 39 Gb of Windows directory. The same principle applies with most programs you run.

I feel if we have that capability in the program and mess too much with it, we will find that the program will not perform like we want it to.

That is just my 2 cents worth.

I agree with this comment.

+1