Romex Software Forum

RobF99 wrote: Thu Jan 20, 2022 12:41 pm HDD was 4K, Primocache was 32K. As far as I understand the block size in PrimoCache doesn't affect performance at all. Instead it determines the amount of data surrounding the cached block that is cached. I confirmed this by lining up the files from the folder on the drive consecutively and when I used 512Kb it cached a fraction above the amount of data in the folder. When the files were randomly about the drive it cached almost twice the amount of data because it was picking up many small blocks and each of their their surrounding 512Kb of data blocks.

Interesting...
It seems PrimoCacche use data locality to pad out the 512kb block.
ie: If the requested 4kb of fata is at position N; also read block N+1, N+2, etc to fill the 512kb block with data that's most likely to be required.

But... perhaps you have this backwards?:
ie: When your data was all sequential, the 512kb block was filled with relevant data, with only a small amount of extraneous, unwanted data ending up in the cache.
Whereas with your randomly placed data; a lot more extraneous data ended up in the cache..?

RobF99 wrote: Thu Jan 20, 2022 12:41 pm I tested 512Kb block size down to 8 Kb and it made no difference to performance regardless of compression method and block size. It so happens that for my processor the sweet spot for transfer rates and compression are at 16 Kb with XPRESS16K.

Er... I'm confused?
Block size made no difference
but
"...the sweet spot for transfer rates and compression are at 16 Kb with XPRESS16K..."
???

RobF99 wrote: Thu Jan 20, 2022 12:41 pm How do I [split my SSD RAID 0 array, Setting 1 drive to cache reads and the other to cache writes] since you can only have 1 L2 cache for each drive you are caching? Or am I missing some neat trick in PrimoCache?

Oops! My apologies:
It's been too long since I've actually been in front of my system, so based on how configurable PrimoCache is is, I assumed...
But!
This idea makes for a very good suggestion!

(I've been out 'In the Sticks' in 'Darkest Africa' for way to long! Thinking on subjects like this keeps me sane in a place where;
they chop down the Mango trees this season, then complain about the lack of Mangos/hunger next season..! )

RobF99 wrote: Thu Jan 20, 2022 12:41 pm Yes I discovered this program [Freaky Compactor] just a few days ago. It does what it does very well!

...

Rob you posted about NTFS Compression on Sun Sep 19, 2021, in my:
Win 10 Compact feature: Better Compatible? compression!
post, dated Fri Sep 17, 2021.

That's a good 2 and a half months after my post on Windows 10 Compact/OS where I linked both Freaky Compactor and CompactGUI:
viewtopic.php?t=5327

I get that people like their peers to think that a good idea is theirs... but come now!
Credit where credit's due wont kill you...
My having given you credit/praise for testing my idea didn't kill me...

Elrichal wrote: Sat Jan 15, 2022 3:55 pm Thank you both Rob and Support for your studies and results, they are very interesting and informative reads.

Support DID NOT post the linked initial post on Win 10 Compact/OS and PrimoCache, I did..!

RobF99 wrote: Wed Feb 23, 2022 5:53 pm Yes, the column with Windows Read Cache are the Windows cache results. So compare 867 with 1121, and it is 23% less.

After more experimenting the reason the systems became corrupted were that a few apparently critical files most likely appeared corrupted to Windows while they were not corrupted just compressed. One of the error related to a key Windows Defender file being compressed (was in programdata folder) and another were some sys files in the System32 folder. I do not recall which files specifically.

That is why I advise to stay away from the programdata folder and the Windows folder. Leave programdata alone, and to compress the Windows folder just use Compact /CompressOS:always and Windows will take care of the correct files. You can then go in if you like and Compress files that won't be a potential problem such as .xml, txt, inf, pnf, ini etc. e.g. Compress /s /c /i /EXE:XPRESS16K *.xml. These are all regular data files. I had no problems even manually compressing all DLL files in the Windows folder. It seems to just be one or two pesky .sys files and that Defender folder that caused the issues.

I have 5 systems in this state, and they are all 100% stable! No blue screens, no random crashes, no data corruption. And they are all fast, fast, fast!

Yes, you will not benefit from already compressed files you regularly work with and yes, your OS and even your applications will benefit from this compression. Compared to NTFS Compression, Microsoft really got their act together with these awesome algorithms.

Apparently a list of files that CAN NOT be compacted is to be found in WimBootCompress.ini or WimBootReCompress.ini found in the Windows\System32 folder.

WimBootCompress.ini is created during WinNTSetup installs on Wimboot mode.
and
WimBootReCompress.ini is created during WinNTSetup installs on Compress mode.
http://reboot.pro/topic/22007-wofcompre ... /?p=211055


If you are into compressing everything that can be compressed Rob (game texture files etc), see:
https://sourceforge.net/projects/nikkhokkho/


Support:
If PrimoCache is in any way interested in integrating this, the open source wimlib compression algorithms generally outperform their windows equivalents:
https://wimlib.net/compression.html#Benchmarks

Hey hi, Use window compression options like XPRESS16K for enhanced system performance without additional software like PrimoCache. Be cautious of potential risks and check adequate RAM for caching. This approach is beneficial for static data files, reducing storage space while maintaining accessibility in native Windows format.

Ushavilash wrote: Fri Apr 05, 2024 11:49 am Hey hi, Use window compression options like XPRESS16K for enhanced system performance without additional software like PrimoCache. Be cautious of potential risks and check adequate RAM for caching. This approach is beneficial for static data files, reducing storage space while maintaining accessibility in native Windows format.

Exactly!
If the underlying drive is faster without caching; its faster when there's a cache miss. (which fits with PrimoCache's faster I/O business model...)
Also; because there's ~2X more data in the cache: the chances of a cache hit are greater.


Then there's all that fun: Match Memory block to cluster, to sector, or page and/or erase block size. (which is reported by the correct storage query commands nowadays)

NB these Atto benchmarks for memory read speeds going up with file = data block size.

https://www.unbxtech.com/2020/02/ram-di ... ained.html

That's counter to the conventional wisdom of '4K is fastest' here, BUT 4K happens to match the:

• Physical 'sector' size of both DRAM and most HDDs.

• The reported sector size of most SSDs (Real physical Page and Erase Block sizes are bigger)

If the minimum block size taken per data block is bigger everywhere... and 'matches' the real, physical, HDD sectors, or un-fragmentable NAND Page Size..???
(Erase Block size ascertained for write caching)
(SSD and RAID parallel channels need to be kept in mind too)


Admin:
This may not make as much sense or be as interesting to you as may well be to the dev/s..?
Do run the idea/s past him/them...
Tell them there's a crazy Microsoft Certified Systems Engineer (ex Netw Admin) that thinks faster overall I/O is your business and [the above].
He'll thank you for taking his mind off of anti-crack for a bit!