Win 10 Compact feature: Better Compatible? compression!

[Logic]

From my understanding PrimoCache works at the block level then it would use the XPRESS8K versions for example. The performance improvement will really depend on the hardware used. In some cases similar to Windows memory compression, it might be handy instead of writing it to the next tier storage. XPRESS4K from a quick test seems to improve storage only if the storage is slower than 1GB/s.

Thats correct steveb

Theres a vast array hardware to consider, but as a general rule of thumb on a 4+ core system faster I/O is found with:

• Xpress16K for HDD
  Xpress8K for SATA SSD
  Xpress4k for NVMe SSD

Readyboost uses Xpress ?K. Probably Xpress4K?
Someone at MS decided: "This compression works very well! Why don't we do the same thing in RAM too!?"
So in Win 10 Superfetch/Sysmain now has 3 cache levels:

• L1: Uncompressed in RAM
  L2: Compressed in RAM
  L3: Compressed to Readyboost if ~random 4K -AND- Compressed to Pagefile if Sequential.

MS wouldnt bother if Xpress compression didnt improve I/O speed to disk...

Support seems to be saying that Primocache is somehow tapping into this compression to compress the data in L1 and L2, but, as usual, does not expound on the subject!??
Maybe he's just trying to shut me up?
He's going to be a brilliant politician one day!

[Support]

Support seems to be saying that Primocache is somehow tapping into this compression to compress the data in L1 and L2, but, as usual, does not expound on the subject!??

I mean if Windows or other applications enable compressions on target drives, PrimoCache L1/L2 also stores compressed data, not uncompressed data. The feature to able to compress data is on our todo list but at a low priority, because to support this feature we have to redesign the base programming architecture which is a big change.

[Logic]

I mean if Windows or other applications enable compressions on target drives, PrimoCache L1/L2 also stores compressed data, not uncompressed data. The feature to able to compress data is on our todo list but at a low priority, because to support this feature we have to redesign the base programming architecture which is a big change.

Ah... Thx Support

IMHO there's no need to change the base programming:
Microsoft's CompactOS and Compact is already in Windows 10. (There's an app, with driver, that works in Win 7 too)

People have already developed simple GUIs for it. (I wont post links again as you aren't going to bother to look Mr Politician! )
I use Compactor by Freaky.

So all that's needed is GUI, an exclusion list and a tester to see if said files are worth compacting as Freaky has done.
Also; My initial research points to data being written in 64K blocks, so it may be an idea to set Primo's block size accordingly.
I will post once I get back home and experiment.

The same could be said for the MyDefrag source code, which is ~ 15% faster than the stock Win Defrag.

One wants ALL I/O optimised in any i/O Optimization software..!?

[steveb]

Microsoft Compact works at the file level not at the block level like Primocache. From my understanding compression for L2 would need to be different to Microsoft Compact. Ideally Romex software should have compression like features for even after reads. Even an old Samsung 960 Pro out performs Xpress4k. If all you need is Compact compression support use one of the tools that already exists on top of primocache. E.g. For spinning disk

[RobF99]

Just compress the files/folders on your spinner. That is what I do. Since PrimoCache works at the block level, the data in L1 and L2 will be compressed. This way you can get on average an extra 50% out of L1 and L2. An important item to consider is that compression is subject to significant fragmentation. So if you compress files, be sure to defragment after you compress. It is not uncommon for a 1 Gb file to be split into 10,000 to 20,000 fragments after compressing and yes, this degree of fragmentation will slow down your L1 and L2 reads even of the fastest processors because with reads, even though cached, the O/S still has to deal with this many clusters and this creates CPU overhead. It will also waste L1 and L2 storage depending upon your block size since 20,000 fragments may require up to 20,000 blocks in L1 and L2 at whatever your block size is. I have done significant testing of this. Also don't bother to compress uncompressible data since it will slow down performance of L1 and L2, read and write.

To do this:

1. Compress the files in the folder

2. Then uncompress the noncompressible files
    i. compact /u /s /i *.jpg
    ii. compact /u /s /i *.gif
    iii. compact /u /s /i *.png
    iv. compact /u /s /i *.zip
    repeat this for all files that you know are uncompressible

3. Defragment. (if you use UltimateDefrag, you can just right click on folder and select defragment, and it will only defragment the files in that folder. I don't know of any other defrag ultility that will let you defragment just one folder)

4. You should be all set and all read compressed data will be stored in L1 and L2 compressed.

[InquiringMind]

...if you use UltimateDefrag, you can just right click on folder and select defragment, and it will only defragment the files in that folder. I don't know of any other defrag ultility that will let you defragment just one folder

CCleaner's Defraggler is one (free) alternative offering this. However, since each defragmenter is likely to have slightly different ideas of the "perfect" file layout, it is best not to mix them - just choose one defragmenter and stick with it.

[Logic]

From my understanding PrimoCache works at the block level then it would use the XPRESS8K versions for example. The performance improvement will really depend on the hardware used. In some cases similar to Windows memory compression, it might be handy instead of writing it to the next tier storage. XPRESS4K from a quick test seems to improve storage only if the storage is slower than 1GB/s.

Thxx for testing steveb

With all the different hardware/configurations out there it's very hard to say what the best configuration will be in general.

The rule of thumb with modern processors:
16K: HDD
8K: SATA SSD
4K: NVME SSD
makes sense as the slower the drive; the more time the processor/s has to decompress the data into RAM and still come out faster than stock uncompacted I/O.

I'm still trying to confirm this but it seems the 4K, 8K and 16K is the compression block size, so aligning that with:
1: Cluster size
2: Primocache's block size
may well increase performance by quite a bit...?

[Logic]

RobF99 has tested Windows 10 Compact (Xpress4K to 16K) here:
viewtopic.php?t=5410

Besides more than halving the space used on his HDD and doubling the amount of data that fits in his L1 and L2 caches;
He shaved 2.2 seconds of load times (averaged) and halved the load times for uncached files on his HDD!

Thx for testing RobF99

[Logic]

Confirmed:

• Xpress4K compresses data in 4KB chunks.

• Xpress8K compresses data in 8KB chunks.

• Xpress16K compresses data in 16KB chunks.

• LZX compresses data in 32KB chunks.

https://docs.microsoft.com/en-us/window ... on_info_v1

This means that matching the Xpress 4K, 8K, 16K to the same cluster size and Primocache Cache Block Size may well have a performance benefit..?
Also the Page Size of SSDs?:
https://site.aleratec.com/blog/2011/09/ ... drive-ssd/

[Ushavilash]

Hey hi, optimize file compression choose XPRESS algorithms based on your drive type and freaky compactor for GUI, image compression use Jpeg Compressor and also check compatibility with primocache deferred write option to avoid crashes.