possible to benefit from PrimoCache for both host and VM?

[quirken]

What sorts of considerations will I need when using PrimoCache alongside virtual machines?

My specific use case:

• Host OS could be Windows 10 or Linux (leaning towards Linux)
• Single Windows Guest OS used exclusively for gaming
• Most of the host OS's data will be on the SSD, so it's not particularly important for the host to benefit from PrimoCache. All the large files on HDD I want to cache are basically Steam games, which will be in Guest.
• Host OS's performance is actually less important than guest OS (but due to some stupid stuff on how Steam In-Home-Streaming works, I need a copy of Windows with lower-than-usual security). Host OS will be primarily web browsing / productivity apps. Nothing super intensive.

Hardware:

• 500GB SSD
  • Partition 1: Host OS
  • Partition 2: Guest OS
  • Partition 3: PrimoCache L2 Cache for HDD
• 2TB HDD (for user data storage and Steam game installs)
• 16GB RAM

Option 1:
In a best case scenario, the Host OS would be running PrimoCache, and the guest OS would benefit from it yet be oblivious. But given the way virtualization is designed specifically to prevent VMs from interacting with the Host, I would not be surprised if this was impossible. Or does the low-level nature of PrimoCache mean that files frequently accessed inside a VM would get cached as expected (as long as everything was NTFS)? Would running two copies of PrimoCache, one directly on hardware and one inside a VM, be a good idea or would that just be a mess of conflicts?

Option 2:
If my host OS is Linux, my guest OS is Windows, and I install PrimoCache inside the VM, barring some minor overhead, will I still gain most of the performance benefits inside the guest OS? Rather than using a file-based VHD, the entire partition (and cache partition) would be made available exclusively to the VM. So while there's a layer of abstraction, as far as the guest OS is concerned, there's a system partition and a partition for caching, which correspond 1:1 to real physical drives.

Option 3:
I'm aware there are similar things to PrimoCache for Linux. Are there any that play nice? Linux Host running _____, Windows Guest running PrimoCache, but shared cache in case access to HDD is needed.

Any other suggestions for my use case?

Thanks a ton!

[quirken]

thought of another way I could configure things...

Option 4:
Host OS is Windows+PrimoCache, guest OS is Linux. I can then remote desktop into the guest OS. This would probably be the best from a performance standpoint, although I would end up not using Steam in-home streaming anymore (which isn't necessarily a bad thing)

[Axel Mertes]

Hi Quirken,

if your host is based on Windows and you use PrimoCache on that one, every VM will benefit from the speed improvement.
In fact, most VMs use drive images, which tend to be very large single files, containing the actual image of the drive content, similar to an ISO image.
PrimoCache does not care about files at all, its all block based. So if if the VM image is on a drive that is being cache using PrimoCache, you will see the exact same speed improvements like on the host itself.

The only way to improve beyond that is to employ something like Condusiv Diskeeper or in your case better V-Locity, which is a version of Diskeeper aimed at VMs only. It optimizes file writes to write larger chunks of data and minimize fragmentation beforehand and less I/Os than without. The combination of both gives you a maximum performance.

Cheers
Axel

[quirken]

Thanks!

[Axel Mertes]

You are welcome!

I should add that V-Locity running INSIDE the VM will optimize file I/O performed by the VM and the software running within. So only an optimized I/O is requested to the physical host OS layer. Now running PrimoCache on the physical layer gives you cached read and writes. You may consider adding Diskeeper on the physical layer too. Diskeeper has caching on its own, but ONLY to RAM and only very few and only for reads. Its not ineffective, but compared to what I can do with PrimoCache its far behind. However, the optimizations in file I/O created by Diskeeper really add up.

I disabled Diskeeper caching, but left its file I/O optimization in tact, while caching with PrimoCache. Then I have daily defragmentation running on my volumes, so they are really clean and fast. I think you can't do anything beyond this. This is by far the best possible strategy to handle your storage. You will see a low cost storage outperforming many brute-force FC SANs this way, with magnitudes in price difference.

Cheers
Axel

[minhgi]

Adding just a tip when using diskeeper and vmware environment, enable VSS with large file efficient defrag for older version. That way it use a different defrag strategy optimize for i/o and shrinking of the VM image. Definitely disable ramcache if you are using diskeeper 16 or V-Locity.

[Axel Mertes]

You mean disabling ramcache due to using PrimoCache RAM L1 and SSD L2 cache?
That is what I do as well, just let Diskeeper optimize in the super low I/O transfers by combining smaller writes into bigger chunks.
Or why else would you disable ramcache?

[Axel Mertes]

Quirken, I just re-read the original post and think I need to clarify a few things:

1. PrimoCache, Diskeeper and V-Locity are all pure Windows based products. They will not be available on Linux systems.

2. The only way to let the Linux system benefit from caching is by having the host OS running Windows and the VM running Linux. The VM file should be on a harddisk that is being cached by the physical Windows host using PrimoCache.

3. PrimoCache can cache ANY filesystem, as is pure block based. Even if you have a true partition in Linux, you can still let that cache by PrimoCache.

So if you are really in need of a Linux OS for productivity and also want a fast Windows for gaming I would recommend the following:

Host OS: Windows
VM OS: Linux
VM either on a true Linux partition on its own drive OR true Linux partition on the same drive as the Windows host OS OR Linux partition inside a virtual disk (either on a dedicated drive or on the Windows host OS drive).
Make sure that both the Windows OS AND the partition containing the Linux partition or virtual partition are being cached by PrimoCache.
Add an SSD for L2 caching. The size should be about as big as the largest total amount of data you usually touch on a single day.
I recommend using NVMe over SATA SSDs whereever possible. If you don't have a NVMe port, but a free PCIe slot, you can buy NVMe to PCIe adapter boards for lower than 20 US$/EUR. SATA SSDs are typically peaking at 550 MByte/s (Samsung 850 EVO), while NVMe goes up to 3500 MByte/s (Samsung 960 PRO). The difference is HUGE. Another option is the Intel P900 Optane Intel Crosspoint memory card, but its far more expensive (but a lot faster).

Let us know which way you went

Cheers
Axel

[quirken]

1) there are linux tools that do similar things, but likely with more of a headache to configure. The linux installation is probably going to be primarily accessing data on the SSD anyway, so I'm not as concerned (my main use case of primocache is likely decreasing game load times, as games are installed on a HDD, but most other programs on SSD)

2) wouldn't I get better typical performance if the linux vm was on the SSD? It wouldn't benefit from primocache, but a L2 cache would be pointless and I don't imagine L1 cache would really be noticibly helpful for web browsing / etc

Optane would require a new computer they require a newer processor

The thing complicating this setup (and the reason I'm leaning towards linux host) is the way Steam's in-home streaming works. You need to have your machine be unlocked in order to stream a game to the other room. It basically functions similarly to remote desktop (with an extra GUI to make it easy to launch games). So my plan is to have the windows install automatically sign in when it is turned on.

Considerations:
1) windows performance matters more
2) windows will be insecure (so VM minimizes risk)
3) if a linux vm is running on top of windows, I need a way to prevent someone watching what I'm doing (or taking control via remote desktop), as the windows install is insecure.

If a windows VM had direct disk access (as in, there's a real NTFS partition on a physical disk, no extra abstractions), couldn't the VM still benefit from primocache (even if there was a slight performance hit)?

[Axel Mertes]

[..]
Considerations:
1) windows performance matters more
2) windows will be insecure (so VM minimizes risk)
3) if a linux vm is running on top of windows, I need a way to prevent someone watching what I'm doing (or taking control via remote desktop), as the windows install is insecure.

If a windows VM had direct disk access (as in, there's a real NTFS partition on a physical disk, no extra abstractions), couldn't the VM still benefit from primocache (even if there was a slight performance hit)?

If you manage to run PrimoCache inside a VM, it would surely accelerate disk access. Never tried, but I think in theory it should work. Condusiv managed to make Diskeeper work as V-Locity inside a VM, so why should it be different with PrimoCache?

That does not require the storage partition to be exactly NTFS (but mostly likely it will be that way for Windows).

You need to divide the SSD in two partitions IIRC. FancyCache (the "Beta" of PrimoCache) had the feature to open kind of a cache file on the SSD. PrimoCache requires a complete partition. FancyCache was a bit more flexible in this regard, but I think for performance reasons that option was canceled out for PrimoCache. I am not sure if its back now on 3.x - did not check for that yet.

For caching your games, you surely need the cache big enough to really have an impact, so a cache size bigger than the entire biggest game seems "required" to benefit from the cache.

For our use case - a server feeding video, image sequences, audio for visual effects post production - we went for measuring the average amount of data transaction per day and sized the cache according to that. In fact we took like twice as much to gain some head room. So during a normal day about everything we touch comes from the L2 cache, which keeps the disk storage quite free from access. So we have lots of time to run defragmentation, so we have never more than 1-2 fragmented files on volumes > 6 million files. Maximum performance when accessing physical disks!