Page 1 of 1

Pointers / Suggestions on new setup

Posted: Mon May 03, 2021 6:50 pm
by uniquepleb36232
Hello

Looking for some generall pointers/tips/suggestions on my planned build for my workflow.

TLDR: A bunch of NVM's mirrored with direct PCIe-CPU access would be preferred, but I don't hate my money.
(I've got backup covered, all redundancy for me is to save time/hassle, uptime/backup is not the issue)

For my storage solution I've read up on all kinds of solutions. I did consider SSD cache with MS Storage Spaces, but long story short I've concluded It's not worth my time and effort as this point. Other then the basic storage spaces volume (no SSD cache, +++) it is still badly documented, full of issues and not officially supported on non-datacenter Win. Things like TrueNAS and other solutions with ZFS and such require pass-through solutions which I prefer to avoid at this point (KISS - Keep it simple stupid)

That being said, I assume I can use a Storage Space mirror volume and use that with PrimoCache?

In theory StabileBit Drivepool looks interesting for my use case. However it seems a bit exotic and using that with my use case would/could generate all kinds of weird issues.

I'm still getting reading up on PrimoCache solution, sorry in advance if my use case is obviously stated in documentation/FAQ.

My use case:
"Power workstation", Win10 running and labbing with VM's, Hyper-V (Win10), (alternatively/also vmware workstation). For me, the host OS is mostly just as a management OS with GUI support and easy/non-complicated GUI support for utils/programs/tools which would be a hassle to run on core Hyper-V2019 or Win2019ServerGUI.

What I want:
  • Higher IO on non-ssd pools/volumes, everything else (transferspeed) is a appreciated bonus.
  • non-exotic, low-maintenance, low-knowledge solution. (AK47>ICBM)
My workflow:
Host OS (Win 10 Pro) is used as management OS for VM's/RDP/VNC. Instead multiple VM's/dockers are running for example Plex, torrenting, Daily use "Office" VM and so on. Meaning storage and disk access will be on large to very large files VDI files (10 GB - 1 TB, Typically VHDX), any smaller file access would be within VDI files and mostly program/os related (generally very little workflow on small files).

PrimoCache works as a virtual disk "man in the middle" ? Running PrimoCache locally on the guest VM's with VHDX volumes might be better?

Specs:
  • Ryzen 5950x (16/32)
  • 128 GB memory
  • 500 GB SATA SSD
    Crucial MX500 - CT250MX500SSD1
    Physical 4096
    Logical 512
  • 500 GB SATA SSD
    Samsung EVO 850
    Physical 512
    Logical 512
  • 2x 18 TB SATA WD Ultrastar HC550
    WDC WUH721818ALE6L4
    Physical 4096
    Logical 512
  • M.2 - NVM 256 GB
    I assume 4096 / 512, can/will find out later
  • M.2 - NVM 1 TB
    I assume 4096 / 512, can/will find out later
(SATA on 6Gbs over chipset bus, one M.2 over chipset bus, one M.2 on CPU PCI-e)

Scenario:
2x 500 GB SATA SSD in stripe volume on Storage Spaces
2x 18 TB SATA HDD in mirror volume on Storage Spaces
The 1 TB striped storage spaces volume as a PrimoCache for the mirrored storage spaces 18 TB ?
Or
Use the 1 TB M.2 NVM as a PrimoCache, I assume CPU-PCI-e lane is better.


I'm still getting my head around the 512 / 512e / 4kn physical disk thing. Apparently SSD caching in Storage Spaces can cause significant performance drop (Source:http://jeffgraves.me/2014/06/03/ssds-on ... rformance/). Any thoughts about this regarding if something like this affect PrimoCache setup?

I might be going over to Hyper-V 2019 Core (free) at a later stage. Does PrimoCache support install on the mostly "non-gui" Hyper-V 2019 Core ?

All suggestions are appreciated.

Re: Pointers / Suggestions on new setup

Posted: Thu May 06, 2021 8:51 am
by Support
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm I assume I can use a Storage Space mirror volume and use that with PrimoCache?
Yes
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm PrimoCache works as a virtual disk "man in the middle" ? Running PrimoCache locally on the guest VM's with VHDX volumes might be better?
Running PrimoCache in VMs should be faster than running it on the host machine. But the former needs more licenses.
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm The 1 TB striped storage spaces volume as a PrimoCache for the mirrored storage spaces 18 TB ?
Or
Use the 1 TB M.2 NVM as a PrimoCache, I assume CPU-PCI-e lane is better.
Storage spaces use 4KN sector format. Currently PrimoCache doesn't support 4KN drives as level-2 cache. So I think you need the latter one. Besides, the M2 ssd is faster.
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm Does PrimoCache support install on the mostly "non-gui" Hyper-V 2019 Core ?
Yes, but you need the PrimoCache Server edition.

Re: Pointers / Suggestions on new setup

Posted: Thu May 06, 2021 8:51 am
by Support
Support wrote: Thu May 06, 2021 8:51 am
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm I assume I can use a Storage Space mirror volume and use that with PrimoCache?
Yes
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm PrimoCache works as a virtual disk "man in the middle" ? Running PrimoCache locally on the guest VM's with VHDX volumes might be better?
Running PrimoCache in VMs should be faster than running it on the host machine. But the former needs more licenses.
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm The 1 TB striped storage spaces volume as a PrimoCache for the mirrored storage spaces 18 TB ?
Or
Use the 1 TB M.2 NVM as a PrimoCache, I assume CPU-PCI-e lane is better.
Storage spaces use 4KN sector format. Currently PrimoCache doesn't support 4KN drives as level-2 cache. So I think you need the latter one. Besides, the M2 ssd is faster.
uniquepleb36232 wrote: Mon May 03, 2021 6:50 pm Does PrimoCache support install on the mostly "non-gui" Hyper-V 2019 Core ?
Yes, but you need the PrimoCache Server edition to run PrimoCache on Window server operating systems.

Re: Pointers / Suggestions on new setup

Posted: Tue Mar 21, 2023 2:14 pm
by 804551
I use Windows Server 2016 with Hyper-V role. Several Hyper-V virtual machines with corresponding VHDX files are placed in HDD, which size is 18 TB. Size of every VHDX is about 4-6 TB.

I`ve never used PrimoCache before. So is it worth in my situation? Where should it be installed? In the host machine or in every VM? What performance rise does it offer? Faster moving files in VM`s? Or better latency in the host?

Re: Pointers / Suggestions on new setup

Posted: Thu Mar 23, 2023 2:48 pm
by tverweij
I run Primo Cache for years on HyperV (Windows 2019)
I use data deduplication to make sure that everything is stored only once - for me this saves about 60% storage and this improves the caching ability a lot.
I run Primo Cache on the host - Primo Cache is for me the most important reason to never look at ESXi, as it won't work on that.

Warning about Data Deduplication: Windows 2019 is the first one where this is 100% stable.
Same goes for ReFS that also improves the speed of HyperV,

My cache on the VM drive is 64 Gb, the drive itself is 3.37 Gb Netto (raid10) and I have stored 4.82 Tb on it, leaving 1.2 Tb free (Dedup rate: 55%)
I use write caching as my servers run in data centers with a 100% power guarantee.

But always make sure that everything is backed up regulary (and the backups are tested).

For the questions about what the results are: Everything in the VM's becomes faster then it was without Primo Cache.

Re: Pointers / Suggestions on new setup

Posted: Thu Mar 23, 2023 3:00 pm
by tverweij
Some more:

- I only use L1 cache as my diks are all enterprise SSD's in Raid 10 (all internal disks) and are formatted with ReFS- very fast on itself, so L2 won't improve that.
- Running it in the VM's not only cost more licenses, but also more memory - for me not an option as memory is one on my primary bottlenecks.
- If you have more than one numa node on your CPU's, make sure that primo cache runs in Numa mode (much more stable and also faster)
- Settings I use for Primo Cache: Shared R/W cache, average write, 1 sec latency (only do the write behind when your power is 100% guaranteed or when your data isn't worth much and only when your backups are 100% in order)

Re: Pointers / Suggestions on new setup

Posted: Thu Mar 23, 2023 3:03 pm
by tverweij
On last one: test your setup with different block sizes on an idle system (all VM's off)
For my system, 32k blocks give the most acceleration (according to Crystal Disk Mark). But that depends on CPU speed and memory speed.