A VFX usecase? SSD+2xHDD with mirroring and speed possible?

FAQ, getting help, user experience about PrimoCache
jacksonwallop
Level 2
Level 2
Posts: 5
Joined: Sat May 01, 2021 3:11 am

A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by jacksonwallop »

Hi all, I appreciate the input.

I'm curious what you'd recommend for this usecase. I have a VFX workstation that reads/writes large and small files constantly (eg: 1000 images from an animation sequence, or multiple uncompressed footage files). Read speed is usually more important than writing (as rendering is usually the bottleneck when it comes to writing).
  • NVMe 1TB: Windows
  • NVMe 2TB: Storage
  • HDD 4TB: Storage
  • HDD 4TB: Storage
  • RAM 128gb
I realize an external networked raid would be ideal, but these are the current constraints.

I'm researching solutions that would let me combine the 3 storage drives into 1, take advantage of the SSD speed, and still allow for data mirroring across drives (similar to raid1).

Is this possible by using DrivePool with Primo Cache? Or should I be looking at other solutions?


Assuming this is a good avenue to pursue, what settings would be optimal?
I'm not sure about a few things here:
  1. DrivePool the 2 HHD storage drives?
  2. DrivePool turn on Pool file duplication x2
  3. PrimoCache task the 3 storage drives together? (not the new drivepool)
  4. Set the entire 2TB SSD as PrimoCache L2 storage? L2 Size set to MAX?
  5. If I dont want PrimoCache using RAM as cache, do I set L1 to 0mb? Or is it necessary to have it use some?
  6. Blocksize? A larger blocksize would seem to make sense given the smallest average files I deal with are a 2-10mb, aside from occasional text file. However, primo says block should match the cluster size or smaller? (usually 4kb right?) How do I check the cluster size of the L2 storage when it doesn't get assigned a drive letter? Dropping the block size makes the overhead skyrocket so...128kb looks like the happy place?
  7. Do i need to partition the L2 SSD 2TB Storage to only use 90% of the drive in order to maintain drive health? Something related to TRIM support...
  8. Enable Defer-write, 60sec Latency, Intelligent write mode, free cache on written?
  9. Drive Specs - should I uncheck L1 cache if I dont want primo to use RAM? or what's advised here?
  10. Other settings I've forgotten?
I really appreciate any guidance here, thankyou!
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by Jaga »

From a pure speed perspective, putting the two 4TB drives into RAID 0 and then using the 2TB NVMe as a L2 read/write against it would probably be the fastest solution. But RAID can be a pain in the arse - when it fails it sometimes destroys data in Striping mode. And if one drive fails, the whole array is lost.

If you used hardware RAID 1 for mirroring you could then put the 2TB on as a L2 read/write cache. Redundancy but less overall speed (for uncached blocks), and 1/2 the storage space. And again, the chance RAID or a drive could fail and crunch data.

My choice would be: If you use folder duplication via Stablebit DrivePool, it will write to one drive and then schedule the write on the other (1/2 storage space, but if one drive dies the other is 100% usable). And Primocache can cache drives that make up a DrivePool just fine, so again - a 2TB L2 against both DrivePool drives. You don't get raw RAID 0 speed, but you have complete peace of mind your data is safe, and you get a L2 covering both drives for read/write caching. 2TB is currently the max L2 in Primocache (I don't think the limit has been raised quite yet), and since it would be split between the two 4TB drives, you'd end up with 1TB L2 cache -per- drive. 25% coverage is probably just fine.

Stablebit Drivepool has it's own "write caching" mechanism, where you can specify a volume as the write target, and then later on the files that were temporarily written to it are moved to the target drive. It's not nearly as fast as Primocache's write caching however, so I'd recommend sticking with Primocache no matter which direction you go.

With the size of the drives you are using, forget about using a L1 to cache them, just use it to cache the boot drive. I'd only use a L1 if the machine in question has 16+ GB of RAM, any less and you're starving the OS/Apps for memory at the same time.

Block size depends on how much RAM the system can afford to waste on overhead. If it's got 32-64GB or more, set the block size to 4KB. If you raise block size, overhead goes down and you get a *tiny* loss of overall speed. You'd probably never notice it however.

And yes, *always* overprovision any SSD drive you use (either standard or NVMe). It really helps with lifespan on heavily used drives. 5-10% is usually adequate.

Only enable defer write if you are going to use a very short setting (5s or less), or you can use higher if you have a UPS on the machine so it never suffers from power loss. It also needs to be a -very- stable machine, so no BSODs or unexpected hard crashes or defer-write is going to make life worse instead of better.
jacksonwallop
Level 2
Level 2
Posts: 5
Joined: Sat May 01, 2021 3:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by jacksonwallop »

Jaga thanks for the reply, really appreciate it. I've come across some of your previous posts while researching this, which have been helpful!

Just a couple clarifications:
  • Over-provisioning: Is the proper way to specify this for the L2 cache would be to format the entire drive as 1 partition using Primo, but specify a number less than MAX in the L2 Cache settings. (MAX for me shows 1.8tb, i'll pick 1.6tb)
  • Level 1 Cache: I do not want RAM used for caching, so setting "OS Managed Memory" to 0mb accomplishes this correct? (the mouse over description seems counter intuitive to me, as in I want my OS to manage all my memory, not primo, but semantically this appears to mean the opposite in primocache as I understand it)
  • Is there a reliable way to measure the performance of a primo-cached divepool or do you just monitor cache hit rate in Primo and Pool Performance in DrivePool? Crystal Disk Mark? Somehow I don't imagine that test would get cached in way that would respond to the speed of the L2 SSD in the same way that regular usage would right?
Thanks again Jaga!
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by Jaga »

For over-provisioning, setup a volume using Windows' Disk Managment. Create a volume of 90% or so. You don't need to format it, but it won't hurt either. When you go into Primocache's L2 management area, you can pick that volume and Primocache will re-format it for it's use.

Correct on the L1 setup.

The best way to check performance on Primocache (in my experience) is to use a utility like CrystalDiskMark, and use a test data size smaller than your cache size. Empty the cache before starting it up of course, or the entire test data set might not fit. Hit rate is a good indicator of real-world cache performance, but only after many hours (or even days) of repeated use of the same apps/games. The first time you launch any software that the cache has no experience handling, hitrate will suffer. But through repeated use, that data makes it's way into the cache for higher hit rates.

With a read test in CrystalDiskMark (for example), read speed is calculated by reading a set of data the program *just* wrote to the drive. So in the case of using Primocache, it should reside on the L2 if it was a small enough set of data, and then the read speed should be boosted accordingly. Write speed is a no-brainer, it will always hit the cache immediately (if you emptied the cache prior to testing) and had defer-writes enabled. Longer defer-write time will help the test since the cache won't be dumping to the target drive mid-test, but again... UPS on the PC! :)
jacksonwallop
Level 2
Level 2
Posts: 5
Joined: Sat May 01, 2021 3:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by jacksonwallop »

CrystalDiskMark used on the Drive Pool volume with PrimoCache gave me the attached results (259mbs read / 174 write)

This was when the pool was at 56% full. At 1% full, a test returned 1900MBs/3770MBs/360MBs/75MBs read for the same tests, 220 write.

Stand alone NVMe returns 6700/4100/480/81 read, 4950/4234/400/193 write.

I'll try using the drive pool primo cache setup for a week, see how it feels, but if crystalmark is accurate that's kind of a bummer :/
Attachments
CrystalDiskMark_.png
CrystalDiskMark_.png (22.74 KiB) Viewed 1545 times
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by Jaga »

Did you tell Primocache to cache the Pool, or it's underlying drives? And after ensuring it's caching the underlying drives, have you tried a test against just one of those drives instead of the pool?
jacksonwallop
Level 2
Level 2
Posts: 5
Joined: Sat May 01, 2021 3:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by jacksonwallop »

Primocache is pointed at the two underlying drives.
Here's a test from one of the two drives, and from the pool volume.

S is the pool volume
E is one of the HDDs in the pool.

Only difference I saw during the test was the cache hit rate on E went up to 80%~
Attachments
Test_S.PNG
Test_S.PNG (141.95 KiB) Viewed 1528 times
Test_E.PNG
Test_E.PNG (164.33 KiB) Viewed 1528 times
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by Jaga »

I think I know what may be happening for you. Since the L2 cache is set to Read Only, any writes to the drive bypass it entirely. So the benchmark creates a test file (which has never been cached), then reads it directly from the drive (into the cache). You aren't getting "cached speeds" with Read Only strategy on your benchmark, at least for the first pass.

Try changing the Cache Task to 100% shared Read/Write, then clearing it and re-testing against the E: drive. I wager your performance will go up along with a better hit rate.

Couple of other suggestions to help Pool performance as well: In DrivePool, click on "Manage Pool" and then on "Performance", and check the following:
- Bypass Systems Filters should be "on"
- Read Striping should be "on"
- Real Time Duplication should be "off"

Disabling Real Time Duplication helps Pool performance, but delays the duplication until a later time. That's a trade-off you have to figure out if you are comfortable with.

At one point in the past I tested Primocache with a L2 against my Pool (9 8TB drives), and it seemed to work fine. Not as fast as a raw drive, but certainly much improved. Keep working it and I'm sure you'll find the sweet spot.
jacksonwallop
Level 2
Level 2
Posts: 5
Joined: Sat May 01, 2021 3:11 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by jacksonwallop »

Nice I think that was it! At least for the Read speeds.
- Turning off Individual read/write cache space made the read test go to 5500/5700/1800/76 MB/s, writes are still at 175~
(similar test numbers on the pool and the underlying HDD)
- In Drivepool, turning off Real Time Duplication dropped the read speed down to 4500~ fwiw

Do you think write speeds can be improved? I was hoping there was someway for this cache/pool combo to write to the SSD first and then shuffle the data to the slower drives later. Is that possible? The CrystalMark tests at least don't seem to show that happening.

Manually copying a few 1gig files from a SSD to the Pool happens at 1300 mb/s then drops down to 170~ half way through (according to the windows explorer)
The same files moved between two SSDs transfers at 2000 mb/s steady. Same speeds if duplicating files on the NVMe.

Manually copying 10,000 files ranging from 300kb-1mb from SSD to the Pool transferred at 350-380 mb/s
The same files merely duplicated on the NVMe moved at 500 mb/s, sometimes dropping to 150~

Not sure what to make of this or if it could be improved. Let me know if you have any ideas, thanks Jaga!
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: A VFX usecase? SSD+2xHDD with mirroring and speed possible?

Post by Support »

To improve the write performance, you have to enable the Defer-write.
Post Reply