Help with setup 256gb ram 48TB HDD drive for video storage

FAQ, getting help, user experience about PrimoCache
312UAV
Level 2
Level 2
Posts: 5
Joined: Mon Oct 29, 2018 8:39 pm

Help with setup 256gb ram 48TB HDD drive for video storage

Post by 312UAV »

Hello
I have tried several different setups but none seem to work well.
Here is what I am trying to do.
I am backing up terra bytes of data (video and video projects) to HDD for storage.
My computer is a HP Z840 win 10 64 with 256gigs of ram. Its an editing station so its packed.
The HDD's that I am backing off too are Seagate red 8tb with 256mb onboard 5400k.
The Reds are mounted internally, once full of backups I swap them out with another etc.
I tried using 30gigs of ram as my cache in the program as read for the red and 10 gigs of ram for the write from the raid.
(I have tried 30 30 for both same results)
folders sizes that I am copying over from raid to red are anywhere from 100 gig to 400+ gigs in size.
Everything goes well for about half the data 300Gb per second to 200Gb a second then boom, back to 0 then 110 then 0 then 50 then 0 then 100 then 0 then 15Mbs. and so on.
Its not much better if I dont use the Primo....?
Doing this production work for years, I have done all the windows settings one needs to do in optimizing the system for a high-end machine and drives etc.
Normally I would be backing off to tape but this is my home system and HDDs are or should be faster to do this with.

What am I doing wrong in primo cache?
Thank you
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Jaga »

Your data-to-cache size is simply skewed too heavily, the data is far too large to be 'cached' for anything other than write purposes. That's why you see speeds start high, then nosedive once the cache is overwhelmed.

I'm not sure a cache will help you much when you read/write 100-400 GB at a time, unless you setup a cache that's around 200+ GB in size.
312UAV
Level 2
Level 2
Posts: 5
Joined: Mon Oct 29, 2018 8:39 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by 312UAV »

So the idea would be keep the data size that you are transferring slightly than the cache size?
I could set the cache size to 200gigs then transfer less than 200gigs at a time...?
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Jaga »

Yep, that would work much better. It would give the cache time to read and/or write out fully, and also not have to resort to "urgent writes" (which you can see the stats for in the Primocache UI). If you get too many urgent writes, it means the cache is too full or overwhelmed and can't keep up, so Primocache starts bypassing it to write to the disk directly.

I don't think you mentioned if you're using deferred writes either - only have that on if the machine in question has a UPS on it. If it does and you are, then try setting the deferred write algorithm to "Buffer".
312UAV
Level 2
Level 2
Posts: 5
Joined: Mon Oct 29, 2018 8:39 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by 312UAV »

Ok I'll try that as well. Thank you.
here is another question if you don't mind.
I have 4x Samsung ssd drives about 2tb as a raid I am not really using
Can they be set up as a 2tb cache for the hdd"s to use instead if using my ram'?
.That would pretty much take care of all my cache needs.
Thank you again
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Jaga »

Absolutely they could. It'll go a long way to help your scenario.
312UAV
Level 2
Level 2
Posts: 5
Joined: Mon Oct 29, 2018 8:39 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by 312UAV »

How would that be done in settings of the program to use the ssd raid instead of the ram.
I can not find it in the instructions or a YouTube video.
Any help would be appreciated.
Thank you.
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Jaga »

You need to setup the SSD RAID as a L2 space for Primocache to use (by comparison a RAM cache is a L1). You can do both L1 and L2 on the same Cache Task if you want, but in your case I think *just* the SSD RAID would be better used as a write cache against the backup drive(s). You can use RAM as a L1 cache against your copy-from RAID drives if you want.

You need to provision that L2 write space on the SSD RAID first, there's a button up top in Primocache labeled "Manage Level-2 Storage". You'll want to just have a standard basic partition on the SSD RAID, then click that button and setup the SSD RAID as a L2 cache (click the volume, then Create Level-2 Storage). A tip: Leave about 10% free (unused) total space, so that each drive's garbage collection can run as a pseudo-TRIM.

When you setup a new cache task for writing to the Reds (add them in the selection dialog), tell it write strategy only. Then don't give it any memory for the L1, and set the L2 to point to the space you just provisioned. Tell it to use MAX space, then click the Advanced Level-2 Cache Options button to the right. Turn on "Individual Read/Write Cache Space" if it lets you, and move the slider all the way to the right (so it's a write cache only). You can also turn on "Volatile Cache Contents" if you want, so that the L2 write cache is cleared on shutdown and not preserved across reboots (a write only cache wouldn't need to keep anything it had written out). Make sure Prefetch Last Cache is disabled, since write only caches don't need to read.

That should force anything going to the Reds to be written to the SSD RAID first, and then it will send that info to the Reds as fast as they will take it. When you shut down, that L2 should be flushed and cleared, and ready for the next boot.

Note: I haven't used this particular method myself, but it should be easy enough to test on your end. Just try a few copies with it, see what the speeds are like, and watch the Primocache UI. The L2 space used in that cache should slowly go down as it's flushed to the drives, finally stopping when totally free again.
312UAV
Level 2
Level 2
Posts: 5
Joined: Mon Oct 29, 2018 8:39 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by 312UAV »

Got it and thank you very much for the details Jaga.
Have a nice night
Stan
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Support »

As Jaga suggested, it's useless to setup read cache in your case since your backup process only read data once. The only thing can be improved by PrimoCache is the write performance by enabling Defer-Write. Of course, in order to avoid data loss with defer-write, a UPS and a stable Windows system is required to help against the case of sudden power loss, system crash or hang.
Since you have heavy write load and L1 cache is not big enough (without L2), the write mode "buffer" is recommended. As Jaga said, "urgent write" shall be avoided. You may reduce write latency or choose the "buffer" mode.
Now, I see you have a spare 2TB ssd raid. Yes, it is able to use it as write cache for the HDD. Jaga suggest using L2 only, but I suggest L1 + L2 as RAM has a better write performance than SSD (raid).
Post Reply