Help with setup 256gb ram 48TB HDD drive for video storage

FAQ, getting help, user experience about PrimoCache
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Jaga »

support wrote: Wed Oct 31, 2018 7:08 amNow, I see you have a spare 2TB ssd raid. Yes, it is able to use it as write cache for the HDD. Jaga suggest using L2 only, but I suggest L1 + L2 as RAM has a better write performance than SSD (raid).
I just wasn't totally sure if when the L1 filled up with write requests, if it would "spill over" to the L2, or go directly into urgent writes instead. So I didn't suggest both for that reason. If the L2 is used when the L1 is overwhelmed with write requests, then absolutely use both. :)
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Support »

Yes, L2 will be used to buffer incoming write-data when L1 is full though L1 will do urgent writes to target disk.
viewtopic.php?f=33&t=4632#p12661
InquiringMind
Level SS
Level SS
Posts: 477
Joined: Wed Oct 06, 2010 11:10 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by InquiringMind »

If you're using the Seagate Reds for backup only, then there is absolutely no point in caching them at all - you are not going to be repeating any reads or writes and you do increase (albeit slightly) the probability of data loss in the event of a system crash or power failure.

The one suggestion I would make is to put your backup drives in external enclosures and connect these via e-SATA or USB 3.0+ in order to make it easier to change drives. An NAS cabinet is another option to consider though you'll want it connected via gigabit Ethernet at the very least (and 10GbE or multi-link GbE would be better since you'd presumably want 500+MB/s write speeds) since that can add RAID remote storage,.
zeroibis
Level 4
Level 4
Posts: 20
Joined: Thu Oct 11, 2018 11:13 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by zeroibis »

You may want to read this thread where I did a lot of testing in moving a mixed test group of primarily video files with various cache settings.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by mell111 »

@ zeroibis - which thread is that? Thanks.
zeroibis
Level 4
Level 4
Posts: 20
Joined: Thu Oct 11, 2018 11:13 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by zeroibis »

Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Axel Mertes »

Hey,

that is pretty similar to what I am doing here, film post production on large 64 TB RAIDs and using disk to disk backup for daily nearline backup purpose.

As Support and Jaga already mentioned, a read cache doesn't really help on your backup need, because you will likely only read the data once from the source disk. On the target disk side, L1 RAM + L2 SSD RAID + read&write cache + defer writes is highly recommended. And its not only the defer write that is speeding up the process, but the fact that the MFT of your target disk is held mostly in the write cache. If you transfer many small files (image sequences for instance...) then you will usually see a huge performance hit in copy operations, due to the fact the HDD read/write heads are flying back and forth between where the new data is written and where the MFT sectors are placed and where the MFT backup is placed (usually 1/2 disk size away...). In turn that leads to heavy load on head seeks and slows down HDD writes significantly. By using PrimoCache you can optimize that behavior strongly.

I just recently made a disk to disk copy from one 64 TB partition to another one using that method.

Btw, I highly recommend using software such as "Syncovery" for this backup process. It can handle incremental backups, versioning and all such neat things you need and it will easily handle really large datasets. You can even offload to cloud if needed/desired.

Using Syncovery you can create many individual backup tasks with very specific settings. You can let these run e.g. every night to make automatic backups of your current projects, including versioning of overwriting files.

Beside that do you create catalogs from your disk to allow offline search for files?
We use "Broken Cross Disk Manager" for this. It creates inventories of your disks, including contents within compress zip, 7z, rar etc. files plus it can handle LTO LTFS to catalog them as well (which is why we use it). Its not a MAM solution, but it works like a charm for me and is really low budget.

Do you regularly defragment your source disk?
We defragment our 64 TB partitions using space defrag and we usually on see less than 10 fragmented files during the day. This makes these volumes/partitions behave significantly faster. You may employ caching your source disk as well for this, as defragmentation speed is significantly better with PrimoCache enabled. However, your risk is always that write cache is currently at risk if you experience an unexpected crash.

Just a few ideas...

Cheers,
Axel
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Axel Mertes »

I should add the wish for PrimoCache to allow read-ahead caching in that context.

When data is sequential on defragmented files, it may improve read speed a lot if PrimoCache makes sure to preload a custom number of blocks BEYOND the last requested block.

Why?

Because whenever the next streaming block is requested it is then very likely it can be fed from the cache instead of waiting for disks to spin around.

I mentioned that before and I am quite sure it would help in this scenario too.

Cheers
Axel
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Support »

Axel Mertes wrote: Thu Nov 29, 2018 4:54 pm I should add the wish for PrimoCache to allow read-ahead caching in that context.
Sure, read-ahead is under our consideration.
Axel Mertes
Level 9
Level 9
Posts: 180
Joined: Thu Feb 03, 2011 3:22 pm

Re: Help with setup 256gb ram 48TB HDD drive for video storage

Post by Axel Mertes »

support wrote: Fri Nov 30, 2018 6:40 am
Axel Mertes wrote: Thu Nov 29, 2018 4:54 pm I should add the wish for PrimoCache to allow read-ahead caching in that context.
Sure, read-ahead is under our consideration.
:!: :!: :!: SUPER :!: :!: :!:

:D
Post Reply