Primocache and Network File Transfers

FAQ, getting help, user experience about PrimoCache
thecodeman
Level 1
Level 1
Posts: 3
Joined: Tue Jan 03, 2023 10:19 pm

Primocache and Network File Transfers

Post by thecodeman »

I've been looking through the forum and I haven't been able to find an answer to a question I have. Can Primocache speed up network file transfers, specifically larger movie files being copied from one target to another? I'm looking at purchasing some 10Gb nics and installing in two PCs, both uplinked to a 10Gb switch. One PC would be a Plex server and the other a processing workstation. I thought about adding a 1Tb NVME drive to each and setting up as an L2 cache, 50% read and 50% write. Would this setup possibly allow for a faster file transfer between the two targets? How wouid L1 cache affect this? I would prefer not to use delayed writes but I suspect this would allow for better results. One other thing I forgot to mention is transfer sizes typically would be 100-250gb per copy and I would be moving files from spinning rust to spinning rust, combination of SATA and SAS drives in a JBOD configuration.

Any suggestions on how to get better speeds moving data between two hosts in this configuration?
RobF99
Level 8
Level 8
Posts: 130
Joined: Fri Sep 19, 2014 5:14 am

Re: Primocache and Network File Transfers

Post by RobF99 »

It will speed up network files transfers only if the data you are copying is already cached in L1 or L2 since your network connection is not the rate limiting factor. Your problem with these large files is that they would need to be cached which would mean that they would need to have already been previously read so that they are cached, otherwise you will only get network performance at the rate of your spinning drives transfer rates. Of course if the data is already cached the throughput will be at the rates of your NVME drive. With files that size, forget about L1 since you would need 100 to 250 Gb from the read side and again that data will have already need to be read to be in L1. Of course if you have write caching on the target PC there will be a slight reduction in latency since the file is written to the SSD first, rather than directly to the spinner. Anywhere where you can reduce latency on a network will have a slight beneficial effect.
thecodeman
Level 1
Level 1
Posts: 3
Joined: Tue Jan 03, 2023 10:19 pm

Re: Primocache and Network File Transfers

Post by thecodeman »

I can see that on the copy from side but for the copy to side, shouldn't all data hitting the target node go through cache before it hits the spinning rust.
RobF99
Level 8
Level 8
Posts: 130
Joined: Fri Sep 19, 2014 5:14 am

Re: Primocache and Network File Transfers

Post by RobF99 »

Yes it will but unless you have a very large amount of RAM, whatever nominal amount you set to L1 cache will rapidly fill up, it will then divert the data to L2 cache or directly to the spinner with urgent writes, depending upon your settings. In this case if you do have L1, then set flush L1 cache to L2 cache so that it does not do urgent writes.

But yes if you have the target node cached it will write to the cache before it flushes to the spinner. Another contingency is that depending upon your settings if it flushes to spinner while still receiving data, you will have a situation where it will be receiving data while flushing to the spinner, so the SSD will now be reading and writing simultaneously. In that even the performance using modern SSD architecture will still perform faster than writing direct to the spinner through the Windows cache. If you still have copies of the data and are merely copying to the target drive therefore, not worried about any risk of data loss, you could set a long time between flushes such as 30 to 60 minutes or so to reduce the odds of a flush while the data is copying to the target which will give you the overall fastest performance. Remember though if the data is not cached on the sending node, then data will only be read and written at regular spinner transfer rates because the read rate is the rate limiting factor in this situation.

So if data is cached on the sending node, you will have SSD performance for the entire transfer. Slightly less performance (maybe 1% or 2% reduction) if the receiving node flushes while still receiving data, but only regular hard drive performance if the sending node is reading the data directly from the hard drive and not the SSD cache, regardless of what is happening at the receiving end.
thecodeman
Level 1
Level 1
Posts: 3
Joined: Tue Jan 03, 2023 10:19 pm

Re: Primocache and Network File Transfers

Post by thecodeman »

If I add a 1TB NMVE cache to two systems and configure it 50% read and 50% write, I'm assuming I will see an increase in speed copying from one system to the other. Correct me if I am wrong but when I call the data to be copied, it will automatically be copied into cache on node 1 as read and then moved across the network to the other system hitting the write cache first before going to spinner. Is that how this should work? Can you recommend any options to tune my settings to increase performance other than adjusting the cache flush time?
RobF99
Level 8
Level 8
Posts: 130
Joined: Fri Sep 19, 2014 5:14 am

Re: Primocache and Network File Transfers

Post by RobF99 »

No, when you call the data to be read, it will not be automatically copied to the cache and then moved across the network. Copying to the cache upon read is a post read process. This is set in the gather interval. It cannot predict what you need to read, it can only see what you have read. Of course if you have performed some previous operation that will have that data in the cache, i.e. if you have been doing things that have caused that data to be cached, e.g. creating that data.

The only other thing you can do to cache that data ahead of time would be with some utility that reads that data into that cache. I have a post elsewhere on this forum that mentions a small utility called readfile. You can use this to precache data. But since you are working with big files the process of reading the data into cache could take a while and in the mean time cause the source system to be under a lot of load while caching that file. It won't be a pleasant experience using that PC while precaching that data but it is the only way to precache the data unless, as I said you have been working on that data thus causing some of it to be cached.

It's hard to suggest appropriate settings for your systems for what you need to do but what should work pretty well would be:

Source:
A nominal amount of L1, Maybe 25% of the system RAM. Have read/write deselected so it shares read/write
L2, 50% read and 50% write is fine. Overprovision the 1Tb so that you have 80 to 90% cache size. i.e. 800 to 900 Gb allocated to L2. (In disk management create a drive that is the size you want and leave the rest unallocated)
Set gather interval to any value but NOT instant as you may slow your reads a little for large files.

Target
As much L1 as you can. 90 write/10 read or you can have read/write deselected so it shares read/write
L2, 50% read and 50% write is fine. Overprovision the 1Tb so that you have 80 to 90% cache size. i.e. 800 to 900 Gb allocated to L2. (In disk management create a drive that is the size you want and leave the rest unallocated)
Set a long flush time (if you aren't concerned with risk of data loss) 30 to 60 minutes.
Also set Flush L1 to L2 in write delay. This is most important.
I would also use write mode Idle Flush. I noticed that this will delay writes up to twice as long as the write delay time if the system is still receiving data.

You can then tweak as needed but these should give you best performance for what you are looking to do. Hope it helps.
tverweij
Level 6
Level 6
Posts: 74
Joined: Thu May 10, 2018 9:27 am

Re: Primocache and Network File Transfers

Post by tverweij »

thecodeman wrote: Tue Jan 03, 2023 10:31 pm Can Primocache speed up network file transfers
No it can't, it is a disk cache, not a network optimizer.

But when your network reads are limited by the read speed of the remote hd, a remote cache might help, but only if the same data is transferred multiple times.
The first time, the data is read from disk and the cache is populated. The second time, it will read from cache (but only when it was not already replaced by other data).
When the network transfer speed is limited by the local HD write speed, a local cache with deferred write might help.

And that is all that primo cache can do in this case.
rajdude
Level 4
Level 4
Posts: 22
Joined: Thu Mar 07, 2019 5:24 pm

Re: Primocache and Network File Transfers

Post by rajdude »

I am in almost the exact same boat here as the OP. Have been doing some testing with PC but does not look like it is caching files being served by the server.
Test:
I copy a large 15 GB file down from server to client, I get approx. 150 MBps
Hoping that this will put it in the SSD cache.
Then I delete the file from the client.
Thereafter, I copy the same file down from server to client again.
The resulting down speed is the same as the first attempt - 167 MBps

Maybe I am limited by the write speed of the SSD on the client?
But if I test my client's local SSD using Crystal Disk mark, Sequential 1 thread, 1 queue, 8 GB payload. It shows 600 MBps

What gives?

I can start a new thread, if that makes more sense.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Primocache and Network File Transfers

Post by Support »

@rajdude, please upload your cache configuration and stats for our reference by following the guidance in the link below, thanks.
https://kb.romexsoftware.com/en-us/2-pr ... leshooting
rajdude
Level 4
Level 4
Posts: 22
Joined: Thu Mar 07, 2019 5:24 pm

Re: Primocache and Network File Transfers

Post by rajdude »

Support wrote: Wed May 03, 2023 6:54 am @rajdude, please upload your cache configuration and stats for our reference by following the guidance in the link below, thanks.
https://kb.romexsoftware.com/en-us/2-pr ... leshooting
Thanks for looking into this. Here is PC's config:
2023-05-03-server - Remote Desktop Connection-000328.jpg
2023-05-03-server - Remote Desktop Connection-000328.jpg (102.34 KiB) Viewed 716 times
PS: Here is the configuration of my server

  • Supermicro motherboard X11SSL-F
    Xeon processor E3-1225 v6 3.30 GHz 8M cache
    32 GB RAM
    Mellanox Fiber Channel network card 25Gbps SFP running at 10gbps
    Dell PERC H730p withe 2GB battery backed cache
    6x4TB Toshiba N300 NAS drives in RAID6 (SATA)
    250 GB SATA SSD used as cache for PrimoCache
    120 GB SATA SSD used as OS drive
    Windows 10 Pro
Here is the configuration of my client
  • i7 processor overclocked to 4 GHz
    16 GB RAM
    2x 250 GB SATA SSDs in RAID 0
    Mellanox Fiber Channel network card 25Gbps SFP running at 10gbps
Post Reply