Page 1 of 1

Large file transfer comes to a crawl

Posted: Sun Feb 14, 2021 6:55 pm
by darksheer
System Specs
I9 9900K
32 GB Corsair Dominator Platinum 3200 MHz Ram
Samsung 970 Evo Plus 1 TB
Samsung 970 Evo Plus 500 GB - Cache Drive
10 TB WD HD - Partitioned in 3

I move large files (20 - 30GB) across the hdd partitions and I cannot find a setting that will consistently move them at a fast speed. I have tried so many different combinations and some I can get 1 GB + transfer speeds for about half the transfer and them it bottoms out to <10mbs while other setting I can get 150 across the entire transfer. Sometimes it will bottom out towards the end on those settings also. What am I doing wrong? I am fixing to build a nas to hold all my files so I want to make this work so I will be able to transfer at good speeds to it.

Re: Large file transfer comes to a crawl

Posted: Sat Feb 20, 2021 7:37 am
by RobF99
You might want to change your L2 gather interval to Idle only. In a large file transfer if your gather interval is at any other setting it will begin populate L2 from the hard drive while also trying to read the file from the hard drive during copy. So you have two reads on your hard drive occurring at the same time and thereby reducing your I/O performance to less than half because two different read threads both have their own individual seek overhead.

I run both L2 gather interval and Defer Write on Idle settings so as to avoid as much of this clashing as possible. With defer write on idle, be careful to ensure that all data is flushed to the target drive before you delete a copied file from another drive.

Re: Large file transfer comes to a crawl

Posted: Mon Feb 22, 2021 4:52 am
by Support
Not easy to get constant fast speed on transferring large files. When transfer a file, it means that 1) reading from the source partition and 2) writing to the destination partition. The HDD's native read/write on large files is about 100~200MB/s. So if you want to get the constant fast speed, the file to be transferred should already be cached, and you also need to enable Defer-Write on the destination partition. You may need use both L1 and L2 for defer-write because your files are quite large and memory (L1) is insufficient to achieve constant fast write speed.
Ok, I think the big problem is that your files to be transferred are not always cached before the transfer because the cache size is only 500GB while you operating large files which are 20~30GB each. So the total speed is limited by the reading from the source.