Primocache and Network File Transfers

[Nick7]

Test:
I copy a large 15 GB file down from server to client, I get approx. 150 MBps
Hoping that this will put it in the SSD cache.
Then I delete the file from the client.
Thereafter, I copy the same file down from server to client again.
The resulting down speed is the same as the first attempt - 167 MBps

While doing this, open Task Manager, and check from which disk are reads.
Also, during this time you can see if cache hit ratio is increasing or decreasing in PrimoCache.

By using this, you will know if you are reading stuff from cache or not.

Oh, also... there is possibility of 0 disk activity - if it's read from OS cache.
Anyway, most likely it's sharing issue.

[tverweij]

I am in almost the exact same boat here as the OP. Have been doing some testing with PC but does not look like it is caching files being served by the server.
Test:
I copy a large 15 GB file down from server to client, I get approx. 150 MBps
Hoping that this will put it in the SSD cache.
Then I delete the file from the client.
Thereafter, I copy the same file down from server to client again.
The resulting down speed is the same as the first attempt - 167 MBps

Maybe I am limited by the write speed of the SSD on the client?

Look at:
- antivirus (on both client and server)
- max write speed on the client
- L1 Cache size (relatively small for a 15 Gb file)
- L2 Max read speed on the server

[rajdude]

Yeah, you are correct. I looked at task manager.
During the first copy from server to client, server's HDD shows activity.
Then after the copy finished, I see activity on both HDD and cache SSD (maybe PrimoCache is doing it caching activity AFTER the file i/o finishes up)
Unpon subsequent copy from server to client, the server's HDD shows no actibity but cache SSD shows activity.

That shows that PC is doing its work.

As for network file copy speeds, something is definitely wrong with my system, For a 15 GB file to be copied from the server to client, I see wildly varying times, on different days, using the exact same tool - robocopy. I actually have a tiny script which simply deletes and copies. So I am sure that the robocopy switches are exactly the same.

23 seconds (lowest ever)
30 seconds, was the usual time when I was testing a couple of days ago
2:18 longest, observed today. I cannot make it go down, Weird!

Note: One one day, the times stay the same. If the machines are rebooted and I test another day, I have another set of values!
This behaviour is very strange!

Test:
I copy a large 15 GB file down from server to client, I get approx. 150 MBps
Hoping that this will put it in the SSD cache.
Then I delete the file from the client.
Thereafter, I copy the same file down from server to client again.
The resulting down speed is the same as the first attempt - 167 MBps

While doing this, open Task Manager, and check from which disk are reads.
Also, during this time you can see if cache hit ratio is increasing or decreasing in PrimoCache.

By using this, you will know if you are reading stuff from cache or not.

Oh, also... there is possibility of 0 disk activity - if it's read from OS cache.
Anyway, most likely it's sharing issue.

[Nick7]

PrimoCache moves data by default when there is no disk activity, so this is I/O you saw initially from HDD&SSD at same time.

Now, you need to dig deeper - when copying again, and data is read from SSD cache, is disk at 100% utilization?
Is disk at 100% utilization on destination computer?
Are any logical cores of CPU at 100%?

Do you have anything else on same network that might have high bandwidth?

[rajdude]

ok that makes sense then
I was thinking PC will cache the data at the time of sending out the data over the wire.

so yeah, I am checking again with the same 15 GB file.

First time download:
Server:
cache SSD shows around 40% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
1 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 1:02 mins


Subsequent downloads:
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
1 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 53 seconds


Upload test (from client to server):
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
2 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 36 seconds!

I think the upload is going to PC's L1 cache which is 15 GB

subsequent upload tests (from client to server):
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
2 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 36 seconds!


again download test:
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 30% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
1 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 1:06 seconds

one last download test:
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
1 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 1:06 seconds



No I do not have anything which takes too much bandwidth on my local network. It is 10Gbps anyway. Mellanox NICs over Fiber channel SFPs going to a Qnap 10Gbps switch.

PrimoCache moves data by default when there is no disk activity, so this is I/O you saw initially from HDD&SSD at same time.

Now, you need to dig deeper - when copying again, and data is read from SSD cache, is disk at 100% utilization?
Is disk at 100% utilization on destination computer?
Are any logical cores of CPU at 100%?

Do you have anything else on same network that might have high bandwidth?

[Nick7]

Two more things, before declaring issue with CIFS/SMB servers:
1) Did you try iperf between two PC's (client/server)?
2) Maybe try different PCIe slot for 10G card? Might be in x1 PCIe, even if slot is x16. Also, it might be over chipset.

[Support]

Subsequent downloads:
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
1 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 53 seconds

Windows has its native file caching, so the subsequent downloads likely read file data from Windows cache since you have about 14GB free RAM. You can check if PrimoCache stat "Total Read" increases when downloading.

Upload test (from client to server):
Server:
cache SSD on shows around 0% active time
HDD RAID shows around 0% active time
All 4 logical CPU in that Xeon show around 15%
Client:
SSD on shows around 20% active time
2 of 8 logical CPUs in that i7 show around 40%
All other 7 logical CPUs show around 10%
Time taken 36 seconds!

I think the upload is going to PC's L1 cache which is 15 GB

You can PrimoCache stat "Total Write (Req)" and "Total Write (L1/L2)" during uploading. If both items increase, it means upload is going to cache.
I see the transfer speed stayed about 400MB/s assuming the upload file was 15GB. This speed seems to be in line with expectations.

[Support]

As for network file copy speeds, something is definitely wrong with my system, For a 15 GB file to be copied from the server to client, I see wildly varying times, on different days, using the exact same tool - robocopy. I actually have a tiny script which simply deletes and copies. So I am sure that the robocopy switches are exactly the same.

23 seconds (lowest ever)
30 seconds, was the usual time when I was testing a couple of days ago
2:18 longest, observed today. I cannot make it go down, Weird!

This seems quite weird. I suspect it's a network problem. You may try iperf first as Nick7 suggested.