How to pre-load the cache on a high-end PC? Topic is solved

[gregfreemyer]

Hi!

A few questions about your setup:

Presumeably you use NTFS, right?
Which block size did you use to format your drives and cache?

Yes, NTFS. I use the default 4KB blocks for the drives. I have the cache set to 256kb blocks. Potentially very wasteful, but mostly I read in large sequential files.

Default has always been 4 KB most of time for NTFS. However, block size that small force PrimoCache to use a large amount of overhead for indexing the potential source disk and L1/L2 cache itself. As you are dealing mostly with really big data files, small block sizes make simply no sense at all and waste only RAM and overhead. In turn, if your cach is too large, you need to use larger block sizes otherwise PrimoCache may not be able to utlilize it due to RAM constraints. I had that scenario in my setup, and as I have to deal with larger files often too (film & video post production...), we have formatted everything using 64KB blocks. That reduced memory overhead by 64/4 = 16 times...

Thanks for the suggestion, that might indeed be very smart of me to do as well.

I currently use a 2044 GByte SSD RAID0 as cache and consider swapping to NVMe RAID0 for higher bandwith reasons. 2044 GByte is the physical maximum cache size PrimoCache 2.x can address. We use only read cache, for security reasons, as a power outtage/crash could leave you with corrupt disks. The cache disk is used to cache several volumes (roughly 150 TByte in total). The cache size is about 1.5 to 2 times the size of the data we touch every day, ie. its very likely the all users will receive data from the cache, or after they first touched it - for the rest of the day and usually in the following days too. Unfortunately fresh written data is not automatically kept in the cache too, so it gets only loaded into cache after a read operation. That should be changed in PrimoCache 3.x and it would be a huge timesaver for us, as about 90% of our writes (rendered image data) will be read immediately afterwards...

I've got write cache setup. Are you saying that it might cause me to loose my newly written data, or an entire volume? I can risk my newly created data, but my entire volume is not something I want to risk losing.

I have high hopes that PrimoCache 3 will introduce a lot of often requested features and may be able to even recover a write cache after a crash/reboot/power outtage. In that context it may hopefully be able to re-use a filled/populated SSD/NVMe cache without reloading/repopulating the data. That would make systems a lot faster after a reboot compared to now.

Maybe its worth to look into your block size settings and try optimizing. With the file sizes you have you should always use a large block size. While PrimoCache itself can use up to 512 KByte, NTFS can only be 64 KB max and ReFS is using 64 KB as default. So 64 KB seems to be the sweet spot for now.

Many thanks for the info. I think I will move to 64KB blocks for both the drives and the cache.

[Axel Mertes]

http://barefeats.com/hard210.html

They got 5900 MByte/s using four 951 M.2 cards. Those are specc'ed at 2150 MByte/s each.
So four 960 Pro (specc'ed 3500 MByte/s each) will presumeably result around 9600 MByte/s in total. And on the HP they got near 12000 MByte/s as I have heard.

[gregfreemyer]

http://barefeats.com/hard210.html

They got 5900 MByte/s using four 951 M.2 cards. Those are specc'ed at 2150 MByte/s each.
So four 960 Pro (specc'ed 3500 MByte/s each) will presumeably result around 9600 MByte/s in total. And on the HP they got near 12000 MByte/s as I have heard.

The article says "When it comes to the Mac Pro tower, the NVMe blades won't run any fastet than AHCI blades since the PCIe 2.0 x4 slots top out at 1500MB/s. Even in the case of the Squid card in the PCIe 2.0 x16 slot 2, each individual blade is limited to 1500MB/s. The 5900MB/s is achieved through multiplication via the PCIe switch."

So, at least with the Mac Pro Tower, it doesn't seem the 960 Pro will be any faster than the 951's (1500 MB/s is the limiting factor, not the SSD).

I've got a AsRock x99 Deluxe. It has 3 Gen 3 x16 slots. I could use one of those for the Amfeltec card.

Is this the 4-SSD card they are testing:

http://amfeltec.com/products/pci-expres ... d-modules/

I was told recently the empty card is about $700. If 2TB turns out to be too small, I might look at that. I don't think I would ever need the speed.

[Axel Mertes]

Yep, the Amfeltec approach is a bit older and very expensive. It costs more than the HP and Dell counterparts - and those are unfortunately locked to their workstations through BIOS.

I hope that there will be more cards like these coming soon.

Anyhow, your limitation is two-fold: Speed and access speed. RAW speed is higher on the NVMe type SSDs. Access speed is at least high with any kind of SSD. Too bad the AsRock x99 Deluxe has not M.2 slot at all. But PCIe 3.0 gives you plenty of speed for such an add on card.

As per what you described, if all access is potentially scattered over the entire disk (I did forensic disk restore quite some times over here, too) then I would love to have the ENTIRE IMAGE on SSD AT ONCE, because otherwise the HDD will chime in all the time, ruin the effect the SSD brings.

Two port cards are much cheaper, so you may use a two port card and start with one or two M.2 Samsung 960 Pro 2 TByte (or was it the EVO that comes with 2 TB?). Best would be if you can have them for a test run.

[gregfreemyer]

Too bad the AsRock x99 Deluxe has not M.2 slot at all.

But it does. I have my current 2TB Samsung 960 Pro in it. Working fine.

I also have e a 1-port PCIe card, so I could go to 2 of the 2TB drives if it was warranted. For now, a single 2TB drive seems sufficient, but not excessive.

As per what you described, if all access is potentially scattered over the entire disk (I did forensic disk restore quite some times over here, too) then I would love to have the ENTIRE IMAGE on SSD AT ONCE, because otherwise the HDD will chime in all the time, ruin the effect the SSD brings.

Exactly. That's why I wanted to do the pre-load of the L2 cache. And from what I can tell I'm succeeding at that now.

Images that exceed 2TB are still rare for me, so I'm going to stick with what I have for a while.

BTW: I had temp folder get corrupted today. No PC reboot involve.ed. Friday I was using the temp folder to hold PSTs prior to parsing them. The extracted emails went into a sub-folder of that folder. I pushed 250 GB of PSTs through that folder last Thursday/Friday. Today, I couldn't even browse to it. I was told by Windows that the folder was corrupt.

This time it was no big deal, but it does make me nervous that the PrimoCache write-cache may not be safe enough for my needs.

[Axel Mertes]

I am using read-only caching on important data until a mechanism exists to really protect write caching (like a crash->resume functionality). Only on unimportant stuff I am using the write cache. On disk to disk copies I use hash code verification anyway.

I never had a corrupted disk/file with PrimoCache ever, and I really move many TBytes of data around (150 TByte online, 1-2 TBytes daily fresh data, even bigger transaction).

But I don't use cache preloading. Maybe there is still a bug left?

[Axel Mertes]

I am using read-only caching on important data until a mechanism exists to really protect write caching (like a crash->resume functionality). Only on unimportant stuff I am using the write cache. On disk to disk copies I use hash code verification anyway.

I never had a corrupted disk/file with PrimoCache ever, and I really move many TBytes of data around (150 TByte online, 1-2 TBytes daily fresh data, even bigger transaction).

But I don't use cache preloading. Maybe there is still a bug left?

[gregfreemyer]

As I continue to work with PrimoCache, I am less and less confident I know how to force data into the cache.

I recently emptied my L2 cache (all 2TB was emptied), now I have a 400GB of data I want to work with in the next 48 hours. I'm attempting to get it into the L2 cache, but it doesn't seem to be working.

A tool provided as part of PrimoCache is truly needed to let users have some control over the cache.

[Babel17]

As I continue to work with PrimoCache, I am less and less confident I know how to force data into the cache.

I recently emptied my L2 cache (all 2TB was emptied), now I have a 400GB of data I want to work with in the next 48 hours. I'm attempting to get it into the L2 cache, but it doesn't seem to be working.

A tool provided as part of PrimoCache is truly needed to let users have some control over the cache.

Is it possible that you paused the cache, and can get it to work by hitting the Resume button?

[gregfreemyer]

No, it isn't the pause.

I've setup a 64GB L1 cache in RAM and 2TB of L2 cache.

If I rapidly read 500 GB of data from a rotating media, most of it doesn't make it to L2.

I've found that if I read in ~60GB of data, then kill the reading app, the L2 cache gets a burst of data. Since my L2 cache is NVME, the L1->L2 flush takes less than a minute.

Then, currently I am restarting me main/fast data reader. It reads the L1/L2 part of the data quickly, then starts on the non-cached data. I let ~60GB of uncached data get read, then kill the data reader app. I get another round of L1->L2 data movement. After that stops, I repeat it.

It's a painful process, but it seems to work.

If there is a way to force all non-cached data reads to immediately get sent to L2 cache, I would definitely like to be using it.