Preload currently accessed files -> improve Audio/Video/Film

[rutra80]

I guess that the cache block size defines the smallest amount of data that can be read/written. So if you want to write 4KB to a cache which has a 512KB block size, you must actually (re)write 512KB.
As for the exFAT, I think it's less secure than NTFS. ReFS is said to be more secure than NTFS. ReFS cluster size is always 64KB.

[Support]

@Axel Mertes, a cache block can be partially filled. So a 512KB cache block may only store a single 4KB disk data.

[Slavius]

If a cache block can be partially filled and can not only hold one disk block (like you can write arbitrary amount, let's say 4k disk blocks to one 64k cache block) then what's the point with being able to set cache block size? Why isn't it always the biggest possible or any fixed - optimal value to prevent overhead?

[Axel Mertes]

Hi Support,

I analyzed the values you presented for the overhead and was a bit confused about some findings.

For some reasons you need more memory per block, the less blocks are indexed in total. Does that make any sense?

See the yellow marked fields in the attached table for details. They show how much bytes you actually need to index a cached block, so how much RAM you consume for this. I have expected a constant number. I understand that in 64-bit mode there might be some values which take more bytes due to memory alignment.

[Axel Mertes]

@Axel Mertes,

The major factor is the block size. Bigger the block size, smaller overhead. See FAQ.
Also currently supported maximum L2 cache size is about 2TB (2044GB). And we will overcome this limit in future.

Thanks.

Hi Support!

Above you mention the current limit is 2 TB of L2 SSD cache.
Given a 64 KB block size it would need about ~2.8 GBytes of RAM.

If I have more RAM available, can I use more cache?
We have currently 4 TByte of SSD storage in the server. That might need 5.6 GByte of RAM to index.
I don't actually see an actual address space limit in 2 TBytes, as this corresponds to 2^41.

Should I give it a try?

[rutra80]

If PrimoCache creates an MBR partition for a cache, then that's where's the 2TB limit from - MBR partitions can't be bigger than 2TB (their size, in sectors which are 512 bytes big, are stored in 32bits).
Anyway, at that many GBs of RAM index alone, I would start to worry about memory bandwidth...

[Axel Mertes]

If PrimoCache creates an MBR partition for a cache, then that's where's the 2TB limit from - MBR partitions can't be bigger than 2TB (their size, in sectors which are 512 bytes big, are stored in 32bits).
Anyway, at that many GBs of RAM index alone, I would start to worry about memory bandwidth...

Ah, I missed that part with the MBR section. OK, makes sense.

But why would you start worrying about memory bandwidth?
RAM is usually hell fast and we have 48 GByte in the server. Without actually knowing how PrimoCache implements the cache, there are several possibilities how they can implement a cache:

They can e.g. have a matrix table "per block of source disk" that indiciates if the block is cached and if yes, to which block of first and/or second level cache.

Another approach would be to just have a list of cached blocks and compare if the required block is in the list.

However, the latter method would be a lot slower - programming wise - while it may be more memory efficient. The first method is a simple look up operation with fixed speed per access, regardless of the size of the cache.

If the cache is filled during runtime, I don't see a reason where bandwidth might become a problem. Rutra 80, if you do, please explain why!

[rutra80]

I don't know. That "index" is a huge data set which needs to be accessed on every I/O operation and maintained. You're putting all the I/O onto CPU(s). There will be more CPU cache misses for actual server's tasks. The same is with bandwidth and CPU power. Are you on NUMA? There's quite a lot of things to consider. I'd first benchmark the whole system without cache and then add some smaller cache and benchmark again. Then increase it and test again before putting such a big overhead to production.

[Axel Mertes]

I am not that concerned about overhead. In fact, the machine is a dual hex core, showing 24 threads with hyperthreading. Even under heavy loads it currently rarely uses two cores at all. As we run on SAN most of the time right now, it handles SAN policy through a dedicated ethernet, while feeding SMB clients through the standart ethernet using dual 10 GBit trunks.

The delays we see right now from local attached SAN RAID drives conected via 4 GBit FC compared to SSD cached RAID drives connected via 10/1 GBit Ethenert is dramatic. Latency is what we want to overcome, IOPS rule, so does bandwidth. SSD cache alway wins. We also don't see any special loads on the server when accessing the SSD caches.

Its just the question which route is the best:

1. Pure SSD storage.
Pro: Super fast, easy to administrate
Contra: Expensive, RAID controllers can't handle the bandwidth at all.

2. Software SSD cache like PrimoCache
Pro: Fast, easy to administrate
Contra: Limited size of cache, risky if used as write cache, occasional performance hit when data is touch first time and not yet in cache.

3. Storage Spaces tiering in Windows Server 2012 R2
Pro: Automatic tiering of most used/recent data to SSD storage.
Contra: A bit complex to administrate at first, not very transparent how the target disk is "organized", occasional performance hit when data is touch first time and not yet in cache.

4. Hardware RAID SAN or NAS systems using SSD caching for tiering
Pro: Similar to storage spaces, but usually easier to administrate. Many run on ZFS, some on proprietary software (QNAP, DataDirect). It appears that some systems support large SSD tiers, such as QNAP with their 40 GBit ethernet enabled boxes.
Contra: Quite expensive - even if in most cases a standart server is used with Linux. However, the experience may justify the price level.

So for us its a question which route to go. PrimoCache is among the cheapest solutions plus it can be switched on/off as required. So it would be fairly easy to deploy in an existing infrastructure. With Storage Space I need to design everything upfront and start from scratch, which can be a lot more complex. Further there is few information yet about the real performance of Storage Spaces systems build for performance and how reliable they are. Hardware RAIDs like QNAP NAS are way more expensive than the "parts" alone, but their software seems to be a great deal. There are only a few NAS boxes that are said to show 3.8 GByte/s real transfer speed.

[Support]

If a cache block can be partially filled and can not only hold one disk block (like you can write arbitrary amount, let's say 4k disk blocks to one 64k cache block) then what's the point with being able to set cache block size? Why isn't it always the biggest possible or any fixed - optimal value to prevent overhead?

1) Though a cache block can be partially filled, its spare space can only store the adjacent data.
2) When all cache blocks is used/assigned (even these blocks may partially filled), PrimoCache has to discard some old cache blocks for storing new cache data. Small block size has better flexibility than big block size.