An idea for getting rid of RAM overhead

[rutra80]

There are users here who use huge caches of huge volumes, which causes huge overhead eating up huge amounts of RAM.
They don't seem to experience significant performance drop due to maintaining the overhead, which means that it doesn't need a lot of read/write bandwidth.
Then, why not use SSD or even HDD for overhead data?
If it's just a bitmap or other kind of index, you could free a lot of RAM by using a bit of SSD space for it.

[Support]

Thank you for your idea!

[Wagz]

I wouldn't think that would be a very good idea. Other technologies have done that in the past and the results have been pretty bad. Bandwidth isn't the issue, latency is. You'd be adding the latency of random reads, cache scanning, and accessing the SSD multiple times to return a single block. For small block random reads, that could be bad. Consider that SSDs are still a thousand times slower than Memory. There are some enterprise storage arrays that tried this in their early days (Sun) only to find customers logging priority one outages from the decreased performance.

[rutra80]

It wouldn't have to be used for L1 cache which is as fast as RAM (and which usually needs much smaller overhead), only for L2 cache which already is as fast/slow as SSD is, I guess it would double the SSD latencies in worst case scenarios, which still would be much faster than HDD.
Also, there could be a small RAM buffer for overhead write/change operations, SSD one serving only for overhead lookup operations/reads.
It could be turned off by default, available only as an option for low memory configurations.

The fact is that there are technologies which don't need overhead at all (well, almost), but I guess that would need major rewrite of PrimoCache.

[Wagz]

Well, what I'm saying is that it is not 2x latency increase mathematically. It's more like 1000x latency increase. ZFS, XtremIO, and other technologies tried spilling their block tracking maps onto SSD and the results were pretty terrible. The reason is because you're scanning a block map that used to be in memory(nanoseconds) on a flash medium that is 1000x(or more) slower. Those block maps can be 3GB or more. You're scanning that map on an I/O. imagine that on every I/O.

[rutra80]

If it is accessed so much, how is that possible that it doesn't significantly slow down memory access in servers with huge amounts of I/O?

[Axel Mertes]

I am sure that optimizing the data structures for controlling the L1/L2 cache blocks plus moving away from MBR for cache disks plus making people aware of using larger block sizes would help more.

We currently use a 16 TByte RAID with NTFS 64 KByte formatting alongside a 2 TByte SSD cache with 64 KByte formatting (so 1:1 block size for SSD vs. HDD) to have maximum performance. The RAM overhead is fairly small, and there is still ~14 GByte RAM L1 cache left.

This screenshot is after running PrimoCache for less than 1 week (we usually restart the server once a week for several reasons).
23 TBytes reads total is reasonable (a bit more than I expected). The biggest improve would be to write a copy of each written block into the read cache for fast re-accessing (which is pretty common in our scenario).

[dhonn]

I have a programming idea on how to reduce overhead:

Let's say for example you have 2048MB for cache and you set up the block size to 4KB. Break up the cache into 4KB blocks so you have:
2048MB Cache = 524288 4KB blocks

Then for each 4KB memory block add an additional 8-byte overhead for the physical address on the disk.
524288 memory blocks * 8-byte disk address = 4194304 bytes or 4MB overhead

When a write or read to the disk you will scan 524288 addresses. Scanning 4MB of overhead should not take any CPU at all and can be optimized further by scanning the lower bytes of each address first.

No matter how big the disk the overhead will only be based on the size of your cache and not the disk size.

[Support]

We have a new design in ver3.0 and will reduce overhead by 30% or more.

[dhonn]

But wouldn't it be cooler to reduce the overhead by over 99% or more?