CHIA and PrimoCache

FAQ, getting help, user experience about PrimoCache
User avatar
Support
Support Team
Support Team
Posts: 3085
Joined: Sun Dec 21, 2008 2:42 am

Re: CHIA and PrimoCache

Post by Support »

azzurro wrote: Sun May 23, 2021 11:35 pm Wouldn't that be something that could be implemented in PrimoCache?
PrimoCache can. You can set L2 cache (SSD cache) to be read-only (100% read which is set by default). And with Defer-Write, you can increase write performance a lot, as well as converting random writing to sequential writing.
Primo Ramdisk | PrimoCache
Romex Software Support
azzurro
Level 2
Level 2
Posts: 5
Joined: Sun May 23, 2021 11:26 pm

Re: CHIA and PrimoCache

Post by azzurro »

Support wrote: Mon May 24, 2021 2:55 am
azzurro wrote: Sun May 23, 2021 11:35 pm Wouldn't that be something that could be implemented in PrimoCache?
PrimoCache can. You can set L2 cache (SSD cache) to be read-only (100% read which is set by default). And with Defer-Write, you can increase write performance a lot, as well as converting random writing to sequential writing.
Cool, so with defer-write the I/Os are inherently being converted to sequential streams? What happens, if a block is requested from the disk which hasn't been written yet, I guess it will then be served directly from L1 cache (RAM), correct? I hope it doesn't get written to disk first only to then serve the request from disk instead directly from RAM?

I guess the only thing left for this to be perfect for chia would be to be able to fine-tune what PrimoCache considers the machine/disk/cpu being idle and maybe put some more intelligence into the caching to maybe keep "hot" blocks in the cache longer. Something like "infinite" delay, but write cold blocks to disk if the disk is idle and delete them from RAM as soon as cache space is needed for new I/Os while still serving the blocks from RAM as long as they are there.
User avatar
Support
Support Team
Support Team
Posts: 3085
Joined: Sun Dec 21, 2008 2:42 am

Re: CHIA and PrimoCache

Post by Support »

azzurro wrote: Mon May 24, 2021 12:55 pm What happens, if a block is requested from the disk which hasn't been written yet, I guess it will then be served directly from L1 cache (RAM), correct?
Yes.
azzurro wrote: Mon May 24, 2021 12:55 pm I guess the only thing left for this to be perfect for chia would be to be able to fine-tune what PrimoCache considers the machine/disk/cpu being idle and maybe put some more intelligence into the caching to maybe keep "hot" blocks in the cache longer.
Mostly, you may just use the native mode for Chia and tune the latency to avoid "urgent write" and meanwhile reach a high "Deferred Blocks". "Urgent Write" means latency is too big and cache is full of deferred write-data. A high percentage of "Deferred Blocks" indicates the fully utilizing the cache. Please note that it takes time to flush all deferred write-data, so you should not tune latency to make "Deferred Blocks" 100%. Otherwise incoming new write-data has no cache space to store and will trigger "Urgent Write". Usually 50%~80% should be good, but it depends on your writing workload, cache space and the write ability of target drives.
Primo Ramdisk | PrimoCache
Romex Software Support
User avatar
Jaga
Contributor
Contributor
Posts: 681
Joined: Sat Jan 25, 2014 1:11 am

Re: CHIA and PrimoCache

Post by Jaga »

I hate to admit it, but based on my testing with Chia (both single and parallel plots), I don't think I'd recommend Primocache for use with it unless you have an Enterprise class server with ~1TB or more of RAM to devote.

The reason is simple: using a L2 cache from Primocache is basically the same as using the L2's SSD as the Temp plot drive. Using a L1 means the cache size has to at least cover a single plot's max data, which is around 260GB. Parallel plotting would require >512 GB of RAM for even 2 plots. With the kind of Cores/Threads you can get out of an Enterprise class server, you could almost never stuff enough RAM into it to fully utilize Primocache's L1 feature to good effect.

The same would go for Primo RamDisk: massive amounts of RAM would be necessary. The cost would be even more prohibitive than using temp plotting drives on Enterprise class SSD's, in my opinion.

The only gains to be made in using a L1 cache would be some marginal reads from the L1, but the hitrate would probably be quite low due to not having enough RAM to cover multiple plots at once. I have 64GB of RAM in my gaming rig (a 4C/8T CPU), and after some trials here I just plain stopped using Primocache for the L1. One of the phases of plotting compresses all of the temp data, re-writing it to the Temp plotting drive, eliminating good L1 hit rate. And since Chia temp plots on a SSD avoid the whole L2 caching algorithm, it doesn't make sense to setup a L2.

Would be interested to hear others' thoughts on it as well, see if anyone has any real world use cases where Primocache would actually make a noticeable difference in plot times.
azzurro
Level 2
Level 2
Posts: 5
Joined: Sun May 23, 2021 11:26 pm

Re: CHIA and PrimoCache

Post by azzurro »

That's why I said it would be cool to be able to adjust what PrimoCache considers to be "idle". I have been using PrimoCache on an older enterprise-class server with a RAID5 (which can do 600MB/s of sequential write speed) and with some SSDs in it.
Regarding using the SSDs as L2 cache I came to the same conclusion as you. Otherwise, I must say that I have seen some "low pressure" times on the RAID where PrimoCache easily could have flushed quite some of the 50GB L1 cache I have assigned to disk but it didn't. That resulted in the cache being full eventually, having to flush it "urgently" in the worst possible moment.

So my conclusion is still: it would be really great to make adjustable what PrimoCache considers the system being idle. Relying on fixed intervals is not good in this case, as you never know when the disks have some spare resources to spend on writing.
Audionut
Level 1
Level 1
Posts: 3
Joined: Tue May 25, 2021 8:49 am

Re: CHIA and PrimoCache

Post by Audionut »

azzurro wrote: Wed May 26, 2021 12:27 pm Otherwise, I must say that I have seen some "low pressure" times on the RAID where PrimoCache easily could have flushed quite some of the 50GB L1 cache I have assigned to disk but it didn't. That resulted in the cache being full eventually, having to flush it "urgently" in the worst possible moment.

So my conclusion is still: it would be really great to make adjustable what PrimoCache considers the system being idle. Relying on fixed intervals is not good in this case, as you never know when the disks have some spare resources to spend on writing.
This exactly. Chia likes to do things in chunks, especially reading and computing. While chia is computing, there's a bunch of idle time on the read task, which is perfect for the write task.

I currently can't get any of the write modes, other than native, to write in idle read time (when the compute task kicks in). Despite being listed as native+, and regardless of the defer-write time, these write modes refuse to write until the buffer is full and it panic's.

Ideally I would like to see a read priority mode for the flush, that throttles flushes during a read, with full write speed during read idle, regardless of what the CPU or anything else is doing.

Image Image
User avatar
Support
Support Team
Support Team
Posts: 3085
Joined: Sun Dec 21, 2008 2:42 am

Re: CHIA and PrimoCache

Post by Support »

Jaga wrote: Wed May 26, 2021 12:23 pm Would be interested to hear others' thoughts on it as well, see if anyone has any real world use cases where Primocache would actually make a noticeable difference in plot times.
We have already received some feedbacks from customers, saying that they can do more plotting at the same time. Usually their rigs have 128G ~ 2T RAM installed.
Primo Ramdisk | PrimoCache
Romex Software Support
User avatar
Support
Support Team
Support Team
Posts: 3085
Joined: Sun Dec 21, 2008 2:42 am

Re: CHIA and PrimoCache

Post by Support »

azzurro wrote: Wed May 26, 2021 12:27 pm That resulted in the cache being full eventually, having to flush it "urgently" in the worst possible moment.
In most scenarios, native mode + a proper latency should be enough. Reduce latency to avoid the urgent write.
Primo Ramdisk | PrimoCache
Romex Software Support
User avatar
Support
Support Team
Support Team
Posts: 3085
Joined: Sun Dec 21, 2008 2:42 am

Re: CHIA and PrimoCache

Post by Support »

Audionut wrote: Wed May 26, 2021 12:44 pm I currently can't get any of the write modes, other than native, to write in idle read time (when the compute task kicks in). Despite being listed as native+, and regardless of the defer-write time, these write modes refuse to write until the buffer is full and it panic's.
For chia mining, 8GB L1 for defer-write is too small. Otherwise you need to reduce latency to avoid "urgent write".
Primo Ramdisk | PrimoCache
Romex Software Support
User avatar
Jaga
Contributor
Contributor
Posts: 681
Joined: Sat Jan 25, 2014 1:11 am

Re: CHIA and PrimoCache

Post by Jaga »

Support wrote: Fri May 28, 2021 3:52 pmWe have already received some feedbacks from customers, saying that they can do more plotting at the same time. Usually their rigs have 128G ~ 2T RAM installed.
Cool, that verifies my assumptions that standard rigs don't really gain much with it. Would be super nice to see the difference on a rig with 1-2 TB of RAM when compared to the same without Primocache.
Post Reply