CHIA and PrimoCache

[Audionut]

I currently can't get any of the write modes, other than native, to write in idle read time (when the compute task kicks in). Despite being listed as native+, and regardless of the defer-write time, these write modes refuse to write until the buffer is full and it panic's.

For chia mining, 8GB L1 for defer-write is too small. Otherwise you need to reduce latency to avoid "urgent write".

Negative. At mining start, I can dump the initial ~35GB (takes around 2 minutes iirc) to the SSD with 4GB L1 + 10 second latency, without urgent writes.

As first my screenshot above shows, if I enable ANY of the write modes other than native, it refuses to flush the L1 while there is CPU load, until the cache is full and then it panics.

edit: If I understand correctly, primocache combines the data, writes out at the block/cluster size, and trims as it goes along. I expect this to have a noticeable positive affect on SSD life, which will be especially useful for chia mining.

[Support]

using a L2 cache from Primocache is basically the same as using the L2's SSD as the Temp plot drive.

Not exactly. You can share one SSD for more plotting tasks at the same time with PrimoCache. For eg. K=33, you need about 521G temp space for each plotting. So a 2TB SSD can support for 4 parallel plots without PrimoCache. But with PrimoCache (Defer-Write enabled), it can support 10+ parallel plots.

[Axel Mertes]

I have done quite a bunch of Chia plotting recently, lik 80 TBytes of plots so far. Outputting over 50 plots per day.

My believe is that PrimoCache will bring you no gain in speed, but a gain in SSD lifetime. I use it only for L1 caching and defer writes to 10 seconds, so multiple writes get combined into one, reducing wear to a certain degree. L2 cache is total nonsense in that context IMHO. Even those having enough RAM to use a RAM disk only for caching say that their speed gains are far less than expected, like a 2 times improvement over SSD or SAS RAID performance.

I am still not sure what this algorithm really does, that it isn't able of utilizing the machines resources fully. I almost never see a full CPU load, disk load, RAM load. I wrote some scripts to stagger creation and wait for enough free SSD plotting space if needed. This runs really smoothly and as I said, 50 or more plots per day. However, running out of disk space and still wondering if that entire nonsense if of any meaningful use at all...

For now, its like playing lottery, nothing else. Still needs 2 month to win it says. currently I am growing faster than the network, however, I am not going to buy additional free HDD space...

I wonder if they could report the FREE disk space on the network. That would be an interesting number to know...

Currently I see up to 25%, sometimes 40% deferred writes, which is significant IMHO.

[Jaga]

I have done quite a bunch of Chia plotting recently, lik 80 TBytes of plots so far. Outputting over 50 plots per day.

My believe is that PrimoCache will bring you no gain in speed, but a gain in SSD lifetime.

I'm using it a bit differently, and see a 70% gain in overall speeds: 52GB L1 (Read/Write) with Infinite Defer Writes. Setup against a 4x 4GB RAID 0 stripe of WD Reds (Temp2 drive for 75% disk activity). My Temp1 drive is my OS SSD which has plenty of write lifetime left.

I used to get ~7 to 8 plots per day using staggered parallel plotting on this particular rig using PSChiaPlotter. With the new configuration (using MadMax plotter) I am getting ~12+ plots per day. And as an added surprise - a 58% hitrate for each plot in Primocache, with millions of trimmed blocks per plot which reduces the need for writes even further. Typical deferred writes are up around 65% at the end of the plot cycle.

It isn't a super rig, and I'm not using SSDs for plotting (meaning I can basically do this indefinitely without any added cost). The trick was figuring out optimal Stripe size, volume cluster size, and Primocache block size. Right now each plot is down to 1:50.

[Axel Mertes]

Interesting, I haven't really followed any new plotters yet, but to me the original implementation always looked pretty slow and un-optimized, without having really looked at the actual code. All the performance I saw, told me there is plenty of room for optimizations.

Anyhow, with nearly all free disks full it makes few sense for me to improve the game any longer as of now.
Might change, if I add more disk space, but never for Chia itself, only as a side effect.

With the Chia value going down and down since its beginning, its questionable how this will end anyway. I am sceptic at this point. After all, as most crypto currencies, a huge waste of resources IMHO.

Maybe I download MadMax and give it a shot to see how it performes.
Anyhow, I do get a plot every 20 minutes using the standart Chia plotter, so maybe my staggering is simply well optmized already - on fairly old computers.

[andriusst]

Just wanted to pitch in as I have been plotting for over a month, created 114TiB plots so far. Currently getting 1 plot out in just under an hour. Rig is 16-core AMD CPU, 2TB nvme SSD, 64GB RAM, Windows10. I found that once plots are optimally staggered then it is down to raw temp drive bandwidth. As much as I like PrimoCache I was unable to set it up to beat Windows native caching (LargeSystemCache=1). Bear in mind that my particular scenario may not be ideal way of using PrimoCache. But with PrimoCache plots finish in roughly 28,000s and without 26,000s (workers launching 58 mintues apart, ending up with 7-8 simultaneous workers). PrimoCache with 32GB L1 cache gives less than 1% hit rate. Cache flushing rate is between 700-1000MB/s (according to windows task manager). Crystal diskmark does above 3000MB/s for writes so not sure why such a big discrepancy. Using trial and error I found that NTFS cluster size of 2048KB works fastest (don't care about write amplification).

But Jaga gave me an idea. Perhaps having HDD RAID with enough disks to beat SSD in sequential bandwidth would be even faster using PrimoCache. I am guessing that using deferred writes should coalesce semi-random multi worker writes into more-less sequential writes. The effect should improve with increasing L1 cache size. According to my very rough measurement 7 well staggered simultaneous workers total I/O is around 350MB/s reads and 450MB/s writes. PrimoCache will do amazing job at caching writes and will give signifficant boost in that respect. But with 35GB L1 cache I am getting less than 1% cache hit rate meaning read burden falls directly on the disk. Multiple workers will create a semi-random read workload which HDDs are likely to struggle with. I suspect that having 7 drive RAID0 is going to be slower than giving each worker own HDD.

300GB or 600GB 15K RPM SAS HDDs are dirt cheap on ebay these days. So perhaps that would be a cheap way to increase the output for someone not willing to fork out on SSD. Will also need SAS HBA supporting 8 or 16 drives of course but they are also cheap on ebay.

[Jaga]

But Jaga gave me an idea. Perhaps having HDD RAID with enough disks to beat SSD in sequential bandwidth would be even faster using PrimoCache.

It definitely can be. That's why I'm using 4x HDDs in RAID 0, to overcome both the individual drive performance limit, and to give Primocache a single plotting target to cache.

Take a look at the screenshot I'm adding. I lowered the L1 size from 52GB to 42GB (so that I can work on the machine, game on it, etc), and then I simply change the cache size before starting overnight plotting. Even with just a 42GB cache size, running for a couple of days it looks like this:



Take note of the Trimmed Blocks: 26.7 million blocks, each 512KB in size. Also take note of the Total Write (L1) vs Total Write (Disk): I've been able to avoid ~47% disk writes due to the Infinite Defer-Writes setting. Then there's the Cache Hit Rate, at ~47%. With a cache size of 52GB, that climbs to almost 55%. And lastly, compare Total Write (L1) vs Total Write (Disk): the system made only 53%% of total writes to the disk.

I am doing serial plots - just one at a time. But I've been able to gain +70% daily plot performance by doing it this way, AND if I lower the cache size just a little I can still work/game on the same rig. Best plot time with this method is 1:50, worst (with lower cache) is 2:45. It'd probably scream if my RAID array (Temp2) for plotting used SSDs instead.

Give it a go, you might be surprised what your system is capable of.

[andriusst]

Nice one. I own PrimoCache already and have been experimenting with all possible settings combinations. Your good trimmed blocks and cached reads come from the fact that large size of the plot file effectively sits in your L1 cache. Have L1 cache large enough to fit it completely and you won't be seeing any disk writes at all and 100% cache hit ratio.

I yesterday upgraded to 128GB RAM on my plotting rig and on my first plot I had 100% cache hit rate with 0 writes to the disk until the second plot started after 1 hour. At that point L1 cache filled up completely, cache hit rate started going down and I started seeing actual disk writes. As additional workers kicked off things just got worse and worse. The screenshot below is the result of about 14-15 hours of runtime.

primo.png (89.19 KiB) Viewed 3735 times

I think the amount of data being created by plotting workers is too much to be cached effectively with so (relatively) little RAM. So I did not expect PrimoCache to do miracles. Yes the trimmed blocks numbers do look nice and I appreaciate this number of blocks won't be written to SSD and will reduce the wear. But I was hoping it could maximize disk write speeds. As I mentioned earlier the disk is capable 3GB/s+ write speeds and I do get that speed with just a single plotter worker running. But with 7 workers in parallel the disk write speed seems to be capped to 1GB/s or less. I guess this is because with this many workers writing becomes somewhat random (as opposed to sequential). I was hoping that PrimoCache would re-order deferred blocks into bigger chunks where possible and prioritise larger sequential sections for writing first. But to me it looks like first-in-first-out strategy is used.

[Jaga]

With 128GB of RAM, you'd want to create a RAMdrive and use the MadMax plotter with it. 0 writes to the drive, only need a ~110GB RAMdrive volume size for the Temp2 volume. Temp1 is where the other 25% of the writes happen, and can be a SSD or HDD.

[andriusst]

I have tried this MadMax plotter and I am damn impressed. But it is not zero writes to the disk. It says 25%. Still it is in the league of its own compared to the vanila plotter. I also tried to use PrimoCache instead of RAM drive and got some amazing results!

SoftPerfect Ramdisk / 116000MB / Plot time 2431.84 sec / disk writes=427,628MB

PrimoCache / 124000MB / Plot time 2368.24 sec / disk writes=372,206MB*

* (I flushed the cache contents to make the test fair but that added about 100,000MB to the final figure. I won't be doing that for normal plotting though so disk writes will be even less)

So just to sum up with PrimoCache plot completes faster AND there are less writes to SSD.
I am sooo happy! Thank you for pointing to MadMax plotter. He deserves thanks too! I just hope the plots are as valid as the ones made by the vanilla plotter.