Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

FAQ, getting help, user experience about PrimoCache
PhattMatt
Level 3
Level 3
Posts: 11
Joined: Sun Jan 05, 2025 6:38 pm

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by PhattMatt »

Nick7 wrote: Wed Jan 08, 2025 2:38 pm For one, why setup array in storage spaces, and use that for cache?
You should use disks themselves, without SS for caching.

2nd - you are confusing some things. Freeing cache on written is freeing cache within L2 cache in Primocache.
This will *NOT* free up space in underlying caching device! Especially in thin provisioned SS drive!

Again, please use SSD's for caching without SS!
I sometimes have to downloads massive amounts of data in large amounts continuously for a day or two at a time. I can download up to 20 TB over two days. I needed a very large very fast cache. The cheapest solution was to get a few 4TB drives and combined them into a logical volume to act as the cache. Primocache wont let me point multiple ssds to the same volume so without using storage spaces, I don't know how to combined their capacity to act as a cache and point them at the same drive. If I can't use storage spaces how would you recommend I go about setting this up?
Nick7
Level 5
Level 5
Posts: 52
Joined: Sun Jun 25, 2017 7:50 am

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by Nick7 »

Ah, you are correct. It is possible to only use one device for caching one volume.

In case of using SS be aware of thin provisioning. When you create volume, leave some free space in pool.
What seems to happen is SS pool runs out of space.
PhattMatt
Level 3
Level 3
Posts: 11
Joined: Sun Jan 05, 2025 6:38 pm

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by PhattMatt »

Nick7 wrote: Thu Jan 09, 2025 9:50 am Ah, you are correct. It is possible to only use one device for caching one volume.

In case of using SS be aware of thin provisioning. When you create volume, leave some free space in pool.
What seems to happen is SS pool runs out of space.
Provisioning would only come into play if i was using a mirror or a parity but i'm not. It was a simple volume with no resiliency. Also even with thin provisioning if there are no deferred blocks to write the pool should only be filled a small percentage to account for the overhead of the provisioning but that's not the case here. The storage space would always climb to %100 filled even if at the time it hit %100 there was only %2 deferred write blocks.
Nick7
Level 5
Level 5
Posts: 52
Joined: Sun Jun 25, 2017 7:50 am

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by Nick7 »

I don't know internal logic of Primocache, but it is completely feasible that it fills up L2 sequentially, and new data is initially always added to end.
This way it'll easily fill up 100%.

But, nonetheless, you need to expect for L2 to fill up all space, or underlying device for L2 to be 'full'. Just leave some free space in pool.
Even with no resiliency you can still overprovision drive in SS.
User avatar
Support
Support Team
Support Team
Posts: 3713
Joined: Sun Dec 21, 2008 2:42 am

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by Support »

PhattMatt wrote: Wed Jan 08, 2025 5:28 pm I sometimes have to downloads massive amounts of data in large amounts continuously for a day or two at a time. I can download up to 20 TB over two days. I needed a very large very fast cache. The cheapest solution was to get a few 4TB drives and combined them into a logical volume to act as the cache. Primocache wont let me point multiple ssds to the same volume so without using storage spaces, I don't know how to combined their capacity to act as a cache and point them at the same drive. If I can't use storage spaces how would you recommend I go about setting this up?
How about making a hardware RAID? Most motherboards support RAID.
PhattMatt
Level 3
Level 3
Posts: 11
Joined: Sun Jan 05, 2025 6:38 pm

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by PhattMatt »

Support wrote: Fri Jan 17, 2025 4:33 am
PhattMatt wrote: Wed Jan 08, 2025 5:28 pm I sometimes have to downloads massive amounts of data in large amounts continuously for a day or two at a time. I can download up to 20 TB over two days. I needed a very large very fast cache. The cheapest solution was to get a few 4TB drives and combined them into a logical volume to act as the cache. Primocache wont let me point multiple ssds to the same volume so without using storage spaces, I don't know how to combined their capacity to act as a cache and point them at the same drive. If I can't use storage spaces how would you recommend I go about setting this up?
How about making a hardware RAID? Most motherboards support RAID.
I am unable to install the drivers for hardware raid (Intel RST), without a fresh install of windows. Is there no other solution? I don't claim to understand the mechanics of how primocache works bit it seems like this should be something i'm just able to do.
User avatar
Support
Support Team
Support Team
Posts: 3713
Joined: Sun Dec 21, 2008 2:42 am

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by Support »

PhattMatt wrote: Sun Jan 19, 2025 1:45 pm I am unable to install the drivers for hardware raid (Intel RST), without a fresh install of windows. Is there no other solution? I don't claim to understand the mechanics of how primocache works bit it seems like this should be something i'm just able to do.
Hardware raid does not need Intel RST. It can be made via motherboard BIOS.

Another way is not to set "MAX" in the L2 cache size setting, but to leave some space.
PhattMatt
Level 3
Level 3
Posts: 11
Joined: Sun Jan 05, 2025 6:38 pm

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by PhattMatt »

Support wrote: Mon Jan 20, 2025 3:40 pm
PhattMatt wrote: Sun Jan 19, 2025 1:45 pm I am unable to install the drivers for hardware raid (Intel RST), without a fresh install of windows. Is there no other solution? I don't claim to understand the mechanics of how primocache works bit it seems like this should be something i'm just able to do.
Hardware raid does not need Intel RST. It can be made via motherboard BIOS.

Another way is not to set "MAX" in the L2 cache size setting, but to leave some space.
When I turn on hardware raid I always get a bsod "Boot device inaccessible". From what i can find on the internet its because I need the drives first.
User avatar
Support
Support Team
Support Team
Posts: 3713
Joined: Sun Dec 21, 2008 2:42 am

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by Support »

PhattMatt wrote: Sun Jan 26, 2025 1:39 pm When I turn on hardware raid I always get a bsod "Boot device inaccessible". From what i can find on the internet its because I need the drives first.
Then I think the way you can try is not to set "MAX" in the L2 cache size setting, but to leave some space.
PhattMatt
Level 3
Level 3
Posts: 11
Joined: Sun Jan 05, 2025 6:38 pm

Re: Using an array of M.2 to cache for HDD array causes M.2s to fill continuously

Post by PhattMatt »

Support wrote: Fri Jan 31, 2025 1:14 pm
PhattMatt wrote: Sun Jan 26, 2025 1:39 pm When I turn on hardware raid I always get a bsod "Boot device inaccessible". From what i can find on the internet its because I need the drives first.
Then I think the way you can try is not to set "MAX" in the L2 cache size setting, but to leave some space.
This was already attempted prior to me posting this. The deferred blocks are only ever reaching %50 but that does not stop the pool from filling up even when the deferred blocks go back to 0.
Post Reply