Caching Options for Downloads Drive

FAQ, getting help, user experience about PrimoCache
hshah
Level 3
Level 3
Posts: 12
Joined: Sun Jun 09, 2019 9:41 am

Caching Options for Downloads Drive

Post by hshah »

I am just having a play around and kind of seeing what my options are. I have a HDD for downloads/torrents etc but Windows/applications all live on a NVMe. Anything download either staying on the HDD for a bit, or gets copied over to my NAS shortly after.

I have ~2GB RAM free and have also tested creating a L2 cache as an image on the NVMe. I had set L1 as 1GB and L2 as 10GB.

Everything appeared alright, but I don't actually know what would be beneficial and wouldn't. Based on what I have said, can anyone suggest anything? I'm thinking that a write cache would definitely help because concurrent downloads tend to put the drive under strain, especially when torrents are involved and they are being seeded. I have Googled a bit and if I read things correctly, L2 for writes could in theory cache a downloaded torrent so that any seeding uses data from there rather than the HDD?
hshah
Level 3
Level 3
Posts: 12
Joined: Sun Jun 09, 2019 9:41 am

Re: Caching Options for Downloads Drive

Post by hshah »

Also, I don't fully understand whether the L1 (1GB-2GB RAM) ought to be set as read or write? If both, should the space be shared or have a ratio defined from the start? My L2 is a 12G file-disk on a NVNe, and was set up with RAMDisk as a SCSI. It was then formatted as L2STORAGE in PrimoCache. Similarly, should this one be read/write and shared or have the space defined?

There would be more seeding taking place when new torrents are involved, so it would be beneficial if L2 could serve those requests. The HDD obviously struggles with multiple concurrent read/writes, and that becomes really noticable when my scripts copy the downloaded media over to a NAS and there are still multiple download/uploads going on. Could there a brief delayed write set and L! reserved for that?

Not sure if L1 passes through L2 to get to the HDD, especially if L2 has been set as 100% read. Overall, I've not got my ahead around all this yet so any help would be greatly appreciated :)
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Caching Options for Downloads Drive

Post by Support »

I think L1 write-only cache (100% write cache space) with defer-write enabled shall help downloading. You may check the cache statistics to see the performance and tune the configuration.

PS. it is not recommended to use a file-disk or virtual disk as level-2 storage. Dead-lock might happen if doing so.
hshah
Level 3
Level 3
Posts: 12
Joined: Sun Jun 09, 2019 9:41 am

Re: Caching Options for Downloads Drive

Post by hshah »

support wrote: Tue Jun 11, 2019 4:29 am I think L1 write-only cache (100% write cache space) with defer-write enabled shall help downloading. You may check the cache statistics to see the performance and tune the configuration.

PS. it is not recommended to use a file-disk or virtual disk as level-2 storage. Dead-lock might happen if doing so.
I am only able to spare 2GB of RAM at most for L1 so was hoping to back that up with a disk based L2. Was also hoping that placing the image on a NVMe would still benefit the standard HDD.

I could partition the NVMe and allocate L2 to the HDD that way?

I've just noticed that I can't copy some films from the HDD to my NAS... just hangs for ages and then Windows shows some sort of error. I am assuming that is is trying to do something with the cache and then falling over... assuming its down to the file based L2 lol.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Caching Options for Downloads Drive

Post by Support »

hshah wrote: Tue Jun 11, 2019 6:49 pm I could partition the NVMe and allocate L2 to the HDD that way?
You may shrink the existing NVMe ssd partition and then create a dedicated partition by spare ssd space for L2 cache.
hshah
Level 3
Level 3
Posts: 12
Joined: Sun Jun 09, 2019 9:41 am

Re: Caching Options for Downloads Drive

Post by hshah »

Just about to try this out now and was wondering what sort of L2 size you would recommend? L1 can only go up to 2GB due to RAM limitations. If it makes a difference, the HDD in question is a 2TB Western Digital Black.
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Caching Options for Downloads Drive

Post by Jaga »

With only 2GB for the L1, it's a waste to put that as a read/write L1 on the 2TB drive - the read hitrate on the L1 would be abysmal. Make it a write-only L1 and then as large a read/write L2 as you can use for the rest of that cache task. If you are partitioning off part of your NVMe (boot drive?) for the L2, remember to leave empty space at the end of the NVMe to serve as overprovisioning (typically 5-10% of the total drive space on the SSD/NVMe).

The problem overall is the size of total data residing on the 2TB drive, and the resulting hitrate with small L1/L2 caches. If the 2TB is fairly full, you'll need a good sized L2 to adequately cache frequently used items.
hshah
Level 3
Level 3
Posts: 12
Joined: Sun Jun 09, 2019 9:41 am

Re: Caching Options for Downloads Drive

Post by hshah »

Roughly 1.5TB is occupied, but at least 1TB of that is like a garden shed... a dump of old data that may never see the light of day lol. In terms of active use, downloads are written to the drive before being moved elsewhere. Torrents are seeded for a while, so the data is copied elsewhere rather than being moved.

After a few days of testing a 2GB L1 write only cache, the hit rate was around 20%. Previously, when downloading a newly released torrent (~1GB), copying to a NAS took a ridiculously long time, and speeds of any active downloads/uploads significantly dropped during this time. With the L1 in place, the copying had no impact on the downloads/uploads and vice versa.

Funnily enough, I now have more RAM available because I found that storing my Plex/Emby database/cache/metadata on a RamDisk was no faster than the NVMe storage, all whilst adding risk/complications.


The plan:

L1 - 8GB RAM
L2 - 12GB NVMe
  • Should I now make both read+write and let PrimoCache manage everything?
  • I don't understand the empty space part, but have left ~500MB at the end. Before I made changes, C: went all the way to the end
Annotation 2019-07-01 084439.png
Annotation 2019-07-01 084439.png (8.41 KiB) Viewed 5685 times
ferrari
Level 4
Level 4
Posts: 27
Joined: Fri May 17, 2019 3:40 am

Re: Caching Options for Downloads Drive

Post by ferrari »

Yeah I'd do read/write for both with that much ram, L2 might not even be necessary. Also it's good health for the SSD to leave maybe 10% of the drive unpartitioned you only seem to have 1% unpartitioned.
User avatar
Jaga
Contributor
Contributor
Posts: 692
Joined: Sat Jan 25, 2014 1:11 am

Re: Caching Options for Downloads Drive

Post by Jaga »

Yep, agree with ferrari on your new changes. Switching from a 2GB L1 to 8GB will dramatically change your hitrate, so give it full read/write on both caches. I'd still do a L2 given the size of your data set (500GB) to keep hitrate high. And it's a NVMe, which means the L2 will give very snappy access times compared to a typical SSD.

The unallocated space is just overprovisioning to help with SSD life. You currently have it set to 1% of the drive space, but it's enough to help even a tiny bit. Typically it would be between 5% and 10% of drive space (5-10 GB).

Give it a test and see how it does, you should be pleased with the results.
Post Reply