Any way to limit defer-write speed/utilization?

FAQ, getting help, user experience about PrimoCache
Extremity
Level 1
Level 1
Posts: 3
Joined: Wed Oct 26, 2022 5:16 am

Any way to limit defer-write speed/utilization?

Post by Extremity »

I'm using a 500GB NNVMe drive as an L2 write-only cache for six 8TB drives that contain media files, as I'm sick of waiting for things to transfer. It's been working great, but I've been trying to figure out one thing.

My write cache is large enough that I don't really have to worry about it filling up as long as data is slowly and steadily moved off of it. I would like to avoid the HDDs spiking up to full utilization when not necessary - I get that the goal is probably to get the files off the cache ASAP once the interval/conditions are reached due to volatility, but still, an option - even if suggested against - would be nice.

So, I'd like an option to "stream" the data from the cache to the destination at a rate of my choosing, either based on a fixed speed or a percentage of total available throughput. For example, instead of having a 4GB file moved to the cache and then a transfer to the destination HDD starting at maximum speed whenever the interval/conditions are met, I would like the ability to do this but limit the transfer speed to, let's say, 50MB/s, or 40% of rated write speed, either or. I could then adjust my interval settings as needed taking this into consideration; perhaps I want to write more frequently but at a lower speed, or less frequently at a higher one.

Is this something that's possible? The other conditions would still be in play, the write interval options, it would simply limit the write speed.

Is this even possible to implement? Basically, it would let me prioritize things I am actively doing on the system while letting the cache drive constantly stream its contents to the destination drives over time. I know that the OS should attempt a balancing act between transfers but it never seems to prioritize what I want. It'd be so, so nice to just be able to say "cache operations transfer at XMB/s" and be done with it.

I was hoping the "Average" write option would do this, lower the write speed to an average over time, but it's not. Apparently it only averages out how long writes are deferred - the disk will still always be utilized at 100%, it's the amount of time they are utilized that is averaged and not the speed. Unfortunately, this does not correspond with my use case.

Any help would be appreciated.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: Any way to limit defer-write speed/utilization?

Post by mell111 »

Interesting request. I can see two circumstances where this would be useful:

1. When you want to read from the disks while they're being written to.

2. For older systems with slower PCIe bus, extended writes to disks may slow down the whole system considerably while they are ongoing.

While you're waiting, you may want to look into a free utility called Ultracopier that allows you to do exactly what you want when copying files. This would limit the rate at which PrimoCache gets the written data, which would allow you to set a short latency interval in PrimoCache, which would, in turn, limit the amount of time it "hogs" the bus and drive. Of course, this only applies when you're copying files. If/when you use an app that does a lot of concentrated writing, you would need to adjust accordingly.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Any way to limit defer-write speed/utilization?

Post by Support »

I'm sorry but currently no. However, we do plan to support this feature in order not to make the whole system busy in flushing. Thanks.
tverweij
Level 6
Level 6
Posts: 74
Joined: Thu May 10, 2018 9:27 am

Re: Any way to limit defer-write speed/utilization?

Post by tverweij »

When I first started using Primo Cache, I was under the impression that Average would do this - but regretfully, it doesn't.

It would second this idea, as it will remove the peak writes from the system - It would be nice if Average would calculate the rate every second instead of using a fixed one as proposed above.
Extremity
Level 1
Level 1
Posts: 3
Joined: Wed Oct 26, 2022 5:16 am

Re: Any way to limit defer-write speed/utilization?

Post by Extremity »

Well, I'm glad to hear this is at least coming one day.
mell111 wrote: Wed Oct 26, 2022 7:47 amWhile you're waiting, you may want to look into a free utility called Ultracopier...
As for the suggested software, I can't tell if you're saying it would limit the speed of the initial write (to the cache) or the second write (from the cache to the HDD). If the former, that's not the goal - everything would get written to cache at full speed, just not from the cache to the final HDD (the flush).
tverweij wrote: Thu Oct 27, 2022 11:24 am... I was under the impression that Average would do this - but regretfully, it doesn't.
I agree that "Average" would make more sense doing it this way. Right now it appears to just change how often the dumps happen, which seems unnecessary with all the other options provided.
mell111 wrote: Wed Oct 26, 2022 7:47 amWhen you want to read from the disks while they're being written to
To give some context, for me specifically it's what you guessed; this is for a relatively large media server and at any point content could be requested from these drives; relying on the OS to negotiate the read/write speeds never seems to work extraordinarily well, and if a video is requested while the cache is flushing to that specific HDD it's just as problematic as if I had simply moved the file there myself without the cache. With large, individual files there's no real difference (outside of it "feeling" faster, which is nice), it's doing the exact same thing just with a delay. If I were dealing with multiple smaller operations I can see the benefit. Limiting the flush speed, however, immediately provides a massive tangible benefit that I wouldn't know how to accomplish otherwise.

In addition, as I add more drives to the server I'll most likely continue using DAS (until I eventually cave and move to a rack-mount setup). These provide substantial throughout, but it begins to diminish if many/all drives are accessed at once. This doesn't happen very often but does now and then, and two HDDs being at 100% due to a cache flush can substantially limit the read speeds of the remaining drives until the flush is complete. Being able to negotiate the maximum write speed would also prevent this problem from occurring, in addition to the aforementioned issue with simultaneous read/write on an individual disk.

I know that development takes time and I don't expect an exact answer, nor will I hold you to it, but is this something you guys planned to implement in the near future, or is it on the backburner and happening whenever you have time over the next few years? Basically, I'm on a trial and am attempting to decide if the functionality will be available within the next year or so, or if I should start looking for other ways to accomplish this.

Either way, thanks for the quick replies. I do really like the software and see a plethora of potential uses.
mell111
Level 5
Level 5
Posts: 48
Joined: Fri Oct 05, 2018 11:16 am

Re: Any way to limit defer-write speed/utilization?

Post by mell111 »

Regarding Ultracopier, yes, it would only limit the rate of the input to the cache - it knows nothing of the cache and vice versa. Still, that should help indirectly if you set a low enough latency in PrimoCache, especially with a very low rate set in Ultracopier, because it would only write in short bursts (each at full speed, of course.) Since it's free and you're on a trial anyway, you may want to give it a shot - it worked for me in a setup similar to what you describe.

Oh, I do agree that this would be a very nice feature to add to PrimoCache.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Any way to limit defer-write speed/utilization?

Post by Support »

Extremity wrote: Fri Oct 28, 2022 1:33 am
tverweij wrote: Thu Oct 27, 2022 11:24 am... I was under the impression that Average would do this - but regretfully, it doesn't.
I agree that "Average" would make more sense doing it this way. Right now it appears to just change how often the dumps happen, which seems unnecessary with all the other options provided.
"Average" can limit write speed to some extent. However, if inbound write amount is huge, it still will try to flush at max speed in order to quickly complete the flush.
tverweij
Level 6
Level 6
Posts: 74
Joined: Thu May 10, 2018 9:27 am

Re: Any way to limit defer-write speed/utilization?

Post by tverweij »

When I look at the graphs (resource monitor, Queue length) it is always peaking.
And indeed, with a huge write for a few seconds, the following physical write is really big.

That is why I said it should be calculated as and average for the Average setting.

Example - a 5 second write delay, as I would expect the "Average" setting to perform:

The writes to cache:
Sec 1: 10 Mb is written
Sec 2: nothing is written
Sec 3: 5 Mb is written
Sec 4: 1 Mb is written
Sec 5: 5 Mb is written
Sec 6: nothing is written
Sec 7: 100 Mb is written
Sec 8: 1 Mb is written
Sec 9: nothing is written
Sec 10: 5 Mb is written
5 sec after that, nothing is written

Total transfer to cache in 15 seconds: 127 Mb - with a write peak of 100 Mb/sec

The writes to disk performed:
Sec 2: The last 5 seconds, 10 Mb was written, so write will be performed at 2 Mb /sec
Sec 3: The last 5 seconds, 10 Mb was written, so write will be performed at 2 Mb /sec
Sec 4: The last 5 seconds, 15 Mb was written, so write will be performed at 3 Mb /sec
Sec 5: The last 5 seconds, 16 Mb was written, so write will be performed at 3.2 Mb /sec
Sec 6: The last 5 seconds, 21 Mb was written, so write will be performed at 4.2 Mb /sec
Sec 7: The last 5 seconds, 11 Mb was written, so write will be performed at 2.2 Mb /sec
Sec 8: The last 5 seconds, 111 Mb was written, so write will be performed at 22,2 Mb /sec
Sec 9: The last 5 seconds, 107 Mb was written, so write will be performed at 21,4 Mb /sec
Sec 10: The last 5 seconds, 106 Mb was written, so write will be performed at 21,2 Mb /sec
Sec 11: The last 5 seconds, 106 Mb was written, so write will be performed at 21,2 Mb /sec
Sec 12: The last 5 seconds, 106 Mb was written, so write will be performed at 21,2 Mb /sec
Sec 13: The last 5 seconds, 6 Mb was written, so write will be performed at 1.2 Mb /sec
Sec 14: The last 5 seconds, 5 Mb was written, so write will be performed at 1 Mb /sec
Sec 15: The last 5 seconds, 5 Mb was written, so write will be performed at 1 Mb /sec
Sec 16: The last 5 seconds, 0 Mb was written, so write will be performed at 0 Mb /sec

Total transfer to disk in 15 seconds: 127 Mb - with a write peak of 22,2 Mb/sec
And all data has been written to disk within the 5 seconds write delay.
tverweij
Level 6
Level 6
Posts: 74
Joined: Thu May 10, 2018 9:27 am

Re: Any way to limit defer-write speed/utilization?

Post by tverweij »

Add to the previous post:

And there should be a maximum physical write (instead of the max speed of the drive itself) - user defined, to prevent the drive from being used 100% for writes at any time. The algorithm should compensate for that by increasing the needed write speed for as long as there is data in the cache that is older than the 5 seconds.
tverweij
Level 6
Level 6
Posts: 74
Joined: Thu May 10, 2018 9:27 am

Re: Any way to limit defer-write speed/utilization?

Post by tverweij »

One last addition:

When the write cache is almost full (90%), The physical write should still be at the user defined maximum instead of the max speed of the drive.
To make sure that the data can be written to the disk, that max speed that the system can write to the cache should be reduced to 80% of that user defined max speed.
This makes the writes really slow at that moment, but as the disk isn't going to 100%, the system won't be fully crippled, as reads are still possible.
Post Reply