Progress during flush or stop

Report bugs or suggestions around FancyCache
edv
Level 3
Level 3
Posts: 15
Joined: Thu May 10, 2012 12:21 pm

Progress during flush or stop

Post by edv »

Lately I have been using FancyCache (0.8.0) with a large amount of RAM allocated to the write cache (60 GB). It's been working well, but I have a small usability suggestion. I am using a slow hard drive (USB2 external) and if I flush the write cache (or stop caching, which does the flush first) I have no idea how long it's going to take (and writing 60 GB of data does take a while; upwards of 40 minutes at a 25 MB/sec write speed). If I have the presence of mind to open the performance monitor before I flush the cache, I can watch the "Deferring Blocks (Current)" count go down, which gives me a pretty good idea of progress, but if I didn't remember to open the monitor, once I click "Flush Write" I am unable to open the performance monitor and am left with an activity indicator on a "Processing, please wait..." dialog in lieu of any meaningful progress bar or number of blocks remaining to write.

It's not a terrible nuisance but given that FancyCache knows at some level how many blocks are left to write, would it be possible to replace the activity bar on the "Processing, please wait..." dialog with an actual progress bar that counts down the remaining deferred blocks?

(By the way, using FancyCache on this external drive makes an astounding difference in usability - many kudos to the developers and whoever thought up the idea of FancyCache originally!)
dustyny
Level 8
Level 8
Posts: 118
Joined: Sun Sep 02, 2012 12:54 am

Re: Progress during flush or stop

Post by dustyny »

This isn't a very well known fact to users but 99% of the progress bars that you see are just arbitrary and they are there only to give the user comfort that activity hasn't stopped. There really is no accurate way to predicate how long it will take to write data to your disk, as the write speed of the disk varies wildly over time. So many variables affect the write speed, file size, file system (NTFS, FAT), block size, platter speed, what part of the platter is being written to (outer vs inner), HDD memory cache size & speed (HW not SW), your storage controller(s), the I/O interface (SATA, SAS, USB, etc), cable length, the OS etc, the list goes on and on.

To be honest this is really an issue of you doing something you shouldn't. If you are talking 40 minutes of write delay, you are taking an extremely HUGE (let me repeat that to drive it home, ridiculously HUGE) risk with your data. If this data is important to you in anyway, you really want reduce your write delay to seconds at most (if it's truly important use no write delay at all), not minutes and certainly not 40 minutes. In the event of crash or power loss, at best you'll lose all that data or (depending on the apps you use) you could corrupt a huge amount of data on your HDD. Now on the flip side, you can have a absolutely tremendous read cache and it is 99.99999% fine.

With this in mind, I'd like to piggyback a suggestion on to this thread. The write cache delay should have an alert that pops up when users try to set excessively large write caches alerting them that this behavior is very risky and that it should only be used by advanced users for very specific use cases (such as temp files).
edv
Level 3
Level 3
Posts: 15
Joined: Thu May 10, 2012 12:21 pm

Re: Progress during flush or stop

Post by edv »

It might not be an accurate measure of time, but it would be an accurate measure of "blocks remaining to write to the drive" The counter in which I am interested is already in the Performance Monitor; it would just be nice to have it on the "please wait" dialog from the main window as well, in case I don't have the Performance Monitor open (alternately, the option to open the Performance Monitor during a stop/flush would serve the same purpose, albeit with some extra clicks).

As to whether it's a good idea, that completely depends on what the purpose of the data is. I can probably go a year without a power failure or other catastrophic data-loss event, and if the worst that happens in that unlikely situation is that I need to re-run the process that generates the huge amount of data, that's acceptable to me. In my opinion, "it shouldn't be an option because most people don't understand what they're risking" is a poor argument for such a technical utility. I use large write caches almost exclusively due to the nature of the processes I am running, and if a warning were added I would at least like the option to disable said warning after viewing it once.

In my current situation, I frequently delete a lot of the written data within a few minutes of writing it, so having it flush to the drive is just literally a waste of time (I used to use a RAM drive for these processes, but FancyCache is far more convenient in that I don't have to guess what size to make the drive; any overflow will simply be actually written to the drive instead of causing the process to receive a drive space failure). It is true that I don't usually have 60 GB of data to write; I am simply testing the limits of the software at the moment. However, even if the cache flush were going to take only a minute, it would still be nice to see the blocks remaining as a tangible measure of progress.
dustyny
Level 8
Level 8
Posts: 118
Joined: Sun Sep 02, 2012 12:54 am

Re: Progress during flush or stop

Post by dustyny »

Have you considered that this is only an issue that is related to your specific setup and your workflow? You have a huge bottleneck and the only reason why a progress bar would be of any use is because you're using a huge write delay and trying to push that data through such a small pipe. Most users if they do use this feature should keep it very small in order to reduce putting their data at to much risk, so writing to disk should be very quick. If they are using the product safely (being a relative term here), the progress bar would go down to zero to fast to be of any real use.
In my opinion, "it shouldn't be an option because most people don't understand what they're risking" is a poor argument for such a technical utility
You do understand when you quote someone you're actually supposed to use the exact words they said right? Abuse fancycache all you like but you leave basic grammar out of this! :lol:

What I said was..
"The write cache delay should have an alert that pops up when users try to set excessively large write caches alerting them that this behavior is very risky and that it should only be used by advanced users for very specific use cases (such as temp files)."

You obviously fall under the special use case category (temp files). Fancycache isn't an enterprise class product and from my experience on the forums I'd say most of it's users are hobbyists & enthusiasts. As a result most users of Fancycache are not technical enough to understand how the write buffer works and the risk that comes with using a very large delay. Hence the reason for the warning.

I actually assumed a technical user wouldn't have any issue with the write delay as they should understand what to expect. I figured most technical people would know how to calculate the cache to disk transfer rate and be able to roughly predicate how long it will take to flush to disk. I use a 9GB cache with a raid 0 array (average R/W 800MBs), so I know that it takes 12-15 seconds to flush the cache to disk. Its a very easily calculation that I do in my head, cache size divided by average transfers speed equals approximate write duration.
I am simply testing the limits of the software
Actually in this case you are testing the limits of your hardware. When you have such a major bottleneck that's the issue, this is causing a problem for you because it's not really a practical thing to do. From what you are telling me it sounds like you are using an USB hard drive or a flash drive, when you should really be using an SATA HDD or better yet an SSD to handle your "overflow" (I'd probably use few SSDs in a RAID array but that takes $).

I'm not trying to offend you, I'm just pointing out that this is a rare situation that is particular to how you are trying to use the product. The problem is your H/W.
edv
Level 3
Level 3
Posts: 15
Joined: Thu May 10, 2012 12:21 pm

Re: Progress during flush or stop

Post by edv »

I don't understand the hostility to the suggestion, frankly. The feature I mention already exists; it is the "Deferring blocks (current)" number on the Performance Monitor dialog.

- I can watch this number decrease during the cache flush if I opened the dialog before clicking "Flush Write" on the main dialog.
- I like to watch the number.
- I would like to be able to watch the number even if I forgot to open the Performance Monitor first.
- So I requested that this number be on the "Please wait" dialog

The only reason I gave all that other information in my initial post was to explain how I got to the point where I was waiting for an operation to complete for which I would have been able to see an indicator had I opened a previous dialog. Why are you responding with reasons why I shouldn't want to see this number that already exists on another dialog? Are you a developer of FancyCache? I don't know of any other way to phrase the request.
dustyny
Level 8
Level 8
Posts: 118
Joined: Sun Sep 02, 2012 12:54 am

Re: Progress during flush or stop

Post by dustyny »

Dude I'm not hostile to this idea, I'm just pointing out that your suggestion solves a problem that only you (and perhaps a very rare few) are experiencing. Write defer is a dangerous feature and someone using extremely high defer length should be very rare. Most people will never see this progress bar and if they do it will disappear so quickly that it's useless, I'm confused as to why you don't get that.
edv
Level 3
Level 3
Posts: 15
Joined: Thu May 10, 2012 12:21 pm

Re: Progress during flush or stop

Post by edv »

I posted the suggestion because I used the software and thought of something I would like to have. When I think of something I would like beta software to do, I don't try to guess whether everyone else using the software might feel the same way; that is the job of the product manager or usability expert. I only mention my individual desire and hope that whoever adds features to the product will feel the same way. I don't see a purpose to the repeated information that my situation is uncommon unless you are the developer or other representative of the project explaining to me why you aren't going to take the time to implement the idea.
dustyny
Level 8
Level 8
Posts: 118
Joined: Sun Sep 02, 2012 12:54 am

Re: Progress during flush or stop

Post by dustyny »

You're not taking it as I intend but I am trying to help you out. :D

If you brought your car to the dealer and complained that you wanted to drive your car 80MPH (128km/h) in the reverse gear, would you be surprised if they told you that it's not designed to do that? Would you argue the point that they should add that feature or would you accept that you're doing something wrong?

You're trying to run fancycache with a huge write delay while handicapping it by skipping the SSD and your trying to use a USB HDD as well. Not only that but you want to be able to write a massive amount of data to the drive that you generate over a short period of time. For some reason you stop the caching often and you like to watch the blocks count down (perhaps you have a good reason but it seems kinda OCD from what you've told me) but you don't want to do the 1 single click it takes to open the performance monitor. Take a step back and think on that for a sec.

I'm not the developer but I build enterprise class storage systems and I've spent the last year doing some deep level testing of this product. Trust me when I say, if you just used the product as it was intended, you'd instantly see why your suggestion doesn't make sense. Fancycache is obviously flexible enough to work like this but because of your bad configuration, you're going to see behavior none of the rest of us will see. That is what I'm pointing out, the only people that will really see this progress bar are people who have the product very badly configured. That is not the sort of behavior that you want to engineer in to a product.

So as I said in the very beginning, you are using the product improperly and what you are seeing is a result of the approach that you have taken. You gave your suggestion but I given you the solution to the problem that your feature would try to address. I'm sorry if you don't like it but this is a H/W issue, give it the H/W it needs and the issue disappears.
Last edited by dustyny on Tue Mar 12, 2013 4:48 pm, edited 1 time in total.
edv
Level 3
Level 3
Posts: 15
Joined: Thu May 10, 2012 12:21 pm

Re: Progress during flush or stop

Post by edv »

I do not need or want help with what you are determined to believe is a problem with the way I am happily using this software; this is a forum for bug reports and suggestions, not technical support. The simple fact is that FancyCache does precisely what I want it to do regardless of how much you might wish that I were uncomfortable using it that way. I will leave my suggestion as it stands and I will continue to use the program the way I am. I will try to always remember to click "Performance Monitor" before "Flush Cache" if the developers decide that it is not worth their time to allow me to open the Performance Monitor during the flush or show me the numbers on the "Please Wait" dialog.
dustyny
Level 8
Level 8
Posts: 118
Joined: Sun Sep 02, 2012 12:54 am

Re: Progress during flush or stop

Post by dustyny »

I sincerely do hope it does perform well for you, it's a great product.

Good luck edv..
Post Reply