Strict FIFO writing

[NickJ]

For system stability purposes, if I'm using FancyCache on my system drive, I want it to be no worse than not using FancyCache on a system drive in case of a power failure.

Currently if a power failure happens, I'm under the impression that the expected outcome may include corruption, since FancyCache seems to reorder sectors on write for efficiency due to the random write results and the supporting quote showing why it's faster:

3. With Defer-Write, FancyCache can combine neighbouring small writes into one large write and align writes with SSD’s erase block boundary. This feature brings the performance boost

Would it be possible to get a toggle for speed vs. strict FIFO writing for data integrity on power loss?

Random writes results showing that Fancy Cache is faster than random writes alone:
http://www.romexsoftware.com/en-us/fanc ... write.html

Thanks

[Support]

Well, I don't think a Strict FIFO writing will do much help for power failure issue. The results is quite unexpected on power failure or crash/hang, when defer-write is enabled. So usually Defer-Write is enabled on unimportant data.

Anyway I'd like to thank you for your ideas!

[NickJ]

Well, I don't think a Strict FIFO writing will do much help for power failure issue. The results is quite unexpected on power failure or crash/hang, when defer-write is enabled. So usually Defer-Write is enabled on unimportant data.

Anyway I'd like to thank you for your ideas!

From my perspective power failures happen on important data - systems bluescreen etc. If a system bluescreens with no caching, data is cut off at 12:23:47. If a system bluescreens with current caching, some data from 12:23:35 can be written, and some data from 12:23:40 can be written, while missing data from 12:23:38. This is a nightmare recovery wise, and the file is corrupted unless it can somehow tell it has to roll back the entire duration. If a system bluescreens with strict FIFO caching, instead of data being cut off at 12:23:47 it's cut off at 12:23:35.

Is there another source of data corruption writing from a cache during a power failure that I'm not aware of? I was under impression that the other risks were present regardless of caching or not. With the scenarios above, on any single user system I see no problem in running write caching on the system drive if and only if it's strictly FIFO.

Either way, I understand if write caching is only designed for non critical applications. I personally was very excited about the idea of a program that would let me write cache with RAM and read cache with an SSD at the same time for an arbitrarily large system hard drive without a stability hit though

[Violator]

Upon power loss and system failure only a battery on the controller and an UPS will save you with a lazy cache in use.
In case that disk caching stability is a requirement, your sitting with a server that includes a proper I/O card to handle the caching.
There is no way that you can archive the same thing in a programmable way.

[NickJ]

Upon power loss and system failure only a battery on the controller and an UPS will save you with a lazy cache in use.
In case that disk caching stability is a requirement, your sitting with a server that includes a proper I/O card to handle the caching.
There is no way that you can archive the same thing in a programmable way.

A proper I/O card will preserve the entire cache. I am not talking about preserving the entire cache - that preserves unwritten data. I'm talking about preserving disk integrity and stability. Preventing corruption of the OS and data already on the system. A caching program can be designed such that moving files on a hard drive itself, or moving from hard drive to hard drive results in no data loss. These are transactions that happen through RAM normally.

Let me try another example in which this will remove *some* (not all) of the instability removed in ram caching:

A set of files in Directory A is copied to Directory B. After each file's data is finished, the OS updates the file's metadata letting it know the file is finished. If the copy is interrupted during a file, one can tell that the metadata was never updated, and the file is likely corrupt.

In FancyCache currently the updating of each file's metadata will be optimized for write purposes, such that if the copy is interrupted during the write of a file the metadata may already be in its final state, even with a corrupt file.

In every case where the metadata of a file (or parts of a file) that is built to be fault tolerant via write order having the data written out of order causes that fault tolerance to no longer fill its objective - to make sure that it says the new data has been written only after it is actually written.

This allows for "FancyCache compatible" programs that are tested to not be corrupted.

As a programmer, I often think of ways to make my code tolerant to a power failure. However, most of them depend on updating metadata before and after possible points of failure for corruption recovery.