Preload currently accessed files -> improve Audio/Video/Film

[Axel Mertes]

Hi!

We do mostly audio/video/film production and therefor work often with streaming data and very large video files.
Unfortunately many video editing and playout systems are designed with a "brute force" method in mind, requiring storage that even under the worst circumstances will not fail to perform the desired bandwidth and throughput without latency.
This is essentially a bad thing, as a simple read-ahead cache would solve so many issues easily.

So, as far too many vendors (include Adobe, AVID, etc.) are apparently not addressing the problem and pointing to the hardware side, I wonder if its possible to work around this.

Two approaches come to my mind:

The first approach would be to identify the desired file using the accessed block number by some kind of a reversed MFT. So if you know which file containes the currently loaded block, you may be able to preload subsequent blocks - regardless of their order on the disk. A policy could make sure to e.g. read ahead a given amount of blocks from any currently open file. This would end in an effective pre-loading scenario OUTSIDE the host software. IMHO that would be a really huge deal, as we currently see nearly no benefit with PrimoCache in this area at all.

The second approach would be to not identify files, but read ahead simply the next x blocks given from the last accessed ones. On a defragmented volume that will immediately result in preloading the right data in most of the cases. The policy would be very simple (and IMHO could be easily integrated short-hand into the existing PrimoCache) by setting a number of blocks to read ahead. Say one uses a specific drive for dedicated audio/video recording (other files are stored on a different drive), then its pretty safe to preload say the next x blocks as soon as possible. In a sequential read attempt on a defragmented drive its very very very likely that these preloaded blocks are the next ones to be accessed. This would immediately overcome many performance hit problems like external user access, directory loading etc. while playing back data.

If someone around does audio / video / film editing / playout as well, please chime in to raise the demand.

I clearly know that the first approach is very complex to address. However, the second one is so simple that is screaming for a implementation as far as I can tell. Call it a specific A/V mode or streaming mode.

Best regards,
Axel

Magna Mana Production
Bildbearbeitung GmbH

[Support]

Hi Axel,

Thanks! The method 2 has already listed on our tasks.

However, actually Windows has its file-level caching mechanism that allows read-ahead. Do you think a read-ahead cache will help to this issue?

[Axel Mertes]

Yes, I would think so having this caching would really help.

A real world example:
We recently needed to playout a HD video stream (ProRes codec, so ~20..25 MByte/s) from a defragmented RAID6 EonStore system with 8 spinning disks in the RAID set, connected with 4 GBit FibreChannel. When we do benchmarks, we measure 360 MByte/s on average for read and write speeds (writing sometimes a bit higher) on this particular system with a 6 TByte netto RAID partition. The RAID configuration is explicitely set to "A/V Streaming" mode. However, when doing a print to tape operation in PremierePro CC, we do actually see latencies that cause PremierePro CC to drop frames and stop the print to tape process. So having a brute force RAID with like 15+ times the required performance does still not solve the problem of dropped frames due to latencies.

When playing, we sometimes see the memory usage grow linearly. I think this is when Windows decides to cache the file, as we see no particular application with growing memory, so it must be a hidden system process. Apparently it does not seem to read ahead, but simply keeps everything already read in memory in case you want to go back to it again (unlikely in a video playout...). At a given point memory is full and then presumably old cache entries are kicked out. However, sometimes we do not see a growing memory consumption at all, but a straight line with say 3.5 GBytes (out of 12 GBytes on that machine) versus the 11.9 GBytes we see when it happens. So we do not even understand when Windows begins caching and when not. And, after all, drop frames occur in both situations.

Btw, where do have the information from that Windows does actually read ahead?

Obviously Adobe should have taken care to use a small amount of memory as cache. Running such an application "just in time" is a pretty bad thing. I need to talk to a friend of mine, who is actually the R&D head for PremierePro, After Effects and SpeedGrade about this. In SpeedGrade (formerly Iridas) this has NEVER been an issue, so its about time he takes care of introducing this into other Adobe apps.

By forcing a read ahead and a caching (actually a read into cache in this scenario) we may be able to overcome the problem for simply ANY application. It is more than worth trying!

Best regards,
Axel

[InquiringMind]

...So having a brute force RAID with like 15+ times the required performance does still not solve the problem of dropped frames due to latencies.

As an aside, have you tried using an SSD drive? Their near-zero latency may be more useful in this case than your RAID's throughput, and if you're working on only a couple of projects at a time, you could use one as temporary storage - with file versioning software like Aphar Backup (free) or FileHamster (commercial) copying it to RAID whenever it is updated, if you're paranoid about data loss.

...When playing, we sometimes see the memory usage grow linearly. I think this is when Windows decides to cache the file, as we see no particular application with growing memory, so it must be a hidden system process.

Windows' cache is not included in overall memory usage by Task Manager (since the memory is given up whenever an application needs it) but its current size is reported in the "System Cache" figure.

If Task Manager isn't reporting which application is using that memory, then Process Explorer or Process Hacker may be worth a try. If some other application is using large amounts of memory, it could trigger pagefile swaps which might (if the pagefile is on the same RAID system as your video data) be causing the frame drops.

The symptoms you report could result from an antivirus scanner (checking/processing those large video files every time you change them) so configuring it to ignore those files might be worth trying.

[Axel Mertes]

Hi,

yes, we do use the SSD (OCZ RevoDrive 3x2) when printing to tape now, but its annoying (manual) workaround, after all.
It would be much more convenient if there is a transparent cache working in the background and you do not need to make double-house-holding or moving data around at all. Its all about the user experience

Currently we use SuperFlexibleFileSynchronizer, but I will have a look at those Aphar Backup and FileHamster.

We do not have any antivirus etc. software activated on the system, though. But I will investigate.

The fact that we see it working nicely from the SSD is the reason I brought it up here. In fact the RAID is by far faster than required - on average. So if we can use the tremendous power to fill an SSD and / or RAM cache "in time" the whole thing should work out of the box, what do you think?

I find it pretty straight forward thing to implement.

Say you read block 1000 from a disk, and your cache policy is set to read ahead the next 50 blocks, then just do so, read block 1001 to 1049 as well. Clearly this will not work very good with fragmented files or will cause unnecesarry caching on files that are simply at their end or short. But the latter doesn't matter if I can playout without headaches.

Best,
Axel

[InquiringMind]

yes, we do use the SSD (OCZ RevoDrive 3x2) when printing to tape now, but its annoying (manual) workaround, after all.
It would be much more convenient if there is a transparent cache working in the background...

Primo Cache's L2 caching (using a specially formatted SSD) looks, on the surface, to be ideal for your situation. Has it been tried?

Currently we use SuperFlexibleFileSynchronizer...

Good heavens, what a mouthful of a name. They've changed it to Syncovery now though. If you're using its Real Time Synchronization feature, you probably don't need to consider anything else (it makes copies after changes, whereas the others copy before change).

[Axel Mertes]

Yes, we have tried PrimoCache, but the problem persist until the whole file got cached.
So a cache policy reading really ahead and storing that read-ahead-data on the SSD transparently would surely solve the problem "out of the box".

Yes about the name, I forgot they changed it to Syncovery... I am still used to that old name

The whole point is - again - I manually need to set up rules and such. Very uncomfortable from a user perspective. My users here simply expect that whatever data they touch should be accessible fast. I can explain them that the first touch will be slower, but then the cache should chime in and next time it should be way faster.

One point to note:
If such a read ahead cache policy is introduced, its clear that you need to maintain the information if a block was accessed by another software and thereby "just cached as needed" and if subsequent blocks were only read by the cache policy forcing them to be read into the cache. The state of each block must the change from "read ahead" to "accessed by software" whenever it has been really touched by any software. In that moment, the cache policy would need to check again if enough subsequent blocks have already been read into the cache. If not, the missing blocks need to be accessed - and so on.

Now don't tell me which other software I should check out - make it happen in PrimoCache and I will be really happy!

Best regards,
Axel

[InquiringMind]

Setting a large block size (the maximum 512KB) may help out, if you've not already tried it.

[Axel Mertes]

Hi,

we have tried 4K blocks on the RAIDs. It basically works, but the speed improvement is rather minimal. Some software is optimized for 512byte blocks instead.

However, after IBC show and looking at many vendors solutions, I am more than sure that having this kind of caching like described in my above idea is really the way to go.

So please make it happen. I bet it'll work wonders!

Best,
Axel

[frosti7]

i'm also a video editor that is following this topic, any news?
did primocache helped somewhat in this situation?