A solution to preloading game data for you gamers?

[Violator]

please be more specific how much overheads it will add in percents?

When you read from start to end C++ is up to twice as fast, since it doesn't need to do any decoding.
Scroll down to file reading: http://www.codeproject.com/Articles/212 ... arp-vs-NET

[Manny]

please be more specific how much overheads it will add in percents?

When you read from start to end C++ is up to twice as fast, since it doesn't need to do any decoding.
Scroll down to file reading: http://www.codeproject.com/Articles/212 ... arp-vs-NET

Very interesting article but we have slight different scenario.
First of all it reads text file, so .Net version converting it to UTF-16. You can read file in binary format, it will eliminate the biggest overhead.
The second main difference that you can't read huge files in "whole file at once mode", so we will need to read it by some pieces, and in that test C# is leading more then 5x.
Third difference is about filesystem speed and IOPs, because guy uses file from system cache. Lets calculate.
400kb * 20 times = 8mb
so C++ in whole file mode = 266 MB/s and 666 iops
C++ in per line mode = 66 MB/s and 166 iops - 2kk lines per second

C# in whole file mode = 160 MB/s and 400 iops
and in per line mode = 133 MB/s and 333 iops - 4kk lines per second

in per line mode he has size of block = 24 bytes (400kb*1024/17000lines)

My laptop hdd results:
512 bytes block = 89 iops, 0.05 MB/s, 1900 lines per second
1 MB block = 36 iops, 36 MB/s

as you can see any solution on any language is more efficient than my file storage.
lets take C# block mode, it is 3.6x faster than hdd, 9x faster by iops than hdd, and 2000x !!! faster in lines.

So any way you will be limited by your HDD than by software platform, an even if you have 8x SSD RAID 0, then you will probably win 1-2 seconds on 8GB file. But if we use binary mode then we will have difference in microseconds, instead of seconds.

[Violator]

Why would you want to read per line if all you need is moving a files raw data to ram?
I'm not saying that C# doesn't have it's advantages, I write a lot in C# myself too.
You got a point thou, it doesn't matter that much for a desktop system with low I/O, but as soon as you start to work with real storage systems it clearly has it's benefits.
Although, most of those already have I/O controllers or accelerators, which would kind of make such an application pointless.

To get back to the topic, creating such an app for that would be useless too.
Using a ram-disk and linking files and directories is the way to go.
Fancy Cache helps on those huge files that most of us don't have enough ram for, so there we just cache the most used blocks instead of entire files that can't fit in anyway.

[Manny]

Because to read whole file at once, you need to keep it all in memory, to read huge file with small amount of ram you need to read by blocks, like that:
var file = File.Open("c:\file.bin");
byte[] buffer;
do
{
buffer = file.Read(4096);
} while (!file.End);

Do you run games on non desktop systems? Do you think that desktop systems are unreal? It is far from the conditions of this threads, don't you think?

You can't dynamically spread ram over cache and ram drive right now, so you need to keep those file in ram all the time, or do a lot of click to load ran drive, unload ram drive, lunch cache.
So idea is to make systems that uses ram effectively and don't need user actions. So most of time we have in cache some program files, documents and browser cache, but when we launch the game, it preloads whole resources to ram, and after we close the game, new files start to replace game resources that is no longer needed.

[Violator]

Still no need to read each line, the app doesn't need to care about them anyway to move entire files to ram.
Each game will perform best with specific files cached, also, you need to check the users amount of ram, consider amount of free ram needed and which game files that will give the end user the greatest performance benefit.
In addition there will be need to constantly poll the process list, plus to know which files that actually can be manipulated while the main game executable is running, pre caching the needed files is a better solution, which means that the application itself also will have to work as the launcher.
Next there will be a need for memory monitoring to avoid to much disk I/O caused by disk swapping and even worse a complete system crash.
As you see, one size will not fit all.
Which supports my stand, ramdisk and mklink is the way to go, unless you got enough time to make an app and test it trough with every game on the planet.

[Manny]

I'm sorry but you said nonsense

First of all you need to know what files to map to ramdrive same as proceed by app.
You need to read not line by line, but block by block. If you don't understand what i mean, then just make some experiments with reading huge file, and you will understand everything.
App can be very simple, just give it a file name, and it will read it. Nothing to test, nothing to be afraid of.
Cache is limited by size, and can't cause system crash in any scenario, same as game crash. Just give to cache same amount of ram that you were going to give to RAM-DRIVE.
To create launcher you will need to create bat file, where you read files that want to preload one by one, then execute app.
To write that app you need 15-20 minutes i think, especially because main body of the app I have already posted.

In my opinion you just do not understand the idea, and look for the fictitious problem where they are not and can not be.
In any case, if you decide to continue to use your mind, it's your business. I don't care much

[dustyny]

To write that app you need 15-20 minutes i think, especially because main body of the app I have already posted.

Looks like you've spent more then 15-20 mins discussing this issue, so maybe you can code something up for the rest of us to test?

[Violator]

I'm sorry but you said nonsense

First of all you need to know what files to map to ramdrive same as proceed by app.
You need to read not line by line, but block by block. If you don't understand what i mean, then just make some experiments with reading huge file, and you will understand everything.
App can be very simple, just give it a file name, and it will read it. Nothing to test, nothing to be afraid of.
Cache is limited by size, and can't cause system crash in any scenario, same as game crash. Just give to cache same amount of ram that you were going to give to RAM-DRIVE.
To create launcher you will need to create bat file, where you read files that want to preload one by one, then execute app.
To write that app you need 15-20 minutes i think, especially because main body of the app I have already posted.

In my opinion you just do not understand the idea, and look for the fictitious problem where they are not and can not be.
In any case, if you decide to continue to use your mind, it's your business. I don't care much

Did I say anything against blocks? Just pushed a bit more to the C# / C++ difference. :p
If you cache 6GB of files on a 4GB system the user will run into performance issues, that shouldn't be hard to get.
So you still need to test (if / else as example) how much memory the user has available an estimate the maximum that should be allowed for caching.
And you need to go trough games to see which files benefit the best from caching, since you can't cache it all either, there is not enough memory on majority of gamers computers to do that.
I don't understand how that can be so hard to get?
Also

So idea is to make systems that uses ram effectively and don't need user actions.

Which never should include bat files or anything else that a user needs to run or set besides the program, correct?
Please do me a favor, go to the SWTOR and LOTRO forums and check out the posts regarding imdisk, mklink and creating a batch file that handles it, maybe then you understand what I'm trying to tell you.

[Manny]

Reading line by line pretty much the same as reading block per block.
I can't understand how you can't understand that you need to know all these things for both solutions. So your arguments can't be used i decesision between FancyCache and RamDrive.
And, no, launch bat file is not a problem for me, and managing ram drives looks like a problem. If you are afraid of bat files - it is your personal problem. Also i don't see any reason to spread such solutions, like giving it friends or posting on forums. So it is up to you. We are discussing technical side of the question.
If you want to show me something then give me a direct link, because i don't even know what doest it means SWTOR or LOTRO. And i'm not really intrested in games at all. But i'm intrested in caching and programming

[Manny]

To write that app you need 15-20 minutes i think, especially because main body of the app I have already posted.

Looks like you've spent more then 15-20 mins discussing this issue, so maybe you can code something up for the rest of us to test?

I will try to. May be next week, this one is crazy.