guys what would be the best configuration to optimise the read and write of 1000's of 50kb html files?
i have a programe that updates them as a batch. I have 20 gigs spare ram capacity and a spare 64 gig ssd, although the files already reside on an ssd.
optimise for 1000's of read/write to 50kb files
-
ohnomatron
- Level 1
- Posts: 4
- Joined: Mon Feb 17, 2014 11:26 am
Re: optimise for 1000's of read/write to 50kb files
None Cache-configuration would guarantee max speed.
What you need is a >50 Megabyte RAM-Drive (1000*50Kb) fixed size files configured to DIO, NTFS, no compression, no searchindex
Make that one automatically load on boot and save on shutdown and symlink your folder to the RAMDisk.
If the data is critical you might consider saving (backing up) that Image in intervalls additionally.
RAM-Disk is perfect for such relative small but specific tasks!
Doing so, the files location realy moves to RAM tricking your filesystem to just look there.
That way its' availability is guaranteed (load at boot).
That way its' acceleration is guaranteed.
That way its' integrity is guaranteed (save on shutdown, save in intervalls).
What you want more?
Please understand that:
CACHE is meant to speed up a "DISK-Partition" generally... not a specific data.
RAMDISK is meant to speed up specific "Folders(or Applications)".
Simple as that. It is important to realize that.
The way you configure a CACHE pretty doesn't matter in fact ... because of the limited speed-up effect Cache can have on your app AT ALL.
If you need specific folders accelerated and you can foresee their overall size...always considere RAMDisk in favor, it's the non-plus ultra.
Note: The optimal settings of PrimoCache are totally NOT related to the tasks you want to do; basically "optimal" cache-configuration depends on hardware-capabilities. The only thing that is a relevant decision regarded to personal preference or what kind of "environment" you are working in is if R or W or R/W. Your cache size optimal settings depends on how much ram you have... higher is better for any task.. just no correlation. Also your cache block-size depends on available CPU-capacity and memory-size too due to increasing overhead... for IO-throughput smaller block size is better for any task, so again no correlation to "task".
What you need is a >50 Megabyte RAM-Drive (1000*50Kb) fixed size files configured to DIO, NTFS, no compression, no searchindex
Make that one automatically load on boot and save on shutdown and symlink your folder to the RAMDisk.
If the data is critical you might consider saving (backing up) that Image in intervalls additionally.
RAM-Disk is perfect for such relative small but specific tasks!
Doing so, the files location realy moves to RAM tricking your filesystem to just look there.
That way its' availability is guaranteed (load at boot).
That way its' acceleration is guaranteed.
That way its' integrity is guaranteed (save on shutdown, save in intervalls).
What you want more?
Please understand that:
CACHE is meant to speed up a "DISK-Partition" generally... not a specific data.
RAMDISK is meant to speed up specific "Folders(or Applications)".
Simple as that. It is important to realize that.
The way you configure a CACHE pretty doesn't matter in fact ... because of the limited speed-up effect Cache can have on your app AT ALL.
If you need specific folders accelerated and you can foresee their overall size...always considere RAMDisk in favor, it's the non-plus ultra.
Note: The optimal settings of PrimoCache are totally NOT related to the tasks you want to do; basically "optimal" cache-configuration depends on hardware-capabilities. The only thing that is a relevant decision regarded to personal preference or what kind of "environment" you are working in is if R or W or R/W. Your cache size optimal settings depends on how much ram you have... higher is better for any task.. just no correlation. Also your cache block-size depends on available CPU-capacity and memory-size too due to increasing overhead... for IO-throughput smaller block size is better for any task, so again no correlation to "task".
Re: optimise for 1000's of read/write to 50kb files
But aside from using RAMDIsk for that task, here is my recommendation for cache-configuration:
Cache Task for C:\ (SSD): R/W
L1-Size: 12 GiB (or higher maybe 18GiB if you do not need another seperate cache-task for HDDs)
Block-Size: 16 Kb block-size (or lower depending how much additioonal CPU you would like to waste for little IO improvement-impact).
No L2 because this one is already a SSD.
If you have additional HDD for media then another Cache-Task comes into play:
Cache Task for D:\, E:\, etc. (HDD): R/W
L1-Size: 6 GiB
Block-Size: 64kb or 128kb (varies on chipset, might try different also)
L2-Size: 64 GiB (perfect use-case for the spare SSD)
Oh wait we only consume 18GB...so 2 left.... hmmmmm, just use more RAMdisk ... these work perfectly seperated side-a-side.
Whatever you put inside a RAM-disk and symlink the folder in NTFS to the RAM-Drive...all of it excluded from CACHE, not interfering with CACHE and is permanently accelerated.
Also If a app uses 2 gib ram from content...and you put these ressources to ram-disk, then it uses no more ram for content itself, only the EXE size
. ... if the exe is also on the ram-disk... system ram usage would show 0 bytes for the process...... try that with cache on block-level , impossible 
Better think of "Cache" beeing a general-on-demand "RAM-Buffer" and "RAMDisk" beeing a specific permanent "RAM-Storage".
If you realise that, everthing gets clear.
I personally promote renaming those
Most people are not aware of the power of "mklink /J" for ram-disks
Hope my words helped you!
Cache Task for C:\ (SSD): R/W
L1-Size: 12 GiB (or higher maybe 18GiB if you do not need another seperate cache-task for HDDs)
Block-Size: 16 Kb block-size (or lower depending how much additioonal CPU you would like to waste for little IO improvement-impact).
No L2 because this one is already a SSD.
If you have additional HDD for media then another Cache-Task comes into play:
Cache Task for D:\, E:\, etc. (HDD): R/W
L1-Size: 6 GiB
Block-Size: 64kb or 128kb (varies on chipset, might try different also)
L2-Size: 64 GiB (perfect use-case for the spare SSD)
Oh wait we only consume 18GB...so 2 left.... hmmmmm, just use more RAMdisk ... these work perfectly seperated side-a-side.
Whatever you put inside a RAM-disk and symlink the folder in NTFS to the RAM-Drive...all of it excluded from CACHE, not interfering with CACHE and is permanently accelerated.
Also If a app uses 2 gib ram from content...and you put these ressources to ram-disk, then it uses no more ram for content itself, only the EXE size
. ... if the exe is also on the ram-disk... system ram usage would show 0 bytes for the process...... try that with cache on block-level , impossible Better think of "Cache" beeing a general-on-demand "RAM-Buffer" and "RAMDisk" beeing a specific permanent "RAM-Storage".
If you realise that, everthing gets clear.
I personally promote renaming those
Most people are not aware of the power of "mklink /J" for ram-disks
Hope my words helped you!
-
ohnomatron
- Level 1
- Posts: 4
- Joined: Mon Feb 17, 2014 11:26 am
Re: optimise for 1000's of read/write to 50kb files
thanks for taking the time to say all that. I did both in the end, ramdisk for the main files and cache for c drive.
-
Incriminated
- Level 6
- Posts: 65
- Joined: Fri May 31, 2013 3:03 pm
Re: optimise for 1000's of read/write to 50kb files
You can use the freeware "LSE - Link Shell Extension" to quickly generate symlinks/junctions/hardlinks from within the explorers right click-menu, so you not have to use dosbox, or external GUI-tools... also it gives you proper icon-overlays so you can detect them more easily.
http://schinagl.priv.at/nt/hardlinkshel ... llext.html
http://schinagl.priv.at/nt/hardlinkshel ... llext.html