Consistent results? How to measure?

[wpcoe]

I just discovered FancyCache the other day, and it has been interesting (and fun!) learning about it both online (here and other forums) and via experimentation/experience. Probably because I'm a brand new member, I am "not authorized to define a signature," so here's a summary of my computer systems:

Desktop
i5-760, 2.8GHz (stock), CM Hyper212+ HSF, GigaByte P55A-UD3P
2 x 2GB Kingston ValueRAM + 2 x 2GB generic RAM = 8GB RAM
Kingston SNV425-S2/64GB SSD, 500GB + 1TB WD Green HDDs 7,200rpm
ATI HD4670, 1GB RAM, onboard sound only
Samsung 203B LCD (20" wide screen, usually in portrait orientation)
Lian-Li Lancool K56 , abee Athlete 600W PSU, Windows 7 Home Premium 64-bit

Notebook
Acer Travelmate TM1872, i3-380UM, 1.33Ghz, 2 x 2GB generic RAM = 4GB RAM
Seagate Momentus XT 320GB hybrid drive, Windows 7 Professional 64-bit

At first, I just experimented with caching my (very) low-end Kingston SSD and got nice numbers from CrystalDiskMark and Anvil benchmarks. However, from the start I noticed that repeating the same benchmark in succession would show varying results, sometimes higher, sometimes lower, sometimes consistent in only one or two items, etc, hence the title of this message thread: What are the variables that cause the reported results to vary so widely? I would shut down every non-essential background process, and have nothing running in the foreground except FancyCache, in an effort to santize the process, but to no avail.

I had already cached the Kingston SSD with the following settings:

Block size: 4KB
Cache Size (Level-1): 1536MB (1.5 GB)
Defer Write Enabled, Latency: 1 Sec*
Level-2 Cache: not enabled

*I'm nervous about deferring writes on a system drive, but found that with even just a token one second of latency that the benchmarks for write performance got a significant boost. Go figure.

Now, for the hard drive:

First I ran CrystalDiskMark once on a partition of the hard drive before caching it with FancyCache.

Then I rebooted and ran one iteration of CDM with the following settings:

Block size: 4KB
Cache Size (Level-1): 256MB
Defer Write Enabled, Latency: 10 Sec
Level-2 Cache Size: 768MB on my Kingston SSD

Then, I rebooted and ran five consecutive iterations of CDM as:

HDD_FancyCache_config.png (36.55 KiB) Viewed 7280 times

After each run with CDM, I would copy/past the screen shot into Photoshop, save it, and exit Photoshop. Then on to the next iteration with CDM with no change to the any settings in FancyCache.

Look at the range of results, especially in the 5 runs of CDM with identical FancyCache settings:

HDD_FancyCache.png (73.5 KiB) Viewed 7280 times

Oy vey. For example, follow the 512k writes: 118.8 to 1619 to 1371 to 1655 to 1307?

I realize this is beta software, so some rough edges can be expected, but is there anything I'm doing wrong, or not doing, that is contributing to such inconsistent results? My Kingston SSD has both TRIM and aggressive garbage collection -- is that affecting the benchmarking? When rebooting, I would let the computer pause at the BIOS screen for between 30 minutes and several hours, hoping to do the next benchmark test with a "refreshed" SSD.

Addtionally, I encountered instability when I tried to cache the HDD on my notebook with 512MB RAM. There is only 4GB RAM on that computer, so I don't want to use too much. I did try using 1GB RAM as a cache, but again the machine was unstable: after completing CDM, the entire system would slow to a crawl (not quite freeze) for over a minute. I would click on the Photoshop icon and nothing would happen. I would open Task Manager (ctrl + alt + esc) and a basic window would open, but with only a short list of operating processes. Then, boom, after a minute or so, my computer would resume working. It was enough to make me uninstall FancyCache entirely. I even "rolled back" the system via System Restore and re-installed FancyCache, but same results.

[wpcoe]

Oh, I just noticed something else. After finally posting that long rambling post above, I decided to check on RAM usage on my computer (8GB RAM installed.)

When I opened Task Manager, it showed 5.63 GB in use.

I then stopped caching the HDD where I had set a 512MB Level-1 cache. RAM usage dropped to 3.45 GB -- it freed up over 2GB RAM?!?

Then I additionally stopped caching the SSD where I had set a 1.5GB cache and RAM usage dropped to 1.86GB -- it freed up 1.59GB RAM, which is more reasonable.

Then I restarted the cache for the SSD and RAM usage went back to 3.45GB, but when I restarted the 512MB cache for the HDD, RAM usage went back to 4.88GB -- a jump of almost 1.5GB.

Why is RAM usage so large ( >1.5GB) when supposedly setting aside only .5GB for a HDD cache?

You can see that the RAM freed/used in connection with the .5GB cache for the HDD is about the same size as used for the 1.5GB cache for the SSD. What's up with that?

[Support]

Well, in your testing case, the data length is 1000MB, L1 cache is about 512MB which is less than 1000MB, so L2 cache also joined in the test. The results are mostly affected by the L2 cache which is on SSD.

And for your reference,
http://www.romexsoftware.com/en-us/fanc ... cases.html

Regarding the memory usage issue when L2 cache enabled, we have noticed this issue. please see
viewtopic.php?f=25&t=1080

Thank you.

[wpcoe]

Thanks for pointing out about CDM using a larger data length than my cache. Duh. I had almost forgotten that one can select the data size in CDM, so I ran a series of CDM benchmarks with a 500MB size with the 512MB L1 cache in FancyCache. The writes were all pretty consistent running CDM six times consecutively. (Unfortunately, I forgot to switch on deferred writes, and the results from CDM was pretty much the same as before I tried caching the drive in RAM.) The CDM *read* results were again all over the map:

HDD_4K_512MB,_no-defer_768MB.png (48.67 KiB) Viewed 7260 times

So, I set up FancyCache with a 1GB L1, 1GB L2 (on SSD), with a 10 second write deferral. Using those settings with a 500MB CDM data size, resulted in more consistent read AND write figures, and considerably above the figures for the uncached HDD:

HDD_4K_1024MB,_10sec_1024B.png (44.39 KiB) Viewed 7260 times

I'm still puzzled by something, and this may well be a CrystalDiskMark issue: When I run the default five iterations of each read and write test, within the five runs for a given statistic, e.g. 512K Read, or 4K Write, the five results jump all over the place. However, the fifth result, which is the one that CDM displays when it's finished, is consistent when running multiple consecutive runs.

e.g. For Sequential Read, the results in one run (3rd one above) of CDM were: 2338 Mbps, 1818 Mbps, 879 Mbps, 2469 Mbps, 2469 Mbps -- why such jumping around, using a 500MB data length in a 1024MB RAM cache?

All-in-all, now that I found a way to get consistent results, I feel more comfortable with FancyCache. Especially when those results for the HDD are so much better than compared to before I started caching.

[Support]

Well, thank you for sharing your test results again.

This testing with CDM just gives a rough concept. It cannot cover most real scenarios. We are still studying the possiblity to improve the performance.

[NickJ]

The CDM read results seem to be all over the place due to the caching algorithm you're using. Try again with LRU instead of LFU-R, and you should see more consistent results. The entire cache isn't available for your test, unless you tell it to replace "everything" older, which is what LRU does. LFU-R will only replace the older "unused" stuff.