Romex Software Forum

Hi there,

what I'd suggest to do is to do here is run a perfmon.msc and have a look at physical disk performance. Interesting figures are: Split I/O, Bytes / read, Bytes / s.
Then adjust your RAID array configuration appropriately. You should in the first place set the stripe size to match your overall throughput/s. If you require e.g. 30MB/s then you should find out either I/O per second or Bytes / read to be able to maximize the amount of drives involved in that operation by adjusting the stripe size. If your SW reads data in 64kB blocks you should set stripe to 64kB so that 64kB are evenly distributed over your RAID drives. Consult your controller documentation about what your stripe size actually means (I have come across some controllers that try to confuse you with segment size, etc.)

Assuming you have RAID6 with 8 Drives without hot-spare (leaving us with 6 actually reading drives) then setting a stripe size to 512kB will result in cca. 85kb segment size (512/6) for a single disk. Operating system partition should be formatted with the same settings as stripe size for block size (NTFS or exFAT) so one read involves all drives at once and does not produce thrashing (reading 64kB on the RAID level requiring only 16kB on filesystem level) or split I/O (reading 64kB in SW requires filesystem to perform 2 read operations of 32kB blocks and/or requiring RAID to read 4 16kB stripes).

Check your product guide on what exactly A/V streaming setting does to your cache settings. I would generally set controller cache to 80/20 for read, maybe even more. Also a cache pre-read should be turned on (if such setting exist).
If you have your disks connected to local FC controller have a look at partition aligning. Improper aligning results in split I/O (filesystem reads continuous 64kB block but underlying RAID has to perform 2 reads as FS block does not start at RAID stripe size start). Usually this is a matter of fiddling and benchmarking as RAID controllers rarely provide real configuration to the OS and they emulate 512b/4k sector behavior for compatibility reasons.

If you're really hunting down the best performance you should consider RAID 10. If properly implemented within the controller you are able to read from all drives (even the mirrored ones) so during reads you have 100% performance (all 8 disks reading in parallel) and during writes 50% which still can be mitigated by write cache settings on the controller.

You can change your windows file caching settings by "fsutil behavior set MemoryUsage". Consult Microsoft documentation on your desired setting value.

Make sure You're not using SATA or NL-SAS drives as these produce very low IOps. Use enterprise SAS 10k or 15k drives in your RAID.

What I assume is happening:
1) Your controller is set to cache writes heavily. It is impossible to achieve higher write than read performance on RAID5/6 as both impose write penalty (by design).
2) Your SW uses parallel reads overloading your controller and disks. What you should do is to use e.g. sqlio.exe, CrystalDiskMark or similar and tune your storage for parallel performance (QD32 or 16, it depends on your SW design).

Best Regards,
Slavius

Hi Slavius,

thanks for the comments, very helpful summary.

I still would love to see someone from Romexsoftware explaining how to calculate the size of RAM required to index an SSD cache.

Here is the point:

With Windows Server 2012 I can create a so called storage space, consisting e.g. of a HDD array and an SSD array. The storage space will use both arrays as one drive and will move the most often or recently accessed data to the SSDs, while longer time not touched data will be on the HDD. However, its really using the SSD and there is no "backup" on the HDD. If anything fails, the whole thing is lost. But you can put in as much SSDs as you like - there is no real limitation.

Performance wise I might get pretty much the same from an HDD array using an SSD array as PrimoCache. However, I am confronted with the problem that the cache needs to be indexed and therefor requires a lot of RAM. So there is a limitation in how many SSDs I can actually use as cache. And its hard to tell how much SSDs space that actually is.

I have 8 * 1 TByte SAMSUNG 850 EVOs which I want to use as SSD array. Maybe we even double that. Using storage spaces it sounds pretty straight to use this, but I remain with the increased failure risk of a multi-controller multi array in worsed case. Using an SSD cache instead would feel more flexible, as the underlying HDD array can be treated "as always". I can defragment it and be sure where on disk the data comes to stay. On a storage space I have no idea, as its up to Windows own policies when and where to move data. No idea if that will be defragmented or not.

Also, using the SSD array as cache is very flexible in theory, as I can disable cache if needed and have an 8 or 16 TByte super fast array "at hand" if needed for some reason. When that reason is over, I can reactivate the cache and my other RAID will get faster again.

However, given all former thoughts I doubt I can build a reasonable computer that can index an 8 or 16 TByte SSD cache and reasonable block sizes.

Any idea how to calculate the space needed for the index?

@Axel Mertes,

The major factor is the block size. Bigger the block size, smaller overhead. See FAQ.
Also currently supported maximum L2 cache size is about 2TB (2044GB). And we will overcome this limit in future.

Thanks.

Hi Support!

Thanks for chiming in into this discussion.

I agree with the block size being important. When I am able to re-format the drive to be cached, I'll choose the biggest possible block size. Thats apparently 64 KBytes per block as of now.

You say the maxium L2 cache supported is 2 TByte. So how much RAM would I need to index a cache of 2 TByte size and fully use it?
Can you please describe/explain how to calculate this?
Simply writing the formula would help.
That would help A LOT!

So we can plan how much SSD cache we can/need to add to have the best compromise.

You say you'll overcome this limit in future. What do you play to change and how much cache will be possible then?
Any idea when this will be possible?

Is it possible to have several 2 TB caches side by side on the very same computer, didicated to different HDD storage systems?
Like 2 TByte SSD cache for drive A: and another 2 TByte SSD cache for drive B: - on the very same computer?
What does that mean for RAM requirements?

Thanks!

Well, memory overhead for the cache size is proportional to the cache size. Given 1TB cache size, here is the table of memory needed. Note: the above overhead doesn't count the memory needed for the target disk size.

In future versions, we'll try to reduce memory overhead using other designs. But I'm sorry that so far I don't know when the improvement will be done.

Hi Support,

many thanks for that overview - very very helpful!

You mention 512 KByte block size. Is there an option in Windows to format drives using that block size?
If I remember correctly, there is only the option to format with 64 KByte maximum.
If there is trick I'd like to know it.

Best regards
Axel

Well, this block size means the cache block size which can be set in the cache configuration. It does not refer to the file system's cluster size. You don't need to format the drive, just change the cache configuration.

exFAT supports cluster sizes up to 32MB

support wrote:Well, this block size means the cache block size which can be set in the cache configuration. It does not refer to the file system's cluster size. You don't need to format the drive, just change the cache configuration.

Does it really make sense to use a different block size for the cache than for the actual disk?

What does PrimoCache do when I have for instance a disk array with 4 KByte NTFS block size and want to cache that using an SSD array with say 4 TByte and a cache block size of 512 KByte?

Will one cache block contain 512/4=128 actual NTFS blocks then?

I doubt that, because you need to a lot of memory to index that - and that was the start of the discussion for me.
If PrimoCache just uses large cache blocks and does not fill them up, so using a 512 KByte cache block to store a single 4 KByte disk block, I'd rather choose to reformat the disk array with the same large block size that the cache is using.

So is there a recommended practice to get the best effect out of the cache, which means to fill the cache as full and big as possible?

Thanks
Axel

rutra80 wrote:exFAT supports cluster sizes up to 32MB

Good to know. I am just not sure yet how safe exFAT is compared to NTSF or the newer ReFS. With huge amounts of data that's important for me.

Thanks,
Axel