Block size vs sector size
Block size vs sector size
Is there any relation between the block size in PrimoCache and sector size on my HDD/SSD?
Re: Block size vs sector size
The block size you give Primocache to use needs to be a multiple of the sector size on the disk. For optimal performance, setting it equal to sector size is best, but it has the highest overhead in terms of memory use. When caching the boot drive with Primocache, I typically use a 4k sector size and a 16k block size to keep overhead down.
On data drives, I format with either 32k or 64k sector size, and set Primocache to use the same. You get great read performance at a low overhead rate.
On data drives, I format with either 32k or 64k sector size, and set Primocache to use the same. You get great read performance at a low overhead rate.
Re: Block size vs sector size
Do you have two L2 disks with matching sector sizes for your system and data disk?
Why not set all disks to same sector size? Say 32k
What are the disadvantages and advantages?
Advantages:
It makes things easier for PrimoCache. Less overhead. Less blocks to manage like also means less CPU usage.
Disadvantage:
Larger sectors tend to waste more space because they often aren't filled completely (as I understand it)
Personal thought:
Is there any speed benefit in having larger or smaller benefits? While reading a 32k sector takes 8x longer than a 4k sector in theory, I think in real life the numbers may be quite different.
The ideal example a 32k file. Takes 1 read on a 32k sector and 8 reads of 4k sectors. No big deal if the sectors are all nicely grouped together. But if the disk is fragmented extra seeks must be done. My theory is that bigger sectors means less fragmentation/seeks.
Of course the above considerations are (mainly) for HDDs
I never tested this but maybe others have. I can imagine data-centers are highly interested in such stuff, so they did test it.
a] What's the average percentage of drive space wasted going from 4k to 32k?
b] What's the average speed benefit of going from 4k to 32k?
c] Other pros and cons of different sector sizes.
Why not set all disks to same sector size? Say 32k
What are the disadvantages and advantages?
Advantages:
It makes things easier for PrimoCache. Less overhead. Less blocks to manage like also means less CPU usage.
Disadvantage:
Larger sectors tend to waste more space because they often aren't filled completely (as I understand it)
Personal thought:
Is there any speed benefit in having larger or smaller benefits? While reading a 32k sector takes 8x longer than a 4k sector in theory, I think in real life the numbers may be quite different.
The ideal example a 32k file. Takes 1 read on a 32k sector and 8 reads of 4k sectors. No big deal if the sectors are all nicely grouped together. But if the disk is fragmented extra seeks must be done. My theory is that bigger sectors means less fragmentation/seeks.
Of course the above considerations are (mainly) for HDDs
I never tested this but maybe others have. I can imagine data-centers are highly interested in such stuff, so they did test it.
a] What's the average percentage of drive space wasted going from 4k to 32k?
b] What's the average speed benefit of going from 4k to 32k?
c] Other pros and cons of different sector sizes.
Re: Block size vs sector size
Drives (especially spinners) typically read/write that data faster with smaller sectors. But when you have a lot to read (possibly even fragmented on the drive due to smaller chunks) the larger sectors are better, though sometimes more space inefficient. When Primocache is told to "go fetch this file" and it isn't in the cache, Primo will try to read the sectors where the file resides (and the sectors next to it, so it fills up it's entire block). If you have that set to 64k blocks and the file is only 1kb, the drive STILL has to deliver a 64kb section of the hard drive - that's how Primocache is caching the drive after all, in blocks.
With L2 caches on SSDs, the block size is better left smaller, IF you can afford the overhead. If not, bump block size up slightly until you're happy with the overhead. It's different than the L1 overhead - check the UI to see. https://www.romexsoftware.com/en-us/pri ... cache.html But again - matching the block size to the sector size is what Romex claims gives the best overall performance, especially on SSDs with high IOPS rates. Never set block size smaller than sector size.
I use larger sectors on spinning data drives, because I store a lot of larger files (movies, music, etc). That makes reading them in much faster. But on the SSD boot drive where disk sectors are still 4k, my L1 is set to only 16k blocks in Primocache. It still has to read in 4 for any file no matter the size, but the overhead is slightly less, and I still get somewhat of a read boost when file size goes over 16k due to most of the files being larger than 4k.
Fragmentation on your spinner disks will make all the difference if you use smaller sectors and larger blocks. The decrease in performance will be far less, and the gains far higher. Keep spinner drives optimized so Primocache can do it's work better.
I think to get the answers to your sector size questions, you might have to do a little Googling. It's a little outside the scope of the forums here, and I don't think those specifics have been answered here before. It can be dangerous to try and change the boot drive from 4k to something larger, and on data drives speed gets trashed when you have media on there and only 4k sectors. The smallest media file I can think of is a short MP3, which usually still weigh in around half a megabyte.
The best setup I've found (and which I still use) is a SSD boot drive that holds nothing but the OS and installed software, at 4k sectors. All of my data (including my user folders which I move to data drives) sits on either 32k or 64k sector size volumes on separate disks, and Primocache is set to use 64k blocks on those.
With L2 caches on SSDs, the block size is better left smaller, IF you can afford the overhead. If not, bump block size up slightly until you're happy with the overhead. It's different than the L1 overhead - check the UI to see. https://www.romexsoftware.com/en-us/pri ... cache.html But again - matching the block size to the sector size is what Romex claims gives the best overall performance, especially on SSDs with high IOPS rates. Never set block size smaller than sector size.
I use larger sectors on spinning data drives, because I store a lot of larger files (movies, music, etc). That makes reading them in much faster. But on the SSD boot drive where disk sectors are still 4k, my L1 is set to only 16k blocks in Primocache. It still has to read in 4 for any file no matter the size, but the overhead is slightly less, and I still get somewhat of a read boost when file size goes over 16k due to most of the files being larger than 4k.
Fragmentation on your spinner disks will make all the difference if you use smaller sectors and larger blocks. The decrease in performance will be far less, and the gains far higher. Keep spinner drives optimized so Primocache can do it's work better.
I think to get the answers to your sector size questions, you might have to do a little Googling. It's a little outside the scope of the forums here, and I don't think those specifics have been answered here before. It can be dangerous to try and change the boot drive from 4k to something larger, and on data drives speed gets trashed when you have media on there and only 4k sectors. The smallest media file I can think of is a short MP3, which usually still weigh in around half a megabyte.
The best setup I've found (and which I still use) is a SSD boot drive that holds nothing but the OS and installed software, at 4k sectors. All of my data (including my user folders which I move to data drives) sits on either 32k or 64k sector size volumes on separate disks, and Primocache is set to use 64k blocks on those.
Re: Block size vs sector size
Getting clearer now. Yeah, it's possibly beyond the scope of this forum. But OTOH, PrimoCache is all about getting more performance and the end result is best if both the disk and PrimoCache are configured in the best possible way.
I understand that for copying a 1GB file a larger sector size is beneficial.
But what for a (read only) database of 1GB? That file has record size access which usually is small.
Or when creating a large file. Let's say converting a movie into a 1GB file but by constantly adding a few bytes?
BTW do you happen to know a program that gives file size statistics of a drive?
Just some statistics that show me the percentage of <4k, <16K, etc files.
I'm planning to build a new PC and might buy/formats SSDs and HHDs with the optimal sector sizes in mind.
I understand that for copying a 1GB file a larger sector size is beneficial.
But what for a (read only) database of 1GB? That file has record size access which usually is small.
Or when creating a large file. Let's say converting a movie into a 1GB file but by constantly adding a few bytes?
BTW do you happen to know a program that gives file size statistics of a drive?
Just some statistics that show me the percentage of <4k, <16K, etc files.
I'm planning to build a new PC and might buy/formats SSDs and HHDs with the optimal sector sizes in mind.
Re: Block size vs sector size
If your DB file has smaller record/field accesses, then yes - a smaller sector and block size would be better for performance (but higher on overhead in Primocache).
Normally software pre-allocates new large files when creating them - it all depends on the software though. In the case of pre-allocation, larger sector/block size would be better.
The thing about average statistics on files, is that many of them never get used past the Windows install (proved by the size of the Windows prefetch). And on a boot volume that's a SSD, having a ton of small sectors is actually advantageous considering the massive IOPS they have. There's no seek time involved with them. I Googled file size statistics, and there are a few utilities out there, along with some Powershell scripts that can do it in vonjunction with something like Excel. Honestly though - where the boot (if a SSD) is concerned, I wouldn't truly waste too much time with it.
Normally software pre-allocates new large files when creating them - it all depends on the software though. In the case of pre-allocation, larger sector/block size would be better.
The thing about average statistics on files, is that many of them never get used past the Windows install (proved by the size of the Windows prefetch). And on a boot volume that's a SSD, having a ton of small sectors is actually advantageous considering the massive IOPS they have. There's no seek time involved with them. I Googled file size statistics, and there are a few utilities out there, along with some Powershell scripts that can do it in vonjunction with something like Excel. Honestly though - where the boot (if a SSD) is concerned, I wouldn't truly waste too much time with it.