Need some feedback on my unusual use case of primocache

[SnowReborn]

I had been a long time user of primocache , from very beginning where it's riddled w bugs and BSOD, to now reliable and fast. always been one of my favorite software. However recently I have pulled the trigger on TrueNas Scale. I have removed all my local client computer's HDDs, and moved them all to my TrueNas Server.

My setup is current like this :
Server(1TB RAM) > TrueNas < Mellanox connectx4(SFP28 dac with nvme-OF RDMA) > client windows PC

I have several questions and concerns:

1. since I am running TrueNas with native read / write caching from openZFS, and client server are bonded via nvme-OF RDMA, which essentially provides direct memory access, does primocache can be leveraged in anyway under this circumstance? (i know i can get better 4k random and IOPS from primo)

2. if I were to enable primocache on my RDMA conencted Zvol network partitions(formatted into NTFS and looked like physical device to windows) are there any inhertant risks of data corruptions, if so, is it from primocache's read cache or defer write? or both?

3. are primocache able to work with RDMA? It seems like to me from testing that, if i turned on defer write on primocache(which puts write queues in client RAM), and flush it, it will directly flush to server's RAM all at once, and server will then start flushing to HDDs. is this correct observation?

4. What if the internet connection dropped out between my client and server while using primocache? are there risks of data corruption that way? if so is it from read caching or write caching? or both?

5. Lastly, this is unrelated to server. Let's say I turned on primocache L1 defer write (infinite) on a local nvme bootable OS drive(and i am fully aware of the risk, have UPS backup, full disk image to rollback if improper shutdown). Right now I have issues with one of my PCIE cards that often freezes and panic bios at shutdown phase or reboot(stupid mellanox nic). So let's say i have 10Gb unflushed defer write in primocache, and I click "restart my pc" on my client PC. The windows goes the normal blue screen (NOT BSOD) of regular shutdown phase, apps closes, Mouse turns off, keyboard turns off, monitor shows black screen, and also loses signal(normal behavior of restarting pc), and then just hangs there frozen, stuck at this very step, no POST from restarts, no mouse or keyboard lights, no bios screens. I assume at this stage primocache has already finished flushing the write cache and the kernel has already shutted down, but bios paniced, so theoretically my local drive is still safe from data corruption right? or my understanding is incorrect? if anything I would like to know, at exactly what stage, precisely does primocache try to flush the defer write into disk, and primocache shuts down from kernel?

Thanks for any feedbacks or suggestions!

[tverweij]

For 1:
The above scenario will only work when you are using iscsi (this will create local drives).
If you use mapped drives, Primocache can not use them (as they are network drives).

For 2 and 3:
Primocache uses the drivers of Windows to write to the disks.
So if it works without Primocache, it will work with Primocache.

For 4: Always - if a network connection to a disk drops, you loose data.

For 5: I don't know as I never used level 2 cache.

[SnowReborn]

For 1:
The above scenario will only work when you are using iscsi (this will create local drives).
If you use mapped drives, Primocache can not use them (as they are network drives).

For 2 and 3:
Primocache uses the drivers of Windows to write to the disks.
So if it works without Primocache, it will work with Primocache.

For 4: Always - if a network connection to a disk drops, you loose data.

For 5: I don't know as I never used level 2 cache.

Heya! Thanks for the reply! I think you have some misunderstandings that I could clarify alittle. Sorry about the ambiguity!

1. nvme-of RDMA is similar to iscsi, but more advanced form of it, windows still sees the remote network zvol target as a physical block device, that is handled by windows, formatted into NTFS.

2,3, RDMA often bypass some kernel level stuff to reduce latency and overhead, which is concerning for primocache driver.
4. not nesscarily, iscsi and RDMA has robust timeout mechanism that allow information to temporarily stored in RAM before re-trying connection and flush to target, and would temporarily halt some of the I/O, which somewhat acting like a buffer zone like primocache defer write.

5. not talking about L2, talking about L1 defer write with indefinite timeout timer applied to a bootable nvme OS partition.

[tverweij]

Ok, NVMe over Fiber using RDMA as protocol (as I understand it after googling it).
That's out of my league....

I am afraid that the only way to know is to test it ...

[SnowReborn]

Ok, NVMe over Fiber using RDMA as protocol (as I understand it after googling it).
That's out of my league....

I am afraid that the only way to know is to test it ...

np friend, appreciate chiming in