Freezes with Dual CPU (2 Numa) Architecture Topic is solved

Found a bug? Report here
savagesmc
Level 2
Level 2
Posts: 5
Joined: Thu Nov 28, 2019 4:18 pm

Freezes with Dual CPU (2 Numa) Architecture

Post by savagesmc »

I'm VERY new to this forum. I purchased the PrimoCache about a month ago based on some research. It is definitely something I need in my setup. Thank you for making this product.

However, I believe there may be a stability issue in my particular configuration. I am running a dual processor xeon 2690 (sandy bridge) setup. I have a total of 16 cores / 32 threads with 8 cores per xeon. This setup also has a total of 128 GB of ram, a ssd (sata) boot drive, a 4 TB spinning disk (sata), and a 480 GB NVME drive (pcie).

I setup PrimoCache with 2 layers of caching, the first layer being 32 GB of RAM, the second layer being the 480 GB NVME drive. I set it up to cache both the SSD boot drive as well as the 4 TB spinning disk. I set it up to do write-through so as to be stable (no caching of writes).

I am experiencing random lockups where the screen 'freezes', and there's no response to any sort of input. These freezes occur for a second to a few seconds, but then will occur where the only recourse is to hard reset the entire system.

These freezes will happen from 10's of minutes to a few hours. Usually withing 3 to 4 hours, I can be guaranteed that the system will freeze and require a reset.

I have eliminated all other possibilities of lockups - i.e. bad power supply, bad ram, bad processor, bad video card etc... The only thing that gets rid of the freeze behavior is to remove the drive caching.

I am wondering if it's my 2 processor setup that may not be something that many other (or any other) users would have used before that may be causing the freezes.

I'm hoping there is some way we can debug or troubleshoot this, because I really like the performance benefits that this tool would give me. I am a professional software developer, so if there's some way I can help you to troubleshoot this issue, please let me know. I have the visual studio toolset, so could try to debug if you wanted me to help you with that.

Thanks,

Steve
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by Support »

Thank you very much for your feedback!

Could you remove L1 cache first and check if it works?
savagesmc
Level 2
Level 2
Posts: 5
Joined: Thu Nov 28, 2019 4:18 pm

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by savagesmc »

I removed the L1 RAM cache, and it has not frozen in 3 days.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by Support »

Thank you very much for your testing and feedback. It seems that the issue is related to NUMA. We have completed the code for improving NUMA support, however we haven't fully tested it and integrated into the release version.
If possible, you may try 8GB or 16GB L1 cache and see if it will cause freeze or not.
Thanks.
savagesmc
Level 2
Level 2
Posts: 5
Joined: Thu Nov 28, 2019 4:18 pm

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by savagesmc »

I will try those sizes and let you know what I find. Also, I am a software developer by trade, so I could help you integrate and test if you'd like. Thanks.
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by Support »

Thank you very much, savagesmc. I'm looking forward to your testing results.
savagesmc
Level 2
Level 2
Posts: 5
Joined: Thu Nov 28, 2019 4:18 pm

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by savagesmc »

I have had the L1 cache enabled using 8 Gigs of RAM along with the 447 GB NVME drive for several days now, and have had no lockups. It was running stable enough that I enabled write-caching, and my diskmark numbers seem to indicate the cache is operating. I may disable the write caching and try the 16 GB size next. I will let you know what I find. Thanks.
Attachments
2019-12-05_10-24-31.png
2019-12-05_10-24-31.png (33.69 KiB) Viewed 9314 times
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by Support »

Many thanks!
savagesmc
Level 2
Level 2
Posts: 5
Joined: Thu Nov 28, 2019 4:18 pm

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by savagesmc »

I'm sorry, but after a few days with no lockups, it started locking again. I backed out each change (in the cache settings) back to only an L2 cache. I even tried going down to only 2 GB of L1 cache at the last try, but still had crashing. It's been a few days again (since turning off the L1 cache completely again) with no crashes. It appears like there is a longer term low-probability real-time interaction going on that could still occur even with the L2 only cache - I don't know if I ran long enough now considering I had to go many days to see the first L1 crash, and then had a number of them over a few days. Here's a screen shot showing all the dates and times that my computer has frozen. I will let you know in a few weeks if L2 only operation has no crashing.
Attachments
2019-12-21_11-46-43.png
2019-12-21_11-46-43.png (52.48 KiB) Viewed 9197 times
User avatar
Support
Support Team
Support Team
Posts: 3623
Joined: Sun Dec 21, 2008 2:42 am

Re: Freezes with Dual CPU (2 Numa) Architecture

Post by Support »

:o Did you notice that system free memory was reducing before the problem happened? We thought the problem was caused because the memory connected to one node was consumed up. I'm looking forward to the testing result.
Post Reply