ETW trace via PerfView and possibly a dump. It could be anything, but this sort of thing over time is often due to a growing Gen2/LOH or thread pool starvation. Start a trace with PerfView with the default settings + Thread Time turned on and the buffer increase to 5000 MB and record for 1 min.

From there look at the GC Stats view to see if its GC time. From the dump open up Parallel Stacks to see how many threads are running and how many of them are blocked.

If you share these two things to me offline, I can take a quick look for you.

From: Craig vN via ozdotnet <ozdotnet@ozdotnet.com>
Sent: Saturday, December 3, 2022 9:51 AM
To: ozDotNet <ozdotnet@ozdotnet.com>
Cc: Craig vN <craigvn@gmail.com>
Subject: Random server slowdowns

We have a problem that is plaguing us and hoping someone can give some guidance.

Our website (ASP.NET MVC, Azure VM) fairly regularly (daily) experiences some random slowdowns. Here is a typical instance from our APM software.

The problem is it is proving very hard to find the source of the slowdowns, I suspect some kind of lock is happening, but when delving into the trace it doesn't really show anything other than everything slowing down, until I restart the server, which is becoming annoying at 5am.

The only thing I can delve from the APM tool is it happening in the MVC layer somewhere.

Does anyone know of any tools or instrumentation that can help diagnose these kind of problems. It is random, sometimes doesn't happen for days, then three times in one day.

Thanks

Craig