We have an application that heavily utilizes the MemoryCache class provided by .NET, and up until recently, everything has been working well on our servers, but now we've started to notice CPU spikes every so often during peak usage times.
After analyzing some of the process dumps with WinDbg, a common theme has emerged, that I think is related to our use of MemoryCache.
Basically, what I am seeing are many threads blocked by one thread using the TimerQueue object, which I believe is trying to expire an item from the MemoryCache object.
Currently, we cache certain levels of database results, for each user, with an absolute expiration of 10 seconds. The server we are on has 16 cores, which I would have thought would be enough to handle a decent amount of concurrency.
So what it seems like is happening is that if we have an increase in concurrent usage on the system (possibly trying to write to the cache), while the automated cache expiration thread is trying to delete an item from the cache, all those threads will be blocked until the delete from the cache is completed?
Does this explanation seem accurate, and if so, does anyone have recommendations to improve the performance?
So far, the only solutions I have are the following:
- Add some load balancing to help spread the usage out, and hopefully avoid this problem
- Increase the expiration threshold to maybe 30 seconds, to help counteract it
- Use a different caching mechanism entirely
