Explain if static readonly ConcurrentDictionary<string, AsyncLocal<object>> State leaks memory?

Question

I have an app that seems to accumulate a lot of memory. One of the suspects is below, and I'm just trying to wrap my head around what it is actually doing. Or, more specifically, how is it cleaned up?

private static readonly ConcurrentDictionary<string, AsyncLocal<object>> State;

Problem context: The idea is to simulate what OperationContext in WCF would do - provide static access to information about the current call. I am doing this inside a Service Fabric remoting service.

Can someone help me understand the nature of this in terms of what happens to the AsyncLocal<object> once the async call ends? I see it hanging around in memory but can't tell if it is a memory leak, or ig the GC just hasn't reclaimed it yet.

I know the static dictionary stays around, but do the values also, or do I need to be manually clearing those before my current service invocation completes to ensure no memory leak here?

*Edit - Here is some more info as requested by Pavel.

Posting relevant code below, but the whole picture is here. Github where the general idea came from. They are trying to make headers work in ServiceFabric/.net core like they used to in old WCF. https://github.com/Expecho/ServiceFabric-Remoting-CustomHeaders

The RemotingContext object is here: https://github.com/Expecho/ServiceFabric-Remoting-CustomHeaders/blob/master/src/ServiceFabric.Remoting.CustomHeaders/RemotingContext.cs

It's use can be seen here (line 52, among others): https://github.com/Expecho/ServiceFabric-Remoting-CustomHeaders/blob/master/src/ServiceFabric.Remoting.CustomHeaders/ReliableServices/ExtendedServiceRemotingMessageDispatcher.cs

Here is a code snippet:

public override async Task<IServiceRemotingResponseMessage> HandleRequestResponseAsync(
    IServiceRemotingRequestContext requestContext,
    IServiceRemotingRequestMessage requestMessage)
{
    var header = requestMessage.GetHeader();

    RemotingContext.FromRemotingMessageHeader(header);

    //Some other code where the actual service is invoked (and where RemotingContext values can be references for the current call.

    return null;
}

I know the static dictionary stays around, but do the values also, or do I need to be manually no, they magically disapear ... seriously, you put something into dictionary so dictionary keeps referenece to this object, if there are object reference it cannot be GCed ... what do you not understand? ... so obviously you have to remove entry fom dictionary if longer needed or you should use WeakReference — Selvin
– Selvin, Commented Nov 20, 2020 at 11:19
Well thanks, but agreed, no need to be rude. It's the AsyncLocal bit that is the value in the dictionary. It is my understanding that when an async call chain happens, the value in the dictionary will be specific to that call. If I have concurrent async calls going (as happens in Service Fabric) each of them would get a unique State["SomeFoo"]. It looks like State will have the key for all threads but the value will be specific to the current execution context? So does that, then mean I need to delete the AsyncLocal before the Task completes otherwise it will get stuck in memory? — Will Comeaux
– Will Comeaux, Commented Nov 20, 2020 at 11:42
Please, share your code and how the State dictionary is used — Pavel Anikhouski
– Pavel Anikhouski, Commented Nov 20, 2020 at 11:42

Phil Rogers · Accepted Answer · 2020-11-20 12:03:41Z

1

The Garbage Collector is a strange beast, so it's worth getting to know about its behaviour. (Note, this is a simplistic view of the GC)

First of all, if just one reference exists to an object, that object is not considered garbage, so it will never be collected. As your Dictionary is static, it is always going to exist, so anything contained within it will always have a reference to it. If there are no other references to the contained object and you remove it from the Dictionary, it will become unreachable and therefore garbage and will be collected. Making sure there are no references to your objects is the way to ensure they will be collected. It's very easy to forget about some reference you made somewhere, which keeps the object alive.

Secondly, the Garbage Collector doesn't run continuously. It runs when memory resources are getting low and needs to release some memory. This is so that it doesn't hog the CPU to the detriment of the main applications. This means that objects can still be in memory for some time before the next Garbage Collection. This can make memory usage seem high at times, even when it isn't.

Thirdly, the Garbage Collector has "Generations". Generation 0 objects are the newest and most short-lived objects. Generation 0 objects are collected most often, but only when memory is needed.

Generation 1 contains short-lived objects that are on their way to becoming long-lived objects. They are collected less often that Generation 0 objects. In fact, Generation 1 collection only happens if a Generation 0 collection did not release enough memory.

Generation 2 objects are long-lived. These are typically things like static objects which exist for the lifetime of the application. Similarly, these are collected when the Generation 1 collection doesn't release enough memory.

Finally, there is the Large Object heap. Objects that consume a lot of memory take time to move around (part of the garbage collection process involves defragmenting the memory after collection has taken place), so they tend to remain uncollected unless collection didn't release enough memory. Some people refer to this as Generation 3, but they are actually collected in Generation 2 when necessary.

answered Nov 20, 2020 at 12:03

Phil Rogers

5734 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Will Comeaux Over a year ago

That's helpful but I'm not quite fully understanding the nature of my specific code. I get the static will stay in memory, but for each service invoked, my code will update State["SomeHeader"] with a new value. But because the dictionary value is an AsyncLocal, it's only updating for that specific task. My service will run "forever" and the dictionary lasts that long too. Does that mean each task, once it's complete, will leave a residue in the State dictionary? See the confusion?

Will Comeaux Over a year ago

Accepting yours as the answer Phil but I'm taking a slightly new approach. I am going to treat my headers as a Stack so as calls come in I plan to Push an item onto the current value and then Pop the item back out. Having the stack flow across the calls will have some value anyway and it makes it clear when I am no longer using the current frame. FWIW, I wrote a test to play with the ConcurrentDictionary of AsyncLocal and honestly, still can't tell if the value of the AsyncLocal itself is gone once the async call is complete. It's not there in the dictionary, but still is in memory...

Peter Bons Over a year ago

@WillComeaux let me know if it does reduce memory footprint or create a PR at the repo

Will Comeaux Over a year ago

@PeterBons will do. Still testing to see if there is really an issue. The more I look at it the more I am convinced there is, but at the same time... Found this tweet earlier but the author gave no real elaboration on if he verified it or not. Might reply to it to see if he dug any deeper. twitter.com/davidfowl/status/1033964051607379968?lang=en In the mean time, still trying to get that RemotingContext to behave the way I want, and not leak memory. :) I'll follow on on the github with what I am thinking there.

Collectives™ on Stack Overflow

Explain if static readonly ConcurrentDictionary<string, AsyncLocal<object>> State leaks memory?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related