How can I debug OutOfMemoryException without stacktrace?

Question

When I run my .NET Core application on linux server I get OutOfMemoryException. I totally don't have any idea how can I debug that because stack trace is not provided. The only output is: "Unhandled Exception: OutOfMemoryException."

At the moment this exception is thrown there is enough space left. 200 GB of RAM is used and 300GB is free so I don't think there should be any problem with memory.

The moment when it happens is that I try to serialize an object into protobuf. The final file should be about 1GB. I have never had this issue before and models serialized pretty well.

I would be very thankful for any insight. I really need to make it work :( I already tried calling this right before serializing because I thought the problem was with fragmentation. After 10 hours of computation I realized it's not the case.

_logger.LogDebug("LOH compaction start");
GCSettings.LargeObjectHeapCompactionMode = 
GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect();
_logger.LogDebug("LOH compaction end");

The code I am executing when the exception is thrown is this:

        public static void Serialize<T>(this ProtobufSerializer protobufSerializer, string filePath, T serializationObject)
        {
            using (var fileStream = new FileStream(filePath, FileMode.Create))
            {
                protobufSerializer.Serialize(fileStream, serializationObject);
            }
        }

Scary numbers. It is never caused by not having enough RAM on a demand-page virtual memory operating system. On Linux you'd first make sure that the system swap space is large enough. — Hans Passant
– Hans Passant, Commented Oct 23, 2019 at 7:30
Can you edit your post to include the code that is executing when the exception occurs? (BTW what on Earth are you serializing that is so big?) — John Wu
– John Wu, Commented Oct 23, 2019 at 7:54
@JohnWu I edited my post. Those are models for forecasting sales into the feature. There is an information about every store and product in the company and which model we should use for such combination for best forecast possible — liamlim
– liamlim, Commented Oct 23, 2019 at 8:10
@HansPassant I never thought about swap size. It's 30GB. Is it ok for 500GB RAM? — liamlim
– liamlim, Commented Oct 23, 2019 at 8:13
That's so low to be pointless. Talk to whomever configured the machine, they might have well picked such a low number thinking that half a terabyte of RAM is always enough. Until it isn't. — Hans Passant
– Hans Passant, Commented Oct 23, 2019 at 8:18

Arif Eqbal · Accepted Answer · 2019-10-23 07:14:04Z

2

You need a very systematic approach to identify and resolve OOMs.

Do not assume that if you have lot of RAM free it is all for your application.
OOMs are very notorious to disguise themselves, you might see the Stack Trace pointing somewhere else and the problem actually would be elsewhere, in the sense, some other code is leaking/consuming large memory and has put the memory usage on the verge and a legit code somewhere else trying to secure some limited memory would push it to OOM.
Use some profiler and identify the area where the memory is increasing/leaking. (Looks like you have identified this as the code to serialize your large data.)
Then work on those areas and try to minimize the memory usage.

edited Oct 23, 2019 at 7:14

answered Oct 23, 2019 at 7:04

Arif Eqbal

3,1681 gold badge20 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

liamlim Over a year ago

I use Google.Protobuf and it looks like it can't serialize these data although it serialized much larger data in past. Is it possible to learn if Serialize method of that protobuf allocates internally?

Morampudi Dhamodhar Raju · Accepted Answer · 2019-10-23 06:58:03Z

0

OutOfMemory exception will be thrown only if you system bit size is not upto the level of size of object you use.

An OutOfMemoryException exception has two major causes:-

1.You are attempting to expand a StringBuilder object beyond the length defined by its StringBuilder.MaxCapacity property.

2.The common language runtime cannot allocate enough contiguous memory to successfully perform an operation

answered Oct 23, 2019 at 6:58

Morampudi Dhamodhar Raju

13 bronze badges

1 Comment

liamlim Over a year ago

Neither (1) nor (2) is the case. I don't use StringBuilder and there is 300GB of RAM free. Even peek of RAM usage is much higher than 200GB at the moment OutOfMemoryException was thrown.

Collectives™ on Stack Overflow

How can I debug OutOfMemoryException without stacktrace?

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related