OutOfMemoryException when loading large string from Xml file

Question

I'm trying to simply load a very large string from an Xml file and save it to a temporary file inside the IXmlSerializable#ReadXml method, the code being used is below.

The problem is I'm getting an OutOfMemoryException on the reader.ReadStartElement("data"); line. It appears that the XmlReader is attempting to preload the value string and as it is ~500Mb in this case it is failing to allocate a StringBuilder for it.

Is there some better way of copying this string into a file, or some way to bypass the preloading of the XmlReader?

public void ReadXml(XmlReader reader)
{
    // Read other elements

    reader.ReadStartElement("data");

    this.dataFile = Path.GetTempFileName();
    FileStream tempFile = File.Create(this.dataFile);

    char[] buffer = new char[CHUNK_SIZE];
    int count;

    using (StreamWriter writer = new StreamWriter(tempFile))
    {
        while ((count = reader.ReadValueChunk(buffer, 0, CHUNK_SIZE)) != 0)
        {
            writer.Write(buffer, 0, count);
        }
    }
    reader.ReadEndElement();
}

Ouch; that is... tricky. I assume it is far to late to suggest that maybe xml isn't the right format for a 500Mb enclosure? — Marc Gravell
– Marc Gravell, Commented Dec 9, 2010 at 21:34
I would not use xml and were ever you are parsing it give it a filestream so it is not reading the entire string at once. Because as a unicode string you are already doubling the size (if it is stored as ascii). — Will
– Will, Commented Dec 9, 2010 at 21:58
Normally it is nowhere near 500Mb, this is just part of the stress testing. It can be 500Mb (or even bigger) though so it needs to support this. Nowhere is this supposed to be getting stored as a string, it was originally but once I realised how big it could get I changed to this chunking and streaming for both reading and writing. — Nemo157
– Nemo157, Commented Dec 9, 2010 at 22:23

Nemo157 · Accepted Answer · 2010-12-09 22:36:01Z

Found a solution, the problem isn't in the IXmlSerializable#ReadXml method, it's actually in the method that calls XmlSerializer#Deserialize. Originally I had this:

private void OpenSavedData(StreamReader strmReader, string fileName)
{
    XmlSerializer serializer = new XmlSerializer(typeof(SavedData));
    SavedData savedData = serializer.Deserialize(strmReader) as SavedData;

    // Process data
}

By default Deserialize generates a XmlTextReader to pass to the ReadXml method. If I had actually made it past the ReadStartElement call I would have found that XmlTextReader doesn't support GetValueChunk.

Instead I need to instantiate an XmlReader myself using the XmlReader.Create method. This will create an implementation that doesn't preload the values and supports chunking.

private void OpenSavedData(StreamReader strmReader, string fileName)
{
    XmlSerializer serializer = new XmlSerializer(typeof(SavedData));

    XmlReaderSettings settings = new XmlReaderSettings();
    settings.CloseInput = true;
    settings.IgnoreWhitespace = true;

    SavedData savedData = null;

    using (XmlReader xmlReader = XmlReader.Create(strmReader, settings))
    {
        savedData = serializer.Deserialize(xmlReader) as SavedData;
    }

    // Process data
}

This then allows the ReadXml call to succeed.

Collectives™ on Stack Overflow

OutOfMemoryException when loading large string from Xml file

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related