Java parse large XML document

Question

I'm trying to parse, and replace values in a large xml file, ~45MB each. The Way I do this is:

private void replaceData(File xmlFile, File out)
{
    DocumentBuilderFactory df = DocumentBuilderFactory.newInstance();
    DocumentBuilder db = df.newDocumentBuilder();
    Document xmlDoc = db.parse(xmlFile);
    xmlDoc.getDocumentElement().normalize();

    Node allData = xmlDoc.getElementsByTagName("Data").item(0);
    Element ctrlData = getSubElement(allData, "ctrlData");
    NodeList subData = ctrlData.getElementsByTagName("SubData");

    int len = subData.getLength();

    for (int logIndex = 0; logIndex < len; logIndex++) {

        Node log = subData.item(logIndex);
        Element info = getSubElement(log, "info");
        Element value = getSubElement(info, "dailyInfo");
        Node valueNode = value.getElementsByTagName("value").item(0);
        valueNode.setTextContent("blah");               
    }

    TransformerFactory tf = TransformerFactory.newInstance();
    Transformer t = tf.newTransformer();
    DOMSource s = new DOMSource(xmlDoc);
    StreamResult r = new StreamResult(out);
    t.transform(s, r);

    } catch (TransformerException | ParserConfigurationException | SAXException | IOException e) {
         throw e;
    }
}

private static Element getSubElement(Node node, String elementName)
{
        return (Element)((Element)node).getElementsByTagName(elementName).item(0);
}

I notice that as I am further along the for loop the longer it takes, and for an average of 100k node's it takes over 2 hours, while if I just break out smaller chunks by hand of 1k, it will take ~10s. Is there something that is inefficient with the way that this document is being parsed?

----EDIT----

Based on comments and answers to this, I switched over to using Sax and XmlStreamWriter. Reference/example here: http://www.mkyong.com/java/how-to-read-xml-file-in-java-sax-parser/

After moving to using SAX, memory usage for the replaceData function does not expand to size of XML file, and XML file processing time went to ~18 seconds on average.

where did dimensionValue come from? where does it go? Same with dimension. — Bob Dalgleish
– Bob Dalgleish, Commented Apr 24, 2014 at 18:32
Ah sorry, forgot to edit that when I was simplyfing the code. Those values were array (int[]) lookups. — Niru
– Niru, Commented Apr 24, 2014 at 18:37
If I've similar problems I use a simple method: I let the code run in debugger and stop it often and randomly. The places where it stops most often are the slowest. Very simple, pretty efficient. Have fun. — Bernd Ebertz
– Bernd Ebertz, Commented Apr 24, 2014 at 18:38
Using the SAX API instead of loading the whole DOM into memory would be way more efficient. Admittedly, the code might not be as readable... — Ralf
– Ralf, Commented Apr 24, 2014 at 18:40
I did not see a way to replace xml values using a SAX parser. At that point wouldn't it be the same as just using a String buffer to read the xml file and replace the values line by line? — Niru
– Niru, Commented Apr 24, 2014 at 19:14

dimitrisli · Accepted Answer · 2014-04-24 19:36:04Z

3

As people have mentioned in the comments loading the whole DOM into memory especially for large XMLs can be very inefficient therefore a better approach is to use the SAX parser that consumes constant memory. The drawback there is that you don't get the fluent API of having the whole DOM in memory and the visibility is quite limited if you want to perform complicated callback logic in nested nodes.

If all you are interesting in doing is parsing particular nodes and node families rather than parsing the whole XML then there is a better solution that gives you the best of both worlds and has been blogged about and open-sourced. It's basically a very light wrapper on top of SAX parser where you are registering the XML elements you are interested in and when you are getting the callback you have at your disposal their corresponding partial DOM to XPath.

This way you can keep your complexity at constant time (scaling to over 1GB of XML file as documented in the above blog) while maintaining the fluency of XPath-ing the DOM of the XML elements you are interested in.

answered Apr 24, 2014 at 19:36

dimitrisli

21.4k12 gold badges64 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Niru Over a year ago

Thanks, I did switch over to using SAX and XmlStreamWriter using an example from here: mkyong.com/java/how-to-read-xml-file-in-java-sax-parser. For the same set of data, reading/replacing/writing is now down to 18 seconds.

Michael Kay · Accepted Answer · 2014-04-24 21:25:23Z

2

Why are you doing this in Java when XSLT is designed for the task?

45Mb is a big file to hold in memory, but still viable. The tree models used by good XSLT processors such as Saxon are much more efficient (both in storage space in in search speed) than a general purpose DOM (for example, because they are read-only). And XSLT has much more scope to optimize your code.

I can't reverse engineer your specification from your code, but I don't see anything in your description that is intrinsically non-linear. I don't see any reason why this should take more than 10 minutes or so in Saxon.

answered Apr 24, 2014 at 21:25

Michael Kay

165k11 gold badges97 silver badges173 bronze badges

Collectives™ on Stack Overflow

Java parse large XML document

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related