Parsing large XML file in Node.js

Question

So I have an XML file that is larger than 70mb. I would like to parse this data for in Node.js to do data visualizations on it eventually. To start, I thought it would be best to use JSON instead of XML, because Node.js is better built to work with JSON. So I planned to use the xml2json node module to parse the xml into JSON but I can't seem to write the xml file to a variable because its so large. I attempted to do this with the following code.

var fs = require('fs');


fs.readFile(__dirname + '/xml/ipg140114.xml', 'utf8', function(err, data, parseXml) {
    if(err) {
        return console.log(err);
    } 
});

I receive a stack trace error. Whats a better way to get this file converted into JSON so I can parse it with Node? I am pretty new to Node so let me know if my approach is wrong. Thanks in advance!

Possible duplicate of Node.js Example to convert Xml to JSON for large Xml file — planet_hunter
– planet_hunter, Commented Oct 1, 2018 at 8:15

Søren Ullidtz · Accepted Answer · 2016-11-18 10:29:04Z

0

Json2xml requires you to load the entire file into memory. You could allocate more memory but I would recommend parsing the XML directly from the file instead.

There are other libraries on NPM such as xml-stream that will allow you to parse the XML directly form the file without loading it all into memory.

My personal issue with xml-stream is that it relies on GYP, which can be a hassle if you're a windows user. I added a very basic parser called no-gyp-xml-stream to NPM, this one only depends on sax. But it's a bit rudimentary and may not suit your needs.
I am however willing to improve it if anyone needs anything: https://www.npmjs.com/package/no-gyp-xml-stream

answered Nov 18, 2016 at 10:29

Søren Ullidtz

1,5241 gold badge16 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Parsing large XML file in Node.js

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related