Parsing a large JSON array in Javascript

Question

I'm supposed to parse a very large JSON array in Javascipt. It looks like:

mydata = [
    {'a':5, 'b':7, ... },
    {'a':2, 'b':3, ... },
    .
    .
    .
    ]

Now the thing is, if I pass this entire object to my parsing function parseJSON(), then of course it works, but it blocks the tab's process for 30-40 seconds (in case of an array with 160000 objects).

During this entire process of requesting this JSON from a server and parsing it, I'm displaying a 'loading' gif to the user. Of course, after I call the parse function, the gif freezes too, leading to bad user experience. I guess there's no way to get around this time, is there a way to somehow (at least) keep the loading gif from freezing?

Something like calling parseJSON() on chunks of my JSON every few milliseconds? I'm unable to implement that though being a noob in javascript.

Thanks a lot, I'd really appreciate if you could help me out here.

160 000 objects is really a lot. You should split it up on server side — Pinoniq
– Pinoniq, Commented Sep 12, 2014 at 14:12
Have you checked this thread? stackoverflow.com/questions/1160137/… — Tasos K.
– Tasos K., Commented Sep 12, 2014 at 14:15
Apply pagination on server side, add pagination numbers and next page etc in json, and then keep calling api on ajax success. — Rohit Awasthi
– Rohit Awasthi, Commented Sep 12, 2014 at 14:16
You could split the data and send the chunks at different points of time if that's applicable to your case. — KiaMorot
– KiaMorot, Commented Sep 12, 2014 at 14:16
You can call your parseJSON using setTimeout with a minimum delay. Refer this SO post for some more info. — Thangadurai
– Thangadurai, Commented Sep 12, 2014 at 14:17

Cyril Duchon-Doris · Accepted Answer · 2014-09-12 14:45:37Z

1

You might want to check this link. It's about multithreading.

Basically :

var url = 'http://bigcontentprovider.com/hugejsonfile';

var f = '(function() {
            send = function(e) { 
                postMessage(e); 
                self.close();
            };
            importScripts("' + url + '?format=json&callback=send");
         })();';

var _blob = new Blob([f], { type: 'text/javascript' });

_worker = new Worker(window.URL.createObjectURL(_blob));
_worker.onmessage = function(e) { 
    //Do what you want with your JSON 
}
_worker.postMessage();

Haven't tried it myself to be honest...

EDIT about portability: Sebastien D. posted a comment with a link to mdn. I just added a ref to the compatibility section id.

edited Sep 12, 2014 at 14:45

answered Sep 12, 2014 at 14:19

Cyril Duchon-Doris

14.2k12 gold badges92 silver badges181 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

nepeo Over a year ago

This is a really good suggestion, but perhaps you could expand on it a bit and add some warnings about portability?

Cyril Duchon-Doris Over a year ago

I don't enough on the subject to be able to expand too much. However I reposted a link with browser compatibility.

nepeo Over a year ago

Me either to be honest, I hadn't even heard of the feature until you posted it! It's quite an exciting api though. Given that I mostly work on mobile devices the lack of android support is depressing, guess it's going to be awhile before I get to try it out...

nepeo · Accepted Answer · 2014-09-12 14:54:50Z

I have never encountered a complete page lock down of 30-40 seconds, I'm almost impressed! Restructuring your data to be much smaller or splitting it into many files on the server side is the real answer. Do you actually need every little byte of the data?

Alternatively if you can't change the file @Cyrill_DD's answer of a worker thread will be able to able parse data for you and send it to your primary JS. This is not a perfect fix as you would guess though. Passing data between the 2 threads requires the information to be serialised and reinterpreted, so you could find a significant slow down when the data is passed between the threads and be back to square one again if you try to pass all the data across at once. Building a query system into your worker thread for requesting chunks of the data when you need them and using the message callback will prevent slow down from parsing on the main thread and allow you complete access to the data without loading it all into your main context.

I should add that worker threads are relatively new, main browser support is good but mobile is terrible... just a heads up!

Collectives™ on Stack Overflow

Parsing a large JSON array in Javascript

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related