Create big json object js

Question

I am using Nodejs to Create a JSON file from a really large JSON object (1GB). In order to avoid memory problems, I'm using createWriteStream :

var writeStream = fs.createWriteStream('./output/outPut.json')

After using Builder(custom function ) it will return a big object JSON.

the final step is to create this file :

 writeStream.write(JSON.stringify(search_index), (err) => {
             if (err) throw err
             console.log('File  Ready... ')
                    })

but unfortunately, JSON.stringify cannot be used with such heavy Object

JSON.stringify throws RangeError: Invalid string length for huge objects

Is there a solution to this issue, please?

whats the purpose of creating such a large file. Can you split the information into smaller files? — Craicerjack
– Craicerjack, Commented Dec 20, 2020 at 21:35
@Craicerjack I am using elasticlunr JS to build the search module in my application, and this file is the search index, unfortunately, it cannot be split into multiple files — ben ftima wajdi
– ben ftima wajdi, Commented Dec 20, 2020 at 21:38
Does this answer your question? Parse large JSON file in Nodejs — Rishabh Deep Singh
– Rishabh Deep Singh, Commented Dec 20, 2020 at 21:38
@RishabhDeepSingh this answer is about how to read large JSON not create it — ben ftima wajdi
– ben ftima wajdi, Commented Dec 20, 2020 at 21:40

Bergi · Accepted Answer · 2020-12-20 22:58:03Z

3

A write stream is of no use if you only write one large string. You cannot use the native JSON.stringify here. There are however quite a few libraries available that will implement JSON serialisation into a stream (and even more for deserialising from a stream), use one of them.

answered Dec 20, 2020 at 22:58

Bergi

671k162 gold badges1k silver badges1.5k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Dharman · Accepted Answer · 2020-12-21 20:40:33Z

1

First step is to init your stream obejct

const writeStream = fs.createWriteStream('./object.json', { flags: 'w' })

then Converting my data to string JSON (stringify ) Using JSON Stream Stringify module

const JsonStreamStringify = require('json-stream-stringify')
const jsonStream = new JsonStreamStringify(Promise.resolve(Promise.resolve(TargetData)))

The last step is to pipe the returned data to writeStream

jsonStream.pipe(writeStream)
jsonStream.on('end', () => console.log('done '))

But It may take a lot of time in my case 10-15 min (1.1GB)

edited Dec 21, 2020 at 20:40

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Dec 21, 2020 at 20:34

ben ftima wajdi

1132 silver badges11 bronze badges

Comments

Barthy · Accepted Answer · 2020-12-21 00:30:47Z

When writing big collections of data into files (i.e. lots of documents in a search index), it is generally best to work with smaller packages in order to find the balance between memory consumption and computation time.

For example, you could create 100 JavaScript objects (n = 100), stringify them, append them to your file and continue with the next 100 objects.

The trick here is to fine a package size n that makes good use of your system's memory without spending too much time with read/write operations on files.

A common pitfall is that your total number of documents might not be an integer multiple of your package size n. When iterating using a zero-based index variable you should then write to the file if:

index % n === 0 || index === (totalCount - 1)

An additional benefit is that you can start/restart this process after an error without having to recreate all of the data (i.e. if some error occurs after writing 800MB of data you can log the last successful index and continue from there).

Collectives™ on Stack Overflow

Create big json object js

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related