How Javascript/Node.js engine store JSON in memory?

Question

I have a 70 MB JSON file, which I read from a Node.js script and assign to a variable.

console.log(process.memoryUsage());
let data=require('../newJSON.json');
console.log(process.memoryUsage());

Output:

{ rss: 28184576,
  heapTotal: 6283264,
  heapUsed: 4199672,
  external: 8252 }
{ rss: 724721664,
  heapTotal: 695595008,
  heapUsed: 663708016,
  external: 8252 }

It seems that 70 MB JSON takes 632 MB of memory. I am interested in understanding how does JSON is stored into memory by Node Js/ Javascript?

I'm curious too! And if you save your JSON as a .js file with the only content being module.exports = <json> then it takes even ~2x more memory! — Lukasz Wiktor
– Lukasz Wiktor, Commented Jul 10, 2017 at 8:49
Yeah! for me module.exports caused FatalProcessOutOfMemory issue. — anand
– anand, Commented Jul 10, 2017 at 9:08
What kind of JSON is it, what's the structure and average element size? — Bergi
– Bergi, Commented Jul 10, 2017 at 9:42
e.g. : {"8 char": {''6 char": {"1 char": { "8 char": {"1 char":<0 or 1> ,"1 char":<0 or 1>} } } } }. (8 char : average 8 characters in key) — anand
– anand, Commented Jul 10, 2017 at 9:53

jmrk · Accepted Answer · 2017-07-10 18:04:40Z

8

First off, JSON is just a string representation of objects. There is nothing special about "JSON objects" -- the JSON parser parses the JSON string and creates regular JavaScript objects from it. This:

var a = JSON.parse('{"foo": "bar"}');

and this:

var a = new Object(); a.foo = "bar";

are completely equivalent.

Object storage in memory is complicated, because modern JavaScript engines have pretty nifty optimizations for various different circumstances depending on what your code is doing.

JSON string length and size of the corresponding object in memory are not strictly correlated; in most cases the JSON representation is expected to be smaller, sometimes by a lot. E.g. for the innermost nesting of your example: "a":0, takes 6 bytes, whereas for one more property in the created object, you need:

one pointer for the property's name, "a"
one pointer for the property's attributes (writable, enumerable, configurable)
one pointer for the property's value, 0
assuming the object is in dictionary mode: on average, approximately two pointers of slack

On a 64-bit platform, that adds up to ~40 bytes.

If you look at an entire object of similar shape: {"a":0,"b":1} is 13 characters, whereas the memory requirement is:

"map" pointer
"elements" pointer (unused)
out-of-object "properties" pointer (unused)
value of first property (0)
value of second property (1)
the object's "map": 11 pointers (could be shared with other objects of the same shape, but if you have only one such object, there's nothing to share it with)
the object's map's property descriptors: 10 pointers

In total, 26 pointers or 208 bytes.

Lastly, there's a chance that some of the memory usage you see is from temporary objects that the GC will clean up over time.

answered Jul 10, 2017 at 18:04

jmrk

42.1k7 gold badges74 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Juan Perez Over a year ago

Do you know if there's a table somewhere that says roughly how much RAM bytes are required by nodejs to JSON.parse() a json string of x bytes?

jmrk Over a year ago

@JuanPerez I don't think such a table can exist (while being useful/accurate), because as my answer states, JSON string length and size of the corresponding object in memory are not strictly correlated. There are examples where the decoded form of an N-byte JSON string takes less than N bytes, and there are examples where it takes 10*N bytes or even more.

Collectives™ on Stack Overflow

How Javascript/Node.js engine store JSON in memory?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related