Storing JSON data from Node.js to MongoDB

Question

I'm pulling weather data in JSON format from Wundground through their API with no problems. I'm trying to store that data in MongoDB for later use. I actually get the data and am able to write it to a collection in Mongo. However, when I do a db.collection.find() it almost looks like each individual character is being saved separately rather than JSON format. Here's the code snippet that gets data and should be saving to Mongo:

// Define the Wunderground method.
var method = "/api/" + apiKey + "/conditions/q/" + state + "/" + city + ".json";

// Define the HTTP post properties.
var options = {
  host: 'api.wunderground.com',
  path: method,
  method: 'GET',
  port: 80
};

// Create the HTTP POST.
var request = http.request(options, function (response) {
  var str = '';

  // Create the listener for data being returned.
  response.on('data', function (chunk) {
    str += chunk;


    // Create the listener for the end of the POST.
    response.on('end', function (){
      db.collection('weathercollection').save(str, function(err, records) {
        if (err) throw err;
        console.log("record added");
      });
    });

A small excerpt of the JSON-formatted weather data:

{ "current_observation": {
    "image": {
    "url": "http://icons-ak.com/graphics/logo.png",
    "title": "Weather Underground"
    },
    "display_location": {
    "full":"My City, State",
    "city":"My City",

I shouldn't have to parse the data before saving to Mongo should I? So what am I missing. As I said, if I output to the console all the weather data displays perfectly I just seem to be doing something wrong between Node.JS and MongoDB.

Thanks.

UPDATE***

I did try to parse "str" in this way with

// Create the listener for data being returned.
response.on('data', function (chunk) {
str += chunk;

var jsonResult = JSON.parse(str);

// Create the listener for the end of the POST.
response.on('end', function (){
  db.collection('weathercollection').save(jsonResult, function(err, records) {
    if (err) throw err;
    console.log("record added");`

That didn't seem to work either. I will look at it again.

Looks like you have a String, parse it!

tymeJV
– tymeJV

2013-11-07 21:00:53 +00:00
Commented Nov 7, 2013 at 21:00 — tymeJV
– tymeJV, Commented Nov 7, 2013 at 21:00

Paul Mougel · Accepted Answer · 2015-01-30 10:38:37Z

13

Yes, you need to give to your send function a JavaScript object (cf. the MongoDB native driver documentation, which it looks like you're using), but you send it a string (which is why you can concatenate it on each data event). You'll have to convert your string to a full object using JSON.parse(str).

If you want to be sure of which data type you're dealing with, print the result of typeof str and typeof JSON.parse(str).

Edit: You have a second issue in your code. The response object is actually a stream, which means it emits data when it receives it. This also means you can receive the data event multiple times. This is why you need to:

Create an empty string
On each data event, concatenate the chunk you just received to the string
Only try to parse it at then end, when you're sure you will not receive any more data.

In the updated code snippet you gave, you tried to parse the string on the first data event, but that might be an incomplete string.

Here is the correct way to achieve this:

var str = '';
response.on('data', function(chunk) {
  str += chunk;
});
response.on('end', function() {
  var myObject = JSON.parse(str);
  // Send the Mongo query here
});

Related to this issue, you also registered a listener to the end event, which is good, but you added a new listener on each data event! Which means if you receive 5 data events, you'll call 5 times the function that will add the object to MongoDB… In the snippet above, notice I've moved the response.on('end', function() {…}) on the outside of the response.on('data') callback.

edited Jan 30, 2015 at 10:38

answered Nov 7, 2013 at 21:03

Paul Mougel

17.1k6 gold badges59 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Godfried Over a year ago

I suspected as much and had tried that as well (see update to original post). I'm still missing something though. I'll go back and give it another shot.

Godfried Over a year ago

Just did Json.parse(str) and Node throws and error: undefined:43 "obser SyntaxError: Unexpected token o

Paul Mougel Over a year ago

Makes sense. See my edit: you get a SyntaxError because you try to parse an incomplete string. Wait for the end event before parsing it.

dur · Accepted Answer · 2016-07-26 12:02:13Z

0

var MongoClient = require('mongodb').MongoClient, format = require('util').format;

MongoClient.connect('mongodb://127.0.0.1:27017/test', function(err,db) {

    if (err) throw err;
    console.log("Connected to Database");
    var document = {name:"David", title:"About MongoDB"};

    // insert record
    db.collection('test').insert(document, function(err, records) {
        if (err) throw err;
        console.log("Record added as " + records[0]._id);
    });
});

Reference from: http://code.runnable.com/UW3ef2Tkq498AABE/insert-a-record-in-mongodb-using-mongodb-native-for-node-js

edited Jul 26, 2016 at 12:02

dur

17.3k26 gold badges93 silver badges146 bronze badges

answered Jul 21, 2016 at 9:36

Ankit Jain

6722 gold badges6 silver badges18 bronze badges

Collectives™ on Stack Overflow

Storing JSON data from Node.js to MongoDB

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related