Making HTTP request and parsing the result as JSON

Question

I want to download a JSON and parse it. I tried the following approach:

    var request = require('request');
    var url = "http://iiif.nli.org.il/collections/danhadani.json"
    var result = request(url , function(error, response, body) {
          console.log("Fin");
          JSON.parse(body);
     });

undefined
> Fin
Fin
SyntaxError: Unexpected token 
    at Object.parse (native)
    at Request._callback (repl:1:81)
    at Request.self.callback (/home/artium/Projects/nlihack-team-m7/node_modules/request/request.js:186:22)
    at emitTwo (events.js:87:13)
    at Request.emit (events.js:172:7)
    at Request.<anonymous> (/home/artium/Projects/nlihack-team-m7/node_modules/request/request.js:1163:10)
    at emitOne (events.js:77:13)
    at Request.emit (events.js:169:7)
    at IncomingMessage.<anonymous> (/home/artium/Projects/nlihack-team-m7/node_modules/request/request.js:1085:12)
    at IncomingMessage.g (events.js:260:16)

I am able to log the JSON string retrieved in the body, it looks ok to me, so I guess I am doing the parsing wrong.

Edit:

The first characters of the body looks like this:

> body.substring(1,250)
'{"@context":"http://iiif.io/api/presentation/2/context.json",\n"@id": "http://iiif.nli.org.il/collections/danhadani.json",\n"@type":"sc:Collection",\n"label":"Dan Hadani Collection", \n"attribution":[{"@value":"The National Library of Israel","@language'

Your code looks correct. What do you get if you console.log(body) with console.log("Fin") — samanime
– samanime, Commented Nov 28, 2017 at 20:37
@samanime I get a string with something that looks like json inside. This is a huge file, I will update the question with the first characters. — Artium
– Artium, Commented Nov 28, 2017 at 20:44
I ask because your code should be fine. Throwing a JSON string into JSON.parse(body) should work fine. If it doesn't, there is likely a syntax error somewhere in the JSON. You might run it against an online parser and see what it says. — samanime
– samanime, Commented Nov 28, 2017 at 20:47
@samanime Yes, I ran it through jsonlint. At least what I got after downloading the json through firefox. — Artium
– Artium, Commented Nov 28, 2017 at 20:52
Ah, yes, that does actually make sense. It looks like that JSON has non-ASCII characters, so when it gets to them, it likely gets confused. Normally the headers should specify that it's UTF-8 when it comes in, but it looks like the headers aren't specified, so you have to provide it yourself. — samanime
– samanime, Commented Nov 28, 2017 at 21:09

Skarlinski · Accepted Answer · 2017-11-28 21:18:56Z

3

Tested & working after specifying encoding:

{encoding:'utf8'}

It seems that the specific url you are requesting does not specificity it's encoding in the response header, so we have to manually set it like so:

 request(u ,{encoding:'utf8'}, 
   function(error, response, body) { console.log("Fin");  JSON.parse(body)  })

edited Nov 28, 2017 at 21:18

answered Nov 28, 2017 at 21:01

Skarlinski

2,48715 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Skarlinski Over a year ago

@artium, still checking why encoding was the issue

Elad · Accepted Answer · 2017-11-28 21:54:40Z

1

Try this:

var request = require('request');
var url = "http://iiif.nli.org.il/collections/danhadani.json";

var options = {
  uri: url,
  method: 'GET',
  json : true,
  encoding: 'utf8'
};

var r = request(options , function(error, response, body) { 
    console.log("Fin");
    // now you have an Array(43515) of objects on body.members without the need of parsing.
    console.log(`The first object in the json file is: ${body.members[0]}`);
});

You will get the data as array of objects (becuse of the format of that json file)

I tryed the code, and it works.

בהצלחה!

edited Nov 28, 2017 at 21:54

answered Nov 28, 2017 at 21:12

Elad

2,4771 gold badge19 silver badges34 bronze badges

Collectives™ on Stack Overflow

Making HTTP request and parsing the result as JSON

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related