Async map with headers for getting multiple url's parallel

Question

I am learning async map . I want to download a bunch of URLs but i want to send a header along with the get request.

If i didn't have a header, I could have just done

var request = require('request');
var async = require('async');

var urls = ['http://myurl1.com', 'http://myurl2.com', 'http://myurl3.com'];

async.map(urls, request, function(err, results) {
        if (err) throw(err);          // handle error 
        console.log(results.length);  // == urls.length
});

Now I do need to send a header {"x-url-key":"myurlkey"} along with every get request. How do I modify the code above to do so?

Terry Lennox · Accepted Answer · 2020-08-19 14:06:26Z

1

That should be straightforward enough to do, we can create a wrapper function requestWithHeader to pass to async.map, this will specify whichever headers (or other options) you wish.

I'm also specifying json: true here, you may not want to do that in your actual code.

In this example I'm using https://httpbin.org/get as the url, this will send back all the request parameters which is useful for test purposes as we can see which headers we populated.

var request = require('request');
var async = require('async');

var urls = ["https://httpbin.org/get?foo=bar", "https://httpbin.org/get?foo=baz"];

function requestWithHeader(uri, callback) {
    request(uri, { headers: {"x-url-key":"myurlkey"}, json:true }, callback)
}

async.map(urls, requestWithHeader, function(err, results) {
    if (err) throw(err);          // handle error 
    console.log("Results:", results.map(result => result.body));
});

To wait for async.map to finish you can create an asynchronous function to call it, e.g.

async function testMapWithPromise() {
    try { 
        let results = await async.map(urls, requestWithHeader);
        console.log("testMapWithPromise: Results:", results.map(result => result.body));
        // Do whatever with results here...
    } catch (error) {
        console.error("testMapWithPromise: An error occurred:", error);
    }
}

testMapWithPromise();

edited Aug 19, 2020 at 14:06

answered Aug 19, 2020 at 9:07

Terry Lennox

30.9k5 gold badges36 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Illusionist Over a year ago

Thank you ! If i want to wait on async.map to finish before doing other stuff, where should that code be?

Terry Lennox Over a year ago

Well, async.map will return a Promise if no callback is passed, so you can use async / await to wait for all the calls to complete. I'll add an example!

Collectives™ on Stack Overflow

Async map with headers for getting multiple url's parallel

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related