Node.js HTTP response streams

Question

Using the native http.get() in Node.js, I'm trying to pipe a HTTP response to a stream that I can bind data and end events to.

I'm currently handling this for gzip data, using:

http.get(url, function(res) {
  if (res.headers['content-encoding'] == 'gzip') {
    res.pipe(gunzip);
    gunzip.on('data', dataCallback);
    gunzip.on('end', endCallback);
  }
});

Gunzip is a stream and this just works. I've tried to create streams (write streams, then read streams) and pipe the response, but haven't been having much luck. Any suggestions to replicate this same deal, for non-gzipped content?

I'm trying to get the response body as it comes, then once it's finished. I thought I could do res.on('data') but that never seems to trigger. — dzm
– dzm, Commented Oct 25, 2013 at 3:20

hexacyanide · Accepted Answer · 2013-10-25 03:21:48Z

34

The response object from a HTTP request is an instance of readable stream. Therefore, you would collect the data with the data event, then use it when the end event fires.

var http = require('http');
var body = '';

http.get(url, function(res) {
  res.on('data', function(chunk) {
    body += chunk;
  });
  res.on('end', function() {
    // all data has been downloaded
  });
});

The readable.pipe(dest) would basically do the same thing, if body in the example above were a writable stream.

answered Oct 25, 2013 at 3:21

hexacyanide

92.4k31 gold badges168 silver badges162 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

dzm Over a year ago

Doh, well it turned out to be due to http not following redirects. Your code is what I have and it works fine, with the exception of redirects. I checked out npm follow-redirects and will test it out.

vkurchatkin Over a year ago

You can also try popular request module, which also follows redirects

dzm Over a year ago

@vkurchatkin request does support gzipped content

Jwan622 Over a year ago

Why do you have to do string concatenation? is chunk not the entirety of the data?

hexacyanide Over a year ago

You need to do string concatenation because the response is a stream. The data event fires when there's data immediately available, which may or may not be the entire response.

|

The Fool · Accepted Answer · 2020-05-23 14:37:33Z

Nowadays the recommended way of piping is using the pipeline function. It is supposed to protect you from memory leaks.

const { createReadStream} = require('fs');
const { pipeline } = require('stream')
const { createServer, get } = require('http')

const errorHandler = (err) => err && console.log(err.message);

const server = createServer((_, response) => {
  pipeline(createReadStream(__filename), response, errorHandler)
  response.writeHead(200);
}).listen(8080);

get('http://localhost:8080', (response) => {
  pipeline(response, process.stdout, errorHandler);
  response.on('close', () => server.close())
});

Another way of doing it that has more control would be to use async iterator

async function handler(response){
  let body = ''
  for await (const chunk of response) {
    let text = chunk.toString()
    console.log(text)
    body += text
  }
  console.log(body.length)
  server.close()
}

get('http://localhost:8080', (response) => handler(response).catch(console.warn));

Collectives™ on Stack Overflow

Node.js HTTP response streams

2 Answers 2

7 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related