Nodejs binary http stream

Question

I need to stream files from a client (nodejs command line) and a server (express nodejs).

This is the client side:

var request = require('request');
var fs = require('fs');

// ...

  var readStream = fs.createReadStream(file.path);
  readStream.on('end', function() {
    that.emit('finished');
  });
  readStream.pipe(request.post(target));

// ...

This is the server side:

var fs = require('fs');
var path = require('path');

// ...

    app.post('/:filename', function(req, res) {
      req.setEncoding('binary');
      var filename = path.basename(req.params.filename);
      filename = path.resolve(destinationDir, filename);
      var dst = fs.createWriteStream(filename);
      req.pipe(dst);
      req.on('end', function() {
        res.send(200);
      });
    });

// ...

All is working, files are saved correctly on the server side... but they are about 50% bigger than the source files. I tried to see difference between the two files with hexdump and the server side file has similar content but with 0xC2 sometimes. I guess this is related to encoding.

Why do you set the encoding? This will convert every single chunk into strings and is mainly intended if you want to read strings from the stream. As you directly pipe the request to a file, I don't think you should be calling setEncoding. — Paul Mougel
– Paul Mougel, Commented Nov 20, 2013 at 17:19
fs.createWriteStream(filename, {encoding: 'binary'}) didn't work. Node api: The encoding can be 'utf8', 'ascii', or 'base64' — Alessandro Pezzato
– Alessandro Pezzato, Commented Nov 20, 2013 at 17:24
Removing 'req.setEncoding('binary');` is the solution. Strange... I added it when it wasn't working — Alessandro Pezzato
– Alessandro Pezzato, Commented Nov 20, 2013 at 17:26
Good to know! Will add this as an answer so that the solution will be easier to find. — Paul Mougel
– Paul Mougel, Commented Nov 20, 2013 at 17:57

Paul Mougel · Accepted Answer · 2013-11-20 17:57:23Z

3

Don't call req.setEncoding('binary').

This will convert every single chunk into strings and is mainly intended if you want to read strings from the stream. As you directly pipe the request to a file, you don't need to do it.

answered Nov 20, 2013 at 17:57

Paul Mougel

17.1k6 gold badges59 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

vkurchatkin Over a year ago

Also it's worth mentioning that binary encoding is deprecated.

Paul Mougel Over a year ago

This isn't mentioned in the documentation. The source code mentions backwards compatibility, but not deprecation. Do you have a link?

Paul Mougel Over a year ago

Which refers to this issue, looks like binary won't go away soon...

vkurchatkin Over a year ago

Well, I think one can tell that binary is a little deprecated, but not deprecated-deprecated) Anyway, it's not a good idea to use it in new code.

Collectives™ on Stack Overflow

Nodejs binary http stream

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related