Read binary data with Node.js stream

Question

I'm trying to read a binary file with fs createReadStream. Assumed that we know the "misunderstanding" of binary and latin1 as value for the encoding option, and that by defaults using toString on the data chunk will use utf-8, I have tried to user a Buffer and buffer concatenation that is the .concat function in this way:

var readStream = fs.createReadStream( graphPath );
                        var data;
                        readStream.on('data', (chunk) => {
                            var b = new Buffer (chunk.length);
                            if(!data) data = new Buffer (chunk.length);
                            else data = Buffer.concat([data, b]);
                        })  
                        readStream.on('end', () => {
                            console.log( "type is", typeof (data ) );
                            console.log("read graph %d",data.length);data.buffer);
                        });

so that to append the new chunk to the data. In this way the typeof object in data is object, but it seems the encoding it is still not a binary format (i.e. it is not application/octet-stream charset=binary to be clear in terms of content type.)

If I just do data+=chunk the output type will be string.

peteb · Accepted Answer · 2017-11-07 16:55:00Z

7

If the data coming out of the file is already 'binary' encoding, then set the readStream encoding to 'binary' using setEncoding()and concatenate all of the chunks into a single string and return that. setEncoding() supports any of the encodings implemented in Node per Buffer docs for Encodings.

const getBinary = (graphPath, asBuffer = false, cb) => {
  let readStream = fs.createReadStream(graphPath)
  let data = ''

  // set stream encoding to binary so chunks are kept in binary
  readStream.setEncoding('binary')
  readStream.once('error', err => {
    return cb(err)
  })
  readStream.on('data', chunk => (data += chunk))
  readStream.on('end', () => {
    // If you need the binary data as a Buffer
    // create one from data chunks       
    return cb(null, asBuffer ? Buffer.from(data, 'binary') : data)
  })
}

edited Nov 7, 2017 at 16:55

answered Nov 7, 2017 at 16:48

peteb

19.5k9 gold badges58 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

loretoparisi Over a year ago

Thanks, the input file is a binary file. The api method that takes its data as input is supposed to load a byte array: public void importGraphDef(byte[] graphDef) throws IllegalArgumentException (this is the api signature). Your code is correct, but for some reason that method throws an IllegalArgumentException error like the passed` data from cb was a nested object. I have used it like getBinary(graphPath,true,(error,graphDef) => graph.importGraphDef(graphDef)); I must use asBuffer=true since it does not expect a string...

peteb Over a year ago

@loretoparisi, then if you always need the buffer then just remove the conditional in the return cb and the asBuffer flag. So your 'end' handler would like readStream.on('end', () => { return cb(null, Buffer.from(data, 'binary') }) and the getBinary() signature would be let getBinary = (graphPath, cb) => {}

Collectives™ on Stack Overflow

Read binary data with Node.js stream

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related