Write raw binary data to Buffer

Question

I am working on a function that iterates over PCM data. I am getting chunks of data of varying size and I am currently handling this by buffer concatenation. Problem is, I am quite sure that this approach is a performance killer.

One of the simplest algorithm consists of chunking 500 chunks of 4800 bytes (= grain), and repeating them 3 times as such :

buf = <grain1, grain1, grain1, ..., grain500, grain500, grain500>

function(){
  // ...
  let buf = Buffer.alloc(0) // returned buffer, mutated

  // nGrains is defined somewhere else in the function
  // example: nGrains = 500
  for(let i=0;i<nGrains;i++){
    // a chunk of PCM DATA 
    // example: grain.byteLength = 4800 
    const grain = Buffer.from(this._getGrain())

    // example: nRepeats = 3
    for(let j=0;j<nRepeats;j++)
      buf = Buffer.concat([buf, grain])
  }

  return buf
}

I feel like these performance heavy operations (1500 mutating concatenations) could be avoided if there were some sort of way to directly write "raw data" from a given offset to a pre-size-allocated buffer. I made the following helper function that gave me HUGE performance improvements, but I feel like I am doing something wrong...

function writeRaw(buf, rawBytes, offset) => {
  for(i=0;i<rawBytes.byteLength;i++){
    buf.writeUInt8(rawBytes.readUInt8(i), offset + i)
  }

  return buf
}

My function now looks like this:

function(){
  // ...
  const buf = Buffer.alloc(len) // returned buffer, immutable

  for(let i=0;i<nGrains;i++){
    const grain = Buffer.from(this._getGrain())

    for(let j=0;j<nRepeats;j++)
      writeRaw(buf, grain, (i * nRepeats + j) * grainSize)
  }

  return buf
}

My question is : Is there a cleaner way (or more standard way) to do this instead of iterating over bytes ? Buffer.write only seems to work for strings, although this would be ideal...

Jonas Wilms · Accepted Answer · 2019-04-21 12:40:00Z

1

There is Buffer.copy.

const buf = Buffer.alloc(len);

for(let i = 0; i < nGrains; i++){
  const grain = Buffer.from(this._getGrain());

 for(let j=0;j<nRepeats;j++)
   grain.copy(/*to*/ buf, /*at*/ (i * nRepeats + j) * grainSize);
}

You could also use Buffer.fill:

  const buf = Buffer.alloc(len);

  for(let i = 0; i < nGrains; i++) {
    const grain = Buffer.from(this._getGrain());
    buf.fill(grain, i * nRepeats * grainSize, (i + 1) * nRepeats * grainSize);
  }

edited Apr 21, 2019 at 12:40

answered Apr 21, 2019 at 12:13

Jonas Wilms

139k20 gold badges164 silver badges164 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

BJRINT Over a year ago

I went for Buffer.copy which is exactly what I wanted ... I guess I was misled by the method name. Thanks!

BJRINT Over a year ago

Please note that you inverted the parameters on Buffer.copy (it is source.copy(target, ...), so grain.copy(buf, ...) in my case). Just a typo I guess.

Collectives™ on Stack Overflow

Write raw binary data to Buffer

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related