Converting Uint8Array to BigInt in Javascript

Question

I have found 3 methods to convert Uint8Array to BigInt and all of them give different results for some reason. Could you please tell me which one is correct and which one should I use?

Using bigint-conversion library. We can use bigintConversion.bufToBigint() function to get a BigInt. The implementation is as follows:

export function bufToBigint (buf: ArrayBuffer|TypedArray|Buffer): bigint {
  let bits = 8n
  if (ArrayBuffer.isView(buf)) bits = BigInt(buf.BYTES_PER_ELEMENT * 8)
  else buf = new Uint8Array(buf)

  let ret = 0n
  for (const i of (buf as TypedArray|Buffer).values()) {
    const bi = BigInt(i)
    ret = (ret << bits) + bi
  }
  return ret
}

Using DataView:

let view = new DataView(arr.buffer, 0);
let result = view.getBigUint64(0, true);

Using a FOR loop:

let result = BigInt(0);
for (let i = arr.length - 1; i >= 0; i++) {
  result = result * BigInt(256) + BigInt(arr[i]);
}

I'm honestly confused which one is right since all of them give different results but do give results.

@Ry- I'm fine with either BE or LE but I'd just like to know why these 3 methods give a different result. I've tried running the for loop from 0 to end instead of reverse but it still was giving me different results. — Wor Chan
– Wor Chan, Commented Jul 31, 2022 at 6:52
There is something wrong with the last example. Why would you multiply by 256? — Konrad
– Konrad, Commented Jul 31, 2022 at 7:50
@KonradLinkowski: Because 256 is 2^8. It’s correct, for little-endian. — Ry-
– Ry- ♦, Commented Jul 31, 2022 at 8:44
That DataView will only work with 64 bits, as the name getBigUint64 suggests. The last snippet works only with an Uint8Array, unlike the first snippet which works with any typed array or buffer - also they use different endianness. — Bergi
– Bergi, Commented Jul 31, 2022 at 10:18

jmrk · Accepted Answer · 2022-07-31 14:18:07Z

9

I'm fine with either BE or LE but I'd just like to know why these 3 methods give a different result.

One reason for the different results is that they use different endianness.

Let's turn your snippets into a form where we can execute and compare them:

let source_array = new Uint8Array([
    0xff, 0xee, 0xdd, 0xcc, 0xbb, 0xaa, 0x99, 0x88, 
    0x77, 0x66, 0x55, 0x44, 0x33, 0x22, 0x11]);
let buffer = source_array.buffer;

function method1(buf) {
  let bits = 8n
  if (ArrayBuffer.isView(buf)) {
    bits = BigInt(buf.BYTES_PER_ELEMENT * 8)
  } else {
    buf = new Uint8Array(buf)
  }

  let ret = 0n
  for (const i of buf.values()) {
    const bi = BigInt(i)
    ret = (ret << bits) + bi
  }
  return ret
}

function method2(buf) {
  let view = new DataView(buf, 0);
  return view.getBigUint64(0, true);
}

function method3(buf) {
  let arr = new Uint8Array(buf);
  let result = BigInt(0);
  for (let i = arr.length - 1; i >= 0; i--) {
    result = result * BigInt(256) + BigInt(arr[i]);
  }
  return result;
}

console.log(method1(buffer).toString(16));
console.log(method2(buffer).toString(16));
console.log(method3(buffer).toString(16));

Note that this includes a bug fix for method3: where you wrote for (let i = arr.length - 1; i >= 0; i++), you clearly meant i-- at the end.

For "method1" this prints: ffeeddccbbaa998877665544332211
Because method1 is a big-endian conversion (first byte of the array is most-significant part of the result) without size limit.

For "method2" this prints: 8899aabbccddeeff
Because method2 is a little-endian conversion (first byte of the array is least significant part of the result) limited to 64 bits.
If you switch the second getBigUint64 argument from true to false, you get big-endian behavior: ffeeddccbbaa9988.
To eliminate the size limitation, you'd have to add a loop: using getBigUint64 you can get 64-bit chunks, which you can assemble using shifts similar to method1 and method3.

For "method3" this prints: 112233445566778899aabbccddeeff
Because method3 is a little-endian conversion without size limit. If you reverse the for-loop's direction, you'll get the same big-endian behavior as method1: result * 256n gives the same value as result << 8n; the latter is a bit faster.
(Side note: BigInt(0) and BigInt(256) are needlessly verbose, just write 0n and 256n instead. Additional benefit: 123456789123456789n does what you'd expect, BigInt(123456789123456789) does not.)

So which method should you use? That depends on:
(1) Do your incoming arrays assume BE or LE encoding?
(2) Are your BigInts limited to 64 bits or arbitrarily large?
(3) Is this performance-critical code, or are all approaches "fast enough"?

Taking a step back: if you control both parts of the overall process (converting BigInts to Uint8Array, then transmitting/storing them, then converting back to BigInt), consider simply using hexadecimal strings instead: that'll be easier to code, easier to debug, and significantly faster. Something like:

function serialize(bigint) {
  return "0x" + bigint.toString(16);
}
function deserialize(serialized_bigint) {
  return BigInt(serialized_bigint);
}

answered Jul 31, 2022 at 14:18

jmrk

42.1k7 gold badges74 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Bergi Over a year ago

"Using hexadecimal strings will be significantly faster" - that's a real shame, isn't it? One would hope that copying the bytes from the internal bigint representation into a buffer (or the other way round) is faster, and more space efficient, than string parsing and serialisation.

Wor Chan Over a year ago

> Additional benefit: 123456789123456789n does what you'd expect, BigInt(123456789123456789) does not.) Could you please explain this? I am unable to use 256n because typescript complains: "BigInt literals are not available when targeting lower than ES2020."

jmrk Over a year ago

@Bergi: a built-in BigInt <-> ArrayBuffer conversion routine could indeed be more efficient; the chunking/shifting/masking you need when building this in userspace is going to be slower. That said, using hex strings is a fine solution, I think; it's not clear whether having some other facility would be worth it.

jmrk Over a year ago

@WorChan: Just try BigInt(123456789123456789) at the console and see what happens. The explanation is that 123456789123456789 is a Number literal and as such has limited precision; converting it to BigInt afterwards can't magically restore the lost bits.

Bergi Over a year ago

@WorChan You should definitely change your TypeScript options to fix that warning if you want to use bigints.

|

Bergi · Accepted Answer · 2022-10-29 16:18:03Z

If you need to store really big integers that isn't bound to any base64 or 128 and also keep negative numbers then this is a solution for you...

function encode(n) {
  let hex, bytes

  // shift all numbers 1 step to the left and xor if less then 0
  n = (n << 1n) ^ (n < 0n ? -1n : 0n) 

  // convert to hex
  hex = n.toString(16)
  // pad if neccesseery
  if (hex.length % 2) hex = '0' + hex

  // convert hex to bytes
  bytes = hex.match(/.{1,2}/g).map(byte => parseInt(byte, 16))

  return bytes
}

function decode(bytes) {
  let hex, n

  // convert bytes back into hex
  hex = bytes.map(e => e.toString(16).padStart(2, 0)).join('')

  // Convert hex to BigInt
  n = BigInt(`0x`+hex)

  // Shift all numbers to right and xor if the first bit was signed
  n = (n >> 1n) ^ (n & 1n ? -1n : 0n) 

  return n
}

const input = document.querySelector('input')
input.oninput = () => {
  console.clear()
  const bytes = encode(BigInt(input.value))
  // TODO: Save or transmit this bytes
  // new Uint8Array(bytes)
  console.log(bytes.join(','))

  const n = decode(bytes)
  console.log(n.toString(10)+'n') // cuz SO can't render bigints...
}
input.oninput()

<input type="number" value="-39287498324798237498237498273323423" style="width: 100%">

Collectives™ on Stack Overflow

Converting Uint8Array to BigInt in Javascript

2 Answers 2

7 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related