Extraneous character while converting string to buffer and back in javascript

Question

In node v16.14.2,

> String.fromCharCode.apply(null, Buffer.from('°'))
'Â°'

This works fine though:

> Buffer.from('°').toString('utf-8')
'°'

I want to know why the first scenario adds an extra character in the output.

Browser example:

console.log(String.fromCharCode.apply(null, ethereumjs.Buffer.Buffer.from('°'))) /// 'Â°'
console.log(ethereumjs.Buffer.Buffer.from('°').toString('utf-8')) // '°'
console.log(new Uint8Array(ethereumjs.Buffer.Buffer.from('°'))) // [ 194, 176 ]
console.log(new Uint8Array(ethereumjs.Buffer.Buffer.from(String.fromCharCode(176)))) // [ 194, 176 ]

<script src="https://cdn.jsdelivr.net/gh/ethereumjs/browser-builds/dist/ethereumjs-tx/ethereumjs-tx-1.3.3.min.js"></script>

EDIT: Another funny thing:

> new Uint8Array(Buffer.from('°'))
Uint8Array(2) [ 194, 176 ]
> new Uint8Array(Buffer.from(String.fromCharCode(176)))
Uint8Array(2) [ 194, 176 ]

The static String.fromCharCode() method returns a string created from the specified sequence of UTF-16 code units. — mplungjan
– mplungjan, Commented Aug 23, 2022 at 10:00
The code unit of '°' is 176. I do not know the implementation details of UTF-16, but it should have same code unit value for both UTF-8 and UTF-16 — Dipu
– Dipu, Commented Aug 23, 2022 at 11:14

MikeM · Accepted Answer · 2022-08-23 16:20:55Z

2

The default string encoding of Buffer.from is UTF-8 which encodes characters in the Extended ASCII range using two bytes. (It can't just use one byte because the highest order bit of that byte is used to signal multibyte encoding.)

UTF-8 two-byte encoding has the form:

110xxxxx 10xxxxxx

which gives 11 bits to represent a character code.

The ASCII code of '°' is 176

> '°'.charCodeAt(0)
176

which in binary is:

> (176).toString(2)
'10110000'

So replacing the x with this binary value we have:

110xxxxx 10xxxxxx
110xxx10 10110000
      ^^   ^^^^^^

which means the UTF-8 two-byte encoding of '°' is

11000010 10110000

or

194 176

as expected:

> [...Buffer.from('°')]
[ 194, 176 ]

> String.fromCharCode(194)
'Â'

> String.fromCharCode(176)
'°'

edited Aug 23, 2022 at 16:20

answered Aug 23, 2022 at 11:25

MikeM

13.8k3 gold badges38 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Extraneous character while converting string to buffer and back in javascript

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related