Encoding.UTF8.GetBytes seems to be inserting a spurious byte

Question

I have the following code, converted to C# from an old VB6 program. The VB6 had used the old Winsock, which could accept a String argument, but the C# program uses System.Net.Socket which wants a byte array.

byte[] msg = Encoding.UTF8.GetBytes(tempString); 
_TCPConn.Send(msg);

tempString has

0x0002 (' ')
0x0000 ('\0')
0x0000 ('\0')
0x0000 ('\0')
0x0080 (' ')
0x006d ('m')
0x0068 ('h')

But msg gets an extra byte

0x02 
0x00
0x00
0x00
**0xc2**
0x80
0x6d
0x68

Where is that "c2" coming from?

What is the receiver expecting? An ANSI string or Unicode string? — Jim Rhodes
– Jim Rhodes, Commented Jan 7, 2013 at 18:48
VB6 allowed storing bytes in a string but those days are over. In particular Unicode normalization can randomly destroy the content, before you even get to converting it back to bytes. You'll need to fix this problem at the core and stop using a string. — Hans Passant
– Hans Passant, Commented Jan 7, 2013 at 18:55

Jim Rhodes · Accepted Answer · 2013-01-07 19:19:14Z

2

That is what UTF8 does. Values from 0x80 to 0x7FF get encoded with 2 bytes. Values from 0x800 to 0xFFFF get encoded with 3 bytes. 0xC2 0x80 tells the decoder to output just 0x80.

Edit: If the receiver is only expecting the low byte of each character and character values 0x80-0xFF are valid, you will have to convert each character one at a time.

int len = tempString.Length;
byte[] msg = new byte[len];
for ( int idx = 0; idx < len; ++idx )
{
  msg[idx] = (byte) tempString[idx];
}

edited Jan 7, 2013 at 19:19

answered Jan 7, 2013 at 18:35

Jim Rhodes

5,1044 gold badges27 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Ilya Ivanov Over a year ago

I've used Encoding.UTF8.GetBytes(" \0\0\0 mh") and printed 7 bytes without 0xc2. Am I missing something?

user316117 Over a year ago

Yikes! So how can I just convert my string to a byte array?

Jim Rhodes Over a year ago

Is every character in tempString guaranteed to be 0x0000 - 0x00FF?

user316117 Over a year ago

Yes, guaranteed - the device it's talking to is a piece of industrial machinery also manufactured by this company that assumes strictly 8-bit ASCII characters. I've discussed with management the long term consequences of assuming 8-bit characters versus the amount of work it would take to rip out all the old VB6 logic and make something more up to date, and they are satisfied to stay 8-bit for this part of the product.

Wolf · Accepted Answer · 2013-01-07 18:39:49Z

0

This is done by UTF8 encoding itself. This is fine.

Then you can use UTF8.GetString(Byte[]) method to decode it.

answered Jan 7, 2013 at 18:39

Wolf

2,1351 gold badge15 silver badges11 bronze badges

Collectives™ on Stack Overflow

Encoding.UTF8.GetBytes seems to be inserting a spurious byte

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related