C#, UTF-8 and encoding characters

Question

This is a shot-in-the-dark, and I apologize in advance if this question sounds like the ramblings of a madman.

As part of an integration with a third party, I need to UTF8-encode some string info using C# so I can send it to the target server via multipart form. The problem is that they are rejecting some of my submissions, probably because I'm not encoding their contents correctly.

Right now, I'm trying to figure out how a dash or hyphen -- I can't tell which it is just by looking at it -- is received or interpreted by the target server as ?~@~S (yes, that's a 5-character string and is not your browser glitching out). And unfortunately I don't have a thorough enough understanding of Encoding.UTF8.GetBytes() to know how to use the byte array to begin identifying where the problem might lie.

If anybody can provide any tips or advice, I would greatly appreciate it. So far my only friend has been MSDN, and not much of one at that.

UPDATE 1: After some more digging around, I discovered that using System.Web.HttpUtility.UrlEncode()to encode an EM DASH character ("—") will hex-encode it into "%e2%80%94".

I'm currently sending this info in aHttpWebRequestpost, with a content type of "application/x-www-form-urlencoded" -- could this be what's causing the problem? And if so, what is the proper way to encode a series of name-value pairs whose values may contain Unicode characters, such that it will be understood by a server expecting a UTF-8 request?

From that result I'd guess you might be ascii encoding the result of utf8 encoding an em dash. — Joshua
– Joshua, Commented Jan 29, 2011 at 0:11
Even with the wrong encoding, it's very unlikely that a 1-character dash whould be translated into a 5-characters sequence. It's probably not only an encoding problem. — Simon Mourier
– Simon Mourier, Commented Jan 29, 2011 at 7:37
@Joshua: I think you're close to the heart of the problem. I just added an update to my original post with some more info. — Mass Dot Net
– Mass Dot Net, Commented Mar 16, 2011 at 15:57
Unfortunately I'd have chosen application/octet-stream and just assume it's in the right format on both ends so I'm not going to be able to help you any farther. — Joshua
– Joshua, Commented Mar 16, 2011 at 22:53

Jacob · Accepted Answer · 2011-01-28 23:51:00Z

3

byte[] test = System.Text.Encoding.UTF8.GetBytes("-");

Should give you

test[0] = 0x2D (45 as integer).

Verify that your sending 0x2D to the target server.

edited Jan 28, 2011 at 23:51

Jacob

79k24 gold badges159 silver badges245 bronze badges

answered Jan 28, 2011 at 23:48

Chauncat

2483 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Marlon Over a year ago

Wireshark is helpful for this kind of stuff

Mass Dot Net Over a year ago

I've never used Wireshark before, but our lead developer is very experienced with Fiddler. I'll give this a shot as soon as I'm back in the office -- thank you for the tip.

Chauncat Over a year ago

Wireshark is very simple to setup. It allows you to see what packets are coming in to your sever. You can filter the data in many ways so you don't get lose in the data.

Mass Dot Net Over a year ago

Btw, what is the C# code that properly outputs the information above (test[0] = 0x2D (45 as integer))?

Chauncat Over a year ago

I use this function to convert byte arrays to hex strings public static string BtyeToHexString(byte[] in_record, int startIndex, int count) { string hexString = ""; if (startIndex + count <= in_record.Length) { for (int i = startIndex; i < startIndex + count; i++) { hexString += in_record[i].ToString("X2") + " "; } } return hexString; }

|

Jacob · Accepted Answer · 2011-01-29 07:18:54Z

1

You may need to add a "charset=utf-8" parameter to your Content-Type header. ~~You may also want to have a Content-Encoding header to set your encoding.~~ The headers should contain the following:

Content-Type: multipart/form-data; charset=utf-8

Otherwise, the web server won't know your bytes are UTF-8 bytes, so it will misinterpret them.

edited Jan 29, 2011 at 7:18

answered Jan 28, 2011 at 23:53

Jacob

79k24 gold badges159 silver badges245 bronze badges

2 Comments

Mass Dot Net Over a year ago

You are correct in that I'm not currently explicitly defining a content encoding type when I send the multipart form. I've just sent an email to the third party, asking if they knew what the default expected content type was -- is that something they'd be able to easily identify? I think they're running Microsoft servers (IIS).

dkarp Over a year ago

UTF-8 is not a valid Content-Encoding value. That header is used to indicate how the payload is compressed. It's not used to indicate the charset.

Collectives™ on Stack Overflow

C#, UTF-8 and encoding characters

2 Answers 2

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related