C# Converting a string to bytes and then back to a string with default encoder mangles the string

Question

I am troubleshooting a strange issue reported by a client which is caused by the application trying to parse invalid XML. I believe the root cause to be related to how the XML string is encoded and then decoded. I have an internal API that gets the XML string (which I know to be valid to begin with), then converts it to a byte array and wraps it with a readonly MemoryStream. Then on the other side, the stream is converted back to a string and then passed to XDocument.Parse(string). The latter call fails, saying "Data at the root level is invalid. Line 1, position 1." Anyway, I believe the root cause has to do with how I am encoding and then decoding the string. In fact, the following line of debugging code returns a different string than what was passed in.

Encoding.Default.GetString(Encoding.Default.GetBytes(GetMeAnXmlString())));

Using Encoding.Default on the way in and then back out yields a different string than what I started with. That's craaaazy. Any ideas?

Note:

I am using an API which I cannot change which retrieves the stream containing the XML, so I cannot alter the use of Encoding.Default. Doing so will risk production issues (a.k.a showstoppers) for clients where everything is working fine.

Encoding.Default will be "the operating system's current ANSI code page", which will not be able to encode all unicode characters. If code outside your control has already encoded the XML with Encoding.Default, and if there were such unrepresentable characters, then you're out of luck - the information is already lost. — Weeble
– Weeble, Commented Nov 12, 2013 at 22:42

trope · Accepted Answer · 2013-11-12 22:45:00Z

7

The long and short of it is that Encoding.Default is sketchy because of the code page aspect that Weeble mentioned.

http://msdn.microsoft.com/en-us/library/system.text.encoding.default%28v=vs.110%29.aspx and http://blogs.msdn.com/b/shawnste/archive/2005/03/15/don-t-use-encoding-default.aspx

You'd likely be better off just deciding to use Encoding.Unicode or Encoding.UTF8.

answered Nov 12, 2013 at 22:45

trope

1814 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

C# Converting a string to bytes and then back to a string with default encoder mangles the string

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related