ASP.NET Core HtmlAgilityPack Encoding errors

Question

There are some posts regarding encoding questions and HtmlAgilityPack but this issue wasn't addressed:

Because the website I try to parse contains Unicode symbols like € or ä, ü I tried to set the encoding to Unicode:

public class WebpageDeserializer
{
    public WebpageDeserializer() {}

    /*
     * Example address: https://www.dslr-forum.de/showthread.php?t=1930368
    */
    public static void Deserialize(string address)
    {
        var web = new HtmlWeb();
        web.OverrideEncoding = Encoding.Unicode;
        var htmlDoc = web.Load(address);
        //further decoding fails because unicode decoded characters are not proper html (looks more like chinese)
    }
}

But now

htmlDoc.DocumentNode.InnerHtml

looks like this:

ℼ佄呃偙⁅瑨汭倠䉕䥌⁃ⴢ⼯㍗⽃䐯䑔堠呈䱍ㄠ〮吠慲獮瑩潩慮⽬䔯≎...

If I try to use UTF-8 or iso-8859-1 the € symbol is converted to � (as well as ä, ö, ü). How can I fix this?

I updated my code example - I hope it contains everything you need. If not - please don't hesitate to ask for more information. — binaryBigInt
– binaryBigInt, Commented Dec 8, 2018 at 21:57
@FalcoAlexander than use other tools. Why do you think we should be interested in the tools you are using. — L.B
– L.B, Commented Dec 8, 2018 at 22:13
@FalcoAlexander Yes, I use Visual studio and write codes to test the questions :) — L.B
– L.B, Commented Dec 8, 2018 at 22:22
also tested with RoslynPad and can reproduce the strange behaviour with huge "chinese" response that freezes the .NET host — Falco Alexander
– Falco Alexander, Commented Dec 8, 2018 at 22:25

L.B · Accepted Answer · 2018-12-08 22:37:12Z

1

Your site is mis-configured and the real encoding is cp1252.

Below code should work:

var client = new HttpClient();
var buf = await client.GetByteArrayAsync("https://www.dslr-forum.de/showthread.php?t=1930368");
var html = Encoding.GetEncoding(1252).GetString(buf);
var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);

answered Dec 8, 2018 at 22:37

L.B

116k20 gold badges189 silver badges229 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

binaryBigInt Over a year ago

Thanks for your reply. Encoding.GetEncoding(1252); gives me a System.NotSupportedException. Do I have to configure something to get this encoding? I am using .NET Core 2.1 and Windows 10 64-bit. Edit: This fixed it: stackoverflow.com/questions/37870084/… Thanks alot!

Falco Alexander · Accepted Answer · 2018-12-08 22:45:09Z

0

instead Encoding.Unicode use:

web.OverrideEncoding = Encoding.GetEncoding("iso-8859-1");

(tested with your website and german umlauts)

to get the right encoding check the header of the target website. it contains the right hint:

<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">

edited Dec 8, 2018 at 22:45

answered Dec 8, 2018 at 22:37

Falco Alexander

3,4222 gold badges26 silver badges45 bronze badges

1 Comment

binaryBigInt Over a year ago

This didn't work for me because the € sign or ä ü are just removed.

Collectives™ on Stack Overflow

ASP.NET Core HtmlAgilityPack Encoding errors

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related