Making sense of Apache HttpClient wire log from response

Question

My application is using Apache HTTPClient 4.3.5 that sends HTTP request and gets response.

I am trying to figure out what response the app received.

Below is the log snippet -

[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "POST /app HTTP/1.1[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "User-Agent: Mozilla/5.0[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Content-Length: 47[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Content-Type: application/x-www-form-urlencoded[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Host: test.com[\r][\n]"
[Jan 04 2015 05:38:14.109 AM] DEBUG wire:72 - http-outgoing-2 >> "Connection: Keep-Alive[\r][\n]"
[Jan 04 2015 05:38:14.110 AM] DEBUG wire:72 - http-outgoing-2 >> "Accept-Encoding: gzip,deflate[\r][\n]"
[Jan 04 2015 05:38:14.110 AM] DEBUG wire:72 - http-outgoing-2 >> "[\r][\n]"
[Jan 04 2015 05:38:14.489 AM] DEBUG wire:72 - http-outgoing-2 << "HTTP/1.1 200 OK[\r][\n]"
[Jan 04 2015 05:38:14.489 AM] DEBUG wire:72 - http-outgoing-2 << "Server: nginx[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Date: Sun, 04 Jan 2015 10:37:31 GMT[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Content-Type: text/html;charset=UTF-8[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Transfer-Encoding: chunked[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Connection: keep-alive[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Vary: Accept-Encoding[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Pragma: no-cache[\r][\n]"
[Jan 04 2015 05:38:14.490 AM] DEBUG wire:72 - http-outgoing-2 << "Expires: Thu, 01 Jan 1970 00:00:00 GMT[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Cache-Control: no-cache[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Cache-Control: no-store[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Set-Cookie: JSESSIONID=0C1444D7E0FA6B0F3CCE20AFBA28237E; Path=/; HttpOnly[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Front-End-Https: on[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "Content-Encoding: gzip[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "[\r][\n]"
[Jan 04 2015 05:38:14.491 AM] DEBUG wire:72 - http-outgoing-2 << "37[\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "[0x1f][0x8b][0x8][0x0][0x0][0x0][0x0][0x0][0x0][0x3][0xb3])J-.[0xc8][0xcf]+N[0xb5][0xe3][0xb2][0x81]1[0xe3][0x93][0xf3]S[0x80][0xfc][0xe0][0xd2][0xe4][0xe4][0xd4][0xe2]b.[0x1b]}4[0x9][0x84][0x80][0x1d][0x17][0x0]<Xn]@[0x0][0x0][0x0][\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "0[\r][\n]"
[Jan 04 2015 05:38:14.492 AM] DEBUG wire:72 - http-outgoing-2 << "[\r][\n]"

At first, I thought "[0x1f][0x8b][0x8][0x0][0x0]..." is hex string but there are non-hex string such as ")J-." and "+N." and each debug line is ending with "[\r][\n]". How would I be able to determine what response the app received.

I have spent quite few hours trying to find the answer but was unsuccessful.

Does it have to do anything with gzip compression or chunked transfer encoding?

Your help would be greatly appreciated if you could answer it.

Thanks,
James

ok2c · Accepted Answer · 2015-01-07 14:40:22Z

3

HttpClient wire log replaces character 10 with [\n], character 13 with [\r], characters < 32 and > 127 (non-printable) with [0xN] where N is char's hexadecimal representation.

The reason you are seeing a lot of non-printable chars is indeed GZIP content encoding.

answered Jan 7, 2015 at 14:40

ok2c

27.9k5 gold badges66 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mikhail Boyarsky · Accepted Answer · 2021-11-15 14:22:22Z

2

http://moznion.github.io/PrettyWireLogViewer/

This tool decodes ugly httpclient format to human-readable.

answered Nov 15, 2021 at 14:22

Mikhail Boyarsky

3,0683 gold badges28 silver badges38 bronze badges

1 Comment

Homayoun Behzadian Over a year ago

Very nice tool! it's very weird that StackOverflow does not have bookmark feature. I definitely want to bookmark this answer

Simon · Accepted Answer · 2021-09-30 11:40:15Z

This is an old post, but I've just spent too much time looking for code to take such a log entry and decode/unzip it to a readable string. So here's my contribution:

public void parseHexReponse() throws IOException {
    // make sure NOT to delete any spaces which are part of the payload!
    final String responseHex = "[0x1f]...[0x0]"; // paste your response here
    final String[] parts = responseHex.split("[\\[\\]]+");
    final byte[] data = new byte[responseHex.length() / 3];
    int idx = 0;
    for (final String s : parts) {
        if (s.startsWith("0x")) {
            data[idx] = (byte) Integer.parseInt(s.substring(2), 16);
            idx++;
        } else if ("\\n".equals(s)) {
            data[idx] = 10;
            idx++;
        } else if ("\\r".equals(s)) {
            data[idx] = 13;
            idx++;
        } else {
            for (final byte b : s.getBytes()) {
                data[idx] = b;
                idx++;
            }
        }
    }
    try (GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(Arrays.copyOfRange(data, 0, idx)));
        ByteArrayOutputStream out = new ByteArrayOutputStream()) {
        final byte[] buffer = new byte[1024];
        int len;
        while ((len = gis.read(buffer)) > 0) {
            out.write(buffer, 0, len);
        }
        System.out.println(new String(out.toByteArray()));
    }
}

Collectives™ on Stack Overflow

Making sense of Apache HttpClient wire log from response

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related