Python struct error

Question

I'm trying to design a system to react to different binary flags.

0 = Error
1 = Okay
2 = Logging
3 = Number

The sequence of this data represents a unique ID to reference the work, the flag and the number. Everything works, except the number flag. This is what I get...

>>> import struct
>>> data = (1234, 3, 12345678)
>>> bin = struct.pack('QHL', *data)
>>> print(bin)
b'\xd2\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00Na\xbc\x00\x00\x00\x00\x00'
>>> result = struct.unpack_from('QH', bin, 0)
>>> print(result)
(1234, 3)
>>> offset = struct.calcsize('QH')
>>> result += struct.unpack_from('L', bin, offset)
>>> print(result)
(1234, 3, 7011541669862440960)

A long should be plenty big to represent the number 12345678, but why is it incorrectly unpacked?

Edit:

When I try to pack them separately, it looks like struct is adding too many null bytes between the flag and the long.

>>> import struct
>>> struct.pack('QH', 1234, 3)
b'\xd2\x04\x00\x00\x00\x00\x00\x00\x03\x00'
>>> struct.pack('L', 12345678)
b'Na\xbc\x00\x00\x00\x00\x00'

I can reproduce this error by adding padding before the long.

>>> struct.unpack('L', struct.pack('L', 12345678))
(12345678,)
>>> struct.unpack('xL', struct.pack('xL', 12345678))
(12345678,)
>>> struct.pack('xL', 12345678)
b'\x00\x00\x00\x00\x00\x00\x00\x00Na\xbc\x00\x00\x00\x00\x00'

Potential fix?

When I use little-endian order, the problem seems to correct itself and make the binary string shorter. Since this is destined for a SSL wrapped TCP socket, that's a win win, right? Keeping bandwidth low is generally good, yes?

>>> import struct
>>> data = (1234, 3, 12345678)
>>> bin = struct.pack('<QHL', *data)
>>> print(bin)
b'\xd2\x04\x00\x00\x00\x00\x00\x00\x03\x00Na\xbc\x00'
>>> result = struct.unpack_from('<QH', bin, 0)
>>> print(result)
(1234, 3)
>>> offset = struct.calcsize('<QH')
>>> result += struct.unpack_from('<L', bin, offset)
>>> print(result)
(1234, 3, 12345678)

Why does this happen? I am perplexed.

You aren't concerned that it is unpacking the flag too incorrectly? — Capt Planet
– Capt Planet, Commented Jan 13, 2016 at 8:00
excuse my ignorance: hex(12345678) is 0xbc614e, which looks quite different than Na\xbc\x00\x00\x00\x00\x00. Why is that? What's the meaning of Na? struct.pack('L', 65536*1024) gives \x00\x00\x00\x04\x00\x00\x00\x00, correctly (same order of magnitude as 12345678) — Pynchia
– Pynchia, Commented Jan 13, 2016 at 8:10
@Pynchia Yea, my point exactly! I don't know what is going on. I read somewhere else that this module might be bugged. — bkvaluemeal
– bkvaluemeal, Commented Jan 13, 2016 at 8:12
FYI: bin() is a builtin function, which you hide by overwriting it. — Ulrich Eckhardt
– Ulrich Eckhardt, Commented Jan 13, 2016 at 8:19
@Pynchia hex() returns an integer, while struct returns a byte string. If you take the bytes from 0x00bc614e and align them in the proper way (=reversed) in a byte string like so: b'\x4e\x61\xbc\x00' then you get back b'Na\xbc\x00'. The Na is simply the ASCII-rendered version of \x4e\x61. — poke
– poke, Commented Jan 13, 2016 at 8:28

poke · Accepted Answer · 2016-01-13 08:20:49Z

7

You are running into byte alignment issues. You need to know that by default the individual parts of a struct are not just placed next to each other but they are properly aligned in memory. This makes it more efficient, especially for other applications, as they have more direct way to access individual bytes from it without having to account for overlap.

You can easily see this by using struct.calcsize to see the required space needed to encode using a format:

>>> struct.calcsize('QHL')
16
>>> struct.calcsize('QH')
10

As you can see QHL requires 16 bytes, but QH requires 10. The L we left off is however only 4 bytes wide. So there is some padding going to on make sure that the L starts again on “a fresh block”. This is because any type requires (with padding) that it starts on a offset that is a multiple of its own size. For QH it looks like this:

QQ QQ | QQ QQ | HH

Once you use QHL, you get the following:

QQ QQ | QQ QQ | HH 00 | LL LL

As you can see, there were two padding bytes added to make sure that L starts on a new block of four.

You can modify the alignment (as well as the endianness) using a special character at the beginning of the format string. In your case, you could use =QHL to disable alignment altogether:

QQ QQ | QQ QQ | HH LL | LL

When I use little-endian order, the problem seems to correct itself and make the binary string shorter. Since this is destined for a SSL wrapped TCP socket, that's a win win, right? Keeping bandwidth low is generally good, yes?

Using an explicit byte order also disables alignment yes, so that’s where the effect comes from. If it’s a good idea to turn of alignment depends though. If you want to use consume your data somewhere else, in other programs, it would be a good idea to stick to native alignment.

answered Jan 13, 2016 at 8:20

poke

392k80 gold badges596 silver badges632 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bkvaluemeal Over a year ago

This data will be packed on one computer with Python and unpacked on another using Python as well. Does that man alignment isn't as important?

poke Over a year ago

As long as you’re using the same platform-independent format on all machines, then yes, disabling alignment is fine.

Capt Planet · Accepted Answer · 2016-01-13 08:25:13Z

1

The correct output in your case:

>>> import struct
>>> data = (1234, 3, 12345678)
>>> bin = struct.pack('QHL', *data)
>>> print(bin)
b'\xd2\x04\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00Na\xbc\x00\x00\x00\x00\x00'
>>> result = struct.unpack_from('QH', bin, 0)
>>> print(result)
(1234, 3)
>>> result += struct.unpack_from('L', bin, 16)
>>> print(result)
(1234, 3, 12345678)

This happens because:

Padding is only automatically added between successive structure members.

Also, the reason your fix works is:

No padding is added when using non-native size and alignment, e.g. with ‘<’, ‘>’, ‘=’, and ‘!’.

answered Jan 13, 2016 at 8:25

Capt Planet

1136 bronze badges

Comments

giograno · Accepted Answer · 2016-01-13 08:29:51Z

1

It is a byte alignment issue. In your case the following replacement will give you the correct output.

result += struct.unpack_from('L', bin, offset+2)

answered Jan 13, 2016 at 8:29

giograno

1,8294 gold badges21 silver badges30 bronze badges

Collectives™ on Stack Overflow

Python struct error

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related