In Python, how can I convert both numbers and strings into byte arrays?

Question

I want to encode a set of configuration options into a long string of hex digits.

The input is a mix of numbers (integers and floats) and strings. I can use binascii.a2b_hex from the standard library for the strings, bit-wise operators for the integers, and probably, if I go and read some on floating point representation (sigh), I can probably handle the floats, too.

Now, my questions:

When given the list of options, (how) should I type check the value to select the correct conversion routine?
Isn't there a library function for the numbers, too? I can't seem to find it.

The serialized data is sent to an embedded device and I have limited control over the code that consumes it (meaning, changes are possible, but a hassle). The specification for the serialization seems to conform to C value representation (char arrays for strings, Little Endian integers, IEEE 754 for floats), but it doesn't explicitily state this. So, Python-specific stuff like pickle are off-limits.

Ignacio Vazquez-Abrams · Accepted Answer · 2011-02-22 23:34:38Z

4

You want struct.

>>> struct.pack('16sdl', 'Hello, world!', 3.141592654, 42)
'Hello, world!\x00\x00\x00PERT\xfb!\t@*\x00\x00\x00\x00\x00\x00\x00'

answered Feb 22, 2011 at 23:34

Ignacio Vazquez-Abrams

804k160 gold badges1.4k silver badges1.4k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Hanno Fietz Over a year ago

Yes, I do. Thanks. :) (Accepted Adam's answer, but upvoted yours, too)

Sven Marnach · Accepted Answer · 2011-02-22 23:13:56Z

2

You easiest bet is to pickle the whole list to a string and then use binascii.a2b_hex() to convert this string to hex digits:

a = ["Hello", 42, 3.1415]
s = binascii.b2a_hex(pickle.dumps(a, 2))
print s
# 80025d710028550548656c6c6f71014b2a47400921cac083126f652e
print pickle.loads(binascii.a2b_hex(s))
# ['Hello', 42, 3.1415]

edited Feb 22, 2011 at 23:13

answered Feb 22, 2011 at 23:07

Sven Marnach

608k123 gold badges966 silver badges865 bronze badges

1 Comment

Hanno Fietz Over a year ago

Sorry, can't use pickle, the consumer of the serialization is an embedded device programmed in C, so I have a specific contract for the encoding.

Hanno Fietz · Accepted Answer · 2011-02-22 23:55:37Z

1

What about using the struct module to do your packing/unpacking?

import struct
s = struct.pack('S5if',"Hello",42,3.1415)
print s
print struct.unpack('5sif')

or if you really want just hex characters

import struct, binascii
s = binascii.b2a_hex(struct.pack('S5if',"Hello",42,3.1415))
print s
print struct.unpack('5sif',binascii.a2b_hex(s))

Of course this requires that you know the length of strings that are being sent across or you could figure it out by looking for a NULL character or something.

edited Feb 22, 2011 at 23:55

Hanno Fietz

31.5k49 gold badges157 silver badges239 bronze badges

answered Feb 22, 2011 at 23:45

Adam A.

5121 gold badge4 silver badges13 bronze badges

1 Comment

Hanno Fietz Over a year ago

The strings are serialized with a fixed length (truncated or padded with null bytes), so that fits well with struct.

Collectives™ on Stack Overflow

In Python, how can I convert both numbers and strings into byte arrays?

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related