Python pySerial read data from arduino breaks when sending "(char)0"

Question

I send some data from an arduino using pySerial.

My Data looks like

bytearray(DST, SRC, STATUS, TYPE, CHANNEL, DATA..., SIZEOFDATA)

where sizeofData is a test that all bytes are received.

The problem is, every time when a byte is zero, my python program just stops reading there:

serial_port = serial.Serial("/dev/ttyUSB0")
while serial_port.isOpen():
  response_header_str = serial_port.readline()
  format = '>';
  format += ('B'*len(response_header_str));
  response_header = struct.unpack(format, response_header_str)
  pprint(response_header)
serial_port.close()

For example, when I send bytearray(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15) everything is fine. But when I send something like bytearray(1,2,3,4,0,1,2,3,4) I don't see everything beginning with the zero.

The problem is that I cannot avoid sending zeros as I am just sending the "memory dump" e.g. when I send a float value, there might be zero bytes.

how can I tell pyserial not to ignore zero bytes.

Brett Lempereur · Accepted Answer · 2014-09-19 15:18:30Z

1

I've looked through the source of PySerial and the problem is in PySerial's implementation of FileLike.readline (in http://svn.code.sf.net/p/pyserial/code/trunk/pyserial/serial/serialutil.py). The offending function is:

def readline(self, size=None, eol=LF):
    """\
    Read a line which is terminated with end-of-line (eol) character
    ('\n' by default) or until timeout.
    """
    leneol = len(eol)
    line = bytearray()
    while True:
        c = self.read(1)
        if c:
            line += c
            if line[-leneol:] == eol:
                break
            if size is not None and len(line) >= size:
                break
        else:
            break
    return bytes(line)

With the obvious problem being the if c: line. When c == b'\x00' this evaluates to false, and the routine breaks out of the read loop. The easiest thing to do would be to reimplement this yourself as something like:

def readline(port, size=None, eol="\n"):
    """\
    Read a line which is terminated with end-of-line (eol) character
    ('\n' by default) or until timeout.
    """
    leneol = len(eol)
    line = bytearray()
    while True:
        line += port.read(1)
        if line[-leneol:] == eol:
            break
        if size is not None and len(line) >= size:
            break
    return bytes(line)

To clarify from your comments, this is a replacement for the Serial.readline method that will consume null-bytes and add them to the returned string until it hits the eol character, which we define here as "\n".

An example of using the new method, with a file-object substituted for the socket:

>>> # Create some example data terminated by a newline containing nulls.
>>> handle = open("test.dat", "wb")
>>> handle.write(b"hell\x00o, w\x00rld\n")
>>> handle.close()
>>>
>>> # Use our readline method to read it back in.
>>> handle = open("test.dat", "rb") 
>>> readline(handle)
'hell\x00o, w\x00rld\n'

Hopefully this makes a little more sense.

edited Sep 19, 2014 at 15:18

answered Sep 19, 2014 at 12:55

Brett Lempereur

8155 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Tobi Over a year ago

but when I change the character to another one, I just shifted the problem to another bit-sequence...

Brett Lempereur Over a year ago

Change the character? The function I posted will happily read b'\x00' and append it to the byte array. There is no need to modify the output from your Arduino, only the way you read from the serial port in Python.

Tobi Over a year ago

yeah, but "\n" has a byte representation also. So I shift the problem only to this representation which is 10 in decimal.

Brett Lempereur Over a year ago

One of us isn't quite understanding the other here. There was a slight typo in my solution (I referred to LF which is a constant defined within PySerial as equal to \n, I've changed this to \n in my code). I've also updated my solution with a representative example (switching the serial port for a file handle, but it's functionally the same).

Brett Lempereur Over a year ago

Ah, I see your problem now. The easiest thing to do would be to change how you send the data entirely. Compute the length of the data you're going to send on your Arduino and send that first (packed as a 16-bit integer). Then, allocate a buffer that size in your Python script and read in exactly that amount of data.

Collectives™ on Stack Overflow

Python pySerial read data from arduino breaks when sending "(char)0"

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related