0

I have a client-server code. On the client side, I want to send a list with = 200000 such as:

my_list = [1] * 200000

by socket programming. However, I do not want to split the list. All the lists should be sent at once.

This is my client code:

packedMylist = b''.join(struct.pack('256s', item.encode('utf-8')) for item in my_list)
Msg.append(struct.pack(f'{len(packedMylist)}s', packedMylist))
client.send(Msg)

The server code:

Msg = conn.recv(25600000)
packedMylist = struct.unpack(f'{len(Msg)', Msg)

My_list = [struct.unpack_from('256s', packedMylist, i*256)[0].decode('utf-8').rstrip('\x00') for i in range(200000)]

But it does not work properly. It generates error like this:

struct.error: unpack_from requires a buffer of at least 98048 bytes for unpacking 256 bytes at offset 97792 (actual buffer size is 97984)

Improve this question
4
  • Shouldn't conn.recv(25600000) be 51200000? i.e. 256*200000 Commented Apr 6, 2023 at 5:18
  • @TannerFirl No, the value is a maximum to receive in one call and can (and probably will in this case) be much smaller. Typically 4K is plenty and the receiver should buffer data until the expected data is received. That's the way TCP works, but most beginners don't read the documentation carefully. Commented Apr 6, 2023 at 5:23
  • There's a typo in your recv code: unpack(f'{len(Msg)' Commented Apr 6, 2023 at 5:45
  • In short: you use a stream based transport (TCP) but expect message based semantics. This does not match. While you might not want to split the list of integers, you will need to do with the serialized list since you cannot send+recv it in one go. Commented Apr 6, 2023 at 6:53

1 Answer 1

Reset to default
0

First, decide what the elements of the list are. For struct.pack, s expects a bytes object where each element is a value 0-255. If you have a list of integers, i might be more appropriate (4-byte signed integer).

I'll assume you want integers.

Here's a server:

import socket
import struct

with socket.socket() as s:
    # set up server
    s.bind(('localhost', 5000))
    s.listen()

    # handle one connection at a time
    while True:
        client, addr = s.accept()

        # Wrap the client in a file-like object to buffer the data.
        # With recv(n), n is a *maximum* to receive, but you must check
        # the return value to see what you really get.
        # The file-like wrapper should be checked as well in case the socket
        # closes prematurely, but assuming the data was sent correctly it will
        # buffer data and receive what you request.
        with client, client.makefile('rb') as rfile:
            # multiple messages could be sent
            while True:
                header = rfile.read(4) # start with 4-byte # of elements in list
                if not header:
                    break   # client hung up
                datalen, = struct.unpack('!L', header) # elements in the list
                # list elements assumed to be 4-byte signed integers (type i)
                data = struct.unpack(f'!{datalen}i', rfile.read(datalen * 4))
                print(f'{datalen=} data=[{data[0]}, ..., {data[-1]}]')

And a simple client:

import socket
import struct

with socket.socket() as s:
    s.connect(('localhost', 5000))
    my_list = list(range(200000))  # [0, 1, ..., 199999]
    # pack the number of elements and the entire list.
    data = struct.pack(f'!L{len(my_list)}i', len(my_list), *my_list)
    # Use sendall() to ensure all bytes are sent; otherwise, you must
    # check the return value of send() because it may not send all bytes
    # requested on one send.
    s.sendall(data)

Start the server and run the client a few times.

Server output:

datalen=200000 data=[0, ..., 199999]
datalen=200000 data=[0, ..., 199999]
datalen=200000 data=[0, ..., 199999]
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Could you please modify your code to send a list of 200000 string instead of integer?
That’s a different question and would involve using pickle or json to serialize the string list.
pickle actually works really well with socket.makefile wrappers as it can read and write directly from the stream. Multiple serializations can be written and read using the dumps and loads methods. It does have security issues, however.