MicroPython and ESP8266 http server incomplete request

Question

I am trying to implement a simple http server based on the basic socket module of MicroPython which serve a static html and can receive and handle simple http GET and POST requests to save some data on the ESP.

I followed this tutorial https://randomnerdtutorials.com/esp32-esp8266-micropython-web-server/ and changed some parts.

webserver.py

import logging
from socket import socket, getaddrinfo, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR

from request import Request
from request.errors import PayloadError

log = logging.getLogger(__name__)

def __read_static_html(path: str) -> bytes:
  with open(path, "rb") as f:
    static_html = f.read()

  return static_html

def __create_socket(address: str = "0.0.0.0", port: int = 8080) -> socket:
  log.info("creating socket...")
  s = socket(AF_INET, SOCK_STREAM)
  s.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

  addr = getaddrinfo(address, port)[0][-1]
  s.bind(addr)

  log.info("socket bound on {}  ".format(addr))

  return s

def __read_from_connection(conn: socket) -> Request:
  log.info("read from connection...")
  raw_request = conn.recv(4096)

  return Request(raw_request.decode("utf-8"))

def listen_and_serve(webfile: str, address: str, port: int):
  server_socket = __create_socket(address, port)

  log.info("listen on server socket")
  server_socket.listen(5)

  while True:
    # accept connections
    client_server_connection, client_address = server_socket.accept()
    log.info("connection from {}".format(client_address))

    req = __read_from_connection(client_server_connection)

    log.info("got request: {}".format(req.get_method()))

    path = req.get_path()
    if path != '/':
      log.info("invalid path: {}".format(path))
      client_server_connection.send(b"HTTP/1.1 404 Not Found\n")
      client_server_connection.send(b"Connection: close\n\n")
      client_server_connection.close()
      continue

    if req.get_method() == "POST":
      log.info("handle post request")
      try:
        pl = req.get_payload()
        log.debug(pl)
      except PayloadError as e:
        log.warning("error: {}".format(e))
        client_server_connection.send(b"HTTP/1.1 400 Bad Request\n")
        client_server_connection.send(b"Connection: close\n\n")
        client_server_connection.close()
        continue

    log.info("read static html...")
    static_html = __read_static_html(webfile)

    log.info("send header...")
    client_server_connection.send(b"HTTP/1.1 200 OK\n")
    client_server_connection.send(b"Connection: close\n\n")

    log.info("send html...")
    client_server_connection.sendall(static_html)

    log.info("closing client server connection")
    client_server_connection.close()

The request module is my self written http request parser with minimal support for what I need.

The served html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>ESP Basic Configuration</title>
  </head>
  <body>
    <h1>ESP Basic Configuration</h1>
    <form action="/" method="post" enctype="application/x-www-form-urlencoded">
      <h3>Network</h3>
      <div>
        <label for="network.ssid">SSID</label>
        <input id="network.ssid" name="network.ssid" type="text" />
      </div>
      <div>
        <label for="network.password">Password</label>
        <input id="network.password" name="network.password" type="password" />
      </div>
      <button type="submit">Save</button>
    </form>
  </body>
</html>

When I run the code on my system with normale Python3.9 everything seems to work.
Running the code on my ESP8266 the length of the raw_request is truncated to 536 bytes. So some request are incomplete and the payload can not be read.

I have read the socket is non-blocking by default and a short read can occure. I've tried using blocking sockets with timeout. But I got timeouts all the time, when I think there should not be one.

I have tried using the file-like socket object like shown here:
https://docs.micropython.org/en/latest/esp8266/tutorial/network_tcp.html#simple-http-server
But the request reading is stopped after the headers because of the if condition with \r\n.
Removing this condition and just checking with if not line holds the loop on the next line read.

I have currently no idea what I can do to get the full request with payload.

EDIT: add MRE This is the minimal example where I can reproduce the issue:

main.py

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

from socket import socket, AF_INET, SOCK_STREAM, SOL_SOCKET, SO_REUSEADDR

def main():
  server_socket = socket(AF_INET, SOCK_STREAM)
  server_socket.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)

  server_socket.bind(("0.0.0.0", 8080))

  server_socket.listen(5)

  while True:
    # accept connections
    client_server_connection, client_address = server_socket.accept()

    raw_request = client_server_connection.recv(4096)

    print("raw request length: {}".format(len(raw_request)))
    print(raw_request)

    client_server_connection.send(b"HTTP/1.1 200 OK\r\n")
    client_server_connection.send(b"Connection: close\r\n\r\n")

    client_server_connection.sendall(b"""<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>ESP Basic Configuration</title>
  </head>
  <body>
    <h1>ESP Basic Configuration</h1>
    <form action="/" method="post" enctype="application/x-www-form-urlencoded">
      <h3>Network</h3>
      <div>
        <label for="network.ssid">SSID</label>
        <input id="network.ssid" name="network.ssid" type="text" />
      </div>
      <div>
        <label for="network.password">Password</label>
        <input id="network.password" name="network.password" type="password" />
      </div>
      <button type="submit">Save</button>
    </form>
  </body>
</html>
""")

    client_server_connection.close()


if __name__ == "__main__":
  main()

When outputting the raw request with the print statement I get the following info:

raw request length: 536
b'POST / HTTP/1.1\r\nHost: 192.168.0.113:8080\r\nConnection: keep-alive\r\nContent-Length: 39\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nOrigin: http://192.168.0.113:8080\r\nContent-Type: application/x-www-form-urlencoded\r\nUser-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\r\nReferer: http://192.168.0.113:8080/\r\nA'

The request is still truncated to 536 bytes and the request end abruptly.

Could you MRE this? Remove the post code and whittle down to the barest to make the error. If nothing has been posted I'll have a try in the next few days. You're probably just hitting the esp8266 limits though: the esp32 is a much better experience with uPy — 2e0byo
– 2e0byo, Commented Oct 1, 2021 at 21:19
@2e0byo MRE? Sorry I am pretty new here. What does that mean? — joshuai96
– joshuai96, Commented Oct 1, 2021 at 21:21

Steffen Ullrich · Accepted Answer · 2021-10-02 01:05:16Z

3

def __read_from_connection(conn: socket) -> Request:
  log.info("read from connection...")
  raw_request = conn.recv(4096)

You assuming that recv will return all the data you need. This assumption is wrong. TCP has not a concept of messages but is a byte stream, so you need to recv again and again until you got all the data you need. What all data means is defined by the HTTP message format - see the HTTP standard in RFC 7230.

Why the amount of data returned by recv differs between ESP8266 and PC: probably because the ESP8266 has a smaller socket buffer and announces this as window in TCP. This makes the peer send the data in smaller packets.

Apart from that the line ending in the responses must be \r\n not \n.

But the request reading is stopped after the headers because of the if condition with \r\n. Removing this condition and just checking with if not line holds the loop on the next line read.

The empty line (\r\n) marks the end of the HTTP header. To find out the size of the body you need to parse the HTTP header and look for the length information, i.e. Content-length header for a fixed length or Transfer-Encoding: chunked when the body is sent in small chunks and the full length is not known up front. Refer to the standard for the details.

edited Oct 2, 2021 at 1:05

answered Oct 2, 2021 at 0:59

Steffen Ullrich

125k11 gold badges155 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

joshuai96 Over a year ago

I will try to implement reading and parsing the header first and decide to read past \r\n when i expect a body. I will get back when I tried it.

joshuai96 Over a year ago

I finally found some time to test this. 1. Rereading: I figured i'll reread the socket in a loop with smaller chunk sizes until one read returns less then this chunk size. This resulted in one read of 512 Bytes and of 24 Bytes. Resulting in the former 536 Bytes Limit. 2. read past: Then I switched to client_server_connection.makefile("rb") and readline on that, parsing each line and anticipate content-length header and allow to read past the header, if data was send. This results in a hanging connection on readline after the header, until the connection is cancled by client.

Steffen Ullrich Over a year ago

@JoshuaIrm: The current question is answered, i.e. the problems you see with your current code are explained. If you have new code with different problems then please ask a new question and include all necessary details there, i.e. code, debug information, expectations, what happens instead, ... . Don't make your question a moving target.

Collectives™ on Stack Overflow

MicroPython and ESP8266 http server incomplete request

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related