Fast way to send value from Raspberry (Python) to Arduino over serial?

I want to send a int value (0-640) from Raspberry to Arduino via RX/TX serial port (not USB).

For the moment I receive the value on the Raspberry over UDP, therefore the received type is "class 'bytes' ". I am converting the bytes to int, encode it to a string and sends it to Arduino.

My problem is that it is extremely slow to convert from ASCII to int on Arduino side. Best case scenario would be, of course, to have something simple like:

Raspberry (dream scenario):

serial.write(data)

Arduino (dream scenario):

RPiMsg = Serial1.read();

Any suggestions? Questions?

My code so far:

Raspberry code:

import struct
import serial

serial = serial.Serial('/dev/ttyAMA0', 115200)

while True:
    # (This is here to show you where it comes from):
    data, addr = sock.recvfrom(4)

    if len(data)!=4:
        continue
    else:
        # Convert data to int
        msg = struct.unpack('i',data)[0]
        # Send to Arduino
        serial.write(str(msg).encode() + str('\n').encode())

Arduino code:

void setup()
{
  // Opens serial ports, sets data rate
  Serial1.begin(115200); // Serial1 = reads from Pi
}

void loop()
{
  if (Serial1.available())
  {
    // Resets RPiMsg
    RPiMsg = 0;

    // Read serial one byte at a time and convert ASCII to int
    while(true)
    {
      incomingByte = Serial1.read();

      // Exit while-loop, end of message
      if (incomingByte == '\n') break;

      // If nothing is in the buffer, Serial1.read() = -1
      if (incomingByte == -1) continue;

      // Shift 1 decimal place left
      RPiMsg *= 10;

      // Convert ASCII to int
      RPiMsg = ((incomingByte - 48) + RPiMsg);
    }

    Serial.println(RPiMsg);
  }
}