What method for live sending Python numpy data to Unity 3D?

You can simply do this : “Take picture bytes stream from RenderTexture (Unity) and send it to python server on websockets” A sample code you can use for connecting to websockets:

void Start() {
      ws = WebSocketFactory.CreateInstance("ws://localhost:12345");
      // ...
      
      // Connect to server 
      ws.connect();
   
     InvokeRepeating("Capture", 1f, 0.2f);
 }

 public void Capture() {
     RenderTexture activeRenderTexture = RenderTexture.active;
     RenderTexture.active = Camera.targetTexture;
     
     Camera.Render();
 
    Texture2D image = new Texture2D(Camera.targetTexture.width, Camera.targetTexture.height);
    image.ReadyPixels(new Rect(0, 0, Camera.targetTexture.width, CameraTexture.height), 0, 0);
    image.Apply();
    RenderTexture.active = activeRenderTexture;

    byte[] bytes = image.EncodeToPNG();
    Destroy(image);

    ws.send(bytes);
}

some important notes:

1- we encode images to PNG format and send them in this sample. you can do it or not, it's your choice!

2- In this solution, you are running everything on main thread in unity.Thus, it may lag and your program's frame rate may be low. also , the way you encode it is important (PNG encoding is huge!).

3- If this solution didn't work, you can also try streaming with FMETP STREAM maybe.

For sending your numpy array from python and receive it in unity, there are multiple ways. One of the simplest ways is that you convert your numpy array to string and send it through socket to unity. Then, convert it back to float in unity. A sample for sending string:

# python code
import socket 

UDP_IP = "127.0.0.1"
UDP_PORT = 5050

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

msg = "sample string!"

sock.sendto(msg, (UDP_IP, UDP_PORT))

//C# code
IPEndPoint anyIP = new 

IPEndPoint(IPAddress.Parse("0.0.0.0"), port);

byte[] data = client.Receive(ref anyIP); 

string text = Encoding.UTF8.GetString(data);

Another way that may be a better approach (but I didn’t try it myself) is you can serialize your numpy python array. Since ndarray is a binary construct, you can use ndarray.tobytes() and then deserialize it in C#.

But for deserialization, pay attention to your numpy data types(int, float, etc.) and Endianness of your machine (little endian/ big endian)

An example for deserializing in C# (an array of bytes to array of doubles):

Buffer.BlockCopy();

You can see this question for more info.