Loading numpy array from http response without saving a file

Question

I have a bunch of files containing numpy arrays at some url (e.g., https://my_url/my_np_file.npy) and I am trying to load them in my computer.

If I download the file manually, I can properly load the numpy array using np.load('file_path'). If I take the url reponse (using the code below), save the content to a file and then use np.load(), it also works.

response, content = http.request('https://my_url/my_np_file.npy')

If I try to load the array from the content string, I get the error bellow. This is likely because the np.load is interpreting the string input as a name for the file, rather than the data itself.

File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 370, in load fid = open(file, "rb") TypeError: file() argument 1 must be encoded string without null bytes, not str

Is there any way to load the array without having to save a file?

LucasB · Accepted Answer · 2020-05-10 18:24:35Z

14

You are missing io.BytesIO to make the string appear like a file object to np.load!

The following is what you're looking for:

import requests
import io

response = requests.get('https://my_url/my_np_file.npy')
response.raise_for_status()
data = np.load(io.BytesIO(response.content))  # Works!

answered May 10, 2020 at 18:24

LucasB

3,6231 gold badge31 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Rocky Li · Accepted Answer · 2018-10-19 01:23:30Z

0

I just worked out a workaround using shutil

import requests
import shutil
response = requests.get('http://path/to/test.npy', stream=True)

with open('haha.npy', 'wb') as fin:
    shutil.copyfileobj(response.raw, fin)

np.load('haha.npy') # Works!

This will first download the file itself though, but in an automatic fashion.

answered Oct 19, 2018 at 1:23

Rocky Li

5,9862 gold badges21 silver badges36 bronze badges

Comments

Henry Kim · Accepted Answer · 2023-12-10 05:55:23Z

-1

Can you add an example of the HTTP response? (from https://my_url/my_np_file.npy)

Maybe you could try np.fromstring after you retrieve the web response.

This is an example from the above reference.

>>> np.fromstring('1 2', dtype=int, sep=' ')
array([1, 2])
>>> np.fromstring('1, 2', dtype=int, sep=',')
array([1, 2])

edited Dec 10, 2023 at 5:55

answered Oct 19, 2018 at 1:13

Henry Kim

5855 silver badges20 bronze badges

Collectives™ on Stack Overflow

Loading numpy array from http response without saving a file

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related