Any workaround to getting a list to numpy array for keras input data?

Question

I'm trying to train a CNN model on keras, my data looks like this

type(datain)
<class 'list'>
len(datain)
35000
type(datain[0])
<class 'numpy.ndarray'>
datain[0].shape
(256,256,1)

And being my input data a list of arrays I get this error when trying to train the network

AttributeError: 'list' object has no attribute 'shape'

but when trying to do something like np.array(datain) as suggested here https://github.com/keras-team/keras/issues/4823 my computer freezes/crashes. defining my input using python list takes like 60 seconds in total, but if I try as numpy arrays from the beginning but takes like 1 sec per (256,256,1) array, and is way too much time if I intent of doing various test and modifications to my network,
is there any work around for this problem?
any way to use lists for keras?
a different way to define a numpy array?
or am I misunderstanding something?

This may be too much for your computer's memory. Work with a generator. — Daniel Möller
– Daniel Möller, Commented Dec 26, 2017 at 14:25
Just to be sure, try first a small amount of data, like np_data = np.array(datain[:100]) and see if the error still occurs. — Daniel Möller
– Daniel Möller, Commented Dec 26, 2017 at 14:28
okay with 100, my computer didnt freeze and my network started to train — Boris
– Boris, Commented Dec 26, 2017 at 14:44

Daniel Möller · Accepted Answer · 2017-12-26 16:02:37Z

4

Creating a generator from your data.

A generator is a python concept, it loops and yields results. For Keras, your generator should yield batches of X_train and y_train indefinitely.

So, a simple generator that you can make is:

def generator(batch_size,from_list_x,from_list_y):

    assert len(from_list_x) == len(from_list_y)
    total_size = len(from_list_x)

    while True #keras generators should be infinite

        for i in range(0,total_size,batch_size):
            yield np.array(from_list_x[i:i+batch_size]), np.array(from_list_y[i:i+batch_size])

Use the generator in training:

model.fit_generator(generator(size,datain,dataout),
                    steps_per_epoch=len(datain)//size, 
                    epochs=...,...)

edited Dec 26, 2017 at 16:02

answered Dec 26, 2017 at 15:54

Daniel Möller

86.8k24 gold badges202 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Any workaround to getting a list to numpy array for keras input data?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related