0

I am trying to create 3 tensors for my language translation LSTM network.

import numpy as np
Num_samples=50000
Time_step=100
Vocabulary=5000
shape = (Num_samples,Time_step,Vocabulary)
encoder_input_data = np.zeros(shape,dtype='float32')
decoder_input_data = np.zeros(shape,dtype='float32')
decoder_target_data = np.zeros(shape,dtype='float32')

Obviously, my machine doesn't have enough memory to do so. Since the data is represented as one-hot vectors, it seems using the function csc_matrix() from scipy.sparse will be the solution, as suggested in this tread and this tread.

But after trying the csc_matrix() and crc_matrix(), it seems they only support 2D array.

Old treads from 6 years ago did talk about this issue, but they are not machine learning orientated.

My question is: Is there any python lib/tool that can help me to create sparse 3D arrays that allows me to store one-hot vectors for machine learning purpose later?

Improve this question
9
  • scipy.sparse matrices are used in learning, such as in the sklearn package. But, no they have not been expanded to 3d. Commented May 21, 2018 at 4:00
  • If it is the case, is there a way to train my network with the scipy.sparse matrices? Since all of my machine learning experience are creating 3D array in the very beginning which contains batch, time_step, and length of the one-hot vector. I am open to new ways to deal with the training data. Commented May 21, 2018 at 4:24
  • I suddenly have another thought in mind (Which is not directly related to this problem). How strong is the machine that Google use to train their translation deep network? Obviously, 50000 training examples are a not huge data size at all, yet, it takes 23Gb memory for each array already. I would expect Google would use a even larger training examples, say 5 millions training examples. By using the same calcualtio, they would need 2000Gb RAM. Is such a machine even exist? Commented May 21, 2018 at 4:41
  • @RavenCheuk the sorts of models trained by giant companies are usually trained on large, distributed computing clusters, not a single machine. And probably significantly more training data than 5 million examples Commented May 21, 2018 at 5:27
  • But yes, you can nowadays buy a machine with 2000GB of ram anyway. Commented May 21, 2018 at 5:27
Related questions
1
Large, sparse list of lists giving MemoryError when calling np.array(data)
12
Numpy memory error creating huge matrix
0
Getting numpy Memory Error on (relatively) small matrix
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.