I'm trying to make a machine learning model in keras that guesses the next word, given a series of words using a LSTM. This is the code for my model:
gen=Sequential()
gen.add(Embedding(vocab_size,embed_dim))
gen.add(LSTM(vocab_size,activation='softmax'))
gen.compile(loss='categorical_crossentropy',optimizer='adam')
gen.summary()
(X_train, X_test, Y_train, Y_test)=train_test_split(X,Y,test_size=.2)
gen.fit(X_train,Y_train,validation_data=(X_test,Y_test),batch_size=batch_size,epochs=epochs)
My X data is an array of padded arrays of integers, like this: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 59, 73, 4] and my Y is a one hot vector with length of my vocab size (which for this case is 1994). Running the above code creates this output:
2020-08-23 11:19:34.390012: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not
load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
2020-08-23 11:19:34.391222: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-08-23 11:19:34.405184: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: LAPTOP-J9AD5M4U
2020-08-23 11:19:34.406474: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: LAPTOP-J9AD5M4U
2020-08-23 11:19:34.407916: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2020-08-23 11:19:34.695283: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x20c4fc4a980 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-23 11:19:34.696499: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (None, None, 32) 75040
_________________________________________________________________
lstm (LSTM) (None, 2345) 22305640
=================================================================
Total params: 22,380,680
Trainable params: 22,380,680
Non-trainable params: 0
_________________________________________________________________
2020-08-23 11:20:30.730043: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:30.943137: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:31.469228: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:32.787365: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:34.014073: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:34.752033: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:35.980107: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:36.790928: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:37.396431: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:38.314946: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:39.111772: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:39.859069: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:40.938812: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:42.485426: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
2020-08-23 11:20:43.183461: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at variable_ops.cc:104 : Already exists: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
Traceback (most recent call last):
File "C:\Users\jsmith\Desktop\Code\JohnWorkspaces\Machine_Learning\Next Word\train.py", line 45, in <module>
gen.fit(X_train,Y_train,validation_data=(X_test,Y_test),batch_size=batch_size,epochs=epochs)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 66, in _method_wrapper
return method(self, *args, **kwargs)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py", line 848, in fit
tmp_logs = train_function(iterator)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 580, in __call__
result = self._call(*args, **kwds)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 644, in _call
return self._stateless_fn(*args, **kwds)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 2420, in __call__
return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1665, in _filtered_call
self.captured_inputs)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 1746, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\function.py", line 598, in call
ctx=ctx)
File "C:\Users\jsmith\anaconda3\lib\site-packages\tensorflow\python\eager\execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.AlreadyExistsError: Resource __per_step_0/gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var/struct tensorflow::TemporaryVariableOp::TmpVar
[[{{node gradient_tape/sequential/lstm/while/sequential/lstm/while_grad/body/_354/gradients/AddN_6/tmp_var}}]] [Op:__inference_train_function_3777]
Function call stack:
train_function
Occasionally it will train on one batch, then get the error, and other times it generates the error before any learning takes place at all. I don't know if it has anything to do with the input and output shapes of the LSTM, however I would expect if that was the problem it would occur at gen.compile, not gen.fit.
