3

I'm trying to train a TensorFlow (version 2.11.0) code in float16.

I checked that FP16 is supported on the RTX 3090 GPU. So, I followed the below link to train the whole code in reduced precision.

https://www.tensorflow.org/api_docs/python/tf/keras/backend/set_floatx

Then, When I was testing the training time of the model, I observed that the training time is not reduced. Interestingly when I manually convert some precision try to change the precision for some parts of code, the training time gets increased (due to the overhead of converting to FP16).

According to the above link it was mentioned that by default the code will run in float16, but is there any option to check if the underlying hardware really runs in float16?

Furthermore, my network is a 3-layer MLP network (which is quite small), is there any change that FP16 works only with larger networks?

Note:

  1. I'm trying to train an Reinforcement Learning algorithm in reduced precision

    (link: https://github.com/openai/maddpg)

  2. I used CUDA 9.0, cuDNN 7.6.5 to train the algorithm

Improve this question
Related questions
5659
How can I access the index value in a 'for' loop?
4484
How can I find the index for a given item in a list?
3982
How do I get the current time in Python?
Load 6 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.