ONNX Runtime Inference using GPU : libcublasLt.so.11 not found

Question

Im trying to run inference using ONNX runtime on my server GPU. However im getting this error:

2024-08-10 23:53:29.404983674 [E:onnxruntime:Default, provider_bridge_ort.cc:1745 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1426 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

After looking into my server cuda files, I noticed that i have libcublasLt.so.12 instead of 11. I have the latest versions of ONNX-runtime and ONNX-runtime gpu installed.

How do i fix this? I dont have admin permissions on the server so cant really downgrade cuda or something of that matter.

phelogges · Accepted Answer · 2024-08-12 09:32:13Z

1

You had noticed that your ort required cuda 11 (libcublasLt.so.11) but your current cuda version is 12 (libcublasLt.so.12).

You may go to ort cuda ExecutionProvider to check your ort-cuda-cudnn compatibility, find the best ort version for your system.

Don't forget to add cuda libs into dynamic library search path, for Linux, you may write as export LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH} , usually the path already added when you install cuda.

answered Aug 12, 2024 at 9:32

phelogges

516 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

NoodleArms · Accepted Answer · 2024-09-23 13:02:21Z

1

If you already have CUDA 12 (as indicated by libcublasLt.so.12), you can try using the onnxruntime-gpu version for CUDA 12.

You can install this version with the following command (refer to https://onnxruntime.ai/docs/install/ Install ONNX Runtime GPU (CUDA 12.x) for more details):

pip install onnxruntime-gpu==1.18.0 --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/

Version 1.18.0 is compatible with CUDA 12 and cuDNN 8.9. For additional versions, see the ONNX Runtime CUDA Execution Provider documentation.

edited Sep 23, 2024 at 13:02

answered Sep 23, 2024 at 12:59

NoodleArms

112 bronze badges

Collectives™ on Stack Overflow

ONNX Runtime Inference using GPU : libcublasLt.so.11 not found

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related