PyTorch not working when using Pytorch with cuda 11.1: Dataloader

Question

So, I'm running this deep neural network. And normally, when i train it with the cpu using the pytorch library WITHOUT cuda, it runs fine.

However, I noticed when I installed pytorch+cuda 11.1, whenever I try to enumerate over the Dataloader, the following error gets thrown out:

OSError: [WinError 1455] The paging file is too small for this operation to complete. Error loading "C:\Users\Name\Desktop\Folder\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll" or one of its dependencies.

But, when i use pip to uninstall pytorch+cuda and install pytorch without cuda, my script runs fine.

both versions of pytorch are 1.8.0

Does anyone know how to go about fixing this? Would love to be able to use this gpu.

Note: This is how i set up the dataloader:

    train_dataloader = DataLoader(
        train_dataset,
        batch_size=args.batch_size,
        num_workers=args.num_workers,
        shuffle=True,
        pin_memory=(device == "cuda"),
    )

James · Accepted Answer · 2021-03-25 13:24:34Z

1

See this issue on the PyTorch github page: https://github.com/Spandan-Madan/Pytorch_fine_tuning_Tutorial/issues/10

Basically you need to turn off the automatic paging file management in Windows.

Windows Key
Search for: advanced system settings
Advanced tab
Performance - "Settings" button
Advanced tab - "Change" button
Uncheck the "Automatically manage paging file size for all drives" checkbox
Select the "System managed size" radio button.
Reboot

answered Mar 25, 2021 at 13:24

James

37k4 gold badges54 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

krpytix Over a year ago

This worked! However, I should note that setting it to System managed size didn't work, rather I had to set some obnoxiously larged number to both the initial and maximum allowed size.

celsowm Over a year ago

@krpytix how did you do that?

Thomas Erdösi · Accepted Answer · 2021-04-29 18:39:51Z

0

This error also occurs when the page file is currently being extended by the OS. Setting the page file to a FIXED SIZE solved the issue for me.

Proceed as follows:

Go to the advanced system settings
Select tab "Advanced"
Click the first "Settings" button
Select tab "Advanced"
Click on "Change" button
Uncheck the "Automatically manage paging file size for all drives"
Select option "Custom size"
Enter the desired size as "Initial size" AND as "Maximum size" (this ensures that the page file is created with the maximum size and does not need to be extended)
Confirm all dialog windows and reboot

answered Apr 29, 2021 at 18:39

Thomas Erdösi

5463 silver badges15 bronze badges

Collectives™ on Stack Overflow

PyTorch not working when using Pytorch with cuda 11.1: Dataloader

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related