The system cannot find the path specified when reading csv with dask

Question

I'm trying to read a csv file with dask read_csv. At the moment it fails with the following error:

FileNotFoundError: [WinError 3] The system cannot find the path specified: xxx.csv

It seems dask changes the path that I pass as a parameter. Here's what I do:

import dask.dataframe as dd

path= os.path.join('x/y/z', 'xxx.csv')
dd.read_csv(path, usecols=cols)

Instead of reading the file in x/y/z/xxx.csv, it tries to find the file in network/workarea/x/y/z/xxx.csv. If I check my working directory, it appears as follows: network/workarea. Whatever I change my working directory to, the path just just gets appended to the working directory.

Interestingly reading the file using the path above with pandas works.

Any help would be appreciated. Thank you.

Note that you are mixing posix-style paths ("/") with the windows separator ("\\", implicit in os.path.join), which is a bad idea. — mdurant
– mdurant, Commented Jan 26, 2022 at 14:34

Michael Delgado · Accepted Answer · 2022-01-26 00:41:17Z

1

When working with a dask LocalCluster or a networked file system, be sure to use absolute paths, not relative paths. This is vital because the workers themselves do not have the same working directory as the client.

In your case:

path = os.path.abspath(os.path.join('x/y/z', 'xxx.csv'))

Should solve the problem

answered Jan 26, 2022 at 0:41

Michael Delgado

15.7k4 gold badges39 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Coder Over a year ago

Thanks for your help. Doing this still appends x/y/z/xxx.csv to network/workarea, which results in network/workarea/x/y/z/xxx.csv

Michael Delgado Over a year ago

hmm. Yeah try @mdurant's idea of separating all path elements in the os.path.join call. you should be getting something like c:\\folder\\project\\x\\y\\xxx.csv. Can you print out path before passing it to dask to make sure you have the absolute path? If this really is an issue on the dask side (unlikely) we'll need a full minimal reproducible example to debug.

Collectives™ on Stack Overflow

The system cannot find the path specified when reading csv with dask

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related