Dask Distributed: How to delete uploaded files from the cluster

Question

I wanted to know if there's a function in dask.distributed that removes the files uploaded to the cluster using the client.upload_file()?

Basically, the opposite of the upload_file() function. best regards

pavithraes · Accepted Answer · 2021-10-07 10:43:02Z

1

There's no function to do this directly as of today. But as a work-around, you can use client.run with os.remove:

client.run(lambda dask_worker: os.remove(os.path.join(dask_worker.local_directory, "file.py")))

where file.py can be replaced by the name of the file that you uploaded.

Note that the input argument to client.run must be dask_worker, relevant docs here: https://distributed.dask.org/en/latest/api.html#distributed.Client.run

answered Oct 7, 2021 at 10:43

pavithraes

8046 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1