How to split a csv into multiple csv files using Dask

Question

How to split a csv file into multiple files using Dask?

The bellow code seems to write to one file only which takes a long time to write the full thing. I believe writing to multiple files will be faster.

import dask.dataframe as ddf
import dask
file_path = "file_name.csv"
df   = ddf.read_csv(file_path)
futs = df.to_csv(r"*.csv", compute=False)
_, l = dask.compute(futs, df.size)

Have you considered splitting it using bash? It might be faster. — rpanai
– rpanai, Commented Apr 30, 2019 at 17:00

rpanai · Accepted Answer · 2019-04-30 17:05:47Z

2

I suspect that when you read df you have df.npartitions is just 1.

import dask.dataframe as dd

file_path = "file_name.csv"
df = dd.read_csv(file_path)
# set how many file you would like to have
# in this case 10
df = df.repartition(npartitions=10)
df.to_csv("file_*.csv")

But as far as I can see it's not faster.

answered Apr 30, 2019 at 17:05

rpanai

13.5k3 gold badges48 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to split a csv into multiple csv files using Dask

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related