Every month I am downloading the whole ledger in csv format and saving the same to a folder. How I can read it in pandas as single dataframe
1
-
1Does this answer your question? Import multiple csv files into pandas and concatenate into one DataFrameRavi– Ravi2020-12-03 19:51:14 +00:00Commented Dec 3, 2020 at 19:51
Add a comment
|
3 Answers
fthomson's answer but the last line should be
import pandas as pd
import glob
files = glob.glob("path/*.csv")
df = pd.DataFrame()
for f in files:
csv = pd.read_csv(f)
df = df.append(csv)
1 Comment
Make42
This answer is NOT it. Never append DataFrames repeatedly in a loop as it repeatedly allocated memory! Every
df.append copies the entire DataFrame. ni1o1's answer is to be used.This is a faster way. Append small DataFrame after large DataFrame will cost a lot. So a better way is to append all DataFrame into one list and use pd.concat to concat all DataFrame.
import pandas as pd
import glob
files = glob.glob("path/*.csv")
df = []
for f in files:
csv = pd.read_csv(f)
df.append(csv)
df = pd.concat(df)
Comments
I'm doing this from memory but this is the general idea
import pandas as pd
import glob
files = glob.glob("path/*.csv")
df = pd.DataFrame()
for f in files:
csv = pd.read_csv(f)
df = df.append(csv)
