Using python pandas script on multiple files

Question

I have a python (pandas) script that reads csv files into a dataframe, edits them, and makes them into a new csv (output):

import pandas as pd

df = pd.read_csv(r'\filename.csv', sep=',')

# I skipped over the dataframe edits

try:
    df.to_csv(r'outputfilename', sep='\t', encoding='utf-8')    
except IOError as e:
    print('Could not make Excel file' % e)`

I would like to use it on multiple csv files at once and get multiple outputs (csv's):

I have tried:

from tkinter.filedialog import askopenfilename

filename = askopenfilename()

df = pd.read_csv(filename, sep=',')
...

and it works (gives me an selection window), but I am only able to choose one file

sudonym · Accepted Answer · 2018-06-12 07:43:42Z

2

If doing it one after another rather than all files at once is sufficient, have a look at the glob module:

1. Create list of files in target directory

target_directory = r'Path/to/your/dir'
file_list = glob.glob(target_directory + "/*.csv") 
# Include slash or it will search in the wrong directory!

2. Loop through files in list and do your magic

for file in file_list:                # Loop files
    df_result = your_function(file)   # Put your logic into a separate function
    new_filename = file.replace('.csv', '_processed.csv')
    df_result.to_csv(new_filename, index = False)

answered Jun 12, 2018 at 7:43

sudonym

4,0384 gold badges40 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

DasVisual Over a year ago

This works for the most part. It goes through the files and edits them, but the last part .to_csv returns: AttributeError: 'NoneType' object has no attribute 'to_csv'

DasVisual Over a year ago

I used:

for file in file_list:                # Loop files         df_result = use_pandas_script(file)   # Put your logic into a separate function 	new_filename = file.replace('.csv', '_processed.csv') 	df_result.to_csv(new_filename, index=False) 	print(df_result.info())

and the result was: <class 'pandas.core.frame.DataFrame'> Index: 0 entries Empty DataFrameNone so you're right the dataframe was empty. Is there anything that can fix that?

sudonym Over a year ago

this means that your use_pandas_script() returns an empty dataframe. please update your initial QUESTION and include the full use_pandas_script(), beginning with def and ending with return

DasVisual Over a year ago

I think I see what happened. I forgot to include return df4 at the end of the function. I have included it and it works so far. Wow thanks! I feel so stupid for missing that.

sudonym Over a year ago

I am happy to help - happy coding!

|

Veera · Accepted Answer · 2018-06-12 08:32:42Z

0

from tkinter.filedialog import askopenfilename
filename = askopenfilename()

In this code you are saying that 'askopenfilename' that is to open one file, where as you should open multiple files you should try the below code

import Tkinter,tkFileDialog
dir = Tkinter.Tk()
mul_file = tkFileDialog.askopenfilenames(parent=dir,title='Choose a 
                                                   multiple file')

'mul_file' will be tuple. if required change it to list by

mul_file=list(mul_file)

this gives all choosed files in a list.iterate list in loop and do your csv manipulation in it

edited Jun 12, 2018 at 8:32

answered Jun 12, 2018 at 8:21

Veera

264 bronze badges

Collectives™ on Stack Overflow

Using python pandas script on multiple files

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related