Text file converted to CSV using Python is different than converting with Excel

Question

I have a program that parses a huge file of data output from an eye tracker. The raw file comes to me in text format, but I need a CSV file to do data analysis on.

What I had been doing is opening the text file in Excel, saving it as a .csv file, and then running it through my parser. That works fine, but it's laborious, so I want to create a piece of code to run at the beginning of my parser: which takes the raw text file, turns it into a CSV file, and then runs the parser on the just-made CSV file.

The code I am attempting to use is as follows and modified from here:

txt_file = subjectNum + ".asc"
csv_file = "subject_" + subjectNum + ".csv"
in_txt = csv.reader(open(txt_file, "r"), delimiter = '\t')
out_csv = csv.writer(open(csv_file, 'w'))
out_csv.writerows(in_txt)

This generates a file just fine, but the parser then fails to process it the same way as it does the "manually-generated" files that I get when doing the conversion through Excel. The parser does create the files, but they are empty.

Also, my source text file is 17.8mb. When I convert it into CSV using Excel, the resulting file is 16mb and contains 237,218 rows. When I use the code above to convert the text file into CSV, the resulting file is 17.8mb and 236,104 rows.

It seems like I am missing something in the code above that happens when I convert manually with Excel.

could you post some of the file you are using for input and your expected output? — dm03514
– dm03514, Commented Sep 30, 2013 at 15:08

Tim Pietzcker · Accepted Answer · 2013-09-30 15:14:58Z

4

You need to close the file after writing to make sure it's written to disk entirely.

Also, you should always open the file in binary mode (Python 2) (or in newline="" mode (Python 3)).

with open(txt_file, "rb") as infile, open(csv_file, 'wb') as outfile:
    in_txt = csv.reader(infile, delimiter = '\t')
    out_csv = csv.writer(outfile)
    out_csv.writerows(in_txt)

edited Sep 30, 2013 at 15:14

answered Sep 30, 2013 at 15:09

Tim Pietzcker

337k59 gold badges520 silver badges572 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Text file converted to CSV using Python is different than converting with Excel

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related