I need to process a large text file (4 GB). Which is having data as:
12 23 34
22 78 98
76 56 77
Where I need to read each line as do some work based on the lines. Currently I am doing as:
sample = 'filename.txt'
with open(sample) as f:
for line in f:
line = line.split()
line = [int(i) for i in line]
a = line[0]
b = line[1]
c = line[2]
do_someprocess()
It is taking huge time to execute. Is there any other better way to do this in python??
If do_someprocess() takes a long time compared to reading the lines and you have extra CPU cores you could use the multiprocessing module.
Try using pypy if possible. For some compute intensive tasks it is dozens of times faster than cpython
If there are a lot of duplicate ints in the file, it can surprisingly be faster to use a dict mapping than int() as it saves the time to create new int objects.
First step is to profile as @nathancahill suggests in the comments. Then focus your efforts on the parts where the biggest gains can be made.
split() returns you a list. And then you are trying to access the first,second and third element by
line = [int(i) for i in line]
a = line[0]
b = line[1]
c = line[2]
Instead of that you can directly say a,b,c = line.split() then a will contain line[0], b will contain line[1] and c will contain line[2]. This should save you some time.
with open(sample) as f:
for line in f:
a,b,c = line.split()
do_someprocess()
An example:
with open("sample.txt","r") as f:
for line in f:
a,b,c = line.split()
print a,b,c
.txt file
12 34 45
78 67 45
Output:
12 34 45
78 67 45
EDIT :
I thought of elaborating on it.I have used timeit() module to compare the time taken by the code to run. Please let me know if I'm doing something wrong here.The following is the OP way of writing the code.
v = """ with open("sample.txt","r") as f:
for line in f:
line = line.split()
line = [int(i) for i in line]
a = line[0]
b = line[1]
c = line[2]"""
import timeit
print timeit.timeit(stmt=v, number=100000)
Output:
8.94879606286 ## seconds to complete 100000 times.
The following is my way of writing the code.
s = """ with open("sample.txt","r") as f:
for line in f:
a,b,c = [int(s) for s in line.split()]"""
import timeit
print timeit.timeit(stmt=s, number=100000)
Outputs :
7.60287380216 ## seconds to complete same number of times.
line has more than 3 elements. Better to say a, b, c = line.split()[:3]
split()andint()are the functions taking the most time?