Parsing a data feed in python

Question

I'm attempting to plot a live (and continuous) feed of data that is being dumped into a csv from another program. I got the plot working but then realized my data only plotted until the time I executed the code. I believe there is something wrong with my parse function (kind of new at this stuff) but I can't figure out what.

import threading
import csv 
import dateutil.parser 
import datetime
import time
import matplotlib
import matplotlib.pyplot as plt
from matplotlib import animation

log = "sample.csv"
data = ""
mytime = ""
thickness = ""
times = ""   

#attempt at continuously reading my data source 
def parser():
    with open(log, 'r') as f:
        global data
        reader = csv.reader(f, delimiter=',')    
        data = reader.readlines()
        time_thickness_data = []
        while 1:
            last_pos = f.tell()
            next_line = f.readline()
            if not next_line:
                time.sleep(1)
                f.seek(last_pos)
            else:
                mytime, thickness = splitter(next_line)
                time_thickness_data.append([mytime, thickness])

def splitter(line):
    mytime = line[0]
    thickness = line[3]
    return (mytime, thickness)
    times = []
    for date in mytime: 
        _ = dateutil.parser.parse(date) 
        times.append(datetime.datetime.strftime(_,'%H')) 

def main():
    a = threading.Thread(target=parser)
    b = threading.Thread(target=splitter)
    a.start()
    b.start()

if __name__ == "__main__":
    main()

#goes on to animated plot using times for x axis and thickness for y
fig = plt.figure()
axes = fig.add_subplot(111)
line, = axes.plot([], [], '.')
plt.show(block=False) #i had to use this to get my plot to show up

def init():
    line.set_data([],[]) 
    return line,

def animate(i):
    a = 50 * i
    xdata = times[:a]
    ydata = thickness[:a]
    line.set_data(xdata, ydata)
    plt.draw()
    axes.relim()
    axes.autoscale_view(True,True,True)
    return line,

anim = animation.FuncAnimation(fig, animate, init_func=init,  
                           interval=1, blit=True)                           
plt.show()

Sample row from csv looks like this: 2015-07-25 14:54:50.786774,1,0,439.85,,,,0,0,

Where is the plotting that you want to be continuously updated? — db1234
– db1234, Commented Aug 12, 2015 at 17:19
Well i was trying not to make this a monster post but I'll edit it back in. — MBlankfield
– MBlankfield, Commented Aug 12, 2015 at 17:28
Why are you using two threads reading same file before trying to process it with only one? What does this line mytime, thickness = splitter(next_line) does (1 param), when you declare def splitter(): (0 param)? — Serge Ballesta
– Serge Ballesta, Commented Aug 12, 2015 at 18:12
@Serge Thats a good point, I attempted to fix that. I appreciate the feedback, I feel like I'm in over my head. — MBlankfield
– MBlankfield, Commented Aug 12, 2015 at 18:53

Serge Ballesta · Accepted Answer · 2015-08-13 14:29:10Z

1

There are too much unused pieces in your code to really understand what you were trying to achieve. Here is a simple function that:

opens a csv file
repeatedly read lines from it, waiting for new lines to be added when reaching end of file

calls a callback function with the decoded list from each csv line

def csv_process(filename, callback, delay=1):
    class waiter:     # wrapper around a file object to wait for new lines on EOF
        def __init__(self, fd, delay=1):
            self.fd = fd
            self.delay = delay
        def __iter__(self):
            return self
        def next(self):  # try to read a line of wait delay seconds
            while True:
                line = fd.readline()
                if line:
                    return line
                time.sleep(self.delay)
        def __next__(self):  # ensure compatibility with Python3.x
            return self.next()
    with open(filename, "rb") as fd:
        rows = csv.reader(waiter(fd, delay), delimiter=',')
        for row in rows:
            callback(row)

Python2 iterators shall implement next, while Python3 ones shall implement __next__. To ensure compatibility with both, above code defines both methods.

edited Aug 13, 2015 at 14:29

answered Aug 12, 2015 at 20:32

Serge Ballesta

150k13 gold badges137 silver badges267 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

MBlankfield Over a year ago

Your code is far more compact than mine and I believe it accomplishes the same thing. I'm still trying to get it to work though. I defined filename and callback but when I run the code I get an error from the csv.reader stating that argument 1 must be an iterator. Is there something I need to do outside the function with the class you created?

Serge Ballesta Over a year ago

@MBlankfield: what Python version are you using?

MBlankfield Over a year ago

I'm using version 3.4

Serge Ballesta Over a year ago

@MBlankfield: My iterator was only compatible with Python 2.x . I've updated my post to be compatible with Python 2.x and Python 3.x

Collectives™ on Stack Overflow

Parsing a data feed in python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related