transferring data between processes in python

Question

I'm not new to python but still looking for the best appropriate way of sharing string-formatted data between processes.

Currently I have 4 processes (1 parent + 3 Childs in multiprocessing), parent is fetching data from long poll server and after few checks sends data to mysql server.

Then Childs (1 for each kind of tasks) are processing the data in the way I need to.

Everything works fine for now, my data is securely stored and always in access, especially if I need to debug or implement new features, but after all it works slower with project growth (as for web app - not good). As you might assume, kind of data stored in db is list or its objects. I know that there are some problems in python with transferring data between processes ('d be better to say, software restrictions)...

I was thinking about temporary storing data in JSON or simple txt files, but concurrency wouldn't allow me to do that. Also I could try using sockets, but is it worth to start callback server for such a purpose?

Asynchronous thing didn't work for me either. So what are my options in this case? I don't need to loose any piece of data, just as I need to keep it working fast. Any other measures are welcome :)

p.s most of related topics are outdated or didn't answer my question, because I'm sure that the way I'm working with data isn't best so far.

Nikolas Stevenson-Molnar · Accepted Answer · 2018-06-19 03:08:05Z

2

The multiprocessing module provides some tools to do this: https://docs.python.org/3.6/library/multiprocessing.html#exchanging-objects-between-processes

Specifically take a look at Queue. Any process can push messages onto or pop messages from the queue. So for example, a parent could push tasks, the children wait for tasks and get them on a first-come-first-served basis. If one child is busy working on that task, the next child will get it.

For example, if you needed your child processes to work on data fetched by your parent process, you could have something like this:

from multiprocessing import Process, Queue

def do_thing(q):
    data = q.get()  # Get data queued by the parent process
    # do something with data

if __name__ == '__main__':
    q = Queue()
    p = Process(target=do_thing, args=(q,))
    p.start()
    while True:
        data = get_data_from_server()
        q.put(data)  # Queue data for a child server

If you want something a bit more basic, you could use a Pipe, which allows two-way data transfer between processes.

edited Jun 19, 2018 at 3:08

answered Jun 19, 2018 at 3:01

Nikolas Stevenson-Molnar

4,7502 gold badges25 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Chickenfresh Over a year ago

So, in simpler words, Pipe is something like two functions where one can call another and able to get value in return while queue is task that transfers a-job-to-do for another thread and, instead of waiting for step-by-step complete, starts working on another step from its cycle?

Chickenfresh Over a year ago

But what happens if my app accidentally crashes? I'm loosing all data I needed to work on? If so, how can I handle such a problem (without loosing a lot of resources)

Chickenfresh Over a year ago

and let's think that processing of data received from server always takes at least twice more time than fetching next, so in order to save time I have to predict how fast child works and create as many duplicates as would be enough to finish in time?

Chickenfresh Over a year ago

I love the queue solution, is there any way I could get a number of objects in queue? It seems that on Mac Queue.qsize() doesnt work

Nikolas Stevenson-Molnar Over a year ago

Correct, this is all in memory, so if the app crashes, you will lose the queue. You might want to take a look at Celery, which is a Python task manager, and depending on the backend you use, can preserve tasks even if the program crases/exits: celeryproject.org

|

a.l. · Accepted Answer · 2018-06-19 02:50:41Z

There're many IPC ways(socket, shared memory, pipe, FIFO...), I think you can try pipe, it is simple. see the example below:

import os, sys


def child(prefix, r, w):
    os.close(w)
    print(prefix + ':read')
    s = os.read(r, 1024)
    print(prefix + ':got:' + s)
    print(prefix + ':close')
    os.close(r)
    print(prefix + ':exit')
    sys.exit(0)


r0, w0 = os.pipe()
r1, w1 = os.pipe()
r2, w2 = os.pipe()

if os.fork():
    if os.fork():
        if os.fork():
            os.close(r0)
            os.close(r1)
            os.close(r2)
            print('p:write')
            os.write(w0, 'hello world')
            os.write(w1, 'hello world')
            os.write(w2, 'hello world')
            print('p:close')
            os.close(w0)
            os.close(w1)
            os.close(w2)
            print('p:exit')
            sys.exit(0)
        else:
            child('c0', r0, w0)
    else:
        child('c1', r1, w1)
else:
    child('c2', r2, w2)

Collectives™ on Stack Overflow

transferring data between processes in python

2 Answers 2

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related