How to stop multiprocess python-program cleanly after exception?

Question

I have a Python program that has several processes (for now only 2) and threads (2 per process). I would like to catch every exception and especially shut down my program cleanly on Ctrl+c but I can't get it to work. Everytime an Exception occurs the program stops but does not shut down correctly, leaving me with an unusable commandline.

What I have tried so far in pseudocode is:

try:
    for process in processes:
        process.join()
except:
    pass #Just to suppress error-messages, will be removed later
finally:
    for process in processes:
        process.terminate()

But as I already said with no luck. Also note, that I get the Exception error message for both Processes, so they are both halted I believe?

Maybe I should also mention that most of the threads are blocked in listening on a pipe.

EDIT So I nearly got it working. I needed to try: every thread and make sure the threads are joined correctly. There is just one flaw: Exception KeyboardInterrupt in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored when shutting down. This is raised in the main-thread of the main-process. This thread is already finished, meaning it has passed the last line of code.

Presumably the main thread in the main process is the place you're waiting to join all the other processes (and threads). Is that right? If so then it should always finish last. Are you hitting CTRL-C to raise the KeyboardInterrupt? If so then you might be better off using a signal handler to catch the SIG_INT and instructing your other threads and processes to terminate then. — ed.
– ed., Commented Sep 28, 2011 at 11:10

ed. · Accepted Answer · 2011-09-27 17:47:28Z

1

The problem (I expect) is that the exceptions are raised inside the processes not in the join calls.

I suggest you try wrapping each process's main method in try-except loop. Then have a flag (e.g. an instance of multiprocessing.Value) that the except statement sets to False. Each process could check the value of the flag and stop (cleaning up after itself) if it's set to False.

Note, if you just terminate a process it won't clean up after itself, as this is the same as sending a SIG_TERM to it.

answered Sep 27, 2011 at 17:47

ed.

1,4039 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nobody moving away from SE Over a year ago

I already wrapped each process into a try block, but the processes are multithreaded so possibly I also need to wrap the threads too. Cleaning up should be fine in the finally statement I think. As about the SIG_TERM: Cleanly for now means that I have no zombies, later this can mean a cleanup operation. What I want for now is only to shut down each process, which does not happen in my current program.

ed. Over a year ago

Can you update your post with a bit more detail, e.g. what the exception is and where it is being raised?

Nobody moving away from SE Over a year ago

The kind of exception does not really matter because at this level any exception should trigger a shutdown. As to where it is being raised, I will edit this in my question.

Collectives™ on Stack Overflow

How to stop multiprocess python-program cleanly after exception?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related