Python- Multiprocessing Daemon

Question

I'm creating a multiprocess, which creates a csv file. When I run the code with d.daemon = False it works fine, ie it creates a file in the same folder. But when compiled and run with d.daemon = True, it does not, ie does not creates a file. Why's so?

My Code

I've a seed list of URLs from which I need to scrape the data.

for url in config.SEED_LIST:
    # starting a new process for each category.
    d = multiprocessing.Process(target=workers.scrape, args=())
    d.daemon = True
    d.start()


def scrape():
    import time
    time.sleep(5)
    # The above part of code takes some time to scrape a webpage, applying
    # some logic, which takes some time to execute, hence I've added a time
    # sleep of 5 secs. But when run with daemon = True, the file is not
    # created. Else it works fine.

    data = [[1, 2, 3, 4], [2224, 34, 34, 34, 34]]
    with open('1.csv', "wb") as f:
        writer = csv.writer(f)
        writer.writerows(data)

Michele d'Amico · Accepted Answer · 2014-12-15 23:46:11Z

29

According to multiprocess daemon documentation by setting d.daemon=True when your script ends its job will kill all subprocess. That occurs before they can start to write so no output will be produced.

answered Dec 15, 2014 at 23:46

Michele d'Amico

24k10 gold badges78 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

xxllxx666 · Accepted Answer · 2021-01-17 23:57:16Z

9

d.daemon = True means that the subprocess is automatically terminated after the parent process ends to prevent orphan processes. join() is helpful by simply adding d.join() after d.start(), so that the parent process does not end before the child process; instead, the parent process will wait until the child process ends.

answered Jan 17, 2021 at 23:57

xxllxx666

3833 silver badges7 bronze badges

Collectives™ on Stack Overflow

Python- Multiprocessing Daemon

My Code

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

My Code

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related