How to asynchronously run functions within a for-loop in Python?

Question

Hi I was wondering how to asynchronously call a function within a for-loop in Python, allowing the for-loop to execute more quickly. bar() in this case is a time intensive function, which is why I want the calls to it to be nonblocking. Here is what I want to refactor:

def bar(item):
    //manipulate item
    return newItem

newItems = []
for item in items:
    newItem = foo(item)
    newItems.append[newItem]

Here is what I've tried:

async def bar(item):
    //manipulate item
    return newItem

async def foo():
    newItems = [bar(item) for item in items]
    newItems = await asyncio.gather(*newItems)
    return newItems

newItems = asyncio.run(foo())

This doesn't seem to work as each function call still waits for the previous one to finish before starting. I would love tips on what I might be doing wrong. Thank you so much for any and all help!

You may need to utilize Python's multiprocessibng capability for what you are doing since, async is usually only useful in speeding things up if you have an I/O operation which needs to wait. seeSpeeding Up Python with Concurrency, Parallelism, and asyncio for further clarification. — itprorh66
– itprorh66, Commented Jul 14, 2022 at 20:59

Artiom Kozyrev · Accepted Answer · 2022-07-16 14:42:46Z

3

If your tasks are really async you can do it the following way:

import asyncio


async def bar(item: int) -> int:
    # manipulate item
    print("Started")
    await asyncio.sleep(5)
    print("Finished")
    return item ** 2


async def foo():
    items = range(1, 10)
    tasks = [bar(item) for item in items]
    new_items = await asyncio.gather(*tasks)
    return new_items

if __name__ == '__main__':
    results = asyncio.run(foo())
    print(results)

answered Jul 16, 2022 at 14:42

Artiom Kozyrev

3,9062 gold badges18 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

plynch714 Over a year ago

Perhaps my issues stem from my ignorance of what makes certain functions "really async" as I thought I could make any function asynchronous. Thank you for the reply.

Artiom Kozyrev Over a year ago

@plynch714 well if my answer solves the question for async tasks, please mark it as solution. Actually if your tasks are not async, for CPU heave tasks use multiprocessing module.

Collectives™ on Stack Overflow

How to asynchronously run functions within a for-loop in Python?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related