Using parallel method instead of sequential loop

Question

I have code where already loop is used and I want to replace loop with multithreading, but not able to understand how to do. below is code.

def run_test_function(arg1,arg2, arg3, arg4, arg5):
   try:
      #loop1
      for x in arg1:
         #loop2
         for y in arg2:
            #loop3
            for z in range(len(arg3)):
               #statements
            #statements
         #statements
   except Exception as e:
      print(e)
def main(argv):
   arg1 = value1
   arg2 = value2
   arg3 = value3
   arg4 = value4
   run_test_function(arg1, arg2, arg3, arg4)

I want every iteration of loop1 should behave as a thread, so that I can replace loop with multithreading. Appreciation and thanks in advance for help. :)

statements are like few kubectl commands and processing their output and validating the output. Lets say statements are like printing x n y of two loops. — Adeep Malmotra
– Adeep Malmotra, Commented Aug 10, 2021 at 11:47

ForceBru · Accepted Answer · 2021-08-10 12:01:50Z

2

To print x and y in parallel (as you said in the comment):

from itertools import product
from multiprocessing import Pool


def run_test_function(arg1, arg2):
    with Pool(4) as pool:
        pool.starmap(print, product(arg1, arg2), 5)

if __name__ == '__main__':
    run_test_function([1,2,3,4,5], "abc")

itertools.product imitates nested loops
multiprocessing.Pool provides parallelism
- Pool.starmap executes functions using the Pool

Output:

~/test $ python so.py
1 a
1 b
1 c
2 a
2 b
2 c
3 a
3 b
3 c
4 a
4 b
4 c
5 a
5 b
5 c
6 a
6 b
6 c
7 a
7 b
7 c

answered Aug 10, 2021 at 12:01

ForceBru

45k10 gold badges71 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Adeep Malmotra Over a year ago

I apologize but not able to understand how to implement it for my code. I have edited my question, can you please look into it and see what is actually happening with my code.

ForceBru Over a year ago

If arg3 and arg4 are constants, you can pass a lambda function: pool.map(lambda x, y: processing_function(x, y, arg3, arg4), product(arg1, arg2))

Adeep Malmotra Over a year ago

what is the functionality of product and Pool(4) and according to you I need to place the statements in different function and have to call that function in run_test_function?

ForceBru Over a year ago

"what is the functionality of product and Pool(4)" - you can find links to the documentation are right after the code in the answer. "I need to place the statements in different function and have to call that function in run_test_function?" - yes

Adeep Malmotra Over a year ago

thank you, you cleared a lot, one more question, what if there is one more nested loop as loop3 and loop3 is like for x in range(len(arg3)) ? I have edited my question accordingly.

|

Collectives™ on Stack Overflow

Using parallel method instead of sequential loop

1 Answer 1

11 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related