Multithreading same API request in Python

Question

Is there any way to speed up the SAME post request to an API using multithreading ? I saw examples with many different requests but not with only one. I try to make an average 20 requests per second with a stop in the script when the request send me a 200 response code

My code :

# coding: utf-8
import sys
import requests
import time
import hashlib
import hmac
import json
from concurrent.futures import ThreadPoolExecutor, as_completed

 
def boucleWhile(statusCommande):
        while statusCommande != 200:
            print("intérieur while")
            host = "https://www.google.com"
            headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
 
            url = '/api'
            query_param = ''
 
            body = '{"text":"t-123456"}'
 
            sign_headers = gen_sign('POST', prefix + url, query_param, body)
            headers.update(sign_headers)
            r = requests.post(host + prefix + url, headers=headers, data=body)
            #r = requests.request('POST', host + prefix + url, headers=headers, data=body)
            print(r.status_code)
            print(r.content)
            print(r.json())
 
            if r.status_code == 200:
                return r.content
 
 
if __name__ == "__main__":
        url_list = [
                "https://www.google.com/api"
            ]
 
        host = "https://www.google.com"
        prefix = ""
        headers = {'Accept': 'application/json', 'Content-Type': 'application/json'}
 
        url = '/api'
        query_param = ''
 
        body = '{"text":"t-123456"}'
        #change XXX value
 
        sign_headers = gen_sign('POST', prefix + url, query_param, body)
        headers.update(sign_headers)
        r = requests.post(host + prefix + url, headers=headers, data=body)
        #r = requests.request('POST', host + prefix + url, headers=headers, data=body)
        print(r.status_code)
        print(r.content)
        print(r.json())
        print(body)
 
        statusCommande = r.status_code
 
        #start = time()
 
        processes = []
        with ThreadPoolExecutor(max_workers=200) as executor:
            for url in url_list:
                processes.append(executor.submit(boucleWhile(statusCommande), url))
 
        for task in as_completed(processes):
            print(task.result())
 
 
        #print(f'Time taken: {time() - start}')

Any idea ?

wombatonfire · Accepted Answer · 2021-05-14 01:16:42Z

1

This may not directly answer your question, but with network IO, where most of the time you are waiting for the response, going async/await route should be a better option, than multithreading.

Have a look at asyncio, it allows you to achieve concurrency in a single thread, and network clients is the perfect use case for it. Replace the requests library with the asynchronous HTTP client that works with asyncio, for example, you can use AIOHTTP or HTTPX, and you should be able to handle much more than 20 requests per second.

And yes, I do understand that learning how to write asynchronous code may not be an easy journey and it requires investing time, but if you are going to do network programming, this effort is worth making.

answered May 14, 2021 at 1:16

wombatonfire

5,6001 gold badge30 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PreDaToR1675 Over a year ago

Hello, thank you for your answer ! I will see what i can do with async requests.

Collectives™ on Stack Overflow

Multithreading same API request in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related