python-requests with multithreading

Question

I am working on creating a HTTP client which can generate hundreds of connections each second and send up to 10 requests on each of those connections. I am using threading so concurrency can be achieved. Here is my code:

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://20.20.1.2/tempurl.html')
            if response1.status_code == 200:
                client_notify('r')
        except(exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout) as Err:
            client_notify('F')
            break
        requestCounter += 1

def main():
    for q in range(connectionPerSec):
        s1 = requests.session()
        t1 = threading.Thread(target=generate_req, args=(s1,))
        t1.start()

Issues:

It is not scaling above 200 connections/sec with requestRate = 1. I ran other available HTTP clients on the same client machine and against the server, test runs fine and it is able to scale.
When requestRate = 10, connections/sec drops to 30. Reason: Not able to create targeted number of threads every second.

For issue #2, client machine is not able to create enough request sessions and start new threads. As soon as requestRate is set to more than 1, things start to fall apart. I am suspecting it has something to do with HTTP connection pooling which requests uses.

Please suggest what am I doing wrong here.

Why are you reusing the same session object? As I can understand, your connectionPerSec actually means the number of parallel threads, and requestRate is actually the number of serial requests on each thread. You might also want to look at concurrent.futures. — Denilson Sá Maia
– Denilson Sá Maia, Commented Aug 5, 2014 at 20:00
session object is reused only in the spawned thread so it uses the same connection for all serial requests. — user3040103
– user3040103, Commented Aug 5, 2014 at 20:26
Seems like ThreadPoolExecutor from concurrent.futures does the same thing what I am trying to do? — user3040103
– user3040103, Commented Aug 5, 2014 at 20:37
Note that in CPython the concurrency of threads is limited: Only one thread at a time can execute Python bytecode. This limitation is enforced by the Global Interpreter Lock to simplify memory management. — Roland Smith
– Roland Smith, Commented Aug 6, 2014 at 0:43

johntellsall · Accepted Answer · 2014-08-05 22:02:20Z

I wasn't able to get things to fall apart, however the following code has some new features:

1) extended logging, including specific per-thread information

2) all threads join()ed at the end to make sure the parent process doesntt leave them hanging

3) multithreaded print tends to interleave the messages, which can be unwieldy. This version uses yield so a future version can accept the messages and print them clearly.

source

import exceptions, requests, threading, time

requestRate = 1
connectionPerSec = 2


def client_notify(msg):
    return time.time(), threading.current_thread().name, msg

def generate_req(reqSession):
    requestCounter = 0
    while requestCounter < requestRate:
        try:
            response1 = reqSession.get('http://127.0.0.1/')
            if response1.status_code == 200:
                print client_notify('r')
        except (exceptions.ConnectionError, exceptions.HTTPError, exceptions.Timeout):
            print client_notify('F')
            break
        requestCounter += 1

def main():
    for cnum in range(connectionPerSec):
        s1 = requests.session()
        th = threading.Thread(
            target=generate_req, args=(s1,),
            name='thread-{:03d}'.format(cnum),
        )
        th.start()

    for th in threading.enumerate():
        if th != threading.current_thread():
            th.join()


if __name__=='__main__':
    main()

output

(1407275951.954147, 'thread-000', 'r')
(1407275951.95479, 'thread-001', 'r')

Collectives™ on Stack Overflow

python-requests with multithreading

1 Answer 1

source

output

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

source

output

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related