Fast http queries in Python

Question

I need to send many http post requests (several thousands) and read the responses in parallel. I use httplib in python, it takes me on average 0.5 ms to send a request, and it is quite slow to read the response. I use coroutines (gevent) instead of threads to make it faster. I thought about the following to make it faster:

write C code (I would need to find a good http library in C), and write a c-extension
use Cython

Which approach would yield the fastest running code? Any sharing of experience with one or the other approach would be much appreciated as well. Any other idea very welcome.

Did you measure where most of the time is spent? If a significant amount of time is spent in creating the network connection to the server, I don't think you'll get a lot faster with a C program. Also 0.5 ms is quiet fast, or did you mean 0.5 s? — Bernhard
– Bernhard, Commented Jun 22, 2015 at 9:04
No, this is just sending the message, excluding the creation of the connection, and excluding the time to connect (three way handshake). Most of the time in spent in sending the data (connection.send, where connection is the httplib.HTTPConnection) — DevShark
– DevShark, Commented Jun 22, 2015 at 9:21
And what is the bottleneck for the data transfer? My (uneducated) guess is that the slow thing is actually network or OS latency and it won't get much faster with optimizing client side code. — dhke
– dhke, Commented Jun 22, 2015 at 9:27
That's a good point. You might be right, I don't know. How could I check that? — DevShark
– DevShark, Commented Jun 22, 2015 at 9:30

Bernhard · Accepted Answer · 2015-06-22 12:32:09Z

1

I would suggest to take a look a PyCURL. Before yo do that you should check if you get higher throughput when you use curl from the command line. If that's not the case, you probably won't get better results with PyCURL.

answered Jun 22, 2015 at 12:32

Bernhard

8,8715 gold badges43 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

DevShark Over a year ago

I had a good look at it, but unfortunately does not improve much, and does not give access to the underlying socket objects, so I lose control. Thanks anyway, it was useful to look at.

Collectives™ on Stack Overflow

Fast http queries in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related