Python requests module, how to issue multiple requests in the for loop?

Question

I wonder why when I call requests.get() method consequentially, like this:

response = requests.get(url.format("set"))
print(response.status_code)
response = requests.get(url.format("map"))
print(response.status_code)
response = requests.get(url.format("list"))
print(response.status_code)
response = requests.get(url.format("vector"))
print(response.status_code)
response = requests.get(url.format("string"))
print(response.status_code)

I got OK status for all requests, but when I do it in the for loop, like:

for word in fIn :
        response = requests.get(url.format(word))
        if(response.status_code == 200):
            print "OK"
        else:
            print(response.status_code)
            print "Error"
            print word

I got 400(Error) for all requests except the last one.

Additional info: there is related question on SO, where are mentioned 2 ways of coping with this situation: wait, headers.
wait doesn't work in my situation
and about headers - I don't know what to provide there.

Update: specific version, that I am trying to implement:

from lxml import html

import requests

fOut = open("descriptions.txt","w")

with open('dummyWords.txt') as fIn:
    for word in fIn :
        print word
        response = requests.get(url.format(word))
        if(response.status_code == 200):
            print "OK"
        else:
            print(response.status_code)
            print(word)

@Tim I am aware of rate limit, I don't know whether they have one. But it seems strange for me that typing requests manually works, while using for loop not. Maybe in the for loop every time new object for session is created and new session is established, while in manual requests one session is used? — spin_eight
– spin_eight, Commented Jul 11, 2016 at 18:39
Is this of any help? stackoverflow.com/questions/31306501/… — Bahrom
– Bahrom, Commented Jul 11, 2016 at 18:41
@spin_eight if you are typing them manually then perhaps you're doing one per half a second? The loop will do all 6 in 0.1 seconds, likely breaking a rate somewhere. — Tim
– Tim, Commented Jul 11, 2016 at 18:42
Looks like you are trying to get the input from a B.S. source like a file (fIn is very unclear here). Try explicitly doing for word in ('set', 'map', 'list', 'vector', 'string'): for a start. — Mad Physicist
– Mad Physicist, Commented Jul 11, 2016 at 18:46

Padraic Cunningham · Accepted Answer · 2016-07-11 19:05:27Z

3

You have trailing newlines that you need to strip off:

with open('dummyWords.txt') as fIn:
    for word in map(str.strip, fIn) :

It works for the last as you obviously have no newline at the end of the last word in the file. "www.foo.com\n" is not the same as "www.foo.com"

edited Jul 11, 2016 at 19:05

answered Jul 11, 2016 at 19:03

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Tim Over a year ago

I think standard is you should have a newline at the end of the last line. If he does or not, is different. stackoverflow.com/questions/729692/…

Padraic Cunningham Over a year ago

@Tim, you are free to edit a file as you please, what you cannot do is add newlines to urls and expect them to work.

Tim Over a year ago

My comment refers to the penultimate sentence - he should have one there, but he clearly doesn't.

spin_eight Over a year ago

@Padraic Cunningham great!!! I initially thought about it on my way home and this idea somehow vanished! Of course that is it!!! Thank you for you attentive look! Have a nice time of a day!! Best regards!

Padraic Cunningham Over a year ago

@spin_eight, no worries, you too.

ajaykools · Accepted Answer · 2018-08-16 10:36:19Z

0

Encoding POST data resolved issue for me.

cmd = urllib.quote(cmds[i])

For my testcases

No newline required
No sleep time required

answered Aug 16, 2018 at 10:36

ajaykools

6858 silver badges12 bronze badges

Collectives™ on Stack Overflow

Python requests module, how to issue multiple requests in the for loop?

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related