Python: requests.get, iterating url in a loop

Question

I am trying to get information from stats.nba.com by iterating requests.get(url) in a for loop, where the url changes at every iteration. If I just iterate it once it works but twice or more seems to give errors and I'm not sure why. I'm new to programming so any info will be helpful. Thanks in advance. Here's my code:

import requests
import json

team_id = 1610612737

def get_data(url):
    response = requests.get(url)
    if response.status_code == 200:
        data = response.json()
        return data
    else:
        print(response.text)
        print(response.status_code)

for i in range(30): # 30 NBA Teams
    base_url = "http://stats.nba.com/stats/teamdetails?teamID="   
    team_url = base_url + str(team_id)
    data = get_data(team_url)

    ## Do stuff ##

   team_id +=1

If I do 'for i in range(1):' it works, but I get status_code = 400 for each iteration if the range is greater than 1. Thanks for the help!

I don't get results even via browser sometimes - it takes multiple page reloads. The furthest I've gotten up to so far is range(4) — Bahrom
– Bahrom, Commented Apr 26, 2016 at 1:21
I don't get reproducible results even with range(1), or for any IDs. It seems to be fairly random on my end. — Alex Huszagh
– Alex Huszagh, Commented Apr 26, 2016 at 1:22

l'L'l · Accepted Answer · 2016-04-26 03:04:16Z

10

The website limits requests per second, so you'll need to include specific request headers or put a delay in your script (the first option being the quickest and likely most reliable of the two).

Headers Method:

'''
add under team_id = 1610612737
'''

HEADERS = {'user-agent': ('Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5)'
                          'AppleWebKit/537.36 (KHTML, like Gecko)'
                          'Chrome/45.0.2454.101 Safari/537.36'),
                          'referer': 'http://stats.nba.com/scores/'}

Then add this to your response get:

response = requests.get(url, headers=HEADERS)

*You shouldn't need to have a delay in your script at all if you use this method.

Delay Method:

~~import time time.sleep(10) # delays for 10 seconds (put in your loop)~~

Seems like hit or miss using a delay, so I'd not recommend using unless absolutely necessary.

edited Apr 26, 2016 at 3:04

answered Apr 26, 2016 at 1:23

l'L'l

47.5k12 gold badges102 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Bahrom Over a year ago

ah yes this must be it - I was trying with a delay of 3 seconds and it wasn't helping.

l'L'l Over a year ago

@Bahrom: Oh, my bad, I was oblivious ... and thanks very much for upvoting!

Bahrom Over a year ago

No worries, great answer :D

pl0222 Over a year ago

Thank you! The headers method worked for me, but the delay method didn't. Much appreciated!

Collectives™ on Stack Overflow

Python: requests.get, iterating url in a loop

1 Answer 1

Headers Method:

Delay Method:

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Headers Method:

Delay Method:

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related