Python Get request fails in Github API

Question

I am trying to get URL for top 2000 Java repos sorted by most stars in Github

Code:

import requests
import json

urls = []

#increased per_page in the request results to 100, so 20 iterations * 100 = 2000 repos
for i in range(20):

   #waiting for 75 secs every 10 requests to respect the github limit
   if i%10==0 and i >0:
      time.sleep(75)
   r = requests.get('https://api.github.com/search/repositories?q=language:java&sort=stars&order=desc&per_page=100&page='+str(i))
   print (r.ok)
   if r.ok == True:
       items = r.json()["items"]
       for each in items:
            urls.append(each["html_url"])

The request fails after 10, (r.ok==False) iterations every time so it does not cross 1000 repos. Any suggestions on the mistake would be great.

What is the status code that you're getting. Check using r.status_code. The likely cause is that you're exceeding rate limits, but I'd check since you're running out after 10 requests, so it could be something else. — Nafiul Islam
– Nafiul Islam, Commented Jul 3, 2020 at 0:23

Rajith Thennakoon · Accepted Answer · 2020-07-03 00:23:20Z

2

I think you are exceeding request limit.According to the documentation,

In rate limits,

The Search API has a custom rate limit. For requests using Basic Authentication, OAuth, or client ID and secret, you can make up to 30 requests per minute. For unauthenticated requests, the rate limit allows you to make up to 10 requests per minute.

In repository search API limit,

Find repositories via various criteria. This method returns up to 100 results per page.

So you get 1000 results per minute and exceed the limitations.

answered Jul 3, 2020 at 0:23

Rajith Thennakoon

4,1402 gold badges16 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Nafiul Islam Over a year ago

I did not know that they introduced just 10 requests as a rate limit. Good catch there.

data_person Over a year ago

I tried to do a workaround with time.sleep(60) every 10 requests, bit even then it fails with status code 422

Rajith Thennakoon Over a year ago

Yep.you have to wait at least 60 seconds.60 + few seconds will be ideal

data_person Over a year ago

i have edited the solution, it fails still with status code 422

Rajith Thennakoon Over a year ago

422 means Sending invalid fields will result in a 422 Unprocessable Entity response.check your request parameters are correct after the sleep.

Collectives™ on Stack Overflow

Python Get request fails in Github API

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related