delay between requests using github3 in python

Question

I'm using python github3 module and i need to set delay between request to github api, because my app make to much load on server.

I'm doing things such as

git = github3.GitHub()
for i in itertools.chain(git.all_repositories(), git.repositories(type='private')):
    do things

I found that GitHub use requests to make request to github api. https://github.com/sigmavirus24/github3.py/blob/3e251f2a066df3c8da7ce0b56d24befcf5eb2d4b/github3/models.py#L233

But i can't figure out what parameter i should pass or what atribute i should change to set some delay between the requests.

Can you advise me something?

Could you clarify what you mean by "my app make too much load on server"? Are you hitting a request limit imposed by the GitHub API? If so, why not catch the corresponding HTTP response and implement a back off? — etemple1
– etemple1, Commented May 18, 2017 at 18:34

Ian Stapleton Cordasco · Accepted Answer · 2017-05-20 14:16:56Z

0

github3.py presently has no options to enforce delays between requests. That said, there is a way to get the request metadata which includes the number of requests you have left in your ratelimit as well as when that ratelimit should reset. I suggest you use git.rate_limit()['resources']['core'] to determine what delays you should set for yourself inside your own loop.

answered May 20, 2017 at 14:16

Ian Stapleton Cordasco

29.2k4 gold badges73 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Hal W · Accepted Answer · 2017-05-26 01:02:20Z

I use the following function when I expect to exceed my query limit:

def wait_for_karma(gh, min_karma=25, msg=None):
    while gh:
        core = gh.rate_limit()['resources']['core']
        if core['remaining'] < min_karma:
            now = time.time()
            nap = max(core['reset'] - now, 0.1)
            logger.info("napping for %s seconds", nap)
            if msg:
                logger.info(msg)
            time.sleep(nap)
        else:
            break

I'll call it before making a call that I believe is "big" (i.e. could require multiple API calls to satisfy). Based on your code sample, you may want to do this at the bottom of your loop:

git = github3.GitHub()
for i in itertools.chain(git.all_repositories(), git.repositories(type='private')):
    do_things()
    wait_for_karma(git, msg="pausing")

Collectives™ on Stack Overflow

delay between requests using github3 in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related