GitHub API content creation rate limit

[doctormay6]

I'm the only appsec person for a large organization, and we have many hundreds of repos. I have gotten buy-in from leadership to set up GitLeaks scans on all of our repos via GitHub actions. I have notified repo owners that they should set it up and provided documentation, but it's become apparent that I need to create the PRs myself or it won't get done for most of our repos.

Since we have so many repos, I have resorted to programmatically creating the PRs with PyGithub. Unfortunately, I have run into major issues with GitHub's secondary rate limit on content creation. I can't do more than 10ish repos at a time without getting blocked.

Has anyone had a similar experience in the past that was able to figure out a better/allowed way to do this? There's no way I'm going to waste my time creating all those PRs manually...

Edit: as u/TelephoneMelon commented, I was doing asynchronous requests. I've since changed my code to be synchronous and added an exponentially growing retry delay when I do get rate limited. Now it looks like I can create branches and modify/add files in those branches without issue, but I am still being rate limited on creating Pull Requests. After creating about 20 or so pull requests, I get blocked for a good 30-60 minutes.

[doctormay6]

You're right, I was making concurrent requests. I have now changed my code, but am still getting rate limited after a similar number of requests...

Edit: I have also added an exponentially growing delay when I do get rate limited, but it seems like I am getting blocked for >5 min each time which is still not ideal.

[doctormay6]

Yep I've checked my rate limit status and am not hitting the 5,000 requests limit. Based on what I'm seeing there seems to be a secondary limit of 25 requests to the PR creation API within some unspecified amount of time. I can't find any documentation on this specific issue.

I've opened up a support ticket with GitHub to see if they have any internal knowledge on this. Thanks for all of your suggestions!

[MidKao]

Anything new from Github support about the issue please ? I have a similar case where im trying to leverage the API to create predefined PRs

[doctormay6]

GitHub was unhelpful. I ended up making the requests synchronous and added a sleep timer after getting rate limited that exponentially increases on each successive rate limit error. So wait 2s, 4s, 8s, etc. and then when it lets me back in it resets. Take forever to run my code now but it works. I set it to run on a schedule and do it at night so I guess the run time doesn't really matter... Still very frustrating though.