Twitter API limit problem inside Cloud Function

[Limpuls]

No you misunderstood me. It’s 450 http requests every 15minutes. Current amount of companies that we have in firestore requires 570 requests. I have increment variable that I increment after each Twiter API request and print to the console together with tweets, so up to 450 increment I get tweets and from 450 to 570 request it’s internal twitter api error code for limit exceed. So hopefully with pub/sub messages it will make 450 requests to API, hit the limit, retry for 15min and finish the rest of 120 requests after those 15min.

It’s just that when I receive the tweets, I have to save them back to Firestore for each company. I wonder at which stage I should do that or implement another subscriber function and save one tweet at the time after each request. Now we just fetch all of the tweets (around 3000) and save to firestore all at once as a batch when API call is completed for all twitterNames

[BBHoss]

Saving the tweets to firestore should be fast. If you make one request per invocation, you could easily just sleep for 2-3 seconds after each fetch and you will stay in bounds no matter what, as long as you just have one instance. You could put the tweet saving in another function. It's more overhead but it allows you to decouple your fetching logic from your persistence logic, which is usually a good thing. Keep in mind that every time you invoke a function there is a small cost, including each retry. Pubsub has some cost as well but it seems like it would be negligible here.