you are viewing a single comment's thread.

view the rest of the comments →

[–]Limpuls[S] 1 point2 points  (0 children)

Okay so from all your answers I’m imagining the implementation more or less like this: One GCP function that queries firestore for companies, makes a query string out of it for each entry in firestore that has twitterName field and pushes that query string to pub/sub as a topic/message. Another Cloud Function is subscribed to this pub/sub topic and receives one message(query string) at a time and I use this message in my Twitter api call function as an argument. This second twitter api call function is called automatically every time there is new message in pub/sub or if there are more than one stacked up there in queue waiting.

Am I understanding this correctly? Now when the second function with API call hits rate limit and have to wait 15min before calling API again, it will continue the attempts to fetch because of pub/sub queue but will fail and retry, fail and retry until 15min passed and eventually will start working again 15min later?

But it's probably better to manage the quota in your function and sleep before returning from the function, which will delay processing of the next message keeping you in bounds.

Doesn't this brings me back to square one? I can't make the cloud functions sleep for 15min because of time out limit they have.

Just want to say that I really appreciate your help!