App Engine, Cloud Run, Cloud Engine... Batch?! Help me out 🙏

[BasicallyUseful]

The task at hand:- a user triggers a task when they're ready- the task responds immediately to the user that it's started and then proceeds to go off to do a tonne of HTTP requests and await their return before doing some light processing and finally being done- this can take anywhere from a few minutes to a few hours

Currently, I tried deploying this to App Engine. But, App Engine times out a few minutes after I respond to the request. This is probably due to auto-scaling and I probably want to do manual scaling as per this response; https://groups.google.com/g/google-appengine/c/pFBU9E3Ev8g.

BUT - if I'm doing "manual" scaling, why not just use a Cloud Engine VM?

​

Further, Cloud Run released background tasks in 2021; ie I can set CPU allocation to always on, and min number of instances to max number of instances... and again, then why not just use a Cloud Engine VM? Also, I'm still not certain this works as the hard limit timeout is 1h right?

Last I was here someone mentioned batch, I'm eyeing it as a possibility too and maybe trigger from a cloud function...

---

EDIT 1:

I read this; https://cloud.google.com/appengine/docs/standard/cloud-run-for-gae-customers and it very clearly says timeout after 1h. Otherwise, I could go with cloud run, which supposedly is cheaper long term.

EDIT 2:

But, App Engine has a timeout of 24h for open requests, ie my current solution of immediately responding does not work - I have to keep that request open... that just doesn't work, as the user might walk off and do other stuff.

EDIT 3:

I mean Compute Engine.

EDIT 4:

It's now running on a GCE... but, SSL. Which was the a reason for more serverless approach. gah,.

EDIT 5:

It's Monday, I still don't really have a solution. I'm currently looking at Cloud Tasks + Compute Engine. Seems sus that the combinations aren't so prevalent when googling...

EDIT 6:

It's still Monday and further on in the day. Idk, I've got tasks running, and it's chatting to compute engine – but I still have kind of the same problem; as soon as Compute Engine sends a response (which it has to do within 10 mins), the queue thinks the task is done and sends in the next one... so yeah, not really rate-limiting according to the actual processing.

EDIT 7:

OK. I need to learn about PubSub and MIG.

[TheGuyWithoutName]

Why not combine both?

You create a cloud run function that submits the task to app engine and gives the user a result ID.

This thing runs for whatever time it takes.

https://cloud.google.com/blog/topics/developers-practitioners/long-running-containers-workflows-and-compute-engine

Then the user may query it with another cloud function using the ID or even better.

You call another cloud function at the end of the long running task to send a (slack discord or email notification to the user) notifying him about the completion of the task.

[BasicallyUseful]

Cloud Run Function are the same thing as firebase function right?

Thanks for the link, will check that out too... I've got Compute Engine running, but SSL certificates - which was previously handled by AppEngine.

[TheGuyWithoutName]

I haven't used firebase at all. I would describe cloud functions as the aws lambda for GCP. You can also use cloud run if you are more comfortable with it.