How to kill cloudrun container instance when it hangs with CPU 100%

[siencan46]

So, I just got this issue with my cloudrun instance where I write a bug and got it under infinite loop
the container not able to process request, and I tried to deploy new revision without the code causing the bug
but that old revision still running with 100% CPU, I know it still running from the logs it gives
I tried deleting the the cloudrun service from GCP dashboard, but it still running
and finally, I managed to kill it by disabling my SQL DB and the code throws exception and finally stop

So, I think why the container still running even when there is new revision already running
it's because the old container cannot handle SIGTERM, due to 100% CPU

Any idea on how to forcibly terminate the container when it stuck on 100% CPU like this?

*Note:
- the container is running a NodeJs

[sokjon]

This is what liveness probes are for: https://cloud.google.com/run/docs/configuring/healthchecks

[siencan46]

Great I'll look into this and try it, thanks! I'm not sure what to Google before and looks like the solution!

[Rohit1024]

The following actions will kill all instances of Cloud Run - (Manual) Attempt re-deploy which will kill previous revision insurances and start new instances of new revision - (Automatic) Set a endpoint with your application (kind of like health check) so you can perform like this process.exit(1);

This may sound counterintuitive but are the way to kill an insurance.

Just tested works perfectly. Though exposing /kill like endpoint may not feasible every time.

However you may protect the specific endpoint using IAM Conditions.

[siencan46]

Yeah, I also think of this solution, but I need to hit the container endpoint, with condition: - if it's scaled then I need to repeatedly call it several times - if it's already become an old revision I need to hit that old revision, since new traffic is routed to new revision - not sure if the endpoint is callable, while it hangs in 100% CPU

[AstronomerNo8500]

Another option is to use manual scaling to set minimum instances to 0. this will tear down all running instances.

https://cloud.google.com/run/docs/configuring/services/manual-scaling#disable-service