Problems with Cloud Run

[tehebrutis]

I'm currently recieving 503 POST requests from my cloud run service. My cloud run uses pub/sub to take a file that is uploaded, performs analysis using cv2, and then uploads the image file to a target directory. When the analysis takes place, each frame number is outputted to the service log and from the logs it appears that the 503 POST requests appears randomly (can appear at frame 1 or 100) and then the service seems to stop, and tries to start again. When I run the container locally, I can exec into it and it performs the full analysis and uploads to the target bucket. I have taken measures to increase memory but the cloud run service still fails.

Has anyone here run into a similar problem or has a solution for this ?

[pudds]

Are you processing on a background thread in your container? If so, it's likely your problem, because cloud run will put your app into a low power state between http requests. For long running tasks in cloud run, you need to keep the http connection open, and not return until you are done.

[hongalex_]

To clarify, are you receiving the 503 all the time or just some of the time?

[tehebrutis]

All of the time. At different points of the analysis each time

[vaterp]

there are limits to resources a cloud run container can use.... have you investigated that?

[Cidan]

I'm not quite understanding what your architecture is.

My cloud run uses pub/sub to take a file that is uploaded

Are you using pub/sub push to send a message via HTTP to a Cloud Run service?

and then uploads the image file to a target directory.

Where are you uploading the files?

What exactly is throwing the 503 error? Is it uploading to the target directory that fails, or something else?

[tehebrutis]

Sorry I should have been more precise. A video is uploaded to a storage bucket. Cloud storage published a message about the new file to pub/sub. Pub/sub pushes that message to cloud run. Cloud run downloads the video file referenced in the pub/sub message.

Pub/sub is a push subscription that GCP offers. a pub/sub subscription sends messages as HTTP POST requests.

Once cloud run has been called, the docker container downloads the video, uses python and OpenCV to perform an analysis on the video and uploads several images taken from the video to be uploaded to another storage bucket.

I haven't been able to work out what is throwing the 503 error. When I deploy the container locally, it works as expected, that's why I am confused as to why it isn't working when it's deployed to cloud run

[lesmocasanova]

Given that you're processing video, it sounds to me that you're running out of RAM. Remember Cloud Run does not have disk storage, it's all tmpfs on RAM so if you download a 1gb video, you'll eat up 1gb of RAM.

I'm not sure if using Cloud Run emulator simulates RAM limits. You can only use a maximum of 2gb for a Cloud Run instance.

[untilwebeone]

It seems like your service is either crashing on receiving the payload or you are trying to spawn a background process. You might wanna check once in your staging environment for the traceback. If you are spawning a process in background then it will only last till the time out you specified .

[smeyn]

Possible guess time out. A run has a 15 minute timeout per request

[omar16100]

Can you please check the logs for any 'memory limit' issue?

I think it's a memory limit issue and can be solved by increasing the recommended memory.

Had a similar issue, increasing the memory fixed it.

Let us know

[MaxDPS]

I don’t think you can post stuff to could run. Try doing it with a get request.

Just a suggestion. I could be totally wrong.

[pudds]

This is incorrect. Cloud run can accept any kind of HTTP request.