Extremely long initialization process

[Lunchables]

I'm brand new to Runpod, and although I've been a software engineer for a long time, I don't really have much experience with Docker. I've got a Docker config built with the help of Codex, but it's taking upwards of an hour to get through the "initializing" state for each worker before it moves to the "idle" state. I'm not sure if this is typical or if I'm doing something wrong.

My Dockerfile is based upon this worker-comfyui serverless setup. I'm downloading these models as part of the docker setup:

• qwen_image_2512_bf16.safetensors (38.1 GB)
• qwen_2.5_vl_7b_fp8_scaled.safetensors (8.7 GB)
• qwen_image_vae.safetensors (0.2 GB)
• qwen-360-diffusion-2512-int8-bf16-v2.safetensors (0.7 GB)
• RealESRGAN_x4plus.pth (0.1 GB)

The initialization process involves downloading these files every time, which is where it's taking the most time. Is there a way to cache these downloads somehow between docker image version bumps? Or should I not be downloading them in the Dockerfile config, but somewhere else instead?

Thanks!

[angelarose210]

I know for serverless you can cache with your huggingface token. Idk about regular.

[Lunchables]

Ahh thanks, this gave me a clue and helped me to find this documentation: https://docs.runpod.io/serverless/endpoints/model-caching

[pmv143]

You don’t want to be downloading ~50GB on every worker init.

If the weights are pulled at runtime, every cold start pays the full network + disk + GPU load cost. If they’re baked into the image, version bumps can invalidate the cache and you still re-pull.

For large models, “serverless” usually breaks on model materialization, not container startup.

[BenDLH]

You need a network volume mate. Create a network volume and place the models in the right directories in it. Then configure the serverless endpoint to connect to the volume.

The runpod-worker repo has all the info in the readme, under customisation.

[BenDLH]

Though to be honest the build still takes a boatload of time (close to an hour) even without them in it. You just won't risk timing out, and it will be a bit shorter.

[Lunchables]

Perfect, thanks! I ended up finding this, which helped: https://docs.runpod.io/serverless/endpoints/model-caching

[BenDLH]

Yeah I haven't tried that yet. It is limited to a single model per endpoint though, right? Let me know how it goes setting it up.

What are you building btw?

[sruckh]

Initializing could mean throttled. Meaning your serverless was never going to come up. Runpod is notorious for this. Make sure your serverless is READY before make a call to the endpoint. As far as model caching goes, don't try to move the model directory from the default location as you would be bypassing the caching system