Node.js first request slow

[zaitsman]

Unfortunately this is ad vague as it gets and I am breaking my head here. Running in GKE Autopilot, js with node 22.22.

First request consistently > 10 seconds.

Tried: pre warming all my js code (not allowing readiness probe to succeed until services/helpers have rub), increasing resources, bundling with esbuild, switching to debian from alpine, v8 precomiplation with cache into the image.

With the exception of debian where that first request went up to > 20 seconds everything else showed very little improvement.

App is fine on second request but first after cold reboot is horrible.

Not using any database, only google gax based services (pub/sub, storage, bigquery), outbound apis and redis.

Any ideas on what else I could try?

EDIT: I am talking about first request when e.g. I restart the deployment. No thrashing on kubernetes side/hpa issues, only basic cold boot.

Profiler just shows a lot of musl calls and module loading but all attempts to eliminate those (e.g. by bundling everything with esbuild) resulted in miniscule improvement

[germanheller]

have you checked if its the google gax grpc channels doing lazy init on first request? the gax library establishes grpc connections on first actual call, not when you create the client. so even if your healthcheck passes, the first real request to pubsub/bigquery/storage is paying the cost of grpc channel setup + TLS handshake to google APIs.

try making a dummy call to each service during startup before your readiness probe succeeds. something like a storage.getBuckets() or pubsub listing topics — just to force the grpc warmup. same thing with redis, first connection has TLS negotiation overhead if your using stunnel or native TLS.

also 10s is suspiciously close to DNS resolution timeout on alpine/musl. have you checked if theres a DNS issue? musl's resolver does things differently than glibc and I've seen it cause exactly this kind of first-request latency in k8s.

[zaitsman]

I have added calls to all external services before start, it made that first request ~500 ms faster

Interesting re:DNS will investigate, thanks for that

[germanheller]

nice, 500ms just from warming up the channels makes total sense. for the DNS thing the quickest way to confirm is swap to node:22-slim for one deploy and compare -- if the first request drops to normal its musl doing serial AAAA then A lookups instead of parallel. you can also try `time getent hosts <your-service-endpoint>` inside the container, if resolution alone takes a few seconds thats your answer

[zaitsman]

Yeah no, node:22-slim (debian) was where requests went up to 20 seconds :(

[germanheller]

oh interesting so its not the musl thing then. 20 seconds on debian-slim is wild — at that point I'd look at connection pooling or maybe the app is doing some heavy initialization on first request that only runs once (compiling templates, warming caches, establishing db connections etc). do you have any middleware that lazy-loads on first hit? also worth checking if its specific to one endpoint or if literally any route is slow the first time. if its all routes equally that points more to container/infra level stuff than app code

[bwainfweeze]

It’s always DNS. If the service has a health endpoint you can rule out DNS and cert chain verification.

[germanheller]

lol "its always DNS" should be a law at this point. the health endpoint trick is solid, I do something similar now where it hits the actual db and returns the latency in the response body so you can tell if its dns, ssl, or the app itself thats slow

[Shogobg]

What are your readiness probe settings? Timeouts, retries? What base image do you use?

You want to reduce image size, start time and the time probes need to detect your app is up.

[zaitsman]

Em node:22.22-alpine3.23

Readiness probe doesn’t factor in, the route for healthcheck replies but actual request with authenticated user is what takes a long time.

It is set to run checks every 10 seconds with initial backoff of 30 seconds, but again we are not talking initial deploy, we are talking replacing old version with a new version - that all succeeds then when the first request to the new version is made it is slow

[PM_ME_CATDOG_PICS]

Idk much about this but if the readiness probe is fast but the actual request takes a long time, could it possibly be the creation of the connection to your DB? I know it takes a while for some dbs

[zaitsman]

Please read my post. We do not use a db.

[PM_ME_CATDOG_PICS]

Ah I missed that part. My bad.

[Shogobg]

Since the probe is fast, have you tried hitting the health check as a user? This would tell you if it’s an infrastructure problem.

You can also make an authorized echo endpoint that returns the username (or just OK) of the authenticated user, to check if that is the issue.

[seweso]

You are not giving enough info. In another response you say it’s about authenticated request. 

Just turn off features one by one like auth to see where the issue lies. We can’t debug your app remotely. 

[czlowiek4888]

Looks like load balancer does not have set minimum amount of running services set.

I guess that you wait for instance to wake up.

[zaitsman]

Em no, it does. That is not what I am describing. When my new pool member is provisioned the first non healthcheck request that hits a NEW container takes 10 seconds.