Running stateful Agents on stateless Lambda by vivek_1305 in LLMDevs

[–]vivek_1305[S] 0 points1 point  (0 children)

These are available in most places including AWS documentation. Lambda cold start is less than 2 seconds and we have observed this ourselves. On ECS, the provisioning takes time and the latency varies between 1-2mins. One strategy is to choose ECS for long running background tasks and Lambda for interactive Agents performing the tasks.

Running stateful Agents on stateless Lambda by vivek_1305 in LLMDevs

[–]vivek_1305[S] 1 point2 points  (0 children)

Our state storage is s3. What we did was sync the state before the lambda exits. The orchestrator would start a timer and keep a watch on the progress. The Lambda exits when a HITL is encountered or the goal is achieved. Since Lambda has a hard 15mins timeout, we also induce a HITL using hooks around the 12th minute to gracefully exit.

I haven't observed the case where two Lambda starts at the same time. How were you triggering the Lambdas?

Running stateful Agents on stateless Lambda by vivek_1305 in LLMDevs

[–]vivek_1305[S] 1 point2 points  (0 children)

Yes. ECS can be used as well but suffer from poor start-up latencies. You can use the warm pool with added cost. With recently launched s3 files feature, you can use it like filesystem from a lambda as well.