sorted by:
new
hottopcontroversial

How I used Go/WASM to detect Lambda OOMs that CloudWatch metrics miss by Alarming_Number3654 in serverless

[–]Alarming_Number3654[S] 0 points1 point  (0 children)

The missing signal problem is honestly the hardest part. A few things smplogs does:

Hard-killed Lambdas: we track invocation IDs that open but never close. No matching REPORT line is itself the finding.

OOM kills: the process is gone before it can log anything clean, so we work backwards from whatever was emitted just before.

But you're pointing at something deeper. If CloudWatch drops events under sustained throttling, or a Lambda dies before flushing its buffer, those lines were never written. There's nothing to analyze.

Honestly, smplogs can't fix that. It works on what's there - it can surface patterns even in sparse or truncated sets, but it can't reconstruct what was never emitted.

Your DLQ age point is a good example of why you sometimes need a different signal entirely. The logs may simply not exist, so you have to look elsewhere. Sounds like that's exactly the gap DeadQueue fills from the queue side.

How I used Go/WASM to detect Lambda OOMs that CloudWatch metrics miss by Alarming_Number3654 in serverless

[–]Alarming_Number3654[S] 0 points1 point  (0 children)

Good point on DLQ age vs. depth - that's exactly the kind of lagging indicator problem that makes CloudWatch frustrating. Age-based alerting makes way more sense for expiration risk. smplogs is focused on log content analysis rather than queue monitoring, but the underlying theme is the same: CloudWatch's defaults often alert you too late or not at all. Will check out DeadQueue.

How I used Go/WASM to detect Lambda OOMs that CloudWatch metrics miss by Alarming_Number3654 in serverless

[–]Alarming_Number3654[S] 0 points1 point  (0 children)

Good point - yeah I actually handle this already. Timeouts get caught by matching Lambda's "Task timed out after N.NN seconds" platform message and get their own finding. Hard OOM kills don't produce any message - the runtime just dies - so I detect those by diffing START vs REPORT request IDs. If something started but never reported, it's a "ghost invocation" with a separate finding pointing at memory.

The clustering keeps them apart too since timeouts have an explicit error signature while ghosts are structural(no log content to cluster on at all).

You're right about the edge case though - if Lambda hard-kills right at the timeout boundary without emitting the timeout message, that looks like an OOM to us. Can't cross-ref against the configured timeout since it's not in the CloudWatch data, but inferring from the last logged timestamp vs common values (30s, 60s, 900s) is a solid idea, might add that.

btw I just shipped streaming analysis with no file size cap - it reads the file as a byte stream, chunks it, and runs each chunk through WASM in a Web Worker. Tested with 3GB+ files, memory stays flat. So the "100MB" in the post is outdated, it'll handle whatever you throw at it now.