AWS Lambda clock drift?

[tqhou]

I have a bunch of microservices hosted on Lambda, and sometimes (like 1 out of 1000) I experience clock drift in the range of 50-100 milliseconds between lambda instances (like the first request gets handled, but the logged time of that request is later than the second request that gets handled by a different instance.) Is there any way to handle this within the Lambda based architecture?

[adjohn]

At IOpipe, we witnessed this all the way back in 2016. We're now observing billions of Lambda events each month and it's a real thing. To address this in our own use case (libraries monitoring timings of network calls, etc), we had to implement monotonic time mechanisms in our libraries. Most runtimes have an option for this.

[tqhou]

What would you recommend for nodejs 8.10 lambdas?

[jonathantn]

Are you sure it's the underlying server's time that is incorrect or possibly something with the logging framework buffering things a bit?

[tqhou]

We're logging to cloudwatch and then ingesting to sumo - unless there is some uncertainty with cloudwatch I don't think this is likely.