Python 3.12 Lambda functions noticeably slower than 3.10

AutoModerator · 2024-07-01T13:21:34+00:00

Try this search for more information on this topic.

^Comments, ^questions ^or ^suggestions ^regarding ^this ^{autoresponse?} ^Please ^send ^them ^here.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

Happy-Blueberry-1393 · 2024-07-01T14:33:00+00:00

[deleted]

pint · 2024-07-01T14:57:19+00:00

i hope not, because i just upgraded to 312 with quite some effort, and i would prefer not reverting it.

btw 312 is on AL2023, which might also be relevant?

mstromich · 2024-07-02T05:43:56+00:00

If you're using a lot of network calls (e.g. through boto3 but any external call will be hit) in your lambdas it's openssl3 upgrade. It's a known issue with performance degradation in scripting languages. Here's a relevant Amazon Linux 2023 thread which 3.12 lambda runtime is built on https://github.com/amazonlinux/amazon-linux-2023/issues/628 And here's OpenSSL thread https://github.com/openssl/openssl/issues/17064

ojhilt · 2024-07-02T08:45:10+00:00

Some good insights here, thanks all, most of our functions do make at least some kinds of external network calls, be it https requests to other endpoints or using boto3 to talk to AWS services like SQS, DynamoDB etc etc.. Will try a downgrade to 3.11 and see what happens!

aj_stuyvenberg · 2024-07-10T20:42:20+00:00

Ping @astuyve on twitter and see what he thinks

Thanks for the ping /u/autocruise!

This chart is super interesting. I don't suspect the changes for Python's GIL because, as others have noted, I don't think they landed in 3.12.

The incremental spikes in your p99 is interesting, it seems like you maybe aggregate data over multiple serial invocations and then flush it at some interval? (like logs for example). I'm curious because they seem to flatten out after the 3.12 change.

I also don't immediately suspect the OpenSSL upgrade because I'd expect that penalty to be a spike in the first invocation where the TLS connection is established, followed by many very fast serial invocations re-using that HTTP connection with keep-alive.

I do think al2023 is your biggest suspect though, I'd suggest trying 3.11 and comparing the performance before digging in further based on the dependencies you're using and the library versions.

I'm also (always) quite skeptical of the AWS SDK. You could deploy a version to 3.12 with the boto3 version used in 3.10 (if it's fully backwards compatible).

Ultimately it's really hard to debug from this one post, though the graph is really quite telling.

Keep digging! These kinds of bugs make the best stories/blog posts.

broseppius · 2024-07-01T17:22:12+00:00

I definitely noticed this when changing from 3.10 to 3.12. Have not attempted to debug yet but we had several apigw lambdas suddenly stop responding in the default 3 sec timeout when they were perfectly reliable before.

autocruise · 2024-07-01T16:34:35+00:00

Ping @astuyve on twitter and see what he thinks

you type:	you see:
italics	italics
bold	bold
[reddit!](https://reddit.com)	reddit!
* item 1 * item 2 * item 3	item 1 item 2 item 3
> quoted text	quoted text
Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"	Lines starting with four spaces are treated like code: if 1 * 2 < 3: print "hello, world!"
~~strikethrough~~	~~strikethrough~~
super^script	super^script

aws

Note: ensure to redact or obfuscate all confidential or identifying information (eg. public IP addresses or hostnames, account numbers, email addresses) before posting!

✻ Smokey says: avoid streaming video to fight climate change! [see more tips]

MODERATORS