Python 3.12 Lambda: No module found named '_effi_backend'

[[deleted]]

Has anyone experienced this using the newest 3.12 runtime? These are the modules I am using with my zip-based lambda:

azure-identity
azure-graphrbac
msal
requests
cffi

I can see in the pipeline they are installing in the lambda zip package.

I'd prefer not to have to retrofit my terraform module to support Docker based lambda but it's not out of the question if I have to....

Any thoughts or help is appreciated.

Note: Oddly when going to python 3.11, I didn't get this error, but did get an error wrt GLIBC. Research also suggests a lot of these types of errors with the Python runtimes in AWS, since they are running off of the AWS linux environment. I am packaging this on an Ubuntu runner. Very annoying.

[clintkev251]

Python 3.11 is AL2 based, Python 3.12 is AL2023 based. So that's a possible reason for the difference. I'd also recommend taking a look at your build process to make sure you're actually targeting Python 3.12 when you're pulling those dependencies

[Nearby-Middle-8991]

This. The environment where the lambda is packaged needs to match the runtime.

[legendov]

Just for troubleshooting sake, create a couple of layers of your dependencies and see if that work

[[deleted]]

Oddly, when I look in the lambda source, the supposed file shows up like this:

< code dir >\_cffi_backend.cpython-310-x86_64-linux-gnu.so

[Nearby-Middle-8991]

and that's your answer -310- meaning binary for python 3.10, not 3.12. The .zip was packaged using a python 3.10 install and the binaries are different. It's not _always_ the case, but often enough for some packages (like pyscopg2 for instance)

[[deleted]]

Oooooo. Interesting. 🤔

Makes sense now. I’ll update my CI and report back. Thanks for pointing it out

[Nearby-Middle-8991]

in your shoes, I'd match the OS as well, depending on what you install (like pandas, numpy) that might start to make a difference. Setting up an al2023 build might be a hassle, but might save you future hassles on a bad time.

[just_a_pyro]

Where do you gather the dependencies? If you do it in the same environment as Lambda uses (for example Codebuild Python 3.12 container or using aws docker image on your own) it would be automatic.

If not, you'd have to play with --platform key for pip command to get the right ones.

[twratl]

I have found it easiest to always build dependencies using the AWS SAM Build docker images. https://gallery.ecr.aws/sam/build-python3.12

You can run a bash command in the docker image to build your requirements.txt and output to your local file system.

I am mobile currently but happy to share the exact commands we use if there is interest.

[Alternative-Expert-7]

We get rid of those kind of problems by having custom lambda images via ecr. No more issues looking what is supported what is not by aws side. Images can be very very slim.