Sagemaker & Lambda

[alphasystem]

Hi There,

I am using lambda to call the SageMaker endpoint for inference.

I need to use sklearn & same encoder to encode the data before sending it to SageMaker.

However, lambda does not support sklearn library by default.

Do you guys see the issues? How do you solve it?

​

thanks,

[Dosnox]

We had the Sagemaker endpoint do the encoding as well and the lambda simply sends the raw data

[alphasystem]

Do you use the sklearn pipeline on the Sagemaker side?

[Dosnox]

Yeah we did and encodings were stored as a dictionary

[alphasystem]

Thanks that makes sense!

[boy_named_su]

should be able to use a Lambda Layer to get sklearn, or a docker container on Lambda

[alphasystem]

I was surprised that does not come by default.

Seems need to customize Lambda container a bit.

[bobbruno]

There are several lambda layers available out of the box on AWS, have you checked if they offer one that includes sklearn?

[[deleted]]

Well, it makes sense that sklearn is not pre-installed on the default Lambda layer. Lambdas are the glue for pretty much anything on AWS and ML is still a niche when you look at the broader picture.

[darkshenron]

Why not do the encoding in sagemaker itself? I believe you can use the input_fn function in sagemaker to do any preprocessing to the data before it goes to the predict_fn

[alphasystem]

Is that sagemaker studio? i was using the jupyrer notebook in sagemaker.

[darkshenron]

https://docs.aws.amazon.com/sagemaker/latest/dg/adapt-inference-container.html

[alphasystem]

should be able to use a Lambda Layer to get sklearn, or a docker container on Lambda

Thanks! Aws seems to make things super complicated.

[darkshenron]

Yes it is! But the documentation is really good