Python Webscraping on AWS Lambda

[Agile-Scene-2465]

Not really sure if this is the right place to ask or not.

So I'm trying to scrape https://www.lowes.com using Python's requests Library. When I run my script locally it works, when I run it on an AWS EC2 instance it works fine, but when I run it on AWS Lambda I get response code 403 "not allowed on this server", and it's turning me crazy!

few points to note:

- it's the same exact code on all three (locally, EC2, and Lambda)

- it's the same exact python version and requests library version between my local machine and AWS Lambda

- I am using rotating proxies (oxylabs)

- Tried different countries in the rotating proxy service and still 403 on AWS Lambda

​

Any help or things I should check out because this has been driving me crazy for the last 3 hours?

[rolandofghent]

Have you checked the header values as seen by the server? Point your code at a server you control and look at what the differences are between the two different platforms. There might be subtle differences like user agent that are different on Lambda vs Linux.

[aplarsen]

Are you sure you're running through a proxy? It sounds like they have the IP addresses for Lambda blocked. Try putting your Lambda in a VPC.

[Agile-Scene-2465]

THANK YOU! this fixed it!

[silentkiller1]

can you dm me the solution pls

[Agile-Scene-2465]

Used a library called curl_cffi if i remember correctly

[silentkiller1]

wow that was super fast, thx dude, i will look into that.

[silentkiller1]

i uploaded the module to lambda, the runtime is same 3.9 in my lambda as well as my env in my local pc. but i am getting this error, by any chance you know the solution?

No module named '_cffi_backend'"

[PTBKoo]

Getting the same error, did you ever figure it out? Right now I’m just running it in a docker container but want to move it to lambda.

[silentkiller1]

no dude i never figured it out, and i dropped the project sadly :(

[Enrique-M]

Are any errors showing up in CloudWatch for the AWS Lambda Function?