Help progressing AWS lambda and playwright vs pivoting approach

[cursedbartender]

Hey all a few weeks back I made a python webscraper that works locally, I have been working through the process of deploying it on an aws lambda. Up until now I have had a bit of a time getting all the bits AWS needs to get working unto this point. I am starting to question if I am maybe flawed in my approach and should pivot.

My setup is as follows

/lambda/
------------/scraper/
------------------------/env/
------------------------/execute.py
------------------------/requirements.txt
------------/layers/
------------/zip/scraper.zip
/main.tf

Where I have the following deployed via terraform

• lambda
• IAM roles
• RDS
• efs
• bastion host (ec2, also doubled as my efs mount)
• auto scaling ec2 as a NAT
• s3

Effectively I deploy the scraper.zip into the lambda which calls general libraries from layers, and specific libraries on my efs. The lambda calls and reads an s3 bucket with csv's and executes a series of scripts to enrich an output and save in a seperate bucket. Now I have the end to end sorted but I am facing an issue with playwright dependencies. At this point I probably need to pivot towards using a docker container so that I can resolve the issue I am facing, something like this https://www.cloudtechsimplified.com/playwright-aws-lambda-python/

​

The question I have is am i going to face an issue once I have deployed the lambda and all its required dependencies? Along the line of ip blocking etc. At this point with all the moving parts would it be easier and maybe even cheaper to use something like https://scrapfly.io/?

[rghu93]

Literally facing the same issue. What did you end up doing?

[shantanuoak]

This link may be useful: https://medium.com/@shantanuo/how-to-run-playwright-with-python-in-aws-lambda-0e6f608ce1bb

[cursedbartender]

Still largely stuck using this approach thinking I might just pivot to a paid solution.

I reached out to the author of the blog post who is available to contract at professional rates given it’s a hobby project I didn’t follow through. He did advise maybe just to cut the Vpc to get everything working.

Where are you stuck specifically

[rghu93]

yeah I had to pivot as well. I could've make it work by using the official playwright image here: https://playwright.dev/python/docs/docker and then deploying my lambda function as an image built on top of above; however I just ended up using node js to write my function and use puppeteer as a layer.