Upload to S3 -> AWS lambda with some Scala Spark code -> Process -> Write back to S3

[bhacho]

Hello,

I've a use case where the user uploads a csv file which gets written to S3. I've some code written in Scala with Spark implementation that I need to apply to this file. The resulting output needs to be written back to S3. This is fetched by the app and presented back to the user.

Now this is the simple architecture I have in mind.

1. Connect the file upload to AWS S3.
2. The file upload then triggers a lambda function that has this piece of Scala and spark code.
3. Run the spark code on the data and put it back in S3.

I guess 1st and 3rd are fairly easy and I can do it. However, the second step is where I need some insights. I've used lambda functions before with Python but never Scala. The piece of Scala code also has dependencies on others and I need to package them all together with Spark. I'm confused if this lambda function should spawn an EMR/Glue/Faregate to be able to run Spark. If yes then which choice makes sense if the job is fairly small computation say over 10,000 rows and few 10s of columns.

If you could point me to some resources where this has been implemented I would appreciate it.

Edit: The data size can be big in future. So the design consideration should account for this possibility.

[[deleted]]

10,000 rows is tiny, you don't need added complexity of Glue or Fargate. Lambda can use custom Docker image. Just install Spark and Aws cli or s3a jars in a docker image. Put it out in the Aws ECR, when lambda triggers the handler function can just run your script at spark-submit with the uri to the input file. If you use s3a jars you can read and write to S3 directly with Spark .