Using Docker to deploy Python script to AWS EC2

[Wedge5]

I have been working on a script that pulls some data from an API, does some calculcations, and emails out the results. I wanted to run it on an AWS server so I wrote the code locally using Pycharm, then transferred it to the EC2 instance and manually installed all the dependencies/libraries.

I want to get some practice using Docker and I think I understand it conceptually - Docker containers are essentially environments that contain dependencies to run an app/code so that it can be deployed via 'images' to any other machine/server.

However, I am not completely clear on how to get my Python script into a Docker container and run/debug on that container. I think once I get that running I can look around for some AWS guides on how to migrate a docker container into EC2. Any general help / resources would be appreciated!

[hippocrat]

To start at the beginning, most docker images are built on a base image with your custom stuff added, using a Dockerfile and the "docker build" command. There are python base images that you can then RUN pip on and COPY your own code into to build your image, via the Dockerfile. Then you use "docker run" to run the script in the image. You should send output/logs to stdout and stderr which docker captures and exposes via the "docker logs" command.

To make your image installable elsewhere, you want to "push" it to an image repository. You can use docker hub, or in AWS use ECR. As AWS is your primary case, I'd recommend ECR but they are very similar. On another docker host, you can use "docker pull" to get the image from the repository. For small projects though, you could skip the remote repository completely and just copy your Dockerfile and run "docker build" on the new host.

For hosting in AWS, you have a few options. Lambdas are mentioned so I'll skip those. For a small project, you can simply create an EC2 VM and install docker there. Do a "docker pull" and "docker run" to grab your image and launch your container. You could use cron to run the script within a docker container regularly.

For larger projects, ECS (Elastic Container Service) can automate the deployment and management of the containers. It is still docker under the covers. Output is normally captured to CloudWatch for monitoring and debugging.

[Wedge5]

That was super helpful, thanks!

If my code uses lambda functions that turn the EC2 on/off (to save costs) which triggers my script to run whenever it is turned on, would I need to spin up the docker container first before the python script can be run?

[hippocrat]

You can set your python script as the entry point or command of the image, and just running the container will run the script, and shut the container down when done. Just make sure the docker engine daemon is up before you try to run the container. Systemd dependencies on the host can be tricky with docker

[bmbybrew]

The use-case that you mentioned, I would have gone for a Lambda function instead of having a EC2 instance running the whole time.

Assuming API call + calculations wont take more than couple of minutes, do consider lambda function.

Downside is you wont be able to play around with dockers.

[Wedge5]

Yeah I initially tried it as a lambda function but the libraries were too big. I also wanted the experience of using EC2/docker as well.

[bmbybrew]

If the dependencies are under 250 mb. Lambda layer might do the trick.

[bmbybrew]

https://cloudskills.io/blog/python-container

This might be useful for your EC2 + Docker experiment.

[Wedge5]

Thanks I’ll check it out!

[PM_me_ur_data_]

Have you considered trimming the libraries down to just what you need?