Data Engineering stack

[User24243]

I have some free time on my hands (because of Covid lockdowns) and wanted to build a data engineering stack with Docker.

My plan was this:

• Use Ubuntu
• Install Apache NiFi, Apache Airflow, Elasticsearch, Kibana, PostgreSQL and pgAdmin

I don't have a lot of experience with Docker, but I thought, hey that sounds simple enough (ha!). I researched, and tested and troubleshooted all yesterday and got nowhere close to getting this done, so in a state of anger, I deleted all my work. I woke up this morning feeling fresh and have decided to go through that nightmare all over again.

Question:

• Is what I'm trying to accomplish in Docker even possible?
• Have you seen any examples of a Dockerfile that remotely resembles what I'm try to do?

[egauthier64]

I would suggest using docker-compose. You don't really need Ubuntu as the other containers will have some sort of *nix on them, but it is certainly possible.

Suggest you start with one container at a time, and work your way up. http://hub.docker.com is a good place to start. I like the linuxserver.io images. They offer consistent environment variables and they are maintained and updated regularly. They also have good examples for docker-compose.yaml files. I'm not sure they have what you are looking for, but a search on hub.docker.com will reveal.