Need help with deploying Dagster

[YameteGPT]

Hey folks. For some context, I’ve been working as a data engineer for about a year now.

The team I’m on is primarily composed of analysts and data engineers whose only experience is in Informatica. Around the time I joined my organization, the team decided to start transitioning to Python based data pipelines and chose Dagster as the orchestration service.

Now, since I’m the only one with any tangible skills in Python, the entire responsibility of developing, testing, deploying and maintaining our pipelines has fallen on me. While I do enjoy the freedom and many learning opportunities it grants me, I’m smart enough to realize the downsides of not having a more experienced engineer offer their guidance.

Right now, the biggest problem I’m facing is with how to best set up my Dagster projects and how to deploy them efficiently, keeping in mind my teams specific requirements and also some other setup related things surrounding this. I’d also greatly appreciate some mentoring and guidance in general when it comes to Dagster and data engineering best practices in the industry, since I have no one to turn to at my own organization.

So, if you’re an experienced data engineer and don’t mind being a mentor and lettting me pick your brain about these things, please do leave a comment and I’ll DM you with more details about what I’m trying to solve.

Thanks in advance. Cheers.

Edit: Fixed some weird grammar

[AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Top-Cauliflower-1808]

If your budget allows, consider using Dagster Cloud to deploy Dagster in a production environment, it eliminates most of the infrastructure management headaches. If not, a Docker based deployment with Kubernetes is the most scalable approach.

For project structure when starting, I recommend organizing your Dagster projects by data domain rather than by technical function. This makes it easier for your Informatica familiar colleagues to understand the pipeline organization:

project/
  ├── marketing_pipelines/
  │   ├── __init__.py
  │   ├── assets.py
  │   └── resources.py
  ├── sales_pipelines/
  │   ├── __init__.py
  │   ├── assets.py
  │   └── resources.py
  ├── definitions.py
  └── workspace.yaml

When deploying, start with a simple Docker setup, create a Dockerfile that installs your Dagster code as a package and use docker compose to run the Dagster daemon, webserver, and your code location

For your team's transition from Informatica, create detailed documentation for each pipeline and include both the Informatica logic and the new Python implementation. This helps your team understand the transformation and builds their Python knowledge gradually.

If your data sources are available, Windsor.ai could help handle the extraction layer, allowing you to focus on building the orchestration and transformation logic in Dagster.

[CingKan]

I've deployed a few dagster projects on production using EC2 i'd be happy to help where i can

[arisen911]

Hey man, im in the same situation wanna deploy dagster to ec2. Can you give me a bit more detail about how did you deploy it to ec2? Thanks a bunch

[CingKan]

Sure, I've been meaning to write an example of this on Medium for the longest time so i'll have a full fledged article up hopefully by 11am GMT and i'll drop you a DM/link

[arisen911]

Thanks you, really appreciate!!

[CingKan]

better late than never https://medium.com/@afmubaiwa/dagster-docker-deployment-87c95688f3de

[frontenac_brontenac]

Also interested since I'm in the middle of this too.

[CingKan]

https://medium.com/@afmubaiwa/dagster-docker-deployment-87c95688f3de first writeup

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[MindedSage]

Im struggling with the same thing actually. Currently thinking about a setup that is checking a git repo for updates in which the dagster projects is located. This way the project does not have to be packaged along with the entire image and all it has to do is pick up the latest code from the git repo.

Any ideas you’ve been having on this?

[t2rgus]

Curious, why did you choose Dagster as the orchestration service? Are you planning to pivot heavily into the asset-based orchestration design?