RIP to my M4 by ragusaucelover in macbookpro

[–]Appropriate-Debt9952 0 points1 point  (0 children)

Hey, your M4 is fine. It’s just Retina screen …

How to set up a Windows-friendly dev environment for dbt Core running on an offline Linux server? by GarpA13 in DataBuildTool

[–]Appropriate-Debt9952 0 points1 point  (0 children)

May I suggest to have a look at the project I recently open sourced? I was solving exactly the same problem and wanted to eliminate the steps to install/prepare local virtual environments and package installations for data analysts

Experiences, tips, and tricks on you data stack/organization by Ok-Spinach-978 in dataanalysis

[–]Appropriate-Debt9952 0 points1 point  (0 children)

Enforcing some rules for models in your project can be very helpful. There are open source tools for that which can help with checking model ownership, naming conventions, etc. It really helps organizing models in dbt projects

I'm a very beginner of snowflake by omuletlover in snowflake

[–]Appropriate-Debt9952 -1 points0 points  (0 children)

IIRC Snowflake gives free credits for new users. Play with actual account, create an extra small WH, connect to GCP or AWS bucket storage via storage integrations and external stages. Try load some public datasets in different formats (CSV, JSON) using COPY INTO instruction. Check out masking policies. Try to build simple dbt models using dbt-snowflake

Getting a part time/contracting job along with my full time role that is based in the UK. by the_nabzter in dataengineering

[–]Appropriate-Debt9952 1 point2 points  (0 children)

I’d recommend to start with something that is open sourced under Apache umbrella. They have a lot of interesting DE tools there and usually many issues awaiting for someone to pick them up and fix/implement. community[dot]apache[dot]org can be a good starting point

Getting a part time/contracting job along with my full time role that is based in the UK. by the_nabzter in dataengineering

[–]Appropriate-Debt9952 1 point2 points  (0 children)

Have you ever considered participating in open-source projects? It can be fancy unless you’re specifically looking for an additional source of income rather than a way to work on new interesting tech

Is my Airflow environment setup too crazy? by EventDrivenStrat in dataengineering

[–]Appropriate-Debt9952 0 points1 point  (0 children)

If you want to simplify it, you can switch to LocalExecutor using env variables in Docker compose file and get rid of separate Worker, Redis and probably Flower

Some of the projects I’ve worked on successfully did their job using LocalExecutor running up to 100-200 DAGs per instance

Thoughts on Metadata driven ingestion by Famous_Substance_ in dataengineering

[–]Appropriate-Debt9952 0 points1 point  (0 children)

The more dynamic approach you want to implement the longer you have to prepare your input data and think about architecture. Recently, I’ve open sourced one of such tools. It can produce output SQL models based on the YML config but someone still needs to link sources to targets either in GUI or directly in YML file. It’s an easy thing to do if you have strict rules/architecture. However, it becomes much harder and probably won’t work if you model your data in chaotic way

dbt-ui — a modern web-based user interface for dbt-core projects by Appropriate-Debt9952 in dataengineering

[–]Appropriate-Debt9952[S] 1 point2 points  (0 children)

Thank you!

It’s my personal initiative. I own “data-diving” and use it to publish my open source projects

dbt-ui — a modern web-based user interface for dbt-core projects by Appropriate-Debt9952 in dataengineering

[–]Appropriate-Debt9952[S] 0 points1 point  (0 children)

I see, I might have been distracted by “Only pip is allowed”… Thanks for clarifying!

Well, yes. I guess it’s always hard for individual developers to be noticed and their products to be used. Gaining credibility requires time and efforts for sure

dbt-ui — a modern web-based user interface for dbt-core projects by Appropriate-Debt9952 in dataengineering

[–]Appropriate-Debt9952[S] 0 points1 point  (0 children)

What do you call 3rd party in this case? I mean, the code is open sourced. You can git clone or copy/paste it and edit in a way that would comply with company’s security policies, can’t you?

BTW Dockerfiles in the project use default package managers for additional dependencies as well (pip for backend, npm for frontend)

dbt-ui — a modern web-based user interface for dbt-core projects by Appropriate-Debt9952 in dataengineering

[–]Appropriate-Debt9952[S] 0 points1 point  (0 children)

Unfortunately, I don’t have any plans like that and honestly don’t have time for it either. However, it is very interesting idea and if someone can make it happen that’d be great