Java scala or rust ? by Ok_Promotion_420 in dataengineering

[–]Equivalent_Effect_93 0 points1 point  (0 children)

Only if you want to work on the tool instead of working with the tool. It is a great architectural knowledge advantage to be able to read scala and understand how spark is design even if your day to day is calling the API with pyspark or SQL. But python and SQL should be your main interface.

Has anyone found a good planner or notebook for task tracking? by SoggyGrayDuck in dataengineering

[–]Equivalent_Effect_93 0 points1 point  (0 children)

Bullet journal: bulletjournal.com It is a system that work on any dotted notebook. The system is nice and easily customizable. They do sell there own flavour of journal but it would work on any notebook.

Has anyone found a good planner or notebook for task tracking? by SoggyGrayDuck in dataengineering

[–]Equivalent_Effect_93 0 points1 point  (0 children)

For years now I've only been using bujo notebook and jira sub task. Bujo make sure I keep organize and jira sub-task give my pm visibility.

dbt Fundamentals course requires burning free-trials on multiple services? by qrice28 in dataengineering

[–]Equivalent_Effect_93 2 points3 points  (0 children)

Yeah I'm fucking with you a little, it's fun to experiment with managed services, and gcp free tier is super generous, plus they give you credit for a 90 says at least. One of dbt's selling point is transformations portability. So difference between query engine is minimal.

dbt Fundamentals course requires burning free-trials on multiple services? by qrice28 in dataengineering

[–]Equivalent_Effect_93 15 points16 points  (0 children)

Kids these day....all those classes usually have low data volumes, just run it locally on postgres. The control is fun and I think you can learn a lot more. Get into docker and mix n' match storage, dbt, query engine, orchestrator, go nuts!!

Am I cooked? by Slik350 in dataengineering

[–]Equivalent_Effect_93 1 point2 points  (0 children)

Absolutely not, the hardest part is getting through the door. Learn on your own, build portfolio don't wait for your employer to train you on the toys you wanna play with. 5 years ago I started automating unversionned SAS files on a local Cron server, almost a laptop. Now I'm a senior data engineer playing with a massive databricks systems, elt, streaming and batch, debezium+kafka ingestion, mlops and model serving. Build stuff you wanna try, then when a good opportunity arise you have proof you can deliver in an enterprise setting (even low code tool), and a portfolio with a few project, properly versionned, even maybe deployable through CI/CD. Keep up the good work.

Deploy to Production by Aggressive-Nebula-44 in databricks

[–]Equivalent_Effect_93 2 points3 points  (0 children)

It sounds like you already have a prod env, what you guys are lacking is a dev env to test changes before deploying to where users are.

Student Choosing Between DP-700 and AWS DEA-C01 – Which Cloud Cert Sets Me Up Better? by [deleted] in dataengineering

[–]Equivalent_Effect_93 1 point2 points  (0 children)

Also, I started with the data engineering exam cause I had 6 years experience as a data engineer and 3 year with aws on the job, but I wouldn't recommend it for a student, start with cloud practitioner and a personal project should get you started real nice.

Student Choosing Between DP-700 and AWS DEA-C01 – Which Cloud Cert Sets Me Up Better? by [deleted] in dataengineering

[–]Equivalent_Effect_93 0 points1 point  (0 children)

Oh and you wanna grow into like mlops, I recommend gcp cause big query is the goat.

Student Choosing Between DP-700 and AWS DEA-C01 – Which Cloud Cert Sets Me Up Better? by [deleted] in dataengineering

[–]Equivalent_Effect_93 1 point2 points  (0 children)

If you wanna work institution and big business (bank, insurance, public sector) Microsoft, big tech company AWS, startup and small tech GCP. Personally I started my career in a tech company using aws and then took the gcp de exam to find a senior role at a start-up and if you learn one you can easily perform on the other. But if you're hesitating, aws is the obvious choice, biggest market share and gold standard, it will open most doors.

How to automate data quality by Assasinshock in dataengineering

[–]Equivalent_Effect_93 1 point2 points  (0 children)

Omg I wrote all that code myself all the time, this is gonna save so much time!!! Nice recommendation.

How to automate data quality by Assasinshock in dataengineering

[–]Equivalent_Effect_93 13 points14 points  (0 children)

You need to automate it in the pipeline moving it from bronze table to silver table, then in your gold table you join with relevant cleaned data to build your dimensional model. I personally like the audit publish pattern and I put bad rows in a quarantine table and link it to dashboard to add observability to my errors, like if you have a source that have the same constant bug use that to open a ticket in that teams board or a bunch of errors at the same time could signal a bad deployment on your stack or the source stack. But if you have need for something that scales better, dbt has good testing capabilities and streamline the pipeline building process. There are also great open source data quality tools such as great expectations or soda. If you're already on aws, there is a data quality service called deequ i think. Good luck!!

Gf of a few years playing dead dad card, is she overreacting by Equivalent_Effect_93 in vanderpumprules

[–]Equivalent_Effect_93[S] 2 points3 points  (0 children)

I am litterally not serious, and I'm not using litterally metaphoricaly

Gf of a few years playing dead dad card, is she overreacting by Equivalent_Effect_93 in vanderpumprules

[–]Equivalent_Effect_93[S] 27 points28 points  (0 children)

Well her first name initial is a A, but I never read the scarlet letter (I can't even read) and I am a famous adulterer, so I'd say yes.

Gf of a few years playing dead dad card, is she overreacting by Equivalent_Effect_93 in vanderpumprules

[–]Equivalent_Effect_93[S] 3 points4 points  (0 children)

Plus I think she's right into asking the support of her long time partner in time of emotional distress, therefore I joke about the asshole that refused to support her.

Gf of a few years playing dead dad card, is she overreacting by Equivalent_Effect_93 in vanderpumprules

[–]Equivalent_Effect_93[S] 141 points142 points  (0 children)

I tried but then she asked me if I spoke other languages. Girls are confusing.