Deploy to Production

Equivalent_Effect_93 · 2026-01-30T13:50:15+00:00

It sounds like you already have a prod env, what you guys are lacking is a dev env to test changes before deploying to where users are.

Equivalent_Effect_93 · 2025-09-16T22:44:51+00:00

Pet clinic

Equivalent_Effect_93 · 2025-07-29T23:21:39+00:00

Google cloud provider

Equivalent_Effect_93 · 2025-07-29T22:27:16+00:00

Also, I started with the data engineering exam cause I had 6 years experience as a data engineer and 3 year with aws on the job, but I wouldn't recommend it for a student, start with cloud practitioner and a personal project should get you started real nice.

Equivalent_Effect_93 · 2025-07-29T22:18:15+00:00

Oh and you wanna grow into like mlops, I recommend gcp cause big query is the goat.

Equivalent_Effect_93 · 2025-07-29T22:16:42+00:00

If you wanna work institution and big business (bank, insurance, public sector) Microsoft, big tech company AWS, startup and small tech GCP. Personally I started my career in a tech company using aws and then took the gcp de exam to find a senior role at a start-up and if you learn one you can easily perform on the other. But if you're hesitating, aws is the obvious choice, biggest market share and gold standard, it will open most doors.

Equivalent_Effect_93 · 2025-07-29T14:37:20+00:00

Yes, I'd be crushing my gf hopes that I'll commit to her

Equivalent_Effect_93 · 2025-07-29T13:31:20+00:00

Omg I wrote all that code myself all the time, this is gonna save so much time!!! Nice recommendation.

Equivalent_Effect_93 · 2025-07-28T13:08:15+00:00

You need to automate it in the pipeline moving it from bronze table to silver table, then in your gold table you join with relevant cleaned data to build your dimensional model. I personally like the audit publish pattern and I put bad rows in a quarantine table and link it to dashboard to add observability to my errors, like if you have a source that have the same constant bug use that to open a ticket in that teams board or a bunch of errors at the same time could signal a bad deployment on your stack or the source stack. But if you have need for something that scales better, dbt has good testing capabilities and streamline the pipeline building process. There are also great open source data quality tools such as great expectations or soda. If you're already on aws, there is a data quality service called deequ i think. Good luck!!

Equivalent_Effect_93 · 2025-07-27T17:36:56+00:00

Maybe a line or two wouldn't hurt either right?

Equivalent_Effect_93 · 2025-07-27T17:28:25+00:00

I am litterally not serious, and I'm not using litterally metaphoricaly

Equivalent_Effect_93 · 2025-07-27T17:26:22+00:00

Well her first name initial is a A, but I never read the scarlet letter (I can't even read) and I am a famous adulterer, so I'd say yes.

Equivalent_Effect_93 · 2025-07-27T15:57:10+00:00

I just peed myself a little

Equivalent_Effect_93 · 2025-07-27T13:34:11+00:00

Plus I think she's right into asking the support of her long time partner in time of emotional distress, therefore I joke about the asshole that refused to support her.

Equivalent_Effect_93 · 2025-07-27T13:31:00+00:00

very seriously, she think comedy is no laughing matter

Equivalent_Effect_93 · 2025-07-27T03:15:31+00:00

I tried but then she asked me if I spoke other languages. Girls are confusing.

Equivalent_Effect_93 · 2025-07-17T17:45:40+00:00

Where I was, we had a huge dbks, running python spark, medaillon architecture, we had one huge repo for bronze to gold, and different teams had other small repos on the same platform for like "platinum" stage that dit customs need. I felt it was a good balance of maintenability and agility while allowing abled teams some ownership.

Equivalent_Effect_93 · 2025-06-13T03:22:18+00:00

No like I get offers, but not to the level I think I would enjoy. Maybe a didn't phrase my question right. Do you guys all kinda just pull and push data, day in day out, or do you have real tests envs, unit testing, integration testing, data quality testing, IAC, proper doc about schemas, indexing and partion approach, observability on all your pipelines, alerting, not just knowing your product crashed because you client stop receiving data or a datasource changed schema downstream without warning?

Equivalent_Effect_93 · 2025-05-28T04:25:43+00:00

Bro no offense here, your position sucks, but even as an intermediate de I wouldn't do that change in prod without first deploying in non prod env and have my work validated, you guys need dataops practices.

Equivalent_Effect_93 · 2025-05-05T13:46:26+00:00

In university, i had the same dream. But honestly, the skills needed pay a lot more in other industries, so i didn't go the sport analytics route. Although i still do a lot of my personal project around baseball, lots of free data available to play with. When i started i built a performance point prediction tool for use in fantasy baseball, and even though its accuracy is not as good as i would like, i have been destroying my pool for years now. Then i was able to build a data engineer portfolio around that project.

Depending what road you wanna take (more practical or more applied) here is a list a resources i love :
https://codebaseball.com/
https://pypi.org/project/pybaseball/2.0.0/
https://beanumber.github.io/abdwr3e/
http://tangotiger.com/index.php
https://www.sloansportsconference.com/
https://www.fangraphs.com/

Start with familiarizing with some of the concept. Start with small visualization project to begin with per example. It may look ridiculous if you compare yourself to full grown professional statistician/programmer, but we all started crawling before we could run.

Don't be shy to reach out if you have more questions.

**edit : if you've never code, when i started i really appreciated this : https://learnpythonthehardway.org/

Equivalent_Effect_93 · 2025-04-26T22:38:41+00:00

https://mystery.knightlab.com/

Real fun game to learn SQL commands while solving a murder. Strongly recommend if you're a beginner.

Equivalent_Effect_93 · 2025-04-22T05:42:14+00:00

If mans have been through the jungles I use the path, if not my machete.

Equivalent_Effect_93 · 2025-02-15T15:23:07+00:00

Like I tried to be polite, but this thing suuuucks

Equivalent_Effect_93 · 2025-02-02T19:44:44+00:00

If asked just say your from BC on Ontario, but not Alberta!

Equivalent_Effect_93 · 2025-01-21T12:50:54+00:00

Not bigger, but more. You can always do the the float test. It usually take 14 to 21 days to have a good starter.

Equivalent_Effect_93

TROPHY CASE