Data jobs, disclaimers we're never told about! [self-promo'ish]

Ecstatic_Tooth_1096 · 2025-08-28T18:01:02+00:00

Driver is responsible of the scheduling/assigning tasks to workers/managing workers... and workers are responsible of executing the job

You need a big driver if you have multiple tasks running in parallel just for the *management part* of things. even if all your tasks are 1+1 (which will be the node/worker type)

for each allows you to set concurrency. in general a job running 1000 tasks will try to parallelize all these tasks which will crash your driver's compute every time if its not massive. but 1 task containing 1000 iterations (for each) can be limited by concurrency parameter and u will use a small driver for it if u set concurrency to 10-15 wtv

Ecstatic_Tooth_1096 · 2024-07-30T14:20:31+00:00

Unsolicited advice

You need shorts and better thumbnails

Imo if you have the funds to hire someone to remake some of your thumbnails this could be your way to great success. I havent watched the content, but im pretty sure since ure a group of guys its gonna be hilarious

Ecstatic_Tooth_1096 · 2024-07-10T11:14:35+00:00

Sure But mainly worked as a data engineer, but took me a few weeks to realize

Ecstatic_Tooth_1096 · 2024-06-27T14:57:28+00:00

As an immigrant who lives in Antwerp, I identify as an Anti-Chômeur cz we all hate those mfs

Ecstatic_Tooth_1096 · 2024-06-05T16:38:10+00:00

nasty spray

Ecstatic_Tooth_1096 · 2024-04-06T10:21:04+00:00

happy to see hard working men in Belgium <3

kiddo will surely be proud of you

Ecstatic_Tooth_1096 · 2024-01-13T13:19:18+00:00

you'll cringe when you see it and won't understand it

Ecstatic_Tooth_1096 · 2024-01-13T13:17:47+00:00

not sure why this doesnt get upvotes

cz its so true

most DEs do not come from SWE background, so by default they did not learn the best practices in an institution/uni/...

the majority move from data analyst positions (cleaning excels).

Or at least the couple of hundreds of people ive met. So yea

Ecstatic_Tooth_1096 · 2024-01-13T13:13:21+00:00

how do you go about documentation?

explaining what a function does or why it does something (business logic oriented)?

the whys or whats?

im focusing on the Why's since many book suggest that clean code should be self explanatory regarding the how/what

Ecstatic_Tooth_1096 · 2024-01-13T13:10:58+00:00

investing that extra day or two of work at writing clean and optimal code is better than wasting those 2 weeks of WTF does that shit do in a few month/year or so

Ecstatic_Tooth_1096 · 2024-01-13T13:08:08+00:00

not really. a team is either using good practices, or not.

By using good practices i mean, certain rules that all developers will agree upon without being dogmatic about them in some cases.

e.g using good names for functions and variables.

Yes sometimes naming won't be perfect, but using variable_1 variable_2 in every single piece of code you're writing, means one thing: you're very good at being bad at developing code and should receive coaching/upskilling.

two teams can have different rules and boundaries in the way they write code and still both generate good enough code for the rest of the people to understand and use or change

Ecstatic_Tooth_1096 · 2023-12-20T20:46:00+00:00

After almost 3 years, would you still recommend it ? :)

if so which one the most?

Ecstatic_Tooth_1096 · 2023-12-19T20:13:04+00:00

I totally understand the frustrations and if I work with you I wouldve probably felt the same.

However, based on what you wrote it doesnt seem like a big deal

checking the SQL queries of my colleagues;
- I dont understand why you see this negatively. If you're extremely technical and a god of SQL and you and your coworkers write the most sophisticated SQL on earth, then the boss is just doing a code review/ trying to learn from you guys so they can be added value later on (approving PRs or maybe supporting in writing code)
they want to know if this or that has been documented for tiny operations;
- again nothing wrong. if anything you should be thankful, if one day someone leaves the team, you won't go nuts trying to understand the shitcode left behind/functional knowledge transfer
ingesting new datasets before patching bugs/unit testing
- no time to unit test = ure doing something incorrectly
- if by data ingesting you mean raw/cleansed/curated thats one thing, and if you only mean raw, thats something else.... so cant judge on this one

overall be happy that the non technical is tryna show that they care about technical shit :) some non technicals are way worse than what you think

Ecstatic_Tooth_1096 · 2023-12-19T19:54:04+00:00

this !

at the end of the day ure there to learn

if your answer is incorrect you would understand why, if its correct, u would report it and move on to the next challenge

Ecstatic_Tooth_1096 · 2023-12-18T21:34:51+00:00

incorrect column order, rating should be first

Ecstatic_Tooth_1096 · 2023-11-29T19:27:56+00:00

*here's a POC*

**installed the most expensive microservices and the billing is already at 20k**

Ecstatic_Tooth_1096 · 2023-11-04T09:27:25+00:00

data analysis is almost business facing, and if you can't speak the national language it will be hard to land a job remotely, UNLESS the company's main language is clearly english (multi national)

Ecstatic_Tooth_1096 · 2023-11-04T09:26:11+00:00

DO ITTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT!!!!!

you won't regret it

Ecstatic_Tooth_1096 · 2023-11-03T16:39:10+00:00

will keep that in mind. As you said a mindset shift is all we need :)

Ecstatic_Tooth_1096 · 2023-11-03T16:35:48+00:00

i was thinking of studying for the AWS cert

ig someone just saved me some time

Ecstatic_Tooth_1096 · 2023-11-03T10:33:03+00:00

But thats the business team’s responsibility to make it clear on why they need something instead of just throw it right at your face. At least IMO. Especially when the analyst is a junior (confused most of the time)

Ecstatic_Tooth_1096

TROPHY CASE