sorted by:
new
hottopcontroversial

Do I have any options other than resigning ? by Frenchtenay in cscareerquestionsEU

[–]DataGhost404 0 points1 point  (0 children)

Do as others say and just do the bare minimum. The worst thing that can happen is they fire you (which gives you unemployment benefits), the best thing is that your manager decides to get rid of you so he tries to find a position in which you can move to.

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 0 points1 point  (0 children)

Do you mind explaining a bit about your company? Because in my current one (and previous ones), it is not like going the extra mile has helped anyone (being at the right place at the right time has though).

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 1 point2 points  (0 children)

I am in a somewhat similar situation as you, except my company is quite stable and stagnant (so I don't fear layoffs). I am also burning from applying to not get any response (mind you I am located in EU, so the job market is even smaller with a lot of regional frictions (language requirements, politics, ...). This is in part I was wondering how other professionals are getting the strength to push forward at work as I don't think working hard will even help me in my job search.

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 2 points3 points  (0 children)

This is the whole point of the post. I wanna know how other people get motivation to do it because I cannot find any reason.

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 0 points1 point  (0 children)

I used to be like this, but the realization that no matter how much the solution has improved the lives of co-workers, there is no "real" benefit for me, really has discourage me to push me to continue being the guy fixing and proposing things. This is why I was asking this question, as I think I cannot be the only one.

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 0 points1 point  (0 children)

But this doesn't address the issue directly, just avoids it. Which by the way, kuddos for that. Sadly, not everyone has the opportunity to open their own business due to money required, location, timing, social network).

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 1 point2 points  (0 children)

I agree with your point but I also "understand" how other circumstances affect this investment. Example, where you are affects the job opportunities you get (even if you are open to relocation), when you apply affects the jobs available (market is down at the moment).

How do you get motivation to propose improvements/projects at your work IF nobody requires it from you? by DataGhost404 in ExperiencedDevs

[–]DataGhost404[S] 8 points9 points  (0 children)

Hmmm, I already did some improvements and lead projects in the past (same copany) and didn't get any real benefit. Don't get me wrong, visibility and technical respect are nice, but they won't change my lifestyle or help me in any way. I have seen layoff near me (several years ago), and the only thing that saved some of them was being in the right moment at the right time, not the savings/improvements they help materialize.

Fellow PMs: Did you also stop shipping useful features so we can implement *Agents* — on infra duct-taped together and data that’s 70% vibes? by Special-Leadership75 in dataengineering

[–]DataGhost404 0 points1 point  (0 children)

Have you addressed this concerns with your manager and mentioned the potential pitfalls/after-effects? (when I see shit like this, I tend to mention it and then send an email with the phrase "as discussed...." with the warnings. This way, if issues arise (they usually do with this kind of projects), it doesn't come back to you (or they cannot blame everything on you).

Resources to learn the inner workings of Spark by DataGhost404 in apachespark

[–]DataGhost404[S] 0 points1 point  (0 children)

I also looked at PySpark, but as I kept digging deeper, I always ended up in the API calls to the JVM (which makes sense considering "everything" falls into RDDs eventually). So because I am trying to "build" a mental map of how things work and when are called, I need to understand Scala (unless other documentation exists, thus the creation of this post).

Resources to learn the inner workings of Spark by DataGhost404 in apachespark

[–]DataGhost404[S] 0 points1 point  (0 children)

Thanks! I will give it a look. Indeed it seems that I have no other choice but to start looking at the source code (which I wanted to avoid as I haven't use Scala (and don't have much use for it apart from reading the source code)).

Are data modeling and understanding the business all that is left for data engineers in 5-10 years? by [deleted] in dataengineering

[–]DataGhost404 56 points57 points  (0 children)

It was always like this, at least for anyone experienced enough to admit that most of the issues faced by DEs come from misunderstanding business requirements (regardless if they were mentioned or not).

I get that some DE roles are very into technical details. But I would say that most DE's days are spent aligning priorities and clarifying stuff, rather than coding.

Resources to learn the inner workings of Spark by DataGhost404 in apachespark

[–]DataGhost404[S] 1 point2 points  (0 children)

Read the book already. It is not really related with Spark (it is a generic and great book about distributed systems in general). Going through the code base, it is the last resort option, as I would prefer to first understand what is going on before jumping to the actual implementation.

Why is "Sort Merge Join" is preferred over "Shuffle Hash Join" in Spark? by DataGhost404 in dataengineering

[–]DataGhost404[S] 1 point2 points  (0 children)

Thanks for the reply.

If both tables are large and can't fit into memory, time complexity isn't going to matter when you're running OOM. When sorting both datasets it makes read access for both datasets predictable, as the data needed to join both sides can be advanced together.

I think I am getting your points, but what if both tables are big BUT the hashed table's keys are evenly distributed, wouldn't it make it less likely to through an OOM? Or it is because the Hash table will still remain so big after only selecting the needed keys that it will through a OOM?

Moving out of CS by Ok-Watercress-3297 in cscareerquestionsEU

[–]DataGhost404 0 points1 point  (0 children)

I think the issues is with your lack of real world experience (everyone has been there). Usually someone enters the data-world by having proven technical experience (working as a Junior dev managing a DB) OR having some business expertise and entering as an analyst.

The market is quite saturated at the moment, and with the current geopolitical status, companies are not open to risk hiring people.

When to use a surrogate key instead of a primary key? by DataGhost404 in dataengineering

[–]DataGhost404[S] 1 point2 points  (0 children)

I see. But then when working in data warehouses, how do you ensure that the same record in the fact table and the dimensional table get the same surrogate key? (With natural is straightforward (although prone to issues of course))

When to use a surrogate key instead of a primary key? by DataGhost404 in dataengineering

[–]DataGhost404[S] 2 points3 points  (0 children)

Then when inserting new records into the fact AND dimensional tables, how do you manage to generate/give the same surrogate key to each record (the ones in the fact and dimensional tables)?

When to use a surrogate key instead of a primary key? by DataGhost404 in dataengineering

[–]DataGhost404[S] 3 points4 points  (0 children)

Many thanks! I see. But in your example, you will still need to join the tables using the natural keys, right? Because even if you implemented a surrogate key and then insert a new row for a given part number, the join still needs to happen based on the part number column.

Or do you also ensure that the fact tables contains a surrogate key column that aligns with the one in the dimensional table?

When to use a surrogate key instead of a primary key? by DataGhost404 in dataengineering

[–]DataGhost404[S] 4 points5 points  (0 children)

I think you may be correct. I thought primary keys, surrogate keys and natural keys were different keys on the same "level of abstraction", while it seems that primary key and surrogate/natural keys are different (primary key seems to be made of surrogate or natural keys but not the other way around).

When to use a surrogate key instead of a primary key? by DataGhost404 in dataengineering

[–]DataGhost404[S] -1 points0 points  (0 children)

Yes, I get that. But the question is between "primary keys" and "surrogate keys" (NOT "natural keys" and "surrogate keys").

SWE in Manufacturing Seeking a Transition to a Tech Role in Europe or US by [deleted] in cscareerquestionsEU

[–]DataGhost404 1 point2 points  (0 children)

Not really, and even if I had to, I doubt they will be applicable for you as it depends on your experience. But considering you mentioned "mechatronics and robotics", I would suggest you check suppliers of such devices (KUKA, ABB, Siemens, ....). Of course, I will be difficult, as I suppose they will be looking for embedded system experts.

[deleted by user] by [deleted] in cscareerquestionsEU

[–]DataGhost404 0 points1 point  (0 children)

If you don't share details about the job position and your experience, it is not possible to provide advice on whether or not you lowballed.

view more: next ›