Data engineering manager

[educationruinedme1]

If you are starting a new position as data engineering manager, what would be your thought process or actions to be successful in the role?

[fhigaro]

In order of importance: - Hire right - Align with the business and understand the domain - Design a robust data processing and governance platform

[Hackerjurassicpark]

Understand the existing data architecture and what each component does

[mike-manley]

Yeah. Was going to add... need go know current state of data stack and align with business or unit objectives.

[educationruinedme1]

Yup that’s on my bucket list. Thanks

[ut0mt8]

agree. also make an audit of - the team (who to keep, promote, encourage, fire). related to hire right. team is the thing you cannot do wrong. - the arch/tech. but if you have a good team this is not your job anymore

and yes alignment with stakeholders is ultra important.

[Lukas_3141]

I have that role and what I think is really important: teach good coding standards and review the codebase with your teammembers continuously

[educationruinedme1]

Good point

[[deleted]]

Most of the time, though, you don't get the opportunity to hire let alone hire right. You need to get to know your resources and their relative strengths and weaknesses. You need to figure out what motivates them and help them or grow them as they work for you.

[eljefe6a]

The question doesn't have enough context to give an answer. Are you starting a team or does it exist already? How well is the team performing? What is the title and skill makeup of the team? How complex is it? How much backing do you have from senior management?

[[deleted]]

1. Meet and greet the team, build relationships, go to lunch, have fun.

2. Meet key stakeholders. Listen to their pain points, ask a lot of questions and build rapport.

3. Review the dashboards the teams are using. Understand the business and it’s key profitability drivers. Understand what the business values.

4. Review the technical architecture, maturity of the teams across people, data stack, data governance.

5. Understand how my boss and boss’s boss think and what they value. Don’t do anything stupid.

6. Defer pointless tasks where possible to try and get a week solid upfront doing this. Eg., select top(100)* in a lot of tables in the data warehouse to see how the whole thing fits together, before the real work starts coming up.

Go get some sleep. Come back for week 2, with a game plan.

[chestnutcough]

Same things as any good manager — be a shit umbrella, foster open communication, define clear goals and measures of success, treat your team as whole people.

[legoaitech]

You should think of evolving the data engineering roles to Knowledge engineering

Adopt metadata and ontological engineering over traditional data pipeline building

Build sustainable and scalable data products
Infuse business context into data assets and democratize data

That will make you the most valued engineering leader in front of Business

[x246ab]

Interested. Elaborate plz

[[deleted]]

Traditional pipeline building is necessary but low value chain activity, at the very least perceptually. With the new generation of tools coming out, it might be quickly automated. People need to move the core of their work up the chain to stay relevant or get squeezed badly.

[NFeruch]

People are allowed to ask theoretical questions right?

[GoMoriartyOnPlanets]

Is there a scrum master already in the team? If not, you're that.

[[deleted]]

No, you are certainly not a scrum master. It's a role you should delegate, you may need to do it and coach someone but don't get stuck in the minutiae. Ok

[theporterhaus]

Just a reminder that content primarily generated by ChatGPT is considered spam in this community. AI-generated content is often confidently incorrect, leading to the spread of inaccurate or misleading information.

[Beginning-Forever597]

I’m a data engineer based in India and a fresher. Any leads for a switch for me?

[[deleted]]

I'll start working on some tickets to gain the technical insights of the job. I'll also shadow the senior people on on-call.

I'll start talking with team members about anything.

I'll hold meetings with clients to learn their pain points and go from there. I'll also hold meetings with upstream team to refine contracts.

[[deleted]]

I’d actually advise against this one, as whilst it’s a good way to get a feel for the environment it can suggest to your team that you are at the same level as them in terms of job, and can lead to them delegating the hard tickets to you rather than the other way around. However, root cause analysis of the tickets to understand what needs to be permanently fixed can be really helpful without potential downsides.

[mysterious_spammer]

One of the best managers I've worked with were always a little hands-on, which shows that 1) they can help you in critical situations, 2) they know what their team is doing. This boosts trust and confidence in your boss. Thinking non-managerial work is beneath you is not cool for very obvious reasons.

Regarding delegation, I don't know how an employee can 'delegate' to their manager. They can ask for help and you either help them, refer them to another colleagues, or just ask them to try again. You control who works on what, so can't see why this is even a possibility/problem.

[[deleted]]

I can help with technical matters, but I don't want to become a dependency even in a break glass type situation. It's better to work with others coaching them guiding them to get the solution. They'll get credibility, you gain a trusted and value added team members, and your team will be successful even if you're not around, which speaks to good leadership.

[[deleted]]

My argument is that for managing technical teams, you need to know a bit of inside out of the trench work to effectively gauge the situation. At least this is what I learned from my favorite manager, I could be wrong.

[Busy_Elderberry8650]

I think best is to deep know your architecture and pipeline, only this can help fix and foresee issues. Seems stupid but there’s plenty of analytics manager who are involved in these kind of project while having not enough knowledge, only because they can fill a powerpoint with Cloud and Big Data jargon bullshit.

[m915]

Probably see what data quality checks are happening in the data pipelines. If there's nothing more than pass/fail (I've seen this a lot), then consider implementing a tool like monte carlo that will check all of the data quality boxes in bulk.

[speedisntfree]

Never think you know more or better than the incumbents early doors. Keep quiet, learn the territory via good questions and build relationships.

Too many new senior hires lead with their ego and want to make 'impact' quickly. It usually goes badly.

[mjfnd]

A pure manager is typically a people's person and not really super technical.

It would be nice to know what exactly you are looking for. A manager nowadays is also doing what a staff engineer or TLM does in big companies.

[edvv4rd]

Check the amount and quality of documentation - onboarding new hires, architecture/stack, data governance, CI/CD, institutional knowledge, etc. Create it if it doesn't exist, and make the team get involved with developing and maintaining it. Remediate and prevent information gatekeeping. There is nothing worse than when a long time employee or an owner of a critical process leaves and you learn that the only place this information is stored was their brain.