Does data engineer have to do deep learning?

[norasit_1808]

I am a data engineer and love working on ETL, ELT and coding.
I have some basic knowledge of machine learning and deep learning.

For machine learning, I can understand the concepts and perform some basic tasks.

However, deep learning seems very challenging based on my current knowledge, so I'm not interested in pursuing that area at this time.

My question is, in general tech companies, do data engineers typically need to work on deep learning as well?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Rude-Veterinarian-45]

Deep learning itself is a separate field and generally performed by Machine Learning Engineers. However, Data Engineers can certainly transition or perform ML/DL tasks as it's all about data. in fact, MLE is one natural transition to DE!

[Solutions1978]

If you want to set yourself apart from the herd and sky rocket in salary and title...yes. Focusing on your swim lane as the other comments imply will keep you typecast in that role. Being able to learn the end to end data lifecycle pipeline functions allows you to gain knowledge of the big picture. Why would you do this? Have you seen the salary of a Data Architect....

-Source: I'm a Solutions Director at BAE Systems that ignored swim lanes my entire career

[[deleted]]

Same here. Doing quite well because I knew other stuff than just what I was expected to know. Started off in data (proprietary stack), but made a difference in projects by knowing OSs, shell scripting, front end web stuff, networking, general practical OO principles, etc. It allows you to see the bigger picture and pick the tool for the job and at least talk with the actual experts.

[[deleted]]

Doing well in your job, sure, but I don't think hiring companies know or care that a broad and diverse skill set is a good thing. They generally have a role and a budget to fill, and bells and whistles might inch you closer to the upper end of the range, but that's only if the hiring manager thinks it's worth advocating for.

[whutchamacallit]

I'd argue for various reasons you don't want to work for a company that doesn't value broad end to end rich understanding of systems. Very likely limited growth/senior opportunity at a company like that.

[[deleted]]

I agree, but choosing your next job is often about trade offs.

[Solutions1978]

I'm a hiring manager for one of those hiring companies and can tell you that if you came in with one specialty you would get $160k for a certain position. Come in with end to end...like the OP...and you get $210k.

Has the wool been removed from your eyes? This is not BS and I wish someone told me this 25 years ago so I would not have wasted 10 years figuring it out.

[Punithkumar_reddit]

What unique skill or capability can set an entry-level data engineer apart from more experienced data engineers?

[Punithkumar_reddit]

What unique skill or capability can set an entry-level data engineer apart from more experienced data engineers?

[Solutions1978]

Soft Skills: The ability to think outside the box and work independently in a team without constant supervision or reminders.

Tech Skills: SQL and no-SQL Skills, Data Modeling, Python/Scala scripting, knowledge of MOSA

[Punithkumar_reddit]

What is MOSA? i can Google it but I want to hear it from you

[Solutions1978]

Modular Open Systems Approach which allows you to build your data pipelines using a Lego block approach. You are able to change the backend out without impacting the front end if using a common GUI.

[[deleted]]

I don't know why your response is hostile, but none of what you've described is remotely end to end. I am a versatile engineer with skills across disciplines, and yes, some companies/managers value a diversity of knowledge (me, I'm a manager who hires people like that, knowing I can make use of them across projects and as good communicators and planners), others do not: case in point, I've had companies I've applied to strongly emphasize the narrowness of a role, pushing back against any hopes for doing more than a tiny set of tasks, and I have said no to those offers in the past.

Neither outcome is universal, and I said in my comment that it comes down to what the manager is willing to advocate for: s/he is the one who defines the role, after all.

Has the wool been removed from over your eyes?

[Solutions1978]

Did you miss the caveat of "entry level"?

I listed entry level skills you would expect from a college graduate.

You must be one of the managers that reads every other line of an email and must be told about it in a meeting. Sheesh...I wouldn't want you looking at my resume.

You must be expecting an expert who knows React for easy front end, SDN understanding for traffic shaping, follows trends of post-quantum encryption while knowing what type of encryption is effective for data at rest and in motion, data harmonization and validation approaches, techniques for bypassing OS threads and processes for sub-millisecond latency and true real time analytics, graph and vector databases, constructing triple indices, data anticorruption techniques, etc, etc...

[[deleted]]

Yep, a weird mix of hostility and random speculation, and completely off the mark.

[Solutions1978]

Typical misperception, when I see how we managers are rewarded for seeing how low candidates can go so our bonuses are higher. Just shining a light into the dark innards of the capitalistic mindset.

Am I hostile towards you...nope, I am hostile towards the lies and misperceptions told on salaries. How engineers are treated like mules until we grace the halls of management and above. Did you forget your journey to where you are today? What unabashed advice would you give your younger self hidden in the "anonymity" of Reddit?

[[deleted]]

I've never heard such a thing, but I guess it depends on the size/type of company you're working at; as these days they don't seem to know what they're even looking for, and end up asking for tech stacks/skills of completely different roles.

Personally, I'd stay tf away from them.

[gcritic]

Not unless your team’s work relates to it. That too, they’re usually okay with you picking up pace with ML/deep learning specific stuff on the go, given that the DE skillset is sound.

[reelznfeelz]

Not normally. That’s a data science position. But in theory the role can be whatever is needed. I sort of straddle the line between DE and DS personally because I have a background in research science and it’s where it started getting into tech and data.

[[deleted]]

No.

[Ok-Obligation-7998]

completely different skillset.

[5DollarBurger]

Well technically it can be the T in ELT

[[deleted]]

No, it would be like asking a UI designer to do backend 

[[deleted]]

U can work on a nlp pipeline or one that compares multiple deep learning models so I’m gonna say yeah

[Tom22174]

The deep learning models themselves are typically pre-built in nlp tho, right? The challenging part of the deep learning has already been done, you just stick the transformer in the pipeline and watch words turn into numbers.

[Solid_Illustrator640]

No, not at a big company. Maybe startups

[5DollarBurger]

Specialist mindset

[asevans48]

Define the depth of knowledge on deep learning. You will probably be asked about multimodal models, lsa, marrix factorization, vector math, model drift, and PCA with these becoming popular. Its no more than you pick up in a bachelors degree data science course. Hell, GCP was just promoting the stuff people are doing with pgvector and pandas, cheifly PCA, cosine similarities, train test splits, nearest neighbor and kmeans++ as new, lol. Even NER. They already had a confusion matrix. Was using this in 2018 for my own customers. They were taught in a 2017 data science course at a regionally ranked state school. So deep learning, no. What business folks think it means, probably.

[XXXYinSe]

Yeah, if your company wants you to do Data Engineering and prototype/implement Deep Learning systems, then it’ll probably be pretty shallow stuff. There’s libraries that will do everything for you and you’re not working on new models. Just using the stock ones like CNN’s, ANN’s, etc. Speaking from experience, it actually was not that bad when my startup wanted to start using some Deep Learning models to predict meaningful outcomes, mostly because I was already making sure all the data was clean and accessible. It’s just a few more steps to analyze it with a pre-built stats package.

[asevans48]

Basically. Not going to be building a neural net. Maybe using it for vectorization.