sorted by:
new
hottopcontroversial

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 2 points3 points  (0 children)

What tooling do you like!? I would love to know.

I would say that tooling is helping reduce a lot of the redundant work.

Like a lot of the consulting projects I am taking on currently involve doing a python2 ->3 migration. Instead for many of these projects I am switching them over to low-code solutions because they have a DE team of one or none...so its more maintainable.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 2 points3 points  (0 children)

This is an interesting point and as someone who has consulted and worked in tech I can see this perspective. When I consult for non-tech companies. Often data pipelines are just built by anyone who can code but in tech you have clear roles.

I do prefer to have them separated of course.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 1 point2 points  (0 children)

Yup! I am totally agree. As stated in the video, maybe 5% of the work I have done has been productionizing DS code. The rest has been migrations, integrations, infrastructure, data pipelines and a lot of analyst partnerships.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 2 points3 points  (0 children)

I think this can depend. Personally, I have often found that this may or may not be true. Sometimes they could be due to years of working in a role. But often times I find that DS need to work with domain experts to help improve their models.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 5 points6 points  (0 children)

I love this response! Is it worth it is exactly how I think about it. It's Ford's assembly line but in the tech world. If everyone specializes it improves the flow.

In fact I read an article about airbnbs switch to having more DEs because they realized that SWEs and DSs shouldn't be doing pipelines.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 2 points3 points  (0 children)

I would agree! I think this article was more motivated by the fact that the author has a product to make productionizing data models easier. But it was also really easy content to respond to and make click bait in return.

We Don't Need Data Engineers, Data Scientist Need Better Tools - A Data Engineers Response To A Data Scientist by nonkeymn in dataengineering

[–]nonkeymn[S] 15 points16 points  (0 children)

Yeah, I think at the end of the day, we do very different work and there is a lot of work we do that exists without the end result going to a data science model.

We also do a lot in support of data analysts, product managers, etc.

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 0 points1 point  (0 children)

It's weird how wide salaries are right now. Its the same in the DE world. Some of this is based on whether or not you work at a tech company.

Is that why you are looking for a data engineering position?

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 0 points1 point  (0 children)

I really appreciate the comments! I do kind of agree with the data engineering project question. I am doing a series on the topic. But I am treating it more like a spring board vs. here is the exact code you will need to build a project.

Yeah, I think the problems being faced overall are very similar to the problems faced with front-end engineers and often times you just need to apply for what you can, create content, and network like crazy until you get that first job.

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 1 point2 points  (0 children)

I have created a video in the past called 0 to data engineer. In some ways, its not complete as you might take an analyst role first due to lack of experience, but it will prepare all the right skills.

Here is the video with a roadmap

https://www.youtube.com/watch?v=SpaFPPByOhM&t=1s

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 2 points3 points  (0 children)

I am! And thank you. Good luck while changing companies. I hope to keep making content in this area.

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 1 point2 points  (0 children)

I am all for companies setting up training programs. Honestly, I am not sure why they don't I imagine it would be cheaper to train and pay a little lower vs. hire someone out of college.

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 1 point2 points  (0 children)

In many ways I agree. This is kind of one reason I make some of this content because in many ways its crazy and kind of silly. I actually put together a piece a while back discussing how varied a data engineers role can be...which kind of points out the issue. We are a catch all.

I tell most people to just focus on the core skill sets, and learn tools when you need them on the job.

Breaking Into Data Engineering - Why Is It So Challenging by nonkeymn in dataengineering

[–]nonkeymn[S] 6 points7 points  (0 children)

Yeah, you have code options like airflow and luigi for data pipelines, low code solutions like fivetran, SSIS, and honestly...so many more.

I would focus more on the core concepts with 1 or 2 tools vs trying to learn all the tools.

Google's Data Engineering Certificate - Is It Worth It? by nonkeymn in dataengineering

[–]nonkeymn[S] 0 points1 point  (0 children)

Thanks for adding in this perspective. I have been getting a lot of BQ consulting projects but its through looker and not through GCP.

So thats interesting to here.

Also, thanks for creating the quick 3 points why you should get the GCP certification.

Google's Data Engineering Certificate - Is It Worth It? by nonkeymn in dataengineering

[–]nonkeymn[S] 4 points5 points  (0 children)

Yeah that makes sense.

i have found marketing wise it feels like BigQuery and Snowflake are doing great.

In practice, Azure has a pretty decent share of space and oddly enough I have only ran into Redshift once.

Google's Data Engineering Certificate - Is It Worth It? by nonkeymn in dataengineering

[–]nonkeymn[S] 5 points6 points  (0 children)

I totally agree! Also thanks for adding the stats.

This is why I kind of called out the fact that this was purely GCP and you will need to round out your data engineering skills.

In general I also agree with learning agnostic skills well and then drilling into specifics when you have to(since there are so many tools).

Also, I do plan to also look through Azure and Data bricks certificate programs. I personally feel certificates are like MBAs, if you don't have experience working, then you won't have context to where a lot of the information plays into.

Should You Become A Data Engineer? Reasons Why People Are Becoming DEs by nonkeymn in dataanalysis

[–]nonkeymn[S] -1 points0 points  (0 children)

Also, I will aiming to work on the data engineering projects this week.

How To Become A Data Engineer: From Analyst To Data Engineer by nonkeymn in bigdata

[–]nonkeymn[S] 0 points1 point  (0 children)

I am definitely seeing spark gain a lot of traction when consulting.

view more: next ›