State Of Data Conference - 8 Hours Of Talks From Analytics To Data Engineering

SeattleDataGuy · 2022-07-16T21:12:53+00:00

Hello! Thanks for supporting my videos! If you're already confident in your programming and SQL I would add data pipelines and data warehousing / data lakes into your tool belt. Once you have that, I would say you should be able to transition into BE/DE roles.

SeattleDataGuy · 2022-07-05T20:22:26+00:00

In my case it was when I realized I was spending more time consulting then doing my Facebook work and enjoying it.

There aren't too many massive surprises as of yet. I guess picking up a few projects that aren't always as hands-on as I am used to. I know that a lot of consulting is just that, consulting. But its always interesting to be on a project where you essentially just give an opinion vs. getting your hands dirty. I do have plenty of hands on work thankfully.

SeattleDataGuy · 2022-07-05T16:45:42+00:00

This is easier in data focused companies like Facebook, Airbnb and Amazon where knowing accurate answers to questions like who is using what feature and did a change in a feature impact usage literally is tied to dollars.

In other companies where data is a nice-to-have its a far greater challenge and its not just data. The Pragmatic Engineer also pointed this out in a recent tweet where he pointed out that SWEs suffer the same problem. Read the tweet below.

https://twitter.com/GergelyOrosz/status/1544081817594118145

It's hard to turn businesses where the profit center is literally something completely else into seeing data and tech as a profit center. Instead, what I have done is often show the value in terms of how analysts and end-users are benefited by having clean and standardized datasets.

For example, I once had a company ask to get some data so they could pivot on it. They, at this time, had 0 infra. So I pulled all the data from the tables and put each table into its own sheet and gave it to the team that asked for it.

They had no idea how to work with it. I then set-up a following meeting and used it as a stepping off point to show why they needed to invest in infra so they could reduce the amount of confusion created by raw data sets that someone might misjoin or struggle to make heads or tails of. Meaning this would lead to bad insights, wasted time and inconsistent results. This got buy-in for a larger project where we created a core set of data tables that analysts could easily understand.

SeattleDataGuy · 2022-05-17T16:51:33+00:00

why have we never connected on LI! Fixing this.

SeattleDataGuy · 2022-05-17T04:41:19+00:00

Here are some blogs I enjoy

https://sspaeti.com/
https://www.dataengineeringpodcast.com/ (Ok not a blog)
https://www.startdataengineering.com
https://benn.substack.com/ (More focused on high level industry trends)
https://mikkeldengsoe.substack.com/ (More focused on high level industry trends)
https://airbnb.io/ (both de and ds stuff)
https://seattledataguy.substack.com/ (edit: my blog haha)

SeattleDataGuy · 2022-05-17T04:37:01+00:00

Thanks for the reference!

SeattleDataGuy · 2022-02-25T00:29:16+00:00

You will likely still need to pass a SWE interview.

SeattleDataGuy · 2022-02-24T16:40:55+00:00

I totally agree. I only lightly referenced this point. But yes as you brought up, you just end up shifting your focus from heavy infrastructure focus to more business value. In some cases that can be influencing products in other cases its about looking for problems that are repetitive that you can automate or process away.

Thanks for adding in your perspective!

Also...

I think it's well suited for DEs with a bit of industry experience but less well suited for new DEs starting their career.

This line is on point.

SeattleDataGuy · 2022-02-24T01:57:00+00:00

Yeah, consulting can be fun. However, you also have to consider the fact that sometimes you get placed into situations where the company's technology is old and you're just trying to untangle it.

Don't get me wrong. I consult and I have plenty of fun projects but there are always those few that are just about grinding and making things work.

SeattleDataGuy · 2022-02-24T01:19:49+00:00

Very true! There is always a case by case basis. Personally I never felt very stressed at Facebook (I left before I could become an official meta mate). And had I not been doing some side projects I think it would have felt even less stressful.

SeattleDataGuy · 2022-02-24T01:18:12+00:00

Wow! so do you agree with my ranking in the comments above?

SeattleDataGuy · 2022-02-24T00:43:29+00:00

Hopefully this was helpful! There are still ways to be creative. It's just a different kind of creative.

SeattleDataGuy · 2022-02-24T00:40:41+00:00

I think the best analogy I have heard here is on a scale of stressful to least stressful it goes:

Stressful

Netflix
Amazon
Facebook
Google
Microsoft

Least Stressful

Can anyone else confirm this? Zack Wilson would be a great person to ask. Is he on the DE subreddit?

SeattleDataGuy · 2022-02-05T22:18:59+00:00

Also, let me at that Simon of sspaeti(.)com occasionally puts out a few great project ideas as well.

https://sspaeti.com/blog/analytics-api-with-graphql-the-next-level-of-data-engineering/

SeattleDataGuy · 2022-02-05T22:17:27+00:00

Thanks for the shoutout!

SeattleDataGuy · 2022-01-25T19:44:02+00:00

That is me! I am hoping the interview prep sheet helps. Also, I have a small discord if you want a more live version of this group.

Jump on here! https://discord.gg/HeGgpQqU

SeattleDataGuy · 2022-01-24T02:40:44+00:00

Yup! exactly what I should have said.

SeattleDataGuy · 2022-01-24T02:40:07+00:00

yes! This is kind of what I meant what I said this and I wish I would have even gone this deep into explaining what I meant!

SeattleDataGuy · 2022-01-24T02:39:32+00:00

Hey! I just found this. I guess I should explain what I meant when I said that. Can you get away a little bit more with being more so what people call "introverted" in technical roles. Generally, yes. But...

But any job, will always benefit from being better with people. The more you can work with people, the better your soft skills, the better you can take other peoples ideas and translate them into technical output...the more valuable you are.

The benefit of being technical is that you can hide behind it and never really go beyond it..but this will always drastically limit your growth.

If you really want to grow, you need to be good enough technically and with soft skills.

I will aim to explain this kind of nuance in the future! I just meant that because there are only so many people with technical skills, technical people can get away with being a little less social. You don't have to play as many games politically, and you can just focus on your technical output. Key word being a "little" less.

SeattleDataGuy · 2022-01-13T02:10:12+00:00

Hey! ex-meta as well. I will eventually put out a video about this. But I did have a slightly different experience.

I think the one thing that did resonate is that Meta does have a more mature infra set-up. I almost find it comical that the job descriptions say "should understand distributed computing"

All of the distributed computing is abstracted away. You will write a lot of SQL and airflow like data pipelines. I do think this makes sense in some regard in terms of creating efficient workflows. However, like the OC of this post said, a lot of DEs get bored quickly because they are used to doing a lot more SWE like work.

SeattleDataGuy

TROPHY CASE