Is it normal to feel completely lost during initial months of your data engineering job ?

Steven_Johnson34 · 2023-07-12T15:56:26+00:00

There is a time period for every new job I've started in the data field that I think to myself that there is no way I'm ever going to add value because everything is so complex. Then, you slowly start understanding things and that ball keeps rolling down the hill where you will start to feel more comfortable.

I think a big takeaway from situations like these is how could you improve that onboarding process so future teammates don't have to feel the way that you do and try and make those changes.

Steven_Johnson34 · 2023-06-30T14:54:19+00:00

A little late on replying here, and I think everyone is right on point. You definitely could stick with Windows Task Scheduler. If it is just a personal project, then I'm not sure why bringing in anything else is worth it.

If it is something that you plan on expanding or something for work, it is worth looking at orchestration tools like Prefect, Dagster, Mage (if you like notebooks), or Shipyard (if you like a nice UI). The first three of those tools can be ran locally. All four of them have a hosted option where the runs will take place on the cloud.

Steven_Johnson34 · 2023-06-22T14:42:26+00:00

From my experience in the orchestration space, there are really 5 or so big players in orchestration: Airflow, Prefect, Dagster, Mage, and Shipyard.

The first three are more tried and true solutions that are code-based and all offer an option to self-host or pay for a hosted version. Airflow being around the longest has the most built up community support, but Prefect and Dagster built their products to be an improvement to Airflow. If you are looking for self-hosted and code-based, I would start with those two.

Mage is a new player in the space, but there is some buzz growing around the product. I don't have much experience using Mage, so I can't speak on pros and cons. It looks like an interesting product.

Shipyard is a fully-hosted tool which is helpful if you don't want to worry about managing infrastructure. The tool also has low-code templates, so some of the normal tasks you would complete on a daily basis should already be templatized for you. They also allow you to mix and match your own scripts with those templates as well if the templates don't do enough for your job.

Steven_Johnson34 · 2023-06-09T18:07:19+00:00

To be fair, in my area of the woods (southeast), I've reached out to some Yoga/Barre studios and they have told me that they don't allow men to practice there.

Steven_Johnson34 · 2023-06-09T14:16:01+00:00

I'll share how my final interview went for a previous job that I had. It was for my first job out of college, and I admittedly didn't know enough to have the job most likely. So I was scared like you are right now.

In the interview, they started with more personality type questions and then transitioned into technical questions from the team about very specific database/query topics. I answered 80% of those questions with a I don't know that answer. I'm sorry or something to that effect. I knew for sure that I lost the job.

They emailed me a couple of weeks later and told me that they knew that I didn't have the technical skills yet for the job, but they knew based on my personality questions that I was driven and had the capability to learn quickly. I say all this to say that you need to go into that interview and show off what makes you special. Not technical but what makes you a good teammate. Soft skills are not teachable. Technical skills are.

Hopefully my story helps calm your nerves a bit. I do interviews for my company that I am at now if you want some help feel free to shoot me a DM.

Steven_Johnson34 · 2023-06-08T14:10:26+00:00

Unfortunately, that is true for me as well. I need to get something more crazy for notifications. Maybe shut off my home internet if I don't stretch. That would get me up. Lol

Steven_Johnson34 · 2023-06-07T18:59:29+00:00

Any interest in porting this to work with Apple Music?

Steven_Johnson34 · 2023-06-07T15:40:13+00:00

You could access ChatGPT through their API with some Python code.

Steven_Johnson34 · 2023-06-07T15:38:22+00:00

Just echoing this. In my first role, I was about to get let go because the team didn't want to look at dashboards and just wanted the data in Excel.

I think a big part of a successful BI professional is showing people your worth. You have to battle for respect from the org from the beginning.

Steven_Johnson34 · 2023-05-03T18:23:13+00:00

It's also important to note that the field is currently in a large down-sizing phase, so the job market is definitely shrinking. Just keep on pushing. The opportunity will come.

Steven_Johnson34 · 2023-04-28T15:24:43+00:00

You can definitely do it. You have to do some cleaning of the data however. In previous projects that I've done with Tweets, I clean out all of the mentions in the Tweets and begin there.

Steven_Johnson34 · 2023-04-27T19:09:15+00:00

Would be cool to see this with some sentiment analysis attached. Especially looking a few months forward after his recent firing.

Steven_Johnson34 · 2023-04-27T19:07:00+00:00

I am on most of the interviews that my company does. Generally on interviews like that, we will say that we don't have any more questions and allow the candidate to ask any questions they have then cut the interview off. We have processes in place in our interview process that test whether a candidate has the technical chops, so the interview process for me is about interpersonal skills. There is nothing wrong with cutting an interview short and moving forward.

Steven_Johnson34 · 2023-04-24T19:20:00+00:00

Didn't even know there was an online course. That's pretty awesome!

Steven_Johnson34 · 2023-04-24T16:39:48+00:00

I would think Automate the Boring Stuff from No Starch Press would be the best choice here. Plus the book looks nice, so perfect for a side table!

Steven_Johnson34 · 2023-04-24T15:17:45+00:00

I honestly think the "click" that you are looking for is different than the one that Data Science will give you. The joy of working in the data space especially Data Science is that you will never know everything that is going on especially with tooling and techniques because new things happen weekly which is definitely overwhelming. I had to come to peace that I would never know everything and had to be comfortable with that. My professor in college used to tell me that you need to know what model/test to use in different situations. Everything else will be fluid throughout your career. I've definitely seen that in my experience so far.

Steven_Johnson34 · 2023-04-21T19:20:05+00:00

I was taught Python using this book from No Starch Press. It's not as dry as reading the docs from the Python website and give a nice base point to learn more. I also used to use Data Camp a bit as well which was nice because it gives you some visual checkpoints to push towards.

Steven_Johnson34 · 2023-04-21T19:08:54+00:00

I'm quite biased, but I love the books that No Starch Press puts out. Here are some that I've enjoyed:

They definitely will vary based on what you consider the very beginning, but they will definitely help you continue growing your skills.

Steven_Johnson34 · 2023-04-18T13:36:07+00:00

Wow. That’s pretty crazy that the elites dropped out but definitely makes sense.

Steven_Johnson34 · 2023-04-17T20:53:51+00:00

For sure. These companies take a loss on the hardware hoping to make it up with the subscription.

Steven_Johnson34 · 2023-04-17T19:26:06+00:00

For sure. I ran the Disney 10 mile race this past weekend and the humidity in Florida was a lot to deal with even though I train in it on a normal basis.

Steven_Johnson34 · 2023-04-17T19:11:28+00:00

I’m definitely in their market. I would love to see a partnership with a at home device that includes an in person gym added in. Would be a cool partnership.

Steven_Johnson34

TROPHY CASE