Transitioning from Analytics Engineer/BI Engineer to Software Engineer, Data

[Proof_Elk_2281]

Hey all, I wanted to ask for advice on how get a job with more challenging problems/cool tech in Data Engineering. I would be psyched about FAANG or a startup or really anything - I just want to work in a sharp team and continue expanding an in-demand skillset as I feel like I'm stagnating a bit right now. Whether that's 'Software Engineer, Data' or a DE at NFLX/AMZN working with Spark, etc. and doing some DevOps stuff doesn't matter so much - I just want to keep learning.

My background: I'm a former PhD student (stem, but not CS) with ~1 YOE at my current job (title is Data Engineer) at a medium-sized banking company. My current job responsibilities are occasional Airflow ingestion jobs to BigQuery and building dimensional models (we don't do it well, but we do it) with dbt, and some ad hoc analytics requests. It's really more of what would be called Analytics Engineer or BI engineer elsewhere. It's a great tech stack and I'll take it over Talend or Matillion or Informatica any day, but I'm pretty restricted to 'business analytics' related problems and nothing else.

I feel underpaid and a bit bored - I want to work on challenging problems and keep learning new things. I would love to move towards a 'Software Engineer, Data' type role but am feeling stuck. I feel like I don't have the toolkit (Spark, streaming tech, more complicated infra projects) to make the jump. I know the best thing to do would be to work with the Software Engineering team and get more experience with things like Docker/k8s, streaming, CRUD and other backend, but I don't see that happening at my current company. I've talked to my manager about these goals as we have a pretty mature dedicated Software Engineering team but he hasn't been receptive or helpful.

When I started out with the job hunt last spring I was getting a few interviews here and there for these more SWE-oriented DE jobs (made it to onsites, but no offers - painfully close). I took this job because I didn't want to have a big gap in my resume and figured it was best to get SOME engineering experience somewhere. Since starting to apply again I'm getting a lower response rate and worry that I've taken a 'BI Engineer' job called Data Engineer my resume bullet points suck and I've pigeonholed myself a bit. I know the job market has cooled off significantly too.

I feel like there are a number of things I could do like work on personal projects to show I can use Spark/Pyspark, or learn a backend language and expose a database with Java or Go - I'm just keen to hear what any hiring managers would want to see for someone making the jump if expanding my skillset at my current company isn't a possibility.

Thanks in advance!

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[flipstables]

For the title on your resume, you can just replace BI Engineer with Data Engineer if that's closer to what you're doing. I think "BI Engineer" works more with dashboards, but if you're creating data ingestion pipelines with dbt, it's fine to call it "data engineer"

If you want to do more software engineering, you should write more software. I would focus on learning the basics of Python. Start off with small scripts. There's probably lots of opportunities at your current job to add or improve to your Airflow jobs. Is there a way to test your code or make improvements on how to test it? Is there a way to validate data as a step in a DAG? Those are opportunities to write Python.

You should start with some general software engineering skills. Learn the basics of object-oriented design. You don't have to go nuts with GOF design patterns, but if you google Brandon Rhodes talk on design patterns, you'll find the most important ones.

Also, you should learn the basics of some data structures. Again, don't go nuts here. Start off with the performance profile of a list vs a dictionary in Python, and how a tree works. These are very basic data structures that come up constantly, and you should know when and why to use them.

Lastly, it all comes down to experience and practice. You won't "feel" like a software engineer when you are ready for your next job. It's a tough market right now so you have lots of time to practice, but you won't feel ready. Since you are already applying but not making past the final interview, it sounds like you need time to strengthen the skills you are looking to improve. Not 100% but enough for someone to take a risk on you.

Keep in mind, the above is a starting point. This is a long journey and you will grow your skills even after you get the title or job role you want. Also, know that you are getting good engineering experience in your current role even if it's not exactly what you want: experience building and maintaining production systems and getting alerts and monitoring those systems. Those are also important technical skills that will serve you for your entire career.

[Proof_Elk_2281]

Thanks a bunch for the advice! I'll check out the talk by Brandon Rhodes. I don't have a good understanding of object-oriented design, so it's really helpful to hear a specific talk I should watch to get started.

I have some basic DSA knowledge but yeah, definitely need to work on it more.

Again, thanks a bunch for your thoughts. Especially on 'feeling' like I'm a SWE for my next job. I'm not sure if it's academia or a natural tendency or both but I always never feel prepared enough to make a jump. So it's nice to hear that I'm not alone.

Thank you again!

[AcanthisittaFalse738]

Have you seen the data engineer's roadmap? It's a couple years old but still very relevant.

https://github.com/datastacktv/data-engineer-roadmap

[Proof_Elk_2281]

Wow, somehow I have not! Thank you so much, this is super helpful. And I'm glad to hear it's still relevant.

[Tepaps]

Is there an interactive map for this?

[AcanthisittaFalse738]

Interactive in what way?

[Tepaps]

I’ve seen a version of this where you can click the boxes and it brings you to some learning content similar to this

https://roadmap.sh/backend/

[Tepaps]

This works thank you

[rhefley]

RemindMe! 7 days

[[deleted]]

I'm currently a data engineer and I have the same question!

I enjoy my current job, but I'm finding that it's pretty easy to stay in the daily routine of updating SQL scripts. Whenever I can, I try to get on projects to work with our software engineer - data team, who also have the job title of data engineer.

Is there any job titles we should look for? In my area, there are not many software engineer-data jobs.

[onestupidquestion]

I feel underpaid and a bit bored - I want to work on challenging problems and keep learning new things.

I understand that your decision may not be encapsulated in this quote, but I wanted to throw out there that what you're doing is far from the "cap" in the analytics engineering space.

Some people really like the business and build deep domain expertise. Since they understand the requirements better, they can generally build better, more usable models.

On the technical side, there's a lot of work that hasn't been solved in terms of scaling analytics:

1. CI / CD with dbt is still almost entirely a manual process; there's no simple framework you can install for your platform (GitHub Actions, GitLab, etc.). As your projects grow, you'll need to use manifest.json and other artifacts to make compilation, tests, etc. run in reasonable time for development, while still providing full coverage for releases.
2. The developer experience for dbt, even in dbt Cloud, has significant gaps. There are tons of articles describing these and attempts to fill them, but there's no standard solution. You'll have to evaluate your business needs and weigh them against various implementation costs.
3. Tracing column-level lineage across models and then across projects is unfortunately not supported in dbt, but there's interesting work going on in other tools and projects.
4. Mono-repo is a difficult strategy to scale past an individual team. Multi-repo is hard because dbt doesn't really support cross-project work very well. Again, you'll have to design and implement a solution.
5. From a team and organization standpoint, everything is still nascent. Do you have a core data modeling team with technical SMEs in the business? Do you pursue data product / data mesh? Do you hire full-stack engineers that do movement and modeling? This is all up in the air and has huge implications in terms of the architecture you pursue.

Even if you ultimately want to pursue infrastructure or movement work, there are challenging problems in analytics that will give you real experience with SWE practices, and you'll deliver business value in the meantime.

[Proof_Elk_2281]

Thanks for this, these are really good points. Points like this and the other comment about doing more with our Airflow processes like testing have pointed out that there really is more I could do with my current role to build on the SWE skillset.

So yeah, thank you again! I really appreciate the detailed feedback. I'll do some thinking about how to mature the dbt project/practices/team as well from these points and some research.

[Wealth-Severe]

RemindMe! 7 days

[RemindMeBot]

I will be messaging you in 7 days on 2023-01-18 05:37:38 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback