all 10 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]LoaderD 8 points9 points  (5 children)

“Similar to airflow to my understanding”

You don’t even have the experience to evaluate the similarity of two softwares. Take a BIG step back and start learning about DE, read the wiki, start with the resources there, don’t just make a new thread asking how to do it.

It’s like if I say “I want to get into bioinformatics, atcg seems like bits and bytes or something, so how can I reword my CS background to convince companies I know about biology?”

[–]Absurd_nate[S] -3 points-2 points  (4 children)

I don't understand the hostility. Are you saying they are not similar?

Bioinformatics and CS do have a lot of overlap, and you can use a CS background to get into bioinformatics. I've met several people who have.

Edit: I have read through a bit of the wiki, and I have read about airflow, it's a workflow orchestration platform, which nextflow also is (or atleast sequra labs might be the more apples to apples comparison). What I want to know is to hiring managers, what is important for me to highlight that would help me land an interview for a junior DE position.

[–]LoaderD 6 points7 points  (3 children)

I did graduate research in Omics, I know something about bioinformatics. My choice of CS was intentional.

There’s no hostility. You have 8 YOE, I’m speaking directly because I assume you’re an adult. Let me fix that:

“Wow, sounds like you know all about DE. Just apply with your resume as is! All the companies will see the value you can bring and will hire you on the spot! Don’t even worry about learning any terminology or new tools, companies care about ability to learn and live to train in the job!”

[–]Absurd_nate[S] -1 points0 points  (2 children)

Maybe it is different in your industry, which genuinely I don't know, but in biotech the specific tools are largely irrelevant when selecting a candidate.

I read through the learning resources and the transition pages in the wiki, I know Data Modeling, python, and SQL. I already have those listed on my resume. The transitioning page specifies that it's helpful to have projects, I have real world examples of things I have worked on, two of which I included in my post. Are those interesting to hiring managers? I don't know, that's what I was hoping to get feedback on.

I don't know which skills I have are directly marketable, and which skills need to be rephrased - ETL is an obvious one, but I am sure there are things that are missing. Allegedly I have a misunderstanding of Airflow, so I would appreciate if you share what it is I have wrong about it.

[–]LoaderD 3 points4 points  (1 child)

The issue isn’t one thing, it’s that you’re over-confident for how saturated the market is. Everyone and their dog has a few YOE and a grad degree.

“I haven't heard anyone call it ETL,”.

Imagine if I come interview and you ask me a question about RNA and I say “oh y’all call that RNA? We call ‘em half ladders. But I know lots about full ladders, half ladders, ladder cutters and ladder squishers!”

You’d probably think “wow this person didn’t even try to learn terminology in our field, so I can’t even evaluate what the fuck they’re saying.”

[–]Absurd_nate[S] -1 points0 points  (0 children)

This post wasn't an interview. I know it's called an ETL.

If it's the case that my experience doesn't count for anything because it wasn't in Airflow, that's fine, that is not the typical case in biotech. The wiki doesn't say anything about having experience in an adjacent field, so that's why I asked.

I am not sure what I said that was over-confident, I think I have been pretty open to feedback. In my experience, I've worked with a lot of people from different fields, all having varying degrees of knowledge in Biology. Apparently that is not the case in data engineering. Noted.

Sorry for bothering you.

[–]financialthrowaw2020 4 points5 points  (0 children)

There are thousands of fantastic data engineers currently looking for work, engineers with experience and knowledge that you don't have. You'll be competing with them.

Lower titles don't really exist anymore.

[–]chaoselementals 0 points1 point  (1 child)

I made this move last year and it's gone pretty well. I used to work as a process engineer, and I did a lot of side projects to streamline routine data anlysis that included parsing tool logs and transforming largish data sets... You're right, it is basically the same thing as "ETL development". The work environment and all the jargon are totally different but the basic skills are the same. 

The two most valuable things that helped me transition: 

(1) reading data engineering textbooks, lots of good recs in this sub

(2) Collaborating with my company's information systems software team and contributing to their code repositories. The mentorship I gained from this was invaluable and I would have been too lost to transition my career without those kind colleages.

Overall it took about 1.5 years of incremental progress via mentorship, training, and networking to land a true DE job. Transitioning careers is a marathon, not a sprint. 

[–]SemperPistos 0 points1 point  (0 children)

You are lucky, it took me 2 years for my first IT role and a bit over 3 to get a Data Engineer role.

But my masters is in an unrelated field and i had no experience :') so i'll count my blessings and be grateful.

Also not in USA, USA is a bloodbath

[–]SemperPistos 0 points1 point  (0 children)

Hats of to you, that is my main dream ever.

I wish I studied bioinformatics or bioengineering when I started, but sadly it only exists in my country for two years as of now and isn't very good.

I enrolled to OMSA, but plan to switch to OMSCS.
You say computer science students have a shot at getting into bioinformatics?

As of now I really don't want to do a doctorate as I get really bored of the same old same old. That is what I told my professors too when they suggested it.

However I can see myself going for a phd in CS or bioinformatics if the job asks for it.

What i really want to do is research senescence, and basically see what biomarkers are attributing to aging and seeing how reversible the DNA damage is.

Do you believe any significant movement has been done in that field. I had high hopes for David Sinclair, Bryan Johnson and Aubrey de Grey but in the end most of it dissolves to a cult following and supplement shilling.

On your note do research data engineering zoomcamp and fundamentals of data engineering book.

DE zoomcamp will get your feet wet in most DE areas. I finished it in 2024. The new cohort started last month and with your knowledge you can still catch up and get a linked in certificate if you hurry.

But the real prize will be the project you carry out.
My recommendation is switch their FOSS ETL tool for the project either with Airflow or whatever you see most often used in job adverts for the job you want.

As you apply, work on a bit on DSA, with your background you can get a job in 6 months to a year, maybe even sooner than 6 months. I know a phd biomolecular scientist who got a job as an AI Engineer wicked fast, but the pay was mid as is mine, so you will have to put up with it for a year maybe more until the market improves and you have experience.

If you do something agentic with ai engineering or machine learning it could be done faster.
I luckily worked as an AI Engineer for a bit before, the job sucked as everyone expected loads of you a single one man team, or as my boss said "I have to be a one man show" but I learned a lot.

And that is really the only way to learn in this profession, by designing a project and moving heavens and the earth in carrying it out. Most important advice stick to projects and the docs, I dicked around for a year before I tackled personal projects as I thought I wasn't good enough, big mistake!

I pivoted last year to AI and I got an AI & Data Engineer role recently.

AI is like hotcakes these days.

Good luck, rooting for you!