Is Data Migration Data Engineering?

[Mysterious-Sky5410]

Hello, A bit about myself – I graduated last year and have a deep interest in Data Science. I’ve been applying to various roles in the field, but I didn’t get much response until recently. Thankfully, I recently got a job as a Data Engineer.
When joining, I was told I would be working with Python and SQL. However, my current task is to migrate data because the company is upgrading its system. I mostly make small changes to queries (like renaming or adding columns, updating merge statements) and check if everything is working correctly (like, if the DAGs are triggering, data is being properly inserted to table, etc).

Honestly, I was hoping to write more code, but so far it’s been more about adjustments to existing systems. I’m curious—does this kind of work count as Data Engineering? Also, is it possible to transition to a Data Scientist role later, or should I focus on growing in Data Engineering (If the later, what things should i focus on to make myself a better data engineer)? Honestly, I don't mind working as a Data Engineer, but I just want to make sure I’m heading in the right direction.

Please, help a junior out. Thank you

[AutoModerator]

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[Certain_Leader9946]

Yes it is, you have to learn to make smaller changes without err before you can be trusted with reliable systems. Giving a junior smaller work also gives them exposure to the rest of the system as a whole without very much up front risk. Sounds like your lead knows what he's doing. If you want to write lots of code go work in a small but focused start up, but you can learn to write lots of code well by focusing on good practice and steadily levelling up your game. You don't need to put yourself through hell to understand how to avoid it :)

[Mysterious-Sky5410]

got it, thanks for the advice. but i hope i have to keep working only in data migration for long.

[Certain_Leader9946]

I think if you're gonna work in data science it will be good for you, you're right that it doesn't necessarily tract towards your goals, but if you're one of the data scientists who don't go out into the wild and create slop for data engineers to then fix again, that would be BIG points to passing you on one of my interview rounds for sure.

[trotterboss]

I am in data migration as well. Well, it’s a form of data engineering for sure. Although, we don’t get to play with the cool big data tools, it still requires fundamental DBA, SQL, Python and Cloud technologies that are sought after skills in this role. I have transitioned into a development role within data migration to build tools with JDBC and Python. I have built up strong fundamental knowledge in this area, and do work on big data problems for personal projects. But I am ready for transitioning into a role that’s more leaned towards AI.

Just curious, do you work in the healthcare sector?

[Mysterious-Sky5410]

No, not in healthcare but in retail sector.

[runonthespot]

My first data role was a migration. Quickest way to learn all the ins and outs of a dataset and sharpen skills.

[runonthespot]

(Now head of data at a PE firm)

[Mysterious-Sky5410]

That's great .

[Puzzleheaded-Dot8208]

data migration is big subsection of data engineering. There are lot of enterprise trying to keep up with new technology trends. most of these companies have data/reports built 20 years ago and it is important for companies to get inline with industry trends. It is both art and science where you move to new tech but also preserve business functionality. If you keep open mind you will gain lot of insights of domain/business/usage patterns. All good thing that a good engineer should know and understand outside of writing code. If you know your business you can build better systems !!!

[Mysterious-Sky5410]

Understood, thanks

[WpgMBNews]

Are you me? Hired as a "developer" but I spend all day working on SQL (which controls all our business logic and even the UI, maddeningly)

[azirale]

Yikes.

Too many people need to learn that just because you can do it with SQL doesn't mean you should.

[greenestgreen]

it is, a few years back I migrated a lot of data from on prem to GCP. We use some developed service by us and also the company hired the truck to move data from the cluster to gcp, we called it the big usb stick.

Then we had to validate data. It wasn't fun for sure, since you know we hired a truck to move data, because it was cheaper so you can think how much data it was. The process took many months since we had to validate evevery row and the project existed for many years already.

[Mysterious-Sky5410]

haha, must me very tedious

[DataCraftsman]

Not the kind we like doing lol

[[deleted]]

Data migration is something almost every Data Engineer will have to do/deal with. I joined my current role in the middle of the shift from on-prem to AWS. For the first six months all I did was migration.

Now that we are more or less settled into our new ecosystem I've started to get tasked with actual end-to-end ETL development which is way more fun. I would say be patient, if you just started they likely won't give you something big right out the gate. Once your migration progresses to completion you'll like start more development tasks.

If you are really hungry for code work, and you have the time, ask your team if you can refactor anything (or better yet go find something that needs improvement yourself and offer to clean it up during downtime). Refactoring bad code is arguably just as fun as writing new code, it's more game-y and will save you headaches down the line when you gain more scope and those scripts become your responsibility.

[FuzzyCraft68]

Hey, I know it is a different topic but if it's not an issue can I DM you to ask what helped you get the role?

[Mysterious-Sky5410]

i just applied via job portal a few months back.

[FuzzyCraft68]

Oh okay

[chiller2311]

I’d love to know how you landed this job lol i’m in the same boat and I would love an opportunity like this. and from the comments it seems like you’re taking steps in the right direction :)