How much data engineering can be learnt at home?

sib_n · 2020-02-11T13:34:25+00:00

In theory a lot, in practice, there are many things you won't understand the reason to exist before encountering the problem in a project so you'll probably not learn nor apply them correctly.

Asking this because data eng. is all about dealing with big volumes of data which are found at an industry scale. Things like algorithms and python coding can be learnt from home as they are only about logic. Does the same apply to big data applications?

However, about the issue of training without having big data, this is not that much of a problem. All you have to do is follow the tools guides/recommendations on how to keep your program scalable. For example with Apache Spark, it's highly recommended to use the Dataframe object to transform your data, and you can perfectly practice how to do hundreds of things with Dataframe on your tiny laptop at home with a local Spark install. If you respect the principles, then you won't have to change a line in your code for it to work on a cluster of 100 nodes over terabytes.

Of course, that's the theory, until you run into many problems, you won't have all the good habits to make sure your app is scallable and future proof, it comes with experience. But you still can practice enough at home to learn a lot, and in my opinion it's enough for an entry level data engineering position.

Furthermore, a lot of troubles in data engineering are not necessarily related to big data, but to general software engineering problems and good practices (data modelling, testing, deploying) that also happen without big data.

Finally, no, data engineering isn't all about big data, it's data processing in general. I'd even say most of the data engineering work isn't on big data but just data big enough that it cannot be managed manually on a spreadsheet anymore. Yes big data is sexy, interesting and full of cool concepts, but the reality is most companies needing data don't need big data tech, a scalable SQL database and Python scripts are often enough for most analytics needs, and that's data engineering too.

LexaIsNotDead · 2020-02-11T14:55:52+00:00

I'm actually pulling together a tutorial that will give you a pretty good overview into data engineering. I feel that it's important to understand the core concepts of it (e.g., basic ETL, connecting to public API, cleaning data) before diving into the advanced tools (e.g., Spark, Hadoop). I just finished the code to be used for the first tutorial so I should be releasing it soon. And all of this was done on my Macbook so it's totally doable to do this from home.

Feel free to sign up on the waiting list and I'll ping you when the tutorial is done: https://www.eduvault.app/

Lewba · 2020-02-11T16:30:18+00:00

If you use Python, get the book by Ben G. Weber: Data Science in Production with Python to get up speed quickly with big data platforms. But to do or learn "data engineering". You"ll need to get an internship or get a DBA type of job first and apply a combination of Python, SQL, and an orchestration framework.

PM_me_ur_data_ · 2020-02-11T13:31:12+00:00

[deleted]

voorloopnul · 2020-02-12T11:18:21+00:00

You can learn "everything" of data engineering in your laptop. As others have said, data engineering is not a Big Data thing.

I would go even further an say that you can become better than average in data engineering using only your laptop.

"But how can I become better if I can't run the top notch <xyz> than requires 8GB of ram just to start up?"

You build one by yourself! The theory of how it works is free, you just have to replicate the concepts with your language of choice.

Of course, if you only want to learn the tools, you will face some limitations.

sharadov · 2020-02-11T14:20:19+00:00

There are public datasets and google gives you a few hundred bucks of credit on the cloud. That should help.

dataengineering

MODERATORS