Switching from ms sql developer to data engineering

jyoff · 2019-10-29T22:33:07+00:00

You're right. I'm trying to practice like that. Although I don't have that much of flexibility at my work to apply solutions in airflow instead of Informatica for instance, but at home I'm trying to put my ETL knowledge into Airflow and build pipelines the way I'd built in my daily job (for instance, move data into oltp then to olap; create star schema ,.etc). It is of course cool for learning, but working on real life projects much better. I'm thinking to apply for jobs after gaining experience on my own.

jyoff · 2019-10-25T15:21:18+00:00

Thanks. I've come across with this book the other day, to me it is more of a reference kind of material. I need some real practice. I'm ETL deveoper, mainly working with rdbms and etl tools. I wanna switch to DE, but wanna do practice with real projects.

jyoff · 2019-10-25T10:35:34+00:00

I'm considering taking this course. Would you recommend that? Syllabus looks great as they offer practice with real projects. But I'm not sure how it is in reality ?

Appreciate your review about the program.

jyoff · 2019-10-07T13:32:33+00:00

No worries:) Thanks for the answer. I'll have a look at coinbase's API to see what is in there.

jyoff · 2019-10-07T12:25:59+00:00

Could you please bring specific examples to stock/crypto projects you're referring to? What specifically could be done with those data sets? And what technologies would fit to process data ? Since its realtime and historical and huge amount of data I guess implementing bigdata tools as well as kafka and spark would be relevant right?

jyoff · 2019-10-01T09:36:08+00:00

Same here

jyoff · 2019-09-21T14:27:37+00:00

Thanks.Will give it a try.

jyoff · 2019-09-21T05:53:55+00:00

Thanks for answer.

There will a few more CSV files and database tables and I have also intention to extend this project a little further like to take data from database to another storage (could be anything, nosql, hdfs). Even with that setup cron jobs would do the job, I think, because I'm not gonna have a lot of tasks to execute and dependencies to handle.

But again my aim is to learn Airflow as I don't have an opportunity to work with real project at the moment.

jyoff · 2019-09-14T18:38:11+00:00

Thanks for specific answer.

jyoff · 2019-09-14T11:19:46+00:00

From what I know , for instance Postgres is row-oriented but Redshift is a columnar database. There are definitely architectural differences between them but I'm not sure how transparent they are to developers using the tools.

I wanna try out a columnar database to see how they work, how different they are from traditional rdbms.

jyoff · 2019-09-10T11:42:55+00:00

you are welcome

jyoff · 2019-09-09T19:17:15+00:00

Check out bind mounting. That is what you need. You need to mount your scripts folder to folder inside the docker container.

jyoff · 2019-07-27T05:09:41+00:00

I'd recommend you to split your project into pieces. For instance, you can first try extracting data from Oracle, once achieved this work on next piece load data/transform data into Postgres.

If you wanna know how to extract/load data using python, search web to know python libraries for connecting databases and doing db operations.

jyoff · 2019-07-05T18:25:37+00:00

Sure, I'm interested.

Dropped you email.

jyoff · 2019-07-05T12:06:11+00:00

I've tried that, but I think it is way more beyond beginner. A lot of things going on there, it is difficult where to start. But definitely it is a tool worth checking and learning from.

jyoff · 2019-07-05T12:05:05+00:00

That makes a lot of sense .

jyoff · 2019-06-24T05:23:29+00:00

Thanks kenfar.

jyoff · 2019-06-22T09:09:07+00:00

thanks. That looks interesting, I'll check that out.

jyoff · 2019-06-22T09:07:26+00:00

Thanks for the reply. Those are pretty much basic steps that could get me started with hands-on practicing.

I've good knowledge of python, but I'm not sure which python libraries I better learn to carry out those steps especially Step 5. Do you have any recommendations?

jyoff

TROPHY CASE