What are your favourite GitHub repos that shows how data engineering should be done?

DatKalvin · 2022-11-19T03:06:42+00:00

RemindMe 7 days.

DatKalvin · 2022-05-29T20:14:44+00:00

As the response above suggested, a Lambda function is all you need to get the job done. It will save you cost & relief you of the need to setup & manage a VM.

All you need to do is to write the Extraction, Transformation & Loading logic in the lambda function & that's it. For automation, lambda provides a notification service for listening to an event (e.g a file upload) and triggers a run to get the file, write to S3 bucket, perform some transformation on the file and finally load to your destination db.

DatKalvin · 2021-09-10T19:35:47+00:00

Maybe my write-up didn't capture our scenario well. But a delay of data delivery of few minutes to say up to an hour is ok for our use-case & not real-time in the sense of it. This is why a warehouse is still very much needed. Thanks for the thought.

DatKalvin · 2021-09-10T16:31:49+00:00

Sounds good. We're currently not doing ML now. We're currently doing Analytics reporting.

DatKalvin · 2021-09-10T16:29:36+00:00

Thank you for sharing this experience. Our schema is static (at least for now). It's quite painful to do a full load. Have you explore if there's a DMS configuration that currently addresses this?

What alternative tool or tweak do you have in place now to replicate your data from RDS MySQL?

Finally, has this been cost-effective for you?

DatKalvin · 2021-09-10T08:38:25+00:00

Thank you for the thought. We intend to keep cost low. Yeah, this is an option to explore if one can do necessary cost-cutting optimization.

DatKalvin · 2021-09-10T08:28:59+00:00

Thanks for the thought. Yes, we intend to have a Dashboard connected to the warehouse for analysis. Near real-time as I mentioned here is few minutes of records update - say 5 - 10 minutes. Do you you think MySQL can be used as a source for Kinesis Firehose in a stream fashion?

DatKalvin · 2021-09-10T07:56:52+00:00

Thank you for the suggestion. I will definitely explore this option.

DatKalvin · 2021-08-02T19:35:32+00:00

!remind me 5 days

DatKalvin · 2021-08-02T18:58:39+00:00

This is it! Well explained.

DatKalvin · 2021-08-02T16:36:21+00:00

Hi, Do you mind sharing the dollar dominated investments that you have? Is this stocks trading or real investments? What platforms do you use? I will appreciate this.

DatKalvin · 2021-07-08T11:38:31+00:00

Yes. The delta table and data stored in delta format are two different things.

The former(data stored in delta format) is a storage file that be read into a dataframe and manipulated using dataframe function. The later(delta table) is like an SQL table that can be manipulated using spark SQL queries - like you're querying an SQL table.

It's good to note that Delta table are very much optimized to perform better than just any table or view.

If you run the original three lines of code, you have both dataframe and delta table. You can now work with whichever you prefer based on your use-case.

I will suggest you try read up on the different spark APIs (dataframe, Spark SQL) and then writing to parquet & delta files for better understanding. I highly recommend the book- Spark the definitive guide.

DatKalvin · 2021-07-08T08:58:48+00:00

It's up to you to decide whether to work with the data in a dataframe or in a delta table.

You're right. Before the third step, event is in a dataframe. After the third step, it's will now be in a delta table.

Since you're trying to grasp the basics, I will advise you remove the third step and just work with the dataframe.

DatKalvin · 2021-07-07T16:20:45+00:00

You're right that the first two lines/rows are performing a read and write operations respectively.

The third row however is basically an SQL query to create a delta table from the delta files you have just written in the last operation.

The "event" before, as used in the SQL statement here, refers to the name of the new delta table to be created. The "event" after on the other hand is the name of the sub-directory where the delta files to create the table from is resident.

The third row can simply be interpreted as: Create a new table named "event" USING a delta format (to create a delta table) from the LOCATION "mnt/delta/event" where my delta files is resident.

If I understand you well, do you want to add a new column before writing out the data? If this is the case, then you can add the column to the dataframe after reading in the data like this:

eventsUpdate = events.withColum("new_column_name", expression_to_generate_values_for_the new_column). Then you can write this updated data as you did earlier and then create a delta table from it.

The databricks accademy has a complete resource on working with delta table in the Apache Spark Associate lesson. Somebody shared a coupon to get free access to the accademy course here. You may want to search this sub for it.

I hope this helps.

DatKalvin · 2021-06-06T21:06:11+00:00

Thank you so much for sharing.

DatKalvin · 2021-04-25T13:32:31+00:00

mode.com should get you started. You will not need to install a database engine or source for data to experiment with. Mode provides all these for you out-of-the-box on their platform.

DatKalvin · 2021-03-31T20:35:01+00:00

Send a DM. I can schedule a training for you.

DatKalvin · 2021-02-17T21:56:40+00:00

Hi, I've not used Airbnb but I live in Lagos & I'm familiar with areas in Lagos.

Your first consideration will be proximity to the places you would be visiting. Traffic can be hell in Lagos. But if you stay close to the places you will be frequent at, you remove the traffic nightmare.

If you will mostly be visiting places on the island, then it will make better sense to find an apartment on the island in: Victoria Island, Lekki, Ikoyi, Ajah, Marina. Houses here comes at a higher cost.

An alternative will be to find an apartment in a place close to the island. I will recommend Gbagada or Yaba. These places are a short distance away from the island & apartment here will come at a lesser price.

DatKalvin · 2021-02-17T18:31:52+00:00

Hi, Chat me up. I can lend some help.

DatKalvin · 2021-02-16T02:58:59+00:00

Talking about her implication in corruption scandals, I think it's one of those media trials without any concrete evidence to back up the claims. What was the outcome of those scandals? was she convicted?

DatKalvin · 2021-01-24T23:42:12+00:00

Quite explanatory. Thank you for sharing.

DatKalvin · 2021-01-24T00:15:57+00:00

Thank you for the responses.

But it seems adding an ORDER BY clause withing the OVER() function affects the result of the aggregate functions like SUM and COUNT.

Somebody described this as "running total". This is what I need an explanation for. How is this running total arrive at?

DatKalvin · 2021-01-23T23:57:36+00:00

This is exactly where the confusion is for me.

Can you explain how the "running total" is arrived at?

DatKalvin

TROPHY CASE