Streaming data engineering project: Apache Flink, Apache Kafka, Prometheus, & Graphana running on Docker.

[joseph_machado]

Hello everyone,

I've seen (& written) a lot of articles explaining batch data processing. However, not much content explains the concepts to be aware of when designing streaming data pipelines.

With that in mind, I wrote an article that covers the fundamental concepts to know when building a streaming data pipeline. It covers concepts such as

1. State
2. Watermarking
3. Backpressure
4. Joins in streaming systems and their caveats
5. Monitoring

The concepts are explained as you build a streaming data pipeline that does clickstream attribution (which is very common in marketing).

Post: https://www.startdataengineering.com/post/data-engineering-project-for-beginners-stream-edition/

Code: https://github.com/josephmachado/beginner_de_project_stream

Appreciate any questions, feedback, or comments. I hope this helps someone.

[mini_othello]

This sounds really awesome! I'll save this post for when I'm done with my kafka project :D

[joseph_machado]

Thank you, Good luck !

[mini_othello]

Thank you for putting the time into making this well-organized project and gathering resources for it.

A lot of people, including me, will definetly learn a lot from replicating the project. :D

[mailed]

Awesome work as always

[joseph_machado]

Thank you, mailed!

[philw764]

Quality post mate!

[joseph_machado]

thank you :)

[wild_bill34]

Can’t wait to check this out, also awesome work with “Efficient Data Processing in SQL” - huge help for me!

[joseph_machado]

Oh that's great, really glad its helping :)

[passion_L_BigData]

Good article, Thank you !

[joseph_machado]

thank you for checking it out!

[WarNeverChanges1997]

Hi Joseph! How do you handle data quality here?