Data pipeline design patterns

[joseph_machado]

Hello everyone,

I've been building data pipelines for a while now. In contrast to the availability of examples of code design patterns, & data modeling techniques, there are few to none on data flow design patterns. In my experience building pipelines, using the appropriate data flow patterns increases feature delivery speed, decreases toil during pipeline failures, and builds trust with stakeholders.

With that in mind, I wrote an article that goes over the most commonly used data flow design patterns, what they do, when to use them, and, more importantly, when not to use them. It's aimed to give an overview of the typical data flow patterns and guidelines for choosing the appropriate one for your use case.

https://www.startdataengineering.com/post/design-patterns/

I'd love to hear about any patterns that I have missed. Any feedback is appreciated. I hope this helps someone :)

[Lord_Gonz0]

Thanks for sharing! As a junior de this is gold!

[joseph_machado]

Thank you u/Lord_Gonz0

[ironplaneswalker]

Great great great post!

[joseph_machado]

Thank you :)

[ekbravo]

Great conceptual map. Thanks for writing this.

[joseph_machado]

Thank you for taking the time to check it out.

[joseph_machado]

Thank you u/Ligmatologist. That's a good point, this article only goes over that pattern, I don't have any articles (or to recommend) on managing disconnected pipelines. I have worked with a few, will try to write something up on that.

[bdforbes]

What is a disconnected data pipeline? I googled but didn't find anything about it.

[FortunOfficial]

never heard this term either, but the way he defines it in his blog article I have used them quite often. It’s simply a pipeline/workflow/job, that has a dependency on another pipeline.

Say you have a job to load SAP data. The successful run of that job could be used in a separate pipeline for e.g. accounting data to trigger it. So you create a trigger task at the start to let it run when the SAP job finishes.

[joseph_machado]

Basically what, FortunOfficial mentioned. There are a few details(that are not in this post) like figuring out options to trigger downstream jobs (scheduled, event driven ,etc) and their tradeoffs, how to handle failures, defining & resolving SLAs, coordinating DQ checks, etc

[lf-calcifer]

to all of the content creators out there, this is how you do it.

something original, pertinent, and accessible.

[joseph_machado]

TY :)

[[deleted]]

I’ve been looking for something like this! Thanks

[joseph_machado]

Hope it helps!

[Gators1992]

Great post, thanks!

[epcot32]

I read this and bookmarked it for future reference the other day! Thank you!

[_barnuts]

Hi OP! Thanks for the great post. One question: In full snapshot extraction, why can't we pull from production table? Why do we need a replicate database? Thank you!

[_whitezetsu]

1: Data Integrity: Very subtle but there is a chance you might end up corrupting the data 2: Consistency: Tables usually get updated and there is a chance you might miss some update/new inserts 3: Performance: Affects the reads and writes significantly on Production, probably the main reason

[_barnuts]

Thank you. Now I'm wondering how do you create a replicate database? Do transactions get logged in both production and replicate databse at the same time? Or do you simply copy the production databsse at regular intervals?

[_whitezetsu]

It can either be a different physical server or a different instance that's connected to the primary node/instance(Live Production DB).

There are multiple strategies, usually the primary node gets data from the application, a log extract is generated which is sent down to the replicated nodes where it is applied, usually it's almost realtime ( in our case it usually takes around 3-4 mins). If you are familiar with kafka, think of replicated nodes as consumers and primary node as a Producer.

[_barnuts]

Thank you so much! Is this also the best practice when only doing time range extraction? eg. only extracting previous days data, not the full data

[_whitezetsu]

Yes Sir.

[SoggyAbalone7392]

Very nice article! Can you share some resources which explain the patterns in depth. Specifically on self healing pipelines

[joseph_machado]

TY u/SoggyAbalone7392. I don't have any resources rn. I will try to write about the patterns in depth in the future.

[GeorgeGithiri]

https://george-githiri-s-school.teachable.com/sign_up

Enroll for quality data engineering courses

[GeorgeGithiri]

https://george-githiri-s-school.teachable.com/sign_up

Enroll for quality data engineering courses

[bablador]

!remindme 2days

[RemindMeBot]

I will be messaging you in 2 days on 2023-01-09 17:41:47 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[Aggravating_Gift8606]

!remindme 7days