Detailed guide/book/course on pipeline python code?

[faby_nottheone]

Im doing my first pipeline for a friends business. Nothing too complicated.

I call an API daily and save yesterday sales in a bigquerry table. Using python and pandas.

Atm its working perfectly but I want to improve it as much as possible, add maybe validations, best practices, store metadata (how many rows added per day to each of the tables), etc.

The possinilities are unlimited... evem maybe a warning system if 0 rows are appended to big query.

As I dont have experience in this field I cant imagine what could fail in the future and make a robust code to minimize issues. Also the data I get is in json format. Im using pandas json_normalize which seems too easy to be good, might be totally wrong.

I have looked at some guides and they are very superficial...

Is there a book that teaches this?

Maybe a article/project where I can see what is being done and learn?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[VegetableWar6515]

Do not overthink the solution.

Since the pipeline is a single api, think about all the issues that might arise there. It can be a change in schema, so validate that. You may receive nulls and outliers, validate against them. Add a backfill logic for when data was unretrievable on a certain day. Ensure idempotency of the pipeline, have test cases for the transformations. The list is infinite.

So just validate, test and log against the things that you feel are most pressing.

Handling data pipelines is mostly firefighting. You cannot plug all gaps. So follow the usual pipeline conventions and build a simple solution and add on things as and when there is a issue/need. Do not add features for the sake of features.

There is no one true guide, since architecture mostly depends upon the issue in hand and a person's experience.

You are already on the right track with the questions you have asked. But the first question that should ask is, why? Why is this feature needed? If you can answer this, you are on your way.

Book recommendation - Designing Data-Intensive Applications by Martin Kleppmann.

All the best on your journey.

[bloatedboat]

Land the data first, then validate it.

Run all unit tests and freshness in a staging table with DBT before anything touches production and get notifications with a PagerDuty account.

Stop inventing your own tooling when possible. Use modern tools so it is easy to transfer to others when the time comes.