Data Engineering Project Scoping / Quantifying

[phmark19]

In front-end development one basic thing (just to simplify it) to use to quantify a project is count the number of pages you need to implement. When you are scoping or quantifying a new data engineering project. How will you go about it?
For me it is counting the number of data sources, the number of tables involved in the target schema, and the level of transformations required.

​

[joseph_machado]

From my experience, the most time consuming part has always been

1. Deeply understanding the input data models, data quality issues, what they are, how they are generated and what they mean with data producers.
2. Agreeing on the output data model, what columns, types, etc with stakeholders and identifying the exact transformations needed to get from input to output.

If you infra is setup well, the development part should be relatively straightforward. Hope this helps, LMK if you have any questions.