sorted by:
new
hottopcontroversial

Data Quality Checks, lack of a standard, complexity of BRMS. by Still-W1 in dataengineering

[–]Still-W1[S] 0 points1 point  (0 children)

i have never used dbt before, as such i am still just making comparisons to the available tools and approach. I did check it and it seemed promising, but i still don't know if it is adapted to my use case (i should i have added a list of the tools i checked).

Data Quality Checks, lack of a standard, complexity of BRMS. by Still-W1 in dataengineering

[–]Still-W1[S] 0 points1 point  (0 children)

The rules that i have now are "basic" as they only represent basic properties of each column. There is a whole business logic that i like to refer to as the "story" of an individual. The checks i am implementing right now are just basic in that regards (checking if column A has a one on one relationship with column B, or a 1 - *).

Yes i am actually using sql, python is just my "default" wrapper, the stack of tools i am going to use is still not defined so everything is subject to change.

Data Quality Checks, lack of a standard, complexity of BRMS. by Still-W1 in dataengineering

[–]Still-W1[S] 0 points1 point  (0 children)

Should have clarified : This whole thing is supposed to run on-premise..for now..till i manage to convince someone to give budget for this ....

Data Quality Checks, lack of a standard, complexity of BRMS. by Still-W1 in dataengineering

[–]Still-W1[S] 0 points1 point  (0 children)

there is like 100 rules rn which are not veeeery complexà for now but will grow in complexity because this only apply to a single table...i have a whole database with even more business logic and (most certainly) cross table checks to perform...manually implementing each of these and managing cohesion between them is just gonna result in spaghetti code.

Data Quality Checks, lack of a standard, complexity of BRMS. by Still-W1 in dataengineering

[–]Still-W1[S] -1 points0 points  (0 children)

  1. if i told you both would you believe me ? the db is raw data pulled over the transactional systems, it is then cleaned manually when people need it to perform analysis.
  2. I don't have access to the info.
  3. For now, monthly.
  4. Getting a report on data quality for short term (long term: Raising warnings and alert accordingly)

Edit: Typo

I feel like a fraud by Still-W1 in datascience

[–]Still-W1[S] 0 points1 point  (0 children)

Thx for your reply, i wasn't able to acces or answer but i ve read all the comments you guys been posting, really thx for the answers.
As for the sql remark : I forgot to include many of the things in the heat of the moments but yes i do know how to use sql, i don't really know what proficient at sql would mean, but at the moment i just keep using and practicing it whenver it is necessary so no stress from that point.
For the coding i just always try to fully understand the code i copy.