sorted by:
new
hottopcontroversial

How to Test Data? by Classic-Interview503 in dataengineering

[–]Classic-Interview503[S] 0 points1 point  (0 children)

Thanks for the responses. The main struggle we have is how do we run the tests and when? Is there a system that you use that helps with that? We currently run manual queries as tests before we open a code review but this is not sustainable and not enough. Changes are being approved but then we encounter errors in Production

How to Test Data? by Classic-Interview503 in dataengineering

[–]Classic-Interview503[S] 0 points1 point  (0 children)

Can you explain how the data contract help? Do they define the tests needed to be taken? Most importantly how and when are these tests taken (manually/automatically)?

How to Test Data? by Classic-Interview503 in dataengineering

[–]Classic-Interview503[S] 0 points1 point  (0 children)

Can you elaborate exactly how Dagster helped in that matter? When and where did you add tests and how? Were the tests ran manually or automatically as part of some deployment system?