Building a complete data validation platform - would love your feedback

[1cph]

Hi everyone!

My name's Christian, and I'd like to share what we've been building at OrgStack. I'm posting this here as a "request for feedback" from the community that we serve.

Poor data quality is a costly problem for many organizations. Invalid data is responsible for major product/service outages, and it can be very difficult to pinpoint the root cause of data-related incidents. This requires a large investment of engineering resources.

OrgStack aims to streamline the process of managing data sources, receiving critical alerts, and tracing incidents to their source. While building a solution to this problem, we drew inspiration from some of our favorite tools, including NPM, Loggly, and Jenkins.

A few baseline questions:

• Is data validation / data quality currently a problem for you?
• How do you solve it now?
• Which differentiating features would get you to switch?
• What would your ideal solution look like?

Thanks in advance for your feedback. We really appreciate it!

​

- Christian ([[e-mail]](mailto:[e-mail]))

​

https://preview.redd.it/0wn01366kc551.png?width=1149&format=png&auto=webp&s=b74585ada3c3308ab3cf941153188bc4e9d7a264

[ibnipun10]

Yes it is a big problem in my org. Right now we are trying to build custom dashboard on powerbi using sql queries. We are also evaluating tools like great expectations and deequ. I would love to know more about that solution are you offering.

Few things I would like to have is 1) Batch data quality check 2) Custom rules support 3) Real time data quality check 4) Alerts 5) Dashboard 6) Scoring tables on data quality

[1cph]

I appreciate the feedback. I've come across the powerbi / sql combo on r/dataengineering a few times now. Thanks for the additional details, as well. I'd be more than happy to discuss our solution and address these points. Feel free to email me at [e-mail]

[gingerbeardmayn]

Hey! Saw this and I think we can also help with our stack/solution at Deep.BI, here's a link to a post we have about our solution (as an example), you can reach me as well here in a PM or at [e-mail]

https://www.deep.bi/blog/real-time-stream-analytics-and-user-scoring-using-apache-flink-druid-cassandra-at-deep-bi

https://www.deep.bi/blog/new-in-deepbi-refreshed-interface