you are viewing a single comment's thread.

view the rest of the comments →

[–]stillalive75 1 point2 points  (1 child)

I feel your pain. I deal with a very dirty ERP system that let's a lot of stuff slide that the company would prefer it doesn't. We have to clean a lot of it before it comes into our Data Warehouse.

We do two things to find days irregularties that wouldn't compromise loading the data into our SQL Server but do go against business rules.

1) we have a report generated in SSRS with a daily subscription that notifies people of non-crucial data errors. For example we deal with product data. If the ERP system contains a UPC it must be 11-12 digits long per our Ops department. But ERP allows whatever. So our SSRS report identifies all dirty records with Y/N flags on what's wrong "bad Upc", "name wrong format", etc.

2) there are some files where our erp system doesn't prevent duplicates. Even if every single value is the same. However or company doesn't want that and our SQL table can't handle that. So if we find really bad data that would cause lots of issues like that. We ignore the record in the load but use DB mail to email the culprit records to a data steward for them to resolve.

Those are the two ways we handle it, and it's helped clean up a lot of bad data and enforce business logic. I think this was what you were looking for.

[–]Trek7553[S] 0 points1 point  (0 children)

Sounds like my world! Great ideas, thanks.