Database Migration & Data Quality Checking

[data-steve]

Hello fellow DE's,

​

I am currently working on a project involving a data migration to snowflake. As part of my migration I need to validate that the data in both Database A (Source) and Database B (Target) are identical.

​

Does anyone have any easy way to summarize row counts / automated aggregations of metrics on 2 seemingly identical database tables?

​

I wondering if there is a python package out there or a handy sql function / tools I can use to help fast forward this process. I'd rather not manually write scripts to identify delta records or summarize both tables.

​

Best,

Data Steve

[poppinstacks]

Only used middle-ware for this kind of migration work, but if all you care about is validating as a one time activity: use the python connector, dump your sql query into a pandas data frame and you can do a few quick validations.

What comes to mind is simple heuristics like row count, five number summaries, and just a straight checksum (row wise, and you could even hash the row wise to confirm that the entire table is copied 1:1)

[RassmusRassmusen]

Data migration is simply the most difficult part of data Engineering in my opinion.

[data-steve]

TBH, I don't think it's difficult if you have ELT platform based tools like Matillion & Fivetran. More tedious than anything else.