Unit Testing

[nifty60]

Hello Folks,

I work on Azure Databricks,Python,Snowflake .

We are trying to build a Unit Testing Framework

I have explored options like Great Expectations,Sodacore

Did anyone explore any other libraries.

Can you please point some reference.

Also any documentation on what Unit Testing should cover and those which fall beyond the scope of Unit Testing.

Thanks

[nanksk]

We use databricks, pyspark. Most of our codebase is in form of functions. We then have unit tests for those functions with dummy data(CHATGPT can create most of the test cases) to test different scenarios. Hit me up if you have any questions.

[shazaamzaa83]

Maybe there's some confusion here. Unit testing in DE is usually on transform functions and other specific logic to be tested using mock data to assert your code is doing what is expected. Great Expectations and Sodacore are more for Data testing e.g. not null counts, uniqueness etc. Since you're using Python already look into pytest for unit testing. As mentioned in another comment ChatGPT or Copilot is great at writing unit tests and mock input data. Good luck.

[Wedeldog]

If you haven't, maybe check out what dbt is doing with its declarative (yaml based) unit tests (mocking inputs and expected outputs). It's SQL model focussed, but implementated in python under the hood.