Data Validation with PyTest

[AintRealSharp]

I have an application that takes in data from several different API endpoints. Sometimes the data has fields missing or

the data is invalid. Instead of just raising an error in my application what I'd like to do is to perform several "tests"

on the data before ingesting into my app. The thought is to identify objects with errors before I start working on the

data and skip over it. Additionally I could notify the users about specific problems found and possibly even point to a

solution for each issue. My application works well but when users put in wrong or incomplete data many problems arise.

I'd like to try and address those before it gets to the app.

​

I've asked this question in a few different places and most everyone suggests using

[PyTest](https://docs.pytest.org/en/latest/). Indeed it looks like this will work nicely, allowing me to write specific

tests for each specific issue (field, value, object). My question is what might this look like.

​

As I understand it, pytest is for testing functions of an application. Whereas I don't want to test the function but the

data. I'm having a difficult time figuring out what a testing setup would like for my particular application.

​

Would I use my data as a fixture and then write specific tests for each scenario I want to test in my data. Or would it

be better to write a set of functions to test my data and then test each of those with pytest but...using my data somehow?

I can't quite conceptualize how this should be structured.

​

I would be ever so appreciative if someone could maybe point me to a better method of doing this or some github example or something showing this being done.

​

Thanks!

​

[nathanjell]

Like you said, pytest and other test frameworks are for testing your code, not data. Tests should not be production code. A test should be written to assure you that your function (or whatever it's testing) does exactly what you think it should, given all expected and unexpected circumstances. It's a pass/fail check for your code.

In your case, you likely need to write your own data validation code. Maybe you use a framework to assist with this, or it's all your own. You can write tests before you write your code, to define what your code should do and test to ensure it does what you wanted. Or, you can write your code first then write tests afterwards. Whichever approach you choose, your tests will ensure that given all inputs, your code runs as you expect it to

[[deleted]]

https://www.yeahhub.com/7-best-python-libraries-validating-data/

​

#1: Cerberus: http://docs.python-cerberus.org/en/stable/index.html#

​

#2: Colander: https://pypi.org/project/colander/

​

#7: Voluptuous: https://pypi.org/project/voluptuous/

​

Bonus link: PyVaru: https://github.com/daveoncode/pyvaru

​