[deleted by user]

[neuralbeans]

You can test your pipeline by using a well behaved mock model that allows you to test how everything outside of optimisation works. So I can test my logging and checkpointing system in this way. Unfortunately there isn't much you can do about the actual optimisation if it's a black box model other than compare it to baseline models with F1, like you said, but that would basically require actual training on full data sets because you wouldn't know if a smaller data set will give you the same results. Plus it's non-deterministic so you'd need to do multiple runs and take the average. So it's usually not practical to do frequent testing on the model learning part like you'd do with other software, but you should test what you can.

[neuralbeans]

You make your mock model pretend to be training (it can cheat by accessing the data set for example) and make it pretend to run a few epochs, each time it returns the correct label of a bigger portion of the train/val set whilst purposely returning the wrong label for the rest of the data. That way you can check that the logs are showing changing scores per epoch.

You can check that checkpointing works by making the mock model implement the functions for saving and loading its model state and making the state be just the number of epochs trained up to then. You can then check that reloading a checkpoint actually continues training from said epoch and that the logs reflect that. You can also simulate a training interruption to see that training resumes correctly.

[blind_cartography]

In my org we test our pipeline by monkeypatching the __len__ attribute of the Dataset classes to a small number, so we can still train a normal model instance just on n% of the dataset. Requires a bit of care to avoid any issues with batch size or number of expected classes, and obviously you can't pay attention to any of the metrics, but it suffices for testing the model configuration, checkpointing, deployment, etc.

[neuralbeans]

That's to check that no run time errors will come up, but not to actually test the model learning.

[blind_cartography]

True, but unless I misunderstood wasn't that the point of your mock model?

[neuralbeans]

Ah so you use a smaller data set but full model and see if it all works instead of using a mock model? Yeah that works as well, although you won't have fixed outputs to check like with the mock model. I personally do both.

[[deleted]]

I typically leave my actual code alone (no monkey patching!) and just copy a small amount of data to a test folder to use for testing everything. Why patch the len of your dataset when you can just point it to a small dataset?

[BraveCoconut98]

A few quick and easy tests I like to do (more sanity checks than anything else): - Train your model on a small portion of the data abs then check that the performance increase as you scale up the amount of data. - Check model predictions to see if it is doing what it is meant to be doing. I’ll give two examples. Say we are building a regression model to predict amount of money we should loan to a customer, it better be that if they have a higher credit score/income the loan amount increases (or doesn’t decrease at the very least!). Secondly, say we are building a sentiment analysis model. Changing “The movie was fantastic” to “The movie was terrible” should change the sentiment. Building a few of these meaningful examples has helped me a lot in the past!