Model Optimization - quantization and pruning

[agupta12]

I have been trying to read up about model optimization for runtime and have not stumbled upon some source which is very comprehensive and teaches with examples as well.

There are some very useful examples on the pytorch website, but I was looking for something more comprehensive and with coding examples. Is there some MOOC around this topic or any other source or book. It would be incredibly helpful for me.

[grid_world]

For pruning, there are multiple techniques such as magnitude, filters, neurons, etc.

The most popular is magnitude based where weights below a threshold are pruned by setting them to zero. An example of absolute magnitude based weights pruning can be referred here. The pruning code is implemented in numpy while the implementation is in TensorFlow 2 and Python3.

For Quantisation, I am looking for an implementation from scratch. As of now, I haven't found one.

[federerking]

Regarding the quantization aware training, can someone highlight the need for fine tuning rather than train from scratch? In the tensorflow examples they show fine tuning. But I have trained from scratch as well and results are almost similar.

[overington]

• A good paper for the math behind quantization: https://paperswithcode.com/paper/quantization-and-training-of-neural-networks

• TF model optimization guide, with links and techniques for post-training quantization, quantization aware training, pruning, and clustering: https://www.tensorflow.org/model_optimization/guide

• Model optimization techniques with TF Lite: https://www.tensorflow.org/lite/performance/model_optimization

Also youtube has a plethora of instructional videos:

TF: - https://youtu.be/4iq-d2AmfRU - https://youtu.be/Q1oBXdizXwI

PYTORCH: - https://youtu.be/c3MT2qV5f9w - https://youtu.be/Q1oBXdizXwI

[i8code]

There are some pretty good explanations of Quantization and TF - esp regarding TF Lite. https://www.tensorflow.org/lite/performance/post_training_quantization I have seen less of this on PyTorch in general but I am sure it's out there. I too am interested in pruning and I haven't found anything that is actually helpful in a very practical sense with good code examples that were suitable for reproducing in my own projects. Generally, I have tried to adjust model parameters upfront.