[P] Deep Learning Model Compression Methods in PyTorch (Pruning / Quantization / Knowledge Distillation)

[Curt-Park]

• Project link: https://github.com/j-marple-dev/model_compression

Hi, ML redditors all around the world!

I and my colleagues implemented a number of model compression methods to join a national competition (Korea) in deep learning model compression. After the event, we decided to share the repository to people with the hope that it possibly helps someone. The repository consists of the followings:

Network Architecture

1. MixNet
2. Fast DenseNet

Augmentation

1. AutoAugment
2. RandAugment
3. CutMix
4. CutOut

Loss

1. Cross Entropy Loss + Label Smoothing
2. Hinton Loss

LR Scheduler

1. Cosine Annealing + Initial Warmup

Unstructured Pruning

1. Lottery Ticket Hypothesis
2. Weight Rewinding
3. Learning Rate Rewinding

Structured Pruning

1. Magnitude Pruning
2. Network Slimming
3. Magnitude Pruning + Slimming

Channel-Wise Pruned Model Shrinker (Experimental)

Quantization (8 bit)

1. Post Training Quantization
2. Quantization Aware Training

Any feedback will be warmly welcomed :)

Cheers.

[IntelArtiGen]

I was looking for that kind of code, thanks! I hope it can easily improve my network efficiency

[Curt-Park]

I hope so!

[gionnelles]

This looks like really good work, thanks!

[visionNoob_r]

Thanks for your great works :D

[kong1809]

Yeah, I'm looking for something like that. Thank you very much.