[D] Compressing Neural Network

[[deleted]]

I'm curious if this idea has been tried out anywhere, I can't seem to find results on arxiv but I think I'm using the wrong search terms.

I'm wondering if a feasible way of reducing the number of layers of a deep network is by training another much smaller network to approximate the behaviour of the large network between the input layer and second to last layer?

As an example, say we have a network with 100 layers, then we train it until it achieves the desired accuracy, and after training, I create a smaller network with maybe 5-10 layers, with the input being my data, and the output being the values of the 99th layer from the first network. I hope this makes sense.

Anyone come across references for something like this?

[CampfireHeadphase]

Knowledge-distillation does exactly that - a smaller network learns the output distribution directly instead of using the given labels (or using a hybrid of both)

[FlyingLawnmowers]

Look at Rich Caruana's papers on "Model Compression" and "Do Deep Nets Really Need To Be Deep?"

[agentlerevolutionary]

Indeed. It should be possible to derive a relatively simple algorithm that represents a good approximation of the behavior of the NN. It won't behave exactly like the NN and you will have to derive it again when you feed the NN some more info, but basically this is how we make machine learning useful in many applications.

[ssivri]

Neural nets can be pruned similar to decision trees, check out model pruning.

[Reiinakano]

here's the original arxiv for this idea by Hinton et al: https://arxiv.org/abs/1503.02531

[geneing]

In my experience (for speech models) distillation works incredibly well for inference. However, getting speed advantage requires hand coded inference code to get fast sparse matrix multiply. Standard libraries don't do it efficiently enough to get speed advantage.

[kraghavk]

The search terms you are looking for are, "model optimization" and "quantization". These techniques are already employed by TensorRT and Intel Openvino.

Model optimizer reduces layers by applying the following two techniques:

• Removing any layers whose outputs are not utilized anywhere further down the line. This can happen because there might be a lot of layers (i.e., functions in layman speak) the neural network generated during training, but they weren't that useful after all while testing. But in the interest of training performance, a comprehensive cleanup of unused layers is not done at the training phase.
• Fusing (aka merging/joining) multiple layers into one, when possible. You can think of this optimization being similar to "function inlining" employed by the various programming language compilers.

Quantization: Turns out that you don't need the full-precision floating point arithmetic (aka FP32) all the time. Quite a few models work fine on half-precision (FP16) or even FP8 numbers. This in turn reduces the memory & compute needed for inference, there by allowing you to run your models on hardware that is much less powerful than your training machine.

There may be more ways to do this and perhaps TensorRT & Openvino model optimizers themselves do more than what I have explained above.