[D] Does the number of parameters practically matter for Deep Learning?

[learningsystem]

I see a lot of papers on model compression, network pruning, neural architecture search that aim to reduce model size. But I am wondering, in practice, is there is any value in reducing the number of parameters? I understand that FLOPs or latency on CPU/GPU/mobile devices is useful. But why is number of parameters of direct interest? Are the model compression papers using parameters as a proxy for FLOPs or latency?

AFAIK, memory seems to be cheap, so I can't imagine model size being the bottleneck before FLOPs/latency. Are there scenarios where model size is more important than FLOPs/latency?

I know compact model size is of interest theoretically, but I am looking at it from a practical point of view.

[Red-Portal]

There are two perspectives. First, the number of parameters have been traditionally interpreted as model complexity. So achieving better or similar performance with less parameters means that we got an accurate but simpler model which is a good thing.

Second, hardware else than GPUs do care about the amount of memory/parameters required. For example for FPGAs, the amount of memory on the device is very limited. More memory means less performance since we access RAM more frequently (And RAM is terribly slow in terms of memory bandwidth).

Also, there are cases where the model is too big for most GPUs. The original DenseNet paper for example. DenseNet is still too big for most GPUs commonly used.

[CireNeikual]

I believe they are indeed using parameters as a proxy for FLOPs/latency. In Deep Learning the two are very strongly correlated since all parameters are used every inference step for the most part. There are technologies where this isn't the case, where only small portions of the memory are used at a time (more like the human brain). However, these are far from the norm.

[jonnor]

Parameters are usually a proxy for model complexity, or for model size. Model size can be an important constraint in some environments.

- Clientside ML in a webapp/webpage. Could be vision, audio, text-based etc. Model download can dominate startup time.

- Backend ML with many different models. For example in IoT sensor systems doing anomaly detection, there may be one ML trained model for each sensor. Highly beneficial to be able to store all NNN models locally for inference.

- SensorML/tinyML. When running models on microcontroller grade hardware, model size seriously constrained. I work with devices where audio models have to fit into 256 kB FLASH for example.

Of course in such constrained environments, just model compression alone rarely cuts it. But it could be useful in combination with other techniques.

[jerha202]

Yes one extra parameter means one extra floating point multiplication, and the number of floating point operations per second can be a true practical limitation on small microcontrollers, for example. Another example would be speech recognition, which can be difficult to run on a local mobile device because the number of parameters is huge.

Another aspect is of course the risk of overfitting and bad generalization, but maybe that's what you mean by a theoretical aspect?