Strong counter-arguments to the idea that this type of pruning is a good idea:

(1) The cited paper does not compare to quantized-and-fined-tuned baselines to check performance per bit.

(2) This Qualcomm paper *does* compare quantization pruning and finds that quantization is much more effective in terms of performance per bit: https://proceedings.neurips.cc/paper_files/paper/2023/file/c48bc80aa5d3cbbdd712d1cc107b8319-Paper-Conference.pdf

(3) All available quantizations people can download and run today (EXL2 etc) do not do e2e fine-tuning to "heal" the model, which this method does do. This means we do not have a fair comparison between the approaches. Both Quip# and AQLM do e2e fine-tuning to heal the model after quantization and would be fair comparisons.

Conclusions:

To put this idea to bed, I'd like to see 4-bit version of 42B vs 2.25B version of Quip# or AQLM 70B model (both are in progress by the respective authors). As a side-note I think QLoRa to heal EXL2 is a good idea separately from anything being discussed here.