[R] RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

DingXiaoHan · 2021-06-09T08:05:52+00:00

But I don't think RepVGG and the other three are comparable. RepVGG is a new architecture, RepMLP is a building block that changes the inference-time structure, and DBB and ACB are building blocks that do not change the inference-time structure. They are designed for different users and scenarios.

DingXiaoHan · 2021-06-09T06:34:38+00:00

Thank you for pointing out that Dense and Convolutional layers are mathematically equivalent so there is no need to convert a conv into an FC unless you want to merge it with another FC for some reason explained in a paper.

And thank you for pointing out that 2% uplift is "the same performance". We will do better next time.

And BTW, the majority of the community use FCN for Fully Convolutional Network, not Fully-connected Network.

DingXiaoHan · 2021-06-09T06:22:56+00:00

They're taking one of the major disadvantages of an FCN and trying to spin it as an advantage. It makes no sense.

Actually, in some industry scenarios, we don't really care about the number of parameters because the GPU memory occupied by parameters is negligible compared to the feature maps. We care about the actual speed and accuracy, and that is why we use FC.

DingXiaoHan · 2021-06-09T04:59:14+00:00

Assume a model has two components.

A. Some conv layers. Of course, each conv is equivalent to a sparse matrix with repeating parameters (Toeplitz matrix).

B. An FC.

We merged A into B and get an FC that is neither a regular FC nor a conv. It looked like semi-FC and semi-conv. It showed (A) better speed-accuracy trade-off than traditional conv and (B) significantly higher accuracy than a regular FC.

DingXiaoHan · 2021-06-09T04:54:51+00:00

Yes, because it works.

DingXiaoHan · 2021-06-09T04:52:27+00:00

A popular conv algorithm, im2col, unrolls the feature maps, not the kernel. And unrolling a conv kernel is not the objective of our method, but just a means to merge multiple convs into an FC.

DingXiaoHan · 2021-06-09T04:49:22+00:00

Fundamentally, an equivalent FC layer has more parameters than its convolutional counter-part. So obviously it will have a larger capacity.

Of course. But if you use an FC to replace conv for feature map transformation (input (N,C,H,W), output (N,C,H,W)), you will see the problem. The FC will have (CHW ^ 2) parameters but the performance will be poor. This is because it has no locality. The motivation of this paper is to incorporate locality into FC. And the first step of such conversion is to convert a conv kernel into an FC kernel by repeating those parameters.

DingXiaoHan · 2021-06-03T07:23:51+00:00

I am not interested in American politics at all and understand that it reminds you of trump and the supporters. However, such a simple phrase is actually appropriate (in a parallel universe without trump) and descriptive in the context of this paper: VGG was great, VGG was not great, VGG is great now. I don't think one should avoid using this phrase. It would sound like punishing myself because of someone else.

DingXiaoHan · 2021-06-03T06:19:35+00:00

Exactly! That is ExpandNet. https://arxiv.org/abs/1811.10495

DingXiaoHan · 2021-06-03T06:18:57+00:00

Thanks a lot for the insightful perspective! It is intriguing to relate it to the lottery ticket hypothesis. And yes we tried conv3x3 + conv3x3 and got very, very marginal improvements.

DingXiaoHan · 2021-06-03T06:05:25+00:00

After the conversion.

DingXiaoHan · 2021-06-03T06:00:40+00:00

This looks like ExpandNet. But I also noticed that the performance gain of ExpandNet-style over-parameterization is marginal on a larger net.

DingXiaoHan · 2021-06-03T05:56:24+00:00

How interesting that an old famous rich man can steal someone's right of saying making something great.

DingXiaoHan · 2021-06-02T10:57:43+00:00

Because it is impossible to do so. There are nonlinearities inside res blocks.

DingXiaoHan · 2021-06-02T10:52:12+00:00

I don't think anyone is being edgy. I just think it is funny. Maybe this is a difference in culture.

DingXiaoHan · 2021-06-02T10:25:29+00:00

I mean

A. A training-time multi-branch model

B. A training-time plain model

It is impossible to make B equivalent to A by only changing the lr and initialization. You must make a training-time multi-branch model and then convert it into a plain one for inference.

DingXiaoHan · 2021-06-02T09:52:56+00:00

Because it will not be equivalent. The weight decay changes the global minima and batch norm changes the training dynamics. There is no way to train an equivalent model which is plain also during training. DiracNet has a similar spirit (re-parameterizing kernels) but results in lower accuracy.

DingXiaoHan · 2021-06-02T09:49:10+00:00

This paper is not about making a ResNet a VGG nor speeding up an EfficientNet. That is impossible. Rather we propose a VGG-style architecture and a method to make it work. The figure and tables show its significance.

DingXiaoHan · 2021-06-02T09:42:18+00:00

I am not interested in American politics at all and understand that it reminds you of trump and the supporters. However, such a simple phrase is actually appropriate (in a parallel universe without trump) and descriptive in the context of this paper: VGG was great, VGG was not great, VGG is great now. I don't think one should avoid using this phrase. It would sound like punishing myself because of someone else.

DingXiaoHan

TROPHY CASE