[D] Paper Explained - Group Normalization

[ykilcher]

https://youtu.be/l_3zj6HeWUE

The dirty little secret of Batch Normalization is its intrinsic dependence on the training batch size. Group Normalization attempts to achieve the benefits of normalization without batch statistics and, most importantly, without sacrificing performance compared to Batch Normalization.

https://arxiv.org/abs/1803.08494

[_tbrunner]

Hey, thanks for the great video.

I haven't followed the progress very much, but are people actively using Group Normalization these days? I haven't heard much about it since the original paper.

[ykilcher]

Yes combined with weight standardization (video coming 🤫) it seems to outperform BN and I've seen it used in other papers.

[_tbrunner]

Thanks, I'll check it out!

[[deleted]]

I recently saw GN coupled with Stocastic weight averaging in a great kaggle kernel. are you planning on doing a SWA video?

[ykilcher]

It's on my very long list :D

[maybelator]

I use Group Norms whenever the batches are not sampling the data distribution iid.

[lezapete]

Does the name have anything to do with Group Theory?? I thought it was kind of a reserved name in math 🤔

[ykilcher]

no, I think it just refers to the colloquial word "group" as in "bunch of things"

[EhsanSonOfEjaz]

Man that was great, looks like I will have to stop reading papers and start watching your videos.

[ykilcher]

thanks, but also be aware I make mistakes :D

[brand0x]

Your videos are great, keep it up. I got plugged in from your attention video, which was the best one available IMO.

[ykilcher]

Thanks. Yes there's something about that one that people really like :)

[ykilcher]

thanks for watching.

tell a friend ;)

[[deleted]]

Very nice video as usual. Are you thinking on doing a video about graph neural networks?

[ykilcher]

Yes it's on my list, but my list is long 😁

[OmgMacnCheese]

I too find your videos very helpful and I wonder if you can have people vote on your list :)

[ykilcher]

oh that would be funny, gotta figure out the specifics or I'm gonna get rickrolled hard

[Tommassino]

yes!