Group Normalization (Paper Explained)

[@hillosand]

Thanks for the video! Note: normalizing *isn't* making the data more Gaussian, it's just transforming it to have mean of 0 and SD of 1. Gaussian data is often normalized and represented in this way too, but the normalization doesn't make your data any more Gaussian. Normalization does not change the inherent distributional shape of the data, just the mean and SD. For example, if your data was right-tailed in one dimension, it would remain right-tailed (and non-gaussian looking), it would just have a mean and SD of 0 and 1, respectively.

[@yes-vy6bn]

yeah it's really called standardization which comes from the equation to convert a normal distribution to a standard normal distribution

[@bluel1ng]

Nice explanation of BN at the beginning! Glad you kept it simple and did not use mythical "internal covariate shift" terminology. ;-)

[@Тима-щ2ю]

it is so true about mythical "internal covariate shift" terminology))

[@rbain16]

Thank gosh you mentioned the other way of thinking about batch norm @ 13:00. I thought I'd misunderstood batch norm the whole time. Like always, top notch content :)

[@mkamp]

Thanks for taking the time to walk us through this so slowly. Much appreciated.

[@bimds1661]

The visualization/explanation of batch norm was really helpful to understand how it works in a CNN! Thanks :)

[@IBMua]

Definitely one of the best NN explanation videos I've seen.

[@vandanaschannel4000]

Thanks man. Perfect illustration to understand the difference between batch norm and layer norm.

[@MiroslawHorbal]

Thanks for the videos. You do a great job of going over the details of papers and summarizing the key points.

[@reginaphalange2563]

"I usually don't believe the experiments that you see in single paper." LOL

[@fahdciwan8709]

Thanks a lot Yannic!! keep the videos coming

[@johnkilbride3436]

Well, I know what I’m adding to my model later. Thank you for the clear explanation.

[@YannicKilcher]

Also add weight standardization

[@andreasv9472]

@@YannicKilcher que?

[@grayleafmorgo]

good review Yannic it helps me a lot to faster understanding the papers

[@indraneilpaul1309]

It seems that the method is motivated by the fact that their might be a few correlated or similar channels but there is no effort to figure out which channels should be grouped together before normalizing them together. I'm surprised that this effect has been replicated across multiple efforts as you mention.

[@makgaiduk]

How does GroupNorm compare with the modern version of BatchNorm with running momentums? Sounds like momentums should fix the problem of small batch sizes on their own

[@eelcohoogendoorn8044]

What I dont quite get here, looking at the paper and the pytorch implementation, is that the batch axis remains unused. As far as making the point that you can get enough samples to compute meaningfull statistics without aggregating over the batch at all, its an interesting experiment. But it does show a tradeoff. If you have a batch thats bigger than 1, wouldn't you at least want to have the option to compute your statistics over the batch as well? Seem intuitively like that would bring you closer to the optimal big-batch-statistics behavior. Am I missing something here?

[@YannicKilcher]

You could consider that. but there are other factors, like with batch norm, you have to track moving averages for test time, and also if you use distributed training, you run into synchronization problems.

[@DerUltraGamerr]

Speaking about normalization, I was wondering about the intuition of LayerNorm in Transformer models. Usually it is applied after the concatenation and projection of the multiheaded self-attention output but wouldn't it make sense to apply it to each head separately to get more fine-grained normalization statistics?

[@YannicKilcher]

There's always a tradeoff. what you're suggesting would also introduce more variance

[@erictian8075]

Great video! Thanks! "But did you really do the experiment?"

[@dimitrisspiridonidis3284]

i love your paper reviews

[@carlosnacher3593]

Thank you so much! This explanation is literally what I needed 🙏🏽🤝🏽

[@naru909]

This is gold. Thank you!

[@Konstantin-qk6hv]

Great explanation. Love your videos!

[@AlterMachKeinAuge]

Awesome content. Thanks!

[@saeidshamsaliei2536]

Thanks a lot Yannic

[@proreduction]

To be sure I am understanding everything correctly: If you are training a fully connected NN (MLP) with only 1 channel, then Layer Norm = Instance Norm = Group Norm, correct?

[@moonryu2806]

15:13 you said you calculate the mean of 3 channel but the picture looks like have 6 different channel. is picture does not represent the 3 channel?

[@YannicKilcher]

In Layer norm, all 6 channels are averaged, in group norm just the 3

[@anynamecanbeuse]

I don't understand clearly. It seems Group norm just group some sequenced channels together, then how can you say they are just the same type of features?

[@YannicKilcher]

The network will make them related by you imposing it.

[@bright1402]

Great explanation! Thank you~

[@Erosis]

I'm not convinced. I think they're going to have to include more of those groupnorm "experiments" :) 24:06

[@nachiketa9245]

Amazing explanation.

[@ekstrapolatoraproksymujacy412]

isn't this simplified version of normalization used in AlexNet from 2012?

[@YannicKilcher]

maybe. all these things look kinda similar, but if an element is repeated often, the precise implementation matters

[@dermitdembrot3091]

I usually even go with batch size of 1 when processing videos 😉 (with my brain)

[@Chr0nalis]

Hmm yeah but what about non-conv networks?. Doesn't really make sense to group features from a specific layer for non-structured data imo.

[@rahuldeora5815]

Nice one! Enjoyed it. Can you do Stand-Alone Self-Attention in Vision Models, it has huge potential impact ?

[@seankernitsman6055]

+1 for the self-attention paper by J. Cheng, L. Dong and M. Lapata. Thanks for the video(:

[@julian390]

Shouldn't Batchnorm allow us to drop the bias term? It seems reasonable in my head but I couldn't find anything on it. Am I missing something?

[@YannicKilcher]

It allows you to drop the bias term of the convolutional/dense layers, yes

[@tropopyte6473]

​@@YannicKilcher Depends if you apply BN pre or post activation, doesn't it? Yes, it seems to be common practice to use BN pre activation, where you can drop the bias, but in my head it just makes much more sense to apply BN post activation. Especially if the argument is "i want unit gaussian like inputs to my layer". Using ReLU after BN will cut off half of my zero centered gaussian distribution, causing it to be neither zero centered, nor have a standard deviation of one, nor look much like a gaussian distribution. ...if my understanding of it is correct...

[@julian390]

Ah that makes a lot of sense, thank you :) This Channel is amazing!

[@guyindisguise]

@@tropopyte6473 I was wondering the same thing, did you find an answer to that question?

[@CppExpedition]

As always, great place to start reading! 🙃

[@JackofSome]

7 minutes in and I'm thinking "is this paper about aggregating statistics around normalization and then using those for each batch". Let's see if I was correct. Edit 1: doesn't seem like it Edit 2: neh, I'm wrong Edit 3: I like my idea better

[@YannicKilcher]

Wouldn't that be batch norm that's always in eval mode?

[@ManishChoudhary-hy5ey]

Nice explained

[@-mwolf]

Thank you

[@evgeniinikitin8843]

great video! I disagree about the uselessness of the group norm experiment though. batchnorm is not the only possible reason of performance degradation in the small batch size mode, so it's a perfectly viable experiment

[@Ronschk]

Thanks for the nice explanation! As it goes into a similar direction, do you know about network deconvolution? Was presented at ICLR this year and looks very interesting: arxiv.org/pdf/1905.11926.pdf

[@oneman7094]

Question. Do you read papers with sunglasses on? Almost in every video that we see your face you have sunglasses and I am always wandering while watching...

[@YannicKilcher]

Sure 😁

[@GenerativeDiffusionModel_AI_ML]

@@YannicKilcher Wearing sunglasses helps understanding the paper!!!

[@YannicKilcher]

@@GenerativeDiffusionModel_AI_ML Yea the white paper is just too bright :D

[@herp_derpingson]

How is group norm reducing internal covariance shift? It is agnostic to the batch moments.

[@YannicKilcher]

It's probably not. It doesn't appear to matter. What appears to matter is that you normalize somehow.

[@herp_derpingson]

@@YannicKilcher Dark magic :O

[@hieuza]

24:07 they reduce the learning rate by 10x at 30, 60, 90 epochs -- that's why the error reduce. Why is it funny?

[@JackofSome]

That's not what he's laughing about. The full battery of experiments done on group norm at that point are kind of unnecessary as they really just needed to show the 2ims/batch case.

[@eco24t]

The point isn't that they decreased the learning rate. The point is that the group norm performance shouldn't be affected by the batch size, but they tested multiple batch sizes anyway, possibly at the request of a reviewer.