Weight Initialization explained | A way to reduce the vanishing gradient problem

[@deeplizard]

Machine Learning / Deep Learning Tutorials for Programmers playlist: kzbin.info/aero/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU Keras Machine Learning / Deep Learning Tutorial playlist: kzbin.info/aero/PLZbbT5o_s2xrwRnXk_yCPtnqqo4_u2YGL Data Science for Programming Beginners playlsit: kzbin.info/aero/PLZbbT5o_s2xo_SRS9wn9OSs_kzA9Jfz8k

[@l33tc0d3]

your explanation is great -- can you please make videos using pytorch in your channel?

[@VinayKumar-hy6ee]

why are the random weights normally distributed with mean of 0 what is the intuition behind it

[@golangshorts]

God sent you to help machine learning learners.

[@raviteja5125]

The explanation is so perfect and clear that there are no dislikes in the video!!. Loved your voice and way of explanation :)

[@deeplizard]

Thanks, ravi!

[@manjeetnagi]

have never come across a video which explains this concept so well. awesome!

[@asdfasdfuhf]

I love your voice, it's so soothing to listen to.

[@torgoron9695]

Some explanatory remarks on the equation var(weights) = 2 / (n_in + n_out): The mentioned var(weights) = 1/n (or = 2/n when using ReLU) turned out to be good for the forward pass of the input (magnitude of the activations are kept approximately constant). However, with this the problem of vanishing gradients still exists during backpropagation. From the perspective of backpropagation it would be ideal to have var(weights) = 1 / n_{nextLayer}. Thus, in an article by Glorot and Bengio (2010) a compromise was found: var(weights) = 2 / (n + n_{nextLayer}), whereas n / n_{nextLayer} is the number of neurons in the layer before / after a considered neuron. This is from the lecture I've attended on deep neural networks.

[@shreyajain1224]

overall, which initalization technique must we use? formula?

[@torgoron9695]

@@shreyajain1224 I'm not sure anymore. But if I remember correctly you initialize based on the normal distribution, whereas you specify the SD of the normal distribution as explained in my comment above (you of course need to transform that variance into an SD). Maybe @deeplizard can help?

[@Viralvlogvideos]

Your angel to me thanks for saving my time and also for reducing my stress levels to understand the concepts

[@loneWOLF-fq7nz]

Never stop teaching us !

[@prashanthvaidya]

This playlist is really good. Grateful to you for your effort! :)

[@sourabhkhandelwal689]

Mind boggling, spellbound, one of the best Computer Science(not just AI/ML) channel on KZbin, hands down. I have watched your videos on Deep Reinforcement Learning and Conv Nets as well, they both are a masterpiece as well(although I wish you continued your series on Deep Reinforcement Learning, teaching other topics like Actor Critique and more stuff). And on top of that, you are funny as hell. Thanks for these videos.

[@deeplizard]

Spellbound... I like it! Thank you, Sourabh! Glad to hear you're finding value in all of the content and that you get my humor too :D Note that the RL series is still being developed with more content to come!

[@shreyajain1224]

clean and simple explanation thanks

[@ashutoshshah864]

This new intro theme music is much better.

[@justchill99902]

Hello! You kicked ass! Question - 3:44 "If the desired output for our activation function is on the opposite side from where it saturated, then during training when SGD updates the weights in an attempt to influence the activation output, it will only make very small changes in the value of this activation output." What does this mean? how is it related ?

[@chetan22goel]

I also have a similar question, what does saturation mean here. A little guidance will be really helpful. Thank you for creating such awesome videos.

[@raven5165]

The slope of the sigmoid is very small when its input is either very large or very small. So its derivative would be very small and it will contribute to the vanishing gradient. But now in general ReLu is used far more often than sigmoid and this is not the case with ReLu.

[@MohamedAhmed-gq7vr]

can I ask a question? if I understand correctly, I don't need the weights to be close to zero because if it is very small number close to zero the gradient will be very small.as well if it is very large. also, I understand why I multiply with 1 over n or any other factors. but the big question why I don't pick the weights form a normal distribution with mean equals to one to avoid vanishing or exploding in big neural network?

[@yuxiaofei3442]

glorious

[@udayannath7091]

If i initialize the weight is it mean that every time the confusion matrix is same? I faced a problem regarding confusion matrix as when i compile the my classification code for credit approval data set, every time it showed different confusion matrix. I fixed the randomization in the training and test set splitting.

[@meetayan15]

by saying variance of random numbers you are actually implying variance of random variables right? I have one more query input_shape=(1,5) means are there 1X5= 5 input nodes?

[@deeplizard]

Hey Ayan - Yes to both questions.

[@VinayKumar-hy6ee]

deeplizard why are the random weights normally distributed with mean of 0 what is the intuition behind it

[@salahfc7971]

0:31 - 0:52 -> Contents of the video.

[@edilgin622]

i write my own neural net so no initialization is done for me :D

[@commelephenix4702]

Speak louder please.

[@arnavsood2116]

IF IT WAS POSSIBLE I WOULD MAKE YOU MY GIRL. PEOPLE WANT MODELS I WANT SOMEONE LIKE YOU WHO TEACHES YOU THOUGH LIFE

[@shauryr]

This series is A hidden gem ! You deserve more views !! Thank you

[@DanielWeikert]

This series is really awesome. Thanks a lot for the detailed explanations. This certainly took a lot of effort so I ( and I guess everyone else who is watching and subscribing to this) highly appreciate that. Could you also cover the other layers in keras e.g. Embedding, Timedistributed,...? :)

[@deeplizard]

Appreciate that, Daniel! You're welcome. Thanks for the suggestions. I'll make sure these items are included on my list of potential topics for future videos!

[@DanielWeikert]

Great. Thanks a lot!

[@Max-lr6dk]

But variance = sum( (x-x_mean) ^2 ) / N so it won't be 250 but 1 And also idk var(z) = 250 it seems to be that x != 1, more like x= 1* (random number between - 1 and 1)

[@Iamine1981]

You have put together a great, concise accessible series for the uninitiated in the field of Deep Machine Learning. Thank you!

[@jamesang7861]

don't get this..."if the desired output for our activation function is on the opposite side from where it's saturated, then during training when SGD updated the weights and attempts to influence the activation output, it will only make very small changes in the value of this activation output barely even moving it in the right direction."

[@amithka9593]

I guess it means that for some odd reason, your random initialisation of weights (while being completely random) would always trigger an activation 1, instead of 0 (In this case, 0 is what is needed) - it would mean that the network would never learn. In very layman terms, the random weights give rise to a standard deviation which always makes the activation fire a neuron incorrectly that what *Learning* desires.

[@Waleed-qv8eg]

WOOW I love the new style! PLEASE add more videos! your way of explanation is so clear. Thanks. KEEP IT UP!

[@deeplizard]

Thank you! I'm glad you're liking the new style. 😎 New videos are coming soon. Hoping to release the next one tonight!

[@deeplizard]

And here we go! kzbin.info/www/bejne/fpbXd5yeqL2Gr9U

[@aaryannakhat1842]

Just amazing! I'll always be thankful to you for providing us these astounding videos!

[@CosmiaNebula]

0:22 intro 0:54 how weights matter 2:48 bad weights can cause vanishing gradient 4:44 heuristic for initial weight 7:24 keras code

[@DannyJulian77]

This video is perfect, THANK YOU SO MUCH!

[@KatarDan]

You are my new hero

[@erfanhasanpoor_zarayabi9338]

thanks for the video, I have a question. when we multiply weights by 1/sqrt(250), the variance problem of the activation function will be solved, but, the weights have the mean around zero and standard deviation around 1/sqrt(250), so weights are smaller than one and vanishing problem can occur for the first layers derivatives, can you help me to understand this problem?

[@junqichen6241]

Thank you for this amazing content. However, I'm having trouble understand how the fact that Z can take on values significantly higher than 1 is going to make SGD very slow. I guess I'm not following this statement - "if the desired output for our activation function is on the opposite side from where it's saturated then during training when SGD updates the weights and ...". Could you clarify a little bit more on this? Thanks a lot!

[@hsa1727]

what about "Bios" i dont get it ,,, you didnt talk about bios and im so confused ,,, is it same for tuning bios as well?

[@im-Anarchy]

bias

[@hsa1727]

@@im-Anarchy helpful 🫤

[@rafibasha4145]

3:06,Please explain how we got 250 as variance

[@im-Anarchy]

yes please

[@BENMADANIHoussemEddine]

thank you so much for this wonderful explanation. What is the default weight initialization in PyTorch?

[@bergamobobson9649]

i guess if we take the weight from a different distribution, the classical CLT will come handy to retrieve the same result as in the Xavier initialization...o am i wrong?

[@KiranSharma-ey6xp]

Nicely explained

[@fernandobaladi6636]

Thank you very much! Now I understand better what goes on with Xavier initialization!

[@tamoorkhan3262]

Before this insightful vid, I didn't know Keras silently doing so much for us without us even knowing. :D (y)

[@tymothylim6550]

Thank you very much for this video! I really enjoyed learning about initialization and how it connects with everything else! Great that it can be used to tackle the vanishing/exploding gradient problem!

[@saileshpatra2488]

Loved this❤️ Please make a video on how ResNet helps in solving the problem of vanishing and exploding gradients??????

[@diegodonofrio]

Thank you a lot for the detailed explanation!

[@parismollo7016]

Amazing video as always! Thank you for your contribution to the machine learning community, it's very valuable and we learn a lot from you.

[@KatS909]

very clear explanation :D thanks!

[@zihanqiao2850]

A start like a game loading, a beautiful woman's voice

[@anandachetanelikapati6388]

{ "question": "Can you identify a way \"Keras\" is helping to quickly train a network? (Select the most appropriate answer) ", "choices": [ "Keras defaults weight initializater attribute to 'glorot_uniform', which protects the variance from becoming larger.", "Exploiting the GPUs.", "Self adjusting the learning factor.", "Keras automatically drops few neurons to reduce the load in training." ], "answer": "Keras defaults weight initializater attribute to 'glorot_uniform', which protects the variance from becoming larger.", "creator": "Anandachetan Elikapati", "creationDate": "2020-08-19T12:01:45.585Z" }

[@deeplizard]

Thanks, Anandachetan! I changed the wording just a bit, but your question has now been added to deeplizard.com/learn/video/8krd5qKVw-Q :)

[@tumul1474]

thank you mam ! you are awesome

[@yepnah3514]

How do we change the values of these weights and bias on each layer?

[@bensaidlucas6563]

In the Glorot initialization I don't understand why this condition should be validated to avoid vanishing/exploding gradient problem: "we need the gradients to have equal variance before and after flowing through a layer in the reverse direction" Thank you

[@user-ep2hv7sv3q]

I think it's because if variance increases, the gradient would have to explode to keep up and shrink if variance decreases

[@lucas.n]

3:45 i believe theres an error on the sigmoid function? Shouldnt it be: fn(x) = 1/1+e^-x ?

[@deeplizard]

The two expressions are equal to each other :)

[@morpheus1586]

Whats the process when backpropagating?

[@longle3928]

Thank you for the video, very clear and easy to understand

[@joaoramalho4107]

You are literally saving my life :)

[@ashwanikumarm9027]

Thanks much for the video

[@AkshayRoyal]

Awesome work. Keep doing it !!

[@vijayendrasdm]

Thanks for awesome video. I have couple of doubts. 1. Suppose I am training a MLP and I have data that I divided into batches b_1,b_2...b_n. Lets say I do Xavier weights initialization. Now after I do back propagation over batch b_1 or after few batches We update the weights , which is in some way proportional to gradients. There is a good chance that this might distort the weights distribution (i.e not Xavier any more) . How do you guarantee the learning takes place after one/few back propagations? 2. To avoid vanishing/Exploding our weights needs to be around 1 (i.e 0.99 or 1.001) some thing like this. In any stage of the learning this has to be the case. My question is if the above statement is true then arent we restricting the weights and is learning actually taking place ?

[@ravirajyaguru5905]

Hi! I have been learning a lot from your awesome video explanations. so first of all, thank you for that. Secondly, I have a question about input_shape. At 7:30, the parameters for the input_shape = (1,5) means that input layer has 2 dimensions with 1 and 5 elements respectively. I am still not clear about that. Thanks in advance.

[@pike1991able]

Great video :-) I have thoroughly enjoyed it, and I recommend it in my circle to follow your series. Out of curiosity, does vanishing gradient also depend on the activation function you choose, and if yes, what are activation functions where this issue happens less? Any suggestions based on your experience! Many thanks.

[@harishpawar2546]

The explanation given is great. I was expecting things much in deep but that's fine. Now I got the clarity on what I need to dissect and I will definitely explore the content in this channel. Love and respect from India. Keep up the good work :)

[@moritzpainz1839]

This youtube Channel is a blessing from god, if there is one :D. By sharing your knowledge in such a easy way, you are seriously doing so much GOOD. Thank you

[@raven5165]

I wonder can we say accumulated variance can contribute to vanishing gradient when we have ReLu as the activation? Also can it contribute to the exloding gradient?

[@whiteF0x9091]

Great videos ! Thanks !

[@JimmyCheng]

very nice video as always! just have a quick question here, if we use the glorot initializer in keras, would it adjust to the activation function we are using? For instance, 2/n or 1/n for relu or sigmoid respectively.

[@deeplizard]

Hey Ziqiang - According to their docs, it appears they've implemented glorot initialization to always use the 2 / (n_in + n_out) method mentioned at 6:38. keras.io/initializers/#glorot_normal

[@roros2512]

I don´t get how var(z) = 250, some help please? thank you for the videos, I've been watchinh this list the last week and then I'll watch the on about Pytorch. thank you again for this great work, you are helping a lot of students

[@karthikd4278]

Even I am not clear on this part. Can someone help us understand this.

[@luismisanmartin98]

She said that in this particular case we are considering we have 250 nodes in the input layer, all with a value of 1. If this is the case, then z is the sum of all the weights. The weights are normally distributed numbers with variance 1. The variance of the sum of normally distributed numbers is the sum of the variances. Since the variance of each weight is 1, the variance of z is then 1+1+1+... as many times as there are weights. Since there are 250 nodes in the input layer, there are 250 weights and hence the variance is 1+1+1+1+...=250.

[@skadoosh7398]

@@luismisanmartin98 variance of each weight that is a single number should be zero right?

[@luismisanmartin98]

@@skadoosh7398 what do you mean? All weights are single numbers taken from a distribution with a certain mean and variance.

[@skadoosh7398]

you had written that the variance of each weight is 1. By the term 'each weight', do you mean a single element of the weight matrix? for example: let the weight matrix be [1,-1], it has mean 0 and variance 1. So do you mean that variance of each weight is 1 i.e var(1)=1 and var(-1)=1. I think I am missing something important. Please clarify

[@sorrefly]

This is extremely useful What I think is missing is the mathematical demonstration

[@aliasgarzakir4779]

Can you please explain why there is 2 instead of 1 in relu activation function?

[@radouanebouchou7488]

Thank you very much , this is such a clear explanation , it is very helfeul .

[@VinayKumar-hy6ee]

why are the random weights normally distributed with mean of 0 what is the intuition behind it

[@shreejanshrestha1931]

in input_shape(1,5) what does 1 stands for and 5 for ?

[@АтанасКостов-к5м]

4:40

[@oguzhanbaser1389]

Hey, the content is dope! sincerely,, yet I would probably prefer less distracting background than the one you've used. Thx btw, thumbs up..

[@deeplizard]

Agree :) We stopped using background noise during explanations in later episodes.

[@philipharman6846]

Love your videos! What are some "symptoms" that indicate your model is being affected by vanishing gradients? Is there a way to determine this, aside from overall accuracy scores?

[@zardouayassir7359]

when vanishing gradients occur, one symptom is that the weights in the first layer get less, or no, update compared to the weights in the later (deeper) layers.

[@anujsharma8938]

what is the value of node ?? how will you define 'value of node' ?? why does node have any value ??

[@deeplizard]

Start at the beginning of the course to have these points clarified: deeplizard.com/learn/playlist/PLZbbT5o_s2xq7LwI2y8_QtvuXZedL6tQU

[@anujsharma8938]

@@deeplizard yes, I have started from 1st lecture of this series. What I don't get it is, the 'value of node' is input of the 'input node' or activation output of 'input node' or kind of constant in activation function

[@deeplizard]

The layer we focus on in this episode is the input layer, which consists only of the raw input data. In this case, we're saying the raw input data is a structure of length 250 with each element in that structure being of value 1.

[@anujsharma8938]

@@deeplizard thank u 💕💕for your patience, but i got all those things except this statement 'element in that structure being of value 1' i got that here element is input node and structure is the input layer but what i don't got is 'node of value 1' how can an input node have any value ?? is that value '1' is assumed input value or assumed output value what do you mean by 'input node of value 1' ??

[@deeplizard]

The input value and output value of the input layer is the same. The input layer is made up only of the raw input data that is being passed to the network. In this case, we suppose we pass input data as a structure of length 250 with each element in that structure being of value 1. This is the input data. The data flows from the input layer to the next layer via connected weights. The input to an "input node" is the same as its output. It is simply the input data. This is not the case for subsequent layers, only for the input layer.

[@rey1242]

Oh So lets call a bunch of weights coming from ALL the neurons on the last layer into ONE neuron on the next, a SET So its the SET that needs to have a variance of 1/N? I thought the weights that needed this variance were the weights that came from ONE neuron to ALL the neurons of the next layer

[@deeplizard]

Hey Reyner - Yes, it's the set of weights you described in your top sentence that needs to have variance of 1/N.

[@rey1242]

@@deeplizard thx for the enlightenment

[@ashishkumar-fk8rh]

Wow.... new logo effect...

[@gamma_v1]

Audio quality is not great. My laptop's speaker volume and the KZbin volume adjuster are at the highest level, but I can barely hear the sound (to be fair there are some background noise at my place, but still the audio should be way louder)

[@deeplizard]

Thanks again for the feedback! We've been experimenting with the audio. Still tuning it trying to get it right. Check out one of the recently released videos, like the one below, and I'd appreciate your feedback on the audio quality on that one. Also, out of curiosity, are you listening through headphones? kzbin.info/www/bejne/fna0dYWNgrBpr7s

[@gamma_v1]

That video seems better, but could be louder. I'm not listening through headphones. Thanks

[@deeplizard]

Thanks for checking it out and letting me know your thoughts!

[@sonalisyngal50]

Hi, great videos! I have a recommendation: Would be great if you could take us through Gradient Boosting in detail, just the way you have done with ANNs. Looking forward

[@EDeN99]

Something you haven't explained so far in the series; in building the first layer, the parameter "input_shape" has not been properly explained. why is it having two parameters and what does each parameter mean? I know it denotes our input data but how does the two parameter it takes works and when do we ignore one of them? Thanks in advance

[@paulbloemen7256]

For a more or less normal, medium sized, WELL TRAINED neural network: - What is the actual range (actual values for maximum and minimum) of the weights and the biases between input layer and first hidden layer, between last hidden layer and output layer, and between the hidden layers? - What is the range of means and standard deviations for the three collections of values mentioned? - Does it make sense to use these characteristics of weights and biases to set the random initial values for the weights and biases for the respective layers? Or has one always have to use the Xavier initialization for weights anyway, but not for biases, to allow for a sensible growth path towards the final values for weights and biases? I truly would appreciate an answer to these questions, thank you very much!

[@EDeN99]

Really awesome series, thank you. I must say your voice is very low in this particular episode. Please try speaking louder.

[@mayurkulkarni755]

Wow a video on machine learning in the middle of advertisements

[@deeplizard]

A lizard's gotta eat!

[@mayurkulkarni755]

deeplizard yeah I can totally understand, but 3 ads in a short video really hinders what you're trying to understand, especially if it's a complex topic. You can try to follow 3b1b: they don't show ads for the first month so that users can watch ad free for first month. Also try to use KZbin SEO tools to increase viewers. I see your channel has a lot of potential. Good luck :)

[@deeplizard]

Hey Mayur - Thanks for the feedback and suggestions. Really appreciate it!