Tutorial 11- Various Weight Initialization Techniques in Neural Network

[@DanielSzalko]

Dear Krish, The formula at 6:00 made my day. My month actually. I build my CNN with the standard library only, cuda toolkit, with a pinch of openCV. Implementing this in my CNN's fully-connected section brought that alive, boosting it's performance. Tonight we will clink glasses to your health! Thanks man!

[@tharukabalasooriya3269]

Gosh!!, India in another level of education man!! from higher Dimension

[@jaskarankaur4971]

I cant express how thankful I am to stumble upon your channel

[@AdityaRajPVSS]

मेरी इतने दिनों की खोज यहाँ आ कर खत्म हुई है। बहुत बहुत धन्यवाद आपका। आप अभी तक के सबसे बेहतर ML DL गुरु हैं । भगवान आपको स्वस्थ रखें।

[@hanman5195]

Your amazing at explaining ,excellent Guruji i found finally in Data science field

[@ayushmansharma4362]

This video is very good, I'm also watching the MIT deep learning videos in there they are just briefing the topics and not explaining the actual working in details, this video is very easy to understand.

[@AdmMusicc]

The video is great. However, could you explain why the respective techniques are good with sigmoid/relu?

[@dcrespin]

It may be worth to note that instead of partial derivatives one can work with derivatives as the linear transformations they really are. Also, looking at the networks in a more structured manner makes clear that the basic ideas of BPP apply to very general types of neural networks. Several steps are involved. 1.- More general processing units. Any continuously differentiable function of inputs and weights will do; these inputs and weights can belong, beyond Euclidean spaces, to any Hilbert space. Derivatives are linear transformations and the derivative of a neural processing unit is the direct sum of its partial derivatives with respect to the inputs and with respect to the weights. This is a linear transformation expressed as the sum of its restrictions to a pair of complementary linear subspaces. 2.- More general layers (any number of units). Single unit layers can create a bottleneck that renders the whole network useless. Putting together several units in a unique layer is equivalent to taking their product (as functions, in the sense of set theory). The layers are functions of the of inputs and of the weights of the totality of the units. The derivative of a layer is then the product of the derivatives of the units; this is a product of linear transformations. 3.- Networks with any number of layers. A network is the composition (as functions, and in the set theoretical sense) of its layers. By the chain rule the derivative of the network is the composition of the derivatives of the layers; this is a composition of linear transformations. 4.- Quadratic error of a function. ... --- With the additional text down below this is going to be excessively long. Hence I will stop the itemized previous comments. The point is that a sufficiently general, precise and manageable foundation for NNs clarifies many aspects of BPP. If you are interested in the full story and have some familiarity with Hilbert spaces please google for our paper dealing with Backpropagation in Hilbert spaces. A related article with matrix formulas for backpropagation on semilinear networks is also available. We have developed a completely new deep learning algorithm called Neural Network Builder (NNB) which is orders of magnitude more efficient, controllable, precise and faster than BPP. The NNB algorithm assumes the following guiding principle: The neural networks that recognize given data, that is, the “solution networks”, should depend only on the training data vectors. Optionally the solution network may also depend on parameters that specify the distances of the training vectors to the decision boundaries, as chosen by the user and up to the theoretically possible maximum. The parameters specify the width of chosen strips that enclose decision boundaries, from which strips the data vectors must stay away. When using the traditional BPP the solution network depends, besides the training vectors, in guessing a more or less arbitrary initial network architecture and initial weights. Such is not the case with the NNB algorithm. With the NNB algorithm the network architecture and the initial (same as the final) weights of the solution network depend only on the data vectors and on the decision parameters. No modification of weights, whether incremental or otherwise, need to be done. For a glimpse into the NNB algorithm, search in this platform our video about : NNB Deep Learning Without Backpropagation. In the description of the video links to a free demo software will be found. The new algorithm is based on the following very general and powerful result (google it): Polyhedrons and Perceptrons Are Functionally Equivalent. For the conceptual basis of general NNs in see our article Neural Network Formalism. Regards, Daniel Crespin

[@saurabhtripathi62]

When to use which initialization technique

[@minderiskrir989]

At 7:30 the Initialization Method is called Glorot, not Gorat.

[@krishnaik06]

Hey thanks I think I misspelt. Apologies

[@hanman5195]

@@krishnaik06 This exposes your open Heart to accept faults or mistakes. Your Awesome Sir :)

[@muppurigopi9576]

You are Amazing Sir.......................

[@cristianchavez5674]

Great Job Krish !

[@aish_waryaaa]

Best video and explanation.Thank You Sir...

[@monikathakur-zu1eb]

hi krishna , please take some inputs and then calculate the intial weights with xavier technique.

[@sandipansarkar9211]

Thanks. I understood the video but what was written below the video.I couldn't relate to what was shown in the video. Need to hold myself till I am able to solve problems on deep learning.

[@anjalisharma6543]

Hi Sir, I have one doubt if we can use the sigmoid activation function on top of the output layer for binary classification and sigmoid can be used in hidden layer but why we can not use the Relu activation on the top of the output layer even though we can use in hidden layer on netron?

[@ethyleneglycol4324]

Because sigmoid's output range is between 0 and 1. So you could take a threshold like 0.5 0r 0.8 and say if sigmoid's output is less than the threshold then we put that data in class A. Otherwise we put it in class B. But relu's output range is between 0 and infinity, so you can't do the same thing. Relu's better suited for regression, i.e. when you want to predict a value for your data, like a country's population in 2030 or the price of product X after the rise in the price of product Y.

[@ArthurCor-ts2bg]

Very lucid and insightful

[@yousefsrour3316]

Thank you so much man, amazing.

[@jagdishjazzy]

Isnt the vanishing gradient problem dependent on the type of activation function used rather than the type of weight initialized?

[@vatsal_gamit]

It depends upon the weights!! You are updating the weights with back propagation.

[@pranjalsingh1287]

actually he wanted to say exploding gradient problem

[@praveensingh-lx4dk]

When you will start deep learning classes? I am waiting for that.

[@Skandawin78]

Pls mention abt bias as well.. not sure how it is changed in backpropagation

[@good114]

Thank you Sir 🙏🙏🙏🙏♥️😊♥️

[@nitayg1326]

Wow! Didnt know about maths behind weight initialization!

[@marijatosic217]

Very helpful! Thank you!

[@blackberrybbb]

Learned a lot! Thanks!

[@soumyajitmukherjee2396]

Can you share the research paper links for reference purpose

[@srinivasanpandian5874]

Hello Krish, Thanks for good videos.. how to know that whether need to use Uniform or normal ?

[@Skandawin78]

Where do we specify the weights initialization method in keras?

[@mittalparikh6252]

you can improve much better by making ur content ready during the video session to avoid lot mistakes in your explanations. At this point lot of people are following you for understanding stuff, a wrong explanation can affect their understanding

[@paragjp]

Hi, Excellent lecture. Can you suggest a book to learn this weight initialization

[@fthialbkosh1632]

Thanks a lot, an excellent explanation.

[@yepnah3514]

hi!! good videos. how would i go about entering the weight values? do i need to use something like : set_weights()??

[@harishbabuk.s4344]

Hi Krishna, What is the importance of weight initialization in MLP and with out weight initialization cant we build a model just using the value of each dimension.

[@RAZZKIRAN]

weight should be small means number of input features ? numeric value of the weight?

[@sumanthkaranam99]

sir ,pls make a video on batch normalization

[@rohitrn4568]

Good stuff

[@rahuldebdas2374]

Its not Gorat initialization, it is GLOROT initialization

[@poojav2012]

Hi Krish, you explain very well and make learning very interesting. May i know which book are you referring to teach so that i can buy that book?

[@help2office427]

Krish use research papers not use any book

[@prasantas2195]

Hi Krish, in some of your videos subtitles are not enabled. With that it would be really MORE helpful.

[@ahmedelsabagh6990]

Simple and useful

[@khushpreetsandhu9874]

What if we have all the features as important, for eg- I have 6 features and I am performing multi class classification, so in this each feature is important for me, how would we intialize class weights in this scenario?

[@riffraff7358]

Bro you are best....

[@vishaljhaveri7565]

Thank you sir.

[@pratyushraj2640]

even if weights same activation wiill not be same...please help anyone

[@himanshukushwaha5064]

we have remeber it for exam

[@ajayrana4296]

how it impact on algorithm...

[@imranriaz9752]

Thanks

[@RnFChannelJr]

great explanation sir, may i get the research paper ? thankss

[@fajarazka9393]

sure sure

[@KenchoDorji-pu6wm]

Anyone watching in 2024. Must watch video😊.

[@louerleseigneur4532]

Thanks buddy

[@alluprasad5976]

thank you ! Will you make videos on cnn please

[@krishnaik06]

Yes I will

[@ruchit9697]

Here you gave example of the weights being same taking all weights as 0 but if we take the same weights having value more than 0 then I dont think the problem would come

[@NOCMG-ht9bd]

nO DEAR AGAIN PROBELM WILL OCURS as it generate large new weights with negtive values after updation . This called exploding gradient problem as teached by Krish in previous lecture.

[@ruchit9697]

@@NOCMG-ht9bd exploding gradient problem will occur when the weights initialized will have a very large value....here I am talking about his example taking same value of weights as zero.....what if we take same normal value of weights....then I don't think same value of weights will have a problem

[@NOCMG-ht9bd]

but by experience we should not take same values of weights again it will create problem later on.

[@NOCMG-ht9bd]

also weights should be according to useful ness of features means if feature is higly important then weight must be high in comparison of other feature weights and vice versa.

[@NOCMG-ht9bd]

also it depends on the problem domain and expert choice too.

[@RAZZKIRAN]

fan-In means number of inputs ? why squreroot of (3)

[@pranavb9768]

In the second layer it would be square root of 2

[@yashchauhan5710]

Amazing

[@atchutram9894]

What is very small or small or large? It makes no sense

even all weights are same but still we have bias activation function never be zero

[@Programming9131]

Sir Hindi me explain ker deta to Jada acche rhe the agr English me he smj na hota to Google se nhi smj leta

[@krishnakanthbandaru9308]

Research paper link please

[@zaidalattar2483]

Great!...

[@abdultaufiq2237]

It's ahhhhhhmaaaazing........

[@haneulkim4902]

While training deep neural network with 2 units in the final layer with sigmoid activation function for binary classification( highly imbalanced) 2 weights of final layer becomes both 0 leading to same score for all inputs since it only uses bias in sigmoid, what are some reasons for this?

[@gopalakrishna9510]

smile please at 11:47

[@ashrafulfuad2967]

sound quality is bad

[@vatsalshingala3225]

❤❤❤❤❤❤❤❤❤❤❤❤

[@shhivram929]

माता, पिता, Kirsh Naik, देवम् _/|\_