Lecture 6: Backpropagation

[@sachinpaul2111]

Prof...stop ...stop...it's already dead! Oh BP you thought you were this tough complex thing and then you met Prof. Justin Johnson who ended you once and for all! The internet is 99.99% garbage but content like this makes me so glad that it exists. What a masterclass! What a man!

[@quanduong8917]

this lecture is an example of a perfect technical lecture

[@odysy5179]

I work in ML and am doing review for interviews, this lecture is extremely thorough!

[@ritvikkhandelwal1462]

Amazing! One of the best Backprop explanation out there!

[@piotrkoodziej4336]

Sir, you are amazing! I've wasted hours reading and watching internet gurus on this topic, and they could not explain it at all, but your lecture worked!

[@ShuaiGe-n3g]

I 've just watched 30 minutes, but I 'm so excited to comment here that it's definately the best course for back propagation!!!!

[@vardeep277]

Dr. JJ, you sly sun of a gun. This is one of the best things ever. 47:39, the way he asks if it is clear. It is damn clear man. Well Done!

[@rookie2641]

Best lecture ever on explanation of backpropagation in math

[@achronicstudent]

Finally!! I understood how to apply backpropagation. Thank you sir! Thank you!

[@dbzrz1048]

finally some coverage on backprop with tensors

[@ryliur]

Future reference for anybody, but I think there's a typo @ 50:24. It should be dz/dx * dL/dz when using chain rule to find dL/dx

[@liviumircea6905]

At 58:56 prof Johnson tells something huge imho , the final equation is not formed by jacobians , finally I got it..simply the best explanation on the backprop .Thank you prof Johnson

[@tomashaddad]

I don't get how back propagation tutorials by 3B1B, StatQuest, etc, get so much praise, but neither of them are as succinct as you were in those first two examples. Fuck that was simple.

[@shoumikchow]

10:02. Dr. Johnson means, "right to left" not "left to right"

[@KeringKirwa]

You earned a like, a comment and a subscriber ... what an explanation .

[@kentu3892]

Such an amazing lecture with easy-to-understand examples!

[@VikasKM]

wooooowww.. what a superb lecture on backpropagation. simply amazing.

[@mihailshutov105]

Thank you very much! I really enjoy this lecture! Hello from Russia with love :)

[@minhlong1920]

Such awesome and intuitive explaination!

[@artcellCTRL]

22:22 the local gradient should be "[1-sigma(1.00)]*sigma(1.00)" where 1.00 is the input to the sigmoid-fcn block

[@mohamedgamal-gi5ws]

The good thing about these lectures is that finally Dr.Johnson has more time to speak compared to cs231n !

[@debasishdas9610]

19:38 Shouldn't 0.39 be 0.4 and 0.59 be 0.6 -- not sure where the rounding errors have creeped in. 49:45 would it not be much easier to use Einstein index notation?

[@arisioz]

At around 18:20 shouldn't the original equation have a w_2 term that gets added to w_0*x_0+w_1*x_1?

[@sainikihil9785]

w2 is a bias

[@apivovarov2]

@49:44 - Mistake in dL/dx formula - 2nd operand should be dL/dz (not dL/dx)

[@훼에워어-u1n]

this is extremly hard. but this is a great lecture for sure. you are awesome Mr Johnson

[@tornjak096]

1:03:00 should the dimension of grad x3 / x2 be D2 x D3?

[@anupriyochakrabarty4822]

how come u are getting the value of e^x as -0.20. Could u explain

[@shauryasingh9553]

I finally understand backprop!

[@smitdumore1064]

Top notch content

[@jungjason4473]

Can anyone explain 1:08:05? dL/dx1 should be next to dL/dL, not L when it is subject to function f2'. Thereby back propagation cannot connect fs and f's.

[@nityunjgoel1438]

Masterpiece!!!!

[@AndyLee-xq8wq]

Amazing courses!

[@נירבןזכרי]

THANK YOU SO MUCH! finally not shallow and excellent explanation.

[@matthewsocoollike]

19:00 where did w2 come from?

[@dmitrii-petukhov]

Awesome explanation of Backpropagation! Amazing slides! Much better than CS231n.

[@MiD-k7u]

Great lecture thank you. I have a question, would be great if anyone could clarify. When you first introduce vector valued backpropagation, you have the example showing 2 inputs to the node, each input is a vector of DIFFERENT dimension - when would this be the case in a real scenario? I thought the vector formulation was so that we could compute the gradient for a batch of data (e.g. 100 training points) rather than running backprop 100x. In that case the input vectors and output vectors would always be of the same dimension (100). Thanks!

[@akramsystems]

Beautifully done!

[@zainbaloch5541]

19:14 Can someone explain computing the local gradient of exponential function. I mean how the result -0.2 comes? I'm lost there!!!

[@beaverknight5011]

Our upstream gradient was -0.53 right? And now we need the local gradient of e^-x which is -e^-x and -e^-(-1)= -0.36. So upstreamgrad(-0.53) multiplied with local grad (-0.36) is 0.1949 which is approximately 0.2. So 0.2 is not local grad it is local multiplied with upstream

[@zainbaloch5541]

@@beaverknight5011 got it, thank you so much!

[@beaverknight5011]

@@zainbaloch5541 you are welcome, good luck with your work

[@Valdrinooo]

I don't think beaver's answer is quite right. The upstream gradient is -0.53. But the local gradient comes from the function e^x not e^-x. The derivative of e^x is e^x. Now we plug in the input which is -1 and we get e^-1 as the local gradient. This is approximately 0.37. Now that we have the local gradient we just multiply it with the upstream gradient -0.53 which results in approximately -0.20.

[@genericperson8238]

46:16, shouldn't dl/dx be 4, 0, 5, 9 instead of 4, 0, 5, 0?

[@kevalpipalia5280]

No, the operation is not relu, its calculation of the downstream gradient. since last row of jacobian is 0 meaning that changes in that value does not affect the output, so 0.

[@kevalpipalia5280]

For the point of passing or killing the value of the upstream matrix, you have to decide pass or kill by looking at the input matrix, here that is [ 1, -2, 3, -1] so looking at -1, we will kill that value from the upstream matrix, so 0.

[@maxbardelang6097]

54:51 when my cd player gets stuck on a old eminem track

[@YoshuaAIL]

Amazing!

[@Nihit-n5n]

great video.thanks for posting it

[@DED_Search]

45:00 Jacobean matrix does not have to be diagonal right?

[@blakerichey2425]

Correct. That was unique to the ReLU function. The "local gradient slices" in his discussion at 53:00 are slices of a more complex Jacobian.

[@qingqiqiu]

Can anyone clarify the computation of hessian matrix in detail ?

[@aoliveira_]

Why is he calculating derivatives relative to the inputs?

[@haowang5274]

thanks, good god, best wish to you.

[@Nur_Md._Mohiuddin_Chy._Toha]

👍👍👍👍

[@jorgeanicama8625]

It is actually muchhhhhh more simpler than the way he used to explain. I believe he was redundant and too many symbols that hides the beauty of the underneath reason of the algorithm and the math behind it. It all could have been explained in less amount of time.

[@kushaagra098]

do you have any resources that explain this better?

[@benmansourmahdi9097]

terrible sound quality !

[@Hedonioresilano]

it seems the coughing guy got the china virus at that time

[@arisioz]

I'm pretty sure you'd be called out as racist back in the days of your comment. Now that it's almost proven to be a china virus...