PyTorch Autograd Explained - In-depth Tutorial

[@weebchina9639]

This is simply amazing man. It should be included as part of the official guide.

[@pradeepadmingradalpharecord]

Hey I made an issue on github here (github.com/pytorch/tutorials/issues/1024#issue-637571028), regarding this, I am having some issues merging, so if someone could help, would be great.

[@ankurgupta2806]

Absolutely

[@elliotwaite]

@@pradeepadmingradalpharecord This looks like a great idea. I'll check it out on GitHub.

[@israrkhan-bj3zh]

You said this is your first video on pytorch but its amazing. Looking forward to have more videos.

[@wanhonglau779]

this explanation is in-depth and original! can't find any other resources that explain pytorch autograd more in-depth than this

[@jiawei8319]

This is awesome. I've been trying to understand the autograd of PyTorch via bared codes for a whole week and I think I learned more in the 10 minutes of your video. Thank you and keep creating.

[@elliotwaite]

Thanks for the encouragement, Jiawei!

[@ishanmishra3320]

Been searching the net for a while about autograd but this is one of the most intuitive explanation with an awesome walkthrough. Thanks!

[@elliotwaite]

Thanks, Ishan!

[@gabrieleruscelli4450]

The part in which he describes the magical box killed me :) goodjob btw!

[@elliotwaite]

😂 Thanks, I'm glad you liked it.

[@Sneha_Negi]

would have never been able to understand such concept in just one video... thanks to you.

[@elliotwaite]

I'm glad you found it helpful.

[@rong5008]

Woudl be amazing if you make more pytorch videos!!!!! this is simply the best explanation of pytorch autograd

[@elliotwaite]

Sirong Huang Thanks! Not sure how soon it will be, but I’ll probably make some more eventually.

[@zilianglin7417]

This video is really helpful. It teaches me not only the presupposition of the 60 minute Blitz but also an overview of the computational/backward graph.

[@bitdribble]

I could finally understand a bit how this works. Thank you.

[@m.y.s4260]

Nicest explanation for autograd on the internet! Thx!

[@forresthu6204]

Mr. Waite makes very complex concept easy to understand, big thanks!

[@elliotwaite]

Thanks, Forrest!

[@ravihammond]

This is the exact explanation I needed to clear up autograd. Many thanks from Australia!

[@elliotwaite]

Thanks for the feedback. Glad you found it helpful.

[@RohitKumarSingh25]

best explanation of autograd available on youtube (Y)

[@elliotwaite]

Thanks, Rohit!

[@AlexeyMatushevsky]

That's magical! Some much effort to explain simple concept! As a result - I really feel like I really understood it. Thanks! Schema's in the hands of the master - priceless!

[@elliotwaite]

Glad you liked it!

[@jackperrin3852]

Awesome explanation. Really helped me understand how PyTorch implements back prop

[@elliotwaite]

Thanks, Jack!

[@jingbolin8835]

From Pytorch Forums, nice and detailed explanation for Pytorcher!!!

[@egecnargurpnar5732]

Elliot, that was an marvelous video. You are explaining such a natural way and fast that you implemented that function. Thank you :)

[@elliotwaite]

Thanks, Cemal!

[@raufbhat8096]

Thanks for the amazing in-depth explanation. Please do more videos on PyTroch.

[@elliotwaite]

Thanks, Rauf!

[@vigneshpadmanabhan]

Glad I got your video as a recommendation... it's amazing. ... Have been using tensorflow for a while and youtube wants me to learn pytorch ...

[@elliotwaite]

Haha. I used to use TensorFlow as well, but I was glad I made the switch to PyTorch. The main thing I like more about PyTorch is that the modularity feels cleaner. For example, the modules feel like legos made of legos, and when I want to try something new, it's easy to switch out any of those legos for my own custom one, regardless of if I'm working at the high level or the low level. And other parts of the framework, not just the modules, work similarly.

[@KSK986]

Nice, crisp and clear explanation. Thanks for sharing this knowledge.

[@MrDarkNightmare666]

after a week trying to understand autograd , now it's clear !! well, clear maybe too much but it doesn't seem impossible

[@elliotwaite]

Glad to hear this video helped.

[@chienyao8799]

Thank you for this wonderful explaination! Looking forward to your subsequent PyTorch videos.

[@mayankpj]

Color analogy is wonderful :) A very nicely done video Elliot !! Well done.... would love to see more of your stuff .....

[@elliotwaite]

Thanks for the encouragement! I'm considering making more videos, just trying to figure out how to do so in a way that will best assist my long-term goals. What kind of content would you find most valuable, useful, or helpful?

[@mayankpj]

@@elliotwaite From presentation perspective, I feel that one of the best ones i have seen out there are from 3blue1brown. I don't mean to say that its the only way. From content perspective, I would believe that videos explaining implementation of ML algorithms using Pytorch like SVM, Linear / logistic regression would be good to begin with. Videos explaining the fastai library's datablock and data loaders (in contrast with pytorch's stock versions) would be really helpful and be probably more viewed too. This can then also lead to explaining more advanced model architectures and contrast them like various vision architectures, nlp architectures especially as they are in most use.

[@elliotwaite]

@@mayankpj Nice. Thanks for the suggestions! I really like 3blue1brown too. I might have to just start experimenting with different video styles to see which are most rewarding.

[@mayankpj]

@@elliotwaite Good Luck ..... my best wishes !

[@shashank3165]

The video is amazing, man. Subscribed and looking forward to other videos.

[@elliotwaite]

Thanks!

[@bijonguha2299]

Superb explanation elliot. Thanks for making this video

[@elliotwaite]

Thanks, Bijon Guha!

[@xzl20212]

Brilliant. I super like your metaphors of colors for "leaves", dried-up leaves and magics😍

[@GoodOldYoucefCef]

Thanks, your graphs made things clear to me!

[@yusun5722]

Great video on the internals of PyTorch. Thanks.

[@SachinKumar-js8yd]

Nice explanation bro !! THanks for this video.

[@TalibHussain-ih1ev]

That is extremely helpful understanding the concept of autograd and backward

[@elliotwaite]

Thanks. I'm glad you found it helpful.

[@jiangpengli86]

Thank you for this fantastic tutorial and the graph. They are really helpful.❤

[@elliotwaite]

Thanks, I’m glad you liked it.

[@terohannula30]

Thanks! This kind of information I was looking for, I want to write own tiny autograd system and searched larger frameworks might have implemented it

[@gabrielcbenedito]

I'm so glad you decided to make this video... this explanation is just perfect! I just subscribed and hope to see more of those!

[@elliotwaite]

Thanks! I hope to make some more machine learning related videos soon.

[@akashupadhyay4373]

Great video , please make more video on pytorch

[@weiwang6706]

That's so great and clear! Thanks Elliot!

[@mohammadhassanvali1942]

A really fantastic and thorough explanation of autograd. Thanks a lot!

[@elliotwaite]

Glad you liked it.

[@eladjohn491]

This is the best video in the world.

[@Flibber2]

Nice video, more machine learning/deep learning related materials please!

[@elliotwaite]

Coming right up. Thanks, Flibber2!

[@muhammadroshan7315]

So much information in such short time. I am amazed just simply startled :o

[@QiuzhuangLian]

Awesome torch `Tensor` tutorial, thank you so much.

[@elliotwaite]

I'm glad you liked it. Thanks for the feedback.

[@parthasarathimukherjee7020]

Great video! More pytorch tutorials please!

[@elliotwaite]

Partha Mukherjee, thanks! I’ll probably make some more soon.

[@tungnguyendinh331]

Just an awesome explanation. Thank you very much

[@elliotwaite]

Thanks, Tung!

[@yassineouali1888]

Great introduction to autograd, thanks

[@kalekalekale]

Beautiful video and great for a beginner level understanding. The explanation of your color choice made me giggle. One conceptual question I'm struggling with: When you wish to not update (freeze) parts of the network, the recommended solution is to set requires_grad to False. I would like to clarify that all this does is avoid unnecessary computation and storage of gradients at those nodes. However, the node will still contain a grad_fn (if it has one) so that in the backward pass, the gradient from that node is still technically passed backward and the chain rule is still maintained? Another solution that was recommended is to not send the parameters you wish to freeze to the optimizer function. However, some recommend to set requires_grad to False as well to save memory storage.

[@elliotwaite]

Thanks. Great question. I should have shown in the video what happens when you freeze nodes, but I'll try to explain here. You can actually only set requires_grad = False on leaf nodes, and leaf nodes don't have grad_fn values, it's the intermediate branch nodes (non-leaf nodes) that have the grad_fn values. So for example: x = torch.tensor(1.0) weight_1 = torch.tensor(2.0, requires_grad=True) weight_2 = torch.tensor(3.0, requires_grad=True) branch_node_1 = x * weight_1 branch_node_2 = branch_node_1 * weight_2 Here branch_node_2's grad_fn will be a MulBackward object that passes the gradient along to an AccumulateGrad object for weight_2 and also passes the gradient along to branch_node_1's MulBackward object, which then passes the gradient to the AccumulateGrad object for weight_1. If we freeze weight_2: x = torch.tensor(1.0) weight_1 = torch.tensor(2.0, requires_grad=True) weight_2 = torch.tensor(3.0, requires_grad=True) weight_2.requires_grad = False branch_node_1 = x * weight_1 branch_node_2 = branch_node_1 * weight_2 This will be the same as above, except branch_node_2's MulBackward won't pass the gradient along to the AccumulateGrad for weight_2, it will only pass the gradient along to branch_node_1's MulBackward, which then passes the gradient along to the AccumulateGrad for weight_1. However, if we freeze weight_1: x = torch.tensor(1.0) weight_1 = torch.tensor(2.0, requires_grad=True) weight_2 = torch.tensor(3.0, requires_grad=True) weight_1.requires_grad = False branch_node_1 = x * weight_1 branch_node_2 = branch_node_1 * weight_2 Then branch_node_1 will not have a grad_fn at all because none of the nodes going into it require a gradient. In fact, branch_node_1 will become a leaf node. And branch_node_2's grad_fn will be a MulBackward that only passes the gradient along to the AccumalteGrad for weight_2. And if we try to freeze one of the branch nodes that has a grad_fn: x = torch.tensor(1.0) weight_1 = torch.tensor(2.0, requires_grad=True) weight_2 = torch.tensor(3.0, requires_grad=True) branch_node_1 = x * weight_1 branch_node_1.requires_grad = False branch_node_2 = branch_node_1 * weight_2 We get the following error: "RuntimeError: you can only change requires_grad flags of leaf variables. ..." So using the tree analogy again, you can only freeze green leaves. And freezing a green leaf is like changing it to a yellow leaf, and any brown branches that lead to only yellow leaves will actually also become yellow leaves. And the backward graph that gets created will only be for the brown branches that lead to green leaves. So when you freeze part of the graph, the intermediate nodes (branches) will update their grad_fn objects to not accumulate gradients for those frozen nodes (which are now yellow leaves), and if all of the inputs to a branch node don't require gradients (are all yellow leaves), that branch node won't have a grad_fn at all, and will become a leaf node that doesn't require a gradient (a yellow leaf), and it won't be involved in the backward graph. A new backward graph is created with each forward pass, so when you change the requires_grad values of any of the nodes involved in the forward graph, it will also change the structure of the backward graph that gets created. This change may just be that some of the AccumulateGrad nodes are not created, but in some cases, larger parts of the backward graph may be omitted if they aren't needed, such as when you freeze the early nodes in a graph. So using the requires_grad = False method to freeze nodes is better than just not passing those nodes to the optimizer, because setting requires_grad = False will be like pruning the backward graph, reducing both memory usage and computation time when the backward gradients are computed. I hope this helps clarify what happens when you freeze nodes. If there is any part that you'd like me to clarify further, let me know.

[@kalekalekale]

​@@elliotwaite Thank you very, very much for the detailed and thoughtful reply, it was extremely helpful. I think the mental block I had was that I was failing to separate the weights (parameters) from the computation nodes, and I thought that setting requires_grad = False in a network.parameter() loop was occurring on the weights AND computation nodes. But as you demonstrated, that is invalid for branches like Mul, and I was considering the computations as parameters. With the tree/leaf/branch analogy (new-ish to Comp. Sci.), it all seems so simple and easy to visualize now. Do I need to set requires_grad = False in every loop? I've read that you do not and can do it once before training (and should remember to set to True after), but your comment of "A new backward graph is created with each forward pass" leads me to think otherwise.

[@elliotwaite]

@@kalekalekale You only need to call requires_grad = True once to freeze the nodes. The backward graph that gets recreated each time is the collection of blue nodes, the MulBackward and AccumulateGrad nodes. The frozen leaf nodes are not recreated and will retain their attribute values.

[@kalekalekale]

@@elliotwaite Thanks again! You have my vote for more PyTorch content! Cheers.

[@ijyotir]

Appreciate the tutorial. Please keep doing this.

[@slavanikulin8069]

Thanks, dude. Didn't understand the last example, will try again later

[@elliotwaite]

Sounds good. If you let me know which part of the last example you didn't understand, I could try to elaborate on it.

[@MithileshVaidya]

Awesome introduction! Thanks a lot!

[@王甯-h2x]

Hi, Elliot Waite thank you so much for creating this, it's really helpful for me!

[@elliotwaite]

I'm glad you liked it.

[@홍성의-i2y]

tensor.detach() is used when we do not need to keep the gradient in that specific tensor. There are also other alternatives, such as .item(), .tolist(), etc. A typical example is the value generated by target-NN within the compute_loss part in the RL literature.

[@elliotwaite]

Thanks for sharing this info.

[@scottvasquez9880]

Great visualizations, dude!

[@elliotwaite]

Thanks!

[@christopherross9214]

this is exactly what I needed. been pulling my hair out. thank you.

[@namanchindaliya1012]

Great explanation. Keep up good work.

[@elliotwaite]

naman chindaliya, thanks!

[@JungeumKim]

Thank you sooo much!! Your first video is super helpful!!

[@andreazanetti01]

thanks for this video, really clarified a lot of things!

[@elliotwaite]

Glad it helped!

[@shantanuagarwal9660]

Thanks a lot Elliot. Excellent work.

[@amitozazad1584]

This is splendid, highly impressive.

[@gunjanmimo]

This video is awesome. Thank you for making this video.

[@elliotwaite]

I'm glad you liked it. Thanks for the comment.

[@ОбезьянаЧичичи-в8л]

My english skills are bad, but this explanation is ultra-understandable🤝

[@elliotwaite]

Good to know, thanks for the feedback.

[@shaswatsingh]

yield_apple = w11 * temp + w12 * rainfall + w13 * humidity + b1 yield_orange = w21 * temp + w22 * rainfall + w23 * humidity + b2

[@EdedML]

Amazing explanation, thank you so much!

[@interlingua2612]

Wow this is what I exactly needed of! thank you so much!

[@AndPacheco34]

Thank you! It was very useful!

[@reemgody5492]

totally amazing explanation. Thanks

[@elliotwaite]

Thanks, Reem!

[@atursams6471]

This is a good video. Thanks for making it.

[@DizzyDadProductions]

Subbed! I hope to see more content soon!

[@elliotwaite]

DizzyDad Reviews, thanks!

[@pierreeugenevalassakis8897]

That's a great walk-through for autograd, thanks!! Something that would be nice for the next video would be if you have any tricks to make sure the graph is setup correctly and the gradients propagate where you want them to ect... for instance how to use crayon, which I personally haven't really got into yet, but I'm hearing is good.

[@elliotwaite]

Thanks for the suggestion, that's a good idea. I haven't tried logging my PyTorch training in TensorBoard yet, but perhaps I'll make a video about how to do it once I learn more about it. Also, here's the link to the draw.io diagrams I made for this video if you're still interested, regarding your earlier comment: drive.google.com/file/d/1bq3akhmA5DGRCiFYJfNPSn7il2wvCkEY/view?usp=sharing

[@pierreeugenevalassakis8897]

Nice, thanks! I realised it was something like draw.io upon re-watching, in the beginning I though it was some sort of interactive front-end to a backend pytorch model!

[@ShahabShokouhi]

Dude! you are the best.

[@elliotwaite]

Thanks!

[@sushilkhadka8069]

Amazing video, thanks!!

[@elliotwaite]

@@sushilkhadka8069 thanks! Glad you liked it.

[@egogo5675]

Excellent :) thank you so much

[@adhoc3018]

Excellent explanation

[@elliotwaite]

Thanks!

[@Aditya_Kumar_12_pass]

thank you. this was the best

[@corentinlingier3549]

Awesome viz! Thanks a lot

[@钟辉-n8c]

it's very clear. Thanks!

[@starlord7383]

Very informative!

[@elliotwaite]

Thanks!

[@yumnafatma9198]

I have some difficulty understanding how DivBackward works, an explanation would be very helpful. Thanks for the nice video.

[@elliotwaite]

To explain, I'll use the example in the video (11:18) of: i = g / h (where g = 16, and h = 2). To understand how DivBackward works, we need to know how to backpropagate gradients through a division. This means figuring out how changing each of the inputs will change the output. To make this equation look more familiar, I'll replace some of the values so that we get the output is represented by "y" and the value we are trying to find the derivative of is represented by "x". So, to figure out the gradient for "g", we replace "g" with "x", and we replace the output "i" with "y", and then we replace any other variables with their current value, meaning we replace "h" with 2, and it becomes: y = x / 2 Now we just find the derivative of this equation (dy/dx) using the rules of calculus: dy/dx = 1 / 2 So the gradient that gets passed back for "g" is half the input gradient to DivBackward. Then we can do the same for "h", replacing "h" with "x", replacing the output "i" with "y", and the other variable "g" with its current value of 16, and we get: y = 16 / x And then we find the derivative of this which is: dy/dx = -16 / (x^2) And now that we've done the derivative part, we can replace "x" with its current value, and since "x" represents "h", we can now replace the "x" with the current value of "h", which is 2. So it becomes: dy/dx = -16 / (2^2) = -16 / 4 = -4 So the gradient that gets passed back for "h" is -4 times the input gradient to DivBackward. Now we can compare these values to what can be seen in the video at 11:18, and we can that these are the values that DivBackward uses. The input gradient to DivBackward is 18, and the gradient passed back for "g" is 18 * 0.5 (9), and for "h" it's 18 * -4 (-72). So how DivBackward works is that it saves the values of the numerator and denominator passed into the division operation and then uses them during the backward pass in the following way: gradient_of_numerator = input_gradient * 1 / denominator gradient_of_denominator = input_gradient * -numerator / (denominator ** 2) These equations represent the same steps we followed above. I hope this helps. Sorry if it was confusing. Feel free to let me know if there is anything you want me to clarify.

[@yumnafatma9198]

@@elliotwaite Thanks for writing detailed explanation. It is clear to me now.

[@elliotwaite]

@@yumnafatma9198 glad I could help.

[@BOURNE399]

Also the ending bgm is a good picking!

[@ghaliahmed]

SUPER GOOD VIDEO!!!!!!!!!!!!!!!!

[@BOURNE399]

!One word: AWESOME!!!

[@bobobopan2354]

excellent video!

[@pabloo.o1912]

Great explanation!

[@elliotwaite]

Thanks, Pablo!

[@ridfieldcris4064]

Very intuitive and straight forward explanation, I am looking forward to further PyTorch video, any update will pop-up in the future?

[@elliotwaite]

Ridfield Cris, thanks! I’m not sure when I’ll make another PyTorch video, but if I do I’ll post it here on this channel, so if you’re subscribed, the video should show up in your subscriptions (if that’s what you were asking about).

[@ridfieldcris4064]

Thank you for replying, I will definitely subscribe.@@elliotwaite

[@khaledsalah9248]

Briliant work : )

[@elliotwaite]

@@khaledsalah9248 Thank you!

[@timharris72]

​​@​Elliot Waite , I noticed that you always use 1 as the gradient when you go backwards from the end of the graph. Does PyTorch always start with a gradient of 1 in all situations or is this just something you used as an example? I'm guessing this is what PyTorch does but I'm not sure. This is the best explanation of autograd in PyTorch I have seen. Thanks for posting the video. I feel like I know PyTorch a little better.

[@elliotwaite]

When you call the .backward() method on a tensor, you can pass in a gradient argument to specify a certain gradient to start with, otherwise, the default starting gradient of 1 is used, which means that you will get gradients with respect to the tensor on which .backward() was called. Glad you liked the vid.

[@timharris72]

@@elliotwaite That makes sense. Thanks for the explanation. Understanding the details really helps when you are trying to learn the concepts!

[@snehotoshbanerjee1938]

Excellent!

[@samyamr]

Thank you for explaining the AutoGrad with such clarity. I am also wondering how you generated such beautiful graphs. Could you share the tool you used?

[@elliotwaite]

Thanks! The graphs were made with www.draw.io.

[@samyamr]

@@elliotwaite I am also wondering, is it possible to modify the backward graph after its construction? For instance, would it be possible to modify it in a way to just compute the gradients of the activations in a neural network, while computing the gradients of the parameters at a later phase.

[@elliotwaite]

@@samyamr I'm not aware of a way to modify the backward graph after its constructed, but you could split up the graph during the forward pass using the detach method. Something like: # Keep a copy of the original activations around that are attached to the first graph. activations_separate_graph = activations # Create a copy of the activations that are detached from the graph. activations = activations.detach() # Set `requires_grad` to True to start a new graph from these activations onward. activations.requires_grad = True ... # Then you could get the gradients of the activations in the first backward call. loss.backward() # And then the gradients of the variables in a second backward call, passing alow the gradients from the first. acivations_separate_graph.backward(activations.grad) That's the idea at least, but I haven't tested this code so there might be errors in it.

[@hiiamlawliet480]

GREAT!!!

[@kuzlovsky12]

Elliot, wanted to find out how autograd works by myself but you have already done the hard work, great video! Could you please let us know how did you dig into the details? Did you use hooks or went to the C++ code? In other words if you hadn't made this video how could we have figured all this out ourselves?

[@elliotwaite]

Thanks. Yeah I mostly read through the Python and C++ source code. I'm not as familiar with C++, so I had to do a bit of deciphering. I also used PyCharm's builtin debugger to test things out, and read through some related discussions on the PyTorch forums.

[@kuzlovsky12]

@@elliotwaite , I am following your footsetps. Managed to get pytorch built in debug mode and can step through the C++ autograd code with CLion. If you want this I can help you set it up.

[@elliotwaite]

@@kuzlovsky12, oh nice, that's a good idea. I might try that in the future if I do any more deep dives in the PyTorch code or other C++ libraries. Hopefully I'll be able to figure it out if I do, but I might try reaching out to you if I get stuck. Thanks for the tip.

[@kuzlovsky12]

@@elliotwaite OK sent you some pointers to follow with the setup. I think it would be a valuable content for a video, there are no resources how to debug the pytorch c++ code with an IDE.

[@elliotwaite]

@@kuzlovsky12, true. I haven't been interesting in making more educational videos lately, but I've added the idea to my list of potential future video ideas, and maybe I'll get to it one day. Thanks for the suggestions.

[@maidy199]

Thank you for the great explanation. I would appreciate if you can create another one for recurrent architectures. I am having a lot of trouble with creating novel recurrent networks due to the retain_grad problems.

[@elliotwaite]

Thanks for the video suggestion. I may not make a video on that subject soon, but for recurrent architectures, you shouldn't need to use retain grad since the weights are leaf nodes and they will accumulate the gradients even if those weights are used in multiple places in the network, which is what you do in recurrent architectures.

[@dasich5338]

the small amount of views strongly indicates how deep you digg

[@elliotwaite]

Thanks. I try to dig a little deeper into topics than the usual beginner material.

[@jaideepkukkadapu2600]

How do we know which variables should be stored in ctx variable? My explanation:- we calculate vjp on forward pass so we know which variables should be stored. Please tell me the correct explanation

[@elliotwaite]

I'm not sure I understand your question, so correct me if you were looking for a different kind of answer. You want to store in the ctx variable whatever values you will need to be able to compute the gradients of the inputs given the gradients of the outputs. In the typical case, you'll want to store the Jacobian of the forward operation, and then during the backward pass, you would multiple that Jacobian and the output gradients together to get the input gradients (making sure to use the correct left or right multiply, and the correct transposed or non-transposed version of the Jacobian depending on how your inputs and outputs are represented). However, some operations don't need to store the full Jacobian, for example, addition and subtraction don't need to store any values since the Jacobian for those operations would just be the identity, so instead, in the backward pass, the output gradients are returned directly as the input gradients without any intermediate multiplication. Also, if your input and output are non-typical tensors, for example, the unbind operation which outputs multiple tensors, I'm not sure if the stored values would still be called the Jacobian, but they would essentially have the same function. At least that's how I think about it, but please correct me if I'm wrong about anything.

[@jaideepkukkadapu2600]

@@elliotwaite Thanks for the explanation to phrase my question again :- How do we know what variables should be stored in the context(ctx) variable?(Do we store every value or we do some calculations in the forward pass so we store only required variables that are used in backward pass)

[@elliotwaite]

@@jaideepkukkadapu2600 only store what data you need in the backward pass and allow the rest to get garbage collected.

[@alexanderlewzey1102]

v good explanation

[@elliotwaite]

Thanks, Alexander!

[@AbhishekSinghSambyal]

Enjoyed a lot. Thanks. May I know which tool did you use to create those figures? Dia?

[@elliotwaite]

Abhishek Singh Sambyal, Thanks! I used this: www.draw.io

[@alalaben]

this video is great, do you have plan to make more like this?

[@elliotwaite]

yang yuan, thanks! Not sure yet what future videos I’ll make.

[@robinranabhat3125]

Hi Elliot. Thanks for this !! . Could you be able to explain on how would these computational graphs diagrams would extend for the higher order derivatives. I am hopelessly trying to figure out and cannot find the answer.

[@elliotwaite]

The first time through the forward graph you get a backward graph that can be used to compute the first-order derivatives. You can then traverse that backward graph to create a second backward graph that can be used to compute the second-order derivatives. And so on again and again for higher-order derivatives. For example, let's say our forward graph is the same as in the example: a = torch.tensor(2.0, requires_grad=True) b = torch.tensor(3.0, requires_grad=True) c = a * b d = torch.tensor(4.0, requires_grad=True) e = c * d This will create a backward graph, which we can then traverse to calculate the first-order derivatives with respect to `e` by calling: [a_grad_e, b_grad_e, d_grad_e] = torch.autograd.grad(e, [a, b, d], create_graph=True) Calling that will be similar to running this code (where `a_grad_e` means the gradient of `a` with respect to `e`): // Same as above, except we don't need to calculate `e` so that line is commented out. a = torch.tensor(2.0, requires_grad=True) b = torch.tensor(3.0, requires_grad=True) c = a * b d = torch.tensor(4.0, requires_grad=True) // e = c * d // Plus this part. e_grad_e = torch.tensor(1.0) d_grad_e = e_grad_e * c // Which equals: 1 * (a * b) = 1 * (2 * 3) = 6 c_grad_e = e_grad_e * d // Which equals: 1 * d = 1 * 4 = 4 b_grad_e = c_grad_e * a // Which equals: 4 * a = 4 * 2 = 8 a_grad_e = c_grad_e * b // Which equals: 4 * b = 4 * 3 = 12 You can see that the extra part is generated by just starting at `e` in the original graph and working backward applying the derivative rules. And if we ran that code it would create another bigger backward graph, which we could then traverse to calculate the second-order derivatives with respect to `a_grad_e` by calling: [ a__grad__a_grad_e, b__grad__a_grad_e, d__grad__a_grad_e, ] = torch.autograd.grad(a_grad_e, [a, b, d], allow_unused=True) Calling that will be similar to running this code (here `_grad_` is used for first derivatives and `__grad__` for second derivatives): // Same as above, except I commented out the lines we don't need. // a = torch.tensor(2.0, requires_grad=True) b = torch.tensor(3.0, requires_grad=True) // c = a * b d = torch.tensor(4.0, requires_grad=True) // e = c * d e_grad_e = torch.tensor(1.0) // d_grad_e = e_grad_e * c c_grad_e = e_grad_e * d // b_grad_e = c_grad_e * a // a_grad_e = c_grad_e * b // Plus this part. Again we start with `a_grad_e` and work backward. a_grad_e__grad__a_grad_e = torch.tensor(1.0) b__grad__a_grad_e = a_grad_e__grad__a_grad_e * c_grad_e // Which equals: 1 * (e_grad_e * d) = 1 * (1 * 4) = 4 c_grad_e__grad__a_grad_e = a_grad_e__grad__a_grad_e * b // Which equals: 1 * b = 1 * 3 = 3 d__grad__a_grad_e = c_grad_e__grad__a_grad_e * e_grad_e // Which equals: 3 * 1 = 3 So the second-order gradients we end up with are: a__grad__a_grad_e = None (it wasn't reached in this backward graph) b__grad__a_grad_e = 4 d__grad__a_grad_e = 3 You can confirm those values by running this example: import torch a = torch.tensor(2.0, requires_grad=True) b = torch.tensor(3.0, requires_grad=True) c = a * b d = torch.tensor(4.0, requires_grad=True) e = c * d [a_grad_e] = torch.autograd.grad(e, [a], create_graph=True) [ a__grad__a_grad_e, b__grad__a_grad_e, d__grad__a_grad_e, ] = torch.autograd.grad(a_grad_e, [a, b, d], allow_unused=True) print(a__grad__a_grad_e) print(b__grad__a_grad_e) print(d__grad__a_grad_e) Hope that helps.

[@endjack218]

Amazingly understandable! Could you help provide the illustrations in the video to github or somewhere? Thanks!!!

[@elliotwaite]

Thanks! Yeah, here is the link to the draw.io file: drive.google.com/file/d/1bq3akhmA5DGRCiFYJfNPSn7il2wvCkEY/view?usp=sharing When you follow that link, you should see a button at the top of the page to "Open with draw.io Diagrams." Clicking it should open the file in draw.io. You might need to be signed in to your Google account to see that button.

[@rahulseetharaman4525]

Thanks for the explanation. I have a few doubts. 1. In the video you used expressions which have only two operands. How does next functions list look like when there are more complicated expressions say d = a*(b^2), requires grad is true for a and b. 2. In the context of activation fucntions like relu or mod (which is not differentiable at x = 0) how does autogard take care of such functions ? And if we define our own activation functions, how does autograd work in this case ? Particularly, if we have some function that is not differentiable but still we use it in pytorch and try to compute backward on it.

[@elliotwaite]

1. Complicated expressions are broken down into operations that can be applied one at a time in operator precedence order. So `d = a * (b ** 2)` is equivalent to `temp = b ** 2; d = a * temp`. 2. The gradient of relu at 0 is 0. The gradient of fmod at 0 is 1. I figured these out just by testing them. I'm not sure if there is a general rule, but how the backward gradients are handled should be defined somewhere in the code, though sometimes it's hard to find this code because it's often written somewhere in the C++ implementations of these operators. If you create a custom activation function, it will just use the combined gradient of all the operations used, or you can define a custom gradient function by defining a backward() method, as explained here: pytorch.org/docs/stable/notes/extending.html Hope this helps.

[@JP-vo7zh]

Thanks a lot!

[@lucanina8221]

This video saved me daysssss

[@elliotwaite]

I'm glad it helped.

[@ことり-c3d]

awesome

[@alexanderlewzey1102]

I'm still a little confused by the difference between requires_grad and is_leaf? Why do we need both it sounds like they are similar, there is obviously a difference I am failing to grasp? Particularly how they both relate to the formal representation of a network ie are the leafs the weights and biases (hence we need to calculate their grad so we can update them)? Are the nodes with requires_grad = True and is_leaf = False the activation function operations? Any help with this would be much appreciated.

[@elliotwaite]

Yeah, it's kind of confusing. Basically, there are three types of tensors (I think of them in terms of colors and as leaves or branches). They are: Yellow leaf (is_leaf = True, requires_grad = False) Green leaf (is_leaf = True, requires_grad = True) Brown branch (is_leaf = False, requires_grad = True) The input data for the model would be a yellow leaf. Any constant values would also be yellow leaves. Any operations where the only input tensors are yellow leaves, the output tensor will also be a yellow leaf. For example, if you wanted to normalize your input values by dividing them by the mean of the dataset (a constant value, so a yellow leaf) and dividing them by the standard deviation of the dataset (also a constant value, so a yellow leaf), the output of that operation, the normalized values, would also be a yellow leaf. Any parameters that you want to be able to update through training, such as weights and biases, will be green leaves. And any operations that have an input tensor that is a green leaf, will output a brown branch. Such as if you multiply your input values (a yellow leaf) with a matrix of weight values (a green leaf), the output will be a brown branch. All branches are intermediate values or output values in the graph. They aren't input data, constants, or trainable variables. Any operations that have an input tensor that is a brown branch will output a brown branch. So if you take the brown branch from the last step (that you got from multiplying the inputs by the weights), and then if you add to it the biases (a green leaf), you get another intermediate tensor (a brown branch). And then if you pass that through a Relu function (an operation that only takes in one value), you get another intermediate tensor (another brown branch), and on and on until you get to the output values or the loss value. Once you get the to the loss tensor (a brown branch) you can call the backward() method on it, and it will backpropagate the gradients through all the brown branches until it has reached all the green leaves, and then it will update the grad values of all those green leaves. Then when you call optimizer.step(), it updates the values of those green leaves based on their grad values. Here are the possible inputs and outputs (on the left side of the arrow are the input tensors, and on the right of the arrow is the output tensor): yellow → yellow green → brown brown → brown yellow + yellow → yellow yellow + green → brown yellow + brown → brown green + green → brown green + brown → brown brown + brown → brown yellow + green + brown → brown If all the inputs are yellow, the output will be yellow, otherwise, the output will be brown. I hope this has helped clarify.