The Fundamentals of Autograd

Autograd is the automatic gradient computation framework used with PyTorch tensors to speed the backward pass during training. This video covers the fundamentals of Autograd, including: the advantages of runtime computations tracking, the role of Autograd in model training, how to determine when Autograd is and is not active; profiling with Autograd, and Autograd's high-level API.
Download the Autograd notebook here: pytorch-tutori...

[@Sami_Wilf]

This is one of the most eloquent and succinct videos on training in pytorch I've seen.

[@kuskofboern]

Great tutorial! Just a small correction regarding the explanation of minimizing a function in the notebook. It is stated in the "## What Do We Need Autograd For?" section that "We want to *minimize* the loss, which means making its first derivative with respect to the input equal to 0: $\frac{\partial L}{\partial x} = 0$.", but this is only a necessary condition for a critical point, not a sufficient criterion for a minimum. For example, the function could have a saddle point or a maximum instead. While the explanation holds for strictly convex and differentiable functions, in general, a minimum is defined by the additional condition that the second derivative (or Hessian) is positive definite. Just thought it’d be helpful to clarify. Keep up the great work!

[@HrishikeshMuruk]

Nice set of videos. I just wish the audio volume as higher.

[@altayyk]

Thank you so much for your clear explanations! Makes the API much easier to understand when the mathematical intuition is already there!

[@FavourAkpasi]

very well explained series of videos. thank you. i however feel bad for your enter key.

[@wayneqwele8847]

awesome that was very succinct and clear.

[@cagdastopcu659]

This is the official doc video but with such a low quality audio?

[@shivampadmani_iisc]

yes, because it's open source

[@5104James]

11:33 Did you mean: the partial derivative of loss with respect to INPUTS (not learning weights)?

[@ayushbachan6113]

yes

[@gebbione]

After running zero_grad the grads are not zeroed but .grad is set to None type. Ie if i just print .grad[0] i get a NoneType object error Maybe change the last cell to ----------------- if model.layer2.weight.grad is not None: print(model.layer2.weight.grad[0][0:10]) for i in range(0, 5): prediction = model(some_input) loss = (ideal_output - prediction).pow(2).sum() loss.backward() print(model.layer2.weight.grad[0][0:10]) optimizer.zero_grad() print(model.layer2.weight.grad) if model.layer2.weight.grad is not None: print(model.layer2.weight.grad[0][0:10])

[@atari1040]

Oooooohhhh! So THAT'S how it works... cool! Thx!

[@mominabbas125]

Explained very well! (Y)

[@myelinsheathxd]

thx!

[@mahdiamrollahi8456]

Can you fix the audio

[@PurtiRS]

Good video. Go softer on the keyboard, please.

[@adnanhashem98]

If you still think you don't understand Autograd, this video (kzbin.info/www/bejne/npvRh3-cq82BZrMsi=Y78dMifRIL_hJwyZ) walks through examples to calculate simple grads by hand and verify them using PyTorch. I understood from it more than any other video.

[@udaynj]

Video is blurry as hell