NeurIPS 2020 Tutorial: Deep Implicit Layers

This is a video recording of our NeurIPS 2020 Tutorial - Deep Implicit Layers: Neural ODEs, Deep Equilibrium Models, and Beyond - by David Duvenaud, Zico Kolter, and Matt Johnson. Full information about the tutorial, including extensive notes in Colab notebook form, are available on our website: implicit-layers-tutorial.org

[@keraeduardo]

I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me

[@alicsir]

Thanks for making this video public. The explanations are very intuitive and clear.

[@alexeychernyavskiy4193]

Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.

[@sippy_cups]

Awesome! Really well presented!

[@khuongnguyenduy2156]

Thank you very much for sharing this amazing tutorial!

[@ezamora1981]

Very cool idea!! Congratulations! and thanks for the tutorial.

[@gewang9770]

I like this tutorial very much!

[@elisim7]

Great tutorial and notes!

[@ansha2221]

Thank you for sharing this.

[@CristianGarcia]

Thanks for the tutorial! I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?

[@jiangao5652]

This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.

[@dominikklotz1035]

Great Idea.

[@Kram1032]

I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...

[@omarsharif4676]

Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment. Cheers!

[@kimchi_taco]

I learn a lot. Thank you very much. There are 2 questions about DEQ. 1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t? 2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?

[@zicokolter9110]

Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.

[@ezamora1981]

Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?

[@adrianbergesenfedaque8016]

Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).

[@vishwajitkumarvishnu3878]

shouldn't the last partial differentiation at 54:00 in backward pass be d1(z*,x,theta) ? its written d2(z*,x,theta)

[@kimchi_taco]

Awesome, but closed caption is little bit out sync. Could you sync it?

[@zicokolter9110]

Thanks for pointing this out! We've re-uploaded them to properly sync. They should work correctly now.

[@DasGrosseFressen]

Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?

[@user-we2zz3gc3l]

50:56

[@cexploreful]

earned like + sub at min. 1.47

[@rohullahalavi]

like