I wonder how much can be done here with stochastic continuous evaluations in the spirit of MCMC or recent "Walk on Stars" style evaluations, where you don't have any discretization error at all, but trade that off with some noise...
@keraeduardo6 ай бұрын
I am a graduate student in Physics. This video is clear, easy to follow and highly informative. Many thanks for making this video public! This is very helpful for me
@alicsir9 ай бұрын
Thanks for making this video public. The explanations are very intuitive and clear.
@Pedritox09532 жыл бұрын
Great video!
@omarsharif46762 жыл бұрын
Thank you for a very informative video. I have a very limited mathematics background and was wondering if there are any good resources to better understand the differentiation in ODE. Please let me know if have such resources if you see my comment. Cheers!
@gewang97702 жыл бұрын
I like this tutorial very much!
@CppExpedition2 жыл бұрын
earned like + sub at min. 1.47
@rohullahalavi3 жыл бұрын
like
@elisim73 жыл бұрын
Great tutorial and notes!
@vishwajitkumarvishnu38783 жыл бұрын
shouldn't the last partial differentiation at 54:00 in backward pass be d1(z*,x,theta) ? its written d2(z*,x,theta)
@khuongnguyenduy21563 жыл бұрын
Thank you very much for sharing this amazing tutorial!
@강수현-b4c3 жыл бұрын
50:56
@ansha22213 жыл бұрын
Thank you for sharing this.
@alexeychernyavskiy41933 жыл бұрын
Thank you guys! Very solid video, and good tempo. You present the material with a smile in a very user-friendly manner, that's a rare delicacy :) I wish new successes for your trio in the coming year. Separate thank you for the website and the code! I think I will try to apply DEQ to image denoising.
@dominikklotz10353 жыл бұрын
Great Idea.
@ezamora19813 жыл бұрын
Hi Zico Kolter, great work! ....What about the inference time of DEQs w.r.t DNNs? Are they similar? ...Another question Do you recommend to use JAX instead PyTorch or Tensorflow2?
@adrianbergesenfedaque80162 жыл бұрын
Hi, I'm just getting started with DILs/DEQs but from what I can tell, their inference time tends to be x2 slower when compared to DNNs. Still, depending on your application it might not be important at all; e.g. in my case we are interested in processing requests on the minute, while a feed-forward DNN takes milliseconds to do inference, so doubling the milliseconds is not going to be a problem. In fact, our hope is that solving the optimization problem directly via this method will save time overall (compared to DNN + optimization algorithm).
@ezamora19813 жыл бұрын
Very cool idea!! Congratulations! and thanks for the tutorial.
@CristianGarcia3 жыл бұрын
Thanks for the tutorial! I have a question about the representations created by DEQs, in normal Deep Networks depth means you can compose features and deeper layers are supposed to have higher level representations, does the same story apply for DEQs or is there a similar way to understand its computation?
@jiangao56523 жыл бұрын
This work is amazing! When I saw GPT-3 use 175 billion parameters to build a language model, just feel hopeless. It's more fair to compete state-of-the-art performance based on model complexity.
@kimchi_taco3 жыл бұрын
I learn a lot. Thank you very much. There are 2 questions about DEQ. 1. Why does equilibrium point z* matter? How is z* better representation than any intermediate representation z_t? 2. ALBERT is BERT but share the weight by all transformer layers. How DEQ save memory sounds like ALBERT computes the gradient of only last layer and update the "shared" weight. ALBERT actually computes all gradients of all layers and update the "shared" weight by average of gradients. Why does DEQ work even though it doesn't care of gradient of intermediate layer?
@zicokolter91103 жыл бұрын
Thanks for the questions. For 1) this is mainly just an empirical issue, but in practice we do see that "deeper" networks (even in the weight-tied setting) do appear to work better, and thus the equilibrium point works best as the final representation (plus allowing efficient differentiation). 2) Yes, ALBERT would store all the intermediate activations, and compute gradients through the whole unrolled network. The idea of the DEQ model is that this is actually unnecessary, though, precisely via the implicit differentiation method we discuss in the tutorial.
@kimchi_taco3 жыл бұрын
Awesome, but closed caption is little bit out sync. Could you sync it?
@zicokolter91103 жыл бұрын
Thanks for pointing this out! We've re-uploaded them to properly sync. They should work correctly now.
@DasGrosseFressen3 жыл бұрын
Really cool. One question though? What is the fuss about neural ODEs? Honestly, I think I am missing something. They look just as taking a fireing rate model as an RNN... What is the difference?
@sippy_cups3 жыл бұрын
Awesome! Really well presented!
@yuhenghuang21314 жыл бұрын
Nice talk, thank you!
@NerdyRodent4 жыл бұрын
Ah, the old single pixel attack...
@양훈민-z2y4 жыл бұрын
Awesome. Can I get slides for this video?
@haiwenhuang94264 жыл бұрын
Just found out that videos on KZbin is much more stable than ICLR streaming.