Neural Ordinary Differential Equations

If you would like to see more videos like this please consider supporting me on Patreon -www.patreon.com/andriydrozdyuk
The PDF of the slides used in this presentation can be downloaded here: github.com/drozzy/Neural-Ordinary-Differential-Equations-KZbin
0:00 - Outline of the presentation
0:38 - Some Cool Results
2:12 - What is a Neural ODE? (Machine Learning Part)
12:15 - Connection to Dynamical Systems
14:26 - Dynamical Systems
20:03 - Pendulum, Example of a Dynamical System
23:22 - Adjoint Method
28:45 - Adjoint Method Proof
30:49 - Gradients w.r.t. theta
32:40 - Complete Backprop Algorithm
34:27 - Concluding Remarks

[@shorray]

Herr YEAH! i was fighting like for 2 Weeks with the adjoint method and nobody really explained like this in detail. Thanks a lot keep going!

[@kodfkdleepd2876]

Or maybe it just took you 2 weeks to get it and you just happen to be watching this video when it "clicked"?

[@keb785]

This is very helpful; I appreciate it as it provides a comprehensive review with detailed explanations

[@tanmayahmed4622]

Thank You very much, Sir. This is by far most easy explanation of neural ODE.

[@vishwajitkumarvishnu3878]

Best video/blog so far on neural ODEs

[@mohamedmusa7149]

Excellent exposition of the paper! Thank you.

[@fbf3628]

This is a truly great explanation!

[@jishnuak3000]

Thanks for explaining the proof, couldn't find it anywhere else

[@siddharthshrivastava5823]

Awesome explanation!!

[@mswification]

I agree with all the previous comments, this was a terrific explanation. I particularly appreciated that you included details of the proof of the adjoint method.

[@Rjsipad]

could you explain the difference between lower case f and theta? I'm a bit confused as to how they are different

[@hannes7218]

great explanation! :)

[@gamebm]

Thanks for the video and detailed derivation. There is a question/comment which really puzzles me. First, if L is a number, which measure the deviation (eg. absolute difference between) of estimation z(t) from the real value, then the mapping z(t) -> L is a functional by definition. We would have, instead, \delta L/\delta z(t) = a(t) naturally defined as a functional derivative. However, as I tried to follow the arguments used in this video (29:28), I realized that it changes a lot (as dz(t+\Delta t)/dz(t) does have its counterpart in the context of functional). So I was forced to understand that the notion of functional is irrelevant here, one defines a number L as a function of another number z(t) which is a function evaluated at a given time, but must not be understood as a function of time. Under such an enforced context, the derivation then makes sense. PS: please do not use "partial" in the numerator and "d" in the denominator, as I don't believe this is the standard.

[@gamebm]

On a second thought, a(t) is indeed reminiscent of the functional derivative defined as a(t)=\delta L/\delta z(t). It is very inviting to state that for z(t) satisfying \dot{z}=f(z,t), one has \dot{a}=-a \partial f/\partial z. Except for its variation the function z(t) is not fixed at both end points as per the definition of functional differential, therefore it one follows that path (which mostly should work), one must introduce some proper modifications.

[@bwan03]

Brilliant! You're really good at explaining I must say. Excellent job! May I please ask what you used for presentation and drawing equations Andriy?

[@AndriyDrozdyuk]

Thanks! I think it was GoodNotes with ipad screen recording and apple pencil. (I just cut the surrounding window borders in the final recording)

[@tenkunvan60]

28:44, i think the backward equation of the adjoint method might be wrong and the integral term should be negative

[@francoisgauthier-clerc6413]

Good job, very clear explanation ! However, it's sad that you didn't introduce some implementation of the function f. How can we design and implement such continuous function ?

[@AndriyDrozdyuk]

Oh that function doesn't exist - that's just for explanation purposes. This is what ODE solver does basically.

[@XupyachkaX]

Здравствуйте, Андрей! Так всё-таки, не могу понять зачем гонять нейронки для дифуров? Дифуры строятся на законах, а нейронки это вроде как хитрые интерполяции которые работают только в узкой, натренированной области значений. Или нейронка может составить краткое дифуравенение по наблюдениям за динамической системой решая обратную задачу?

[@boriscrisp518]

illegible hand written scrawl... just like my undergrad days

[@danielschwegler5220]

Thanks for the superb explanation!