Reinforcement Learning Series: Overview of Methods

This video introduces the variety of methods for model-based and model-free reinforcement learning, including: dynamic programming, value and policy iteration, Q-learning, deep RL, TD-learning, SARSA, policy gradient optimization, among others.
Citable link for this video: doi.org/10.52843/cassyni.jcgdvc
This is the overview in a series on reinforcement learning, following the new Chapter 11 from the 2nd edition of our book "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Book Website: databookuw.com
Book PDF: databookuw.com/databook.pdf
RL Chapter: faculty.washington.edu/sbrunt...
Amazon: www.amazon.com/Data-Driven-Sc...
Brunton Website: eigensteve.com
This video was produced at the University of Washington

[@EkShunya]

I deeply appreciate the quality of knowledge you are providing to the community. please continue to democratise knowledge.

[@alliwant8383]

Superb. One of the things I always I struggle with when learning something is having a well structured map in my head of the topic and subtopics and this does an extremely good job of doing that. Many thanks.

[@tljstewart]

This russian doll of dichotomies has always been a mind bender, often it seems the literature has nebulous definitions and the boundaries aren't so clear. Thank you for the great insights in this lecture, the graphic is superb.

[@Eigensteve]

Thanks!

[@saitaro]

OK, this year is gonna be better than I thought. Thanks, professor!

[@akino.3192]

Wow! Steve, you've managed to break this all down into bite-sized chunks. Thank you 🙏

[@complexobjects]

I just started getting back into RL so this comes at a perfect time! Looking forward 👌

[@jett_royce]

Fantastic opening video. You're a talented teacher and I appreciate this content. Looking forward to watching the entire series.

[@MrAndreaCaso]

Finally! Thank you for you posting. Can't wait to see the whole playlist.

[@eduardolussi104]

Such a high quality course and a free book in description? You're awesome!!

[@Brian-ft4dh]

Really really great overview for those new to learning about reinforcement learning! Thanks so much!

[@MrRafaelSencio]

It's been a great series of videos on RL! I'm updating my research interests and now I want to combine MPC with RL in such a way that the resulting control structure can be safely implemented and has some stability guarantees. Thank you very much!

[@cornevanzyl5880]

You helped me during my undergrad, now you're an inspiration to me during my masters.

[@thrinayreddy3379]

Please make a separate playlist for reinforcement learning :-)

[@Eigensteve]

Good call -- will do

[@rudzanimulaudzi7947]

@@Eigensteve please put the videos in order, the current order is not correct. But great content.

[@quantum01010101]

An excellent arrangement of a very tough topic, logical and in the proper flow, keep up the very good job Thank you.

[@shashwatpandey4213]

This is series is life saver ! Thanks Steve !

[@Ceznex]

Coming to this video after a while. Really great video, thank you!!

[@GaetanLepage]

Great synthetic and dense video ! Thank you very much for sharing !

[@XenoZeduX]

What an amazing start to the new year! 😍

[@samirelzein1095]

Great teacher and master of the art!

[@cabbagecat9612]

Great video! Though imho the on/off-policy distinction explained at 14:24 might be a bit misleading. I believe both on/off policy can explore sub-optimal actions with something like epsilon-greedy.

[@apurvdhir7062]

Needed this..... Thank you Professor

[@saeedparsamehr9884]

I am really grateful for your eye-opening videos, especially this one

[@Eigensteve]

Thanks!

[@kuanxD]

this explanation is just beautiful! Thx so much

[@ianlorondealmeida9680]

Great didactic, congratulations! I used to confuse myself frequently when dealing with these concepts.

[@zhehaoli1999]

Great overview, which is just what I need, thank you sir!

[@virgenalosveinte5915]

Steve, you are amazing.

[@haotianhang3997]

Thank you! Happy New Year!

[@MCMelonslice]

This is amazing. Thank you, steve!

[@resalatbinafsar7907]

Hi Mr. Brunton, Your videos are impressive and thank you for making the content. A small suggestion though, it will be better for us to navigate if you make separate playlists with orders in a particular content.

[@ZZ-dr7uf]

Thank you prof for providing your book in PDF format

[@mohammadabdollahzadeh268]

Dear steve its amazing category to classify the reinforcement learning thanks alot

[@user-ie6tf4wf7p]

You really invest perfectly the time in your lessons, very very useful! great series!

[@Eigensteve]

Thanks! :)

[@jiaqint961]

The dichotomy break down are so awesome...

[@anonymous-tt2lm]

The heart of AI is reinforcement learning, it is the only most interesting in whole AI/ml. Basically original AI . Thanks professor 🤝👍

[@Ajay-xd7zq]

Thanks so much for such great video. Can you please tell where does Proximal Policy Optimization (PPO) fit in these categories. For my case a small game, I know that I will be using model free RL, but not able to decide what else to use apart from Q-Learning.

[@danish32100]

Great as always.

[@jadavdas5405]

Nice lectures and lots of stuffs to learn. Thanks for sharing. Are the On policy and Off policy somehow related to exploitation and exploration concept?

[@Spiegeldondi]

I am a happy owner of you "Data Driven Science and Engineering" book. That fact that there will be much more content on RL in the 2nd edition is really good news! Will there also be a print version of the 2nd edition of your and Kutz' Book?

[@Eigensteve]

Thanks! Yes, the print edition should be out sometime later this year.

[@AlexandreGirard87]

Very nice overview video! There is a small typo in the non-linear dynamic equation, the superflous dt on the right. Regarding how MPC fit in the whole DP framework, I remember Prof. Bertsekas was presenting it as a way to approximate cost-to-go online.

[@Eigensteve]

Good catch on the typo! And interesting perspective on MPC -- thanks!

[@rajmeetsingh1625]

Thanks, Sir, Please add some robotics-related examples in the upcoming series also.

[@Eigensteve]

Thanks for the suggestion!

[@RasitEvduzen]

Thnx for video professor.

[@C7ZR1]

Good stuff! You need to remove the "dt" on the right side of your nonlinear dynamics equation.

[@drsandeepvm5622]

Great presentation 👏

[@Janamejaya.Channegowda]

Thank you for sharing.

[@thanh315960000]

Thank you!

[@sinarezaei218]

thanks for your great videos

[@bonaldli]

Dear Steve, great explanation. However, just wanna confirm: I thought Actor-Critic is a model-free model?

[@tienphatbui9827]

At 20:07 Steve said about the "model of system" that if we have a "model of system" we use the model-base, and if we don't have a "model of system" just use model-free. So, can you explain me more about "model of system". What is it? Example? and Why? Thank you so much.

[@ShahFahad-hj1ps]

Great job Prof. Steve. How about multi-agent based DRL especially graph learning based RL. That can be a remarkable addition to your playlist.

[@Matlockization]

I enjoy your broad strokes topics. I was wondering can an AI write to its memory once it learns or discovers something new ? Or it doesn't work like that.

[@vermashwetank]

Great series! Can you make a contro bootcamp like series for non-linear control theory? Would love to see some simplified explanations for topics like PDE backstepping, reference governors, lyapanov stability criteria etc.

[@Eigensteve]

Would love to do a bootcamp on this -- maybe a goal for the new year! :)

[@alimustafa2682]

I would like to have RL as a career , and you would be the best lecturer for a kickstart.

[@mortezaaliyari8818]

Thanks for the wonderful videos. It would be great if you add real code to the end of the main videos. it would be very easier to understand with detail.

[@paria4393]

I have Energy data and I need to implement RL on these data (Inverter) to achieve the best result (when charge/discharge battery, when is the best time to feed in grid,etc.) which algorithm should I use for that ?

[@justlaugh8804]

Actor critic should be in the policy gradient optimization no?

[@kundankumar-dt5uu]

Sir, it model free algorithm uses Marakove decision proce (MDP)?

[@karthiknn8836]

Hello Professor, where will DDPG algorithm sit in this chart?

[@marofe]

There is a typo at 10:08, the dynamic model in continuous time should be dx/dt=f(x,u,t) only

[@Eigensteve]

Good catch, thanks!

[@PippyPappyPatterson]

0:00 Intro 3:00 Background 7:54 Model & Model-Free Reinforcement Learning (RL) 8:29 Markov Decision Process (MDP) 10:25 Nonlinear Dynamics 13:02 Gradient & Gradient-Free RL 14:05 Off-Policy (Q Learning) & On-Policy (SARSA) RL 17:23 Policy Gradient Optimization 18:05 Deep RL

[@ArmanAli-ww7ml]

I was reading 1 journal article and found that the author claimed following a model-free RL problem but they have used Markov Decision process to model the problem? they have not mentioned probabilities for states. What does that mean? Also there is another paper which used probabilities for state transitions and solved the problem using Q-learning, so it's all confusing again.

[@hamidhussain5488]

What is difference between deterministic policy and stochastic policy?

[@Oliver-cn5xx]

hi steve, I think actor critic are usually considered model-free

[@Moonz97]

At 14:20 to 14:41, you state On-Policy is always playing the best game possible. Is this approach the same as greedily picking the best action at each state? If so, would On-Policy algorithms not include exploration such as epsilon-greedy? The way I understood On-Policy vs Off-Policy here is that On-Policy is purely exploitation whereas Off-Policy is both exploration & exploitation. Am I misunderstanding it? Thanks!

[@kunqian6243]

I was also a bit confused at that point. But, I think you have probably misunderstood it since, for RL, we have to always ensure exploration & exploitation. I feel that what I understand is completely the opposite to Prof. Steve's description: on-policy uses the to be improved policy to select an action (meaning not always the best action), while the off-policy uses a different policy to decide which action to take (you may always choose the best action). I hope Steve will elaborate more on it. :)

[@WhenThoughtsConnect]

oil lamp except the clear liquid is a gradient and the bubbles are parameters that perform an action. then its p>p' goes to q, then flipflops p q>q'. p'=p or q'=q if the AI improves at a particular task. two randoms cancel each other out on an error function and acts like an implicit rolles theorem without explicitly stating d/dx=0.

[@pedrowangler97]

The distinction between On-Policy and Off-Policy explained in this video seems to be different from other sources on the internet. I'm trying to get my head around reinforcement learning and I have noticed that different people have different understandings of certain concepts. Model-free and Model-based are also given a different distinction by others, and this really throws me off. I'm not saying the explanation in this video is incorrect, but that there are different explanations elsewhere and I'm not sure which one is correct.

[@georgefarnon2432]

Excellent. What other topics will be included in the 2nd edition?

[@Eigensteve]

Updates throughout, all code in Python and Matlab (with R and Julia online), and new chapters on RL and physics informed machine learning

[@mohammadabdollahzadeh268]

Dear steve we can use ls algorithm instead of gradient algorithm isn’t it

[@ArmanAli-ww7ml]

Can anyone explain RL by comparing it with ML mathematically? I know much about ML but getting trouble understanding RL.

[@andreas-lebedev]

Why is "Actor Critic" assigned (only) to the left side? Isn't it (also) a combination of gradient free and gradient based algorithms, e.g. the Critic is a DQN and the Actor is a Deep Policy Network?

[@ayushroy6208]

I just have one doubt, is A3C a model free one?

[@mohammadabdollahzadeh268]

Dear professor, please explain to us about how to use reinforcement learning to tune pid gains ❤️ Im looking forward to hearing from you Sincerely mohammad

[@mohammadabdollahzadeh268]

Dear professor please explain to us how to use reinforcement learning to tune pid controller gains I’m looking forward to hearing from you Sincerely mohammad

[@1812aks]

Off plolicy..on policy is slightly confusing here...isnt off policy, a setup where you have the prior data and cant continuously interact with the environment

[@csalahuni]

The link for the new chapter of the 2nd edition of the book is not working for me. Can someone post the correct link in the comments?

[@AliRashidi97]

Yeah it doesnt work :(

[@Eigensteve]

Maybe use databookuw.com/databook.pdf

[@csalahuni]

Thank you, this link is working, but it still shows the 1st edition of the book I think.

[@Eigensteve]

@@csalahuni Shoot, sorry, here is the chapter: faculty.washington.edu/sbrunton/databookRL.pdf added to description too

[@AliRashidi97]

@@Eigensteve tnx professor 🙏

[@Shaunmcdonogh-shaunsurfing]

Excellent sunmary

[@alexanderskusnov5119]

Will we see the programs teaching each other? (like chess)

[@nightsailor1]

Sound level is low.

[@johnalley8397]

Bated breath. No, really. Hurry uuuuh-uuuup!