Reinforcement Learning Series: Overview of Methods

[@EkShunya]

I deeply appreciate the quality of knowledge you are providing to the community. please continue to democratise knowledge.

[@tljstewart]

This russian doll of dichotomies has always been a mind bender, often it seems the literature has nebulous definitions and the boundaries aren't so clear. Thank you for the great insights in this lecture, the graphic is superb.

[@Eigensteve]

Thanks!

[@alliwant8383]

Superb. One of the things I always I struggle with when learning something is having a well structured map in my head of the topic and subtopics and this does an extremely good job of doing that. Many thanks.

[@alireza202]

The main difference between on-policy and off-policy methods is the way the generated data is used during learning. If the method uses the data from the current policy, it is on-policy learning. But if the method uses data from another policy (e-greedy or older policies like replays), it is an off-policy method.

[@cabbagecat9612]

Great video! Though imho the on/off-policy distinction explained at 14:24 might be a bit misleading. I believe both on/off policy can explore sub-optimal actions with something like epsilon-greedy.

[@cornevanzyl5880]

You helped me during my undergrad, now you're an inspiration to me during my masters.

[@akino.3192]

Wow! Steve, you've managed to break this all down into bite-sized chunks. Thank you 🙏

[@thrinayreddy3379]

Please make a separate playlist for reinforcement learning :-)

[@Eigensteve]

Good call -- will do

[@rudzanimulaudzi7947]

@@Eigensteve please put the videos in order, the current order is not correct. But great content.

[@saitaro]

OK, this year is gonna be better than I thought. Thanks, professor!

[@complexobjects]

I just started getting back into RL so this comes at a perfect time! Looking forward 👌

[@eduardolussi104]

Such a high quality course and a free book in description? You're awesome!!

[@MrAndreaCaso]

Finally! Thank you for you posting. Can't wait to see the whole playlist.

[@anonymous-tt2lm]

The heart of AI is reinforcement learning, it is the only most interesting in whole AI/ml. Basically original AI . Thanks professor 🤝👍

[@Brian-ft4dh]

Really really great overview for those new to learning about reinforcement learning! Thanks so much!

[@MrRafaelSencio]

It's been a great series of videos on RL! I'm updating my research interests and now I want to combine MPC with RL in such a way that the resulting control structure can be safely implemented and has some stability guarantees. Thank you very much!

[@GabrielBYPF]

You really invest perfectly the time in your lessons, very very useful! great series!

[@Eigensteve]

Thanks! :)

[@saeedparsamehr9884]

I am really grateful for your eye-opening videos, especially this one

[@Eigensteve]

Thanks!

[@PippyPappyPatterson]

0:00 Intro 3:00 Background 7:54 Model & Model-Free Reinforcement Learning (RL) 8:29 Markov Decision Process (MDP) 10:25 Nonlinear Dynamics 13:02 Gradient & Gradient-Free RL 14:05 Off-Policy (Q Learning) & On-Policy (SARSA) RL 17:23 Policy Gradient Optimization 18:05 Deep RL

[@jett_royce]

Fantastic opening video. You're a talented teacher and I appreciate this content. Looking forward to watching the entire series.

[@rajmeetsingh1625]

Thanks, Sir, Please add some robotics-related examples in the upcoming series also.

[@Eigensteve]

Thanks for the suggestion!

[@quantum01010101]

An excellent arrangement of a very tough topic, logical and in the proper flow, keep up the very good job Thank you.

[@jiaqint961]

The dichotomy break down are so awesome...

[@Ceznex]

Coming to this video after a while. Really great video, thank you!!

[@AlexandreGirard87]

Very nice overview video! There is a small typo in the non-linear dynamic equation, the superflous dt on the right. Regarding how MPC fit in the whole DP framework, I remember Prof. Bertsekas was presenting it as a way to approximate cost-to-go online.

[@Eigensteve]

Good catch on the typo! And interesting perspective on MPC -- thanks!

[@bonaldli]

Dear Steve, great explanation. However, just wanna confirm: I thought Actor-Critic is a model-free model?

[@Spiegeldondi]

I am a happy owner of you "Data Driven Science and Engineering" book. That fact that there will be much more content on RL in the 2nd edition is really good news! Will there also be a print version of the 2nd edition of your and Kutz' Book?

[@Eigensteve]

Thanks! Yes, the print edition should be out sometime later this year.

[@mohammadabdollahzadeh268]

Dear steve its amazing category to classify the reinforcement learning thanks alot

[@alimustafa2682]

I would like to have RL as a career , and you would be the best lecturer for a kickstart.

[@ianlorondealmeida9680]

Great didactic, congratulations! I used to confuse myself frequently when dealing with these concepts.

[@jacekbudzisz380]

Great channel. You have a knowledge in the field that I was looking for that is mix on AI/NN and control theory.

[@resalatbinafsar7907]

Hi Mr. Brunton, Your videos are impressive and thank you for making the content. A small suggestion though, it will be better for us to navigate if you make separate playlists with orders in a particular content.

[@XenoZeduX]

What an amazing start to the new year! 😍

[@pedrowangler97]

The distinction between On-Policy and Off-Policy explained in this video seems to be different from other sources on the internet. I'm trying to get my head around reinforcement learning and I have noticed that different people have different understandings of certain concepts. Model-free and Model-based are also given a different distinction by others, and this really throws me off. I'm not saying the explanation in this video is incorrect, but that there are different explanations elsewhere and I'm not sure which one is correct.

[@vermashwetank]

Great series! Can you make a contro bootcamp like series for non-linear control theory? Would love to see some simplified explanations for topics like PDE backstepping, reference governors, lyapanov stability criteria etc.

[@Eigensteve]

Would love to do a bootcamp on this -- maybe a goal for the new year! :)

[@apurvdhir7062]

Needed this..... Thank you Professor

[@marofe]

There is a typo at 10:08, the dynamic model in continuous time should be dx/dt=f(x,u,t) only

[@Eigensteve]

Good catch, thanks!

[@GaetanLepage]

Great synthetic and dense video ! Thank you very much for sharing !

[@C7ZR1]

Good stuff! You need to remove the "dt" on the right side of your nonlinear dynamics equation.

[@mortezaaliyari8818]

Thanks for the wonderful videos. It would be great if you add real code to the end of the main videos. it would be very easier to understand with detail.

[@kuanxD]

this explanation is just beautiful! Thx so much

[@WhenThoughtsConnect]

oil lamp except the clear liquid is a gradient and the bubbles are parameters that perform an action. then its p>p' goes to q, then flipflops p q>q'. p'=p or q'=q if the AI improves at a particular task. two randoms cancel each other out on an error function and acts like an implicit rolles theorem without explicitly stating d/dx=0.

[@AksGu2]

Thanks so much for such great video. Can you please tell where does Proximal Policy Optimization (PPO) fit in these categories. For my case a small game, I know that I will be using model free RL, but not able to decide what else to use apart from Q-Learning.

[@MCMelonslice]

This is amazing. Thank you, steve!

[@samirelzein1095]

Great teacher and master of the art!

[@Moonz97]

At 14:20 to 14:41, you state On-Policy is always playing the best game possible. Is this approach the same as greedily picking the best action at each state? If so, would On-Policy algorithms not include exploration such as epsilon-greedy? The way I understood On-Policy vs Off-Policy here is that On-Policy is purely exploitation whereas Off-Policy is both exploration & exploitation. Am I misunderstanding it? Thanks!

[@kunqian6243]

I was also a bit confused at that point. But, I think you have probably misunderstood it since, for RL, we have to always ensure exploration & exploitation. I feel that what I understand is completely the opposite to Prof. Steve's description: on-policy uses the to be improved policy to select an action (meaning not always the best action), while the off-policy uses a different policy to decide which action to take (you may always choose the best action). I hope Steve will elaborate more on it. :)

[@jadavdas5405]

Nice lectures and lots of stuffs to learn. Thanks for sharing. Are the On policy and Off policy somehow related to exploitation and exploration concept?

[@ShahFahad-hj1ps]

Great job Prof. Steve. How about multi-agent based DRL especially graph learning based RL. That can be a remarkable addition to your playlist.

[@zhehaoli1999]

Great overview, which is just what I need, thank you sir!

[@georgefarnon2432]

Excellent. What other topics will be included in the 2nd edition?

[@Eigensteve]

Updates throughout, all code in Python and Matlab (with R and Julia online), and new chapters on RL and physics informed machine learning

[@Oliver-cn5xx]

hi steve, I think actor critic are usually considered model-free

[@tienphatbui9827]

At 20:07 Steve said about the "model of system" that if we have a "model of system" we use the model-base, and if we don't have a "model of system" just use model-free. So, can you explain me more about "model of system". What is it? Example? and Why? Thank you so much.

[@jakal0282]

Model of the system means you know every possible state of the system, every possible next state and the actions that take you there (& with what probability) as well as the rewards earned after every state & action pair. You can have a model of a maze, you CANT have a model of a chess game (unless you make assumptions about the opposing players strategy).

[@paria4393]

I have Energy data and I need to implement RL on these data (Inverter) to achieve the best result (when charge/discharge battery, when is the best time to feed in grid,etc.) which algorithm should I use for that ?

[@RasitEvduzen]

Thnx for video professor.

[@sinarezaei218]

thanks for your great videos

[@drsandeepvm5622]

Great presentation 👏

[@thanh315960000]

Thank you!

[@virgenalosveinte5915]

Steve, you are amazing.

[@andreas-lebedev]

Why is "Actor Critic" assigned (only) to the left side? Isn't it (also) a combination of gradient free and gradient based algorithms, e.g. the Critic is a DQN and the Actor is a Deep Policy Network?

[@ayushroy6208]

I just have one doubt, is A3C a model free one?

[@mairios521]

I am a beginner in RL field... i think A2C is an actor critic algo as well (and It's a model-free RL)

[@justlaugh8804]

Actor critic should be in the policy gradient optimization no?

[@haotianhang3997]

Thank you! Happy New Year!

[@Matlockization]

I enjoy your broad strokes topics. I was wondering can an AI write to its memory once it learns or discovers something new ? Or it doesn't work like that.

[@danish32100]

Great as always.

[@hamidmirza333]

What is difference between deterministic policy and stochastic policy?

[@kundankumar-dt5uu]

Sir, it model free algorithm uses Marakove decision proce (MDP)?

[@ArmanAli-ww7ml]

I was reading 1 journal article and found that the author claimed following a model-free RL problem but they have used Markov Decision process to model the problem? they have not mentioned probabilities for states. What does that mean? Also there is another paper which used probabilities for state transitions and solved the problem using Q-learning, so it's all confusing again.

[@mohammadabdollahzadeh268]

Dear professor, please explain to us about how to use reinforcement learning to tune pid gains ❤️ Im looking forward to hearing from you Sincerely mohammad

[@mohammadabdollahzadeh268]

Dear steve we can use ls algorithm instead of gradient algorithm isn’t it

[@csalahuni]

The link for the new chapter of the 2nd edition of the book is not working for me. Can someone post the correct link in the comments?

[@AliRashidi97]

Yeah it doesnt work :(

[@Eigensteve]

Maybe use databookuw.com/databook.pdf

[@csalahuni]

Thank you, this link is working, but it still shows the 1st edition of the book I think.

[@Eigensteve]

@@csalahuni Shoot, sorry, here is the chapter: faculty.washington.edu/sbrunton/databookRL.pdf added to description too

[@AliRashidi97]

@@Eigensteve tnx professor 🙏

[@Janamejaya.Channegowda]

Thank you for sharing.

[@karthiknn97]

Hello Professor, where will DDPG algorithm sit in this chart?

[@Shaunmcdonogh-shaunsurfing]

Excellent sunmary

[@ArmanAli-ww7ml]

Can anyone explain RL by comparing it with ML mathematically? I know much about ML but getting trouble understanding RL.

[@mohammadabdollahzadeh268]

Dear professor please explain to us how to use reinforcement learning to tune pid controller gains I’m looking forward to hearing from you Sincerely mohammad

[@1812aks]

Off plolicy..on policy is slightly confusing here...isnt off policy, a setup where you have the prior data and cant continuously interact with the environment

[@alexanderskusnov5119]

Will we see the programs teaching each other? (like chess)

[@nightsailor1]

Sound level is low.

[@HarshPatel-g2q]

Actor critic is model based ???? I dont think so.

[@johnalley8397]

Bated breath. No, really. Hurry uuuuh-uuuup!