Q-Learning: Model Free Reinforcement Learning and Temporal Difference Learning

Here we describe Q-learning, which is one of the most popular methods in reinforcement learning. Q-learning is a type of temporal difference learning. We discuss other TD algorithms, such as SARSA, and connections to biological learning through dopamine. Q-learning is also one of the most common frameworks for deep reinforcement learning.
Citable link for this video: doi.org/10.52843/cassyni.ss11hp
This is a lecture in a series on reinforcement learning, following the new Chapter 11 from the 2nd edition of our book "Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control" by Brunton and Kutz
Book Website: databookuw.com
Book PDF: databookuw.com/databook.pdf
Amazon: www.amazon.com/Data-Driven-Sc...
Brunton Website: eigensteve.com
This video was produced at the University of Washington

[@thiagocesarlousadamarsola3990]

I personally love the big picture perspective that Prof. Brunton always shows. Please, continue to make these high quality videos!

[@Mohammad-gj8ir]

👍👍👍

[@alirezatavakoli7325]

The videos are wonderful! Thank you, professor.

[@TheSinashah]

CS PhD student here. This video provides such amazing content. Highly recommended.

[@usonian11]

Thank you for the outstanding production quality and content of these lectures! I especially enjoy the structure diagram organizing the different RL methods.

[@davidelicalsi5915]

Professor I must sincerely thank you for the astonishing quality of this video. You were able to clearly explain an advanced concept without simplifying, going into the details and providing brilliant insights. Also I sincerely thank you for saving my GPA from my R.L. exam 😆

[@kalimantros845]

I was hoping that your next video would have been about Q-learning, and here it comes!

[@jashwantraj2987]

Prof. Burton, you are amazing. I never expected someone to take so much of time to explain a concept about TD. I'm one of the few people who hate reading text books to understand concepts. I rather see a video or learn about it from class. Thanks a lot

[@haotianhang3997]

Thank you! It's a great video. My understanding in TD learning was deepened a lot.

[@OmerBoehm]

Thank you dear Prof Brunton for this outstanding lecture. The detailed explanations and focus on subtleties are so important , Looking forward to your next videos.

[@imanmossavat9383]

I enjoy your talks. They are very clear and well structured and have the right level of detail. Thank you,

[@areebayubi5469]

Thank you so much for using very relevant analogies and very clear explanations. I think I have a much better grasp of the concepts behind Temporal Difference learning now.

[@BoltzmannVoid]

this was the best explanation ever! thank you so much, professor!

[@cruise0101]

Excellent class! Extremely easy to understand!

[@anlehoang7030]

More casual example for TD-learning: Imagine a curious robot exploring a maze, searching for a hidden treasure. Unlike other methods that wait until it finds the treasure to learn, TD learning is all about learning on the fly. It uses what it already knows (like the estimated value of different paths) and immediate feedback (rewards) to improve its predictions about future moves. - The robot keeps track of a Q-value (Q(s_t, a_t)) for each path, which tells it how good it thinks that path is based on its past experiences. - When it takes a path and gets a Q-value (or reward) (like finding a clue), it compares that reward to what it expected (based on the Q-value - r_t + \gamma Q(s_{t + 1}, a_{t + 1})). This difference is called the prediction error. - If the reward is better than expected (positive error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) > 0), the robot increases the Q-value for that path, making it seem more attractive next time. - If the reward is worse than expected (negative error, or r_t + \gamma V^{old}(s_{t + 1}) - V^{old}(s_t) < 0), the robot decreases the Q-value, steering it away from less promising paths.

[@multiversityx]

You gave the best explanations I've ever seen!

[@krullebolalex]

Thanks a bundle Steve, this was really well explained!

[@farhadebrahimzadeh3420]

Thank you for the clear picture. It was really well explained and others already mentioned, now I can say that I understand these techniques quite fairly well. 🙏

[@FRANKONATOR123]

Hi Prof. Brunton. Great vídeo as always! Please keep producing quality ML content

[@marzs.szzzzz]

These are fantastic lectures, I use these as an alternative explaination to David Silvers DeepmindxUCL 2015 lectures on the same topic, the different perspective really suits how my brain understands RL. Thank you!!

[@TheFitsome]

This is the best RL tutorial on the internet.

[@denchen1950]

The video quality is incredible lol and all the concept is discussed extremely clear OMG!! Brilliant masterpiece bro KEEP GOING !!

[@antimon40]

Somehow I find that the explanations given by Prof. Brunton are easier to understand than those provided by video lectures from Stanford (which are also available on KZbin).

[@prateekcaire4193]

Thanks a lot. Not just math but also the intuition that i was looking for

[@jicabe577]

Thanks a lot, Prof. Brunton!

[@vamsimanoharreddy1468]

Excellent Explaination

[@user-cj5ff8mw6v]

I very like your descriptions about deep learning

[@maria4880]

Thank you so much for these lectures sir!

[@anirudhthatipelli8765]

Thanks for the fantastic explanation!

[@Julsten3107]

Such a great video, thank you!

[@jimlbeaver]

Great explanation..very clear!

[@nahuelpiguillem2949]

Thanksssss steveeee. I couldnt understand nothing before this video.Thanks again

[@complexobjects]

I do like the description of Q Learning. I had come up with another analogy for why it makes sense. If you took the action of going out to a party, and then happened to make some mistakes while there, we wouldn't want to say "you should never go out again." We'd want to reinforce the action of going out based on the best* possible outcome of that night, not the suboptimal action that was taken once there.

[@r.d.7575]

Read the chapter, and I've been waiting for this video for a while. Happy to know I'm the first to comment :) Thanks, Steve. Can you explain more the bias (and especially the variance) in RL in a later video ?

[@Throwingness]

Loud and clear.

[@ajaykumar-rh2gz]

Hi Steve, Thanks for Amazing! lecture. I think the mail challenge in RL is designing our own custom environment (Multiple states and actions). It will be a great help if you can upload some lecture, suggest some link to do this job. Other comments are also welcome. Currently, I am doing some experiment on Retail pricing optimization using offline data. Looking forward.

[@stuartferguson11]

This whole series was good, but this one pushed me past my confusion. My neural network finally learned to play tic-tac-toe!

[@mariogalindoq]

Steve: thank you again. I appreciate your work. Trying to help, let me say that I believe there is a small typo, at minute 5:29 you wrote π(s,a) = argmax_a Q(s,a), should it be written π(s) = argmax_a Q(s,a)? Also, at time 22:35, the first equation has a sum over k, should it be over n? Anyway, this is a very good video.

[@somethingirreversib]

Great lecture!

[@yuktikaura]

Very well explained.

[@BipinOli90]

Brilliant! Thanks 🙏

[@caiolp4]

Great lecture! Concerning the difference between SARSA and Q-Learning, I didn't get the emphasis on Q-Learning being better for exploration. In principle, one can choose a epsilon-greedy for both methods. As a matter of fact, the SARSA method is defined in Sutton's book with an epsilon-greedy policy. I get the point that the TD target of Q-Learning does not depend on the policy itself and, therefore, is called an off-policy method. However, if one can choose a exploratory policy (e.g., epsilon-greedy) for both methods, why would SARSA be safer or less exploratory?

[@duncanw9901]

I got the impression he was asserting that the updates to the quality function can/will/often become an undesireable feedback loop when non-optimized states are used, and I would infer that means the training steps done on those states would have an undesireably high probability of entering such states. What you said does seem to be convincing evidence otherwise though.

[@jorge-george6958]

That is a correct comment. The difference between the two lies in the Q-function updates. The way you choose your action is orthogonal (and can be more/less exploratory in either method). Also, from the video, Q-learning comes off as "better" method than SARSA, at least in problems where you don't need safe exploration, which is not accurate. It's more like a trade-off, where no method is clearly better. I love your videos in general. I think though, that this particular one needs a bit of a revision. Hope you don't see this as a critique, but rather as constructive feedback.

[@polinagrinko1678]

brilliant explanation

[@Chetan_Hansraj]

woww thank you , so well explained with lot of patience .. god bless

[@martinschulze5399]

being a phd student of you must be a gift :D

[@faqeerhasnain]

In value able Content.. Cant thank enough.

[@51nibbler]

ty 4 great explain greeze from switzerland :)

[@Pedritox0953]

Great video! Would be nice a simple example

[@dbracale]

Nice talk al always. A question: at minute 5:00, does pi depend on a? It shoud not, right? The same holds for the previous video.

[@miminh98]

Hello :) thank you for the video. I have a small question. I don't fully understand the difference between TD(0) and SARSA. Indeed, if SARSA uses the optimal action 'a' at each time step 'k', doesn't the Q-function in SARSA equal the Value function in TD(0) ? Or was there an error in my understanding ? Can you please help me see more clearly ? :)

[@chymoney1]

Great stuff

[@samirelzein1095]

top prof!

[@farhadebrahimzadeh3420]

there is a question that pops up in my head; if SARSA is an on-policy method, then is it OK to use e-greedy algorithm in SARSA? as you mentioned it always take into account taking the safe and on-policy action rather than random and off-policy actions?

[@anupamadhikari139]

Can you explain how off policy q learning can take a suboptimal route sometimes if you are always taking the max of the actions presented?

[@XandreClementsmith]

In the Monte Carlo method, you have the reward discounted by gamma. Why do you discount a reward function for the entire episode? Furthermore, 1/n(R) would not the average reward for any step k.

[@chuanjiang6931]

One question, in terms of updating the Q function using the observed(real) reward at state k + 1, how do we know the observed(real) reward at state k + 1 since it is one timestamp in future?

[@David-nw6rz]

Great lecture, but I guess your definition of on/off policy is different from the definition of Sutton/Barto. On policy doesn't necessarily mean you always take the optimal action. "[on-policy] learns action values not for the optimal policy, but for a near-optimal policy that still explores" [excerpt from a different chapter but also valid for TD]. SARSA usually still follows a epsilon-greedy strategy.

[@tearistovic]

Thank youuu !

[@titaniumsheepdog]

what is the reward for r_k if we don't receive a reward after every action? Is it just assumed zero and the value is based only off the quality function?

[@linyidai9076]

Help a lot with my AI course final!!!!

[@preston748159263]

I would like to know what is being used to project the equations. It is apparently not video editing because he points directly to them.

[@JustinMasayda]

One thing that seems to be either an error or just inconsistent notation is the use of TD(N) to mean an N-step TD. It seems like the value in the parentheses is supposed the value of lambda, not the number of steps of TD. TD(0) apparently should be read as, "N-step TD when lambda = 0, " while TD(1) means, "N-step TD when lambda = 1." I'm basing this off of the book "Reinforcement Learning: An Introduction - Second edition" by Sutton and Barto.

[@bevansmith3210]

For MC you only get the reward at the end and then divide it up among all the states. But for TD, if you are only taking one step forward, where does the reward come from? A little confused here.

[@tubege]

Pi of s,a should only be a function of s since the RHS calculates a. What am I missing?

[@stevenchiu8560]

Could you please explain it with some examples, that will be really helpful to understand these formulas, thanks!

[@npr1m991]

This is amazing content. I just have a question (still struggle with the concept of on and off policy)... at 30:18, the max Q (in Q-learning) .. which Q is it old or new ?

[@eugeneL_N1E104]

old, but at next state $s_{k+1}$

[@JustinMasayda]

22:35 As someone else mentioned, the first equation has a sum over k, shouldn't it be over n?

[@rev0cdevs38]

I cannot access the new chapter in the 2nd edition. Has anybody accessed the link?

[@user-jg4mh6hb2g]

You are the best:))))))

[@sunaxes]

For TD0, Why not say we do alpha x the new value (s_k+1) + (1 - alpha) x the old value (s_k). It's a very basic update method...

[@dam-ib9fs]

very useful

[@dominic_lee]

wow, nice

[@--JYM-Rescuing-SS-Minnow]

👍

[@Throwingness]

A+

[@dihancheng952]

I don't think the comparison of q learning and sarsa is accurate.

[@mayfields5092]

isnt q-learning, learning to play master yi

[@djsocialanxiety1664]

too much talk too few examples

[@robert-dr8569]

Instead of using so many words to explain, why couldn't you just use a couple of examples to explain the relationship between Q(s, a) new vs Q(s', a')? It would be so easy to understand through examples.

[@sharannagarajan4089]

It’s a theoretical formula.

[@stevewu1920]

Instead of using so many words to complain, why couldn’t you just make a video illustrating these yourself?

[@romxpl4885]

@@stevewu1920worst take ever