A friendly introduction to deep reinforcement learning, Q-networks and policy gradients

A video about reinforcement learning, Q-networks, and policy gradients, explained in a friendly tone with examples and figures.
Introduction to neural networks: • A friendly introductio...
Introduction: (0:00)
Markov decision processes (MDP): (1:09)
Rewards: (5:39)
Discount factor: (8:51)
Bellman equation: (10:48)
Solving the Bellman equation: (12:43)
Deterministic vs stochastic processes: (16:29)
Neural networks: (19:15)
Value neural networks: (21:44)
Policy neural networks: (25:44)
Training the policy neural network: (30:46)
Conclusion: (34:53)
Announcement: Book by Luis Serrano! Grokking Machine Learning. bit.ly/grokkingML
40% discount code: serranoyt

[@-xx-7674]

This is probably the most friendliest video and still covering all important concepts of RL, thank you

[@zentootv4687]

Hands down, this explanation of reinforcement learning is like winning a dance-off against a robot-smooth, on point, and utterly unbeatable!

[@SerranoAcademy]

Thanks! Lol, I love it!

[@reyhanehhashempour8522]

Fantastic as always! Whenever I want to learn a new concept in AI, I always start with Luis's video(s) on that. Thank you so much, Luis!

[@achyuthvishwamithra]

I feel super fortunate to have come across your channel. You are doing an incredible job! Just incredible!

[@srinivasanbalan5903]

One of the best videos on RL algorithms. Kudos to Dr. Serrano.

[@colabpro2615]

you're one of the best teachers I have ever come across!

[@wfth1696]

One of the clearest explanations of the topic that I saw. Excellent!

[@renjithbaby]

This is the simplest explanation I have seen on RL! 😍

[@EshwarNorthEast]

The wait ends! Thank you sir!

[@jeromeeusebius]

Luis, great video. Thanks for putting this together explaining the most important concepts and terms in Reinforcement Learning.

[@therockomanz]

I'd like to thank the creators for this video. This is the best video to learn the basics of RL. Helped a lot in my learning path.

[@riddhimanmoulick3407]

Thanks for such a great video! Your visual descriptions combined with your explanations really presented a wonderful conceptual understanding of Deep-RL fundamentals.

[@geletamekonnen2323]

I Can't pass Without appreciating this great great Lecture. Thanks Luis serrano. 😍

[@alsahlawi19]

This by far the best video explaining DRL, many thanks!

[@jjhj_]

I've been bingewatching your "friendly intro to" series since yesterday and it has been amazing. I've worked with ML models as part of my studies and my work over the past two years, but even so, you've enriched my conceptual understanding by so much more than any of my professors could. Really appreciate your clever visualizations of what's going on "under the hood" of the ML/DL algo's. Great videos, awesome teacher!

[@SerranoAcademy]

Thank you, so happy to hear you’re enjoying the series! :)

[@user-cy2hk8bb3x]

Yeah

[@zhangeluo3947]

Yes@@user-cy2hk8bb3x

[@LuisGonzalez-jx2qy]

Amazing work fellow Luis! Looking forward to more of your videos

[@TheOnlyAndreySotnikov]

Great video!

[@emanuelfratrik1251]

Excellent explanation! Thank you!

[@Andy-rq6rq]

Amazing explanation! I was left confused after the MIT RL lecture but it finally made sense after watching this

[@NoNTr1v1aL]

Absolutely amazing video! You are my saviour!

[@shreyasdhotrad1097]

Very intuitive as always. Expecting some more intuitions on semi supervised learning,energy models. Thank you so much sir!!🙏

[@miguelramos3424]

it's the best video that I've seen about this topic, thanks.

[@saphirvolvianemfogo1717]

Amazing explanation. Thank you, it gives me a good starting point on DRL

[@eeerrrrzzz]

This video is a gem. Thank you.

[@mariogalindoq]

Luis: congratulations! Again a very good video, very well explained and with a beautiful presentation. Thank you.

[@alexandermedina4950]

Great starting point for RL! Thank you.

[@prakashselvakumar5867]

Very well explained! Thank you

[@beltusnkwawir2908]

I love the analogy of the discount factor with the dollar depreciation

[@piyaamarapalamure5927]

This is the best tutorial so far for the Q learning .. Thank you so much 😍😍

[@karlbooklover]

best explanation I've seen so far

[@charanbirdi]

Absolutely brilliant, specially Nural network and loss function explanation

[@zamin_graphy]

Fantastic explanation.

[@yo-sato]

EXcellent tutorial. I have recommended this tutorial to my students.

[@code_with_om]

After a day of searching I found a great explanation 😀😀 thank you so much

[@zeio-nara]

An excellent explanation, thank you

[@nishanthplays195]

No words sir! Finally found another great yt channel ✨

[@pellythirteen5654]

Fantastic ! Having watched many teachings on this subject , your explanation really made things clear. Now my fingers are itching to try it out and write some Delphi code. I will start with your grid-world first , but if that works I want to write a chess-engine. I have already written a chess-program using the alfa-beta algoritme and it will be fun to compare it with a neural-network based.

[@mutemoonshiner]

Huge thanks , for a nice and lucid content. specially for how to train the network, loss function and how to create datasets.

[@overgeared]

excellente como siempre! thank you from an MSc AI student working on DQNs.

[@pandharpurkar_]

Luis is master man of explaining complex things easily..!! thank you luis for such a great efforts

[@lebohangmbele283]

Wow. I can show this to my pre-school nephew and at the end of the video they will understand what RL is all about. Thanks.

[@mustafazuhair2830]

You have made my day, thank you!

[@scooby95219]

great explanation. thank you!

[@DrMukeshBangar]

great video. easy explanation! thank you.

[@flwi]

Wow - that was a very understandable explanation! Well done!

[@infinitamo]

You are a God-send. Thank you so much

[@shreyashnadage3459]

Finally here it is....been waiting for this for ages! Thanks Luis! Regards from India

[@francescserratosa3284]

Excellent video. Thank's a lot!!

[@ahmarhussain8720]

amazing explanation

[@KathySierraVideo]

Thank-you for this 🙏

[@ahmedoreby2856]

very good video with excellent elaboration for the equation thanks you very much for this

[@kr8432]

I am not stupid but AI still does not come easy to me. Sometimes I wonder, besides having more slots in the working memory, how a genius or simply more intelligent people think about this subject so that it comes more naturally to them. I feel like this video was a very good insight on how easy such a complicated topic can appear, if you just have a very good intuitive understanding for abstract concepts. Very nicely done!

[@AyaAya-fh2wx]

You are a genius!! Thank you!

[@adrianfiedler3520]

Incredible video, I love the animations!

[@bostonlife8589]

Fantastic explanation!

[@sricinu]

Excellent explaination

[@CrusadeVoyager]

Nice vid with gr8 explanation on RL.

[@randomdotint4285]

Oh my god. This was god level teaching. How I envy your real world students.

[@paul-andrejacques2488]

Just Fantastic. Thank you

[@Shaunmcdonogh-shaunsurfing]

Excellent video! Hoping for more on RL.

[@diwakerkumar5910]

Thanks 🙏

[@sergeipetrov5572]

Thank you so much! Very useful!

[@svein2330]

This video is brilliant!

[@roshanid6523]

Thanks for sharing

[@debobabai]

Excellent explanation. I dont know why this video has so low views. It deserves Billion views.

[@faisaldj]

I wish I had atleast my bachelors Math teacher like you but I would like to be like you for my students.

[@li-pingho1441]

this the best rl tutorial on internet

[@seraphiusNoctis]

Loved the video, quick question on the policy network section, because something still seems a little “disjointed” in the sense that the roles for both networks do not seem to be clear - I might be missing something… I don’t understand why we would use a decreasing/recursive “gain” function instead of just using the value network for the purpose of establishing values for the policy. Instead, doesn’t the value network already build in feedback mechanism that would be well suited to this?

[@ZirTaaah]

best vids on the subject for suuuuuuuure im mad that i didnt see it earlier nice broo

[@teetanrobotics5363]

Amazing. Could you please make a course on RL and Deep RL?

[@ishwargowda]

This is perfect!!!

[@AlexisKM100]

God damn it, this explanation was just straightforward, I loved it, it helped me to clarify many doubts I had, thanks :D Just how every explanation should be, concise and with practical examples.

[@antonioriz]

This is simply GREAT! I would love to follow more video on the issue of Reinforcement Learning. By the way I'm really enjoying your book Grokking Machine Learning, but I would like to know more on RL

[@banaiviktor6634]

Yes agree, no clear explanation on this topic apart from this video , thanks a lot, it is awesome ! :)

[@nothing21797]

Wunderbar!!!

[@bjornnorenjobb]

wow, extremely good video my friend! Big thanks!

[@ahmedshamz]

Thanks for these videos Luis. Are these from a course?

[@honghaiz]

Nice presentation

[@msantami]

Thanks, great video. Bought the book!

[@SerranoAcademy]

Great to hear, thank you! I hope you like it!

[@brok4498]

great job!

[@aliza207]

in love with your videos😍

[@AI_Financier]

Great video, a question: if i go for value network, do i still need the policy network too or vice versa? because by having only one of them, i can get to my target? thanks in advanced

[@william_8844]

WTF!!! Like I am half way through and I am already blown by the way you explain content. This has been the best video so far explaining RF..... Wow. New sub❤❤😅

[@Lukas-zl5zs]

amazing video, good work!

[@lucianoinso]

Truly great video and explanation! Loved that you went deep (haha) into the details of the neural network, thanks!

[@SerranoAcademy]

Thanks! Lol, I see what you did there! :D

[@Alpacastan21m]

Amazing.

[@paedrufernando2351]

cool...it took so long to drop this vid..I was earlier expecting RL videos from your site..but then I turned to Prof Oliver Siguad and completed RL there..Now I understand how DDPG works and internals of it..But I defintiley would want to see your take and perspective on this topic..So here I go again to watch this Video on RL ....

[@studgaming6160]

Finally good video on RL

[@AlexandriaLibraryGame]

I don't understand how to train the NN at 34:09, what are the features and what is the label?

[@RobertLugg]

You are one of the best teachers around. Thank you. What if the grid is different or the end goals change location? Do you need to start training over?

[@SerranoAcademy]

Thank you! Great question, If the environment changes in general you do have to start again. However, there may be cases in which you can piggy back from having learned the game in a simpler situation, so it depends.

[@ottodgs4031]

Very nice video! When you say that the label of the new dataset is a "big increase" or a "small decrease", what is that in practice? Just the gain?

[@elimelechschreiber6937]

Thank you. Question: In the last section you use the term 'gain' but actually use the 'value' function i believe. Shouldn't the gain be the the difference of the value (in your example, always positive 1 then)? The gained value associated with the given action?

[@rohitchan007]

This is by far the best explanation.

[@wilmarariza9020]

Excellent! Luis.

[@SerranoAcademy]

Gracias Wilmar!

[@outtaspacetime]

1234's vote up! thanks for this great overview

[@jaivratsingh9966]

@Luis Serrano - thanks for this. Excellent! At 30:15 shouldnt (4,0) be -2 and hence (4,1) be -3 and so on. A Query on policy train: If you freeze video at 28:52, and look at the table. I see it as random walk where you end up to a reward location, and kind of infer the value (subtracting 1) from next value point and come up with 3,2,..-1. Why would you say the one should decrease p(->) for 0,0 ? At 0,0 (or any chosen node on simulated path), the moves always increase the value (better value state), then change should never be "decrease"). Also while training the net you dont use "Change". Then why are we discussing "Change" at all ? Shouldn't it be simply the probability of actual step each step to be higher than rest as it points to path leading to a reward?

[@AyaAya-fh2wx]

Thanks!

[@SerranoAcademy]

Thank you so much for your contribution Aynur! And I'm so glad you like the video! :)

[@joselee5377]

i fucking move this video. oh my goodness... the level of satisfaction of understanding something that i struggled to grasp ;)

[@aditimahajan9278]

Kindly make video on Response Surface Methodology

[@kafaayari]

Great lecture Mr. Serrano, thx. But some parts are inconsistent and confusing. For example at 29:49, for the state (3,1) the best action is to move left and agent went left. However you try to decrease its probability during training as seen in the table. That doesn't make sense.