Reinforcement Learning Course - Full Machine Learning Tutorial

Reinforcement learning is an area of machine learning that involves taking right action to maximize reward in a particular situation. In this full tutorial course, you will get a solid foundation in reinforcement learning core topics.
The course covers Q learning, SARSA, double Q learning, deep Q learning, and policy gradient methods. These algorithms are employed in a number of environments from the open AI gym, including space invaders, breakout, and others. The deep learning portion uses Tensorflow and PyTorch.
The course begins with more modern algorithms, such as deep q learning and policy gradient methods, and demonstrates the power of reinforcement learning.
Then the course teaches some of the fundamental concepts that power all reinforcement learning algorithms. These are illustrated by coding up some algorithms that predate deep learning, but are still foundational to the cutting edge. These are studied in some of the more traditional environments from the OpenAI gym, like the cart pole problem.
💻Code: github.com/philtabor/KZbin-...
⭐️ Course Contents ⭐️
⌨️ (00:00:00) Intro
⌨️ (00:01:30) Intro to Deep Q Learning
⌨️ (00:08:56) How to Code Deep Q Learning in Tensorflow
⌨️ (00:52:03) Deep Q Learning with Pytorch Part 1: The Q Network
⌨️ (01:06:21) Deep Q Learning with Pytorch part 2: Coding the Agent
⌨️ (01:28:54) Deep Q Learning with Pytorch part
⌨️ (01:46:39) Intro to Policy Gradients 3: Coding the main loop
⌨️ (01:55:01) How to Beat Lunar Lander with Policy Gradients
⌨️ (02:21:32) How to Beat Space Invaders with Policy Gradients
⌨️ (02:34:41) How to Create Your Own Reinforcement Learning Environment Part 1
⌨️ (02:55:39) How to Create Your Own Reinforcement Learning Environment Part 2
⌨️ (03:08:20) Fundamentals of Reinforcement Learning
⌨️ (03:17:09) Markov Decision Processes
⌨️ (03:23:02) The Explore Exploit Dilemma
⌨️ (03:29:19) Reinforcement Learning in the Open AI Gym: SARSA
⌨️ (03:39:56) Reinforcement Learning in the Open AI Gym: Double Q Learning
⌨️ (03:54:07) Conclusion
Course from Machine Learning with Phil. Check out his KZbin channel: / @machinelearningwithphil
--
Learn to code for free and get a developer job: www.freecodecamp.org
Read hundreds of articles on programming: medium.freecodecamp.org

[@MachineLearningwithPhil]

Here are some time stamps folks! Intro 00:00:00 Intro to Deep Q Learning 00:01:30 How to Code Deep Q Learning in Tensorflow 00:08:56 Deep Q Learning with Pytorch Part 1: The Q Network 00:52:03 Deep Q Learning with Pytorch part 2: Coding the Agent 01:06:21 Deep Q Learning with Pytorch part 3: Coding the main loop 01:28:54 Intro to Policy Gradients 01:46:39 How to Beat Lunar Lander with Policy Gradients 01:55:01 How to Beat Space Invaders with Policy Gradients 02:21:32 How to Create Your Own Reinforcement Learning Environment Part 1 02:34:41 How to Create Your Own Reinforcement Learning Environment Part 2 02:55:39 Fundamentals of Reinforcement Learning 03:08:20 Markov Decision Processes 03:17:09 The Explore Exploit Dilemma 03:23:02 Reinforcement Learning in the Open AI Gym: SARSA 03:29:19 Reinforcement Learning in the Open AI Gym: Double Q Learning 03:39:56 Conclusion 03:54:07

[@firelordzaki1600]

I litetally love this guy i subscribed right after i watched this. Best guy to watch while in college.

[@sushiboy4595]

Ooooo

[@Rayskidschannel3950]

Ppppp

[@user-zn5ir2ll5h]

Зю

[@666lucanator]

😢😢

[@InturnetHaetMachine]

This is a great video if you already understand the topic, understand the code and just want a guy saying what he's typing out aloud, kinda explaining bits and pieces here and there.

[@henrypowell3496]

yah and people in the comment section be like: thank you, what a great tutorial for free, great explanation, while they get nothing and just being smart in typing a comment

[@ramtinnazeryan]

He said at the beginning no need to know about this and that. 14 minutes into the video he is typing the line 123. Honestly why didn't he copy and paste it? :))))

[@surajbherwani5488]

Worst video

[@elasdebastos235]

yeah, i hae the same feeling, i didn t undesrtand a crap @@ramtinnazeryan

[@Short_Clips44]

Its a "course" - so it is what it is :D . Take it as an overview to the topic, watch him code to see whats up, but not code urself here.

[@amitbuch]

Extremely well explained. Kudos to the tutor. Simple explanation to workign code in less than an hour is amazing and yet very clearly laid out. Thanks for this upload.

[@SmartassEyebrows]

One minor correction for those watching at 1:19:12 and trying to follow along (like myself): on line 77 after the "else", the "memStart = int(np.random.choice(range(self.memCntr - batch_size - 1)))" should actually be "memStart = int(np.random.choice(range(self.memSize - batch_size - 1)))". The self.memSize is needed here instead of self.memCntr because at this point the self.memory list is now full (the "else" branch), but the self.memCntr value is continuing to grow and is now larger than the max self.memory size. That leads to line 78 giving miniBatch an empty list, [ ], leading to memory being an empty array, because memStart will be a larger value than the self.memory list length, while then being used as the index for grabbing the miniBatch from that same self.memory list -- no good. Ultimately that leads to an exception: "too many indices for array" on line 81 (since we are trying to forward an empty 1D numpy array and call 2D indices that don't exist). With self.memSize for line 77, that no longer happens, and memStart stays within the bounds of the self.memory length/size. With that, everything works, and you can watch the agent play :)

[@petermcfarland9236]

This video has helped me find clues that ultimately helped me to understand machine learning. Thanks blue Steve.

[@Johniakson]

This is AWESOME! 👍👍👍 Thank you for this!

[@Bill0102]

I'm immersed in this. I read a book with a similar theme, and I was completely immersed. "The Art of Saying No: Mastering Boundaries for a Fulfilling Life" by Samuel Dawn

[@user-hk2jx5mj6z]

Thank you profoundly for sharing your knowledge!

[@barbellbender4232]

The length of the flatten outputlayer can actually be calculated from first conv layer tracing the data through the network. Just use the function: ((dimension length - kernal size for the dimension + 2*padding)/stride)+1 = output length for the dimension do this for each dimension for each conv layer and multiply by number of outputs in the end to find the length of the flat dimension as such: 1st conv layer: ((185 - 8 + 2*1)/4) + 1 = 44 (acutally 44.75 but you always round down, since there are no 0.75 pixels) ((95 - 8 + 2*1)/ 4) + 1 = 22 (rounded down from 22.25) 2nd conv: ((44 - 4 + 2*0)/2) + 1 = 21 ((22 - 4 + 2*0)/2) + 1 = 10 3rd conv: ((21 - 3 + 2*0)/1) + 1 = 19 ((10 - 3 + 2*0)/1) + 1 = 8 this means the 3rd layer outputs 128 frames with each having dimensions 19*8 and therefore if you wanted to flatten them into one you would get one dimension with 128*19*8 vectors. Just neat little trick for those who want it

[@lsagar]

I'm a beginner and the Background loop seems more interesting than what he's talking. I hope I understand what he's saying someday

[@musiquelover9268]

The same thing

[@aaryasankhe5973]

Anyone interested in learning the terminologies of what he is talking about should go check out the video lectures Stanford did on MDPs(Markov decisions processes and RL), they're about each an hour long and do go in depth behind the math for a lot of this stuff. Cheers!!!

[@AnthonyAnalog]

Underrated. Thank you.

[@evanstark4823]

man i just fell asleep on youtube and now i’ve been watching this for 2 hours 19 minutes

[@danilkos7411]

I sleep to this all night. KZbin knows I’m using it as white noise at bedtime and offers sleepy stuff like this 😬

[@vsp540]

⭐️ Course Contents ⭐️ ⌨️ (00:00:00) Intro ⌨️ (00:01:30) Intro to Deep Q Learning ⌨️ (00:08:56) How to Code Deep Q Learning in Tensorflow ⌨️ (00:52:03) Deep Q Learning with Pytorch Part 1: The Q Network ⌨️ (01:06:21) Deep Q Learning with Pytorch part 2: Coding the Agent ⌨️ (01:28:54) Deep Q Learning with Pytorch part ⌨️ (01:46:39) Intro to Policy Gradients 3: Coding the main loop ⌨️ (01:55:01) How to Beat Lunar Lander with Policy Gradients ⌨️ (02:21:32) How to Beat Space Invaders with Policy Gradients ⌨️ (02:34:41) How to Create Your Own Reinforcement Learning Environment Part 1 ⌨️ (02:55:39) How to Create Your Own Reinforcement Learning Environment Part 2 ⌨️ (03:08:20) Fundamentals of Reinforcement Learning ⌨️ (03:17:09) Markov Decision Processes ⌨️ (03:23:02) The Explore Exploit Dilemma ⌨️ (03:29:19) Reinforcement Learning in the Open AI Gym: SARSA ⌨️ (03:39:56) Reinforcement Learning in the Open AI Gym: Double Q Learning ⌨️ (03:54:07) Conclusion

[@claude.detchambila]

This is one of the best free RL videos available. Please make some more.

[@yushuowang7820]

Indeed

[@dummypg6129]

This is the only issue that i often see on any "basic tutorial" videos. There's no explaination on the terminologies during the intros.

[@constantinetikh]

The terminology and concepts are explained in the two blocks starting 03:08:20

[@sunmustbedestroyed]

Heads up: this one isn't for beginners.

[@opkp]

@Black Hole lel , got him good

[@fabrizioantonazzo3113]

grazie mille ho iniziato da poco a seguire il tuo corso ben fatto

[@firelordzaki1600]

Wow fcc really stepping up

[@imtiazahmad7826]

thanks, so helpful video

[@kerryd2060]

Thank you. Great job in explaining the content.

[@mehershrishtinigam5449]

thanks for taking time out your busy day to teach us RL fellow martian!

[@nicklansbury3166]

You had me at "Atari"!!!

[@natashasamuel9346]

Great class. Keep up the good work. Thank You, Natasha Samuel

[@henrypowell3496]

so you understood everything in the tutorial?

[@charlespeck5493]

easy to understand, much pleasure

[@remoncomputer]

Good awsome view and awsome channel, good work

[@ShapeShifter369]

Thank You a lot !!!!!!!!!!!!!!!!!!!!

[@that_endo]

2nd time waking up to this guy

[@CaribouDataScience]

Reinforced learning, is like when you have to write 1000 times "I will not talk in class"?

[@user-ey5ej1pd3i]

I saw this video I'll see this video again Thank you

[@VaIonty]

Bro I fell asleep watching a different video and woke up this morning to this video playing and I was an hour deep in it 😭

[@finarwa3602]

I really appreciate the work you are doing . Could you mention which the development tool you are using for the whole series?

[@Beny123]

Vscode

[@zoerycroft4300]

for anyone watching, inheriting from object is implied, you haven't needed to type that since even the oldest versions of python 3, save yourself some time ;) `class foo(object):` is exactly the same as `class foo:` the reason he types it here is probably for intercompatability between py2 and py3, but not even a year after this was uploaded py2 went end of life, so you shouldn't need to worry about that anymore :))

[@imrielseraphis]

I am currently creating an agent-based model that will generate x numver of agents. Each agent has a step function. I would LOVE to incorporate this reinforced learning method into the model. How would you adjust it from taking a visual frame like from a game to using only the global environment variables? Is it simply as easy as swapping out one for the other?

[@freetellurian]

✍️👍 Больше спасибо

[@sebtaciak]

keep going with Sutton but please move to DLR too!

[@user-or7ji5hv8y]

what can you recommend to watch of your other youtube videos before watching this one?

[@matthimuller]

python guides and some linear algebra to understand what is happening

[@kae4881]

Check out his channel his original one.

[@0GRANATE0]

When I wanted to implement a multi agent reinforcement learning envirenment for a soccer game (multiple agents, separately trained, with separate models) what algorithms would I use for a continious envirenment (not a grid world) where the players can walk/run/shot everywhere? - DQN Reinforcement Learning?

[@lalomega7966]

This is awesome..

[@haneulkim4902]

Amazing course, thanks alot Phil! One question, you were comparing policy gradient methods with reinforcement learning however after few searches it seems like policy gradient method is an algorithm within RL. Could you clarify?

[@pororo3653]

ж0

[@jansprlak4157]

Beta tester od roku 2016 oceňujem,,💖💖💝

[@KushChoudhary]

Let me tell you It is so good :)

[@user-cv5kq9hy7x]

អរគុណ

[@vasanthrs1140]

Can you please make a full tutorial in flutter? Thanks, I've been watching your tutorial for a long time. Excellent work.

[@freecodecamp]

We're posting a full flutter tutorial next week. :)

[@rbalachandran007]

Thanks

[@vasanthrs1140]

@@freecodecamp Thank you so much, I'm looking forward for that tutorial.

[@angrycoder]

Clearly the video is only for people who have already researched about RL. Not for beginners at all!

[@billchenxi]

Do you know why his kernel size is even number? Normally we use odd number for easy calculation.

[@underlecht]

why is that a difference?

[@billchenxi]

​@@underlecht Because a non-symmetric kernel (even number) yields a non-symmetric filter response. In the example above, this non-symmetry leads to a shift of the blurred image by half a pixel.

[@carlosalbertoleyva6460]

How exactly do I run these python programs? I'm using Atom IDE

[@kae4881]

You can run via the terminal(linux and macOS)/command prompt(windows) or you can download the Run Script addon in Atom(that's what i use)

[@devsinharoy8811]

when you do def(build network) wouldnt it be easier just to use keras

[@mafiamustafa]

The stacking frame reshaping code is wrong, it messes up the entire array. I am trying to debug it.

[@mafiamustafa]

Hello Phil, I think there is another mistake in the code, in the learn function it should be reward_batch + gamma*np.max(Q_next, axis=1)*(1-terminal_batch) instead of just terminal_batch. Since we are passing int(done) as a stored observation. Therefore for done=False, int(done)=0 and vice versa. And if episode does not end that is done equals False then we need to add the next Q_value otherwise we only add reward. What do you think? Am I correct?

[@kae4881]

yeah i do think so. but i encourage you to try it both ways. sometime it trains the agent to finish the episode as fast as possible, that's not what we want.

[@krishnendubose2620]

Could you post the link to the lectures here?

[@boxsewatiodoxsewstio1355]

nice

[@honkmangoose6567]

Steve from blue's clues wasn't joking when he said he was going to college

[@umairnehri8459]

💯💯💯

[@timraiser4811]

I get errors when i run the pip install commands for box2d-py, tensorflow-gpu and torch. Is there something i'm missing?

[@luci0rullz]

Phil, you are a fucking boss :>

[@mmanuel6874]

YEET!

[@YMARihab]

Can I really get a job after learning stuff like machine learning and etc online? I did BSc in EEE but as I was not interested I did no do well and did not learn much, only thing I somewhat enjoyed was the C and C++ courses and digital logic design and verilog/vhdl courses . Later I did python course from MITx and enjoyed solving all the exercises. Pls give me hope.

[@YMARihab]

wild doggo appears. wild doggo regrets wasting his time.

[@abhishekshah11]

How much linear algebra and statistics should I know for this track?

[@MachineLearningwithPhil]

Not much is needed. I explain it all as we go along.

[@kae4881]

not much alegbra and statistics i can say, but more about the basic reinforcement learning terms

[@BharatwaajShankar]

Why are we onehot encoding the actions?

[@user-or7ji5hv8y]

are you using tensor flow 2.0?

[@kae4881]

No, it's tensorflow 1.4.10. If you want to look at tensorflow 2.0 or keras, go check out Phil's KZbin channel

[@hninnwe6015]

Please use some illustrations so that can understand more easily

[@gorimor]

12 minutes in and scratching my head. Hi yes I'm lost

[@cactus_duke1639]

What program do you use to run your code?

[@kae4881]

It's the linux distribution native terminal, not sure about the exact distribution. Maybe ubuntu

[@fuckooo]

Can you please provide an environment.yml file for this?

[@FundamentalistLohar]

raise error.UnregisteredEnv("No registered env with id: {}".format(id)) gym.error.UnregisteredEnv: No registered env with id: Breakout-v0

[@jordanolson11]

so it seems I can't even install this or get started, I'm on windows 10 and I have python 3.7 working, I installed pip, gym, but when I got to tensorflow it's telling me I have "no matching distribution found for tensorflow-gpu", some people suggested that it's because I got the latest version of py, others suggest to use anaconda, what should I do?

[@jordanolson11]

so after a few hours I was able to figure out that anaconda and miniconda are the same thing and going through the github repo for box2d-py it just says to use anaconda, but I'd rather follow how you're doing this vs using a stupid VM with py on it, seems really ridiculous that installing a few things you show in your command line running so fluidly, like what am I missing? I reinstalled python 4 times and made sure my env variables were working correctly, each with a different python version each time and they all gave the same error with pip installing box2d-py and tensorflow-gpu, including 3.6 which is the version you're on. What shell are you using? I'm just using command prompt and I wonder if that's the problem.

[@jordanolson11]

lol after buckling and trying anaconda tensorflow fails, but everything else runs properly. Really fun tutorial.

[@MachineLearningwithPhil]

@@jordanolson11 Sorry, just seeing this now. To the best of my knowledge, Tensorflow only works with python 3.6. You can just install 3.6 in parallel with other versions, without nuking the whole install, I believe.

[@1ycx]

Use Google colaboratory

[@1ycx]

Use Google Colaboratory

[@joebarco5674]

Is this something someone can learn without previous experience?

[@MachineLearningwithPhil]

You sure can, I'll cover the basics as we go along.

[@kae4881]

Not really, i suggest you read about reinforcement learning basics before you watch this.

[@tariqshah2767]

Is there anyone who knows which version of TensorFlow is used in this video?

[@lawrence4876]

Looks like TF1

[@emanuelepapucci59]

my utils package do not have any plotLearning func...

[@sihabportal8623]

machine learning is really trending matter

[@ShahFahad-hj1ps]

Is there any course (Reinforcement Learning code based) for beginners ?

[@thomascarstens2729]

Coursera has a really cool 4-course series "Reinforcement Learning Specialization"

[@0GRANATE0]

How comes that I am not capable to UNDERSTAND this stuff? Is it OK to accept that I am dumb? Serious question... (I am a software engineer, did computer science, but had always problem with math, but did always the best mark when I had to deliver a project..)

[@renaton3992]

Yep... He is theaching for who already knows.....

[@isalihkilic]

Information is so good but the background is grabbing the attention away from the information.

[@user-uy6rg9fn6y]

새해 첫날 듣는 노래가 그 해 운이 달렸다는거 몰라서 2023년 첫곡을 니소식으로 들었다가 이틀만에 사귀던 남친이랑 헤어졌었는데 2024년은 내가제일잘나가 듣고 완전 끝내주게 살아보려고! 1년내내 연애도 끝났고 직장도 안좋게되서 속상했는데 겨우 [빠]칭코닷[컴]하면서 잊고지내고 우울한거도 잊으면서 지냈는데 2024년에는 더 잘될 수 있을것같아! 이제 돈도 잘 벌고 내가 더 행복한 사람되야지!

[@mindflayer99]

ok guys where is the code available

[@mafiamustafa]

self.q = tf.reduce_sum(tf.multiply(self.Q_values, self.actions)) Why are you doing this? I fail to understand the meaning of this line?? Thank you in advance :)

[@mt345678]

he made a mistake.... the github code is corrected.

[@mafiamustafa]

@@mt345678 I see so many other tutorials which do the same thing, and I fail to understand why.

[@robihamdani4017]

i feel uncomfortable with his hands

[@niklasdamm6900]

25.10.22 22:30

[@MuhammadAhmed-vi2xt]

please make complete course on WordPress 5.2

[@AbhishekJain-zu1uf]

i can hardly hear anything

[@subhadeepbanerjee8408]

Add something with udemy

[@joeljoseph26]

When you mentioned a course on Full Machine Learning Tutorial - Reinforcement Learning and there is no proper order to it. I don't recommend watching this thing. There are tons of materials that are a lot simpler than this.

[@motbus3]

Is this tf1 or tf2?

[@kae4881]

tf1

[@saeedalafifi0]

Thanks for your information Can you help me to set an environment to trade forex and how to set it with metatrader 4 Please help me

[@tejasjani2544]

I'm second

[@tejasjani2544]

No I'm First

[@shirish3008]

Congratulations you both got prize money of 5k dollars

[@rbalachandran007]

Lol @shirish

[@tejasjani2544]

@@shirish3008 no thanks.

[@shirish3008]

@@tejasjani2544 we will give you even you don't want...

[@ayushshukla1438]

I have that T-Shirt

[@marcosdiego7965]

[@thegamechanger7157]

Just need to get this over. After I find her, That's it. I will share the technique. She must code it. Then its done. I will go to desert. If your thinking nuclear reactor? In the middle of the congested city in the world with rascal scientist like me . It will result to catastrophe.

[@ankitkpandey369]

Not useful. Creator should at least run a code once before wrapping up a section. Small syntax/logical errors are fine but when there is error in code itself (eg: new_state_batch in first section) and there isn't much explanation about it, whole hour spent in coding goes for nothing.

[@freetellurian]

🤝🌈👽🖖

[@shahinashaikh8189]

okay this is not for beginners at all. wow.

[@morzariadeep]

this is not for beginners !!!

[@anmolgupta3076]

First

[@ashok.rathore22]

1st viewer😎😎

[@KushChoudhary]

Sorry to tell but it was me

[@ashok.rathore22]

@@KushChoudhary i was second dude sorry👍

[@KushChoudhary]

@@ashok.rathore22 😅 It's great bro

[@entwaze]

LMAO, im just watching him code at this point pretending i understand

[@henrypowell3496]

lol yah

[@TJ-hs1qm]

Classic example of a great coder who can't teach :)

[@moulikgupta6979]

terrible tutorial, he's using outdated methods of tensorflow library for literally no reason

[@alexeysmirnov5481]

1.4 hours of coding without any execution. this is not a good way of learning

[@SachinYadav-vl2wi]

dislike

[@wyphonema4024]

this is a terrible tutorial, he is just typing. I think it would be better if he typed all the code already and save the time to explain the codes deeply

[@Ak-qq9it]

Could you do a better job? No