Python + PyTorch + Pygame Reinforcement Learning - Train an AI to Play Snake

In this Python Reinforcement Learning course you will learn how to teach an AI to play Snake! We build everything from scratch using Pygame and PyTorch.
💻 Code: github.com/python-engineer/sn...
✏️ Course developed by Python Engineer. Check out his KZbin channel: / @patloeber
🎨 Art by Rachel: rachel.likespizza.com/
⭐️ Course Contents ⭐️
⌨️ (0:00:00) Part 1: Basics of Reinforcement Learning and Deep Q Learning
⌨️ (0:17:22) Part 2: Setup environment and implement snake game
⌨️ (0:40:07) Part 3: Implement agent to control game
⌨️ (1:10:59) Part 4: Create and train neural network
🎉 Thanks to our Champion and Sponsor supporters:
👾 Raymond Odero
👾 Agustín Kussrow
👾 aldo ferretti
👾 Otis Morgan
👾 DeezMaster
--
Learn to code for free and get a developer job: www.freecodecamp.org
Read hundreds of articles on programming: freecodecamp.org/news

[@murtazabadshah8747]

Everyone's commenting that its an excellent video but IMO this tutorial is awe-full! The instructor does not explain the process, hes all over the place going back and forth and just rushes through the concepts. If you want to blindly follow an online tutorial watch this video, if you want to actually learn the concept I would look somewhere else....

[@30DaysMonkMode-ft1kf]

Exactly.

[@ShtBall5]

any video recommendations ?

[@30DaysMonkMode-ft1kf]

@@ShtBall5 You can check this video: "Learn Pytorch for deep learning in a day. Literally."

[@murtazabadshah8747]

@@ShtBall5 Yeah I ended up watching this video that helped me a lot to understand the basic concept of the Q-learning algorithm and the basic concept of how the state and agent work kzbin.info/www/bejne/pqLbqneLaMunmdU&ab_channel=Bits%26Neurons

[@HRDmonk]

Thank you so much for this tutorial. I have always wanted to introduce snake with ML. and now looking forward to learning more about pytorch

[@edwintjoa6099]

Thanks for the awesome video! Really fun to see agent improving during the games.

[@mihailmihaylov988]

A nice video. My only critique is that the presenter kept writing code for almost an hour without running it. That's not giving a good example.

[@user-de3oj1xw8u]

Need to be patient 😊

[@DroopTheSnoot]

Debugging code should always be done every function or so

[@pedroklain9375]

Bro is just too good at coding, he runs it in his head 😂

[@mihailmihaylov988]

@@pedroklain9375 It's not about what he can do or cannot do. It's about what example he sets. After all, this is an instructional video.

[@electricspeedruns6121]

friend ai and programming code is a bit different @@mihailmihaylov988

[@willchantemduang5871]

I’ve always wanted to do this, thanks alot for the tutorial

[@Oncopoda]

Excited to try this!

[@adrian46647]

Awesome, so hard to find that type of explanation of dqn. All clear, great balance between theory and coding part for beginners in rl.

[@nesimtunc]

Always very high quality videos when I just need it :) Thanks a lot! Looking forward to finish the video 😎

[@veronicasalazarpereira9933]

I did it!!! Thanks for showing this.

[@khalidelgazzar]

Watched the first 4 mins and the game and the learning process is fantastic! 🎉 Will go on it with the rest hopefully soon.

[@khalidelgazzar]

Part 2 - 17:24

[@khalidelgazzar]

21:30

[@techarchsefa]

That is so smooth bro, thanks

[@dabyttamusic7308]

Super great video! I run it and after more than 2000 games it seems that the average score reached a plateau of 30. It Avoids boundaries very well. But it hit itself when the tail is too long. But this movement should be predicted before.

[@user-fb8tr3kb8j]

To avoid the self collision we have to add snake parts positions to the game state somehow, in the current model it doesn't know where's the body so it will never learn how to avoid it

[@user-de3oj1xw8u]

A video as valuable as a playbook👍🏻👍🏻👍🏻

[@Hiyori___]

Very interesting tutorial. I was familiar with Snake code implementations, Tensorflow basics and RL principles and I could follow along. For some reason the plot doesn't appear, and I only get a small blank window where the plot should be. Probably some bug while typing the code down. Other than that everything works for me. Thank you very much!

[@lostsoul4393]

just incase anyone else has this problem, you need at add plt.pause(.1) to the bottom of the plot function

[@stillrunning9841]

Hey there, same happened for me! Did you find the error? I'm still trying to find it :)

[@lillymoreau7721]

@@stillrunning9841 this code worked for me: import matplotlib.pyplot as plt plt.ion() def plot(scores, mean_scores): plt.clf() # Clear the current figure plt.title('Training...') plt.xlabel('Number of Games') plt.ylabel('Score') plt.plot(scores, label='Scores') plt.plot(mean_scores, label='Mean Scores') plt.ylim(ymin=0) if scores: plt.text(len(scores)-1, scores[-1], str(scores[-1])) if mean_scores: plt.text(len(mean_scores)-1, mean_scores[-1], str(mean_scores[-1])) plt.legend() plt.draw() plt.pause(0.1)

[@radiatian4908]

Super late reply, but i had this issue and it turns out the video is missing two lines of code. In his files he also has plt.show(block=False) plt.pause(.1) as the last two lines. Hope this works for u too

[@sergiomollo]

Thanks for this video, I have solved all my doubts

[@devnull711]

Incredible work, thank you Patrick! PS: It is very funny to spot the typos/bugs before you do :)

[@dollarfree531]

Very cool video! Highly useful!

[@pactube8833]

Thanks for freeCodeCamp for making this possible.

[@kevas777]

nice algo, but how to solve a problem with self destroy, when the closest cell to move is "inside body circle". I think a state must be all field with each body part,head, food etc, but its endless states and all of them unique and it will never learns or may be i wrong?

[@vikramganesan]

how to perform UCB, optimal initial values and dynamic programming approaches in this model?

[@Fr4nk4000]

Following this since I wanna make a pygame project of mine that I poured a lot of time into and have no idea what to make the game about play itself. Wish me luck.

[@jaibhagat7441]

have you made something

[@aliengineroglu8875]

Thank you for your great work. I couldn’t understand the equation you created in the video about the simplified Bellman equation, Q_new = reward + gamma * torch.max(model(next_state)). In this equation, model(nextstate) gives us a probabilistic action prediction. I couldn’t understand why we added one of the action probabilities to the reward. This is totaly different than the Bellman Equation. I would be very happy if someone could explain how the original Bellman equation was simplified in this way. Thanks in advance to everyone.

[@alanlam6302]

Same here. Appreciate if someone can provide a reference to this implementation

[@wangking7427]

The Liner_QNet model gives three actions' Q value vector . It's not the probablistic of each action but the QValue of each action. We take the action which has the highest QValue each step. The QValue in this tourtorial is the reward of the action.

[@markkiryaflex]

I think this project is awesome

[@simpleepic]

Great tutorial

[@pyshine_official]

Nice effort and seems like the convergence is achieved upto some extent!

[@manojkothwal3586]

Brilliant 🔥🔥🔥🔥🔥🔥

[@python0tutorial100]

Thanks for the video it was great but I have only one question if u can help after I run the agent and after a few try like 3rd game the game stuck and in the console suddenly start to count the number of game without the game really play

[@codingfan]

Really interesting course !

[@gattorwichar3984]

This instructor is the best

[@Coding_Destini]

What to install before creating an environment ,iam confused

[@filoautomata]

What about using GA to train the NN itself ? Will be a very interesting comparison no?

[@goldenmarketing3179]

Does this tutorial teach how to write "inference" by myself, not using library?

[@lukasgamedev]

Hello! Is there a way to save the state of the neural model? So I can load later a trained enemy AI, ready for being the player opponent? Thank you for the video!

[@TanmayBhatgare]

i think u can you torch.save(model.state_dict(), 'rl_model.pth') to save model and model = YourModel() model.load_state_dict(torch.load('rl_model.pth')) to load it. Hope this helps.

[@vinayakbhandage8319]

You never fail to amaze us✌

[@vespervenom2343]

I copied it code for code and it doesn’t work. It gives me no errors but just runs and then ends with no result🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️

[@soraaoixxthebluesky]

i just want to know after training, how can we load back a trained model?

[@ismailmatrix1]

What's your VSCode theme? It looks gooood

[@stefanb4340]

This might be a stupid question, but how would one go about saving this trained model and accessing it for further training?

[@brayanfernandes371]

you can save the weights

[@sanghoututorial3878]

@@brayanfernandes371 how

[@segovemoc4776]

@Biglyp I actually wanted to ask a similar question. I save the weights and load them through: self.model.load_state_dict(torch.load(model_saved_path)) and it works, but the problem is that the snake is then underperforming. It learns then much quicker than before, but still it is significantly worse than during training.

[@segovemoc4776]

never mind. I found answer in the later comments - it is necessary to adjust the epsilon value to something very small after loading the model

[@gmancz]

@@segovemoc4776 It works, but you need to set the Agent.n_of_games to something big value (eg.: 280). self.model.n_of_games = 280 As you remember, there is exploration and exploitation in the get_action() function. We are updating the epsilon value using the self.n_of_games variable and then get a random number between 0 and 200. This line makes your snake so bad.

[@cesarortegonavacerrada9065]

Very good video. I need help with the homework. How can you avoid the loops?

[@Miyuru_]

same problem it's getting stuck in the same loop

[@hcc3904]

@@Miyuru_ overfitting... just add another condition point like if u dont eat apple in 10 seconds, minus -10 point

[@Radu]

Really nice!

[@sidheshwartiwari9834]

Look who is here, none other than mighty Radu sir🙏🏻

[@Radu]

@@sidheshwartiwari9834 haha :-) funny!

[@carolinab9945]

Is it reinforcement learning even if you give some instructions about the movements?

[@ckelsel8198]

thank you very much.

[@GenkiKuri]

Awesome!!

[@angelosorte5464]

Is there a limit to when it stops learning? I mean the quality of the intelligence will stay the same at some point, or will it improve even more and more after those 12 minutes? Thanks.

[@damienlmoore]

At 34 mins, it breaks out of the game and starts gobbling up your filesystem. At 762, it breaks out of the machine and comes for you. 😅

[@user-zo4cx8yi3g]

thanks, intresting!

[@luthermillamuculadosreisec3844]

Thank you

[@yavarjn2055]

Lot of the code needs more explanation. There is a disconnect from theory and implementation. We copy all the parameters to this and to that without understanding why. The memory part is not well explained.

[@30DaysMonkMode-ft1kf]

Yeah. He isn't explaining, just repeating what he is writing.

[@smkzachatac]

So, my array for state is apparently returning NoneType. Anyone know a fix?

[@fitybux4664]

38:13 "It has to be [0,0,1]"

[@sunyog]

What are the prerequisites of the video ?!

[@cadewzan]

Theres no need of waiting a lot of time to train, on the game script you can just change the varaible from 30 to 1000 so snake goes much more fast and trains intself on less time.

[@MyArtsProduction]

He actually mentions that 1:38:00

[@spyrav]

thanks!

[@eugenmaljas7237]

I want to modify the program: how do I make 4 outputs? I would like to integrate the snake length.

[@arnavgupta8376]

I think you can do so if you change the line 20 ie 'self.model = Linear_QNet(11, 1024, 3)' change the 3 to 4. and in line 90 'final_move = [0,0,0]' add another zero. Maybe there are some more things you need to do.

[@kabrailchamoun8974]

Could you bring us a crash course of blender (3d modelling program)?

[@slamsandwich19]

Python has nothing to do with 3d modeling though

[@muhammadnaufil5237]

@@slamsandwich19 it is. Blender has a python library called pyblender. I am looking forward to a good blender tutorial and how to integrate this with python

[@joe_hoeller_chicago]

Cool vid 😊

[@gukeskl3671]

How can I get this terminal? :o

[@S3R43o3]

So far so good... but how about load the old 'brain' if u exit the game and want resume later ? Oo iam bite confused tryd with state if os.path.exists('model/model.pth'): self.model.load_state_dict(torch.load('model/model.pth')) self.model.eval() in the Agent init. but that dosent work

[@vikramganesan]

did u find out a way to load the previously trained model?

[@S3R43o3]

@@vikramganesan nope sorry

[@RORoMiguel]

it seems like we need to remove the epsilon to load the model, otherwise it will keep doing random moves

[@jeremiebouchard6950]

@@RORoMiguel You're right, it works for me :) thanks! first game : 33 score

[@polllloigyv]

i think it doesn't work because you you wrote / instead of this\

[@canerunafraid9491]

The snake moves smoothly, but when it hits the first wall, the interface closes idk :/

[@Lenslyfe]

It's not meant to wrap. That's the correct behaviour.

[@sakshirokade6321]

WHICH ALGORITHM IS USED IN THIS

[@NationalistVietnamese]

I use your code and train it with speed 60000 (just modify the "game.py" file)

[@xccds]

very great course! thanks! A little question is , in class QTrainer maybe target = pred.clone().detach() ?

[@superfact8751]

hi bro kaha se ho

[@usernumber334]

true

[@pranjalshukla8096]

18:58 for me it is not showing as (pygame_env) ... .... i am a windows user (edit) it worked for I had to open a CMD outside of VS CODE and then follow the steps of conda pygame_env now I am getting "(pygame_env) C:\Users\........."

[@bobjeff6779]

how did you get it to say pygame_env

[@pranjalshukla8096]

@@bobjeff6779 when you are creating a venv (virtual environment) you can name it so to make venv with a different name you can do that by python -m venv myproject

[@feliuswyvern7189]

so i had the program run over 2000 games and it couldn't get past the 80s mark. how would i get it to improve past that?

[@taraskhan4755]

there is no way to improve it unless you got coding knowledge

[@mtijohn9274]

yes

[@philtoa334]

Crazy.

[@deepakpaira0123]

how to save the model...? where to know?

[@baslifico]

I think handing the fundamental flaw in the design to others as "homework" is a bit stiff... You're asking a neural network to solve an NP-Hard problem.

[@bipinmaharjan4090]

Found new hobby

[@googlyeyes3237]

Can we speed up the game soo fast so ai can play faster and learn faster. Basically it can finish 100 games in a minute or two.

[@sidheshwartiwari9834]

Absolutely, you just have to increase the render rate. Good question though.

[@tlefevre2918]

it does that when it is drawing on the chart, I speed it up by making it only update the chart every 100 games.@@guillaumelotis530

[@keethesh2270]

Just change the tick speed to make it a higher one. It is at the top of the script and is called "SPEED"

[@dexterbarrot5226]

What text editor

[@planetoday9169]

can someone help me how to create a multiple snakes?

[@fizipcfx]

why not just give the ai full board information as the state

[@Miyuru_]

someone tell me how to save trained data?

[@marcbringas1006]

where is he typing the commands? is this a CMD?

[@science_horizonts]

Do you mean the source code of the program or the command to run the program If to start, then yes, this is the command line(CMD), but you need to go through it to the folder where you have the project, CMD is somewhat similar to the explorer that goes through the folders

[@__________________________6910]

omg patrick wow

[@patloeber]

yeah :)

[@joekanaan2548]

Hi I have a question. I followed along and wrote everything. When I run it in the terminal it works but the plot doesn't. It gives me an empty minimized white screen. If someone else experienced this please help.

[@AlexandruTunschi1]

There are 2 lines missing: plt.show(block=False) plt.pause(.1)

[@RANDOM_DUD-qj3jd]

Where do these go?@@AlexandruTunschi1

[@nescreation]

Wow

[@godswillhycinth9809]

I'm a JavaScript Developer that knows nothing about python, but I must say I'm jelous 😭😭. This is really cool

[@Ragnarok540]

Python is way easier than JavaScript. Give it a try.

[@Loug522]

Yeah, if you already know other languages(especially OOP) then Python shouldn't take more than a few hours to lean most if not all of its basics.

[@delete7316]

I’m jealous of you. Python is super easy to learn.

[@Ragnarok540]

My snake became self aware, any ideas to stop it from taking over the world?

[@matthewwisdom426]

😂😂😂😂😂 Mine has hacked Russia, it's now impersonating Putin and is about to start world war 3.

[@sidheshwartiwari9834]

🤣

[@sidheshwartiwari9834]

@@matthewwisdom426 lmao 🤣🤣 i am drying 😂😂

[@RANDOM_DUD-qj3jd]

no windows opened when I ran. not even any errors. how do i fix it?

[@bobjeff6779]

did u get it

[@piyushsrivastava7636]

Can we create AI bot to play dice game?

[@sidheshwartiwari9834]

Well, it's already done. Almost every Ludo game has a bot. In fact you don't even need AI and ML for it as the conditions are so few that it can be hard-coded

[@serverautism2268]

after 6000 games it averages at 33.4

[@user-wz2xn4ni2r]

I want it to play minesweeper

[@yarinh8417]

somoone knows hot to imporve the snake so he does not collide with itself and loop over itself?

[@aliabbas-xs6qm]

Discard the AI an use a hamilton cycle kind liek how codebullet did in his video, It wont be an AI but it wont loop or kill itself

[@pepeCastillo]

Jobs Is alive?

[@aoebb5021]

5:31

[@cryptomoney6901]

There is a small bug where the snake will eat itself if you go in the opposite direction that its going in. To prevent this from happening change this line of code: if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT : self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT : self.direction = Direction.RIGHT elif event.key == pygame.K_UP: self.direction = Direction.UP elif event.key == pygame.K_DOWN : self.direction = Direction.DOWN to: if event.type == pygame.KEYDOWN: # 2nd conditional to prevent snake from eating itself! if event.key == pygame.K_LEFT and self.direction != Direction.RIGHT: self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT and self.direction != Direction.LEFT: self.direction = Direction.RIGHT elif event.key == pygame.K_UP and self.direction != Direction.DOWN: self.direction = Direction.UP elif event.key == pygame.K_DOWN and self.direction != Direction.UP: self.direction = Direction.DOWN

[@slamsandwich19]

He fixed that later in the video

[@omidelahi6196]

I got this error: ValueError: not enough values to unpack (expected 3, got 2) for line 118 in agent.py

[@ericb9056]

I got that as well I forgot to type out the first value of the associated return in the game file. check that you have, return reward, game_over, self.score at around line 82 and 96

[@omidelahi6196]

@@ericb9056 Thanks dude

[@dereinedudeda5298]

Grüße aus Deutschland

[@Peaceful-er4vf]

12:00

[@mohammedismail6872]

dumb question ik, but what is the box he is usint to code, is it CMD, or windows power shell etc ?

[@melvinliew2426]

It might be cmd,but for me is anaconda prompt

[@washyb]

hes using some CMD on a Mac, I think.

[@AhmadBakdash07]

Not first 😑

[@bindiberry6280]

Do you legally own the AI you trained?!!

[@mesh8349]

yes

[@aoebb5021]

12:27

[@chandramoulidasari3946]

What was the theme?

[@Agesilas2]

set video speed at x1.25 or x1.5, thank me later.

[@unionid3867]

The training time is very very long

[@xrhstos1330]

Very interesting topic, but a very bad teacher. He doesn't explain almost anything and the little he explains he talks about them very briefly like we already know all of those things and we make a revision.

[@30DaysMonkMode-ft1kf]

Exactly. Do you know some good tutoriors for this?

[@the_person]

0:00 white man jumpscare