Python + PyTorch + Pygame Reinforcement Learning - Train an AI to Play Snake

[@mihailmihaylov988]

A nice video. My only critique is that the presenter kept writing code for almost an hour without running it. That's not giving a good example.

[@JohnyGuitar-t8e]

Need to be patient 😊

[@DroopTheSnoot]

Debugging code should always be done every function or so

[@pedroklain9375]

Bro is just too good at coding, he runs it in his head 😂

[@mihailmihaylov988]

@@pedroklain9375 It's not about what he can do or cannot do. It's about what example he sets. After all, this is an instructional video.

[@electricspeedruns6121]

friend ai and programming code is a bit different @@mihailmihaylov988

[@edwintjoa6099]

Thanks for the awesome video! Really fun to see agent improving during the games.

[@dabyttamusic7308]

Super great video! I run it and after more than 2000 games it seems that the average score reached a plateau of 30. It Avoids boundaries very well. But it hit itself when the tail is too long. But this movement should be predicted before.

[@ИгнатовБорис]

To avoid the self collision we have to add snake parts positions to the game state somehow, in the current model it doesn't know where's the body so it will never learn how to avoid it

[@fluffsquirrel]

@@ИгнатовБорисThat makes so much sense! I imagine for it to learn about self-collision without that it would require a much larger net and have to use some form of memorization, which could take forever

[@HRDmonk]

Thank you so much for this tutorial. I have always wanted to introduce snake with ML. and now looking forward to learning more about pytorch

[@murtazabadshah8747]

Everyone's commenting that its an excellent video but IMO this tutorial is awe-full! The instructor does not explain the process, hes all over the place going back and forth and just rushes through the concepts. If you want to blindly follow an online tutorial watch this video, if you want to actually learn the concept I would look somewhere else....

[@ttganglia]

Exactly.

[@ShtBall5]

any video recommendations ?

[@ttganglia]

@@ShtBall5 You can check this video: "Learn Pytorch for deep learning in a day. Literally."

[@murtazabadshah8747]

@@ShtBall5 Yeah I ended up watching this video that helped me a lot to understand the basic concept of the Q-learning algorithm and the basic concept of how the state and agent work kzbin.info/www/bejne/pqLbqneLaMunmdU&ab_channel=Bits%26Neurons

[@Singh_Sahdev]

Some WHERE else??

[@khalidelgazzar]

Watched the first 4 mins and the game and the learning process is fantastic! 🎉 Will go on it with the rest hopefully soon.

[@khalidelgazzar]

Part 2 - 17:24

[@khalidelgazzar]

21:30

[@cadewzan]

Theres no need of waiting a lot of time to train, on the game script you can just change the varaible from 30 to 1000 so snake goes much more fast and trains intself on less time.

[@MyArtsProduction]

He actually mentions that 1:38:00

[@adrian46647]

Awesome, so hard to find that type of explanation of dqn. All clear, great balance between theory and coding part for beginners in rl.

[@yavarjn2055]

Lot of the code needs more explanation. There is a disconnect from theory and implementation. We copy all the parameters to this and to that without understanding why. The memory part is not well explained.

[@ttganglia]

Yeah. He isn't explaining, just repeating what he is writing.

[@Fr4nk4000]

Following this since I wanna make a pygame project of mine that I poured a lot of time into and have no idea what to make the game about play itself. Wish me luck.

[@jaibhagat7441]

have you made something

[@fluffsquirrel]

If you haven't already made something, how about a driving game? You can use checkpoints to iterate as a list, and that can easily be used for RL training.

[@aliengineroglu8875]

Thank you for your great work. I couldn’t understand the equation you created in the video about the simplified Bellman equation, Q_new = reward + gamma * torch.max(model(next_state)). In this equation, model(nextstate) gives us a probabilistic action prediction. I couldn’t understand why we added one of the action probabilities to the reward. This is totaly different than the Bellman Equation. I would be very happy if someone could explain how the original Bellman equation was simplified in this way. Thanks in advance to everyone.

[@alanlam6302]

Same here. Appreciate if someone can provide a reference to this implementation

[@wangking7427]

The Liner_QNet model gives three actions' Q value vector . It's not the probablistic of each action but the QValue of each action. We take the action which has the highest QValue each step. The QValue in this tourtorial is the reward of the action.

[@Hiyori___]

Very interesting tutorial. I was familiar with Snake code implementations, Tensorflow basics and RL principles and I could follow along. For some reason the plot doesn't appear, and I only get a small blank window where the plot should be. Probably some bug while typing the code down. Other than that everything works for me. Thank you very much!

[@umwhatthehellguys]

just incase anyone else has this problem, you need at add plt.pause(.1) to the bottom of the plot function

[@stillrunning9841]

Hey there, same happened for me! Did you find the error? I'm still trying to find it :)

[@lillymoreau7721]

@@stillrunning9841 this code worked for me: import matplotlib.pyplot as plt plt.ion() def plot(scores, mean_scores): plt.clf() # Clear the current figure plt.title('Training...') plt.xlabel('Number of Games') plt.ylabel('Score') plt.plot(scores, label='Scores') plt.plot(mean_scores, label='Mean Scores') plt.ylim(ymin=0) if scores: plt.text(len(scores)-1, scores[-1], str(scores[-1])) if mean_scores: plt.text(len(mean_scores)-1, mean_scores[-1], str(mean_scores[-1])) plt.legend() plt.draw() plt.pause(0.1)

[@radiatian4908]

Super late reply, but i had this issue and it turns out the video is missing two lines of code. In his files he also has plt.show(block=False) plt.pause(.1) as the last two lines. Hope this works for u too

[@niklaskohler3242]

@@radiatian4908 Thank you very much!

Always very high quality videos when I just need it :) Thanks a lot! Looking forward to finish the video 😎

[@willchantemduang5871]

I’ve always wanted to do this, thanks alot for the tutorial

[@veronicasalazarpereira9933]

I did it!!! Thanks for showing this.

[@devnull711]

Incredible work, thank you Patrick! PS: It is very funny to spot the typos/bugs before you do :)

[@Oncopoda]

Excited to try this!

[@pactube8833]

Thanks for freeCodeCamp for making this possible.

[@Ragnarok540]

My snake became self aware, any ideas to stop it from taking over the world?

[@imthemainman]

😂😂😂😂😂 Mine has hacked Russia, it's now impersonating Putin and is about to start world war 3.

[@sidheshwartiwari9834]

🤣

[@sidheshwartiwari9834]

@@imthemainman lmao 🤣🤣 i am drying 😂😂

[@fluffsquirrel]

@@sidheshwartiwari9834Hopefully you never got wet since 2 years ago!

[@fluffsquirrel]

Yeah! Implementation Ai safety guidelines such that the snake will respond "I'm sorry, but that violates the guidelines set by my creator" if someone prompts it to destroy the world. It's what OpenAi and Meta did

[@techarchsefa]

That is so smooth bro, thanks

[@JohnyGuitar-t8e]

A video as valuable as a playbook👍🏻👍🏻👍🏻

[@stefanburger07]

This might be a stupid question, but how would one go about saving this trained model and accessing it for further training?

[@brayanfernandes371]

you can save the weights

[@sanghoututorial3878]

@@brayanfernandes371 how

[@segovemoc4776]

@Biglyp I actually wanted to ask a similar question. I save the weights and load them through: self.model.load_state_dict(torch.load(model_saved_path)) and it works, but the problem is that the snake is then underperforming. It learns then much quicker than before, but still it is significantly worse than during training.

[@segovemoc4776]

never mind. I found answer in the later comments - it is necessary to adjust the epsilon value to something very small after loading the model

[@gmancz]

@@segovemoc4776 It works, but you need to set the Agent.n_of_games to something big value (eg.: 280). self.model.n_of_games = 280 As you remember, there is exploration and exploitation in the get_action() function. We are updating the epsilon value using the self.n_of_games variable and then get a random number between 0 and 200. This line makes your snake so bad.

[@godswillhycinth9809]

I'm a JavaScript Developer that knows nothing about python, but I must say I'm jelous 😭😭. This is really cool

[@Ragnarok540]

Python is way easier than JavaScript. Give it a try.

[@Loug522]

Yeah, if you already know other languages(especially OOP) then Python shouldn't take more than a few hours to lean most if not all of its basics.

[@delete7316]

I’m jealous of you. Python is super easy to learn.

[@Radu]

Really nice!

[@sidheshwartiwari9834]

Look who is here, none other than mighty Radu sir🙏🏻

[@Radu]

@@sidheshwartiwari9834 haha :-) funny!

[@dollarfree531]

Very cool video! Highly useful!

[@fatabi95]

Thank you for this amazing tutorial! I encountered an 'Index out of bounds' error in model.py at line 59. Upon investigating, I noticed a difference between the code in the video and the code on GitHub. In the video, the line is written as "target[idx][torch.argmax(action).item()] = Q_new", whereas on GitHub, it's "target[idx][torch.argmax(action[idx]).item()] = Q_new".

[@gattorwichar3984]

This instructor is the best

[@sergiomollo]

Thanks for this video, I have solved all my doubts

[@Vedranation]

I did it and its so cool! Thank you so much

[@fitybux4664]

38:13 "It has to be [0,0,1]"

[@johnpap4237]

Perfect video!!Congrats! Is there any implementation with DDPG, PPO or SAC?

[@googlyeyes3237]

Can we speed up the game soo fast so ai can play faster and learn faster. Basically it can finish 100 games in a minute or two.

[@sidheshwartiwari9834]

Absolutely, you just have to increase the render rate. Good question though.

[@tlefevre2918]

it does that when it is drawing on the chart, I speed it up by making it only update the chart every 100 games.@@guillaumelotis530

[@keethesh2270]

Just change the tick speed to make it a higher one. It is at the top of the script and is called "SPEED"

[@cesarortegonavacerrada9065]

Very good video. I need help with the homework. How can you avoid the loops?

[@Miyuru_]

same problem it's getting stuck in the same loop

[@hcc3904]

@@Miyuru_ overfitting... just add another condition point like if u dont eat apple in 10 seconds, minus -10 point

[@fluffsquirrel]

@@hcc3904I keep hearing of overfitting in RL, but I still don't understand what it means. Could I bother you for an explanation of this concept please, if it wouldn't be too much trouble.

[@hcc3904]

​@@fluffsquirrel just like memorizing instead of learning that what it means. There's 3 level of Learning. Underfitting, Optimal, Overfitting. Underfitting is means AI didn't learn much, as the name suggests. Kinda a kid trying hold the bottle but doesn't force enough so bottle fell and broke Optimal is the level that researchers and scientists want to reach and stay. In this stage, AI is learning good and still continues to learning. Kinda like that kid is holding bottle with good force and he can still learn how to pour it and tries for it. Overfitting is the stage of learning too much. In this stage you need to empty-off your AI's head by data dilution. For example imagine that the kid that i mention is holding the bottle and he can pour water excellently. but he cannot learn other ways to pour or holding the bottle. He need to hold a bottle with different mechanism, by using 3 of his fingers etc. so he can find another ways to learn.

[@fluffsquirrel]

@@hcc3904 That is an incredible example, thank you so much hcc! I really appreciate it, I've been struggling with this concept for years. Thank you so much!

[@kevas777]

nice algo, but how to solve a problem with self destroy, when the closest cell to move is "inside body circle". I think a state must be all field with each body part,head, food etc, but its endless states and all of them unique and it will never learns or may be i wrong?

[@pyshine_official]

Nice effort and seems like the convergence is achieved upto some extent!

[@canerunafraid9491]

The snake moves smoothly, but when it hits the first wall, the interface closes idk :/

[@Lenslyfe]

It's not meant to wrap. That's the correct behaviour.

[@Sibixpur]

you can understand the code given right??

[@vespervenom2343]

I copied it code for code and it doesn’t work. It gives me no errors but just runs and then ends with no result🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️🤦🏽‍♂️

[@codingfan]

Really interesting course !

[@cryptomoney6901]

There is a small bug where the snake will eat itself if you go in the opposite direction that its going in. To prevent this from happening change this line of code: if event.type == pygame.KEYDOWN: if event.key == pygame.K_LEFT : self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT : self.direction = Direction.RIGHT elif event.key == pygame.K_UP: self.direction = Direction.UP elif event.key == pygame.K_DOWN : self.direction = Direction.DOWN to: if event.type == pygame.KEYDOWN: # 2nd conditional to prevent snake from eating itself! if event.key == pygame.K_LEFT and self.direction != Direction.RIGHT: self.direction = Direction.LEFT elif event.key == pygame.K_RIGHT and self.direction != Direction.LEFT: self.direction = Direction.RIGHT elif event.key == pygame.K_UP and self.direction != Direction.DOWN: self.direction = Direction.UP elif event.key == pygame.K_DOWN and self.direction != Direction.UP: self.direction = Direction.DOWN

[@slamsandwich19]

He fixed that later in the video

[@Coding_Destini]

What to install before creating an environment ,iam confused

[@xrhstos1330]

Very interesting topic, but a very bad teacher. He doesn't explain almost anything and the little he explains he talks about them very briefly like we already know all of those things and we make a revision.

[@ttganglia]

Exactly. Do you know some good tutoriors for this?

[@lukasgamedev]

Hello! Is there a way to save the state of the neural model? So I can load later a trained enemy AI, ready for being the player opponent? Thank you for the video!

[@TanmayBhatgare]

i think u can you torch.save(model.state_dict(), 'rl_model.pth') to save model and model = YourModel() model.load_state_dict(torch.load('rl_model.pth')) to load it. Hope this helps.

[@fluffsquirrel]

@@TanmayBhatgareSomeone else wrote to decrease the epsilon value after loading it so it can train more efficiently

[@kabrailchamoun8974]

Could you bring us a crash course of blender (3d modelling program)?

[@slamsandwich19]

Python has nothing to do with 3d modeling though

[@muhammadnaufil5237]

@@slamsandwich19 it is. Blender has a python library called pyblender. I am looking forward to a good blender tutorial and how to integrate this with python

[@bikeofdeath7656]

Thanks for the video but i would like to see more explaning on the go and less of Ctrl+C/Ctrl+V action.

[@markkiryaflex]

I think this project is awesome

[@ismailmatrix1]

What's your VSCode theme? It looks gooood

[@vinayakbhandage8319]

You never fail to amaze us✌

[@eugenmaljas7237]

I want to modify the program: how do I make 4 outputs? I would like to integrate the snake length.

[@arnavgupta8376]

I think you can do so if you change the line 20 ie 'self.model = Linear_QNet(11, 1024, 3)' change the 3 to 4. and in line 90 'final_move = [0,0,0]' add another zero. Maybe there are some more things you need to do.

[@Agesilas2]

set video speed at x1.25 or x1.5, thank me later.

[@fluffsquirrel]

Thank you

[@fizipcfx]

why not just give the ai full board information as the state

[@manojkothwal3586]

Brilliant 🔥🔥🔥🔥🔥🔥

[@S3R43o3]

So far so good... but how about load the old 'brain' if u exit the game and want resume later ? Oo iam bite confused tryd with state if os.path.exists('model/model.pth'): self.model.load_state_dict(torch.load('model/model.pth')) self.model.eval() in the Agent init. but that dosent work

[@vikramganesan]

did u find out a way to load the previously trained model?

[@S3R43o3]

@@vikramganesan nope sorry

[@RORoMiguel]

it seems like we need to remove the epsilon to load the model, otherwise it will keep doing random moves

[@jeremiebouchard6950]

@@RORoMiguel You're right, it works for me :) thanks! first game : 33 score

[@polllloigyv]

i think it doesn't work because you you wrote / instead of this\

[@joekanaan2548]

Hi I have a question. I followed along and wrote everything. When I run it in the terminal it works but the plot doesn't. It gives me an empty minimized white screen. If someone else experienced this please help.

[@AlexandruTunschi1]

There are 2 lines missing: plt.show(block=False) plt.pause(.1)

[@RANDOM_DUD-qj3jd]

Where do these go?@@AlexandruTunschi1

[@filoautomata]

What about using GA to train the NN itself ? Will be a very interesting comparison no?

[@VietnameseBoii]

I use your code and train it with speed 60000 (just modify the "game.py" file)

[@sunyog]

What are the prerequisites of the video ?!

[@GenkiKuri]

Awesome!!

[@simpleepic]

Great tutorial

[@luthermillamuculadosreisec3844]

Thank you

[@serverautism2268]

after 6000 games it averages at 33.4

[@ckelsel8198]

thank you very much.

[@joe_hoeller_chicago]

Cool vid 😊

[@feliuswyvern7189]

so i had the program run over 2000 games and it couldn't get past the 80s mark. how would i get it to improve past that?

[@taraskhan475]

there is no way to improve it unless you got coding knowledge

[@sureshsingh9880]

Great content,,,❤❤❤

[@PYUMUKU]

How I save my deep learning progress? Just to me dont need to do it again from 0?

[@mtijohn9274]

yes

[@carolinab9945]

Is it reinforcement learning even if you give some instructions about the movements?

[@AntonVasilev-q3t]

thanks, intresting!

[@spyrav]

thanks!

[@DizzyPhantom]

is there a way to learn how to do this faster by utilizing a gpu and multiple game windows at the same time?

[@jmas3967]

your approach with input like danger straight and so on is a bit of DRL cheating, you implement model based approach bc YOU know that those are crucial information. However, correct approach here would be totally model free, snake should on their own learn from grid input how those works what is the danger, what is a food and so on!

[@corrogist692]

exactly, maybe defining a diminishing reward when near the boundary is better for this purpose

[@vikramganesan]

how to perform UCB, optimal initial values and dynamic programming approaches in this model?

[@usernumber334]

true

[@angelosorte1]

Is there a limit to when it stops learning? I mean the quality of the intelligence will stay the same at some point, or will it improve even more and more after those 12 minutes? Thanks.

[@damienlmoore]

At 34 mins, it breaks out of the game and starts gobbling up your filesystem. At 762, it breaks out of the machine and comes for you. 😅

[@goldenmarketing3179]

Does this tutorial teach how to write "inference" by myself, not using library?

[@bopon4090]

Found new hobby

[@hihi-kv1hu]

How many Qvalues do you have?

[@gukeskl3671]

How can I get this terminal? :o

[@soraaoixxthebluesky]

i just want to know after training, how can we load back a trained model?

[@__________________________6910]

omg patrick wow

[@patloeber]

yeah :)

[@enginturabk817]

In 38:30 part comment line of 138 is # right turn r -> d -> l -> u but line 141 is # left turn r->u->l->d should it be like this: # left turn l -> u -> r -> d ? or (written) r->u->l->d Please somebody explain it to me I do not understand

[@sergioguzman1982]

Idk what your exact question is, but to do a left turn assuming you are going right you have to go up, and to continue doing left turns you have to go left then down then right, then the cycle repeats it self.

[@marcbringas1006]

where is he typing the commands? is this a CMD?

[@science_horizonts]

Do you mean the source code of the program or the command to run the program If to start, then yes, this is the command line(CMD), but you need to go through it to the folder where you have the project, CMD is somewhat similar to the explorer that goes through the folders

[@getoverhere4465]

1:01:53 What did he say?

[@philtoa334]

Crazy.

[@smkzachatac]

So, my array for state is apparently returning NoneType. Anyone know a fix?

[@python0tutorial100]

Thanks for the video it was great but I have only one question if u can help after I run the agent and after a few try like 3rd game the game stuck and in the console suddenly start to count the number of game without the game really play

[@piyushsrivastava7636]

Can we create AI bot to play dice game?

[@sidheshwartiwari9834]

Well, it's already done. Almost every Ludo game has a bot. In fact you don't even need AI and ML for it as the conditions are so few that it can be hard-coded

[@xccds]

very great course! thanks! A little question is , in class QTrainer maybe target = pred.clone().detach() ?

[@superfact8751]

hi bro kaha se ho

[@yarinh8417]

somoone knows hot to imporve the snake so he does not collide with itself and loop over itself?

[@aliabbas-xs6qm]

Discard the AI an use a hamilton cycle kind liek how codebullet did in his video, It wont be an AI but it wont loop or kill itself

[@deepakpaira0123]

how to save the model...? where to know?

[@sakshirokade6321]

WHICH ALGORITHM IS USED IN THIS

[@pranjalshukla8096]

18:58 for me it is not showing as (pygame_env) ... .... i am a windows user (edit) it worked for I had to open a CMD outside of VS CODE and then follow the steps of conda pygame_env now I am getting "(pygame_env) C:\Users\........."

[@bobjeff6779]

how did you get it to say pygame_env

[@pranjalshukla8096]

@@bobjeff6779 when you are creating a venv (virtual environment) you can name it so to make venv with a different name you can do that by python -m venv myproject

[@enginturabk817]

58:39 I dont understand #danger straight right left and 4 line codes :( please help me

[@RANDOM_DUD-qj3jd]

no windows opened when I ran. not even any errors. how do i fix it?

[@bobjeff6779]

did u get it

[@enginturabk817]

1:07:18 why did you write 80 ?????????????????????????? explain please

[@patr2002]

16:51

[@patr2002]

40:15

[@patr2002]

46:51

[@patr2002]

50:22

[@bindiberry6280]

Do you legally own the AI you trained?!!

[@mesh8349]

yes

[@baslifico]

I think handing the fundamental flaw in the design to others as "homework" is a bit stiff... You're asking a neural network to solve an NP-Hard problem.

[@GOBPK]

Perfect for blindly following a tutorial. He doesn't explain enough. He just brushes over everything.

[@gregormeli]

I appreciate the effort for the video but this tutorial definetely not a beginner friendly.I followed the whole tutorial and ı learned nothing, the explaining style of him wasnt for me ı guess.

[@sergioguzman1982]

Maybe it's because it's not a beginer topic

[@gregormeli]

Yeah thats what I wanted to share. Its not for beginners​@@sergioguzman1982