AlphaZero from Scratch - Machine Learning Tutorial

In this machine learning course, you will learn how to build AlphaZero from scratch. AlphaZero is a game-playing algorithm that uses artificial intelligence and machine learning techniques to learn how to play board games at a superhuman level.
🔗 Trained Models + Code for each Chapter: github.com/foersterrobert/Alp...
🔗 AlphaZero-Paper: arxiv.org/pdf/1712.01815.pdf
✏️ Robert Förster created this course.
Website: robertfoerster.com/
⭐️ Contents ⭐️
⌨️ (0:00:00) Introduction
⌨️ (0:01:35) Overview - Part 1
⌨️ (0:05:43) MCTS-Explained
⌨️ (0:27:03) AlphaMCTS-Explained
⌨️ (0:39:05) Overview - Part 2
⌨️ (0:45:14) Chapter 1: TicTacToe
⌨️ (1:00:32) Chapter 2: MCTS
⌨️ (1:34:54) Chapter 3: Model
⌨️ (2:03:09) Chapter 4: AlphaMCTS
⌨️ (2:16:39) Chapter 5: AlphaSelfPlay
⌨️ (2:35:13) Chapter 6: AlphaTrain
⌨️ (2:47:15) Chapter 7: AlphaTweaks
⌨️ (3:08:18) Chapter 8: ConnectFour
⌨️ (3:21:48) Chapter 9: AlphaParallel
⌨️ (3:55:59) Chapter 10: Eval
🎉 Thanks to our Champion and Sponsor supporters:
👾 Nattira Maneerat
👾 Heather Wcislo
👾 Serhiy Kalinets
👾 Erdeniz Unvan
👾 Justin Hual
👾 Agustín Kussrow
👾 Otis Morgan
--
Learn to code for free and get a developer job: www.freecodecamp.org
Read hundreds of articles on programming: freecodecamp.org/news

[@robert.foerster]

❤I hope you enjoy this course. :) At least I had a lot of fun building it. Feel free to ask questions below in case something is hard to comprehend. Additionally, I've also built MuZero from scratch. Would you be interested in that kind of video? Please let me know. Much Love.

[@ryomensukuna9513]

😍🙏

[@herotrojan1645]

awesome and amazing as i'm a chess player myself.. alphazero is a very interesting and important subject to learn.. thanks!

[@saurabh_pande446]

Yes I am interesting in MuZero , plz make a video 😊

[@rohitchan007]

Yes, please.

[@54peace]

Thanks for this course!! why don't you start a channel tho?

[@Yeard491]

Modern-day Library of Alexandria! Definitely one of the best channels on youtube.

[@PeterMartens98]

Thank you for releasing this, this is awesome!

[@peter-tm4ex]

hope you finished the course

[@cs0576]

Thanks and alles Gute nachträglich!

[@oosmanbeekawoo]

This video hit my head like a hammer! Another pathway into Machine Learning. Yeah Pandas can sometimes be difficult.. Thank you Robert!

[@harshmishra7774]

So excited to watch this!!

[@hunchen9236]

Robert der einzig wahre❤️❤️❤️

[@Planeflake]

I am building my own board game at the moment and I was really struggling with how to switch from my current qtable based setup over to MCTS. Your video helped me a huge amount. Thank you.

[@robert.foerster]

Nice :)

[@stephenmontague6930]

Which board game? How'd it go? Good luck, if still working on it...

[@XxXRedSn0wXxX]

No f*ing way! I've been dreaming about this exact video for a year. I even read some books about neural networks in chess. Will watch every bit of this video. THANK YOU A LOT!

[@robert.foerster]

♥

[@mathmo]

@XxXRedSn0wXxX you might like this book: Pumperla and Ferguson "Deep learning and the game of go" published by Manning.

[@XxXRedSn0wXxX]

@@mathmo Thank you for the suggestion! Will take a look

[@electronresonator8882]

just found it now, amazing, to think that someone would teach me this for free

[@Fetrah2]

⭐ Contents ⭐ ⌨ (0:00:00) Introduction ⌨ (0:01:35) Overview - Part 1 ⌨ (0:05:43) MCTS-Explained ⌨ (0:27:03) AlphaMCTS-Explained ⌨ (0:39:05) Overview - Part 2 ⌨ (0:45:14) Chapter 1: TicTacToe ⌨ (1:00:32) Chapter 2: MCTS ⌨ (1:34:54) Chapter 3: Model ⌨ (2:03:09) Chapter 4: AlphaMCTS ⌨ (2:16:39) Chapter 5: AlphaSelfPlay ⌨ (2:35:13) Chapter 6: AlphaTrain ⌨ (2:47:15) Chapter 7: AlphaTweaks ⌨ (3:08:18) Chapter 8: ConnectFour ⌨ (3:21:48) Chapter 9: AlphaParallel ⌨ (3:55:59) Chapter 10: Eval

[@heavysaur149]

Thank you so much man, it's not like you copy it from the description. Not at all... not at all

[@Fetrah2]

@@heavysaur149 And who told you that I didn't copy it from the description?

[@heavysaur149]

@@Fetrah2 Well, same exact emojis at the same exact place + same exact orthography + same exact name for the parts + you have the same exact parts (not even 1 more or 1 less) so I can be SURE at 100% You have copied it from the description

[@Fetrah2]

@@heavysaur149 I mean everyone knows that I copied it from the description Why are you arguing with me? This is a silly topic to argue about, how old are you? 5 or something?

[@DeveloperJS314]

Thank you so much! I was searching for this exact tutorial.

[@roodyg5352]

Thank you for this wonderful Machine Learning Tutorial!

[@robert.foerster]

TY :)

[@AlessandroOrlandi83]

Thank you for this video, I’d really love to understand how alpha zero works!

[@nocopyrightgameplaystockvi231]

This is next level.

[@user-vj3gp4hk9k]

thank you bro!!! 完美的教程，非常感谢

[@mko.3971]

Bester Mann Robert ist ein genie der Informatik kunst

[@boristong2094]

I have implemented some other rules by myself and succeeded, it is rly helpful! btw, I wonder how to adapt the code so that it can support a different rule where the first player drops two black pieces and one white piece at the beginning at once, and then the other player chooses the color to play. And then the one that plays the white piece does the next drop and the game continues as usual. How to implement this? If each combination of the first three drops counts as an action, the action_size will be extremely large when the board is large. On a 7*7 board, it will be about (7*7)^3. Is it possible to implement this without increasing the action_size but by changing the game process? I rly want to know! Thx a lot!

[@smokyboy3536]

In the original MCTS example, I believe the w and n values in each node should be kept from the perspective of the player whose turn it was before reaching the node. This makes sense because when doing the selection step, the UCB of the children should be evaluated from the perspective of the player making the move. Alternatively, the UCB can be calculated by using (n-w)/n for the value part of the formula in nodes where the opponent is to move. Nice work though!

[@robert.foerster]

Yeah. I wanted to make the initial example as simple as possible. Thus there is only a single player planning for the future.

[@tksnail6837]

truly an exceptional tutorial

[@robert.foerster]

ty

[@gouravsaha7548]

Amazing

[@its_saber1525]

Oh boy, now were gonna see more alpha zeros in chess

[@daironperezfrias7819]

This is what i need just now

[@mrdbourke]

AlphaZero from scratch?? This is epic!!!

[@dipeshdas3]

Bro make some tutorial on deep learning please.you are awesome teacher.

[@surajjain7119]

Thanks for everything FCC

[@gattorwichar3984]

Great 😃

[@chatrughanprasad7778]

Thanks guys

[@alphatensor]

Thanks!

[@robert.foerster]

TY :)

[@ujjawal..720]

Thank broooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo ❣️

[@urthogie]

In the AlphaZero paper, they only have 2 feature maps output by the convolution in the policy head, and 1 in the value head. You used 32 and 3, respectively.

[@rikanghuang7714]

Nice presentation. Learning how AI play games is my dream. Thank you for making it happening

[@enisbenchahed6267]

Thanks a lot for the video! I have one doubt about the theoretical section at the beginning of the lecture: how come a node ceases to be a leaf node as soon as it expands in two successive child-nodes? What if, for example, the possible actions from one given state are three or more? I don't understand why the maximum expansion from one given node is two. Is that just given as an example? Because for instance, as far as tic tac toe is concerned, from any board position, I would expect a number of actions which is equal to the possible "player moves". Am I correct? Thx in advance

[@robert.foerster]

Yeah, you are right. Two as the number of possible actions is just an example illustrating the way the MCTS works.

[@FalguniDasShuvo]

This tutorial is amazing! Loved it and learned a lot from this! One question: [Timestamp: 2:54:30] Setting p=temperature_action_probs raises ValueError: probabilities do not sum to 1. What would be an appropriate approach to solve it?

[@robert.foerster]

To solve the problem, you must divide temperature_action_probs with its own sum again. | temperature_action_probs /= np.sum(temperature_action_probs) I noticed that bug after the video was uploaded. Therefore, I have added a comment about the bug in the given jupyter-notebook in the GitHub repo. Thank you for mentioning the problem! :)

[@FalguniDasShuvo]

@@robert.foerster Thank you.

[@tim5749]

Can I do this tutorial with visual studio? If so, what suggestions do people have? Thanks!

[@rimurusama5070]

I have a project on coding something like this, but for chess. The only catch is that the rules are a bit different. Say the pawns can move diagonally. Like how much different would it be? Especially in the reinforcement learning part. Like I am asking this question before actually going through this video. To understand the scope actually. From the first few minutes itself it seems quite, so regardless of the answer I would definitely watch this.

[@robert.foerster]

The algorithm is built so that you only have to create your Chess class and tweak the other hyperparameters. The rest can stay the same.

[@percyquevedo5636]

Great video♥. I watch several youtube videos on how to trade in the market but haven't made any headstart because they are either talking some gibberish or sharing their story of how they made it and I do not want to make mistakes by taking risks in my own hands

[@joelgarcia4507]

The best strategy to use in trading is to trade with a professional who understands the market quite well, that way maximum profit is guaranteed, I'll highly recommend Katrina susan, she is my current trader and her strategies are working

[@tishadin752]

I'm amazed you mentioned katrina susan, she is the best and her strategies works like magic. I've been making over 80% of my investment weekly since I started investing with her trading service

[@sophiabrown2005]

This is not the first time i am hearing of Katrina susan and her exploits, how she handles investments and generates good profits, she has really made a good name for her self, but i have no idea how to reach her

[@joelgarcia4507]

You can reach her through TELE

[@joelgarcia4507]

GRAM

[@user-fh2ts6eq7b]

@robert.foerster, thanks for the video, amazing! I have a question tho, im currently working on project where a board game with hidden info is a subject of work and implemented ismcts (everything is done in java, its a uni project. but for ml part we are allowed to use python). My question is, would the approach be the same if I first generate training data (state, policy list, value) and store it in a .csv let's say?

[@stephenmontague6930]

Could you restate the question? Not quite clear what's being asked, but I am interested. Interested, particularly in using MCTS, or related, for a hidden info board game.

[@Taaz2]

I too am trying to apply ISMCTS to a game with hidden information but I am unsure if the NN model/policy based evaluation can even be used? In my card game the opponents hands (remaining cards for given state) are randomized each MCTS iteration (1-. 4.) but I am unsure how something like that fits into the game state (literally the shapes).

[@urthogie]

Hey, isn't there supposed to be 2* ln(N) in the numerator of UCB? You left out the 2?

[@icantorus5091]

I followed along until we hit chapter 3, then it just started going over my head a little bit. The MCTS all makes sense and was described well, however I just don't understand how you come up with the NN architecture for the given problem. Say I wanted to model the NN for a game of chess, what would be different? Is there any videos that help to explain this a bit better and how you approach the solution?

[@robert.foerster]

The link to the original paper can be found in the video description. In the case of chess DeepMind set the number of ResBlock to 20 while having 256 planes per ConvBlock. I tried adjusting the number of layers according to the game's difficulty. (Probably the amount of blocks could be even lower for TicTacToe though).

[@eduardomartin136]

Great! But be careful because I think there is a misunderstanding problem with MCTS (if I'm not seeing it wrong). I see that in each play this tree is created again from that node, which means that the generated tree with stats from previous plays is not saved. From the quote in the alphaGoZero article: "The search tree is reused at subsequent time-steps: the child node corresponding to the played action becomes the new root node; the subtree below this child is retained along with all its statistics, while the remainder of the tree is discarded. AlphaGo Zero resigns if its root value and best child value are lower than a threshold value vresign"

[@amritnalam9994]

What is the UCB formula about?

[@ngawangpxint.cloudyy444]

Robert Wir Lieben dich!🍔🐒

[@user-vj3gp4hk9k]

I want to know whether evaluating the quality of an agent can only be done through the feeling of playing games? Is there a way to quantify it?

[@robert.foerster]

You could write your own elo function in which your current net plays against random or previous versions of itself. Then you could evaluate the information such as the win rate or the game's length.

[@samatvak4472]

I have a partially trained model, but want to train it further. When I am calling the training code, can I initialize `model` with the model that I already have, or do I need to create a new instance using ResNet()?

[@robert.foerster]

If you want to load your model from a file that was created using this script, then you need to create a new instance of ResNet. In this implementation the model file only stores the weights and not the model itself.

[@samatvak4472]

@@robert.foerster Thanks for the response! So when I create the new instance of ResNet to train, I can still initialize it with the weights in the saved model file right?

[@robert.foerster]

@@samatvak4472 yes.

[@TIENTI0000]

thanks

[@diniworks2881]

Thanks @robert.foerster for this video! Am I right that unlike in the real-life tictactoe, where X and O are associated with the same players, here 1 and -1 are not like that player1 is 1 and player2 is -1 but "1" is always associated with the node's move, and "-1" is associated with the child nodes and in the next round, when the child becomes THE node, it turns into 1?

[@stephenmontague6930]

i think so... each node acts as a player1 here.

[@AcidiFy574]

What's the difference between Machine Learning & Artificial Intelligence ? BTW, you guys are the best

[@mlguy8376]

ML is a framework for solving some AI problems - so if you have an AI problem such as determining if a picture of a cat or a dog, then you can use ML to solve that problem. As it stands today we can use AI and ML interchangeably , but in the future this may not be the case.

[@umarmuhammadi429]

Thanks for the video i have two questions and will be happy to be answered The first is what's the prerequisites to this ML The second is regarding a project i want to implement which is computer vision using python language but i don't know where to start Thank you

[@mwanikimwaniki6801]

Prerequisites-Some linear algebra, multivariate calculus, stats and probability. You don't have to be an expert at them just need to know enough to have an intuitive understanding of algorithms... Everything you need on machine learning you could learn here

[@samatvak4472]

After training the model, the model.pt file is created, but an optimizer.pt file is also created. What is the use of this? In the video I don't think he used this file.

[@robert.foerster]

In case you train your model further the script will use the optimizer file to load the parameters of your optimizer that were tuned or directly set during previous training. E.g. this is helpful if your optimizer has an adaptive learning rate.

[@ivankirchev7348]

What prerequisites would be recommended before taking on this course?

[@robert.foerster]

Here is a list of recommended prerequisites: fundamental / preferably intermediate python skills fundamental numpy skills fundamental pytorch skills + brief understanding of a jupyter notebook Best Regards

[@SketchSavvy01]

Robert

[@M8Miss]

can i create the same thing but with javascript?

[@ameyapotdar461]

Please make a video of CRUD using django and vanilla javascript

[@MysticLGD]

nice

[@kemal2806]

Thanks God and simulation

[@anjuvaze]

This is my humble request ..Kindly add transcript.

[@dErWixxer63]

I always thought that "leafs" in the tree structure are nodes that dont have any children, not even one. Is this not the case? 10:25

[@robert.foerster]

en.wikipedia.org/wiki/Monte_Carlo_tree_search | "a leaf is any node that has a potential child from which no simulation (playout) has yet been initiated". Oftentimes you directly expand in all possible directions though. Therefore, there are also cases in which your statement is still correct. Best Regards

[@dErWixxer63]

@@robert.foerster Thank you!

[@Pratimasingh-mg9lc]

Please make video on advanced JavaScript

[@user-gf9ri4wj5h]

Don’t do ai with js😂

[@Pratimasingh-mg9lc]

I need advanced JavaScript course by freecodecamp that's why I have written this comment and I am doing AI with js

[@miguelmyers9546]

i wonder how hard this would be if you did the same thing for a 3d first-person game

[@robert.foerster]

I trained MuZero on a CarRacing-Game. Would you be interested in that kind of tutorial?

[@miguelmyers9546]

@@robert.foerster ABSOLUTELY

[@anjo5752]

Robert will be the pinnacle of the next generation of coders 🧗

[@Paragon_Reason]

HERE'S THE DEAL; I'm intrigued by Machine Learning, A.I. and it's ability to to develop superior methods to achieve a goal such as what Alfa did to StarCraft II, or how the Fighter Pilot Simulations learned to battle better than any human such as in the Documentary on Netflix called "Unknown Killer Robot". I do not know how to code.... My education is in safety, not computer science sadly.... Is it possible for a newbie, to learn how to take data, input it into computer, and develop parameters and run it over and over to develop best methods like Alfa? How many years will this take? Or does that take millions of dollars and an entire company to do over the course of years? Thanks.

[@mabrur4304]

please Java EE course

[@TJ-hs1qm]

while we play with ourselves 😅

[@pranavramesh2893]

Day 3 of asking for latex coding.

[@marcelkruger5187]

*typesetting

[@IsxaaqAcademy]

It's easy and there are plenty of resources about it

[@jinwoo78]

There's this thing called "Googling" and there's also this thing called "learn it on your own". 😒

[@japaneseninya]

robert ein macher. neue generationen

[@viddeshk8020]

One request please don't use white screen. It hurts my eyes.

[@robert.foerster]

I'll keep that in mind for the future. TY

[@nicoleparedes5041]

I don't know a thing about coding or informatics or computer science. Can someone recommend me some videos from this channel to get started? There are too many and I'm not sure what is supposed to be more advanced or more beginner friendly

[@benehasler]

Ignore the other spam comment. What really helped me out was just sitting down and watching the whole course on Python by Giraffe Academy. It's also available here on freeCodeCamp's channel. That really kicked off my interest in programming and helped me understand the core concepts. greetz :)

[@oosmanbeekawoo]

Read a book called ‘Python Crash Course by Eric Matthes’. The guy who made this video read this book too! I'm sure he'd recommend it. There is also a video on this channel called (kzbin.info/www/bejne/m4i1l5uQit-ld8U) which I don't recommend cause he treats Classes badly. Then you should take a course on Numpy before coming back to watch this video!

[@ondrejlukas1121]

excellent hair

[@tamerkaratekin9074]

Hey Robert, what is your email? We are building an curriculum for reinforcement learning, maybe you can shoot a few videos, prepare a few lessons. We have some budget.

[@BAMBAMBAMBAMBAM-]

We are now living in a time where math can do math

[@Kalaphant]

50:20 How do you know the player's gonna be a he?

[@Kalaphant]

Oh hey this was 1 year ago! (Random lol)

[@SketchSavvy01]

Hiii

[@yoyoshi2833]

30:00

[@mija_531]

He hast the typical German accent

[@CryptoLando]

so much code lol

[@EPrimeify]

This guy needs a haircut

[@robert.foerster]

facts

[@EPrimeify]

@@robert.foerster sorry, bruh

[@ngawangpxint.cloudyy444]

@@robert.foerster 😂