Deep Reinforcement Learning Tutorial for Python in 20 Minutes

[@supankanlavanathan463]

Video Summary (Made with HARPA AI):- 00:30 🧠 Core concept: "Area 51" summarizes Action, Reward, Environment, and Agent in reinforcement learning. 01:00 🐍 Python setup: Use OpenAI Gym, TensorFlow, Keras to create and train reinforcement learning models. 04:52 🏞 Gym environment setup: Import dependencies, set up the environment, and extract states and actions. 08:27 🧠 Build deep learning model: Construct a model with TensorFlow and Keras. 13:48 🤖 Train the model: Compile and train with KerasRL, monitor progress. 15:06 🎯 Test the model: Evaluate performance in the Gym environment. 17:30 💾 Save and reload weights: Save and reload model weights. 19:49 🔃 Reuse the model: Rebuild, load weights for further testing or deployment.

[@oleksiy2090]

I do not know what to say. I think closes words what can describe my feelings now are "wow, that was amazing and very very simple that even I understood what is going on there". Going to play with code and try to solve more problems. I wish I found your channel earlier. 👍🏻

[@NicholasRenotte]

Thanks so much @Alex, you've found it now 😊! I've got way more reinforcement learning and game AI coming in the coming weeks!

[@andreapalladino7999]

The best tutorial on how to start with reinforcement learning that I have ever seen!

[@coded6799]

For your content, 6.5k subs are too little. I have been scouring the internet for reinforcement learning courses ever since AlphaGo beat the world champion, and today I found your video. And I'm glad I did.

[@NicholasRenotte]

Yooo, thanks so much! I've got a bunch more RL stuff coming soon!

[@coded6799]

@@NicholasRenotte Cool!

[@freydunthanos3155]

Seriously, I'm recommending this channel to my data science class

[@NicholasRenotte]

@@freydunthanos3155 yesss, thanks so much!

[@rmt3589]

A Go fan. Didn't expect to see another person of culture here.

[@prhmma]

nice pace and simple work through, love it man.

[@NicholasRenotte]

Thanks so much 🙏! Got another run of RL tutorials coming up soon!

[@MK-ol9gv]

I usually don't write comments on KZbin videos but wow! I've watched some of your videos and they are extremely helpful. The number of views on this video and subscribes on your channel are so underrated thx for the great content and hope u keep making good videos like this one!

[@NicholasRenotte]

Thanks so much for your kind words @M K! Truly appreciate it!

[@richard_franks]

tl;dr if you're watching this in 2022, make sure you pip install gym==0.17.1. I'm sure this is due to the age of this video/updated code being released, but I had the following errors in case anyone else comes across this. First was - ValueError: too many values to unpack (expected 4) - for the line n_state, reward, done, info = env.step(action). For some reason adding a 5th parameter so it looked like this - n_state, reward, done, info, test = env.step(action) - made it pass. Next was ValueError: Error when checking input: expected flatten_input to have shape (1, 4) but got array with shape (1, 2) on line dqn.fit(env, nb_steps=50000, visualize=False, verbose=1). I was able to fix this by downgrading to python 3.8, downgrading protobuf to 3.9.2, and explicitly installing the versions of all traces found in the pip install trace of the jupyter notebook. When I changed the gym version to the one found in the video, it allowed env.step(action) to actually take 4 parameters, instead of the 5th I had to add in to make it pass, and the code ran. After all that I went back to python 3.10, explicitly installed gym 0.17.0, then installed keras, keras-rl2, and tensorflow, and it worked again. Thanks for the video, the issues obviously aren't your fault, just wanted to pass this info off. I learned a ton about pip, library versions, and all kinds of other stuff in this process.

[@CrossyChainsaw]

This worked for me aswell i downgraded protobuff which downgraded tensorflow aswell. After i upgraded tensorflow to the correct version and everything worked. I think the origin of the problem is not having the correct version of TensorFlow in the first place

[@lotus.css_IV]

TYSM YOU'RE SUCH A G

[@nahiyanalamgir7056]

If you can, try upgrading to gymnasium, a drop-in replacement for gym. gym is no longer maintained.

[@sagnikroy6405]

I watch your videos and feel like you taught us a very important topic like no one did. I do believe this is how no one shouldn't. Better to follow written documentations!!!

[@jumiduss]

Commenting for the algorithm. Started looking into deep learning recently and eventually got here, great intro and explanations. Looking forward to the other videos

[@NicholasRenotte]

Thanks so much!! Whatcha working on?

[@_FLOROID_]

As far as I can tell this tutorial sadly is already outdated since some of the API has changed now and some functions may require different arguements. And updated version of this tutorial would be great!

[@bogdanoleinikov8002]

Thanks for explaining the code, I saw this example online already but with the step by step explanation of this scenario it was much better for learning while running the code alongside the video :)

[@NicholasRenotte]

Heya @Bogdan, thanks so much! I'm building up to more sophisticated examples of RL. I'll be doing a lot more with different environments in the coming months!

[@kalkies2008]

Just discovered your course, amazing! Thank you very much. It is still very relevant. Some of the gym environments have newer versions but all still works. Thanks again!

[@raihankhanphotography6041]

I am so glad I stumbled across your channel. Best tutorial ever! THANK YOU!!!

[@NicholasRenotte]

Thanks sooo much @Raihan!

[@kashyapbrahmandam3586]

Bro's got a taste for classic music. Beethoven and Dvorak in the beginning. Nice!!

[@paulburnett1963]

Sweet.. love the explanation.. That was a lot to take in but what a clean explanation... Thanks for the video. paul.

[@BRUNO12059]

I am from Brazil and your video was very useful for me !!! I hope you to continue to make more videos like that. Great video !!

[@NicholasRenotte]

Glad you liked it @Bruno! Definitely, got a special one on Reinforcement Learning coming up!

[@tomgolf2624]

Thank you Nicholas.. Your video is very informative, nice pace and entertaining. I am now hooked with RL.

[@NicholasRenotte]

Thank you so much @Tom 🙏

[@MaximeAntoine97]

Awesome video! I just started my master in AI and seeing your videos helps a lot to remember a couple “key” things before the start of the semester! I also just started a YT channel, if you’re down we could maybe see how we could create something together, might be fun! Have a good day 👋🏼

[@NicholasRenotte]

Hey thanks co much @Maxime, glad you enjoyed the video!

[@tommclean9208]

If anyone had the same issue as me, using keras-rl saying that model has no attribute __len__, I just modified the model code to: def build_model(states, actions): model = Sequential() model.add(Flatten(input_shape=(1, states))) model.add(Dense(23, activation='relu')) model.add(Dense(23, activation='relu')) model.add(Dense(actions, activation='linear')) model.__len__=actions return model and it worked (notice the additional line model.__len__ = actions Probably not the best practice, but worked without having to downgrade tensorflow

[@NicholasRenotte]

Thanks so much for helping out the fam @Tom!

[@mohammedbasheer581]

Thank you Nic! Very helpful of you to make such informative videos for all! Wish you lots of success and joy!

[@NicholasRenotte]

Thank you so much @Mohammed!!

[@BlinkDrive555]

In 2022, the code might not work well Instead of : from tensorflow._api.v1.keras import Sequential from tensorflow.keras.layers import Dense, Flatten from tensorflow.keras.optimizers import Adam you need to use : from tensorflow.python.keras import Sequential from tensorflow.python.keras.layers import Dense, Flatten from tensorflow.python.keras.optimizers import adam_v2

[@caleboleary182]

Thank you!

[@RajaSekharaReddyKaluri]

Thank you! This would be my first motivation to explore RL!

[@alexeysmirnov1678]

Superclear! Keep doing your stuff, man!

[@MrDaFuxae]

Hi Nocholas, you did a great job there, thanks for sharing your knowledge! I would like to mention, that in my case I had a problem running the code, because I got a value error in the line "n_state, reward, done, info = env.step(action)". Adding a fifth value "observation" on the left side (so that it looks like "n_state, reward, done, info, observation = env.step(action)" got the code up and running :-) Nevertheless your videos are really helpful and please keep going! You're doing an amazing job!

[@FrancescoPalazzo26]

dude I can't import the agents and policies, basically keras doesn't have rl.policy or even rl.agents, what should I do?

[@Nobske]

# new version with terminated and truncated episodes = 10 for episode in range(1, episodes+1): state = env.reset() #initial for each episode terminated = False score = 0 while not terminated: env.render() # render the CartPole action = random.choice([0,1]) # 0,1 left or right observation, reward, terminated, truncated ,info = env.step(action) score+=reward #based on our step we get a reward till it's done print('Episode:{} Score:{}'.format(episode, score)) Docs observation (object) - this will be an element of the environment’s observation_space. This may, for instance, be a numpy array containing the positions and velocities of certain objects. reward (float) - The amount of reward returned as a result of taking the action. terminated (bool) - whether a terminal state (as defined under the MDP of the task) is reached. In this case further step() calls could return undefined results. truncated (bool) - whether a truncation condition outside the scope of the MDP is satisfied. Typically a timelimit, but could also be used to indicate agent physically going out of bounds. Can be used to end the episode prematurely before a terminal state is reached. info (dictionary) - info contains auxiliary diagnostic information (helpful for debugging, learning, and logging). This might, for instance, contain: metrics that describe the agent’s performance state, variables that are hidden from observations, or individual reward terms that are combined to produce the total reward. It also can contain information that distinguishes truncation and termination, however this is deprecated in favour of returning two booleans, and will be removed in a future version.

[@MrDaFuxae]

@@FrancescoPalazzo26 I had the same problem. In my case I could solve it by importing the modules from a different path, which is 'tensorflow.python'. so the import commands look Like 'from tensor flow.python.keras.models import sequential'. Hope this solves your problem!

[@ashkanforootan]

episodes = 10 for episode in range(1, episodes+1): state = env.reset() terminated = False score = 0 while not terminated: env.render() action = random.choice([0,1]) n_state, reward, terminated, truncated, info = env.step(action) score+=reward print('Episode:{} Score:{}'.format(episode, score)) run these for new versions

[@islam6916]

Thank you so much ❤ searched a lot for that kind of video and finally found a good one 👏

[@NicholasRenotte]

Thanks sooo much! There's some more reinforcement learning stuff coming this week, hopefully a video on Atari and (assuming my GPU doesn't catch fire) one on CARLA!

[@islam6916]

@@NicholasRenotte looking forward to seeing that

[@NicholasRenotte]

@@islam6916 awesome stuff!!

[@GeraLdario]

If you get "ValueError: Error when checking input: expected flatten_input to have shape (1, 4) but got array with shape (1, 2)", Install the package 'rl-agents==0.1.1'. It works for me.

[@amoghlakkanagavi10]

yoooo thanks mate

[@Patiencelad]

Great video. Thanks for explaining everything with the step by step. Excellent Job!

[@NicholasRenotte]

Anytime! A heap more rl videos coming, it's going to be a big focus this year!

[@omarsinno2774]

Really nice and simple explanation. Cheers!

[@NicholasRenotte]

YESS! Thanks @Omar, glad you enjoyed it!

[@sdtcuce]

Wow! such a wonderful lessons with practical example. I loved it. I want to learn more about self control action mechanism for multivariate industrial control using RL. Kindly put some light on it

[@NicholasRenotte]

Nice, got more RL stuff coming in the weeks coming @Suvankar!

[@siddharthmanumusic]

Thank you!!! You rock! Such a well made video! Short and fully informative.

[@Tprakh-iw6qt]

Very helpful but could not understand how to visualize the Cart Pole animation. Please let me know how to visualize it

[@Officialnorio]

Hey there! I am having the same issue with *'Sequential' object has no attribute '_compile_time_distribution_strategy'* but in my case *del model* doesn't help at all. If i want to delete it before *model = build_model(states, actions) I receive the error that I want to refer to a var before declaring it (which makes total sense to me xD). Any ideas how to fix this? :) btw. this video is amazing! Keep the good work up :)

[@NicholasRenotte]

Heya @Norio! Try deleting it then running the cell that creates the model again. I show it here: kzbin.info/www/bejne/nnTIe5inbbpjotE

[@Officialnorio]

@@NicholasRenotte didn't work for me. But thanks for your help :/ I now use Tensorforce and don't have any problems :D

[@NicholasRenotte]

@@Officialnorio awesome work! What did you think of Tensorforce, I checked it out earlier on but switched to stable baselines a little later on!

[@Officialnorio]

@@NicholasRenotte Sometimes my code threw some weird output but changing the agent-type fixed it. Tensorforce is pretty easy to use and does its job (so far) pretty well. I am using Tensorforce for my bachelor thesis about MTSP solutions :D

[@NicholasRenotte]

@@Officialnorio awesome, will need to give it a second chance!

[@BrunoVasco]

Thanks man! Nice pace and objectiveness.

[@NicholasRenotte]

Thanks so much @Bruno!

[@xinanwang9379]

Hi Nick, Thanks for your tutorial, it really helped me kick off the field of RL. There is an issue of the keras-rl2 package you used, specifically the NAFAgent, which fails all the time even using the example given in the official repo. Could you please spare some time and take a look at it? Many thanks and wish your channel gets better and better! Best, Tony

[@neelkanthbhavnagarwala6001]

When I try to run "dqn.fit(env,nb_steps...)" command I am getting ValueError : Error when checking input : expected flatten_2_input to have shape (1,4) but got array with (1,2) can you please help me out??

[@lahaale5840]

Nice introduction. It seems the DQN method is value-based even you are using BoltzmanQPolicy. BoltzmanQPolicy is like epsilon-greedy, a method to balance exploitation and exploration. Methods like DPG, PPO, A2C, and DDP can be considered as policy-based methods.

[@NicholasRenotte]

Thanks for tuning in @Laha, good to note!

[@whataday3910]

Hey! Thanks for the video. I would love to see how I can solve a problem with my own environment. Or how to build a specific environment and an agent with specific actions. I am at the moment not familiar with OpenAI but I think it would be interesting to see something more custom. :)

[@NicholasRenotte]

Heya @WhataDay, check this out: kzbin.info/www/bejne/mHWZh2aomNeSa5Y

[@muditrustagi5775]

great job man love from India !

[@gusinthecloud]

great job, ypu saved me a lot of time. Support from Argentina!!

[@文泽宇-x6p]

Hi Nick .Thanks for your tutorial it really helps me a lot.However , i am getting an error saying :"ValueError: Error when checking input: expected flatten_input to have shape (1, 4) but got array with shape (1, 2)",So ,i am wondering why this error didn't happen in your case

[@vietle6099]

I'm having the same issue

[@evolutionXXVII]

Did you ever find a solution to this issue? I'm having the same problem.

[@GeraLdario]

Install the package 'rl-agents==0.1.1'. It works for me.

[@sebastianrada4107]

@@GeraLdario It worked!

[@oneredpanda9933]

I keep running into errors because I don't have the right things downloaded. Ive been trying to to fix it for about an hour now and I can't figure it out! If anyone has done this more recent than 2020, and would be willing to help me I would greatly appreciate it. Thanks so much!

[@skippergiggletush8739]

The video could have been better, by talking a bit more about the input. But overall, it's a great video. Thank you for your time!

[@dineshkrishnasamy1628]

Nice content. We're waiting for ML trader series... thank you

[@Andy-rq6rq]

great tutorial! keep making more

[@NicholasRenotte]

Thanks so much @Andy, definitely plenty more coming!

[@choptran]

Thank you for such simple and easy to follow video. 🙏

[@NicholasRenotte]

Thanks so much @Chop, glad you enjoyed it!

[@Rose-ro7wz]

Thank you for the video, would you please make a video about DDPG?

[@rezagolipour9821]

Hi, thank you for the video. My question is : is there any specific reason behind you have installed Tensorflow 2.3.0? Can version 2.9.0 work without error?

[@jakes-dev1337]

Did ya try it? Try things.

[@edzme]

You're a great teacher thanks for making these!!

[@NicholasRenotte]

Thanks so much @Ed, glad you're enjoying them!

[@nanoluisi]

AttributeError: 'Sequential' object has no attribute '_compile_time_distribution_strategy' when i try dqn.compile, any idea? i tried copying the code itself but the error continues.

[@NicholasRenotte]

Heya @nn aa, definitely can help out!! Quick one, what version of tensorflow are you using? and how are you importing keras/tf.keras?

[@hninpannphyu8567]

@@NicholasRenotte First of all, thanks a lot for your great tutorial videos. i have got the exact same error as @nn aa. I am importing as below. my TensorFlow version is 2.3.1. Could you please take a look into it? Thanks.

[@hninpannphyu8567]

Would be great if you create the more RL learnings tutorials with custom environment rather than using OpenAI gym?

[@NicholasRenotte]

@@hninpannphyu8567 anytime! Can you try dropping the tensorflow. from your imports like so: OLD CODE: from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Flatten from tensorflow.keras.optimizers import Adam TEST CODE: from keras.models import Sequential from keras.layers import Dense, Flatten from keras.optimizers import Adam

[@NicholasRenotte]

@@hninpannphyu8567 I'm actually planning some more RL stuff soon. Anything in particular you'd like to see?

[@krvignesh6323]

Great tutorial Nic.. I was trying to implement this and encountered an error when I run the line "dqn.compile(Adam(lr=1e-3),metrics=['mae'] Error: 'Sequential' object has no attribute '_compile_time_distribution_strategy'. Can someone help me resolving this?

[@abulfahadsohail466]

ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none) ERROR: No matching distribution found for tensorflow this error is showing while instaling

[@tzhern]

short and clear! thanks a lot!

[@NicholasRenotte]

Thanks so much @Tom!

[@NingAaron]

Your video is amazing! You make learning RL fun! However, I have some questions, maybe some rookie-type questions, about the best strategy for reinforcement learning, can I just extract this part, such as a vehicle turning right at an intersection, and then turning left is its best path, Can I extract only this one path among many paths? Or is it possible to convert the results of RL into text? Does this RL training log include the actions selected for these trainings? Please take the time to take a look, thank you very much!

[@salmankhalildurrani]

can you please guide me solve the problem I am getting while working on this example TyperError: _set_agent() missing 1 required positional argument: 'agent'

[@wilamshields]

Thank you for replying to my previous comment on how out of date this code is, can not be used

[@InteliDey]

Hi Nicholas, why did you use "linear" as the activation function in your last layer instead of "softmax"? How would it differ if I choose "softmax" as activation function instead of linear for this case? Will it be possible to mention this, please? Or may be make a video on it? (When to choose linear and softmax activation function for what type of target cases)

[@xnyu254]

softmax is great for classification, but the experiment shown in the video is more of a regression problem. In this case, it makes more sense to use linear. Doesn't mean you can't use softmax, but your dqn will most likely don't work as you would expect it.

[@70ME3E]

@@xnyu254 here you have two states as the output (either go left or go right). It _is_ a classification problem, and not a regression one at all.

[@randomizer272]

Hello, Thanks for the great tutorial step by step video. Quick question. When I run dqn.fit(env, nb_steps = 50000, visualize = False, verbose = 1), I get this error: "'Sequential' object has no attribute '_compile_time_distribution_strategy'". How do I overcome this? and why did this happen? Thanks again

[@randomizer272]

I checked your other video. Deleting the model and reloading the kernel works. This comment is for anyone with same issues

[@NicholasRenotte]

Awesome work @Sriram, yep that's the solution the majority of the time!

[@ts717]

@@randomizer272 thank you for your answer, i have the same problem. But my knowledge is still quite limited so i don't know how i delete my model and reload the kernel. Would be nice if you could explain it a little bit more. Thanks in advance!

[@randomizer272]

@@ts717 You can just do a new line del model and create the model again. It worked for me fine. I will attach the video in which he explained about this error. kzbin.info/www/bejne/nnTIe5inbbpjotE

[@rodolpheredoute809]

in the section "def build_model" you have to change the line : model = Sequential() into : model = tensorflow.keras.models.Sequential() i checked and it seems that's because python can misinterpret it with keras, and not tensorflow's keras (but i have no clue why) this worked for me

[@pranavprasad5661]

@Nicholas Renotte This is such a well-explained video! Thanks for making it, I was looking for something exactly like this. I wanted to know whether you can make a video on custom environments using different types of observation_spaces and action_spaces (Discrete, Box, Dict, MultiDiscrete). I am trying this for a problem and I'm struggling a bit to understand how to use Dict and MultiDiscrete, most examples use Box and Discrete.

[@NicholasRenotte]

Thanks @Pranav, definitely will do! Got it on the list!

[@varunsharma7706]

It brings tears to my eyes😂😂 Awesome

[@NicholasRenotte]

Yesss! Thanks for checking it out @Varun!

[@evolutionXXVII]

FYI the pygame package needs to be installed for env.render() to work. Took me a little while to figure that one out.

[@mmc3790]

thank you!!!

[@McRookworst]

Great video! Got it up and running in no time. One question tough: What exactly does the value of 4 out of env.observation_space.shape[0] represent? Isn't the state supposed to be a pixel vector? Or is this some kind of abstraction openAI makes?

[@NicholasRenotte]

Heya @McRookworst, for CartPole we don't use a pixel vector (moreso used in the Atari envs). In CartPole the four values represented in from the observation space are: [position of cart, velocity of cart, angle of pole, rotation rate of pole].

[@RaviKumar-ub2ng]

Hi, what an amazing video! You are a great teacher and you make the learning of RL fun! However, I have some question and it might be some rookies type of questions because i am not that experience with python. You said that we can reload the trained model, but how can i do it in VSC? Create a new Python file and import the one we created? And also, when i run the " _ = dqn.test(env, nb_episodes=15, visualize=True)" and want to change episodes(just for testing), it has to go through the process all over again, but in your case it just used the rewards already generated and printed it right away. These questions might be so easy that maybe someone in the comments can provide an answer. Thanks :)

[@NicholasRenotte]

Should be able to reload the weights by running this when you open up again: dqn.load_weights('dqn_weights.h5f') Then to chaneg the number of episodes just change the number set to nb_episodes e.g. For 30 episodes run this: _ = dqn.test(env, nb_episodes=30, visualize=True) For 40 episodes run ghit: _ = dqn.test(env, nb_episodes=40, visualize=True)

[@fatemehkiaie7612]

Hi. Great Video. I am wondering to know if it is possible to create a custom environment?

[@NicholasRenotte]

Sure is! Check this out @Fatemeh: kzbin.info/www/bejne/mHWZh2aomNeSa5Y

[@abdelmalekdjamaa7691]

Hi 👋 Can you make a Q learning agent with just Keras and Tensorflow ? Creating the agent seems more interesting ⚡

[@NicholasRenotte]

Definitely, I've got the code 80% of the way there! Should be out in the coming weeks!

[@manishalifestyle7863]

Sir can you please help me in doing MountainCar-v0 and frozenLake as well because these not have same properties as cartpole

[@Bobstrer]

Hi Nicholas, Thank you so much for the great content! I'm running into an error "AttributeError: 'Sequential' object has no attribute '_compile_time_distribution_strategy'" I couldn't really find anything online to help me solve it, do you have any idea where this is from? thank you!

[@NicholasRenotte]

Heya @Olivier, try running del model, then rerunning the cell that creates your model.

[@adrianchervinchuk5632]

@@NicholasRenotte it worked for me, but why it acts in such a strange way?

[@NicholasRenotte]

@@adrianchervinchuk5632 I think there is conflict between tensorflow and keras. Seems to happy pretty frequently.

[@frankgiardina205]

I tried del model but i am getting an error in step 3 Keras symbolic inputs/outputs do not implement '__len__'. i researched in stack overflow and the answer was to downgrade to TF 1.14 don't want to do that. Any help greatly appreciated thanks

[@NicholasRenotte]

Heya can you try 2.3.1, that's the version I'm using in the video!

[@terrezwells9056]

Bro this is not working on my side I followed every step and still having problems

[@ahmedwaly9073]

Wooow this is an awesome tutorial

[@NicholasRenotte]

Thanks so much @Ahmed!! 🙏

[@thiennguyenthanh5004]

Thank you so much for this video!

[@NicholasRenotte]

Anytime!

[@igorperessinotto5774]

Great video! Quick question: how can I ask the model for the actions he took during the tests? Is there a way of getting a list or an array of all the left/right choices he makes?

[@NicholasRenotte]

I don't think it's available through Keras-RL but if you work with the environment directly you can get it based on the generated action. Want a video on it @Igor?

[@igorperessinotto5774]

@@NicholasRenotte I was able to improvise by making him print the action in the env.render(), so I use visualize = True when testing and it returns the action taken in each step. I guess it kinda works. Thanks!

[@NicholasRenotte]

@@igorperessinotto5774 awesome, glad you got a workaround!

[@sangeeth77]

ValueError: too many values to unpack (expected 4) can you please help

[@joaobentes8391]

i have the same exact erro can someone pls help! thanks

[@hocgh]

you can unpack only the first 4 values from the returned tuple and ignore the rest, so you can use the star: state, action, reward, next_state, *_ = env.step(action)

[@joaobentes8391]

@@hocgh i already got it thanks a lot!

[@abdikarimibrahim7078]

Thank you for the amazing content. I would like to know how to define multiple agents? As an example, if we have an environment with more than one agent taking action, how we can define the other agents and return their value?

[@NicholasRenotte]

Heya @Abrikarim, working on something in this space now. Will share it once it's ready :)

[@abdikarimibrahim7078]

@@NicholasRenotte Thank you. Can't wait for it, because am doing my PhD on multi-agent deep reinforcement learning, so am excited for this.

[@NicholasRenotte]

@@abdikarimibrahim7078 woah, awesome space to be working in!

[@abdikarimibrahim7078]

@@NicholasRenotte yeah, thanks alot

[@adrian46647]

had issues with rendering on m2 chip. Those pips helped !pip install tensorflow==2.13.0 !pip install gym==0.25.2 !pip install numpy==1.24.4 !pip install keras==2.10.0 !pip install keras-rl2==1.0.5 !pip uninstall protobuf -y & pip install protobuf==3.20.0 !pip install "gym[classic-control]" !pip install tflearn !pip install ipywidgets !pip install matplotlib pyglet !pip install pygame !pip install numpy --upgrade

[@frankgiardina1360]

Thanks Nicholas i use colab and will try TF 2.3.1

[@NicholasRenotte]

Awesome @Frank, let me know how you go!

[@hardikkamboj3528]

Hey Frank, I am also using Colab but facing a lot of difficulties in rendering gym. can you please help me with that @Frank

[@NicholasRenotte]

@@hardikkamboj3528 hmmm, I normally avoid colab for that reason. Rendering anything outside of the notebook is problematic. You can try training on Colab then rendering on your local machine.

[@hardikkamboj3528]

@@NicholasRenotte thanks a lot mate. I have started reinforcement learning, following your videos. Awesome work mate, really appreciate it

[@hardikkamboj3528]

@@NicholasRenotte I will try it

[@rchlam27]

I have an issue about "AttributeError: 'Sequential' object has no attribute '_compile_time_distribution_strategy'" when I am running dgn.compile. Any ideas?

[@NicholasRenotte]

Heya @Chi Hang Lam, try deleting the model using the code: del model then continue running the rest of the code!

[@BilalKhan-sx9eu]

Best crash course ever :D

[@NicholasRenotte]

Thanks sooo much @Bilal, check this out as well :) kzbin.info/www/bejne/g6bXkKhqZbiksJY

[@jaydevsinhzankat8872]

i am having one error in training part """AttributeError: 'Sequential' object has no attribute '_compile_time_distribution_strategy'"""

[@NicholasRenotte]

Heya, try deleting the model and then rerunning the cell e.g. del model then rerun the model creation cell.

[@jaydevsinhzankat8872]

@@NicholasRenotte it solved after sometime but thx for reply. 🤗🤗

[@niomartinez]

By anychance with the recent AI advancements the past year, is this still relevant or are there newer much easier way now? (I mean this is easy following though your video but there might be new technologies now that allows us to do this better? )

[@josefelgueiras8404]

Great tutorial. Is there a way to graphically visualize the convergence of the agent's scores throughout training? Or should I just use a longer method like alternatively training the agent for, say, 1000 steps and then use agent.test() to get some scores for validation. My goal here would basically be to have a robust method for training an agent while observing its progress throughout the training process.

[@NicholasRenotte]

Take a look at the full rl course on the channel, progress it output via Tensorboard.

[@junjieshi3900]

Geat video! As a starting learner, I have one stupid question here: how can we build a model that can hanld 2-dimensional state, e.g., recording several histirucal data, or even 3-dimensional state data?

[@peter-holzer-dev]

Amazing video, thanks! 🥳

[@abramswee]

great video! thanks for sharing.

[@pranavraut3998]

hello sir your video is really helpful ,but i m trying to run the code but at the time of pip install tensorflow it showing the error that ERROR: Could not find a version that satisfies the requirement tensorflow (from versions: none) ERROR: No matching distribution found for tensorflow please help

[@juliangiles8130]

try using a older version of python it worked for me

[@yasinsahin2962]

I also wanna study master degree in computer engineering as a control engineer graduated. My topic will be RL which includes control theory and machine learning. Any advice? It is a good path to go?

[@NicholasRenotte]

Sounds awesome! Check out Linear Programming and MIPs as well. This might kick you off rom an RL side kzbin.info/www/bejne/g6bXkKhqZbiksJY, there's also some amazing stuff from DeepMind.

[@navketan1965]

SEEKING YOUR WISDOM, SIR, I have 2 choices--1) buy one given forex pair every new day at the open at market price wth take profit(TP) 50 pips and stop loss(SL)50 pips on one ticket as one order 2) second choice is to buy same pair but order is placed as pending order at the open of every new day--buy the same forex pair 100 pips below the open price as pending order with take profit of 50 pips and stop loss of 50 pips all on the same ticket.And after one year of every day trading which strategy is more likely to make any money?And what answer SUPERCOMPUTER would give to my question?

[@alimashoud6294]

Hy! Nice video, How would we do it in PyTorch? Please make a video on deep Reinforcement Learning with PyTorch as well. Cheers!

[@alishafii9141]

I like you, your video and your teach. keep go on

[@saurrav3801]

Bro can you make a detailed video and explanation of making chatbot using reinforcement learning

[@btkb1427]

Hey thanks for the vid! it's great, I get the error AttributeError: 'Sequential' object has no attribute '_compile_time_distribution_strategy' when trying dqn.compile(...) I have the same version of tensorflow as you

[@NicholasRenotte]

Heya @Bartlomiej, delete your model (i.e. del model) and then rerun the code and it should clear it up!

[@vagnermartin4356]

@@NicholasRenotte I also have this same problem. But when you say to delete your model. It is to add the code "env.close ()", because if that didn't work.

[@dralbertus]

Hello! I still don't now why, but I solve this issue writing `del model` a cell below `def build_agent(model, actions):` and then ` model = build_model(states, actions)`. Regards!!

[@NicholasRenotte]

@@vagnermartin4356 nope use 'del model', here'a an example where I do it: kzbin.info/www/bejne/nnTIe5inbbpjotE I think there is conflict between Keras andtf.keras versions perhaps but this seems to resolve the error.

[@NicholasRenotte]

Thanks @@dralbertus!

[@richchizzl5020]

Great video! Can you explain the target model update + the 10_000 displayed by Keras in the verbosity? It seems like, the target model still updates every 10.000 steps, even though the target model update was set to a soft rate of 0.001. What am I missing? :D

[@NicholasRenotte]

Heya @Richchizzl, this is due to a DQN being an off-policy reinforcement learning algorithm. When you train, there are actually two models being trained in tandem, one is consistently being trained based on the latest values (think of this one as a fast learner) the other is being used to generate the next set of states (think of this one as a slow learner). Every 10,000 steps the model weights from the fast learner are copied over the the slow learner, this helps ensure you get more reliable training.

[@richchizzl5020]

Thank's a lot for your reply! Do you know, if it's possible to change the mark of 10,000 steps within the keras framework? I'm trying to implement the connectx kaggle challenge and it seems like that a slowly increasing reward is completely shaken after the update of the slow learner...

[@NicholasRenotte]

@@richchizzl5020 hmmm, did you try changing that parameter? TBH, I'm now using stable baselines over kerasl-rl as it actually seems, well, a lot more stable. You can set the target model update frequency for a DQN pretty easily using the target_network_update_freq parameter: stable-baselines.readthedocs.io/en/master/modules/dqn.html I did a bit of a crash course on setting up experiments with it here: kzbin.info/www/bejne/pIOrm6yji5eDjpo could swap out the algorithm used there for a DQN and set the paramater there.

[@mohammadrezazavvarsabegh6663]

Great Video. If I want to change the max reward (200) to 100, which line should I change?

[@NicholasRenotte]

With the current code base it's a little tricky but you could do it with a custom environment, check this out: kzbin.info/www/bejne/mHWZh2aomNeSa5Y

[@mohammadrezazavvarsabegh6663]

@@NicholasRenotte Thank you so much

[@NicholasRenotte]

@@mohammadrezazavvarsabegh6663 anytime, let me know how you go!

[@mrjzkhan]

I am stuck at !pip install keras-rl2 Once I run this line the whole thing gets stuck, and shows "Kernel busy" status. Shows no errors. And I can't run any code after that.

[@NicholasRenotte]

Heya @Jajanzeb, can you try stopping the kernel, then running the pip install at a command line? It might just be hanging.

[@Sarah-lr6vp]

Hi, I am stuck at the line dqn.compile(Adam(lr=1e-3), metrics= ['mae']) I am getting an error 'Sequential' object has no attribute '_compile_time_distribution_strategy' Any help would be appreciated, thanks!

[@NicholasRenotte]

Heya @Sarah, can you try deleting the model and reinitialising the variable? It normally clears up the error.

[@Sarah-lr6vp]

@@NicholasRenotte okay, thanks! My friend suggested me to make another file and just copy the required methods for the final function, that worked too

[@NicholasRenotte]

@@Sarah-lr6vp awesome work!

[@Sarah-lr6vp]

@@NicholasRenotte thank you!!

[@yalcinimeryuz5414]

Great video! However, I am getting an error saying "TypeError: only integer scalar arrays can be converted to a scalar index" on the line "model.add(Flatten(input_shape=(1, states)))". How do I solve this?

[@thiagobastani6663]

The variable states is probably not an integer scalar array. are you testing the same eviroment?

[@mrwallstreet3438]

Hi Nicholas thanks again for your videos , i have learn and start coding , with your help , since 3 to 4 weeks now ...., and i am loving it ... thanks ....can you do a videos with capsule net .... greetings God bless you Bro

[@NicholasRenotte]

Woah, thanks so much! Awesome to hear about your journey, any particular use case you were looking at with a capsule net?

[@mrwallstreet3438]

@@NicholasRenotte You are welcome Nicholas , all the credit is to you , because you explain us ,what you are doing and why you are doing it , so thanks again :-)) Regarding the capsule Net , can we use this example ( lucenaresearch.com/2019/07/15/capsule-networks-deep-learning-for-stock-forecasting/ ) for time series classification just for BUY and SELL pictures from TRADING CHART with the OS and CV2 importation folder. Looking forward for your reply .... Greetings

[@NicholasRenotte]

@@mrwallstreet3438 definitely, might need to do some research into it. But I'll take a look!

[@mrwallstreet3438]

@@NicholasRenotte , thanks very much , looking forward to that , since then , i will keep learning with your videos , thanks again , bro !!!

[@JohnGriffith1]

I'm trying to run your notebook in a docker on a Windows host. Any idea how to get it to render the plots in the notebook? For instance, step 1 seems to work, but how do I see the visualization? In the video, you go to another window to display. How does that work? Thanks for the tutorial!

[@NicholasRenotte]

Heya @John, it looks like it can be done with a virtual display but I haven't tested this out myself. Check this out: towardsdatascience.com/rendering-openai-gym-envs-on-binder-and-google-colab-536f99391cc7

[@NicholasRenotte]

I'm running it directly on a bare metal installation, the visualisation component automatically pops up in a new screen when the code is run. I'm going to look into rendering on a cloud or non-local environment this week as I'm keen to run it non-locally for a few clients.

[@khalidbinhida]

Excellent sir!

[@goktugozleyen9766]

Super video but i have a question. First i tried but i got an error then I copy your code and paste it. But i still got an error. Error : FailedPreconditionError: Could not find variable dense_5/kernel. This could mean that the variable has been deleted. In TF1, it can also mean the variable is uninitialized. Debug info: container=localhost, status=Not found: Resource localhost/dense_5/kernel/N10tensorflow3VarE does not exist.

[@goktugozleyen9766]

I solved it. Use tensorflow.keras instead of keras

[@muhammadattaurrahman1365]

Model output "Tensor("dense_2/BiasAdd:0", shape=(?, 2), dtype=float32)" has invalid shape. DQN expects a model that has one dimension for each action, in this case 2. How to fix this issue sir?

[@Guskiller1]

Wow, this was very interesting. Great video. I've been interested in trying to use Deep-RL on android games. Do you know how one could go about this? I was thinking of using screenshots as inputs to the DQN. i'd have to create a custom environment, right? Is this something you are familiar with? Thanks again for the video.

[@NicholasRenotte]

Yo @Gustavo Lorentz! Ya, you'd need a custom environment. You could also try out some of the pre-built games from OpenAI though as a kick start, LMK if you want me to make a video on doing it with games! Also, looks like there's a ton of third party envs you could use as a baseline though:github.com/openai/gym/blob/master/docs/environments.md#third-party-environments

[@Guskiller1]

@@NicholasRenotte that sounds like a good video, I'd watch it! I'll try building my own custom environment. Thanks for answering :)

[@NicholasRenotte]

@@Guskiller1 anytime, I've got it on the list for some future videos now!

[@MaximePerrain]

Hi Nicholas, thanks for all your greats video. i've problem with this line of code : model = build_model(states, actions) only integer scalar arrays can be converted to a scalar index and Error converting shape to a TensorShape: only integer scalar arrays can be converted to a scalar index. do you have an idea of what can be the issue?

[@NicholasRenotte]

If you sample your states what does it look like? Are they non-integer values?