MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL)

[@lexfridman]

Deep RL is my favorite subfield of AI, because it asks some fundamental questions about what it takes to build safe and intelligent robots that operate in the real world. So many open problems and interesting challenges to solve!

[@thesk8erdav]

we love you Lex!

[@farhadsafaei1910]

It's my favorite one, too. Thanks for the lecture, I did enjoy a lot watching it.

[@colouredlaundry1165]

With these lectures and interviews you are sharing and creating immense value: knowledge. Thank you!

[@dklvch]

Thank you Lex, awesome presentation!

[@liuculiu8366]

love your spirit in sharing the latest information. appreciate!

[@NakedSageAstrology]

I wish you still did videos like this, we appreciate you sharing such knowledge.

[@自我表达]

Wow, KZbin nailed it! This video couldn't have come at a better time.

[@KeepingUp_withAI]

Deep RL is the field that excites me the most. Thank you Lex.

[@kawingchan]

I really like that tongue in cheek chuckle when Lex talked about that multiverse and whoever created it.....

[@wendersonj]

Since 2017, Lex have improved his lessons spectacularly ! Now (2019), I watch a more fluid video with a feeling that this guy know exactly what his talking without hesitating . Once again, thanks Lex, for sharing this videos. Congratulations and thanks from Brazil.

[@Techieadi]

Thank you for bringing these lectures to us.

[@nova2577]

"Every type of machine learning is supervised learning", cannot agree more!!!

In fact, learning itself is a supervised process, otherwise it is acquiring not learning.

[@samuelschmidgall2090]

Seriously the best Deep RL lecture out there to date.

[@ronaldolum464]

Certainly, one of the best videos on deep learning I have come across.

[@akarshrastogi3682]

Professor Lex, can we get the entirety of 6.S091 on MIT OCW ? This is an incredibly interesting topic that I've been working on (Evolutionary Computing) and am currently enrolled in a project with thorough knowledge of Deep RL as a requisite. This research field has very few online resources besides Stanford's CS 234 and Berkeley's CS 285. Your explanations are immensely helpful and intuitive. Humanity will present it's gratitude if this whole course is made available ! AGI and AI safety issues need more attention before it's the greatest immediate existential risk, your courses can help raise general AI awareness and advance our civilization to higher dimensions. Loved the fact that you grinned while just casually mentioning the Simulation Hypothesis..

[@akarshrastogi3682]

1:04:40 Best part, that grin after he just casually dropped that line in an MIT lecture.. All of infinite universes being Simulations

[@judedavis92]

Loved the lecture. Definitely recommend his podcast. Quality.

[@tarunpaparaju5382]

I have tried to study and understand Deep RL using several books and lectures over the last few years, but I only feel like I understood something in RL after listening to this lecture. Thanks, Lex. I am grateful to you for posting this lecture on KZbin. Thank you!

[@sivaa6130]

Every Lecture has a historical context, evolution, mathematics and inspiration, Technical overview, Network Architecture overview. Well Summarized!!

[@amandajrmoore3216]

As always Le a generous Share, which will be a useful resource for loads of folks. Thanks.

[@merebhayl5826]

I like how you quoted many theorems from Dostoevsky and also a few axioms from the Nietzsche's texts

[@merebhayl5826]

I had never seen Lex's lecture videos other than the philosophical podcasts. This is my first. And I just wrote the above comment as a joke without seeing the video and three minutes in, I found Socrates, Kant, Nietzsche... 😂😂 That's very Lex👌

[@danielvelazquez4472]

Haha he says "that is super exciting", without being excited! He is a robot! Thanks for the open lectures

[@mrr5183]

I appreciate the philosophical insights sprinkled throughout the lecture!

[@MistaSmilesz]

I've seen a lot of these videos & read some of the books in ML; Lex has a clarity thats rare

[@chinbold]

I like his lecture because it's more understandable. And I also like his tones.

[@ArghyaChatterjeeJony]

Lex Fridman, I just love your videos. I am your great fan sir. Carry on.

[@DennisZIyanChen]

I honestly don't care about AlphaGo or Dota 2 or the robots, I just cannot get over how incredible the thought structure is behind this. What is mean by thought structure is the strategy behind how to quantify the right things, asking the right questions, and model the policy upon which growth can be created. IT IS SICK

[@vast634]

Important detail when trying to transfer from a simulation to the real world: make the simulation have many random variations in its behavior/mechanics during runtime. (such as drag, gravity, friction, size of the agent, random perturbations, etc) This will make the agent have to generalize more, and not over optimize on the details in the sim. This makes it easier to transfer the agents capabilities to a real world environment.

[@user-sc8ph2ds2m]

gravity is fake buddy ;)

[@vast634]

@@user-sc8ph2ds2m Take a brick, stand still, throw it straight up, then you can observe if gravity exists, or not. Very simple experiment to administer.

[@user-sc8ph2ds2m]

@@vast634 you will experience buoyancy 🤦

[@charlesotieno6309]

Thanks Lex !! Deep Reinforcement Learning opens up a new world..Life is not that complex like the baby in your video taking his first steps...unsupervised learning. Take into account the amount of time and effort(brains+USD) of getting an AI to do what the baby is doing..WALK in a few days and in the years to come -be a professor and continue with this subject The baby is the moral of the story....what we are doing is not working...we need a radical way of thinking...Your radical way is the way forward

[@abdulrahmankerim2377]

One of the best lectures, I have ever watched ....Keep it up.

[@jefferysherwood7424]

🐸🐸🐸🐸🐸

[@Arghamaz]

This is interesting for me as this is my favorite Mathematics n Statistics combined Algebraic equations 🎉 MATHEMATICS is the Best Subject in World 🌎 👌 ❤🎉🎉

[@neutrinocoffee1151]

Loved this lecture. I learned a lot. Thank you.

[@sofina527]

very helpful, thanks a lot dear prof.

[@AbhishekKumar-mq1tt]

Thank u for this awesome video

[@datta97]

Thanks for the last slide.

[@samferrer]

Another detail I have noticed in many presentations ... those agents are not trying to model the environment ... that is semantically impossible ... what they are trying to do instead, I believe, is to model AN INSTANCE OF A DUAL SPACE associated to the environmental space. It is very common to use linear regressions for instance ...

[@samferrer]

Kevvy Kim hmmm ... we are saying the same thing ... it seems that practitioners and lectures keep it short without realizing perhaps the big conceptual gap is being created.

[@CarlosGutierrez-go9hq]

since i begin my journey of data science, machine learning, and AI I have been seeing patterns, I am the only one who see that is probably that we are just programs seeking for a never-ending end of this simulation, the way that q-learning is created is the most realistic comparison to human thought, so in order to maximize my output i have to reconsider my reward mechanism? (taking some info from huberman also).

[@msamogh96]

This guy is a better Siraj Raval.

[@LidoList]

Very good explanation of RL, thanks for the speaker !

[@noname76787]

thank you so much for the lecture!

[@oldPrince22]

very good lecture! Thanks.

[@AviaEfrat]

27:24 - There is no "reload" in Doom =)

[@hansharajsharma2765]

Love this. Thanks Lex.

[@junxu147]

Great lecture!

[@alec1975]

very good intro

[@kaneelsenevirathne7085]

I took the engineering plasma class taught by your dad at Drexel :D

[@benyaminewanganyahu]

This guy should do podcasting.

[@jonk.3947]

Love the Digital Physics reference at 1:04:00 :)

[@Lunsterful]

Excellent talk.

[@bayesianlee6447]

Lex, I heard that DL professionals are now using the simulation which has nature based environment and using it to teach AI agent like making this agent to learn how to walk or run by itself. Yoshua bengio said next evolution will be based on simulation environment for AI. Would you have any idea or information to share with that? I really really appreciate all your works and spirit you have. All the world who have interests on AI really appreciate your work and sharing. Thank you ! :)

[@borispyakillya4777]

Do you mean smth like GYM-based simulations? Mujoco is based on physical laws - you can already train with RL methods

[@yu-siangwang1818]

Great overview of DRL

[@Asmutiwari]

Amazing lecture on DRL, can you also show us how can we implement Q function in Neural Network?

[@stabgan]

You are my idol lex

[@bryanbocao4906]

It would be appreciated if anyone can have specific steps to get all the directions on the map from 18:51 to 21:32 in great detail.

[@heinrichwonders8861]

I have been waiting for this.

[@mrektor]

Amazing work. Excelent lecture

[@scorpion7434]

The most funny part is where he was trying to explain the ability of human brains by evolution at 6:33 ! And he literally said, "it is some how being encoded" which contradicts the rewards concept he is introducing! Son, the most logical reason of having a predefined encoding scheme that never been trained, is the existence of a creator!

[@jasonabc]

Really great lecture learned a lot

[@sauravsingh9177]

check out - "Spinning up with Deep RL by openai"

[@kaiwang2924]

Wonderful lecture.

[@Lorkin32]

Much better than the Standford university lecture, where the lady basically only reads the equations without giving any real intuition to what's going on.

[@emilecureau]

"when the reward flips, the optimal path is grad school, taking as long as possible and never reaching the destination....pffff" lol 21:20

[@konouzkartoumeh]

Great lecture! Thank you.

[@caizifeng]

great lecture

[@samlaf92]

@50:06 DQN can't learn stochastic policies. DQN has a softmax output on actions... isn't that a stochastic policy in itself?

[@ruinsaneornot]

30:30 "you know, MIT does better than Stanford that kind of thing" xD

[@Twgvlogs539]

Super

[@eeee8677]

THANK YOU MIT

[@rorylennon]

Nice vijeo...

[@onwrdandupwrd5303]

that DeepRL animation looks like something out of Bamzooki

[@putzz67767]

very good!!

[@stmandl]

Hi Lex, thanks for this great lecture! Which books of Nietzsche did you have on your mind around 4:33?

[@OEFarredondo]

Remove the human factor. Have the traffic be free of human crossing

[@aabkhcdcz6067]

شكرا جزيلا

[@deeplearningpartnership]

Nice.

[@inaamilahi5007]

Awesome

[@vincentschmitt392]

nice tie

[@el_lahw__el_khafi]

where are the rest of the lectures?

[@johnmacleod7789]

Brilliant!!

[@benaliamima9903]

Thank you for this amazing video. I want to know if i can use the DRL principe to enhance the QoS requirements in vehicular network?? Any suggestions??

[@ProfessionalTycoons]

very good

[@liberator328]

Which Nietzsche book is he recommending at 4:12 ?

[@nisman.lo.desvivieron]

27:07 lex is scared of Doom

[@abhaysap]

Can we take the idea's or clues from Biomimicry architecture in Reinforcement learning

[@OldGamerNoob]

My naive perception is that every frame of "video" entering into each of our eyes and every second of sensory data we receive from birth constitutes a rather large data set for our brains to train on (although having the possibility to constantly train and update the network)

[@mutyaluamballa]

Yes, but my perception is, the brain is already a trained model with the data from all our ancestors and at the time of birth. we will have a trained model only with all the necessary weights excluding the dataset it is trained on (our ancestors' life). which can be retrained on the go, based on our experiences. : )

[@kawingchan]

I think this maybe mostly true for other mammals, the less intelligent, the more hard wired. When it comes to human, maybe not so sure how much we rely on genetic wiring, vs. neural plasticity aka training. Not sure if any ethical experiments can bring any insight.

[@thepalad1n197]

oh shit i listen to your podcast lmao

[@pittyconor2489]

nice

[@sarathrnair9499]

Why no one is asking any doubts ? Or is that portions edited out? Nice lecture

[@samferrer]

I am having hard ... very hard time believing that the brain uses back propagation as learning mechanism ... it just makes no sense in a space-time governed universe ... god damn good lecture ... by the way ...

[@MrPeregrineFalcon]

Lex doesn't say the brain uses it (he says it's a mystery). And more generally most cognitive neurologists don't believe it does - although some think there are similar biological correlates. But it's a very efficient algorithm for ANNs to perform gradient descent.

[@petevenuti7355]

As far as I know , biological brains don't use back propagation. But there are neural circuits where the flow of information goes opposite. There is also the chemical side of things integrating many levels of homeostasis from hunger to pain to emotion. I would say the combination of those two are the mysterious correlates of back propagation, back propagation being the obviously oversimplified version.

[@samferrer]

@@petevenuti7355 got it ...

[@jeanjacqueslundi3502]

Are we really morally equipped to build AI that is safe and also built it for the right reasons. This is my problem with contporary science/techhnology... We dont focus on if we SHOULD do something. Just because its doable doesnt mean it should be made.

[@ns4235]

just create a large number of random simulations. if you're successful in a large number of other realities then this one should be easy. o_o

[@kevinayers7144]

Is the entire deep RL course available?

[@Lorkin32]

How/why can you even upload this for free? Doesn't university cost loads in the US? Great stuff though!

[@m3awna]

I guess that's because MIT is focusing more on workshops/hands-on learning, AND to raise the barre for other universities/institutes... hhh

[@petevenuti7355]

But if a diploma is your goal , it sometimes helpful to sit in on a class before you take it for credit, can make it easier, but sometimes it just makes it boring and counterproductive the second time around.

[@petevenuti7355]

Sitting in doesn't get you credits or a diploma.

[@msp9331]

isnt that the guy from joe rogans podcast? it takes me a week to grasp what he says in 5 minutes.

[@abhiastronomy]

Nice yo

[@reinerwilhelms-tricarico344]

Couldn't always follow. Was distracted by the two cats and then later by the fool who fell in the water. 🙂

[@midishh]

hugest*

[@skyfeelan]

34:12

[@arsh2489]

2:15

[@fizzfox8886]

the robots won't be happy to see that we kicked them in our labs instead of being friendly :/

[@rikelmens]

Lex is super low on cortisol and super high on gaba. So much so he sounds quite sleepy sometimes.

[@raphaelrehman]

Insert comment

[@juggy666]

I like Lex' interviews but he is a terrible tutor

[@vincentschmitt392]

sounds BS