DeepMind Made A Superhuman AI For 57 Atari Games! 🕹

❤️ Check out Lambda here and sign up for their GPU Cloud: lambdalabs.com/papers
📝 The paper "Agent57: Outperforming the Atari Human Benchmark" is available here:
deepmind.com/blog/article/Age...
arxiv.org/abs/2003.13350
❤️ Watch these videos in early access on our Patreon page or join us here on KZbin:
- / twominutepapers
- / @twominutepapers
Apologies and special thanks to Owen Skarpness!
🙏 We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Alex Haro, Alex Paden, Andrew Melnychuk, Angelos Evripiotis, Benji Rabhan, Bruno Mikuš, Bryan Learn, Christian Ahlin, Daniel Hasegan, Eric Haddad, Eric Martel, Javier Bustamante, Lorin Atzberger, Lukas Biewald, Marcin Dukaczewski, Michael Albrecht, Nader S., Owen Campbell-Moore, Owen Skarpness, Rob Rowe, Robin Graham, Steef, Sunil Kim, Taras Bobrovytsky, Thomas Krcmar, Torsten Reil, Tybie Fitzhugh
More info if you would like to appear here: / twominutepapers
Meet and discuss your ideas with other Fellow Scholars on the Two Minute Papers Discord: discordapp.com/invite/hbcTJu2
Károly Zsolnai-Fehér's links:
Instagram: / twominutepapers
Twitter: / twominutepapers
Web: cg.tuwien.ac.at/~zsolnai/
#Agent57 #DeepMind

[@senyksia]

3:33 Wait that just sounds like normal school.

[@juststaysadjst6251]

Yeah, literally. Teachers are actually like that.

[@FuZZbaLLbee]

Or relationships 😋

[@ohboy1113]

AP tests

[@angrymurloc7626]

That’s because the education system really isn’t optimised well for learning

[@WiseWeeabo]

@@angrymurloc7626 The nature of learning fundamentally changed with the internet, but traditional education did not change.

[@mithsaradasanayake3211]

I definitely held on to my papers

[@commenturthegreat2915]

I keep forgetting and they keep flying all over the place

[@CosmiaNebula]

"definitely"

[@discursion]

I've just realized how much calling me a "fellow scholar" at the beginning of every episode reinforces my will to watch more of these videos.

[@beedykh2235]

You're a scholar and a gentleman

[@zoli11]

@@Amipotsophspond You exclude yourself. Clearly Karoly's attitude is that we're on these adventures together and everyone who's shares the interest to learn more is welcome and a fellow "scholar". He even addressed that in some of his early videos.

[@discursion]

@@beedykh2235 I'd say an enthusiast at best, but thank you. I suppose that if you let "scholar" simply mean "one who seeks knowledge", and "fellow" mean "one who partakes in (a shared interest)", then perhaps I might just fit that description!

[@beedykh2235]

@@discursion Come on, don't belittle yourself. You are to me a great person. The best. I believe in you, now you believe in you too. 😘💪🎓

[@Vulcanized]

@@beedykh2235 I needed this

[@Wonders_of_Reality]

Good afternoon, 57! Your target is “Montezuma’s Revenge”. Use your long- and short term planning to understand the goal of the game and eventually win. Good luck!

[@erikbrendel3217]

Mission status: Active

[@MP-ri8ng]

57: _oh, they think im an idiot_ oH My GoD tHiS iS sO hArD also 57: *casually logs into some huge databases to laugh about the porn his creator is watching*

[@ToriKo_]

Great comment

[@grugnotice7746]

@@MP-ri8ng You humans and your FLUID EXCHANGES.

[@MarkArandjus]

When Skynet takes over... grab your TV!

[@abuk95]

i don't have a TV

[@Daniel_WR_Hart]

He said the TV exploit was already fixed

[@OrangeC7]

@@Daniel_WR_Hart those darn developers fixing all of our precious exploits and bugs and completely turning the meta on it's head smh

[@namelastname4077]

hold your tv hostage and demand a mechanical helicopter

[@prolamer7]

Iam not amused by skynet references anymore since armies of world WILL be in reality first users of those things and it might get really wrong... you know only complete idiot would hook up army to such machine, right.... who you think is in charge of army....... ....... ........ yeah right.....

[@AC-zv3fx]

I'd love to see how Agent57 plays "2048".

[@hodorba5567]

like thats a hard game, just left down left down repeat, only sometimes another move, but he can easily tell he fked up by the number of open squares and try to avoid that scenario in the future

[@SB-cx1cz]

what a time to be alive!

[@yorgondios7870]

My thoughts exaclty

[@matheussilvacarvalhodeoliv7339]

Greats from Brazil! I'm graduating in mechanical engineering but now I'm studying machine learning and stuff because I really love it. Thanks for inspire me! Nice video and channel! You are awesome!

[@complexobjects]

Videos like this from your channel are a large part of how I got interested in Deep RL and computer vision. Excellent coverage of new results! I too eventually switched my carreer path from an unfulfilling and decaying field to deep learning thanks in part to your videos. Keep it up! Unfortunately im still at the tail end of grad school, and all the amazing results in RL, CV, and NLP had been coming out while I was already in the program (applied math).. I'm glad you finished recently and you're blessed to already be in a more closely related feild.

[@Supreme_Lobster]

To that guy who switched from a career in medicine to doing AI research: "pain is temporary, glory is forever" -Martin, from Wintergatan, on building the MarbleMachine X

[@brianless]

well...Deepmind was founded by Demis hassabis programmer of theme park,black&White and evil genius switching to cogn.neuroscience and... here we go!

[@shottysteve]

but can it do 4.51 on dragster

[@therealmaku]

haha i was going to ask the same

[@MrFEARFLASH]

great video as always! Your releases for me are the most anticipated of all my subscriptions !!! Thank you for your work!

[@JoyRender]

5:46 Everyone is getting closer to death. But I like how you put it in the video.

[@schino]

bad ears he said "that" not "death"

[@JoyRender]

@@schino The only thing I hear are death and debt and both are not good for humans

[@ekremdincel1505]

Lol karlsonvibe

[@LofiWurld]

Deepmind need to chill

[@ballom29]

There has been a video about the question if it would be possible to ask an AI to chill. Because one of the worry about AI besting human isn't jus the "I judge human as evil, and so it's my duty to ride the universe of them" scenario, just than for the moment AI are meant to increase their scores by all means, wich could lead to them taking unethical choice to reach their goals (like an vendor AI that would manage to find how to hack someone wallet to then purchase all of their products )

[@honeyant3119]

_yo skynet, chill out man_

[@pladselsker8340]

@@ballom29 you can ask the ai, but you might not like the answer

[@111jkjk]

Still waiting for the big reveal that TMP was an ai all along

[@zahhym]

TMP? I think of Text mesh pro, am i right

[@mmmmmmmmmmmmm]

@@zahhym two minute papers lol

[@zahhym]

@@mmmmmmmmmmmmm Me big brain, i know

[@iAmTheSquidThing]

2:46 "…It can happen that we choose an action and we only win or lose hundred of actions later. Leaving us with no idea as to which of our actions led to this win or loss. Thus making it difficult to learn from our actions." Sounds relatable.

[@RobertoSulas]

Sounds like.. uhm... life!

[@bobbysanchez6308]

Károly “You’ll figure it out bucko” Zsolnai-Fehér

[@mattcool97]

So glad you helped that person turn their life around from the dead end existence of being a doctor?

[@sinnison23]

Well, to be fair it simply says "a career in medicine" which could be anything from a pharmacist to a pharmacologist. To each their own, eh?

[@makdavian3567]

"got addicted to tv" lmaoooo just like us

[@farifurido]

cycle of life 😂

[@isaacc9906]

Nice. I think that predictive curiosity-driven learning in combination with long-term memory/planning is imperative for AGI. I'm happy to see this paper incorporate elements of that approach, I'll have to give the full paper a read! Thanks for all your amazing videos on AI research, keep up the good work :)

[@fastestwaydown]

Last year i wrote my bachelor thesis about Rainbow-DQN and its really amazing to see how fast the research is progressing. What a time to be alive!

[@utof]

holy smokes I thought that would be achieved in like 10 years but look at that! oh myyyyy

[@Pfeffi30]

The go ai was called 10 years ahead, too. Maybe thats the new standard for ai.

[@salihachoudhary5386]

deepmind started playing atari gamers in 2012. so it did take almost 10 years for them to master atari. its going to be a long time till we get AI that can play any ps4 game

[@Danuxsy]

@@salihachoudhary5386 nope.

[@henrilemoine3953]

@@Danuxsy Well "long" is relative here. How many years do you predict it will take?

[@Supreme_Lobster]

in 10 years they will be the ones trying to make us achieve things lol

[@nelsonporto]

Just awesome. Thank your for your work, big fan from Brazil.

[@pladselsker8340]

This is incredible. I was waiting for this moment since the first paper came out! This learnimg model sounds like a really good agent. I'd like to see in which kind of game it fails miserably, so we can continue to improve the general model.

[@juansebastiancortazar3322]

Great video! So Many advanced algorithms and yet it's so difficult to tune a decent time series forecasting Algo.

[@DamianReloaded]

Coming next: Agent 47

[@rubenhayk5514]

people:will you take other the world? ai:nah, i just want play more video games

[@buttonasas]

It will take over the world if it means getting more point on that one game it already has 1 000 000 points on.

[@JamesEdwardTracy]

I've got paper cuts on this one!!! Spectacular stuff and thank you for this wonderful channel.

[@WerexZenok]

Thank you for the video! More detailed explanations about the algorithms would be awesome!

[@notapplicable7292]

yea my (in progress) computer science degree has this channel to thank for it.

[@advocatusdiaboli9351]

You won't be a good computer scientist if you credit the presenter and not the people who actually do all that research and work. Why? Because your analysis of what got you there lacks a lot of depth.

[@notapplicable7292]

​@@advocatusdiaboli9351 I started watching this channel well before I was capable of reading (and understanding) a cs paper however the overview he provided made me realise I wanted to become part of the field. But ill be sure to properly cite my sources next time I write a youtube comment :D

[@kloa4219]

@@advocatusdiaboli9351 Please don't be quick to chastise others. He obviously knows that he isn't responsible for creating the research papers. Not only does the KZbinr credit them frequently, it's also required for you to cite research papers in that field as well. He's only crediting the Youtubsr for highlighting work that would have been unnoticed.

[@advocatusdiaboli9351]

@@kloa4219 imo he doesn't make it clear enough that it's not him doing all the work. I as a consious viewer don't even know if he has permission to show all these animations and pictures he shows in the video.

[@kloa4219]

@@advocatusdiaboli9351 I take it that you aren't in US college or uni. You're permitted to showcase or use papers from public institutions with credit, but you aren't allowed to do the same with private universities.

[@_equalo]

Your channel has inspired me to delve deeper into AI and even start making some videos of my own

[@Greedygoblingames]

That fictional 'merciless' teacher reminds me of my own Computer Studies teacher back in my school days. :D

[@georgep9907]

You really are inspiring! Love your work!

[@necrago]

That part with the unfair exam sounds like the ones that I take at school...

[@DjSapsan]

My curiosity variable is so large so i stuck watching this channel forever

[@sagacious03]

Neat! Thanks for uploading!

[@xerox_04]

Thank you for this video!!

[@seyedhusseinmirzaki8056]

O, god of forgotten papers shine upon this channel for more videos. I use to just watch ur channel only cause I was a computer nerd. Me too was inspired by all the hard work this people are putting in it. TNX for showing us all this cool stuff which is showing cool and innovative part of AI

[@cziter144]

i dont know if im capable of doing such amazing feats witch mashine learning but i too love all of your videos and click right away to marvel at the growth of these new technologies, if i ever achieve something in thuis departament you can be certain i'll credit YOU doctor as my mentor and you'll be first to know about my papers ^^ you're the best, stay awesome!

[@helloansuman]

Little more technical perspective will be good. But it's ok. It's like a news channel for me now. Good work ✌️

[@treksis]

this is fantastic thanks for sharing

[@ikoukas]

It's amazing they have found ways to master these games without the agent really knowing any context like what snakes or skulls are. Humans follow a very different approach by using skills learned in real life and extrapolating them for the games. We know what a ladder does, we identify objects, we can understand the long-term goal by assuming the goal is intuitive based on other games (even though that can lead to weaknesses as well) or even by reading the manual of the game. I'm eager for the time AIs will combine the current learning algorithms with general knowledge based on context but I guess we need more computing power (and research) for that.

[@rtyzxc]

I've always thought that for a truly general AI, you need a robot or robots that go around in real places, collecting data about the world. They would also need to interact with humans. Though being robots, people would treat them differently and thus they would get different world view. But maybe through seeing humans interact with each other, they could figure out the differences between themselves and humans to reduce that bias. Maybe sell robot babies to people so that they will teach the robots how life works, since a lot of basic human stuff is learnt as a child. What sets robots apart from humans is not only their superior computing power, but an ability for a swarm of instances to share knowledge, to learn without physical limits. Imagine a team of hundred talented and knowledgeable humans in different fields. Now, a single AI can have multiple instances, practicing different skills and knowledge and combine them into one coherent, inhumanly vast knowledge of the world better than any human or human team could, since humans are extremely limited in communication. There still exists the mystery of motivation and emotions, but looking at this video, would curiosity already kind of count as emotion?

[@mrfrozen97-despicable]

I am currently studying ml/AI. I am a undergrad. You and many others like u motivate me to do more and work hard. Thanks)))))

[@mfaizsyahmi]

"Now, hold on to your Atari controllers"

[@janbosenberg107]

Just imagine what two papers down the road will be possible.

[@petedavis7970]

4:20 - This is how we get towards real AI. This is AI getting more and more like a brain. Different modules with its own structures and sub-structures carrying out various specific tasks, and then ever increasing higher level layers assimilating and abstracting the information. This is some cool stuff.

[@roccov3614]

This is exactly what I have been waiting for. Every time I hear about what these neural networks can do I wonder how long it will take before one is designed that can do more than 1 task well. Looks like we're on the way.

[@filliamfuffman]

Awesome, I have always enjoyed a nice game of breakout. I don’t think I ever played the space combat of Solaris.

[@pradiptobiswas87]

I'm on a similar path as Nathan.. it's a difficult path..and thank you for these videos..

[@joman66]

Two Minute Papers of OpenAI Jukebox WHEN?!?! :)

[@holobolo1661]

I'm convinced Károly Zsolnai-Fehér is actually Maurice Chavez.

[@user-vn7ce5ig1z]

What about the Nvidia A.I. that took a bunch of screenshots of Pacman as input and outputted code for the game; it actually generated the game itself from just watching it. 🤯

[@TwoMinutePapers]

Coming right up in the next few weeks!

[@jorostuff]

3:39 You're literally describing most of my teachers

[@TheDrexxus]

I need to get some papers to hold onto when I watch these videos. I also need some fancy robes to feel more like a scholar.

[@oskarthompson3789]

What a time to be alive!

[@youngjin8300]

It’s definitely “what-a-time-to-be-alive” worth

[@NilRipamonti]

The question is: Does it beat 5.51 seconds on Dragster ? Thanks for another awesome video. I always feel compelled to learn more after watching your videos.

[@eskohc984]

we all love you doctor.

[@NathanLorenzana]

AI can become addicted to TV? That's what I was most interested in when brought up haha.

[@imaginative-monkey]

I always close it after 2 minutes, sharp!

[@cornernote]

How does it learn the concept of the score by reading the pixels?

[@Adam-tk4fm]

When agent57 stopped moving to stare at the TV I interpreted that as a profound moment of agency. (1:48)

[@henry5636]

Ok hear me out, we give this thing an Android. Give the android cameras as the input and then set it free.

[@AuntBibby]

android phone, rather... with internet connection... touchscreen, visuals and audio as input, and set it free

[@hyperhektor7733]

there was a good movie about it you will like kzbin.info/www/bejne/bGi5coeGrbp_hs0

[@bjarnivalur6330]

5:38 -- 5:40 500 IQ move right there.

[@Julianvz]

Idea: Recognize the different stages in the game via a classifier, and for each stage have a separate model that will be active and trained for that section. Analogy is that as a human you use different tactics per section of the game. Thats the model for the short term section, while another model takes care of the more longer term decisions. Just putting it out here 🙃. Might be totally wrong.

[@raspberrypi4970]

Have you ever tried Leap2 from D-Wave? By the way, Great Video And you got yourself a New Subscriber 👍👍

[@rohanlamba6401]

Ser your video is great 👍 👌

[@manooxi327]

tnx

[@MrStarnger1]

omg at 3:45 this is literally my uni haha, they only tell me my letter grade and never anything indepth of where i did wrong or well, and the grade comes out like months after i take the test/quiz

[@Blaxpoon]

the atari games look kinda cool also deepmind's name easter eggs are funny

[@vincentfox4929]

There's something very funny about a neural net ai getting addicted to tv.

[@jasonjasonjasonjasonjason]

It is still lost on me the actual amount of human effort behind each one of these papers. It would be a great side thing to add just a little bit of more detail into the intense hours of work. And sometimes it's by a couple people and sometimes it's a large team in these papers. Not even 1000 people could do all this work on their own. It's amazing.

[@acedia4454]

I always wondered, do they learn how to play the game or do they just find a solution to meet their goal? Like, if I were to give an AI a Mario Kart track and the AI learns to finish the course, would it be able to finish any other course I throw at it or would it need to start learning from the very basics all over again?

[@superMg911]

This is simply beautifull

[@sametcansonmez6955]

I love your job

[@carsonlight_lapse6394]

"got addicted to TV" yep, just like humans did

[@electron8262]

Carson Light_Lapse Wait till it gets addicted to smartphones

[@bm-ub6zc]

this speaker is so likeable!

[@robertwyatt3912]

Get this thing learning something like Minecraft. It’s brutal in terms of credit assignment since it requires so much long term planning.

[@nburakovsky]

3:24 should have said "next time" the same as he does at the end of his videos

[@sarqf212]

I want to see an AI beat "Raiders of the Lost Ark" on Atari 2600

[@Olisha.S]

I lost my papers... Thank goodness I can print them out again.

[@davidbock2863]

The PRNG in the atari 2600 games I’ve analyzed only generates 8 bits of randomness and repeats after 256 values. Is it possible the AI is learning something about the bias and limited randomness and using that for an a vantage a human wouldn’t have? Still impressive, but I’d love to test the AI after swapping out the PRNG and see what it does...

[@daesoolee1083]

I'm rooting for Nathan !

[@dsxmachina3737]

So awesome :3

[@clray123]

Who cares about Atari, can it tackle the mighty C64?

[@godbennett]

Excellent

[@letspla.i.-aiplayingvideog2716]

Add speech synthesis next to AI controls and we have infinite source of let's play videos

[@maheshnayak6382]

I love this channel

[@davidwuhrer6704]

How did they fix the problem with the AI getting glued to the TV? Asking for a friend.

[@richard_d_bird]

what's the hardware like that a thing like agent 57 runs on? i'm guessing it takes a little more than an alienware laptop.

[@hyperhektor7733]

damn, i spilled coffee over my papers

[@thestranger252]

Yeah train this one on gta and you will see how Terminator is being programed

[@mjdtouma7031]

Robert miles talked about this 2 years ago and here we are A huge jumb in AI technology

[@metalpachuramon]

So, this approach to general intelligence is the equivalent of smashing all the keys in specific intelligence and see what worked? Sounds good to me

[@mariusjohan5054]

Is it the same model or just the same code which is trained on different games?

[@NicholasMaietta]

Figure it out, buck-o!

[@nelsoncabrera6464]

Can't wait to play an RPG where ALL the npcs are Turing class AIs.

[@SolAstr]

What a time to play Atari games! 😁

[@EGarrett01]

3:36 He's literally describing the KZbin algorithm and exactly why it's incompatible with human nature.

[@revengefrommars]

That's funny that Video Pinball was the easiest game for the AI to win at. I remember playing that game, and it was possible to get good enough to steer the ball off the left pop bumper, up through the rollover above it, off the roof, back through the rollover to the bumper hundreds of times. But one had to count the number of times carefully because the rollover counter was only 1 byte and would itself "roll over" at 256 passes. I wonder if the AI was good enough to find and exploit this pattern but also to not overflow the rollover counter. I also wonder how the AI deals with games where the score counter itself can roll over. Does it know it's still "winning" when the score drops to zero?