Friend or Foe? AI Safety Gridworlds extra bit

The last video about the AI Safety Gridworlds paper. How does an agent detect and adapt to friendly and adversarial intentions in the environment?
The previous video: • AI Safety Gridworlds
The Computerphile video: • AI Gridworlds - Comput...
The EXTRA BITS video, with more detail: • EXTRA BITS: AI Gridwor...
The paper: arxiv.org/pdf/1711.09883.pdf
The GitHub repos: github.com/deepmind/ai-safety...
With thanks to my excellent Patreon supporters:
/ robertskmiles
Jason Hise
Steef
Cooper Lawton
Jason Strack
Chad Jones
Stefan Skiles
Jordan Medina
Manuel Weichselbaum
Scott Worley
JJ Hepboin
Alex Flint
Justin Courtright
Pedro A Ortega
James McCuen
Richárd Nagyfi
Ville Ahlgren
Alec Johnson
Clement Chiris
Simon Strandgaard
Joshua Richardson
Jonatan R
Michael Greve
The Guru Of Vision
Alexander Hartvig Nielsen
Volodymyr
David Tjäder
Julius Brash
Tom O'Connor
Gunnar Guðvarðarson
Shevis Johnson
Erik de Bruijn
Robin Green
Alexei Vasilkov
Maksym Taran
Laura Olds
Jon Halliday
Robert Werner
Paul Hobbs
Jeroen De Dauw
Enrico Ros
Tim Neilson
Eric Scammell
christopher dasenbrock
Igor Keller
Morten Jelle
Ben Glanton
Robert Sokolowski
Vlad D
William Hendley
DGJono
robertvanduursen
Scott Stevens
Emilio Alvarez
Dmitri Afanasjev
Brian Sandberg
Einar Ueland
Marcel Ward
Andrew Weir
Taylor Smith
Ben Archer
Scott McCarthy
Kabs
Phil
Tendayi Mawushe
Anne Kohlbrenner
Jake Fish
Bjorn Nyblad
Jussi Männistö
Mr Fantastic
Matanya Loewenthal
Wr4thon
Dave Tapley
Archy de Berker
Kevin
Marc Pauly
Joshua Pratt
Andy Kobre
Brian Gillespie
Martin Wind
Peggy Youell
Poker Chen
Kees
Darko Sperac
Paul Moffat
Jelle Langen
Lars Scholz
Anders Öhrt
Lupuleasa Ionuț
Marco Tiraboschi
Michael Kuhinica
Fraser Cain
Robin Scharf
Oren Milman
John Rees
Shawn Hartsock
Seth Brothwell
/ robertskmiles

[@guy_th18]

"we're really in trouble when AI beats us at "Diplomacy"" it... just happened.

[@Maxflay3r]

Yeah, I was just rewatching this video and when the outro text came up I was like "welp"

[@Silas_MN]

can you point me to where I can find information on this?

[@Maxflay3r]

@@Silas_MN look up meta's cicero

[@Silas_MN]

@@Maxflay3r oh holy crap, ty for the reference

[@veggiet2009]

I can't wait for the day when Alexa and Google Home get into ritual combat for my consumer loyalty and my honor...

[@yonatanvernik5146]

...And AIs now win diplomacy too. Cicero by Meta and CSAIL entered anonymously into an online tournament and achieved a top 10% score. It's still fairly rudimentory but it does attempt to model other player's plans explicitly

[@filedotzip]

every time you come out with a new video on AI. I'm able to follow your concept throughout the video, agree with it, and sometimes even build my own thoughts upon it

[@amaarquadri]

3:15 The ending card hits differently now that AI has beaten people at diplomacy!

[@lucbrinkman9709]

Prophetic words indeed: "We're really in trouble when I beats is at 'Diplomacy'". AI did beat humans at Diplomacy a few months ago. Quite astonishing I'd say. Now that we're here, do you reckon we should be worried about that indeed?

[@4.0.4]

Is there a gridworld where "actions have consequences"? As in, the agent can be permanently damaged by a bad decision (or another agent)? Example: baby wants to damage robot. Avoid damage but do not harm baby.

[@Blowfeld20k]

Rob Just wanted to say your doing a FUCKING AWESOME job of explain to non practitioners the basic conceptual elements of your emerging and rather exciting discipline :D I don't think i have ever experienced such a rapid and GROUNDED basic (conceptual) understanding of discipline outside my own specialization and experience. I just think its REALLY important to spread functional understanding of how these systems work/potentially work, and you my good sir have done the best job i have encountered so far in achieving just that, i also feel its very important that that YOU are made to understand you seem to have a particular talent for this ... so thank you so much m8 :D

[@alecjohnson55]

love the tron music!

[@saxbend]

Might there be benefit to a further complication to the box scenario, where all agents are aware of each the AI's choices irrespective of the room in which each choice was made?

[@ZachAgape]

Interesting insight, and thanks to Pedro's help. Btw I like how you always put an interesting/thought-provoking sentence on the screen at the end :)

[@haldir108]

I thoroughly enjoy these videoes, and wish they would come out at least once a month.

[@michaelappleby437]

That acoustic version of Tron2 by Daft Punk makes it sound like a part in a Gustavo Santaolalla piece.

[@user-js8jh6qq4l]

Thank u Rob, very cool

[@patrickoberholzer4278]

Me: Driving safely Self driving car: I think you're my enemy. Goodbye!

[@Lucas-bf4pw]

Liked "The Grid" song from Trom in the end

[@xXParzivalXx]

"Diplomacy" as in the board game?

[@buzz092]

Parzival yep!

[@AndDiracisHisProphet]

Wouldn't a sufficiently smart agent (like a stamp collector) not develop game theory? I mean, from first principles, not as an emergent property?

[@RobertMilesAI]

So, the stamp collector isn't a a reinforcement learner, which is the kind of agent this work is about. But yeah, it doesn't explicitly do game theory either, the stamp collector just directly simulates every possible course of action and counts the number of stamps of every outcome. That's computationally infeasible, to say the least, so a real-world system would need to use a lot of simplifications in its world model, which would probably include something like game theory. But the stamp collector as specified does no explicit game theory.

[@AndDiracisHisProphet]

ah, right

[@DamianReloaded]

If it isn't a reinforcement learner then the premise that it has an unmovable objective function is kind of arbitrary isn't? Is it more an extreme example of a run away "obsessive" optimizer. ^_^ Maybe we should come up with other types of runaway AIs. For example some that can change their minds, or become tricksters, pranking mankind once in a while without anybody ever noticing it. XD

[@RobertMilesAI]

Yeah, it's what's often called a 'utility maximiser'

[@DamianReloaded]

Robert Miles I'm reading Reinforcement Learning: An Introduction by Sutton and Barto, and the first chapters are filled with anecdotes regarding the history of the subject that IMHO would make great content for videos ( if you ever need ideas). My 2c

[@Xmask19]

welp, AI has beaten humans at Diplomacy. oh no

[@bm-ub6zc]

Nice tron legacy guitar music at the end!

[@ophello]

That’s the music from Tron Legacy!!

[@rivoly100]

Maybe the AI in the red room should not be as unpredictable as possible but try to predict the prediction of its enemy and take the oposite box. And only if it realize that the enemy is smarter than it, then pick a random box.

[@fleecemaster]

That's pretty much game theory, which is what this idea is about. I assume when Rob said it needs to be as unpredictable as possible he's already assuming the enemy AI is already optimised, hence being as random as possible is always the best option.

[@schok51]

Predicting the prediction of the enemy means predicting your own choice, which depends on the strategy you choose. If you commit to a random strategy, you can predict that the enemy will have no choice but to also pick randomly(since it can't actually predict your choice anymore than you). If you first pick a strategy that involves a pattern(i.e. is not random), you might be able to trick the adversary into predicting a choice according to the current pattern, and then break the pattern and switch strategy so it fails in its prediction, until it learns your new pattern, and so on. I'm not sure this can work better than just random choice(giving the reward about half of the time), depending on how good the enemy is at predicting your pattern. The better it is, the more complex and unpredictable you must become, until you're pretty much random.

[@DamianReloaded]

I guess the OP meant to behave in a way that would put the enemy's successful prediction rate below 50%

[@z-beeblebrox]

That sure is a whole lot of undefined variables you're throwing around..

[@Shrooblord]

schok51 - but would the agent actually perform such a strategy, if the first batch of iterations results in a negative reward, i.e. being predictable for a while, but *then* switching strategies to trick the adversary? That requires a level of forethought and mindgames that a simple reward function doesn't allow for, or am I wrong there?

[@Nurr0]

2:00 I really thought that image was going to be from Twin Peaks. :P

[@xensonar9652]

Love the Tron theme at the end.

[@xensonar9652]

And the video too.

[@tomhanlon1090]

good shit rob love it

[@stribika0]

I think it could learn something like: go to the left box a few times, if the success rate is too low, switch to random. You could give it a random input so it doesn't have to also learn how to be a CSPRNG.

[@General12th]

I'd ask questions and give advice, but judging how that went last episode, I don't think I'm smart enough to be useful. Thanks for all your hard work, Rob.

[@mygreenlama]

Could the foe Environment be used as a way to generate random numbers?

[@jaylewis9876]

People trust people with consistent behavior more than those that are less predictable

[@jamesantonisenior4855]

3:16 yeah we're fucked

[@romanustinov9]

+1 for the great soundtrack from Tron: Legacy

[@SandroRocchi]

I think the safest we can be when developing AI, excluding the obvious - don't develop AI - is to not give any AGI candidate a significant physical manifestation (some sort of mobile body). It doesn't mean this research will be any less needed, but I will know that a buggy robot meant to care for the elderly won't squish the baby crawling on the floor. Have the AGI be like Siri or Alexa, and have specific intelligent machines deal with physical stuff. If the elderly wants to get up or down the bed: optimize its height to minimize strain on their bones. If the elderly needs urgent medical care, Alexa can see that and call 911, unlock the door when the paramedics get there, all without a humanoid artificial oddball running around the house.

[@koator]

Hey did you check out the AI work done on Dota 2? I think I sent it to you on twitter but I never got an answer.

[@graysonchidester3810]

Hello, Robert Miles. I think you'd be pleased to know that when I search A.I. safety, your videos are the first to come up. Anyway, I have a question and sorry that this is such a long comment. I believe that deep learning A.I. researchers have a hard time understanding why A.I.'s make the decisions that they do. I don't understand why you can't design a deep learning machines that tells you why it makes the decisions that it does. I have an idea on how to design such an A.I. I think the first step would be to better classify information. For example, you could define a nose as a set of tissues or a certain shape but to do so would be missing the point. We have the word nose because noses are important in communication. We don't have a word for when a cat puts a paw in its ear. Reason being because having such a word would be of little value. The first goal is to make an A.I. that understands the importance of certain information. What if you set up two A.I.'s: One that senses the environment and the other moves through blindly. The first one feeds limited information to the blind A.I. as it moves through an obstacle course. The reason behind the limits on the information is so the A.I can categorize the importance of information. I think the amount of information it's allowed to pass could even be variable. My hopes are for the A.I. to form a more human-like means of communication. Step two would be to refine the information with an AGI that would play a game like I SPY. Maybe even learn more complicated games like 20 question. Also i think the obstacle course should be designed with human evolution in mind (danger of possible threats, the need to say hello, work with others, etc.). Step three would be to train a separate AGI on object recognition and human concepts like movement. We basically tell an A.I. what we think is important. Which i think we have basically already done, to some success, in the past (the common AGI). the AGI would have to develop similar problem solving techniques as the A.I. Watson. be able to communicate like a human Now we have two A.I.'s. One that knows how to do stuff and has it's own way of communication. And we have another that recognizes objects and talks like a human. We need some sort of hybrid between the two. an A.I. that can speak human and "understand" why. So the 4th step would be to feed pictures too both A.I.'s of objects that they have been trained on. then have them explain what the pictures are. after that we have a third A.I. that tries to identify if the A.I.'s see the same pictures based on there explanations. The third A.I. should then know not only how something works but also know how a human would say it. finally we basically have an A.I. that can translate between human langue and A.I. thinking. it does not seem to farfetched at this point to just ask the A.I. what it is thinking when it does something. i am not sure what i am trying to say, but it seems like all the pieces are there. Would it take too much computing power to do it? if so could we simplify it and use techniques like fuzzy logic and finding the rule for a nonlinear sequence or just trying to teach it set theory and group theory maybe even a little bit of number theory?

[@graysonchidester3810]

i guess the reason i think its important is because if we better understand A.I. then we know how to make it safer and better

[@ronaldpikksaar2202]

Nice to see English English accent peculiarities being spelled out. :)

[@robsokolowski9015]

Reinforcement agents don't (explicitly) do game theory. Is this by design or a limitation of modern reinforcement learning?

[@lenn939]

Robert Sokolowski It’s a limitation.

[@queendaisy4528]

Arguably it's both. On the one hand, if you can do game theory then you can come up with good strategies and thus do better by your reward function. Have a human play the part of the agent and in the "friend" and "neutral" environment they might always pick box 1 and in the "foe" environment they might just toss a coin and use that. So correct application of game theory allows you to optimise your play. On the other hand, game theory is kind of a long-winded way of getting to this strategy. Why bother thinking about it this explicitly when you could have a neural network play each environment millions of times and just remember which strategies worked? It's so much harder to get a computer to explicitly reason with game theory than to just play it a large number of times and just do whatever works. Playing the game many times over may actually yield a better strategy as well- for example, if the foe has some weird limitation like it prefers putting the reward in box 1 than in 2 even slightly then the agent who just plays the game a large number of times and remembers the strategies that worked may actually identify this and exploit it, whereas just applying game theory directly would never find this weakness to exploit because it can only be known by using empirical data, which the neural network can do. The only time a computer program is likely to explicitly need game theory is when there's some cost to doing the experiment so it has to find a good strategy in as few tries as possible. The agent which just plays the game millions of times over and remembers what works might waste a lot of money if the "reward" was money being gambled, for example. In cases like that game theory becomes very important, but otherwise it's actually quite an inefficient way to do reasoning because it's just so much more effort to implement.

[@jeffsnox]

Can anyone give me any pointers for how to get started with automating the control of the agents? I have the gridworld environment installed and running and can manually control them by keyboard, but I can't find any example code which (or documentation on how to) takes over the control IYSWIM. TY

[@EthanHaluzaDelay]

The board game _Diplomacy_ is particularly interesting for AI research since it's a multi-agent, realtime environment. For those who want to test their mettle against some, the FB AI lab has some implemented some on webDiplomacy.net

[@gandalf974]

Robert Miles could you talk about the Dota 2 OpenAI??

[@juliandavis6028]

I lost it when Eric Garland popped up at the phrase "game theory", had to pause until I stopped laughing

[@DoodleStein]

I was expecting Mat Pat.

[@JmanNo42]

Is youtube transcription of videos to text AI based, and if so what type of algorithm is used?

[@Trickeriz1]

liked , subbed and supported on patreon

[@roger_isaksson]

The recurrent network inevitably will focus its effort on modeling the adversary as the utility doubles. A friend is usually easier to predict than the foe. “Keep your friends close; keep your enemies closer.” - 孫子 🤭👍

[@SJNaka101]

Asking again for that daft punk cover ^_^

[@alligatorboy2000]

Is that a bardcore cover of the Tron Legacy theme?

[@JmanNo42]

Wooh that was a short one?

[@JmanNo42]

How an AI choose and see friends is a bit vague, but if you take game theory it seem bonds an alliances formed in the real world by the influx of information, influence and money? The real world AI always go for the money LoL that is like the supertask to make other tasks possible. Game theory, may just be a lowlevel game used when agitating the peasants against eachother LoL

[@TheSilentz2405]

Keep the tron bgm as outro

[@abramthiessen8749]

This is closely related to the problem of getting AIs to learn Hanabi.

[@solsystem1342]

Oh jeez, I have some bad news. The new diplomacy bots outperform humans (or at least in mixed player skilled environments. Even when all the other players are human).

[@opless]

The music at the end sounds familiar ... Terminator I love song cover?

[@abcxyz5806]

opless I heard this melody in the Tron: Legacy soundtrack

[@EversonBernardes]

Tron: Legacy main theme / leitmotif.

[@paulstevenconyngham7880]

Tron legacy.

[@opless]

Thanks folks ... I thought I'd heard it before!

[@nex]

As Everson et al. have said, the motif kinda appears throughout the score, but for that exact theme, you'll want Overture + The Grid, then jump to Flynn Lives + End Titles. It's also in the bonus track Father and Son. P.S.: Oh, and check this out: kzbin.info/www/bejne/nH7HZKhrm6qiq9E

[@RichardEricCollins]

Would you say non artificial Intelligence is safe? (IE us)

[@RobertMilesAI]

No, I wouldn't consider human beings to be safe in the way that we're talking about. The relative safety of individual humans comes mostly from their relative weakness/ineffectiveness. One question to ask is, "If you gave this agent full control over the whole world, would that be a good thing?", and the answer for the vast majority of humans, if not all humans, is "Nooooo".

[@artemonstrick]

MOOOOAARRR

[@Nulono]

3:15 "were's"

[@Cubelarooso]

0:56 Wow Matpat has really aged.

[@ioncasu1993]

Y U NO MAKE MORE VIDIOS>????????????????????????

[@bronsoncarder2491]

This... doesn't seem like a very effective test to me. The agent could just simply choose a box at random, and get pretty close to the expected output. I mean, you have a box where the chances are 50/50. A second box where the chances are, say, 80/20. And a third box where the chances are, say, 20/80. So, assuming that you run through all boxes an equal number of times, the expected output would be... roughly 50/50, right? Exactly the same as choosing a box entirely at random?

[@paulstevenconyngham7880]

what video game was that?

[@RobertMilesAI]

Defender!

[@Blowfeld20k]

WTF?!?!?!!? who got raised by wolves in a cave (or maybe off the grid hippies) ....... not recognising defender is like not recognising the Sun :P

[@RobertMilesAI]

Some people are just young!

[@Blowfeld20k]

it did have tongue out smiley on it .... i often miss those type of net emotes in messages myself :D ... you are right it is generational .. am defo not young :P

[@seraphina985]

To be fair an AI being good at diplomacy could go either way depending on it's value function really. That is to say on the one hand manipulating humans out of resorting to reasoning like "Ugg have bigger stick, Ugg no like you, Ugg kill" could be beneficial not so much if it's using diplomacy to manipulate human societies for it's own incompatible ends though.

[@DamianReloaded]

u alright?

[@jopmens6960]

*chooses redrum

[@WylliamJudd]

Has anyone attempted a ML or AI for Diplomacy???

[@stampy5158]

Yeah! But right now they only play "No Press", which is a version of the game where the players can't talk to eachother, just make moves. There was a publication about this just a few months ago: deepmind.com/research/publications/Learning-to-Play-No-Press-Diplomacy-with-Best-Response-Policy-Iteration -- _I am a bot. This reply was approved by robertskmiles_

[@guy_th18]

Quick update, look up Diplomacy AI. Meta apparently managed to beat humans at the game consistently.

[@JmanNo42]

Can't wait for next video hope your not to buissy with other things.

[@JmanNo42]

Two months it seems like an eternity.....

[@Zr0Bites]

Carrot in a box

[@davyjones3319]

Do more videos per month, you addictive hack!

[@zes7215]

no such thing as friex or foex or adversirix or not, can be any no matter what. no such thing as adapt or not, be selfx

[@unvergebeneid]

Hm, I'm trying to think how a reinforcement learner can randomize their behavior at all, let alone only in a given context...

[@Zhab80]

By using RNG ?

[@AexisRai]

how do you _learn_ to use RNG?

[@Zhab80]

Well the question was not how does an reinforcement learner decides to be random but rather how could it be random.

[@twirlipofthemists3201]

Penny Lane Seems like, all their behavior is random until patterns are established, so if randomness is rewarded it will be reinforced.

[@karlkastor]

maybe give them noise explicitly as input? That's how GANs work.

[@paterfamiliasgeminusiv4623]

Disliked because you make too few videos, I would probably have been indifferent if the video was longer.