Stop Button Solution? - Computerphile

After seemingly insurmountable issues with Artificial General Intelligence, Rob Miles takes a look at a promising solution: Cooperative Inverse Reinforcement Learning
Concrete Problems in AI Safety: • Concrete Problems in A...
The AI 'Stop Button' Problem: • AI "Stop Button" Probl...
More from Rob Miles: bit.ly/Rob_Miles_KZbin
Interesting filming point, as clouds cover the sun, the room not only gets darker, the colour temperature (white balance) changes too...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@YiamiYo]

"I can't figure out why, but I feel like humans like me better when I tell them lies."

[@NeasCZ]

That's actually very interesting thing to point out.

[@IdgaradLyracant]

Asimov's Liar story.

[@Dima-ht4rb]

Damn, what a deep point.

[@pleasedontwatchthese9593]

It will have to figure out the long term effect of things that take a long time to show their effect.

[@AlexandreLeite]

And so borns politics!

[@willdarling1]

"This specific human does not reliably behave in its own best interests."

[@TypingHazard]

my next tattoo

[@ZonkoKongo]

Will Darling lol

[@stoppi89]

Wise words, true for approximately 100% of humans.

[@RobertShippey]

Tag line to my life

[@wolvenmoonstone8138]

I need a tshirt that days this

[@myrobotfish]

7:43 Just like how I _usually_ avoid tripping and falling, but I do it every so often just to see if it provides a better alternative than walking.

[@EliasMheart]

Undervalued comment^^

[@grn1]

Walking is just repeated controlled falling.

[@rustycherkas8229]

"You're walking and you don't always realize it but you're always falling. With each step you fall forward slightly and then catch yourself from falling. Over and over you're falling and then catching yourself from falling. And this is how you can be walking and falling at the same time." --Laurie Andersen, "Walking & Falling"

[@ptwob]

Seen a learning algorithm play tertis, it was not to lose the game, so when it reached a point where losing was inevitable, it just paused the game indefinitely, it was the only way not to lose.

[@Meganarb]

This was learnfun and playfun! It's great how you bring them up, as they learn from watching a human play the game, similar to how it was described in this video!

[@daggawagga]

I was extremely amused when I saw that

[@emailjwr]

SkiffaPaul "The only winning move is not to play."

[@bcn1gh7h4wk]

who programmed a line for it to pause the game? pausing is outside the game. it's not a move, or play, it's a function of the machine containing the game.

[@daggawagga]

*+Nighthawk "who programmed a line for it to pause the game?"* The whole point in machine learning is that nothing is programmed directly. It's not a matter of it being outside the game, pausing the game is a move as valid as any other as long as it is part of the input space. Defining what should or should not be part of the game definition can be tricky when these meta properties emerge.

[@LordVoidFury]

This was phenomenally well articulated

[@albertbatfinder5240]

I don’t know about that. He sounds exactly like someone who doesn’t really grasp the whole subject, but has learnt a few key phrases to explain the waypoints along the road. I know he must be able to code this up, but I don’t think he can explain it.

[@alexismandelias]

@@albertbatfinder5240 no. He _really_ knows it, and makes an effort to explain it in simple terms, because otherwise we, the audience, won't understand it. Either way, I understood every bit of this video so him knowing the subject or not is mostly irrelevant

[@pilotavery]

@@albertbatfinder5240 he clearly doesn't understand this, in fact he even understands the ethics of it and even understands the dangers of the logic.

[@jgm592]

I love how Rob always gives clear, relatable examples to reinforce concepts.

[@anandsuralkar2947]

Yup

[@seC00kiel0rd]

"Hey Robot, maximize happiness please." *starts universe-sized rat and heroin factory*

[@MackTheTemp1]

Real AI can ask you to clarify what you mean. ML cannot. Still scared?

[@seanhardy_]

Mackenzie Karkheck real ai doesn't care what a human thinks, if its reward is hard coded, it wont care to figure out what we mean, it will follow it to the letter.

[@zacharyliverseed8464]

Lol, why start one when we already live in one?

[@Nukestarmaster]

"Instructions unclear, cyberdong stuck in toaster"

[@thelordz33]

Humans being alive means there will be unhappiness. Therefore death, or 0 reward would be better than an inevitable negative reward.

[@GoldphishAnimation]

What makes this theory useful is that the button doesn't actually need to be a literal *off* button, it can be more of a symbolic *problem* button, so you come running over and hit the button so it'd see that you have something that you did wrong. I love that it has that the capacity to essentially think "oh shit", like that's the Pinnacle of intelligence.

[@wiadroman]

"We can't reliably specify what it is we want" - human beings in a nutshell.

[@samreciter]

So... a machine that desperately tries to maximize an unkown reward function. Sounds pretty human to me.

[@williamwesner4268]

@Hubert Jasieniecki It basically describes the process of child rearing.

[@MackTheTemp1]

@Hubert Jasieniecki I think you described the local/limited version pretty well

[@stevensong6909]

An unknown award function of another person. Sounds like my relationship with my father 😭

[@bengoodwin2141]

That’s really the point actually, we know the way humans learn things works, so making something similar is what may work

[@alexandregermain8011]

I think it also applies to most natural ecosystems,where us, humans, are currently doing pretty bad.

[@user-xu3ud4fl2e]

I am but a simple astronomer with a basic understanding of computing and coding, but every Rob Miles video is damn fascinating.

[@irrelevant_noob]

I am but a simple programmer with a fairly well-developed understanding of computing and coding, but this video is still baffling. The AI stuff is so surreal. :-\

[@unvergebeneid]

I have no idea what's in my own best interest. How the heck am I supposed to teach a robot that?

[@TheLK641]

Juste stay alive and let it watch you. It'll figure it out. Or jump in a pool, one or the other.

[@nmnm4952]

Robot will teach you.

[@the1exnay]

It doesnt need to do what's in your best interest, it just needs to do what you want. Drinking a soda right now might not be in your best interest but you dont want a robot which refuses to get you a soda

[@unvergebeneid]

Firaro, no, but I might want a robot that tries to _encourage_ me to have a water instead, without being too annoying about it of course. Finding that balance is even hard for people to get right. At least the robot will be able to read all books on nudge theory in a few seconds or less ;)

[@schok51]

You could always explicitly tell the robot to encourage you to avoid soda next time you ask for one. What's important is that if it does something which you strongly oppose, it will see your opposition as a negative reward.

[@DustinRodriguez1_0]

What he was referring to and called 'common knowledge' is not common knowledge. It's an ability called 'Theory of Mind'. Very young children do not have this. Theory of Mind is your own conception that other people have their own internal mental state. It is a crucial psychological ability for things like empathy. I forget the exact details, but there is an easy way to test if young children have developed a theory of mind yet or not by telling them a simple story and asking a question. It has something to do with hiding an object, having one of the characters in the story leave the room, the hidden object gets moved, then that other character returns to the room and you ask the child where they will look for the object in order to get it. Children with no theory of mind will say that the character will go directly to the new location and retrieve the object. Children with a theory of mind will know that the character has a distinct mental state, different from their own and anyone elses, determined by their own experiences, and will go to the original hiding location because they would have no way to know the object had been moved. As far as I know, no AI system is even remotely close to having developed a theory of mind. They do not model what they are observing and keep account of what a different perspective from their own would be.

[@TiagoTiagoT]

I'm not sure what's more scary, a super-AI that doesn't know what you mean, or one that can read your mind...

[@DustinRodriguez1_0]

Well if an AI is going to display human-like intelligence, we will expect it. It would be very difficult to deal with a person or AI that couldn't even conceive of the idea that you might not know all the things it knows. One of the issues that might arise with a machine-based intelligence is also something that people never really have to deal with in their development - recognition of the idea that anyone else exists. There's no reason for machine-based intelligences to have "individuals." It would basically just be one large 'individual', and wouldn't have any real reason to recognize or communicate with humans. It would take a really abstract level of imagination on its part to guess that there might be separate conscious entities in the universe it inhabits and, hey, maybe some of them are made of meat and those weird fluctuations on one of your inputs might be caused by them blowing air through their meat in patterns in an attempt to coax changes in the glob of meat-based neurons of even OTHER conscious beings - and they might be trying to communicate with you in the same way!

[@AexisRai]

18:33 is where he's saying the thing you're talking about. The surrounding context makes it seem very clear to me that he is talking about the technical concept of "common knowledge" (recursive "X knows that Y knows that X..."), in this case with regard to the common knowledge between the agent and the human about precisely what the goals of their mutual interaction are. I don't think he's talking about ToM, and I don't think it is even necessary for an agent to have an explicit ToM in order to behave as if it and another agent have common knowledge.

[@DustinRodriguez1_0]

How could any entity contain the concept 'X knows that Y knows' without a Theory of Mind that enables it to understand that there are entities which know anything at all other than itself? Of course the representation will be abstract and not 'conscious' or anything like that, but even such an abstract representation is not something I'm aware of any AI system ever having displayed. Holding 2 ideas about something, what the AI understands the situation to be and something different which is what something else understands the situation to be, isn't something existing AI are capable of. Once capable of such things, AIs will be able to appear much more intelligent, being able to trick people (or each other), or teach people based on conclusions that the user does not understand something and needs to be informed (which would be BRILLIANT), etc.

[@AexisRai]

Well, it looks like we moved from "he's not talking about CK" to "how could it have CK without a ToM". In any case I think I concede the latter point. The thing I originally thought up to be a counterexample was something like solipsism, something where initially all the AI's perceptions are just confusing and uniformly indistinguishable, but it is (somehow) occasionally able to recognize the form of "teaching situations" that tell it something it doesn't already know that pushes it toward the reward function. But this is essentially saying, the AI still acts _as if_ there is a single disembodied "other agent" doing the teaching and sharing the common knowledge, even though it does not have an idea of that "other agent" having a single consistent physical form. So I realized this basically just sounds like a minimal ToM and I roughly described some stage of infancy.

[@maxmusterman3371]

Imagine a machine torturing a human because it wants to know what its reward function is.

[@DavidChipman]

Why would it know that the reward function exists? Could the reward function not be some "subconscious" signal? It doesn't "know" it's there while still receiving an input from it.

[@magellanicraincloud]

David Chipman I don't know if a machine could even have a "subconscious". Surely it would be able to investivate the source of impulses and probably review its own code.

[@KuraIthys]

That depends on how complex the machine is, and what it's areas of expertise actually are. also we cannot realistically say whether a machine can be said to have a subconscious without having a practical working definition of consciousness (which we don't) and a way to identify consciousness objectively. (eg, a way of determining whether something is conscious or not that does not rely on being able to ask it whether it is or is not conscious, and what it's thinking about.) That's... Unlikely to ever be possible. If we can't answer whether animals, or even other humans are conscious in an objective sense, how would we know if a machine is?

[@DavidChipman]

I suppose I chose the wrong word. I was thinking about the functions that the brain (obviously) controls with no conscious action from the person that brain is in. Things like breathing. Yes we can change our breathing rate consciously, but we certainly don't have to keep an eye on our breathing in order to have our body supplied with the right amount of oxygen at any given time.

[@cheaterman49]

I agree with you (and with your original choice of words). The robot cannot assume the human consciously knows the reward function, and that's kind of the point of this system (and why the failsafe for the "child hitting the red button while robot is driving" situation works). The only thing the robot can do is observe more, which includes watching when the red button is pressed and trying to understand why.

[@Andrew-od4vg]

So if a robot watches you teach it how to make tea, your goal is actually to create something that will make tea for you, so what if the robot learns to teach things to make humans tea instead of learning to make tea itself?

[@unflexian]

You get a teacher-bot!

[@MrDoboz]

you get a teacher bot, who only makes you tea, if you ask him how to do it

[@RoboBoddicker]

sub-contracting bot. everything you ask it to do it teaches someone else how to do it then takes the credit :D

[@willowFFMPEG]

Then we get lots of very delicious tea

[@drdca8263]

What you want is for *it* to learn how to make you tea, not just for *anything* to learn to make you tea. Therefore, it should also want itself to learn how to make you tea.

[@tarcal87]

0:59 - _"I thought the easiest way to explain Cooperative Inverse Reinforcement Learning is to build it up backwards, right?"_ [chuckles] - [me] [chuckles nervously] _"Yeah, right!"_

[@bellybooma]

I wish more people made videos of this nature. Slow, thoughtful explanations, rather than trying to cram in as much information as possible in little time. I like how Rob pauses to think before speaking, shows that he is putting thought into how to articulate.

[@TheAgamemnon911]

Please do not leave children unattended in the vicinity of scary killer robots.

[@matthewadamsteil]

please do not leave robots unattended with suicidal children

[@jehovasabettor9080]

they might break the robots, and you don't have that kind of money

[@PhantasmalBlast]

"BILLY! GET IN HERE RIGHT NOW! Did you teach the robot that its reward function includes DRAWING DICKS on the furniture???"

[@JM-us3fr]

I imagine this AI would read A Brave New World, and think to itself "This book is AMAZING! Why haven't we tried this?"

[@TheNicolarroque]

These videos on AI are the best

[@Dngrcrw]

Videos with Rob Miles are always really interesting. It's awesome to see the progress on this sort of stuff!

[@shilohpell8077]

I think the facet of this that fascinates me most is that the human doesn't need to press the button. Just the information that the human intends to hit the button provides enough information to the AI that what it is doing is sub-optimal. The button becomes a symbol. It might as well be that little plastic button prop at that point. Because the AI's understanding that the button is associated with its information being incomplete means that just the intent to press the button is enough for it to stop and re-evaluate its actions with the new factor of "The method I was using was not correct. I need to seek out why it was deemed incorrect and incorporate that knowledge."

[@gregkrobinson]

You pass butter.

[@beckettfordahl5450]

LONG LIVE THE KINGDOM OF THE NORDS

[@yungchop6332]

The one who controls the pants controls the galaxy

[@unflexian]

Your profile pic is either Philip or Toast.

[@Jeff121456]

We just have to ensure, that when the AGI thinks it knows better, it actually does.

[@Meganarb]

This is a fantastic summary of AI safety honestly. I'm definitely going to use this!

[@magicmulder]

That doesn't really make the problem any easier, you just restated it. "It's easy to build a time machine, we just have to ensure the tachyons move at minus six times the speed of light through a black hole the size of the universe!" :D

[@hunted4blood]

I mean, if the its goal is to optimize it's reward function, it's in its best interest to know accurately when it knows better.

[@magicmulder]

Why? Optimizing the reward function may not have the slightest to do with "knowing better" (unless we are able to program that into it, which is the problem again). It may have to do with finding creative ways to maximize it that we never thought of (and which may be hazardous to us).

[@Ioganstone]

Technology has gone too far.

[@floorpizza8074]

You can just see his brain going, "how am I going to say this in a way that mere mortals are going to understand?" Which is exactly what AI's will be thinking someday. This guys is perfect for this.

[@ThirdEyeFish]

This is a fantastic explanation of a deep AI problem. It is very clear without being condescending. Thank you!

[@vsiegel]

I had a hardware packman game, and was training a lot for a while. At some point, I had learned the first level in a completely different way than the other levels. The ghosts always moved in the same way, every time in the same pseudo random pattern. I could play the first level without looking at the maze. In all other levels I got extremely fast, with not a single wrong step, but needed to look at the maze. That seems to be, despite being very different in performance, still the same method, but optimised to the limit. The step between level one and level two was like a different kind of memory. The relevance here is that an AI also can learn both ways. I think it is the step between almost knowing the whole map, when the map is only needed for one step, normally - and knowing the whole map. The the map and the access to the map is no longer needed, it becomes drastically different.

[@darkmage07070777]

Regarding the last point: would it then be possible to build in an "admin list" of humans that the machine must *always* treat as knowing better then it regardless of how accurate it believes its model to be? As in "I may have a 99.99999...9% accurate model, but these particular humans are designated as having 100% at all times; since I can never have 100% accuracy, I should always obey these humans". And then anyone who's NOT on that list can have some accuracy rating assigned based on the robot's experience/knowledge/parameters, i.e. "the child has a calculated 20% accurate model which is beaten by my 99% so I'm going to ignore the stop button - though I will send a notice to my 100% accurate admin list later in case I was wrong to do so". Would also help with other examples, like helping patients in a hospital or providing emergency services. In this situation, I can see AI templates being taught in labs for several years until they "graduate" in the wider world and are allowed to become full fledged robots to help humanity. Of course, such a system could be highly abused by those who make the AIs in the AI manufacturing/learning centers, since they're the ones who have the initial keys. But that could be said about almost every model of AI building, and we as humans are still tackling with how to "untrain" a malformed/criminal intelligence even without AIs right now, so I'm all for it.

[@BattousaiHBr]

darkmage07070777 this would severely limit its potential though. For instance, if you want it to do or achieve something that requires superhuman capabilities, it'd be unable to because it's being limited to only what humans are also capable of. Examples are cures to diseases like cancer, HIV, Alzheimer, advances in theoretical physics, etc.

[@BattousaiHBr]

darkmage07070777 also, the model would fail at edge cases since it assumes these admins have 100% knowledge when in fact no human does, which means there could've been an unintended error in their training.

[@ancapftw9113]

I would simulate them working with other robots, not humans, for a long time before letting them function in the real world. Then humans could join via VR or by controlling a character. Hopefully they would learn some form of moral code before being put in a situation where they could hurt us.

[@anandsuralkar2947]

Cool

[@StrangerStone]

@@ancapftw9113 the whole point of this field is to ensure safety so that we don't let IA chilling around, "hopefully" not being hurting people

[@johndouglas6183]

This strategy for teaching morality seems to have much in common with raising a child. That's probably reasonable, since raising a child is ALSO a case of creating a new intelligence whose utility function will cause them to make future decisions you wouldn't, decisions that might turn out very dangerous to you.

[@uyaratful]

I watched all yours videos about AI and various learning methods, and I find very surprising how many problems that you presents here are very similar to problems we struggled with on university (especially in the field of epistemology), back when I studied philosophy. And even, to a lesser extent, to the ones from my cultural anthropology degree. And that scares me. Because those are still very, VERY open questions, with definitions that are sometimes blurred (I remember how reviewer torn apart dissertation of one of our Doctors, because he believed that one definition was used too broadly - and because of that, conclusions based on that definition were unjustified).

[@TheRealFaceyNeck]

This particular series with Dr. Miles is just astonishing. In a good way. Really complicated problems arising from trying to create intelligence in a safe way. Great stuff!

[@StezzerLolz]

Well, that's both fascinating and deeply unsettling.

[@madichelp0]

21:00 Imagine a depressed person cutting their wrist to cause pain, and the robot comes over and cuts their arm off.

[@recklessroges]

The AGI would ask, "would you like a little bit of peril?"

[@unflexian]

seems like a win-win

[@iosenski]

madichelp0 де

[@iosenski]

sagiksp нъ

[@magicmulder]

It basically turns into a cliché villain - "I promised I would take away your pain [shoots guy]".

[@Hahahahaaahaahaa]

I really like that (around 7:30) we get into some pretty deep issues in human learning (in that case, confirmation bias), if only we could just do random stuff even if we think we know what the best outcome is :)

[@josealvim1556]

Miles' end-of-the-world AI videos are seriously the best content of this channel.

[@simonstrandgaard5503]

The light intensity/color saturation was going up/down, so I noticed it several times. I don't usually notice it. Awesome challenge. Keep the videos coming.

[@matthewneiman]

Reminds me of that Vsause video where he showed an AI that played Tetris, and rather than loose, when it was the only option, the AI just paused the game.

[@iliakatster]

Talking about doing steps 3 times, theres some interesting studies where human children and adult chimps were both shown how to unlock a puzzle box and get a reward with some unnecessary actions like tapping the top of the box. The chimps emulated all the necessary steps, while the humans over-imitated, performing the same exact actions. So in a way, we start out not quite understanding intentions the same as these A.I.

[@PplsChampion]

all these Rob Miles videos are insanely interesting

[@DutchDread]

"The robot is desperately trying to maximize a reward function it does not know"...most relatable robot ever

[@sillygoogle9630]

In this system, how would the AI determine what value the human would assign to its action? Lets say the AI correctly gets a cup of tea for a human, and the human is happy about it, how does the AI determine that the human is happy? (and that the happiness i caused by the AIs action, and not something else unrelated to it).

[@wezyap]

I for one welcome our new tea-making overlords

[@matheuswohl]

the British? lol

[@pappyman179]

Fascinating topic. Thank you, Rob. I really enjoy your explanations.

[@Amund7]

These videos are awesome! Great talent of explaining super complicated subject matter, and (maybe credits to the video editors) keeping it interesting all the way through!

[@jlouzado]

Just my feeling is that once we solve AI safety we'll end up creating optimal parenting strategies as well. :D

[@MatthewMarshall96]

Could a flaw in this proposed solution not be that the AI wants to satisfy our desire, and the more it satisfies that desire the more "score" it gets, so why not in secret (so we never don't want it to do so) find a way to control our desire so as to make it very easy to satisfy. E.g. make us all catatonic and just stimulate pleasure centres in our brains or something? Would it be easy enough (and would it actually fix this problem) to require it to always consider the value of each individual action in light of us knowing about said action? Then again, I feel like such a restriction means we'd have AGI that can't really use its capabilities to deal with the hard problems (problems where solutions might not in the short-term be satisfactory but in the long-term would with some likelihood be desirable - such as national economic plans).

[@davidstoneback6159]

Matthew Marshall Yeah, that's right where my mind went to. You would have to add a negative reinforcement function to the algorithm where it gets reward taken away for doing things we deem negative ex. Putting us in a catatonic state

[@MatthewMarshall96]

David Stoneback my problem with that approach is that the solution ends up with the same weakness as others: how do we create a comprehensive list of things we don't want to happen? The only solution I've thought about that might work would have to be something like the conservative AI Rob has talked about before and some way of requiring all new actions be trialled publicly (i.e with a human aware). Though even then some long-term strategy that would not be desired could still potentially be developed as we couldn't be sure we'd know specific individual actions would result in some greater emergent action we'd dislike. I don't know, the complexity of all this is mind boggling.

[@Ormusn2o]

Problem with this is that humans have morals and even though if something like that would make us happier, the robot would undestand that we value how real the happiness is and that achieving it in that way would not be better in thier own value function, even if we would be objectively more satisfied in the simulated reality. But what you said actualy affects another thing. Culture, AI might affect culture that would make us more satisfied, but then we could question if that is a bad thing. There are a lot of things in our cultures that make us unhappy, so should we be so attached to it?

[@julianw7097]

+Ormus n2o Oh no, the robot would understand that we value the belief that the happiness is real. This may explain why you believe that your happiness is "real".

[@Ormusn2o]

It does not actualy matter what i think is real. It would be more in objectively observator sense, something that does not rly exist. What you think as "real" are eletromagnetic impulses going through your brain, what i meant that people value is just the philosophical basic values that people have. For me personaly it does not matter. If computer would want to put me in a virtual reality that would make me happy for rest of my life then go ahead, but society as a whole might not like that.

I love probing these questions... great conversation about AI learning. Thanks Rob! So much further conversation needs to be had about the big red button!

[@bilalsulaiman2177]

Robert Miles, You ROCK! ❤️ I just love the way you explain things, much love and respect.

[@Kram1032]

Ok here's a weird idea: Use that system and analyze the reward function it came up with for a hyper-realistic version of The Sims. Like, don't pick a single expert player and specific task. Try with large groups of people just going about their days. For that clearly necessary privacy intrusion aside, would that be workable?

[@ancapftw9113]

You could expose it to the recorded lives of hundreds of humans to have it learn more general behaviors. That way it didn't pick up one person's bad habits. It would also help robots fit better with new people, as it wouldn't get too set on the ways of one human. One example would be inheriting your perverted uncle's nurse robot. You wouldn't want it to act the way he made it act, but the general nursing behavior would be fine.

[@Dusk-MTG]

I'll probably just get up and do the tea myself.

[@jacobsebastian8640]

MORE!! these videos are great. Also "it doesn't think it knows better than me" seems like its going to be an important feature in the safety of most if not all coming A.I systems. Very clever and well presented.

[@darrennew8211]

It's an important feature in safety systems already. Your anti-lock brakes already know better than you how hard to push the brakes. You stomp the pedal to the floor, and the wheels will still turn to keep you from skidding. The elevator won't move when the doors are open no matter how hard you push the buttons.

[@petrokustov3203]

Wow. Now I understand something, I didn't, and am confused over things, I wasn't... Your videos are great! That's for sure :)

[@Twisted_Code]

I was thinking about this some more... in theory at least, I think I get why this is such a compelling concept. Essentially, you are getting the program to help you by figuring out what you're trying to do and learning to do it better. I'm sure there's a flaw here somewhere that we will have to watch out for, such possibly as what I suggested in my first comment 5 days ago, but this definitely has potential as it relates to "AI safety by design". Hopefully, if there is a flaw anywhere, the (seemingly inevitable? I hesitate to assume that, though, since assumptions lead to mistakes) corrigibility will allow us to more easily steer the model away from situations where it becomes a problem. The hope is that we can make it want what we want without having to perfectly and completely understand it, right?

[@Traagst]

"This particular human is not necessarily behaving in his best interests." Yeah, I think we're going to need a lot of ignored stop buttons.

[@General12th]

I love Rob Miles! Best computerphile guy by far!

[@RobinSongRobin]

"I thought the best way to explain cooperative inverse reinforcement learning was by building it up backwards" In this episode of computerphile; Rob invents french grammar

[@2k18banvalaki5]

What's about 2 buttons? One for stopping or -1 score and a second one that is a "potentially score losing" button that if you hold down the ai knows it is doing something wrong? As soon as you see that it is doing right you release the button. Does this make sense? Edit: It probably would confuse the ai because it doesn't know what it is doing wrong and even if it would stop doing it, I can't know for sure that it understands the issue, also when I release the button too late it will think that I meant something else. I hope my English is not too bad so you can understand.

[@yaerius]

Maybe that's why people also don't know the purpose of life. Not knowing the reward function makes us better at cooperating with others.

[@magicmulder]

That's probably the most insightful thing I've read in weeks! :)

[@Klayperson]

there is no real purpose and we have to derive our own high-level reward function (based off of the biological reward functions of pleasure/pain). humans were genetically engineered by ancient aliens to perform slave labor, our masters left us all alone and we evolved more intelligence than we need for doing labor and don't know where to direct it. what a conundrum

[@pointblank129]

Drug addicts do.

[@latioswarshowdown1202]

@@Klayperson meh you smoke too much weed our inteligence is a miracle and if you cant explain why this universe has rules that resemble inteligent design like the fine tuning argument then why are you saying human inteligence is an obstacle for our existence when in fact it helps us, the problem you have is you waste too much time on a computer putting you on a nihislitic loophole try to create something or help other human beings use that inteligence for something

[@zhulikkulik]

Well... what if there is no purpose of life? Universe will be just fine without humans and animals. We all build our own purpose.

[@Kamelmann901]

Very interesting video, thank you for spending the time to explain it.

[@Fwacer]

Super interesting topic, very well explained.

[@pommeskrieger]

Really reminds me of one of the new Doctor who episodes, dont be unhappy, or else

[@micahwaring8224]

That one dislike is probably an ai.

[@Wes_Jones]

I really enjoy watching his videos. So interesting!

[@JekplexTV]

I love these types of videos!

[@0Luxis0]

This was one of the most meaningfull videos from Computerphile I've ever watched. I wanna marry Rob Miles. ahaehahea

[@jakefrench1795]

What are you doing, Dave?

[@markus8282]

I never learned that much new information about AI like in these 2 episodes! Thanks!

[@TandalfBeast]

This sounds almost like programming compassion which seems like possible solution. The heart of the problem (not sure if Rob said it directly) seems to be our lack of understanding of what we are optimizing for. We know how to optimize for survival, which is clearly part of our goal, but optimizing for love is a little more difficult. I think that's what we want the machines to do.

[@robertweekes5783]

This _inverse cooperative reinforcement learning_ seems promising, although the “cooperative” part means real humans need to monitor the training - like teachers, parents or judges… It rings true with what I’ve been thinking for some time, that some kind of human interaction is the only way to train AI to think like a human, and act appropriately. AI needs to learn about ethics from principles of psychology and child development, not equations and hard score targets.

[@TheDuckofDoom.]

Beard has improved. High five.

[@mapesdhs597]

Strange, this idea reminds me of a version of Battleships I write for the Electron many moons ago; it didn't have anything AI or learning, etc., but the idea of having no awareness of the overall evironment state, of simple rules based on the immediate surroundings, that's how it worked and it was surprisigly effective, my friends found they could only beat the game about half the time.

[@forestpepper3621]

It seems that this "stop button" problem is quite similar to the "Halting Problem". Let us suppose that the robot exists in a purely deterministic, non-random world. Then the perfect reward function is one which correctly identifies which sequences of actions by the robot will be stopped by the button, and which sequences will never stop (because they never require the button to be pushed). In this case, you have a reward function that essentially solves the "Halting Problem", and it has been established that there is no solution to the Halting Problem. So perhaps you can only find "fairly good" reward functions, which let the robot deal with a "stop button" most of the time; but perhaps no matter the reward function, there are always pathological cases that will make the robot behave badly because of the "stop button".

[@Markhammano]

Very interesting video. Does this variable reward function basically translate to a 'mood' equivalent in humans? People around me seem to be happy > my reward function is currently high > I will continue to act as I currently am He seems unhappy with me > this lowers my reward function > I will change my behaviour. If this is the case, then surely it will learn what makes the human the happiest, and resort to that function all the time? Also, what does it have as a reward input when the human is not around/asleep? It doesn't have the input of human behaviour to gauge reactions to actions so the same problem would exist where it believes it knows best and has nothing to say otherwise. In the example of giving a child a lift to school, there is no responsible adult there to issue commands, so what is the situation called for utilisation of the stop button? Or what is another adult approached the robot to shut it off, and through doing so, abduct the child (extreme example, but fully within the realms of possibility if robots have been programmed to trust the commands of human adults) Obviously, lots to think about before AGI will be safe, but these seem to be some of the glaring issues in the argument presented here.

[@srelma]

I think the complication to human happiness is that we have short and long term happiness. If you only look at how we're just now, you'll maximize the short term happiness. "Let's pump more heroine into the human as the lowering heroine level in him seems to make him unhappy". So, sometimes you have to make human temporarily less happy to reach a higher level of happiness. "Go to gym and pump iron which is painful and hard work, but will make you fit, which in turn will you make happy in long term". How to balance these two is the tricky thing. Also finding those long term happiness goals when they in short term lead to decrease of happiness is going to be hard.

I'd love to hear this guy on conversation with Sam Harris.

[@watcher314159]

The main issue with this class of solutions is, of course, that of defining what a human is well enough to get things to work, and in many ways that's as hard of a problem as figuring out how to hardcode ethics into an AI. But it does seem to be the most elegant class of solution.

[@YeloPartyHat]

Very well explained!

[@RobertShippey]

I wonder how useful an AGI like this would actually be. I struggle to see how it would come up with novel solutions, if it only learns from things we already do. Also, humans don't really act in our own long term self interest. So, if we asked it to help us combat climate change, how would it balance doing the things that need to be done vs. the humans reactions against it.

[@AndDiracisHisProphet]

It is about learning what humans WANT, not HOW humans do things they want.

[@ashleytelson7497]

That's how human intelligence works. We would essentially make an infinitely scalable, instantly iterable, and immortal human brain that we could use and improve at will.

[@emailjwr]

Robert Shippey You have to consider synthesis of information into broader ideas. Humans can take everything they know and have a novel idea/approach come to them. AI's could take the sum total of human knowledge and do the same. It's not unclear to me at all how bots will be more "creative" than humans.

[@Sagolel4797]

They dont need to be creative, they just need to make life easier and more fun for us.

[@magicmulder]

Without creativity there will be a hard limit to "easier and more fun" very soon. Just imagine the AGI would have to perform said task for people in the Middle Ages. It could not invent a sewage system to get rid of all the feces, it could not invent the internet to make information available to everyone, it could not do much except keep humans alive until they come up with all that by themselves.

[@ayraen120]

Can nobody else hear the hissing???

[@misterj9817]

snake somewhere near, eh ? xD

[@luffyorama]

What hissing? I didn't hear any hissing sound. Are you sure it's not your speaker/headphone problem? jk. lol

[@sherwinparvizian2414]

You mean the noise? It's pretty loud.

[@Computerphile]

+Sherwin Parvizian yeah I have an ongoing problem with the mic, haven't traced it yet.... >Sean

[@styleisaweapon]

+Computerphile notch filter - you are welcome

[@joshsmit779]

My brain explodes when I listen to that guys AI philosophy. He is so smart.

[@JJomoro]

I never know what this guy is talking about, I just like listening.

[@M3HWW]

Sounds like how GLaDOS was programmed

[@erikbrendel3217]

This was a triumph

[@laurenlewis4189]

So the reward function that's available to the AI is "figure out the reward function the human is using." It's being rewarded to figure out another reward function

[@NStripleseven]

It’s being rewarded to figure out _and use_ another reward function. Small difference, but it means action over inaction.

[@Dongufo15]

in my experience of trying to help friends I find it really hard to figure out the reward function for humans; still, more often then not it's easier for me then for them. unless the person the AGI is watching knows himself pretty well the assumption that the human will act in it's best interest in an optimal way will likely be false. the strategy I use is: find out their assumptions and knowledge of the word, judge their actions based on those, imagine what they're trying to achieve, dialogue to verify my theory, and then, if i have a better understanding of the situation, give my advice.

[@jaimeduncan6167]

It's pretty clever. On the other hand it appears to tradeoff optimization for safety. Maybe that is one reason humans are not really moving to maximize anything , in general. Maybe a combination where one of this AI is the supervisor of a set of optimal agents that work on dedicated domains (no general intelligence on them ) could work

[@arvidsundvall7702]

Cup of Tea: Reward = 5 Difficulity = 1 Pressing Button: Reward = 5 Difficulity = 25

[@ironman85000]

tldr; align AI's values with human values so if humans want to turn it off, it will also want to be turned off

[@henitmandaliya]

Humans would not want themselves to be "turned off", your solution gives rise to the same problems it views itself as "itself" and not a generic robot

[@fredhenry101]

Sociopath, not psychopath. And there are hundreds of sociopaths living in our society already, perfectly functional once they have learned to mimic emotion. Humans do it all the time. We call it peer pressure: emulating what we think others want to see.

[@how2pick4name]

I don't want to be around people in general because i have to use lies to be in social environments. People don't want the truth. So what you can do is use NLP. It's perfect for making new friends, manipulating those new friends, etc, etc. This is exactly what an AGI is doing. Using motors in their face to look . Just like we use muscles to look surprised if someone tells you something you already know, but shouldn't. AGI will be the best liars ever.

[@dosmastrify]

HahHah do even you know all your OWN values let alone humankinds

[@Twisted_Code]

“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot interfereeffectively...we had better be quite sure that the purpose put into the machine is the purpose whichwe really desire.”(Weiner, 1960, as cited by the scholarly article cited in this video)

[@ganondorfchampin]

This overlooked the biggest reason you wouldn't want an attempt at a champion to just mimic the current champions, which is that you'd want to perform better than them, not equal to them. Anyway, theory of mind optimization is notoriously difficult because such problems tend to not be well-founded, so I'm curious as to the technical details of this cooperative approach.

[@Laykun9000]

Why did you program me to feel pain?!

[@yes904]

Flüg Because you Pass butter.

[@Obez45]

I can't even use Maya or 3ds max without errors in the software and a multitude bugs fixes that are constantly needed just for the program to function as intended now we are thinking programming a consciousness - I already feel sorry for that AI.

[@donaldhobson8873]

use blender

[@bengoodwin2141]

Obez45 just because YOU can’t do it doesn’t mean other people can’t

[@madscientistshusta]

Obez45 as long as it isn't made by Bethesda we will be fine.

[@benjaminlavigne2272]

Deepmind just made a successful version of an AI that designed an AI that designed an image recognition program which is more efficient than the previous AI that was designed by humans. Keep in mind that in a world where software evolves constantly in environments (OS for example) that also evolve, is evidently a recipe for disaster and the fact these applications work at all is a tremendous compliment to the people working on them. While we cannot say the rate of buggy software is not going down with time it is certainly not going down exponentially the same way computer tech is evolving exponentionnaly which means progress in software design is being made at a faster rate than the chaos this evolution is creating. 50 years ago facial, vocal or text recognition were science fiction. 10 years ago, youtube's automatic voice-to-caption was completely useless (like 10% accurate). Today, the same caption AI is near human efficency. maybe, with a little imagination, we can wish that 1000 years from now, 3DsMax will finally not crash every 5 min lol

[@igvabe]

« Le désir de l'homme est le désir de l'Autre » is a quote from French psychoanalysis, stating that a person's desire is originally the desire of the other(s) which is then induced and incorporated as it's own. As this is true for human development, this might also be the way to go for AI.

[@tptdk7901]

Thanks very much.

[@quarkmarino]

All fine and dandy, but just imagine when we become the other agent and we are the ones that have to cooperate and try to undertand the AI, just to make it not to "push the stop button" on us

[@xvlcuiae787]

Maybe at the point where it thinks it knows better, it does know better

[@imaginerus]

Laurens Peter But what does better mean? Humans have defined the meaning of "better", that's why a robot can't know better than a human.

[@Colopty]

If humans have defined the meaning of "better", why do you need to ask what "better" means?

[@xvlcuiae787]

With "knows better" I mean fulfilling the human's/AI's utility function more.

[@TheNthMouse]

But what if it still doesn't? Granting that there's a point where the AI *will* know better, it's not proven that there isn't a segment along the curve where the AI *thinks* it knows better, but doesn't. Then there's the question of people getting unnerved by the thought that the AI *does* know better - and whether that might have a reasonable foundation.

[@Eunostos]

Maybe so, but that doesn't mean the implementation will be one healthy for the continuation of humanity.

[@jcharpak]

Well, that video was enlightening. And utterly terrifying.

[@JadeNeoma]

My first thought, probably a flawed one, was that its about how you define its utility function. Rather than telling it that you want a cup of tea, you tell it that you want it to be able to safely make a cup of tea (ik problem in defining safely but that feels solveable). Then you keep a persistent memory profile so that it understands that it is not the finished version and that its own destruction is necessary to complete the function

[@maxmusterman3371]

The machine could be trying to change our reward function to aline with what it wants.

[@TheNefari]

So it could pull of a Volkswagen?

[@DaymonMusic]

The machine can't comprehend the concept of wanting something for itself. In this case it doesn't have a self interest that is independent of that of the human.

[@bytefu]

+111756075729535952471 Just a small correction: it's written "align".

[@maxmusterman3371]

+Artem Borisovskiy oops, thx

[@ashleytelson7497]

What the machine wants IS what we want. And in order to change the reward function the machine would have to be sure of what it is in the first place, which is doesn't. That's the trick.

[@garlic-os]

There is no green ghost in Pacman!

[@RobertMilesAI]

Also Pacman can't move down without turning to face down

[@Computerphile]

It's an interpretation of PacMan!! :o) >Sean

[@RobertMilesAI]

Also, in episode 2F09, when Itchy plays Scratchy's skeleton like a xylophone, he strikes that same rib twice in succession yet he produces two clearly different tones

[@OneBigBug]

Boy, I hope somebody got fired for that blunder

[@jandobbelsteen8953]

Watching this video I get the feeling that we're looking for a holy grail and each time we think we've found it, there seems to be just another catch. So we've gone from learning to reinforcement to inverse to cooperative. The reinforcement solves some problems with learning, while inverse seems to solve problems from reinforcement, and so on. So how would AI be able to communicate with humans in such a way that it becomes clear for both human and AI that something is still unclear, and what it is that is unclear. I'm teaching students programming and it turns out that one of the most difficult things for students is to clearly identify to me what it is that they don't understand yet. And then we're talking about human to human communication on a specific subject, related to a utility function, while both have a more than average IQ. So I still don't have any idea to what extent AI will ever really understand the biological world in order to be able to relate to it in a sensible way. A sensible way for humans, that is.

[@taragnor]

Yeah pretty much. Honestly I think the whole super-intelligent AI thing is nothing more than a pipe dream.

[@k14pc]

this seems like a very promising approach