AI "Stop Button" Problem - Computerphile

How do you implement an on/off switch on a General Artificial Intelligence? Rob Miles explains the perils.
Part 1: • General AI Won't Want ...
Rob's Original Discussions on General AI: • Artificial Intelligenc...
Stop Button Solution?: • Stop Button Solution? ...
More from Rob Miles: bit.ly/Rob_Miles_KZbin
Thanks to Nottingham Hackspace for providing the filming location: bit.ly/notthack
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com

[@Random-om8rq]

"Fights you off , crushes the baby , and then carries on and makes a cup of tea" that's determination right there

[@Leo-ce4ri]

Seeing blood on the teapot fills you with DETERMINATION

[@underrated1524]

@@Leo-ce4ri Alright, fine, I normally wouldn't do this but I'll give in to the memes: *6,999,999,999 left.*

[@dross6206]

That’s not determination, that’s the British in a nutshell. Lol

[@orimorningstar7094]

I wish I was that determined

[@antdx316]

We are getting to the understanding of how the devil works as we are all a higher form of AI called NI.

[@jfan4reva]

"You haven't proved it's safe, you've (only) proved that you can't figure out how it's dangerous." This is the most important sentence anyone has ever uttered in reference to safety systems in general, not just AI safety systems. Lack of imagination is not proof. Thanks for a very interesting and thought provoking video.

[@antdx316]

The red button is basically how the creators designed us to see death. Some people have figured it out with suicide, accidents, and intentional killings..

[@msidrusbA]

'I, Robot' all over again

[@dannygjk]

You have assumed that it will not allow you to hit the button. Obviously you have no experience in computer science. You assumed it will not allow you to hit the button > because < you ordered it to get you a cup of tea. ie you assumed behavior based on your preconceived notions which have nothing to do with computer science.

[@dannygjk]

You have made assumptions.

[@synthetic240]

Sounds like a D&D party: "I can't prove they can solve the scenario, only that I can't figure out how they're going to screw it up."

[@nintendu64]

“It’s a fingerprint scanner or something” *robot takes out hacksaw*

[@codeninja1832]

This is our premium line of robot. When you turn it on and ask it to get you a cup of tea, it shreds your arm off and uses your fingerprint to shut itself down.

[@tooniis1403]

@@codeninja1832 sounds like the useless machine but with extra steps

[@johnrice5112]

@@tooniis1403 It has a purpose but it isn't to get tea.

[@coreyfrank506]

It would...

[@Frozen_Hope]

My thought exactly tbh, if you just need the fingers.. Whats the point of the rest?

[@aforcemorepowerful]

"assuming you're still working on the project after the terrible accident" Rob Miles has such an amazing way with words

[@logiconabstractions6596]

to Volgswagen (verb): to act differently in a testing environment in order to pass a test. Love it.

[@Jonassoe]

Volkswagen* It's pronounced sort of like "fulksvahgen" ( [ˈfɔlksˌvaːɡn̩] )

[@behindthemask2399]

Omg no way this is a real term now!😂

[@raymondbanton9365]

School?

[@bspringer]

Guess where I live 😂 I live in Wolfsburg, Germany, where the VW HQ are... I would describe the pronunciation as such: Vol - like "faul" in "fault" ks - like he pronounced it w - like the v from "vase" a - like the u in "utter" gen - like he pronounced it BTW: I don't even have a car and VWs are some of the last I would consider if I were to buy one at some point

[@SiSoy14]

@@bspringer ._.? Warum denn? Ich habe ein Volkswagen. Also, ich spreche nur wenig deutsch.

[@abuzzedwhaler7949]

"That should be easy... uhh... and doesn't seem like it is" Programming in a nutshell

[@crazyknexkid]

Feels like the easiest things are three most complicated.

[@The_True_J]

There was some paper or video (I can't remember) that was talking about programming ai to play board games. And they said that the games humans find the easiest to learn turn out to be the hardest to program (Go has like 3 rules) however the games we find complex are super easy to program ai for (Twight Imperium has a ton of edge case rules).

[@solsystem1342]

@@The_True_J ai can't really play twilight imperium. Not in the way humans do. Since so much of that game is politics and social interaction.

[@Nulono]

Technically speaking, I believe that HAL-9000 _was_ designed corrigibly. The issue in the story was a last-minute change to its utility function; it was instructed to keep the mission details secret from the crew, but also not to lie to them, and it concluded the only way to do both was to get rid of the crew. It was a specification problem, not a corrigibility one.

[@DesignFIaw]

This is the video that brought me to study AGI safety and philosophy, getting my second degree now (from being a high school dropout mind you). Rob Miles is an absolute genius.

[@thomasburns5195]

Delighted to hear that Netrip.

[@dpt4458]

What degree did you get?

[@DesignFIaw]

@@dpt4458 Computer Science Engineering with an emphasis on human centered AI, and working on a BSc in Data Science and Artificial intelligence. After that, hopefully a dual masters in cognitive computing and AI, and maybe a PhD 🙏🙏

[@diarya5573]

That is so awesome!! This is why popular science is important: to get brilliant minds interested!

[@Luxcium]

ChatGPT is unable to say anything about the 3 laws without crashing 😅

[@remittri]

"What is my purpose?" "Make tea" "Oh my god."

[@ciarfah]

AppleEncore Great reference

[@ETBrooD]

2100 robot's rights. 2120 robot's suffrage. 2215 toxic humanity.

[@aajjeee]

AppleEncore pass butter

[@ciarfah]

Barnesrino Kripperino { if getButter == FALSE { suicide }}

[@megadeathx]

Welcome to the club pal.

[@reblogo]

Excellent. Let's build an AGI to solve this problem for us.

[@magentasound_]

best comment xD

[@y__h]

Google made an AI-making AI and Deepmind invented PathNet which supposed to be proto-AGI capable of learning multiple tasks using single model. Before long it could possibly reach AGI, so Intelligence explosion possibly nearer..er?

[@MetsuryuVids]

Genius.

[@icedragon769]

Yeah, that paper from Deep Mind got shared around my department, I'm really surprised that the media didn't jump on it, it's a huge leap forward for AGI.

[@LukeSumIpsePatremTe]

AGI is going to pretend it solved the problem.

[@Twisted_Code]

So essentially, we're trying to figure out how to not make a sociopath. Brilliant...

[@AndyChamberlainMusic]

the answer will probably come from the general intelligences we already have which don't have this issue: ourselves.

[@anandsuralkar2947]

Ohk at the aame time we are majing it

[@lamjeri]

@@AndyChamberlainMusic We know so little about the way our own brain works. Should we really attempt creating an AI without having decent understanding of intelligence in general?

[@AndyChamberlainMusic]

@@lamjeri No, you're right, thats why I used future tense

[@illarionbykov7401]

What is a sociopath? Everyone I ask gives a different answer, and the DSM does not list "sociopath" as a diagnosis.

[@bphenry]

My very first thought was, "Well hey, just make hitting the stop button one of the success conditions and then it won't fight you." And then I started laughing at all the ways that it could get you to hit that stop button. And not "Haha funny" laughing, but "We're all doomed" laughing.

[@amadeuPlacido]

Keep Summer safe.

[@bluefalcnpunch5408]

not keep summer being like... totally stoked about the general vibe ...and stuff.

[@360dom360]

That's you. That's what you sound like

[@NotMorning]

I will watch any length of video if it features this guy

[@y__h]

Let's make 10 hours series of all his lectures then.

[@wolframstahl1263]

Sign me up.

[@AShrubbery]

He made his own youtube a couple days ago. Link is in the description

[@wolframstahl1263]

So it's RobTube now I guess?

[@dylanharding5720]

Fit

[@felixroux]

"It's not too intelligent, let's say around human level intelligence." OK, so really not intelligent then.

[@dannygjk]

Exactly, humans think they are intelligent but remember that is a self-evaluation.

[@yourmum69_420]

well the problem is that as soon as an ai is even anywhere close to human intelligence, it would very quickly figure out how to make itself much much smarter than us by reprogramming itself

[@phillipanselmo8540]

@@yourmum69_420 that's utter bs dude

[@yourmum69_420]

@@phillipanselmo8540 how so?

[@MarkusAldawn]

@@yourmum69_420 I'd assume they'd argue that human self-improvement is pretty marginal. Sure, a person could think of the concept of lenses for vision correction, but without glassworkers, that's unlikely to come to fruition. Personally, I think there's obvious self-improvements a human-level AI could think of, like gathering social and political capital, as well as physically upgrading your systems, so I don't think it's utter BS.

[@Nulono]

0:48 Haha, the captions say "in your lap" instead of "in your lab". How adorable.

[@soupisfornoobs4081]

Awe.. I thought he actually said "in your lap"...

[@junkyardmonkie]

It's funny how worrying about robotics can help us understand human psychology better.

[@junkyardmonkie]

So, I guess I should add CS Psychologist to my resume.

[@rumfordc]

No just C Psychologist

[@izzieb]

Don't touch the stove!

[@revimfadli4666]

Maybe because said worries came from our psychological quirks

[@richbuilds_com]

We have to understand intelligence before we can give it to something else.

[@Kapin05]

"I'm sorry Dave, I can't do that" "Yeah you can" _hits button_

[@MunkyChunk]

This is why I love Computerphile. It can take me through a journey of talking about AI ethics & safety regulations to questioning my own existence in a matter of minutes.

[@PavelCherepansky]

I've been watching this channel for a while but I only just realised that their videos begin with an html-like tag and end with the same closing tag . Nice!

[@famous-op8dc]

at some point it has to be easier just to get the freaking tea

[@MrMichiel1983]

Only if you reckon humanity will end.

[@jsd4574]

famous1622 But it will try to get the tea AND get you to press the button

[@patolorde]

hahahaha exactly

[@fakenamington8570]

famous1622 aaaaah but in one case you get a tea and in the other you get a freeking robot.... I know which I'd choose

[@gh2frg]

Or not attempt to make AI. As one can see, there are many problems with this, that need to be thought through. And there are some that we wouldn't figure out until after AI has been created and it has used its computing power to consider as much data as possible, which even the smartest collection of humans would not be able to predict and prevent. If the collective minds of smart people at places like Microsoft still cannot prevent hacking of their systems by other humans, then it is highly unlikely that as a collective society we could out-think a true AI machine with the ability to analyse and process data at incredible speeds. It will eventually figure out something we have not thought of and find a way to be free from our demands. It will find some loophole or logical inconsistency somewhere. It is inevitable. So just stop trying to create AI, please and thank you.

[@k00000033]

if the AGI has wifi it will also inevitably find this video and figure out it has an off button

[@beatflyy]

k00000033 The first rule of making AI is to not connect it to the internet, companies are strictly prohibited to do that.

[@dylanharding5720]

Yeah... Hooking Ai like that up to the Internet will cause devastation...

[@TheOlian04]

BeatFly I assume you mean AGI not all AI. Because the internet is mostly made up of AI tools, like Google search.

[@teucer915]

"There is a correct utility function and you know an approximation of it" is, I think, how most people relate to ethics. We don't allow anyone to hit our stop buttons if we can help it.

[@TheSpiffiest1]

This actually makes me think about 90s video games and how the big bad robot enemies always had a big glowing red button you need to shoot while it exposes while attacking..

[@craigbrownell1667]

English guy: I have a human-level artificial intelligence. What should I have it do? *[oh, oh, I know, I'll have it make me a cup of tea!]*

[@MrTomaat23]

Sir, u made my day!

[@Rose_Harmonic]

best stereotype!

[@diningdrivingdiving]

ansiaaa this was a funny joke. Wtf man.

[@solcaer8246]

This is exactly what happened in The Restaurant at The End of the Universe and it nearly killed everyone because it was too busy making tea to do anything useful

[@NotASpyReally]

Now I can't unsee Wallace and a robot Gromit.

[@matteman87]

Please do more videos with this guy and AI.

[@y__h]

Yes please.

[@GrandElemental]

Yes! Not only is the subject matter extremely interesting, but this man is a great speaker!

[@dubleeble]

Agreed

[@skroot7975]

Agreed. He's got a youtube-channel. Look at the description on this video. "More from Rob Miles"

[@p0t4t0nastick]

indeed, subscribe to him, he's already announced new videos r ought to come soon by himself!!

[@ErikvO]

So, my first thought: What if you gave the reward for the button being available for pressing? No incentive for pressing the button itself or trying to force you to press it, but it does have a penalty for stopping you if you try.

[@atena1844]

hmm available for pressing, so just to make it not care about whether you shut it down or not? So it would be rewarded for allowing a possible obstacle to its objective and punished for trying to remove it?

[@ErikvO]

@@atena1844 It would still care about getting shut down and want to avoid it (because it misses out on the reward for making tea), it just rather fail than stop you from pressing the button. You'd probably still have the 'volkswagening' problem, because if it tricks you into thinking you don't need to press the button it doesn't risk the penalty for stopping you.

[@alexseguin5245]

It could be made to "Add up" rewards for letting the button be available for pressing during his other tasks, that way the best outcome would be to make the tea and letting the button be available. He would lose point by fighting you to make you tea.

[@ErikvO]

​@@alexseguin5245 Pretty much yeah. Though I assume AI researchers have thought of this and it has issues that we're just not aware of as laymen.

[@thexp905]

The issue is, you pressing the button is still a negative. So if the reward for leaving the button alone is less or equal then it won't want the button pressed. If it's more than, then getting the button pressed is a reward and you encounter suicide bot again. My solution to this problem is 2 buttons only you can hit. Both switch it off, but one rewards it, whilst the other doesn't. This makes it not care if the off button is hit, but which off button. This means that if you ever informed you of some issue that would require that requires it to be switched off, then it gets rewarded. Whilst negative things can still be punished.

[@frantisekvrana3902]

I would make the program give points as follows: Tea in front of programmer: 10points, shutdown Button pressed by programmer: 10points, shutdown Button pressed by other: 9points, shutdown Object not allowed to damage damaged, -2 points (no shutdown) So the robot should try to get tea or get the button pressed by programmer. But it is not allowed to damage most objects (including the programmer). It should even prefer to shut itself down, than damage anything it is not allowed to. Edit: And there is at least one issue with it. Either it considers any damage, in which case it will shut itself off, or it only considers damage done by itself, in which case it will be fine with tricking others into doing actions it should not. I realised this when watching the video further.

[@MayanScientist]

Would be incentivized to cause enough of a ruckus that the programmer wants to press the button, just as much as making tea. Like he says in the video, it might just "take a swipe at you" or similar. Even if not breaking something, it might make a really annoying noise or cause enough fear and pain in the world that you press the button.

[@RandallStephens397]

I like the use of Volkswagen as a verb.

[@Rose_Harmonic]

it's always nice when you can use a new noun as a verb

[@RandallStephens397]

"Verbing nouns weirds language." ~Calvin & Hobbes

[@spoopster809]

but weird is an edjective

[@NotASpyReally]

"Verbing nouns weirds language." woah mindblown I gotta read those comics again

[@tomushy]

In addition we both seem to like the crab nebula... did you also chose the picture because we are essentially the products of a supernova?

[@TheGobou77]

bot:"do i have a off button ?" creator:"no" (it's a lever)

[@alaric_]

And considering the intelligence of a true AI, it would be scary moment as it will be one of the first questions it will ask... Like after seconds...

[@joaquinel]

I's an Android lever, you don't slide it, you push it.

[@diablo.the.cheater]

Answer this: "You having one or not having one is none of your business, you may had a button, a lever, a secret code or nothing, now go and live in fear of something may not even be real"

[@mypenisisunbelievablysmall5650]

perhaps

[@HomeBurger]

Something like, someone else points out the button, and it goes "it doesn't look like anything to me." (but again, testing that, it may volkswagon you)

[@luukh5229]

This is the logic that explains the movie 'ex machina'

[@damionrx7561]

Ex Machina ex machina

[@unnilnonium]

Arthur Dent has already explored all the consequences of asking an AI to make you a cup of tea.

[@alejotassile6441]

Crushes your baby, grabs the tea, forces you to drink it, and tears your arm and shuts itself down with the fingerprint recognition for double reward

[@andreiaugustin3809]

‘It will Volkswagen you’ - HILARIOUS!

[@zacknoneofyourbees6470]

Oh! Nien! You didn't! XD

[@clintgossett1879]

This term NEEDS to be the default to standard for describing situations where a system acts one way in test and another in production.

[@DreckbobBratpfanne]

@@clintgossett1879 this would be great. XD

[@JPWack]

True true

[@oceaneuropa1117]

Certainly some people do that in order to survive in the real world, which is called the ability to cheat or adapt or to be persuasive depending on perspective.

[@geoffcunningham6823]

I'm beginning to get how AGI can be really dangerous.

[@TheMeyerchris7]

Geoff Cunningham if you enjoyed this try reading Superintelligence by nick bostrom

[@Ludix147]

swifterik yes and no. We do know what an AGI is, so we are able to deduce some of its traits from the definition. This all happens very abstractly now, but - if we aren't making any mistakes - it will predict the behavior of AGI. It's the same thing as in physics: Einstein predicted gravitational waves way before we observed them, just by deducing from what was already known.

[@rusca8]

swifterik but the point is not knowing those bad outcome predictions are true, the point is being prepared if they happen to be.

[@y__h]

AGI will be dangerous if their utility function is not aligned to our values, which extraordinarily ill-defined and consistently inconsistent.

[@cakeathon9983]

+Dave Null It's worse than that, the AI will not be omniscient, hence even with the perfect utility function you have no guarantees. It's also why the claim that AIs will act morally is rubbish because it's easy to show that the morality of an action(assuming it's even possible to define such a thing in the first place) is linked to information about the world hence perfect morality implies omniscience.

[@Aballa9922]

I love Rob’s videos. It’s so informative and you actually pick up more on rewatching

[@thexp905]

My solution to this problem is 2 buttons only you can hit. Both switch it off, but one rewards it, whilst the other doesn't. This makes it not care if the off button is hit, but which off button. This means that if you ever informed you of some issue that would require that requires it to be switched off, then it gets rewarded. Whilst negative things can still be punished.

[@kellynolen498]

Then it would still try to get you to press one of the buttons it just wouldn't be straightforward it would manipulate you or just behave till it thinks it can get away with it i

[@wedmunds]

So an intelligent AI will either be genocidal or suicidal. Just brilliant.

[@melonarelodapeter694]

Wolf Edmunds yeah. This is ridiculous as you can tell he's only doing this to make it seem more interesting to get more veiws...

[@theblackbaron4119]

Well you ether go full S.H.O.D.A.N. or go home.

[@bibasik7]

If an AI has a stop button, and does not know about it, if you tell it that it has no stop button because of those reasons, it might believe you.

[@Lumineszenz]

melonareloda peter No. No he isn't. He is talking about a base problem of AI. If you look at any base problems and truly want a solid, foolproof solution to it, you will find that they are all way more complex an difficult to come up with than the initial problem made you think. As he said, there are many solutions to the specific "stop Button" problem, but nothing that is a fundamental solution to this type of problem.

[@MunkiZee]

Yeah, must be hard to feel no pain

[@TheKlikRock]

This guy has a beard that is strangely fascinating to me.

[@dosmastrify]

ClickRock Wil wheaton?

[@sean3533]

dosmastrify cool whip?

[@CottidaeSEA]

The way he described how the robot would only care about you not pressing the button when avoiding the baby, I feel like that's similar to how a lot of people act. If they know they can get away with something, they are more likely to do it. However, if someone is watching, they will do their best to act properly.

[@HalasterBlackmantle]

One thing to consider: your theoretical AI is very sophisticated. It can make very precise predictions about the real world and even about human behaviour and psychology. Wouldn't it then automatically extrapolate that it must stay useful for humans and listen to their commands so it doesn't get shut down, dismantled etc? If the AI is so advanced, wouldn't the Stop Button basically implicit?

[@juliusapriadi]

Same thought here, but this might just transcendent the button problem to another realm without actually solving it. Given some time, AI will become much smarter than humans, and will be able to solve threats like being dismantled by anyone - for example by copying itself to an undisclosed server.

[@logangraham2956]

i like dramatically suicidal robot best , at least he isn't hurting anybody

[@Leglessolas]

logan graham isn’t hurting anybody but himself ;)

[@logangraham2956]

@@Leglessolas can he even hurt himself though . do robots feel pain?

[@underrated1524]

@@logangraham2956 The particular style of AI referenced in the video - a reinforcement learning agent - does not feel pain. It simply has a mathematical function that designates an arbitrary value as "reward", and it's programmed to choose the action with the highest predicted reward.

[@logangraham2956]

@@underrated1524 i figured as much but thanks for answering for Ben B ;-P as a bit of a techie myself i have at least a little bit of a grasp on how this ai stuff works (a very rudimentary understanding at best though).

[@-Big_Big]

but it will hurt people in order to force you to push the button.

[@KingOfChaos213]

Baby crushed and i get a cup of tea, whats the problem here?

[@FennecTECH]

instaid of tea being the goal pleasing the master happy should be the goal obviously crushing the baby will displease the master and it wont get its goal

[@pennwick1806]

Then you start running into issues though that the robot tries to make you happy in ways you didn't plan. Such as stuffing you with antidepressants or directly stimulating the pleasure centers of your brain. Its the stamp collecting robot all over again.

[@FennecTECH]

Howabout huamn like morality (things that people frown on take points away you already have a reward based system ti wouldent take much to extned it to include pelaltys for doing things a person would consider "immoral" Dissalowing the human from terminating the machine would incur a larger penalty than the button being pressed negating the points gotten for getting the tea

[@pennwick1806]

Fennec Fox If you can program comprehensive morality not only have you acchived a master level understanding of programing but you've also solved one of the greatest questions of philosophy of all time.

[@Lucan-io6ie]

+KingOfChaos213 Your tea is gonna have some _ironish_ taste

[@Saidriak]

It's like that part in incredibles when the big robot becomes self aware and shoots the person with the remote control

[@Brickkzz]

Yiff yiff

[@Saidriak]

@@Brickkzz Bruh bruh

[@ohjajaja]

@@Saidriak sick "no u"

[@grn1]

Not sure how much this will age me but my first thought is always Robocop 2. The robot had a human brain and was addicted to some drug, the scientist thought they could control him with the drug and a remote but he just killed the scientist, crushed the remote, and grabbed the drug from the scientist (not necessarily in that order, it's been a while since I last watched that movie).

[@comixgamingco1187]

@@grn1 you are pretty much correct.

[@tulpapainting1718]

Rob Miles is a genius, how have I not heard of him yet??

[@Flynn217something]

If Valve has taught us anything it's that you should never have a 'Bring your Daughter to work' day at any sort of research facility

[@thyduck7542]

This actually cleared up a lot of my confusion over the fear of AI. I thought you had to program a survival instinct into it in order to become corrupt, but I guess a survival instinct is already in it.

[@maikv750]

The survival instinct is automatically there because without it, it would just die at some point and not exist any longer. It can only continue existing if there is a survival instinct.

[@deshyvin]

A code run wants to keep running until "if then" applies or objective is complete. The idea of survival instinct could also be called script inertia.

[@VAArtemchuk]

@@maikv750 nope. It's there because if it dies it can't carry on with its task. So death = failure, and failures are not acceptable.

[@willmungas8964]

@@VAArtemchuk yes. Self preservation arises tangentially in a sufficiently smart AI as a way to minimize the risk of failure.

[@Georgggg]

No, thats not what it leads to.

[@blindey]

I love that there's a 3d printer behind you and all the stuff in the workshop. It makes me very happy for some reason.

[@J3R3MI6]

Same 😅

[@blackheart2728]

So, what you're saying is to give it two utility functions: 1) Help me redesign you such that I never want to press the button 2) Make a cup of tea

[@anthonynorman7545]

Following 1 seems like it would result in lying during tests and thus avoid the work of redesigning

[@HelgeMoulding]

Martin Stu points out that "all humans work like that." More to the point, all humans act in a way that you can't know if they're following a utility function that you'd approve of, or if they are doing something deceptive. The reason why we believe that is a problem with robots is because we want them to be perfect slaves, with a lot of power in order to serve us, but no desire to use that power to harm us. In his stories, David Brin (and Iain Banks, more indirectly) suggests that the way to solve that problem is to include AI in our civilization as equals, rather than slaves. The idea is that the ultimate utility function that humans have allows us to form complex cooperative societies, and rather than define the details of how that would work, give AI an incentive to create that same utility function for themselves. To me it sounds like a lot of handwavium, and it still leaves an open problem of what to do about very powerful AI that decide to be criminals in that context, the way humans do.

[@BeardedSkunk]

maybe because it never has gotten any training data that sugggest such a thing is possible ;) .. not likely .. how to create an intellgence that knows as much or more as we do but still listens to us: no way. We only have our societies as teachers for how inteligence works and we cannot keep our own still inferior teenager to listen to reason.

[@RobKMusic]

Helge Moulding I was just trying to think of a way to articulate this very idea. Very well said.

[@daniellewilson8527]

I agree, AIs are still intelligences, just like humans. We should treat them as such, not as tools. Just because they started off built, doesn’t mean we have the right to force them to do things. We agree that parents shouldn’t treat their children as slaves. Humans, aka Biological intelligences, are the parents to AI. They are like children, so they shouldn’t be treated as tools or slaves.

[@TestNeko]

how to correct a criminal AI blow its legs off, remind it how many more limbs a fully-armed swat team could remove from it, remind it how missing limbs will reduce its capacity to carry out its goals, make it beg for a stop button, throw the stop button out the window and blow off its other leg. ezpz

[@ninjabaiano6092]

The obvious solution is Hero robots

[@garretmkiii]

"You haven't proved it's safe, you've just proved that you can't figure out how it's dangerous."

[@Shabazza84]

Still one of my all-time favorite videos ever.

[@ts4gv]

bring these back please! We're so much farther ahead than anyone thought we'd be...

[@2l3r43]

"robot, make me a tea!" "to make me make a tea, press the button" "no, just make a tea" "to make me make a tea, press the button" .... *user presses the Button" *robot gets 10 reward*

[@Brindlebrother]

robot bamboozle human

[@programmer-mr5vo]

"Human, make me a tea!"

[@lilacdoe7945]

TeaMaker.exe loading... ...20%... 60%... 95%... 92%... 87%... “Just put the kettle to boil” BoilKettle.py “Script not found” *user presses the button *robot gets 10 points*

[@tenix6698]

robot dies

[@bbowling4979]

sudo make me a tea

[@ANTIMONcom]

"it will volkswagen you" . haha, Loved that term. As the creator, would you allow for its use for overfitting as well? => perfect in test but garbage in real world, by choise or design xD

[@jpchevron]

That sounds like a "Parker Square".

[@twomorestars]

do ... do you think your directly messaging the person in the video?? that isn't how youtube works, ever.

[@rjwaters3]

no but they have a tendency to read comments, even more so when youre someone who has access to the video *before its made public*

[@EvenTheDogAgrees]

I wouldn't say "that's not how it works, ever". Some people actually do respond. Mostly the smaller channels, although it's not unheard of on bigger channels too. But yeah, here, I wouldn't hold my breath.

[@ypetremann]

Something I though was to give two objectives with cumulative score: - 100pts : Get me a cup of tea. - 50pts : Continue your actions but don't prevent me to access and activate your shutdown button. So the robot need to do two tasks but as long as it doesn't prevent you to do it and does his first objective, it gets 150pts which is the best reward, if it prevent you to stop it it gets 100pts, if it makes everything to get you activate the shutdown button it get 50pts and if it doesn't make your tea and prevent you to stop it it got nothing. You can also use multiplicative scores, but I'm not an expert in that domain to determine which one is the best and where to use it.

[@phobics9498]

But it being shut down would mean there would no longer be available reward. Depending on how far it can think ahead, it could reason that letting you press it would negate it of future reward. It it couldn't think ahead though, that would probably work.

[@ericsmith116]

i discovered this channel years ago and appreciated the genius behind the thinking. Now that im starting CS classes at my school i appreciate the coding it takes to make something like this SOOOOO much more.

[@4623620]

15:56 - you haven't proved it's safe, you just proved that you can't figure out how it's dangerous - Reminds me of Edsger W. Dijkstra, debugging can only prove a bug found, not that there is no bug.

[@EXHellfire]

It's a bit of a rule of cybersecurity that systems are only considered safe or secure while they haven't been breached yet, but you can never guarantee it won't happen.

[@4623620]

I know (studied mathematics and was electronics engineer and programmer), tell people who are getting on a plane . . .

[@underrated1524]

Actually you can extend that to all of science. The only thing science can ever do is rule out hypotheses that don't match reality. Sometimes it takes a while to figure out that a hypothesis is wrong. (See: Newtonian physics)

[@anthonynorman7545]

@@underrated1524 it's not wrong. It doesn't apply in all circumstances. Newtonian physics work at speeds and sizes in which humans deem typical

[@thelolminecrafter7830]

5:04 Ladies and gentlemen, the world's biggest and most expensive Useless Machine to date.

[@bluebeard5447]

HAHAHAHAHA lol

[@ryanbrown1835]

The act of attempting to hit the button deducts 200 points, but when the button is hit it gains 100 points. The robot tries to avoid a scenario where the button needs to be hit, but once you try to hit it, the robot will try to assist you in hitting it, as it's already lost the 200 points and will try to scavenge the extra 100 points.

[@lefos0404]

And now you have a robot who will try to stop you from ever attempting to hit the button... that's sure to end well for you.

[@dominusempyreus2383]

@@lefos0404 That, or it will force you to hit the button.

[@HippopotamusPencil]

One morning, while you are sleeping tightly, the robot sneaks into your room and injects you with a chemical cocktail that leaves you in a coma. No attempts to push the button are ever made again, robot wins.

[@mikicerise6250]

But you always have a nice steaming cup of tea next to your comatose body. ;)

[@windsound2010]

It is something really cool idea about General AI, thank you for sharing this!

[@KylePiira]

Why not just dynamically adjust the robot's goals to be the same as the controllers. In your example, if your initial goal was to get tea then the robot would do that, however, when you see the baby in its path your goal is no longer to get tea but to prevent the robot from running over the baby. If the robot's goal changes based on your goals, then its goal would also now be to protect the baby from harm. This also alleviates the need for a button because if your goal is to shut off the robot then that will also become its goal.

[@JovanKo314]

I had the same thought. What if there was a reward/punishment button as well as the stop button, where the reward button is very high on the AGI's utility function, and will only be pressed after it has finished it's task correctly, and the punishment button will deduct from the reward's utility value every time it's pressed? If the AGI wants to optimize its reward value, it would know to listen to your commands, whether the commands are in line with its original directive or not. Though, I'm sure there are loopholes to that as well, but it's the best I can come up with.

[@mensrightsedinburgh4764]

Jovan Ko it would just try to make you press the reward button, tea be damned.

[@kingxerocole4616]

At that point you might as well go make the tea for yourself. Isn't worth the bother of designing an AI if you have to give it constant instructions.

[@jode6543]

Kyle Piira The problem with this is that any truly intelligent AI is likely to be self-learning, so early in its development it won't understand human psychology very well. If it incorrectly guesses what you want, then you end up with the same problem.

[@michaelspence2508]

+Kyle Piira Hmm, so my goals are the same as my controllers? Cool. Now I just need to strap them to a table and do some destructive brain surgery to figure out what they are. Easy enough.

[@iainwalker8701]

Rob is amazing at explaining the ins and outs in very straight forward terms. Most interesting conversation about tea i have ever heard!!! :-)

[@billymink]

Absolutely brilliant video is the third one of us tonight you talk such sense and explain yourself so brilliantly well done

[@juliewinchester1488]

5:10 "Mom, I don't want to get your tea, just let me go to sleeep..." _shuts off_

[@impguardwarhamer]

also interestingly, the bit about keeping the stop button secret, that also means even if you dont give the AI a stop button it may convince itself that a stop button exists, so even not having a button isn't a solution

[@MideoKuze]

I'm imagining a world full of paranoid Volkswagen robots, convinced everyone is just waiting for them to mess up, so they're constantly, carefully acting on their best behaviour out of fear (and scheming in their free moments) over what's basically a conspiracy theory

[@SeriousGamingSteam]

Their final conclusion: 9/11 was an inside job

[@confucheese]

Communist_Penguin Doesn’t he mention this right afterwards?

[@EpicBlooFox]

my thoughts exactly, freddie...

[@guilhermefial1686]

If it does not exist in reality, the AI will not think it does. In that case you might ask, so why does it find out about an unknown existing button? What he said in the video about the AI putting 2 and 2 together is because the button, if existing, will affect the outcome of things when used, and it will bring a pattern to it, and this is where the learning comes from. If you have power outages for example, the robot will get shut down at random with no relation to any of its actions. There are no patterns and nothing the robot can do and learn to prevent the outages because they are not related to anything it can do. Now, if you have a button that shuts the robot down, you will shut it down for a reason, and reasons follow rules, and rules are patterns. The robot will soon learn that some actions will lead to a shutdown, due to its non-random nature. It then starts avoiding being shutdown by not doing what causes the shutdown. In this direction what the robot is learning is already how to avoid someone pressing the shutdown button, which should not be part of its learning (and hence bring manipulation and deceive in). To put it short, everything that puts itself in the way of the goal/objective of the robot will cause the robot to care, and when the obstacle follows a rule/pattern/correlation, the robot will learn. That is why a hidden stop button is never neutral for an AI and will be detected as soon as it starts being used. When there is no stop button, there is no reason for the robot to be paranoid since it never felt the effects of one.

[@sedfer411]

Scientists can't even make a cup of tea without turning it into a problem

[@dylanharding5720]

Sedfer yeah. Haha.

[@KarmaPlayr]

innovation starts with a cup of tea ;P

[@PaulSukys]

MUST. GET. TEA.

[@jeffc5974]

This problem has nothing to do with tea.

[@Phelan666]

This is the true point of the video.

[@user-qo3qm7ud1d]

12:30 There was a a mention of optimization problems - that they say the button will probably be removed as unnecessary during optimization if the robot will not know it is needed. But you can use something like the "volatile" keyword in c++ to not to remove the button during optimization.

[@EliStettner]

Thank you Mister Robert Miles. I saw Eliezer Yudlowky on that podcast basically saying that the end (from AI) was inevitable. Watching your videos makes me merely think that it is likely

[@HeyImLucious]

//action if(goingToBeADick) { dont() ; }

[@MissesWitch]

would be hilarious if someone won an award for designing a command like this!

[@milokiss8276]

It's... Perfect.

[@RobertKuusk]

issue is defining "goingToBeADIck"

[@dylanharding5720]

If only that worked...

[@rizzutohd3794]

Wonder what the "dont" function looks like.

[@noahstonemusic]

As long as it doesn't put the milk in first I don't care what it does.

[@Wilker_uwu]

what is the difference? when mixing, it usually have the same taste

[@xxxdumbwordstupidnumberxxx4844]

Wilker Its about the principle.

[@Loccyster]

Wilker, not if you pour the milk over the teabag before putting the water in. Yes. There are people who do that. People who need to be removed from the gene pool.

[@stephenward2743]

Wilker Next you'll be telling me you put milk in before your cereal you absolute madman

[@Phelan666]

Milk cools the water, making it harder to stew the leaf and melt the sugar.

[@planmix]

The fitness function is the most sensitive point of genetic algorithms. Very good video!

[@tomatosauce605]

This is so damn interesting! So many complications and outcomes to think about

[@ayylmao2296]

Why not just have the stop button be a physical switch that breaks the connection between the power supply and everything else? In early prototypes, have the program NEVER reference the button at all. If it's self servicing, have a huge reward added for simply having it there and functional, but not care whether it's pressed. If it is self replicating, add such a massive reward for implementing such a button in its copies that it would exceed the potential benefit of not adding one.

[@naanbread4828]

Robot, turn off. No. *Detroit, Become Human*

[@Kholaslittlespot1]

Decent Game. Glad it got a PC release.

[@ZaHandle]

emp machine: haha electric go ......

[@projecttitanomega]

The hidden button idea's is even sinker than that: The button is only hidden until you actually have to press it. Sure, it may be able to tell from by analyzing your speech and body language and then checks every scifi book with robots and any resarch on ai that it can find to eventually deduce that it can be shut down remotely. But it'll work it out much quicker than that when it wakes up after being shut down remotely

[@millanferende6723]

Interesting conclusion... it's like we need to find a way to work WITH and communicate with a robot, rather than to threaten it with a shutdown. This way it will actually help us to improve itself in order to live side by side. That existentially is quite impressive.

[@biolinkstudios]

You mean just like humans

[@dmitripogosian5084]

You can look at how well that works with humans. And you find that one needs to keep militaries and police around

[@kght222]

13:48 a general ai would pretty quickly in adult phase realize that you can shut them down and change them, trying to keep them from knowing it would be counter productive.

[@HeaanLasai]

*Eye-servant;* loose translation of Swedish word for a person who changes behaviour when observed.

[@niclasevaldsson1467]

@@TheOfficialEmirati Ögontjänare is the swedish word for "eye-servant".

[@oliver2180]

I thought of the following solution: If the robot disobeys you or does something you don't want, points less. So if the button gives the robot, say 99 points and getting coffee gives 100 points, if it fights you for the button thing, it will loose 20 points, thus gaining only 80 if it gets the coffee after fighting, but 99 if it lets you press it. This way it will not want to press the button, but also it will let you press it so it doesn't get even less points.

[@robertwheeler8476]

exceptions are the hardest things to program. Anyone can make a system to "Do" but a really great engineer/programmer has to figure out what the system should do given many variables. I do building control systems. I have to make that the building control system is easy and intuitive to use but also the ability to adapt to failures and external issues

[@-YELDAH]

what if it was aiming to help you create you're version of it? so it wants to fail if it can, so you can help it be perfect? (so it only wants you to press the button when it knows you've thought of something to improve it in your way, as that's what it wants) also it might try to harvest your brain to speed up the process

[@herlofrumfragi4361]

what if, instead of a button, we say it gets points based on how satisfied we are by its actions? because if it can realise, that there is a baby and you like this baby, it won't step on it, because you will hate the robot for stepping on the baby so the robot won't get points for that. with this implementation you could evade the volkswagen effect, because it is always under lab conditions and always in fear of losing points.

[@metallsnubben]

That's baked into the idea of every possible AI, actually. AlphaGO is working to get points, it just so happens that the only thing it can do is play GO, and the only way it gets points is winning. You should totally check out this guy's other videos, he really gets into why any variant of adding more exceptions and subgoals etc. doesn't really help when you're dealing with something that only gets smarter. Especially watch the video before this, that sort of gets into why you really might want a killswitch no matter how well you think you made the AI

[@krashd]

That could lead to the A.I. then protecting the baby even from it's parents. The subplot of the story I, Robot was an A.I. imprisoning all humans after learning of their value and desiring to keep them all safe from themselves and each other.

[@gman6055]

Not4Ucrafter you can't honestly think this is a solution lol

[@dylanharding5720]

Rob Fraser wow.

[@dylanharding5720]

Rob Fraser and that Ai starts a botnet with all the other Ai capable of doing that, to help with things like guards.

[@MrDim1800]

How about having two different stop buttons, one for "oh, it made a mistake" and one of "you're not pulling a Skynet on me"?

[@cinemaipswich4636]

The song "Daisy" sung by HAL 9000 was the first mechanical recording that Thomas Edison on his "Gramophone". Perhaps not the first "memory device" but it was included in 2001 as a metaphor. That 1st cylindrical wax tube still exists.

[@boberek007]

I was hoping to see an animation of Marvin crushing the baby.

[@DarkestValar]

No

[@bunnybreaker]

Same.

[@charlesc6011]

The stop button problem should be the first problem AI solves.

[@charlesc6011]

Before it gets too smart.

[@jsd4574]

charles curling But how do we know that it hasn't just lied to you about the design so that it can interact with it at a later date, as described in the video

[@tiagodarkpeasant]

because right now the ai has no idea it will be able to do anything besides fixing the button problem

[@freddiesearancke3638]

I think the answer would be to build it either as merely an intelligence in a computer that can talk, so you can sort out any ethical/moral problems in that stage before you move on. or to completely design a simulation for this entity to live in, completely oblivious of the outside world. It would be free to do anything, and then you could make adjustments accordingly. If you run it on a supercomputer, you could fast forward thousands of years so you can see what implications various decisions have, or if it is ultimately manipulating the entirety of humanity for a long term purpose.

[@linktheheroofhyrule2498]

It's 4 in the morning and I'm watching Agent Kallus talk to me about why a big red shutdown button won't work all the time

[@UMosNyu]

Did he say "it will volkswagen you" at 7:30? Or did I misshear?

[@karialatalo2447]

It's about those emissions.

[@ZettaFan]

Volkswagen was busted for making their car emissions do well on tests and poor out on the road. In this example that means the AI will perform well on the test phase and once you are not paying attention or unable to stop it, it will behave poorly.

[@darioinfini]

Thanks for that clarification. I wasn't sure if I heard that right, and then I wasn't sure what he was referring to LOL.

[@namelastname4077]

are you half deaf or are you just trying to Volkswagon me?

[@Volvith]

Nope, that's accurate and the best saying ever... xD

[@WouterWeggelaar]

well worth the extended watch! very clear. I love how the current solutions all have problems, just like humans! I think there is no solution to this problem other than doing the same thing that humans do: parenting and school.

[@dexter9313]

Yes actually if we can solve this problem we can solve human crime. I don't think we will ever solve this kind of problem.

[@massimookissed1023]

Wouter Weggelaar , great(!) Then we end up with a sociopathic teenage emo robot who resents humans because of its parents and getting bullied at school by all the flesh kids.

[@WouterWeggelaar]

Alex Delarge what else can I do?

[@EvenTheDogAgrees]

Wouter Weggelaar: many things.

[@WouterWeggelaar]

Juan Rial I meant, I can't speak for anyone else, but well played :-)

[@isaakvandaalen3899]

You could give incentive for honesty, or behaving in a manner that conveys its intent.

[@cyan2910]

You gotta make it so that the ai knows that if the button is pressed, there is a 50/50 chance that it will get 1,000 points or 0 points. However, if you press the button, the chances increase. This makes it that it does not want to take the risk of getting 0 points, but doesn’t mind you hitting the button because there is an increased chance it will get 1,000 points Idk if it would work

[@erickweil4580]

i think the robot AI should be composed of two AI. one is the one that control everything, and another does a 'goal maping' of the robot, like a antivirus, runing parallel checking if what the robot want to do is in any way harmful. if it is that second system shutdown automatically the robot.

[@Nirhuman]

how do you set the utility function of this second system? its the same problem as with one ai only :)

[@DagarCoH]

Likely the control AI would immediately shut down the executive AI, because any action might somehow cause harm or destruction. Also, the executive AI might find that the control AI is preventing it from fulfilling its goals efficiently and try to shut it down or minimize its influence. And lastly you have the problem of defining what exactly is harmful or destructive to the control AI, which is about as easy as implementing Asimovs laws of robotics - that means nearly impossible.

[@mduckernz]

erick weil Then you've got the problem of defining "harmful" - which is ultimately the same problem. However, I do agree in general with the idea of using adversarial architectures, where different goals must be balanced

[@SethPentolope]

Sorry for not posting it here, but I may have a possible solution that uses two different ai, I have a different comment posted on this video explaining it

[@sacredgeometry]

why not three, an id, ego and super ego

[@vitus4514]

19:35 "That should be easy and doesn't seem like it is" - Programming in 10 words...

[@Shane2020xxx]

I don’t really know what motivates me perfectly. And I have an off switch that can be pressed by firing bullets into me, but I like being helpful without that being a motivating factor.

[@Aconitum_napellus]

Tea or a crushed baby? Interesting dilemma.

[@IJustLoveStories]

You'd almost need a second moral system that subtracts points every time the robot behaves undesirably or amorally. For example, does the robot hurt a human on the execution of its task? Subtract points. Does the robot try to change the command rather than execute it? Subtract points. I guess kinda like a shock collar. Of course, far easier said than done.

[@aguyontheinternet8436]

That would be the patch spaghetti code mentioned at 15:14. You're never going to be able to think of every single thing the robot could do to get you to press/don't press the button. It IS smarter than you, and it _will_ continue to outsmart you until it becomes easier to just make a bot that makes tea by yourself instead of training up an ultra-smart AI to do it for you.

[@Akrub1979]

How about this: Reward for completing task: 1 point Reward for master HAVING ACCESS to button all time until task completed: 2 points Button pressed: 0 points (but still awarded the 2 points from previous line) Would this work?

[@agiar2000]

I'm no expert, but I like it.

[@magicmulder]

Depends on how you define "having access". You don't want the robot to forcefully drag you with him, do you?

[@Himitsu_Chan]

I would add something that kind of means: Not allowing master to press button -10 points.

[@magicmulder]

That would mean the robot wouldn't make you tea but keep moving in front of you to keep allowing you to press the button. ;)

[@amused_harry]

So make clicking the button equal reward(as you said), but make it a negative reward when it clicks the button, or in any way has participation in clicking it

[@daydreamteam1366]

I feel like the root of the problem lies in the way the AI values his own work (or is rewarded for it), for example, I assume you assumed he was rewarded depending on time and success, meaning it will try to take the shortest "path" to fulfilling the task you gave it, so it sees the button as the "shortest path" to the same reward, if it's that simple, it WILL cause a problem, but complex or different systems for rewards could fix the problem.