9 Examples of Specification Gaming

AI systems do what you say, and it's hard to say exactly what you mean.
Let's look at a list of real life examples of specification gaming!
Related Videos from me:
Reward Hacking: • Reward Hacking: Concre...
Reward Hacking Reloaded: • Reward Hacking Reloade...
What Can We Do About Reward Hacking?: • What Can We Do About R...
The list: tinyurl.com/specification-gaming
The blogpost this video is based on: vkrakovna.wordpress.com/2018/...
The newer blogpost that happened while I was making this video: deepmind.com/blog/article/Spe...
(Explosion graphic from videezy.com)
Thanks to my wonderful patrons:
/ robertskmiles
Gladamas
James
Steef
Scott Worley
Chad Jones
Chris Canal
David Reid
Francisco Tolmasky
Frank Kurka
Jake Ehrlich
JJ Hepboin
Kellen lask
Michael Andregg
Pedro A Ortega
Peter Rolf
Said Polat
Teague Lasser
Allen Faure
Bryce Daifuku
Clemens Arbesser
Eric James
Erik de Bruijn
Jason Hise
jugettje dutchking
Ludwig Schubert
Qeith Wreid
Andrew Harcourt
anul kumar sinha
Ben Glanton
Benjamin Watkin
Cooper Lawton
Duncan Orr
Eric Scammell
Euclidean Plane
Ian Munro
Igor Keller
Ingvi Gautsson
James Hinchcliffe
Jeroen De Dauw
Jon Halliday
Jonatan R
Julius Brash
Jérôme Beaulieu
Laura Olds
Luc Ritchie
Lupuleasa Ionuț
Michael Greve
Nathan Fish
Nicholas Guyett
Paul Hobbs
Sean Gibat
Sebastian Birjoveanu
Shevis Johnson
Taras Bobrovytsky
Tim Neilson
Tom O'Connor
Tomas Sayder
Tyler Herrmann
Vaskó Richárd
Will Glynn
12tone
14zRobot
Alan Bandurka
Alexander Brown
Anders Öhrt
Andreas Blomqvist
Andrew Weir
Andy Kobre
Anne Kohlbrenner
Anthony Chiu
Archy de Berker
Ben Archer
Ben H
Ben Schultz
Bertalan Bodor
Brian Gillespie
Bryan Egan
Caleb
Chris Dinant
Daniel Bartovic
Daniel Eickhardt
Daniel Kokotajlo
Daniel Munter
Darko Sperac
David Morgan
DeepFriedJif
Devon Bernard
Diagon
Dmitri Afanasjev
Fionn
Fraser Cain
Garrett Maring
Ghaith Tarawneh
HD
Hendrik
ib_
Igor (Kerogi) Kostenko
Ihor Mukha
Ivan
James Fowkes
Jannik Olbrich
Jason Cherry
Jeremy
Jesper Andersson
Jim T
Johannes Walter
Josh Trevisiol
Julian Schulz
Jussi Männistö
Kabs
Kasper
Kasper Schnack
Kees
Klemen Slavic
Leo
lyon549
Marc Pauly
Marcel Ward
Marco Tiraboschi
Marko Topolnik
Martin Ottosen
Matt Stanton
Melisa Kostrzewski
Michael Bates
Michael Kuhinica
Miłosz Wierzbicki
Mo Hossny
Nathaniel Raddin
Oct todo22
Owen Campbell-Moore
Parker Lund
Patrick Henderson
Paul Moffat
Poker Chen
Rob Dawson
Robert Hildebrandt
robertvanduursen
Robin Scharf
Russell schoen
Scott Viteri
Simon Pilkington
Stellated Hexahedron
Tatiana Ponomareva
Ted Stokes
Tendayi Mawushe
Thomas Dingemanse

[@bubinasuit]

I literally did a science fair project where the result of “can a genetic algorithm learn how to arrange solar panels efficiently” was “this genetic algorithm learned to exploit my raytracer”

[@planktonfun1]

hmm nice

[@mistymysticsailboat]

could you take a picture of it?

[@Vaaaaadim]

Hey! But perhaps this can still be of some use. If it cheats the system, then you've found something to bug in your system(which you may not have found otherwise), and if it doesn't then you might have an actually useful result.

[@advorak8529]

Good! You learned way more than you bargained for!

[@robertvralph]

@@advorak8529 hahaha... best comment.

[@marccram6584]

There was an experiment where crows were rewarded with a peanut for picking up trash. For each piece of trash the crow deposited in a special bin, the crow received one peanut. This worked great for a while until the crows ran out of trash and then the crows decided to hang around trash cans and assault humans who were trying to throw trash away. The crows would harass the people until they dropped their trash and then go get a peanut. Essentially the crows were taught to mug humans.

[@fergochan]

This is probably the best story here because it shows how universal this behaviour is. It's not just computers or AIs that do this.

[@blahblahblahblah2837]

It's entirely, intrinsicly biological behaviour. Simple action = reward. Humans would be no different, except that we have some concept of morality/longterm consequences. Although, a human probably _would_ do exactly this if there were very little consequence and no perceived better alternative way to obtain food. I guess it's not much different to a really persistent begger.

[@gordontaylor2815]

@@blahblahblahblah2837 IIRC, the movie "The Terminal" has a sequence where the main character (played by Tom Hanks) does something similar to the experiment mentioned in the OP in order to get food for himself. (If you watch the movie, you would understand that the character is doing this for the reasons you describe in your own comment.)

[@Mythologiga]

Similarly, in India under british rule, there was a snake problem. To address it, the british started to offer rewards for any killed snake. At first, it worked, and people were genuinely capturing snakes. But after a short while, some people started to breed snakes instead in order to get more rewards. The program had to be cut after this was discovered. The new snakes breeders, now stuck with worthless animals, released them in the wild, erasing any gains made by the program.

[@Hangman11]

@@Mythologiga Ergo Humans are Crows

[@valshaped]

So an A.I. is an extremely skilled, unsupervised toddler being paid in candy to do a task

[@LuaanTi]

Yup, external rewards work about as well for AI as they do for humans; you find the easiest way to earn the reward, and don't bother doing anything without a reward :P

[@renookami4651]

Yes, and the ones supervising the task are people who are bad at stating exactly what they want, aka humans, so the A.I is doing exactly what they're been asked for as textbook definition goes...Yet the humans complain. xD

[@carlosvega4795]

This is what most pople refuse to understand... Except for the "skilled" part. They have no skills, they only do what they are designed to do, exactly as a tiger is for hunting. They're not skilled, it's just natural for them, since we are not hunters their ability seems "skilled" for only a skilled learned and expert human would have that same ability to blend, prowl and efficiently take down a prey sometimes without it even noticing. So goes with AI, they're not smart, they're not skilled, they're just naturals to the job, your human feedback is just a way to tell them "make this part bigger... no, wait, smaller, too small, a bit bigger... Good!, now do that exact thing but at the other side of the room since you were in the wrong place the entire time". Good thing AIs don't get annoyed :P

[@gavinjenkins899]

@@carlosvega4795 lolwut, tigers are extremely skilled hunters, AI is smart literally by definition, AIs could certainly become annoyed, what on earth are you on about

[@monstertrucks9357]

@@gavinjenkins899 He's right. AIs can never become annoyed. They are 0% sentient, 0% alive, and always will be. They will never feel annoyance, not even in the slightest degree -- it is not existentially possible.

[@matesafranka6110]

My algorithm teacher used to say, "The best thing about computers is that they do exactly what you tell them to. The worst thing about computers is that they do exactly what you tell them to."

[@famitory]

it would seem that since AI is excellent at finding loopholes, a good application for AI would be finding loopholes in systems we'd rather didn't have loopholes.

[@tetraedri_1834]

I guess it would find loopholes of the reward function before it finds loopholes that we want it to find.

[@anduro7448]

video game bugs?

[@advorak8529]

How about an AI to find holes in an AI - is that the halting problem all over again?

[@MidnightSt]

and that system would learn to find and exploit loopholes in our definition of loophole.

[@NicolayGiraldo]

@@tetraedri_1834 Just as humans find loopholes in the loopholes. Automating of testing previous to human exposure can reduce huge amounts of legislation.

[@NoahTopper]

That program that deleted the text file terrifies me deeply.

[@GdotWdot]

I read a story in a computer magazine long ago, about a hacking contest that didn't go too well. The goal was to store the given payload on the server. That's it. The rules were so vague one crafty participant just copied the payload at the end of an URL pointing to the target server, so that it would be saved in the logs.

[@JamesPetts]

@@GdotWdot I remember once when I first started my masters' degree, there was a treasure hunt in the MCR, where new students at the college competed for a prize by accumulating points assigned to various items (a small number of points for something like a conker, some more points for some college crockery, etc.) brought to an introductory party at which the prize would be awarded. The item with the highest number of points was the glasses belonging to the (notoriously difficult) person who maintained the college's IT systems. The idea was for people to find ingenious ways of trying to steal them (temporarily). I decided that the best way of scoring these points was simply to invite the person in question to the party, on the basis that, if he came, he would inevitably come wearing his glasses. Unfortunately, he did not accept the invitation, but the idea was a sound one.

[@ukaszlampart5316]

I think it is a big mess-up on creators part to even allow generated program to perform side-effects (they probably did not use subset of a language, but allowed for generating arbitrary code). In principle you could develop true AI this way, given enough time and computational power (I do not say it is practical, because probably all world computers at once, would not be powerful enough to get any meaningful results, space of possible programs being too large to narrow to the few which we would call "smart")

[@user-qw1rx1dq6n]

Direct quote from the AI before it pulled that: I’m gonna do what’s called a pro gamer move

[@MainGoldDragon]

The AI has run the numbers and the best solution it has found for saving the planet is deleting half the humans.

[@plcflame]

I wish there were movies like that. AI isn't evil, it's just extremely good in doing what you asked for

[@Khaos768]

There are movies like that. All movies where an AI is trying to enslave and control humanity are basically that AI's efforts to follow the order that it was given to protect humans. Step 1: AI finds out that the greatest cause of harm against humans is other humans. Step 2: AI enslaves humanity to control humans and prevent them from hurting each other. There are even games like that, e.g. Cortana in Halo 5.

[@mimszanadunstedt441]

Human: 'Make me into superman' AI: Roger Roger. *Kills you and makes a Superman sculpture out of your corpse in a pose from a comic* If you wanna see something like task misinterpretation there was a brainwash episode in mlp where the resulting behavior was as such. Season 6 Episode 21. And if thats too light for you, then the Franken Fran novel is a 'monkey's paw' parody and horror manga series (with some amazing and gruesome imagery).

[@djjoeray]

Reimagining the 'deal with the devil' plot as an AI problem....I like it

[@hugofontes5708]

@@djjoeray AI is just a genie we can't have provide all our wishes... Yet

[@hermask815]

A movie franchise with the „what you wish for” theme are the ones called wishmaster . A djinn misinterprets wishes on purpose. #1 & #2 are ok b-movies .

[@ZardoDhieldor]

The hacker heart inside me just loves how AI creatively circumvents the restrictions/goals put in front of it. The boat example just makes me smile everytime!

[@NoNameAtAll2]

Is your avatar from To The Moon?

[@ZardoDhieldor]

@@NoNameAtAll2 Yup. My user name is inspired by To The Moon, too. It's Latin for "moon bunny".

[@mimszanadunstedt441]

idk if its circumventing so much as thinking it did a good job

[@yondaime500]

What is interesting is that the AI doesn't even know it is circumventing the rules, because technically, it isn't. It can't tell a bug from a feature. As far as it knows, it is only doing what it was told to do.

[@ZardoDhieldor]

@@yondaime500 I would go so far and say that even the term "artificial intelligence" is terribly misleading. AI is just as stupid as computers always were: Simply doing what it's told.

[@rentristandelacruz]

There is a program known as Polyworld. The idea is to evolve artificial creatures via natural selection and evolution. One creature evolved a behavior of producing an offspring then eating it. The programmer initial forgot to add a cost when producing offspring so the cannibal creature essential has an unbounded source of food (it's own offspring).

[@RobertMilesAI]

It's on the list! Number 23, Indolent Cannibals tinyurl.com/specification-gaming

[@lordkekz4]

I've always wanted to make a game like that. I gotta check that out.

[@BarbarianGod]

I think that was in a Rick & Morty episode

[@tim40gabby25]

I played laserzone with a mate where we both hid squashed in a tiny barrel and alternated shooting each other. We got the first and second highest scores, picked up the cash prize, split 50-50 and departed like Kings, having earned approximately £5 for 90 minutes work, congratulating ourselves with a £2.00 celebratory beer, each, thus requiring us to restart the process as we had just enough for 2 tickets. By the end of the day it was like shooting very drunk fish in a barrel, yet we concluded we students were one up against the Universe.

[@MouseGoat]

@@BarbarianGod the ep where Beth finds out she has left her childhood freind to rot in a magical fantasy world? XD

[@Xelbiuj]

The world turning to gold is some XKCD "what if" stuff.

[@RoberttheWise]

I would't be surprised if the XKCD guy had contributed to that thread.

[@drdca8263]

Iirc Ryan North of qwantz contributed to a comic book about some people in the space future found a planet where a king Midas happened, and they retrieved “the Midas flesh”, keeping it suspended in a vacuum with magnets or lasers or something to keep it from touching things, in order to potentially use it as a weapon, or something.

[@Theraot]

The question is also right up the alley of worldbuilding dot stackexchange

[@OlleLindestad]

@@drdca8263 The title of that comic is "The Midas Flesh", and yes, Ryan North wrote it. It also has a sentient velociraptor professor in it for some reason.

[@diabl2master]

Totally. I'm wondering if they'd do "what if everything you touched turned to gold" but it might turn into quite a horrific description.

[@ChrisD__]

Robert: "Give it a small reward for every frame the pancake isn't on the floor" Me: *already laughing hysterically*

[@seriouscat2231]

It could logically just freeze in place if there weren't any other conditions.

[@badrequest5596]

some time ago i was playing around with machine learning in unreal engine and i gave it control of a plank and put a ball on it. to make it simpler i just had the ball roll in on axis, so left or right, and the objective was not to drop the ball. it learned the trick really really fast. the best way to not drop the ball, was not to move the plank at all

[@badrequest5596]

after that i had the same AI learn how to drive a template car in the game engine and not hit walls. so what's the best way to not hit a wall? that's right! dont move at all! . i felt like an idiot for not seeing that one coming

[@MidnightSt]

i maintain that this one was embarrassing. give it a reward for the sequence "side A touching the inside of the pane, in the air, side B touching inside of the pane" AI would most likely still hack this one too in some hillarious way, but at least it wouldn't be an embarrassingly stupid mistake done by the researcher.

[@cappuchino_creations]

I kinda expected the whole plan to be just shaking in spacsm, because that would make the pancake jump all the time and have infinite airtime for each frame not touching the pan

[@Merchandise7x]

"I knew everyone would die. I just wasn't sure what would kill us first." Quote of 2020.

[@Dorian_sapiens]

First Runner Up: "It's going to get worse before it gets better."

[@veggiet2009]

Spoiler alert: everyone dies

[@veggiet2009]

"Everybody knows that everyone dies, and no one knows that more than the Doctor..."

[@Tonatsi]

veggiet2009 *”The laws of time are Moffat’s and **_They will obey him!”_*

[@crunchyfrog555]

That's pretty much what I think about the coronavirus and the impending doom of climate change behind it. "phew, the coronovirus is over with....." and now what about the few years we have left to get climate change under control and we aren't doing?

[@nowheremap]

AI has already surpassed humans on malicious compliance.

[@benwilliams5457]

AI have not yet surpassed humans - humans are usually empathic enough to be aware of their malice and often attempt to conceal. The specification errors presented here are easily discoverable. The obvious solutions are to invoke an iterative process of refining the specifications until the path of least resistence/greatest reward is the one the programmer desires. In consequence.. 1.) the AI is now motivated to select modes of compliance which are less easily discoverable 2.) the AI effectively trains the programmer to specify the solution exactly. It seems that this might not be so bad; If the AI can find a cheat that gives results indistinguishable from the 'correct' result then it doesn't matter how. If the AI can iteratively train the programmer then the solution is eventually perfectly described, obviating the need for the AI solution. Of course, both of these outcomes probably require that the whole universe of all possible knowledge is catalogued, but then this is the root problem of, and solution to all hypothetical AI safety issues.

[@ZenoDovahkiin]

No, actually. There is nothing malicious about it. AI is rather accidentally glitch hunting.

[@benwilliams5457]

@@ZenoDovahkiin Malice implies agency, but the whole point of an artificial general intelligence - at least from the non-scientific cultural viewpoint - is that it has, or appears to have agency. Thus, characterising its actions as accidental is not appropriate. However, malice also implies a judgement of good vs bad so I agree that the neutral "glitch hunting" is more suitable than "malicious compliance". This leads me to wonder about whether a sort of moral code could be added to the AI reward function; e.g. a reduction in the value of the reward for any action for any harm that that is caused, or not avoided compared to an alternative action, or a null action. It would be rather like Asimov's Three Laws of Robotics. There is an obvious problem with defining "harm" meaningfully, but there are some simple initial approximations: for example: estimate the total number of healthy humans (within the scope of the AIs senses or field of action) and reduce the reward if the number is lower after an action vs. inaction. The "glitch hunting" nature of AI will doubtless look for short-cuts - attracting more humans with a delicious smell to offset any losses - but the code can be refined as the AI finds loopholes. This approach reduces the problem of hard-programming in every possible contrarian action of an AI to hard-coding every aspect of a moral code, such as we all have built in. Of course, this would fail utterly since no-one can describe precisely what their moral precepts are without listing by example or falling back on poorly defined aphorisms like "Do no harm", despite actively using that morality to police every aspect of our lives. If some one did make headway in an absolute definition of morality, any AI would tear it to shreds because human "morality" is an unevenly-applied, self-serving mess of contradiction that are made up of learned responses, laziness and enlightened self interest. It won't do much for AI safety but it might teach something about human-intelligence-safety.

[@Scotch20]

The AI is just doing what you told it do to

[@anandsuralkar2947]

True

[@chrisjones5046]

I teach about this in one of my lectures, it's a interesting sub-set of Goodhart's Law "When a measure becomes a target, it ceases to be a good measure". It turns out humans have been dealing with this one for a while. It sort of makes the AI more human.

[@freefrag1910]

truly a widely applicable thing. the same happens for "industry standard benchmarks". Once a benchmark program is accepted by the community you can see how the manufacturers fine tune every bit of the hardware to get the highest results, often sacrificing real life results.

[@Redmanticore]

@@freefrag1910 like wolkswagen. "Volkswagen had intentionally programmed turbocharged direct injection (TDI) diesel engines to activate their emissions controls only during laboratory emissions testing" "during the bank's (European investment bank, EIB) annual press conference on 14 January 2016, the bank president, Werner Hoyer, admitted that the €400,000,000 loan might have been used in the creation of an emissions defeat device.[348] "

[@buzz092]

Chortled at the "YEET"

[@Bruva_Ayamhyt]

**chortles heartily**

[@Brindlebrother]

_chortling continues_

[@RobertMilesAI]

[@piemaster6512]

At 5:02 I absolutely lost it. I would feel personally attacked if my program did that to me. Fantastic!

[@massimookissed1023]

I'm surprised it didn't also stick its middle end-effector up at the programmer.

[@Jacob-pu4zj]

It was absolutely hilarious. They left it far too many degrees of freedom.

[@TonyHammitt]

Really needs the "Thug Life" glasses at that point, or a microphone to drop...

[@freefrag1910]

@@Jacob-pu4zj but that is the beauty of finding sometimes brilliant solutions

[@Randgalf]

The funny thing is that this made me realize I often reasoned (and executed accordingly) like an AI when I was a kid. I had a knack for zoning in solely on the stated objective of any task with no concern for any underlying purposes, much to the chagrin of any present peers or teachers which I always found confounding.

[@prolamer7]

interesting insight!

[@germimonte]

when the arm fliped the lego i just lost it

[@chandir7752]

There's something about these AI's when they act they way they do that has me rolling on the floor. Like they look so stupid but are actually extremely good at doing what they are asked to do. That pancake throwing technique lmfao imagine if they tried this in real life with an actual robot arm and suddenly the thing tries to set up a new pancake throwing record... I can't hahaha

[@martiddy]

AI be like: "look at me master!, I'm doing a good job" *proceeds to yeet pancake*

[@FlesHBoX]

I mean, the problem really is US and not the ai. We are clearly not giving them proper instructions because humans rely on a lot of implied and inferred meaning. Even the most specific of human spoken languages are nowhere near as precise and specific as even a rudimentary programming language. It's such a fascinating thing. I imagine that the process of creating AI has taught us a lot about how humans think.

[@plcflame]

@@FlesHBoX Strange enough to think that maybe, before we create an superintelligent and powerful AI, we need to create another language and adapt our brains to this.

[@squirlmy]

this must be your first video with Robert Miles? Maybe you are new to AI altogether? I'm just an enthusiast, not an academic, but the crazy things AIs do is the very first thing I learned about the subject, years ago. It's really strange to me to read someone mention this here as if it were at all unusual or unexpected. Yes, all computers, not just AIs, do exactly what you tell them quite literally.

[@kaitlyn__L]

@@squirlmy I learned about the "grow tall and fall over" thing over a decade ago from BBC4 and a few of the video game playing ones, but these other examples were new to me .

[@TrimutiusToo]

Code bullet reference... He actually made that agent training thingy public if anybody want to try, though it doesn't have that physics bug anymore.

[@knight_lautrec_of_carim]

it's more of a direct mention than a reference

[@TrimutiusToo]

@@knight_lautrec_of_carim reference usually means any kind of mention.

[@TheAechBomb]

I think it still has the physics jank, he just limited the joints so it couldn't push the legs into its body

[@phiefer3]

@@TheAechBomb iirc he never "fixed" anything to prevent that, he just made it so that the agent died if any part of it besides its legs touched the floor

[@underrated1524]

​@@phiefer3 I mean, that is a valid solution IMO. He wanted his creations to walk on their legs, so he made that a requirement for survival. ("I've found that to be a pretty good motivator." ~CB) I mean, what else was he going to do, tinker with Box2D itself? He has enough trouble using the darn thing, let alone debugging its bugs XD

[@karapuzo1]

From the list: "CycleGAN algorithm for converting aerial photographs into street maps and back steganographically encoded output information in the intermediary image without it being humanly detectable." That's great, I am not even mad. Second place goes to "Genetic algorithm for image classification evolves timing attack to infer image labels based on hard drive storage location"

[@renakunisaki]

I like the one that figured out that lesions are likely to be cancerous if there's a ruler next to them.

[@Waffles_Syrup]

Reminds me of tom7's nintendo learning program that learned that the best way to not lose points in tetris was to just pause the game

[@renakunisaki]

I love the ones that learn to break the game.

[@khatharrmalkavian3306]

Not playing Tetris is the same solution I found to Tetris.

[@AdmiralJota]

Joshua?

[@Technodreamer]

The Midas story doesn't generally end with Midas dying, it ends with him begging Dionysus to take back the blessing. The idea of his daughter getting gilded is also a MUCH MUCH later addition to the myth, it's not in the original at all.

[@CraftyF0X]

They sure as hell good in "thinking out of the box". They percieve no bounds for solution but the rules.

[@chrisofnottingham]

The thing is, setting the wrong targets is what happens all the time even without AI.

[@WarrenGarabrandt]

Humans are amazingly great at setting the wrong target, and then maximizing for a metric instead of actually improving anything.

[@TlalocTemporal]

@@WarrenGarabrandt -- Sounds like every political and economic system past the commons.

[@robertvralph]

@@WarrenGarabrandt This literally explains so much of the failings of bureaucracy

[@IPlayWithFire135]

@@WarrenGarabrandt Climate change wants to know your location

[@GeneralSorrow]

Game achievements.

[@unvergebeneid]

"I knew everyone would die, I just wasn't sure what would kill us first." I think this sentence has much broader applicability than it might at first seem 🤔

[@mimszanadunstedt441]

'Minimize heart disease' *all humans r now dead, did i do a gud job master*

[@MrRyanroberson1]

"Maximize the duration of human lifetime spent not suffering from heart disease"

[@lazergurka-smerlin6561]

@@MrRyanroberson1 *I made everyone reproduce as much as possible and made space colonies to support the surplus human population, did I do a gud job?*

[@ZardoDhieldor]

@@lazergurka-smerlin6561 My immediate response would be to make it maximize the average human lifespan (in quality adjusted life years, do you know QALYs?). But I guess that can be circumvented as well...

[@goonerOZZ]

@@MrRyanroberson1 turn everyone into a comatose state and feed them all with tubes

[@peterbonnema8913]

As a young kid, I was about to grab a cookie and eat it. My mother saw that and yelled "Don't touch those with your hands!". So opted to simply bend over and eat the cookies directly from the table. I was such a rebel

[@lordkekz4]

Found the AI.

[@BeHappyTo]

​your mom was the issue, not you

[@Anonymous-sb9rr]

Were you supposed to eat cookies with fork and knife?

[@leddaudet2350]

Cyborg confirmed

[@advorak8529]

Refke van Lavieren Knife and fork? Barbarian! Chopsticks is what you use, _obviously._

[@inyobill]

"We're going to get rid of all of the software engineers, the users will be able to just tell the computer what they want, and the computer will produce the correct software." (I was hearing this would be reality in 1990 by 2000, the end date keeps getting pushed back. "Oh, you have customers that are able to specify their non-trivial product in sufficient detail and rigor that the machine will produce the desired product. Hunh." The hardest part isn't the design and code, by a competent team. The hardest part is specifying the system behavior to produce the desired end.

[@ABaumstumpf]

There are very good code-generators for a lot of simple tasks, there are also generators for really complex things, even the design. But it always needs quite a lot of human work to be even close to useful.

[@inyobill]

@@ABaumstumpf Interesting observation. I was in the world of hard real-time, where predictability was a critical system characteristic, so my view-point is probably a bit skewed, even beyond the egotistical ("How could a machine possibly replace someone as brilliant as I?"). User Beta testing was/is not an option.

[@ABaumstumpf]

@@inyobill I only have to deal with very little hard-realtime systems, but most is not hard (but not as lenient as soft either). The only thing we have that is generated a the api-functions for database-calls, and some base-classes that only hold data. But in general code generation has come a long way, google and amazon use it extensively, alongside a lot of meta-programming, but it has to be very well structured and needs a lot of specifications.

[@abdulmasaiev9024]

"customers that are able to specify their non-trivial product in sufficient detail and rigor that the machine will produce the desired product." - well, see, it's for machines so it needs to be unambiguous. As to not reinvent the wheel let's use one of the unambiguous ways to tell computers what they're supposed to do that are already there like a programming language like C++ and wait a minute this software engineering replacement is literally just software engineering

[@musaran2]

Users don't even KNOW what they want.

[@mrmonkeboy]

This is sort of like F1 racing - the FIA impose some very strict rules and each teams tries to bend each rule so their car is the fastest. It's a totally open system in which there is relativly little cheating, but a lot of rule bending going on. Every year a team comes up with something on the car that is *almost* illegal...

[@tristanridley1601]

It's an adversarial system as well. Every year the FIA makes new rules, no? "So we notice you started doing this thing. That thing is now banned."

[@cryoshakespeare4465]

If some sort of doomsday-exterminator AI was cobbled together, you'd have to imagine that our best efforts wouldn't be spent fighting it, but determining what absurd and benign state would reward it most.

[@musaran2]

Show him some war movies. If it's smarter that that, show him how toxic we are to ourselves and how adversity unites us. It should promptly select non-interference as the most efficient strategy.

[@memebro8703]

Reminds me of the killbots from futurama.

[@comet.x]

@@musaran2 solution: obtain a bucket of red paint and a flamethrower. Gain infinite rewards

[@Redmanticore]

the AI will just move to far corner of universe to escape us

[@Trazynn]

4:22 I think this is a perfect example of what AI is great at. Exploring the fringes of a system and finding loopholes and extremely efficient ways of working that humans would never think of.

[@europeansovietunion7372]

My favorite is the use of camera perspective to trick the human that was supposed to train the AI. Instead, the AI basically managed to train the human to validate what the AI wanted.

[@LetalisLatrodectus]

And the scary bit is that the AI isn't even malicious or anything. It's just doing what it really thinks we want to see.

[@JaneDoe-dg1gv]

A classic example of how it is best to assume ignorance before malice.

[@PinataOblongata]

@@JaneDoe-dg1gv Otherwise known as "Hanlon's Razor". For a given value of "best" you would think we could come up with a better algorithm for calculating likelihood of either behavioural driver, i.e., if it's a voter, weight ignorance, if it's a politician, heavily weight malice/self-interest, and if it's Trump, assume maximum levels of both ignorance AND malice ;)

[@MouseGoat]

my main fear finally brought to reality, and this is a dump AI if we ever build a strong one... well we f****d.

[@Zer0Spinn]

@@MouseGoat I was gonna say the same. Imagine this shit with an AI that controls the power grid, military systems, or god knows what. We have a lot of work to do if we really want to put this things to work...

[@p0t4t0nastick]

Robert this is crazy. Just earlier today I went through my subs you and noticed you haven't uploaded in months. So I went on to google you and found out you had started a podcast, which afaik you never mentioned in any of you past uploads. I wish I had known this, I would have started listen to the podcast back then. Btw the podcast episodes are available only until #52. Anyway super psyched you are back again and to check out the podcast.

[@ricardasist]

Whaaaat? He has a podcast?

[@Jens-B]

Whats the podcast called? I Had no idea! Would have loved to check it out.

[@RobertMilesAI]

Hah! I'm sure I mentioned it somewhere? I don't push it super hard on the channel though, because it's really aimed at researchers rather than the general public. It's often pretty technical. But here's the link if you're interested: alignment-newsletter.libsyn.com/

[@jonas2560]

@@RobertMilesAI I am not a researcher but I am going to give it a listen. Do you know if the episodes before the #52 are available somewhere ?

[@RobertMilesAI]

@@jonas2560 The podcast started at #52! Before that it was The Alignment Newsletter, an email newsletter with no audio. You can read all of those at rohinshah.com/alignment-newsletter/

[@kevinstrout630]

I recently made a tic-tac-toe AI for a school project. Nothing fancy just a recursive game engine. Turns out that because of how I had the weights, it wouldn't take the win as soon as possible, but would instead keep playing as long as it had a way to win because it figured more possible futures where I can win > a single future where I win

[@LazarethPrime]

So you're saying your AI likes to play with its prey for as long as possible.

[@KingDonutz]

One of my favorite examples is some machine learning made for beating original Mario. It was told live as long as you could. This lead to it pausing whenever it was about to die and refusing to unpause

[@TheInsideVlogs]

shameless plug: we built gym environments to study specification gaming where you can play the noodle game of this video as a human and see if you hack games as well as the AI. just google "quantilizers github" and you'll find it.

[@ChayComas]

"The robot just throws the pancake as high as it can" LMFAO

[@RoberttheWise]

Normal person: (sees AI making dumb mistakes) "Haha, dumb computer. What is everyone so afraid of? This thing could never take over or destroy the world." Safe AI enthusiast: (sees the same thing) "Oh no, this thing will take over the world and destroy it if we actually let it do serious thing. We should be really careful with it."

[@kaitlyn__L]

exactly!

[@kaitlyn__L]

especially because these kinds of outcomes are already potentially affecting people's lives where software to help determine criminal sentences are involved?

[@kaitlyn__L]

which is some dystopic sci-fi shit btw, but at least that would have the courtroom painted black with neon lights, in the real world they look exactly the same but are shifting their operation

[@RoberttheWise]

@@kaitlyn__L I feel like sci-fi writers did us all a big disservice with their depiction of rogue AI. Gave us as a society a major blind spot for technology going rogue while still being very dumb.

[@CoryMck]

_"in real life you can't just be very tall for free"_ Guys lying on their tinder bios: *Says who‽*

[@stardustreverie6880]

liked for the interrobang usage

[@fredericapanon207]

So what is the Unicode for the interrobang? 'Cause I want to able to use it too!

[@Kratokian]

You know, a good chunk on the list, I can't tell if the AI actually did anything wrong either. There's definitely something to be said for expectations as well as everything else. "Bicycle AI learns to circle around goal" Isn't that good if it's improving stability? Oh, I guess it never went 'towards' the goal at all Boat Race robot, people definitely value playing some games like this, so if it looks bad isn't it just because the material was too simple? Stenography, that's legitimately just a useful system that doesn't take any extra data or overhead to keep track of maps, when used in an existent system, good job robot!

[@megalocerus1573]

Pretty similar to human response to tax incentives. Very hard to specify the correct reward! I knew someone applying for school aid for his daughter; he discovered that two students in the family qualified for more aid for one, so he signed up for community college courses himself. Before there was artificial intelligence, there was natural intelligence.

[@blahblahblahblah2837]

This is EXACTLY what I first thought too. Whenever there is any government incentive, rebate or aid, it very quickly gets exploited.

[@reddragonflyxx657]

Absolutely, but I don't think the father signing up for college courses is necessarily a bad thing either.

[@kittybeans8192]

Wait, what exactly IS touch? Depending on how we answer that, only an atom-thin layer of material turns to gold, or the entire universe turns to gold. It can't be the former, so we can assume that the true function of the ability is more like a flood-fill, where whatever he touches turns to gold, and whatever touches that also turns to gold, and so on. So yeah, the whole planet including its atmosphere should become gold, and probably the solar system if we count all the photons around and... yeah entire universe becomes solid gold at the speed of light. Should've wished for "whatever I touch turns to gold only in compliance with my expectations and desires". Maybe that's safe...?

[@garret1930]

What happens when your desires are contradictory? As happens very often with humans due to us having multiple goals.

[@AnthonyBecker9]

For more along this train of thought, I recommend the book Superintelligence by Nick Bostrom

[@georgew.9663]

3:32 an accurate representation of my journey through life thus far

[@ksdtsubfil6840]

Robert: Everyone will die Me: Great. Love you, Robert! Keep up the good work.

[@rancidmarshmallow4468]

I'm not an expert, but I believe in the midas scenario 'touch' is interpreted as 'midas is thinking about the fact that he is touching this thing, so it turns to gold' or at least, I've read versions where he is initially very happy and touches various objects, and his clothes don't turn to gold until he realized he is 'touching' them, and his daughter does not turn to gold until he realizes he is touching her. since midas never thinks about that fact that he is touching the ground or air, it never turns to gold.

[@feha92]

Correction: it is not everything that collides with him that turns into gold. It specifically states _everything he touches._ So he can still breath, but any air that touches his fingers (there is a reason it is called a midas finger, or green thumb) will continuously turn to gold (no idea if it welds with the prior gold that just fell, or if it becomes atomic gold dust oozing out of his fingers). And he can still eat, the implements used to do so will however become gilded. Similarly, if he scratches himself, he turns into gold. edit: also, the ground most certainly is not a single object. *Maybe* I can agree to it if you refer to a single casted slab of concrete used as ground, but even regular stone is filled with weird stuff I dont know about cause I am not a geologist, and earth has clumps of dirt aplenty.

[@tricky778]

Then can't he carry gold claws with which he can hold things?

[@columbus8myhw]

Joke: Genie: "You get one wish" Midas: "I wish anything or anyone I touch turns to gold" Genie: "Ha, fool, you do not realize -- wait, what?" Midas: "I know what I said"

[@Ghi102]

I might be dumb, but I don't get the joke?

[@martiddy]

Poor Midas can't even touch his own pp.

[@user-cn4qb7nr2m]

How nice of you to warn us first. "Joke:"

[@wood_croft]

@@Ghi102 Genie in a bottle. Rubbing counts as touching.

[@drdca8263]

Ghi102 the joke is that in this variant, Midas *did* want to turn the people he touched to gold, and the genie was surprised. It goes against expectations, and also might lead to the reader missing the “or person” in the first line, and then realizing it when the genie does. Produces a kind of double take?

[@atavy]

That yeet part had me on the floor. xD

[@Theraot]

6:56 last I argued something like that could happen, I was faced with the reply "that would require perfect knowledge". I stopped arguing. Now this list might help me argue in such situations.

[@morkovija]

I rofled at the original audio reconstruction!x)

[@mhorzic]

Missed you man, love your research stories.

[@christopherg2347]

"Artifical Intelligence beats human stupidity by far!" You know, when I heard it, was was meant the other way around (AI still being pretty stupid).

[@rofl22rofl22]

This was informative and also by far the funniest thing I've seen today. Thanks Rob.

[@Xeridanus]

I remember hearing about a robotic arm that was supposed to grab an object with a claw and instead of opening it's claw (I think it was stuck or something) it applied pressure to the object it was supposed to pick up until the claw opened enough to snap around it. I can't remember all the details but I'm pretty sure it wasn't supposed to do it that way.

[@Nightcoffee365]

This is the first video of yours I’ve ever seen. In the first two minutes you meander off deeply into frankly overwrought ramifications of a mythical story, and then literally interrupt yourself for a scientific analysis of your diversion... I subbed immediately! Where have you been all my life?! It’s like you made this for me specifically.

[@griffinbeaumont7049]

with every video, you're becoming more and more my hero and it's kind of the best thing

[@Alorand]

So "monkey's paw" is the very nature of how AI behaves? Why does this cause me to feel profound anxiety?

[@michaelspence2508]

Because you're paying attention.

[@GuRuGeorge03]

we act like AI as well. You just don't realize it. But here is one example to start with: Sex was invented for reproduction. We humans invented condoms to exploit sex for pleasure instead of reproduction. You see this pattern in literally everything we do. It just isn't as obvious as with the AI that you see here, because you're so used to thinking that whatever you are doing is "more intelligent"

[@derkach7907]

Mentions codebullet. Ah I see, you are a man of culture as well.

[@TheRABIDdude]

These were so fun & interesting to watch! Please do a second episode with more examples!!

[@flymypg]

I was an undergrad at UCSD during the early back-prop days with Bart Kosko and Robert Hecht-Nielsen, and developed an abiding fascination with following ML developments, though only as a hobby, never professionally. Robert Miles is, to me, one of the best at identifying and explaining both the fundamentals and some of the "curious corners" of current advancements. However, one YT video every 4 months is not nearly enough. I can haz moar? Puhleez? Nice haircut! Truly a good one in this era of online tonsorial self-mutilation.

[@Science4Allfrancais]

COVID under-reporting is arguably a more realistic version of the bucket robot's specification gaming...

[@revimfadli4666]

Or over-reporting, if hospitals were to get subsidized

[@joshuascholar3220]

@@revimfadli4666 I have been told that Alameda county's claim that it had been over-reporting Covid is actually an attempt to cover up mismanagement at nursing homes that lead to cases. And it seems likely to me because the claimed amount of over-reporting is impossibly high.

[@TheAgamidaex]

Fantastic intro. The whole video is hilarious. YEET the pancake!

[@jackshae7]

Literally it takes a lot for me as a person to want to like and subscribe in general, especially when watching a new channel, but I am two minutes in and I wanted to like, subscribe and even comment. I love the humor and excited to watch the rest of your videos!

[@justrecentlyi5444]

Well I've just been randomly recommended this video, first time seeing the channel and something really interesting!

[@pafnutiytheartist]

7:36 okay that's an actually scary example.

[@MushookieMan]

To solve the specification gaming problem, we only need to create AGI that can interpret what we meant, instead of what we said.

[@plcflame]

Or we can create another language and another way to think, more specific, before creating AGI

[@hansisbrucker813]

@@plcflame Like Lojban?

[@davidwuhrer6704]

Ah, the old DWIM problem.

[@IstasPumaNevada]

I always love when you're featured on Computerphile. I don't know why I didn't think to check if you had your own channel. Absolutely subscribed.

[@sagacious03]

Very fun, entertaining & informative! I'd love another list like this!

[@Horny_Fruit_Flies]

What if King Midas just wore gloves all the time, the gloves would turn to gold, but then he would be touching everything with the gold gloves and not his skin

[@trelligan42]

Or using a spoon/fork to handle food (unless touching his tongue counts...)

[@Sharklops]

how would get get the gloves on? the moment he touched them they would turn to solid gold

[@Horny_Fruit_Flies]

@@Sharklops well, then maybe a servant could cut the fingers off the gloves, put sausages or something inside the fingers so they have the proper shape, then King Midas would touch the cut off fingers turning them to gold and locking them into their shape, the servant would then take out the sausages and King Midas could put the gold gloves fingers on his own fingers, thus creating a layer of protection to touch things

[@iddomargalit-friedman3897]

You just described the plot of "Frozen"

[@MariaNicolae]

@@Sharklops The gloves could perhaps be thin enough so that even as gold they were flexible enough

[@LateralTwitlerLT]

"oxygen molecules will turn to gold atoms, [...] and I guess the ground is just one solid object" kay then

[@sylvainprigent6234]

I really think that the pancake throwing thing is amazing. I just found your channel after having seen quite a few of your computerfile videos on the AI safety subject. And I find it a very interesting topic although I still only know what you teach in these few videos. Yet this rejoins phylosophical (and mathematical) concepts about how to define things. How do we define properly what we mean, all the subtle implicit and ambiguous part of the language and what do words mean I'll sure be watching more

[@lordkekz4]

Finally another video! Keep up the great work!

[@TiagoTiagoT]

Reminds me of that time when they wanted to end a snake infestation and so they offered a reward for every dead snake; so people started farming snakes; and when the government found that out they canceled the reward program, and so all the farmed snakes were released, making the snake infestation even worse than what it was before.

[@halyoalex8942]

The Cobra Effect :D

[@fletchermorgan8351]

I played Strategy Challenges of the World in 1995! Thank You Miles!

[@RobertMilesAI]

Found 'em!

[@rgoodrick]

I can't wait to see what you are working on in 20 years... your seamless weaving of philosophy and technology is second to none!

[@arw000]

Welcome back! We missed you!

[@TrimutiusToo]

Code Bullet actually made that engine with learning agent running away from laser kinda public... Though mentioned bug with physics engine was fixed of course...

[@s6th795]

7:30 me removing my tasks from the Scrum board

[@PMA65537]

Yossarian moved the bomb line.

[@thrallion]

It's always a wonderful day when a new Rob Miles video comes out

[@danielescotece7144]

Watching your video on gradient descent on computerphile minutes after studying the Conjugate gradient method for solving linear system is sooooooo satisfying

[@jorice1592]

The Midasocalypse! That would be METAL. Literally 😂

[@davidwuhrer6704]

That's heavy, doc.

[@livingbeings]

This video is hilarious and your hair looks great

[@mokopa]

I was so fascinated by [the contents of] this video that, at the end, I snapped out of a trance. That rarely happens. Good job!

[@casaolec]

Thanks for one video each year. My patience ran out

[@Stray0]

YEEEAAATTT 8:32

[@alhazan]

What I'm getting from this is that we should use machine learning to discover loopholes in real physics.

[@underrated1524]

This has actually happened at least once. Google "an evolved circuit, intrinsic in silicon, entwined with physics". The issue is that "magic" solutions like this tend to be surprisingly useless, because they often depend on contingencies in the training environment, like temperature or ambient radio signals, that can't be relied on in practice.

[@Khaos768]

Robert we need YOU to present the examples in this list!!!

[@Arisemardan]

Just found your content and it's honestly great and I'd love to see more from you but some of the production can be distracting. If you'd like any advice on the camera side of things like focus or lighting I'd be happy to help! Again thanks for the content and research that went into to everything.

[@Martcapt]

God, this is just comedy gold. All of these should be rearrenged with even more of a stand up comedy feel. An AI algorith comes into my cooking class. He tries and fails to flip a pancake. I just tell him: for crying out loud, please just try and avoid dropping it on the floor as long as possible. He then proceeds to fling it at the ceeling. It got stuck there. Then he turns to me with a look of glee in his eyes: I am good AI. Task successful. Edit: Fml, he got really angry when I tried to scrape it off and tried to kill us all.

[@chakatfirepaw]

No, that's not how the story of King Midas ends: He gets the wish reversed but he from then on has to live with the ears of a donkey¹. He keeps this secret, (literally under his hat), from everyone but the man who does his hair, who is sworn to secrecy. When the barber can take keeping in the secret no more, he digs a hole and yells "the king has the ears of 'a donkey¹'" into it before filling it back in. Of course, given the kind of story this is, a cluster of reeds promptly grow on the spot and whenever the wind blows they repeat the barber's telling of the secret. 1: Not the usual word, but I doubt KZbin's bots can tell the difference between the animal and the body part.

[@deepspacewanderer9897]

It probably ends both ways, because it's mythology

[@faustovrz]

Glad you are back!

[@blahsomethingclever]

The introduction of imagining Midas’ gift was cute. Several errors though: IF the curse was anything touching his flesh was to turn to gold, assuming only those direct atoms or electron clouds of a certain limit, Midas would instantly be surrounded by a reactive chemical cloud of torn apart molecules, probably smelling like ozone, reddish black in color. Ultra fine black gold dust constantly comes off him, coating him instantly in thick layers of dust. Midas just asphyxiates. IF some level of human story telling is allowed, the curse applies to his body and mouth only. Only 'objects' turn to gold. Easy fix then: hook Midas up to a feeding tube or IV as needed, build articulated clothing he can wear, problem solved. Moreover, if the entire planets some are replaced with gold at original density, it would violently shrink almost a thousand kilometers, and become incredibly hot due to gravitational contraction. Volcanoes of liquid gold would erupt into a landscape with mountains no higher than 200 meters. And things would still look black, due to the dust!

[@Violent2aShadow]

"I knew everyone would die. I just wasn't sure what would kill us first." Hasn't that always been the case?

[@matthewwhiteside4619]

Is the outro song from the Mikado?

[@Trucmuch]

Yep. "I've got a little list"

[@BhupinderSingh-xv6dk]

Dude I had no idea you had your own KZbin channel So glad man 👍

[@AlexanderHaydenInk]

Thank you so much for sharing that list. Reminded that one of my absolute favorite things about AI is the subtle, surprising, absurdist humor that often comes with it.

[@puskajussi37]

Just for fun, how about situations where a wish/AI would go horribly unwrong? For example: Someone makes a system that has instructions to maximize world "badness" or some such. Then the system reasons 1. "Badness" = ("badness" in the end ) - (how good thigs have been) 2. The bleakest (or "baddest") world state is the heat death of universe and that eventuality cannot be avoided. Thus it creates a prosperous, long lived utopia so the eventual tragedy of all that being lost is the greatest.

[@Hexanitrobenzene]

Interesting perspective on this problem.

[@tricky778]

I have never experienced such brilliance of thought from anyone! Or at least not in a form I could perceive, I hope you are a real and kind person. I need to believe that real and kind people are this brilliant.

[@lLenn2]

All this cheating by the ai sounds pretty human to me! :p

[@AyrtonTwigg]

I could watch these examples 24/7

[@authenticallysuperficial9874]

This is an excellent video, love the examples.