Training AI Without Writing A Reward Function, with Reward Modelling

How do you get a reinforcement learning agent to do what you want, when you can't actually write a reward function that specifies what that is?
The paper: arxiv.org/pdf/1706.03741.pdf
The blogpost: openai.com/blog/deep-reinforc...
Thanks to my wonderful patrons:
/ robertskmiles
James
Gladamas
Steef
Scott Worley
Jordan Medina
Simon Strandgaard
JJ Hepboin
Pedro A Ortega
Said Polat
Chris Canal
Jake Ehrlich
Kellen lask
Francisco Tolmasky
Michael Andregg
David Reid
Robert Daniel Pickard
Peter Rolf
Chad Jones
Richárd Nagyfi
Jason Hise
Phil Moyer
Shevis Johnson
Erik de Bruijn
Alec Johnson
Clemens Arbesser
Ludwig Schubert
Bryce Daifuku
Allen Faure
Eric James
Qeith Wreid
Jonatan R
Ingvi Gautsson
Michael Greve
Julius Brash
Tom O'Connor
Robin Green
Laura Olds
Jon Halliday
Paul Hobbs
Jeroen De Dauw
Lupuleasa Ionuț
Tim Neilson
Eric Scammell
Igor Keller
Ben Glanton
anul kumar sinha
Sean Gibat
Cooper Lawton
Will Glynn
Tyler Herrmann
Tomas Sayder
Ian Munro
Jérôme Beaulieu
Nathan Fish
Taras Bobrovytsky
Anne Buit
Vaskó Richárd
Sebastian Birjoveanu
Euclidean Plane
Andrew Harcourt
DGJono
robertvanduursen
Dmitri Afanasjev
Marcel Ward
Andrew Weir
Ben Archer
Kabs
Miłosz Wierzbicki
Tendayi Mawushe
Jannik Olbrich
Anne Kohlbrenner
Jussi Männistö
Wr4thon
Martin Ottosen
Archy de Berker
Marc Pauly
Andy Kobre
Brian Gillespie
Poker Chen
Kees
Darko Sperac
Truls
Paul Moffat
Anders Öhrt
Marco Tiraboschi
Michael Kuhinica
Fraser Cain
Robin Scharf
Seth Brothwell
Kasper Schnack
Klemen Slavic
Patrick Henderson
Oct todo22
Melisa Kostrzewski
Hendrik
Daniel Munter
Graham Henry
Duncan Orr
Bryan Egan
Robert Hildebrandt
James Fowkes
Alan Bandurka
Ben H
Tatiana Ponomareva
Michael Bates
Simon Pilkington
Dion Gerald Bridger
Petr Smital
Daniel Kokotajlo
Fionn
Yuchong Li
Diagon
Parker Lund
Paul Emmerich
Russell schoen
Andreas Blomqvist
Bertalan Bodor
David Morgan
Jeremy
Ben Schultz
Zannheim
Daniel Eickhardt
lyon549
HD
Ihor Mukha
14zRobot
Ivan
Arne Strasser
Jason Cherry
Igor (Kerogi) Kostenko
Isaac Boates
Thomas Dingemanse
Davy Ker
Alexander Brown
Devon Bernard
Ted Stokes
James Helms
Matheson Bayley
/ robertskmiles

[@NoahTopper]

"If you squint, the training process is sort of like a compiler." Totally brilliant statement.

[@shouldb.studying4670]

I had to squint AND tilt my head but I see what he means 🤣

[@ZyTelevan]

Code is data is code

[@BlargVison]

yeah that was a fantastic comparison that i won't forget

[@kacperozieblowski3809]

I agree

[@filipgara3444]

„No.”

[@TheLifeInMotion]

According to Strongbad: "Technology is anything that you don't understand how it works and if you break it you have to buy a new one."

[@chrisdaley2852]

So retractable pens are technology. Got it.

[@CH-bd6jg]

@@chrisdaley2852 pen goes in, pen goes out. you can't explain that! just buy a new one!

[@columbus8myhw]

Chris Daley I mean, yes… but also I was willing to say scissors are technology so maybe I'm not a good judge of these things

[@OrchidAlloy]

@@chrisdaley2852 Yes they are

[@diablominero]

So my desktop computer isn't technology because I built it and could replace a single broken component rather than the whole thing?

[@atimholt]

“Are scissors technology?” Me: yeah, of course. “Most people would say no.” ¯\_(ツ)_/¯

[@totally_not_a_bot]

Those of us who watch these videos don't really qualify as most people.

[@pauljs75]

Even sticks can count as technology, if implemented as tools in some way. (Combination of tools and methods to achieve some goals. Usually making a task easier, or doing something else that improves conditions for the tool user.) Obviously such is not the latest and greatest technology, which seems to be the definition this video is going for.

[@lucar6897]

I also think of calculators as artificial intelligence...

[@shayneoneill1506]

Yeah the part of my brain that did those anthropology units would never let me think scisors arent technology

[@NoahTopper]

When I was a kid I definitely would have said no. But I remember at some point being taught that anything along the line of a pencil or chair was technology, and that sunk in. But I imagine a lot of people still have that initial instinct.

[@columbus8myhw]

"Like, there's no point asking for feedback if you're already pretty sure you know what the answer is, right?" …Do you want me to answer that question?

[@discosteve]

Your point still stands, but neverless the scissors have a butt load of tech in the background that us normies aren't aware of (material science). Just wanted to mention that the humble pair of scissors deserves some praise.

[@DDvargas123]

I was thinking the same thing. We take for granted a lot of the cool tech around us all the time. Levers and Pulleys and other simple machines most of all. But rob makes a good point that people dont commonly think of them as tech even though perhaps they should. Language is a cruel mistress.

[@infinummjb]

scissors are relatively low-tech, but a tech nonetheless.

[@columbus8myhw]

Would you consider a scissors company a "tech company" the same way you'd consider Apple and SpaceX tech companies? What about post-its? Is 3M a tech company?

[@DDvargas123]

@@columbus8myhw 3M's company description is literally: "applies science and innovation to make a real impact by igniting progress and inspiring innovation in lives and communities across the globe." That sounds really tech company to me

[@RobertMilesAI]

I think if you took someone to a scissors factory and showed them all the machines and equipment of the production line, they'd call that technology. But not so much the scissors themselves

[@FortoFight]

If you think about it, this is a lot closer to how a human learns. A human won't constantly bug you for feedback every single time it does something, nor will it learn how to do something properly from a standardised function (e.g. exam mark schemes). A human will independently use its available knowledge, and occasionally ask for help when it's unsure what to do.

[@dannygjk]

Do you have any children? Kids seek approval.

[@owenpawling3956]

@@dannygjk no, but he is right. Kids are just unsure more often.

[@Nico-ur2po]

@@dannygjk You don't correct a kid every time they talk using improper grammar or mix up word order. You correct them every now and then, and they learn over time combined with observing how other humans talk.

[@dannygjk]

@@Nico-ur2po I didn't, (I have two kids).

[@Henrix1998]

I can already imagine the Indian ML farms where thousands of people just evaluate learning

[@TurkishLoserInc]

Sounds a lot like the premise for The Matrix. "On a scale of 1-10, how real do you think this is?"

[@Encypruon]

It's called Amazon Mechanical Turk.

[@Verrisin]

Damn, that actually sounds likely... - Here is my idea: since AI will take all our jobs... There will be one job of the future: *Specifying preference.* - I actually don't hate it. :D

[@Verrisin]

... thinking about it: It kind of is the ideal job, isn't it? Do we, as humans, even want to do anything more than that? - Our job will be saying what we want in the world, and how we want things to work... It will even work as a voting mechanism for policies since they will be run by AI - that figures out how to best match our preferences... - I think this is the way... (or at least a good direction for now ^^)

[@benalias5766]

I can already imagine a complex AI which is surprisingly good at a wide variety of tasks... and turns out to have hired a load of people in India to do its work for it.

[@riccardoorlando2262]

So in a couple years captchas will be reward predictor training? "Which of these is the better shoe design"?

[@toxicpsion]

nah, i'd bet they do it already; just more subtly than that.

[@LoveScreamTrue]

@@toxicpsion Like Google CAPTCHA? - "Select all traffic lights"

[@johnnymellon7414]

"Select all the pictures with Sarah Connor in them" ... wait what?

[@z-beeblebrox]

@@LoveScreamTrue Except it'll become "Select your favorite traffic lights"

[@stribika0]

Which of these places do you prefer as a shelter during a robot uprising?

[@Noxeus1996]

Definitely one of the best educational channels on KZbin.

[@zacharieetienne5784]

hold on to your papers and i'll see you, next time!

[@CynicatPro]

@@zacharieetienne5784 TwoMinutePapers is also super good X3

[@hypebeastuchiha9229]

@@CynicatPro he sucks

[@stefano8936]

Robert Miles: "what is technology?" Me: move the finger to calibrate the amount of video to skip Robert miles: "don't skip ahead" Me: humbly obey

[@GrixM]

I feel betrayed because the next 5 minutes were just repetition of previous videos so I wish I had in fact skipped ahead.

[@jnevercast]

Yeah he got me too. I was about to skip just as he said don't skip. "Well okay!"

[@Atariese]

The thing is... the question he poses after that leads me down that rabbit hole and away from his video... definitely not the intent i would say

[@riperian8954]

@@GrixM lol i did exactly what you and OP did, only I was like 'okay okay that's enough of that' after about 2 minutes. still a brilliant video overall though xd.

[@Macieks300]

"in a later video" well... see you in 3 months then

[@IstvanNagy86]

This channel always worth the wait :)

[@griest5493]

IKR, what a tease.

[@MatthewStinar]

You can't rush this kind of quality! Do you know how long it takes to read and digest all those research papers?

[@IstvanNagy86]

... almost there.

[@Macieks300]

@@IstvanNagy86 to be fair Robert was on Computerphile in the meantime kzbin.info/www/bejne/aWLVhmCMr6tordk

[@TheMan83554]

The thing about your channel is the little touches of 4th wall humour. Having backflip you say "wait I don't have to do a backflip?" Was brilliant.

[@sharkinahat]

I wouldn't mind an ad. YT trained me how to skip paid promotion.

[@weirdal3333]

KZbin Vanced vanced.app

[@rr.studios]

@@weirdal3333 lol im using this app rn

[@HansLemurson]

What sort of reward function did you use?

[@megajor232]

Whatcing your videos make me feel smart without actually having to be

[@benalias5766]

Sounds like you're gaming your reward metric.

[@ephemeralvapor8064]

Maybe your evaluation of his teaching is: Good teacher = true Because he brings understanding lesser teachers could not in the same time and effort on your part.

[@zeikjt]

8:50 That backflip part was super enjoyable :D

[@MrCreeper20k]

17:25 Don't worry Robert, at least I don't mind an ad at the end. And if anyone should get that bread, it's you.

[@briandoe5746]

I am in a room by myself and I audibly cussed when I heard that openai and deepmind we're working together on something. Google's apparent lack of concern with safety is one of the reasons I want your videos sir

[@daniellewilson8527]

Brian Doe why is two AIs with different modes of thought working together a problem? Humans have different modes(parts of the brain specialized for different tasks) that combine the inputs from these disparate programs into a coherent idea of the world. Imagine trying to learn about your surroundings when the only sense you have is the ability to differentiate temperature and you will understand why certain AIs need others to help with things.

[@briandoe5746]

@@daniellewilson8527 my main concern with AI is not the expediency that it gets to general intelligence. My concern with a i is the safety mechanisms and their capabilities when it gets to general intelligence. Google has multiple times proven to be unconcerned about the safety question in This is highly concerning

[@igordmitriev7211]

>We'll talk about them in a later video //Gets hyped, realises that it's the latest video on the channel, gets reminded of Patreon, enlists to see the video a bit sooner

[@Varue]

Humans being able to simulate problems in their head to predict different outcomes is one of their greatest strengths, it means they can be confronted with new experiences they haven’t evolved specifically for and come up with a solution from a list of possible solutions and stand a much greater chance of overcoming the problem without dying

[@OrioPrisco]

Hey it's really cool for the viewers that you turned down that sponshorip offer, thanks

[@dontfeo]

Nah he should've taken it. U can skip it anyway and it would help him bring more content.

[@brendanjackman3600]

"Hmm, reward functions are a limiting factor on some ML capabilities. This is a problem. How do we solve problems? WITH ML"

[@DDvargas123]

Sometimes a solution is so good it can solve its own cons

[@MichaelWBauer]

It's definitely funny when you frame it this way, but it's also interesting to note the similarity here with the brain. The brain is a system of interconnected neural networks which each are responsible for certain aspects of our thinking capabilities. It's not too hard to imagine the connection between the logical extension of the results in this video and the architecture of the human brain.

[@default632]

@@MichaelWBauer Remember where the word neural network came from. Duh

[@MatthewStinar]

I think you're describing a Generative Adversarial Network. en.m.wikipedia.org/wiki/Generative_adversarial_network

[@amyshaw893]

just replace the human with another ai, and get the human to rate that ai. not good enough? MOAR AI!!11!!

[@DDvargas123]

It's AIs all the way down!

[@thehypnotoad5184]

Just make an AI trained on footage of people doing back flips, no need for human input Even if the AI is "only" 99% accurate it should be enough

[@DDvargas123]

@@thehypnotoad5184 "footage of people doing backflips" IS human input

[@thehypnotoad5184]

@@DDvargas123 I mean the input already exist its just need to be collected, its kinda going full circle but it would be interesting to see if you can speed up the reward model that way

[@rumplstiltztinkerstein]

@@thehypnotoad5184 but the ai will find ways to exploit it. Nothing stops us from giving the footage and having a human checking it from time to time telling it to stop using it's head as a catapult when the ai was supposed to be running

[@DamianReloaded]

I would define intelligence as "the ability to autonomously identify problems and search for solutions to achieve goals"

[@jessgold551]

I have watched all of Robert's videos several times. Its perfectly paced, well considered and clearly communicated. There is so much there its interesting to watch, sleep on it, and watch again later to catch more. I also enjoy the presentation and multiple interesting ways of presenting things like word popups and cut to screen as well as some graphics and clips. If it helps with demographics I am a former software engineer and still work in I.T.

[@Felixkeeg]

I am actually a bit dissappointed that you didn't go for the backflip lol

[@ruvimlashchuk6134]

My disappointment is immeasurable, and my day is ruined.

[@ruvimlashchuk6134]

My disappointment is immeasurable, and my day is ruined.

[@Suush]

He forgot to program a reward function :P

[@Alex2Buzz]

Miles: "What is technology?" *VSauce music*

[@ohokcool]

Did u go to Palms Middle?

[@fish_wizard618]

It seems like this method of evaluation could also help AI's learn to do much more arbitrary things. Like if you wanted a “pretty” pattern, you could train it to make more patterns that you find pretty using this.

[@FrotLopOfficial]

That last few minutes of your video will go unnoticed but for those who do, we very much appreciate it.

[@the1gip]

You, sir, remain one of the most interesting educators in KZbin. The effort you've put in to making this video watchable and entertaining really shows. There's not too many people I can watch for nearly 18 minutes in front of a beige backdrop and still be hooked.

[@NoahTopper]

12:19 I approve very greatly of your use of "eachother" as one word. The world needs this change. I don't know if you and I talked about this at all at the EA Hotel, but I've been trying to convince everyone to write it like that.

[@squirlmy]

I started to do that, but "spell correct" too often comes on and I've gotten used to following automated corrections. I'm wondering if automated (or even AI writing assistants) will slow the evolution of language and grammar, and perhaps even pronunciation will remain in stasis not because of any changing dialect cues of social status, origin (or adopted location), or otherwise, but because of how our "correcting" algorithms are programmed in communication devices.

[@qwertyTRiG]

@@squirlmy You've reminded me that I really need to create a dictionary with Oxford Spelling (en-GB-oed).

[@discipleoferis549]

I've been writing "eachother" for 15 years now. I've even told off some of my English teachers for trying to correct me. Heck... I remember back in 6th grade, I think, telling off my teacher for incorrectly correcting another student that had written "ain't". I was an opinionated 11-year-old, haha.

[@NoahTopper]

@@discipleoferis549 I told my high school English teach that I was attempting to turn "eachother" into one word, and if she'd be willing to not mark it wrong when I used it. She was super on board.

[@qwertyTRiG]

@@NoahTopper It definitely makes sense. Similarly, I tend to distinguish between "alright" (acceptable) and "all right" (completely correct).

[@cmoxiv]

Mate, you are brilliant. Great content with a philosophical flavour. The last part about Patreon is probably the only thing that actually convinced me about supporting content creators on Patreon. Well done mate. Well done.

[@rosborr4330]

I subbed because you knew I'd skip ahead the moment you said 'What is technology?'. You win this round, Robert.

[@wilhem13]

A video upload ?? My day's already better. Great content my friend, THIS is why I don't watch TV anymore.

[@crypticnomad]

When people ask me what AI is I generally say that it is a universal function approximator.

[@gus2747]

"If you squint the training process is sort of like a compiler " --- great sentence!

[@geronimomiles312]

You choose to tackle issues which really clarify the meat of the process , and do fantastic. Really good stuff👍

[@explogeek]

Loving your videos, I understand it takes time to research and script and edit, but I wish they came out more often...

[@dontyoufuckinguwume8201]

The guy has a full time job, the only way to get him to make more videos is to donate ^^

[@firefoxmetzger9063]

hmm. If samples are chosen based on unusual examples where the ensemble disagrees, what happens if the exploiting strategy has high agreement among members of the ensemble? It would never show up to the human for "correction" right, because the ensemble is confident about it? So rather then having to trust the network that performs the task, we now have to trust the ensemble training the reward function?

[@MatthewStinar]

I was thinking you would still want to throw in some strong matches just to verify.

[@DarkPrject]

This continues to be one of the most interesting channels on KZbin. Fascinating video. Can't wait to see the next one.

[@n4th4ni3lmc5]

Awesome explanation and sounds like great progress in the field! Thank you very much, sir.

[@arthurguerra3832]

I've been so long without your videos. Please upload more frequently so we can drink your intelligence and knowledge.

[@Telhias]

With regards to puppeteering the robot to perform a backflip. There is a whole community of the Toribash game who do exactly that. It is a game in which every time period (measured in ms) you decide which joints to flex, extend, hold rigid and relax.

[@sjeses]

Absolutely fascinating. Thank you for putting in all the time and effort to introduce me to all these ideas in such an effective way.

[@frib75]

An amazing video. Never heard such a beautiful explanation of what reinforcement learning is. Thank you !

[@StromyYTA]

These videos are awesome. Feel almost like I can keep up to date with AI progress.

[@wiktormigaszewski8684]

This is what I always thought of making a good robot - you give a feedback to it, while it learns, just like parents to a child. Very good, that this concept has been put into practice. It is definitely going to be helpful for AI companies making robots for their clients, who do not know exactly, what they need. The guy from "two minute papers" would say "what a great time to be alive!" :-)

[@reneko2126]

Yeah, why not just raise AI like kids? kzbin.info/www/bejne/m5K8eohsjr2ladk

[@circle688]

what a time to be alive

[@Metrolonx]

Love how the video quality grows with every video! Keep it up!

[@daviddawkins]

Incredibly well presented and articulate, thank you.

[@xxThabaxx]

This is something I've been thinking a lot about as it could work similarly to how we tend to train children. It seems like you could first train a machine learning algorithm to recognize social cues (lingual and physical responses) regarding it's behavior and build a reward function based on that. I think you still run into some complicated reward hacking situations like the machine wanting to force certain reactions. But it seems like it would get us closer.

[@eathonhowell7414]

This way of thinking is exactly what's getting me interested in this field. I cannot help but feel there is a comparison to be made between the in-exact nature of child raising, and trying to "teach" artificial intelligence. General or otherwise. Hell, think of an individual cell within the body as an AGI and the totality of what humans are seems like a miracle.

[@eathonhowell7414]

This way of thinking is exactly what's getting me interested in this field. I cannot help but feel there is a comparison to be made between the in-exact nature of child raising, and trying to "teach" artificial intelligence. General or otherwise. Hell, think of an individual cell within the body as an AGI and the totality of what humans are seems like a miracle.

[@gwen9939]

@@eathonhowell7414 You should probably watch the video called Why not just Raise AI like Kids.

[@morkovija]

Been a long time Rob! Hope you brought the sauce!

[@non_complete]

I agree wholeheartedly with your name.

[@wilhem13]

Most videos I MUST watch them on, at least x1.25.

[@morkovija]

@@wilhem13 means that your content information density is quite high. No way I can speed up mathologer for example. But easily 2-3x some non-narrated restoration videos

[@johnopalko5223]

Thank you for not accepting sponsorship from a company that wanted you to do a 60-second spiel. There are companies who sponsor videos and are happy with just having their logo displayed in the corner once or twice. At most, they have the presenter start out with, "This video is sponsored by So-and-So. [One or two brief sentences.] Link in the description below." These are the companies that get it.

[@esquilax5563]

Good to see you on here again! You have some of the most fascinating content on KZbin

[@mrWade101]

Scissors would be Old technology, whilst when most people say Technology they mean New technology.

[@fergochan]

Great video, but there's still one thing I'm confused about: how do I tell if that simulated robot is doing a back flip or a front flip?

[@zachkrakower172]

Dude these videos are awesome. Thank you for taking the time to educate all of us!

[@EU_DHD]

I like watching you talk about AI safety more than I like learning about AI safety. And I really like learning AI safety!

[@unvergebeneid]

Shade much? So you're not learning AI safety by watching him talk about it?

[@EU_DHD]

@@unvergebeneid Those are two aspects of the same thing. I just like the one aspect more than the other.

[@dsdy1205]

When you realise you've reinvented the parent-child relationship

[@AugustusBohn0]

nature wins again

[@dsdy1205]

God coming back to this comment a year later it sounds so stupid

[@AsmageddonPrince]

Your voice is so soothing, and videos so informative.

[@panstromek]

This is really on point for a problem I am trying to solve now. I do some computer vision for which it is way too complicated to create training data and way too complicated to write reward function, but it's the "You know it, when you see it" type of thing. Thanks for making this video ;)

[@BinaryReader]

Technology is just another word for "Tool". Everything created by humans of some utility is a tool, and is therefore technology. I wasnt aware there was confusion around the definition.

[@oldvlognewtricks]

Queueing was created by humans and is of some utility. Queueing is not technology. Stand-up comedy was created by humans, and is of some utility. Stand-up comedy is not technology. It is difficult (or perhaps impossible) to write a definition that doesn’t raise exceptions, which I suspect was the point Robert was trying to make. Your example only confirms the point.

[@BinaryReader]

Not to get into a huge discussion here, but both of those could be loosely defined as technologies. What are jokes if not tools of social interaction? What is queuing if not a tool for social order (assuming you mean standing in line and not the computer science definition, which is also a technology)

[@oldvlognewtricks]

@@BinaryReader I continue to agree, and disagree. A joke and a queue might be tools, but 'technology' is more of a push. technology /tɛkˈnɒlədʒi/ - noun the application of scientific knowledge for practical purposes, especially in industry. "advances in computer technology" machinery and equipment developed from the application of scientific knowledge. "it will reduce the industry's ability to spend money on new technology" the branch of knowledge dealing with engineering or applied sciences. There is perhaps some science to comedy, but a social convention like queueing is hardly an application of science, so much as an emergent social expediency, or whatever. I'm not getting 'engineering' from either, except in the loosest sense. Alternatively, to take the definition to its logical conclusion, all human action is technology and the definition loses its usefulness. But you're right - no potential for confusion whatsoever ;) At best, there is comparative 'technology-ness' - a joke might be technology, but it's less technology than a smartphone. Maybe moreso than a punch to the face. Maybe it depends on context. Still works to make the 'this is not straightforward to define' point.

[@squirlmy]

@@BinaryReader Perhaps it's an Americanism, but there's another definition of "tool", and you're well on your way towards demonstrating it. Both of you actually, because none of us need or want an in depth discussion of the definitions of either word. Rob's brief mention of it doesn't warrant further commentary.

[@drdca8263]

Rob’s definition kind of closely matches Strong Bad’s definition, of “anything that’s really cool and you don’t know how it works”. Ryan North’s definition includes language, and I think basically any technique which has been invented. But yeah, like Rob says, it isn’t a big deal how we define it. Slightly different definitions can can be used in different social circles, or even in different conversations among the same people.

[@jayteegamble]

meh, we don't mind a 60 second spiel if it gets us more of your awesome content (and we can skip forward anyway). Grab that bag imo

[@diribigal]

This is a tough problem since watching to the end is probably valued by KZbin's AI, and even though you and I wouldn't mind, some would. So how do the short term gains of the sponsorship compare to the long term dividends of the KZbin algorithm and extra subscribers, which increase visibility over time (perhaps by a minor amount) ?

[@sevret313]

@@diribigal That's why you don't put the sponsor at the end, but the start.

[@MrKohlenstoff]

Great video, super well and clearly explained! 👌

[@maloxi1472]

Thank you for bringing this idea to my attention ! Holy cow ! This is such a simple, yet beautiful idea !

[@Laborejo]

"It is easier to write a program to evaluate a solution". This is also why artificial music composition does not produce even half-decent outcomes yet. Creating an artificial listener (or many of them) is still far down on the to-do list.

[@postvideo97]

There have been no research (that I know of) that uses human reward modeling for music generation. It could be the next breakthrough in music generation!

[@Sceleri]

this method could work for that tho you just tell it which beat is more fire

[@ToriKo_]

Sceleri exactly

[@dasc000]

emily howell: hold my beer

[@jameswalker9403]

Have none of you heard of Emily Howell?

[@bencrossley647]

This sounds like a method to solve NP problems. Easy to verify Hard to solve.

[@4.0.4]

The year is 2069. A computer is granted the prize for solving the P vs NP problem. Despite the judges being unable to confirm that the overly-complex thesis the computer came up with was correct or not, it looked quite correct to all experts. A mathematician was quoted saying: "...I mean, in the two new branches of mathematics that the computer invented, the math does check out." It is unknown what the computer will do with the prize, but several paperclip factories report being contacted shortly after the prize money was deposited.

[@bencrossley647]

Chrysippus +1 for paperclips (assuming you’re referencing the game) It will work it’s way to a galactic army at some point.

[@Kevin________]

@@4.0.4 Alright... you win this comment section.

[@griest5493]

I was thinking the same thing when he said that. Also, the halting problem is a thing. The catch is that NNs are just making approximations.

[@default632]

@@4.0.4 universalist paperclips, hours of waste time for a reference on the interwebs. Worth it

[@MidnightSt]

...i don't know much about this area of IT, but the first thing that came to my mind after reading the video title was: "oh, yeah, what's a better idea than creating a black box that nobody knows how and why it works, and what its boundary conditions actually are? why, yes, creating such a black box without even explaining to it what is good and what is bad! BRILLIANT!"

[@V1ctoria00]

Damn. I dont usually find a new channel by its latest video. I was hoping I could binge this topic here.

[@xenoblad]

You've been playing Raid: Shadow Legends for 10 years?!

[@BubbleManxx]

I laughed at the Vsauce reference.

[@Hexanitrobenzene]

Could you provide a timestamp ? Looks like I missed it.

[@BubbleManxx]

@@Hexanitrobenzene Lol, it's at the very start of the video. When he pops up from the lower half of the screen and asks "What is technology?".

[@Hexanitrobenzene]

@@BubbleManxx Oh, that one :) Looks like I'm rusty on VSauce, haven't watched him in awhile...

[@andersenzheng]

@@Hexanitrobenzene Not your fault. There hasnt been one for a while

[@bscutajar]

This is one of the best channels of youtube. The guy's explanations are extremely well done.

[@haldir108]

I am EAGERLY awaiting that video about self-teaching or whatever it is.

[@sk8rdman]

"Mattresses and VPNs." Someone watches SmarterEveryDay

[@DigitalicaEG]

"Don't skip ahe..." Me: **skipping**

[@bibasniba1832]

Priceless knowledge, swift explanation. Bravissimo!

[@AlejandroPiad]

This is the first of your videos I see, and you almost got my subscribe with the first philosophical half, but the second half was plain brilliant, so you definitely got my subscribe now.

[@Deez-Master]

We are getting close to having P=NP

[@governmentofficial1409]

Silicon Valley spoiler

[@realityChemist]

"How do you learn when there's nobody who can teach you?" Read a textbook or a WikiHow article?

[@Vode_ika]

That is someone teaching you, via a book.

[@realityChemist]

@@Vode_ika True, I was thinking in the context of someone sitting there teaching you, like in this video. So I guess the answer is just unsupervised learning? Although I could have sworn Rob already did a video on that... Maybe it was someone else on Computerphile?

[@drdca8263]

Isn’t the answer “think very hard, write things down, and when you can do so safely, try many options, test your previous ideas both by the results of the options you took and by more thinking, repeat”?

[@Biped]

@@drdca8263 but that all requires some way of evaluating your results (aka having a reward function that teaches you)... It seems weird that there would be a way without that. I mean... the information has to come from somewhere...

[@SimonBuchanNz]

I would suspect the answer is, in fact, something like googling it, but this, of course, requires a pretty complete internal model of the world to start generating and testing against your own predictions. I'm struggling to think of alternatives that aren't just this in disguise though: the best I have is looking at a small set of successful examples and trying to break down from the solution used what the problem is, so you have something to test your own solutions against. If there's a decent way to describe that that isn't going to fall prey to small training data issues like overfitting, I'm excited: that's starting to really sound like the casual meaning of learning!

[@bensonmiakoun7674]

Highly interested for the next video! Thanks

[@injinii4336]

Surely scissors are an example of some of our most cutting-edge technology. Ba-dum-tss!

[@roberthoople]

"Training AI Without Writing A Reward Function..." *Capitalism Drools*

[@MatthewStinar]

Watching this video made be realize how much corporations are like poorly programmed artificial intelligence, like the stamp collecting AI that decided to "Kill all humans." We take our instrumental goal of maximizing profits and assign that as the corporation's terminal goal. In pursuing it's terminal goal of maximising profits, the corporation decides to "Kill all humans." 😲

[@Karpata1]

Hey if I have to hit the "L" button a couple times so you can get a couple hundreds or even a couple thousands of pounds I'm fine with it.

[@mygreenlama]

Thank you for another great video! I am very much looking forward to the continuation ;)

[@rerere284]

9:00 There's a game called Toribash where you do exactly this, but with a more complex body. It lets you specify the states of all the joints in 1 second tine segments, playing out like speed clocks from chess when playing multiplayer.

[@Havermeijer]

I remember that game! You could pull someones head off and stuff. Pretty difficult to master though. Also, the game kept sending me happy birthday emails for years and years. I didn't get one last time :(

[@hypnotourist]

Very clear presentation for a fascinating topic ! Your "patreon/human discussions" reward function has trained you well, so to speak :-)

[@alexlamson]

Excellent work Rob, this is a good one. I think there's some content potential in reviewing these big RL papers

[@augustinaslukauskas4433]

I'm not surprised this result is amazing considering both OpenAI and DeepMind worked on it. I dream of working for one of them after uni. Thank you for explaining the paper so clearly and in an entertaining way!

[@sam-you-is]

did you make it sir

[@stefx5994]

Hi Rob, Firstly many thanks for the amazing videos you produce - as a fellow Dev and Techie i find your content and delivery style some of the best and most informative on KZbin. Could i request a future video in which you explain the coding side of developing a basic AI Agent? It would be great to learn how to explore some of the concepts and interesting problems your videos highlight. There's a lot of frameworks, open source projects and tutorials out there already, but they present a very black box, end result focused approach rather than explaining what components we have and how they are working together to reach the end result..the type of complexity you seem to be fantastic at explaining :)

[@RobertMilesAI]

I've been thinking about a "Write an AGI from scratch" series, but it would be a lot

[@Kobriks1]

Excellent explanation! Thank you.

[@MrLuMax5]

In my opinion you could have done the sponsorship. It helps you as you help us, 60 seconds is like not that much and you deserve it for all the work.

[@gabrote42]

These are brilliantly designed! I want more!

[@derasor]

Hi Robert, that was a very interesting and very well done video, it got me to activate notifications, something that I've never done before in any other channel. I just wanted to say I can't be a patreon but I wouldn't mind watching advertisement on your videos, I know this in no way compromises the integrity of your great content, and if it helps you making more of these amazing videos more often, I think we all benefit.

[@weirdsciencetv4999]

This channel is so underrated. I had to do just what he proposes in one of my experiments in college. The technique most definitely works!

[@orcu]

I liked this explanation very much. Great work!

[@rewrose2838]

Lovely stuff here, very clear explanation and great use of the graph

[@SandwichMitGurke]

Omg I think I just saw one of the best videos about ML so far. I’ve never heard of this and your explanation was just phenomenal. I am so hyped to try this out because I often make my ai really bad just because of a crappy reward function. (I do AI programming just as a hobby)

[@ZachAgape]

The first videos I saw u in were the computerphile videos on AI which I enjoyed a lot, and thanks, this video was very interesting too! Also thank you for not wanting to waste 60 seconds of our time ^^

[@tedstokes57]

I like that there's a hint about the next video at the end

[@SkyboxMonster]

Patreon feedback. Congrats you are now smarter than many hundreds of Advertising specialists. Just think of every advertising blunder that the public caught. but the designers did not.

[@andybaldman]

*Woohoo, new vid. You do not post enough! Love all of your vids. You should post more often.*

[@DahVoozel]

Fascinating stuff as always.