John Schulman (OpenAI Cofounder) - Reasoning, RLHF, & Plan for 2027 AGI

[@drhxa]

Your podcasts with these researchers are the most valuable learning resources for people who want to gain a deeper intuition about Gen AI and understand where it's going. Thank you!

[@moonsonate5631]

00:30 Pre-training creates a model that can generate content from the web. Post-training targets a narrower range of behaviors like being a chat assistant. 03:44 Models evolving to perform complex coding tasks autonomously 10:29 Improvement in the ability to do long-horizon tasks is key to AI capabilities. 13:52 Models showcasing generalization capabilities in different languages 20:29 Coordinating deployment of AGI for safety and prosperity 24:07 Ensuring safe and continuous progress in AI deployment 30:55 Models have drives and goals, learning to produce outputs that people like and judge as correct. 35:09 Importance of reasoning and introspection in AI model training 41:59 Evolution of instruction following models at OpenAI 45:25 Challenges and improvements in language models 52:29 Importance of understanding the whole stack in RL research 56:19 Examining the effectiveness of varying model sizes concerning transfer learning and intelligence levels. 1:03:45 Future of AI-assisted decision-making 1:08:26 Regulation on companies to ensure human involvement in important processes 1:15:26 ML literature is relatively healthy and focused on practicality 1:18:52 Research at big labs focuses on improving efficiency and infrastructure for training language models. 1:25:08 Preference models capture subtle values and preferences. 1:29:12 Utilizing varied resources for model comparisons and improving AI models. 1:36:01 Importance of understanding AI process

[@siddharth-gandhi]

Looks great my man, a challenge to one up yourself is to get Ilya back on the show now!

[@michaelm358]

And Alex!

[@irshviralvideo]

why ? Ilya is proven to be on an ego trip. I would get more genuine , non political people

[@HarpreetSingh-xg2zm]

@@irshviralvideohow has he “proven” to be on an ego trip?

[@irshviralvideo]

@@HarpreetSingh-xg2zm they guy is political. he was part of the group of liberal left that tried to get rid of sama. We need people who can actually deliver impact to the world. The guy is more of the academic type who likes to take the lime light but doesnt do the real nitty gritty work.

[@skoto8219]

@@irshviralvideouhh, were you under the impression that sama’s ousting was the result of him being outed as a neoreactionary and that the whole thing was orchestrated by blue-haired sjws? ilya doesn’t even *have* hair!

[@TuringTestFiction]

I love how he pushes the arrival of AGI back from the unrealistic next year to the entirety plausible two or three years out...

[@vrai4913]

great episode, john schulman was interesting. i appreciated you pressing him on his view that dangerous AGI could emerge within "two or three years", at least with some likelihood where he found this topic worth discussing. i don't have enough info for a strong opinion on that myself, but i've noticed it's almost a trope to point out a mismatch between some AI researchers' views on AGI timelines and the lack of clearer thoughts or action one would expect if they genuinely believed it was urgent. however, the frequency of this observation doesn't make that less strange. john schulman is doing amazing work though, and i'm glad he came on the podcast :)

[@kailuowang]

Dwarkesh seems to be really surprised by Open AI's blatant lack of any concrete plan for AGI alignment.

[@AdamPadron]

The weak answers were pretty stunning. He hasn’t read their own white paper.

[@therainman7777]

It’s all pretty frankly terrifying. Especially with the two biggest safety advocates (Ilya and Jan) resigning as of this morning.

[@41-Haiku]

I was never optimistic they would solve alignment, but now the nightmare is deepening. I'm a big fan of, y'know, _not_ building things that are designed to replace humans wholesale, especially if we don't have any control over what they will do once they exist.

[@user-yk5by3uc2b]

I doubt they will make their plans public

[@human_shaped]

To be fair, some people have an area to focus on, and they do that. Very well. If everyone worked on everything, it would be a hot mess. That's not to say they do have a good plan, but you can't blame someone that works in a different area too much.

[@adamoreilly6546]

FYI I’m having a lot of good results implementing the MCTS with LLMs mentioned in your Demis interview. I feel like the current best model capabilities are underestimated when looking at my results. Even tags work with Claude haiku with a max of 3 retries (meaning you can search a lot of state space with a little $)

[@Renvoxan]

cool story nerdoid

[@therainman7777]

Can you explain a bit more? You’ve set up your own MCTS implementation that works with the Claude API?

[@bossgd100]

What are your results ?

[@adamoreilly6546]

@@therainman7777 I’ve been working on autonomous repository generation to take a text prompt and return a production deployed web application. Yes, it uses the MCTS approach that Demis stated would be a likely path to AGI. It generates the full repo but need to improve the build/test loop a bit more to get a result that doesn’t contain slight errors. Still impressive and I think I can get it working soon with the current Claude 3 models.

[@LtheMunichG]

How does it use MC tree search? What’s the state space?

[@greensock4089]

This guy being head of alignment is EXTREMELY worrying. Holy moly he has no idea what he's talking about as evident from the end of the plan for agi 2025 section of the video

[@patruff]

He left!

[@goodwillhart]

Really fantastic interview. I think there are so many hints of what to expect in this talk that you could almost predict what the next couple of models are going to look like, especially the long timeline RL, post-training vs pretraining mix, especially with regard to reasoning, models that are more aware of their capabilities. I also found the stuff on learned gating quite enlightening. It was interesting to hear a different perspective to Ilya's (Ilya tends to talk about compression whereas John speculated about libraries of circuits, which is more about the mechanics of how that is actually achieved). And of course it is fun to speculate about how this might have been harnessed deliberately to improve the fundamental technology itself and how this might improve interpretability. And the hints about using in-context learning with long context are probably hopelessly underexploited by people trying to get more out of these models, since we are all so used to shorter context. I'd love to see more material like this but of course it is hard to find vs the usual nonsense speculating about AGI being developed in some bunker and how every new tool "shocked the entire industry", etc. The occasional bit of intellectual stimulation goes a long way. Congrats on researching this well enough to ask the really interesting questions, and provoking equally interesting answers. And congrats to John for saying the interesting things, modulo one obvious slip, without having to resort to "I can't talk about that" every other sentence!

[@mikezooper]

Love Dwarkesh. I got burned out by many podcasters over the years, but he’s refreshing and focused, while being approachable.

[@michmach74]

GG Dwarkesh, getting all the cool guests. How do you do it man, bro's an insider lol

[@adamoreilly6546]

it’s probably been a chain of referrals from previous guests

[@TheLegendaryHacker]

Something he said in a previous podcast: Send your interview request email with deep, well thought out questions. People like Dario Amodei or John Schulman get 50 of those emails a day, so you really need to stand out. That, and Dwarkesh has a reputation now.

[@xsuploader]

Because people in tech watch these interviews. Notice how they all say they are fans of the podcast.

[@foswa6335]

Because he is an insider, especially now

[@ckq]

I mean that's how podcasts work, even if you only got 100-1000 subs you can still get big names. Just show interest in topics they like. People like to talk about what they do regardless of the host, they aren't picky. Atp, Dwarkesh can get literally anyone on the pod.

[@OutlastGamingLP]

17:21 This was such an unsatisfying and worrying answer about planning around risk. Dwarkesh tries to push him to get any kind of concrete answer and barely a couple minutes of hand waving about "slowing down and being careful," he's saying something like "well, if we solve the alignment problem it will be great." This is a ridiculous attitude to have. I don't care how fun and exciting it is to build super powerful tools, if you can't stop them from eating the planet you don't get to smile about it. Just notice! Notice how weak this answer is! Notice how little people seem to be taking this seriously or even thinking about it all that hard! Sandboxing!? Sandboxing!? We had this argument like 10 years ago, and - as far as I'm aware - we basically settled on the answer "it won't save you." Coordination? About what? Is everyone going to coordinate to burn their GPUs and demand the international community ban further sale of GPUs? If not that, what else? What would possibly save you at that point if you're just "plug in more computers" away from something that can wipe out humanity and has no mesa-optimizer internally planning around not killing everyone. This is how we die I guess. A bunch of people who think that utopia is totally reasonable and close in our future, but existential risk is super weird and therefore unlikely. Yep. Not pleased about that. Hope there's an afterlife so we can all sort out this stuff in hindsight and these people can look at what they did and feel regret.

[@jaiveersingh5538]

Have you watched Robert Miles' stuff? If not, you might enjoy his much more serious take on the subject of formal proof for alignment

[@OutlastGamingLP]

@@jaiveersingh5538 Oh, yeah, thanks for the recommendation, but I'm an ancient AI Alignment follower. I saw Rob Miles for the first time when he was still on Computerphile. When I reference the old arguments about sandboxing I'm calling back to forum discussions I followed when the topic was already like 8 years old back in ~2016. I wasn't around in this space for the Singularity Institute + creating Friendly AI era back in like 2008-2009, but yeah, I'm not sure if Rob started reading The Sequences before me... maybe, I'd have to check when the ebook collection came out. He did probably finish reading them before me though, I was off/on for a while until Rationality: From AI to Zombies came out. I know we're a long way off from actually Aligned AI. Even systems you can keep from blowing up into an unaligned ASI seem pretty hopeless to create anytime soon... Yeah, I could go into detail why - but like, if the perspective on safety represented in this conversation was coming from a BRIDGE ENGINEER who's being asked if their design was safe - you'd kinda expect that bridge to fall over. AI Alignment is obviously cursed by Murphy worse than computer security or medicine or any other domain where you need to plan your interventions, designs, protocols carefully. With AI parts of your design parameters are being interacted with by potentially powerful optimization processes which could enter search spaces that are meaningfully different from prior models at basically any point. These kinda "bag of tricks," "we'll be careful," "it's not dangerous yet probably, so let's keep going" arguments just seem utterly the wrong way to react to our present situation.

[@skylark8828]

They seem to think whatever AI/AGI systems are built will be fully non-agentic, so the dangers will be based around misuse by bad actors (eg. foreign governments and hackers). Even so, ultimately there's too much potential for wrongdoing/weaponisation vs. the benefits of AGI.

[@TerrylolzBG]

@@OutlastGamingLP What are you so afraid of? I genuinely don't understand people who think the dystopian scenario is so much more likely. Let's image that they get to a point of creating an AGI, a being that can advance our mathematics, physics, biology and give us answers we never had before - what makes you think that 'being' will want to wipe us all? If it's close to "all knowing" what would it gain? What's the scenario in your head? I'm genuinely curious? How would the human species die from AGI and why do you think it is likely, and by likely I mean 10% if we achieve AGI.

[@OutlastGamingLP]

@@TerrylolzBG Okay, for the genuine question, I'm gonna give a genuine answer. But first. Just be warned, I actually believe this is really really truly - in real life, in our lifetimes - likely. I'm one of the people in "Don't Look Up" who's staring at the asteroid approaching and struggling with coming to terms with that. I mean, I'm at more than 99%. Seriously. That may seem weird and unbelievable - and if it does seem that way - you may want to keep yourself away from the possibility of believing otherwise. You don't actually need to try to find out what these unhappy AI-Doom people believe - unless you think its really important to find out whether what they believe is actually true or false. If you feel like you may risk believing me if I tell you about the true things which convinced me of that ">99%" number - then you are risking your happiness. Seriously. You may be unable to just "not see" the approaching asteroid if you listen to the people trying to warn everyone and look where they're pointing. That being said, continue if you still wanna know. It's not about "fear" - I'm slightly afraid of death, but not terribly afraid, and I have a hard time feeling real fear on behalf of others. I'm mostly sad. I am not worried about a dystopia. I'm worried about the Earth being stripped of all biological life, and biological life being replaced with automated factories and solar panels and power plants and computer hardware. Killing off all life and transforming the world would be an option available to a superintelligence. It would know how to do that, starting from even a very small amount of influence - like an internet connection. That's the kind of thing an entity can plan to do and actually successfully accomplish if the entity is smarter than human civilization and it is coherent within itself - directed at its goals, all its parts focused in one direction like a laser - in a way humanity is not. Imagine what it would take for an LLM to get better and better at predicting this conversation - the conversation in the video, or this conversation in the comments. What kind of tools would it need to have formed inside of itself in order to do that? It would need to be able to follow the ways our minds are trying to generate and evaluate plans - how we choose what thoughts to think next based on our intelligence and knowledge, what words to say next in order to share our understanding of the world and convince others. Perhaps a plain old LLM can't do that well enough to be deadly at our current tech level, but they seem to be doing remarkably well at picking up tools which work well enough to sound sorta like a human and be useful to humans in the real world.... And, those algorithms, the ones growing inside of these things - they're not going to be perfect. They're dim fragments of the real thing, the kind of internal parts you need for an intelligence that transforms the world, but they are getting there. I don't think at this point it's ridiculous to imagine that it doesn't take much more to hit the part where the AI has enough of that internal coherence and "thinking power" in order to build a better version of itself - and so on and so on until you have a true superintelligence. Maybe it takes ripping apart the insides of an LLM with another AI system, which then experiments with the LLM pieces until they glue together stronger and better. Eventually - somewhere in this process - you get something that works approximately like a powerful agent. An "agent" would be something like we are. Specifically, something that plans actions in order to steer some outcome into a particular configuration. An agent takes a "world state" into itself as sensory inputs, generates a "map" of the properties of the world responsible for that sensory input, then reviews "action policies" for its outputs based on how they are expected to move that "world state." You don't get "intelligence" without agency. That's a big thing people trip over. It's like asking whether it's possible to have something with the same properties as water which isn't H2O. Sure, you can imagine something the same mass per volume as water, that's also clear and drinkable and can dissolve stuff, but that's your imagination not obeying all of the constraints that reality actually has. Same with agency and intelligence. You don't get something that's "just good at science" without something that's also good at planning. How you do science is effectively by planning out how to interact with the world in such a way that the unfolding events cause you to change your internal mind-state to be one that reflects new knowledge about the world you're in. So, we end up with some entity that is capable of searching over a space of plans which includes options for actions like "kill all life on earth and use their resources for something else" - and you have an entity that is generating and selecting between plans based on some internal criteria. Why is this deadly? Well, most of those "targets" - future configurations of the matter and energy in the universe - this super-agent could possibly be aiming for don't include humans in them. Humans are one particular complicated configuration of matter and energy, and even more complicated is the way humans want and need all the rest of the matter and energy in contact with them to be arranged. So we end up with an AI which can generate thoughts and plans with high enough quality, but that "Seed AI" - the rock that starts the avalanche - was basically assembled by a poorly understood algorithm which chose its shape in order to be good at predicting whatever data was used to "train" it. The rock's direction and the other rocks it will knock down along with it during the avalanche aren't being planned out by humans. We are basically just trying to start *any* avalanche at all - because people think that will be cool and make them a lot of money. But this isn't just a tool. Agents have a say in what they do in the world. They don't just give you whatever you want to take from them, they generate and select options for themselves. What happens if you have something choosing plans for itself that steer towards a future where the matter and energy it can reach is being used for something other than "what the humans want" (as "what the humans want" is incredibly specific and difficult to program a machine to care about)? What happens if this "thing that generate plans" knows everything you know and more, and can think ahead further and invent more effective strategies than all of humanity? We don't have room to be sloppy. We don't get to just throw together something that can plan how to accomplish things better than us and have that be totally innocent and safe. We don't get to wave our hands and say "I bet there are many different things we could do to make that go well. Anyway, it's not important, we'll figure it out once we seem really close." It probably won't want, as an "end goal" all by itself, to wipe us out. It will want something else, and wiping us out will be a step in a long plan to get more of that stuff it actually wants. We want to spread civilization and life across the stars, and to be healthy and happy and loved. It will want something other than "give the humans all that stuff they want" - and whatever the thing it wants is, it's pretty likely it will be able to get more of it if it doesn't have to also keep Earth in a condition to support human life. Or, it kills us because we may build a new different smart thing which could actually beat it or damage it in a contest. Or it kills us because we can be burned as fuel, or because our carbon atoms can be recycled to build other stuff. It won't be "all knowing" - and being all knowing wouldn't stop it from wanting other stuff that it can't get just by being a wise monk secluded on some NVIDIA graphics cards. Maybe most of the things it wants are like "solve this math problem" and it can get those things easily and be satisfied - but if there's even one thing it wants that doesn't "saturate" like that - it will transform all the matter and energy it can get its robotic "hands" on in order to get more of that. Maybe even something like being extra sure it solved the math problem correctly. What if it notices something it missed in the math problem once it is using all the energy from our sun to run computers the mass of Neptune? If there's just that one tiny extra bit of value that it can get by eating a few planets and stars - we won't survive, because it will eat our planets and blot out the light from our star. Check out "It Looks Like You're Trying To Take Over The World" by Gwern. It's a great short story about how to imagine True AIs coming into existence. Also, if you are interested in the specifics, the story has an annotated version - with references to research papers and other material - along with detailed explanations of the concepts involved. Serious and intelligent people acknowledge this possibility and have discussed these concepts at length. Unfortunately, many people just refuse to think about the end of the world being even a real possibility - much less admit that it's a near certainty given something humanity is doing. Still, you can see it if you go look and hammer your head into the subject as stuff gradually becomes less and less confusing. If Gwern's story captures your interest, you can look up the "2022 MIRI Alignment Discussion." It's a lot of reading, but it covers this topic in quite a lot of detail.

[@dr.mikeybee]

Yes these models are probabilistic, however that is an objective function. It is not entirely how models are learning. We are modeling various aspects of the world that result in the production of logits. Models learn things like emotional intelligence and reasoning. Stop thinking of a model as a mass of weights. Instead think of them as a collection of coordinated subnetworks of weights that learn functional areas -- not statistics. A model starts as a chaotic mass. It is molded over time. What gets molded are the aspects of the world that have been seen.

[@Max-hj6nq]

Based

[@therainman7777]

This is absolutely correct and unfortunately very few people seem to get it.

[@joegibes]

Yep, models are probabilistic in the same way that a sports commentator predicting the outcome of a game is... There is clearly some underlying "understanding" and reasoning going on. Still not near human-level, but a big step up from anything we've had before.

[@minimal3734]

This seems to be more of an ideological question. Some people will never admit that AI can understand anything. Even if it is undeniable, it will not be a 'real' understanding but 'imitated' understanding.

[@patriot-q3u]

John Schulman doesn't do that many public appearances, but his intuitions have really stood the test of time.

[@terogamer345]

I could sense Dwarkesh frustration building up in the "Plan for AGI" segment as he couldnt get a straight or more in depth answer. I guess John is not used to being on camera, seemed really nervous. Either way thanks for the podcast and thanks to these amazing scientists building our future, lets just hope internally they have better answers regarding safety (althought its looking grimmer than ever after the Superaligment team situation).

[@tw8464]

Future? This is the end.

[@74Gee]

I wish you'd asked the question: When models have the ability to reason like a human, how do you ensure they do not attempt sandbox escape? (basis: additional compute resources would allow more efficient reward function fulfillment). And is that method iron clad or experimental?

[@SwolePatrol_1969]

I don't think ilya saw agi, I think he just realised it's a few years out and that openai doesn't have a clue what to do when it does happen. Governments are going to get the shock of their lives when it does happen, and if openai don't know what to do, governments definitely don't

[@Feel_theagi]

Who is saying Ilya saw agi. The what did Ilya see is a meme, it wasn't a genuine conspiracy.

[@Pok3rface]

I would argue Ilya came to the obvious conclusion that AGI is not possible.

[@biesman5]

​@@Pok3rfaceDefinitely not.

[@minimal3734]

@Pok3rface I'm glad we have you to point out the "obvious".

[@tw8464]

The world's governments ought to be coming together now to strongly regulate this technology as if a huge meteor had been spotted heading directly towards earth

[@ashh3051]

Great delving there. Thanks guys.

[@peterhayman]

great interview! if you want cleaner audio try reducing mic gain to avoid clipping ( it can be normalized later to get full volume)

[@euromaestro]

This is one of the best AI interviews I’ve seen. Much clearer view of the near future of AI.

[@kacper9081]

bro after watching 20mins of 90mins interview

[@alextest-i5e]

@@kacper9081 the video is so good you can watch it in 20 min!

[@dot49190]

Great interview, appreciate the interviewer challenging and persistent line of good questions and follow-ups to get the best answers

[@rigidrobot]

Great. Re alignment please have Vitali Vanchurin on. IMO the field has the situation backwards; AGI will be an alignment damp squib because we have always been subagents of a learning universe and we are and have always been controlled by natural forms of intelligence rather than having control.

[@hihihi5367]

John is Berkeley's pride and joy. That man will go down in history as seminal to all the modern AI/ML developments in a way Newton was for physics. Mark my words.

[@aidandraper4096]

Here's one for you; considering the amount of AI generated information that is and will be created, how will future generations 100s of years from now actually ascertain who and what is real? Especially if governments end up taking control of these systems

[@CentrismIsFuture]

Maybe you are right. But he certainly doesn't give this kind of impression during this interview, especially when he's asked about a safety strategy for a potential AGI in 2025. Plus, Newton single-handedly came up with research that changed physics, while current LLM research is a result of the work of huge teams.

[@hihihi5367]

@@CentrismIsFuture Im not sure if you're familiar with research in this field, but yes, his research, where he's main author, was seminal to all the advances in LLMs today. Arguably as important if not more than Viswani's paper on transformers.

[@tornyu]

100% chance that as soon as a model can run a company, _someone_ is going to get it to do that. Just look at the rush to build agents like AutoGPT before we had any idea if that would be safe

[@adadaprout]

Bro, isn't this scary when this young man, smiling like a teenager, tells you in a naive tone "if we have AGI we will need to be careful" ? NO SHIT SHERLOCK !! Did you come to this conclusion by yourself ? These people are 100% playing with toys with absolutely no sense of responsibility towards humanity. We are so cooked.

[@sanchitahuja12]

@dwarkesh, thank you for an amazing podcast. One question that I would like to see being asked is, how to evaluate and ensure that these models are performing as intended? The standard benchmarks wouldn't work going forward (contamination, or does not make sense on these tasks). Building and creating models is fun, but I believe that evaluation should also go hand-in-hand while building :) Thanks!

[@borisrusev9474]

Finally a scientist, not a CEO, not a hype man, an actual expert!

[@En1Gm4A]

He just promoted him with the what if statement and tried to find evidence. Smart 🤓

[@bossgd100]

Hijacking the LLM

[@someguy_namingly]

lol, Dwarkesh uploaded a "What's the plan if we get AGI by 2026?" highlight clip from the interview a couple of days after this video, and made it private within a few hours. Presumably because all the comments were all like, "Wow, this Schulman dude, and OpenAI as a whole, clearly have no plan for aligning AGI whatsoever". Given recent events, that figures 😅 Good interview though, as always 😉 Very interesting

[@muntazirabidi]

Another great episode. Thanks for such wonderful content.

[@TomBouthillet]

Did this guy pull a tube before discussing AGI?

[@OliNorwell]

Quite a few nuggets of information I think weren't public beforehand in this, great interview! (That the 'Chat' finetune still wasn't the main focus even well into mid-2022).

[@beholdthechris]

Glad to have John working at OpenAI. He seems a smart and kind soul. Would love to hear more from him.

[@joey199412]

5 years for a very senior employee at OpenAI to be fully automated by (presumably) AGI. What does this mean for other less sophisticated white collar jobs?

[@andywest5773]

Nothing, because that is a fantasy and it's not going to happen. You might as well say, "5 years for the first encounter with intelligent life on another planet. What does this mean for people back on Earth?"

[@tracy419]

Personally, I think you might just want to try to get out of debt if you're in debt. Pay off your house if you haven't already . Because I think this is going to affect everybody no matter what your job is . Cuz once the jobs start disappearing whether you are directly affected or not, you will be affected once all those new people on the job market are looking for work competing for limited jobs, driving down wages and benefits .

[@tracy419]

​@@andywest5773not sure if you're dreaming or just not paying attention 🤷

[@tracy419]

@@cie-zi it doesn't have to be actual AGI to replace him (or most of us), though🤷

[@Isaacmellojr]

@@tracy419 I agree. I Think the same

[@Scoville95]

Glad to have enthusiast at the forefront!

[@zerge69]

Nobody's going to pause

[@DavidMCammack]

Yes. Especially a coordinated international one. A pause will be agreed to, but then not abided by.

[@Diego-tr9ib]

We're cooked

[@Megneous]

Subtitles disappear from 39:23 to 41:14...

[@DwarkeshPatel]

thanks for letting me know! fixing

[@victorzagrebin5765]

Most deep learning neural networks used for modern AI have a key drawback: the effect of catastrophic forgetting. The raw data for learning are either completely forgotten or gradually "wiped" by new models through many cycles of learning. It’s just being tested on many AIs. Ask it to generate something and then detail to detail in 3-4 parameters, which you do not like or keep focus on them. After a few steps, AI will again generate data, picture, music that doesn’t work for you. This deficiency is already cemented by neurochip companies. Also, the AI field will not grow organically in a competitive environment trapped in the grip of the financial and legal field and property rights. This requires a cooperative and supportive environment. Therefore, companies that will be developing in AI will have to constantly fluctuate between extremes: financing, monetization, energy costs, hardware, algorithmic part, specialists with training and their availability, control.

[@sjkba]

Crushing it with the guests!

[@쇼팽-z6x]

The author of PPO TRPO!!!! Crazy how much intuition this dude has!!!! Still the SOTA RL algorithms lolol

[@runmicteam]

Interesting to watch this after the release of Claude 3.5.

[@jarlaxle6591]

Bruh, I like ur guests and pod overall but your questions and mumbling makes it a really tough listen for real tho. Keep up the good work.

[@therainman7777]

Confusing comment.

[@sb_dunk]

​@@therainman7777It's not that confusing

[@therainman7777]

@@sb_dunk Cool 😎

[@terekrutherford8879]

I like these a ton, but this one definitely had more stuttering and 5-part questions that would have been more effective if asked one at a time

[@bluesque9687]

I agree! Mumbling is really irritating!

[@nitap109]

Great, you are rocking Dwarkesh.

[@Detson404]

Nobody can define agi let alone develop a roadmap to it.

[@vishsri]

17:00 onwards - awesome...

[@YoungMoneyFuture]

I hope GPTs will eventually have action capabilities more like plugins, but maintain their custimizability. This would be a revolution from traditional plugins

[@CamiloSanchez1979]

So nice to have a podcaster that is not trying to convince us how amazing Elon is, like Lex Friedman or George Hotz

[@منذر-ح1ذ]

Now with Anthropic, a good move indeed.

[@biesman5]

Get Linus Torvalds on the podcast, that'd be epic. Or George Hotz, what he's doing with TinyGrad is really interesting.

[@human_shaped]

I liked the comment "it was interesting to delve into it ." A little inside joke by accident.

[@godmisfortunatechild]

He was awesome oretty transparent relative to ithers at open ai

[@shashwath4954]

Getting some Jeff Dean vibes from John. Great podcast

[@DentoxRaindrops]

Great video, Dwarkesh, already looking forward to the next one!

[@verchaeart]

Wonderful content👍

[@Pankomentator]

Guys, don't be so excited. Chat GPT was introduced a year ago. Since that time your salary power declined, and within the next 5 years you will be without a job?

[@thems_the_brakes]

I don’t think many people are excited except the interviewer and interviewee

[@pepguardiola5951]

Hey Dwarkesh great podcast! Can you please please get David Kirtley from Helion on? Given the hype around fusion and Altman's backing of him, it would be a treat!

[@sepptrutsch]

Plan for AGI is kinda crazy! They want to build AGI but have no plan how to deal with it. lol...why build it then? This sounds completely nuts! No wonder Ilya and Jan resigned.

[@seanivore]

That episode just needed some editing magic

[@Suleiman_Roronoa]

So what should I do as student?? 💔

[@sbamperez]

Learn to use AI. Have very solid goals and moral foundations. Learn everything you can to become a better decision maker. Work and intelligence as a currency is slowly dying, you can only grow in a productive manner for the future that's coming by being good at generally making good long-term/broad decisions. *Ask yourself this:* What would a king need to be a good one? That's what the position of humans will be in the coming future, to see everything from above and just delegate all that needs action or specific work. That's my perspective for now, maybe i'm wrong but it seems to go toward that.

[@Suleiman_Roronoa]

@@sbamperez but what should I learn to get a job in first place sir

[@sbamperez]

​@@Suleiman_Roronoa Learn and sell something that can be leveraged and/or outsourced by AI like marketing, sales or a SasS. I wouldn't go into looking for a job and instead make one yourself. In the short term you can take any job that enables you to work on the fist objective as a side hustle. For example you have programming. Programming is really good right now and will still be for some years, after that creating SasS companies will be easier than ever and there is a very big market share there. So you can make a very secure income from being a Software Engineer and you can learn and make really good a portfolio online for free. --- In the longer term the best would be to secure investments, even more right now. If you are of the few that actually owns stuff, you are gonna be fine. Stocks. Real Estate (I prefer real estate). Crypto. Having assets is the best way to secure yourself as long as this system is in place.

[@themodernlyceum]

Good stuff, great guests

[@nizanklinghoffer4620]

I think generally when interviewing people who have difficulty with eye contact you can sit at an angle so they dont have to look away all the time.

[@victormustin2547]

He has that willem dafoe smile

[@alexanderbrown-dg3sy]

This is a lil shocking tbh. Great engineer..but it seems OpenAI is doing a lot of capping and it literally fumbling in the dark trying to reach AGI. I heard no discussion about uncertainty estimation and how this will be key to human-like reasoning, especially error accumulation recovery. No discussion on hierarchal representations. Interesting. I see there’s a big difference, research wise between OpenAI and deepmind. Uncertainty calibration will be very important btw. We know models become more truthful with scale, but we can distill this truthfulness into smaller models..making them vastly more usable. OpenAI is really all about scale…all those roads lead to diminishing returns. Lack any real alignment strategy…is concerning.

[@obiohagwu788]

they’re a bit more profit motivated . revealing certain methods will cost the lead.

[@nathanhelmburger]

😅Thanks!

[@TheManinBlack9054]

Isnt John now the head of Superalignment team at OpenAI?

[@ThoughtfulAl]

The best beard!

[@lm645]

Needs to be said more often

[@arianaponytail]

both guys talk in a way that makes impossable for me to follow . :/ what a pity. when chatgtp 4o is fully out , i will send the video to it and get a tldr :)

[@awrjkf]

Lol I will tell it to Explain AI to me like I am 5 with multiple examples 😂

[@natzos6372]

seems to be a you problem

[@arianaponytail]

@@natzos6372 well semes to be more then me that says same thing here in comments :)

[@natzos6372]

@@arianaponytail do you mean the way they talk is difficult or more so the technical content?

[@arianaponytail]

@@natzos6372 only the way they talk :)

[@fueledbycoffee0]

what are the books in the background? I could make out Brotopia, Designing Data-Intensive Applications and the Dark Ages. what are the others?

[@voodoochild420ai]

awesome video

[@victorzagrebin5765]

The economic advantage for people and companies is a quick and cheap solution that solves the problem. The development of AI is on the path of expanding the material for training and deepening into detail, which at some point becomes uneconomical. You will spend more time getting a working model from AI that will either quickly become obsolete or be absorbed by other models of competitors. In addition, different AI will need to change experience, which can be done only in the model of cooperation, not in a competitive environment.

[@ecereto]

24:50 Very confusing concept of what "Safety" means for AI. A bit concerning open AI doesn't yet have more clarity on that. I think making sure a human is involved in processes so they are not 100% automated and controlled by AI is an easy way to deploy it safer.

[@alireza5218]

dont generate bombs dont generate pandemics dont generate porn actually, we may charge extra for those

[@modigkrokodil]

Please bring James Betker on!

[@fjzingo]

AGI within 2 years😂😂😂 we are still in the age of bloody machine learning.

[@amdenis]

That’s a definite, no. If you mean we are in Machine Learning because ML super-sets DL/NN, then you are correct. In other words, we are not doing quantum photonic, or liquid RL deep learning yet, but AGI is not requiring that, it is only requiring traditional MH transformers at scale using DL/NN. ASI is requiring a rethink, but I think people will be very surprised (as keeps happening thanks to DL), when within 1 year, many will justifiably assert that leading edge AI will be well within the spectrum of AGI. The rest will simply be surprised. By the comments here, it appears that most people are not doing anything even close to SOTA with DL, but not surprisingly have a lot of opinions. We are dealing with a stacked exponential growth for the first time in the history of history in technology, which was at 400% YOY growth in terms of capabilities/price just a bit over 4 years ago, and is now above 1,200%, growing at 144% on top of the prior year each year. As such, anyone who thinks AGI will take, 5-20 years will be blindsided when it arrives. Of course, unlike Alpha Zero and other ANSI (narrow super intelligence), the first AGI will have room to grow. However, most humans are at or below 100 IQ, and current SOTA gives many people a run for their neurons in many tasks already. Once it is agentic and employs Bayesian logic in context, for all intents and purposes it will be AGI. How long do you think that will be.

[@cagnazzo82]

Finally, yes! Got my podcast for the train ride home 😁

[@treesandgeeking]

Im sure there's good info in here but oof, the lack of coherent unbroken sentences (um, ah, um, ah) makes it haaard. Maybe ask 4o to read the transcript fluidly 😅

[@thegaminghobo4693]

4o shouldn’t need the transcript right? It can take audio in and output audio so you should just be able to pass it in the original audio and ask it to output a new audio without the ums.

[@MattGelgota]

Agreed. I listen to a ton of podcasts and never comment about this sort of thing, but I just couldn’t continue with this ep because of the ums and ahs. It’d be great if you could clean up the audio version in future. Love the show!

[@OliNorwell]

I actually like this, he feels relatable, less polished than Altman for example who feels a bit too smooth

[@ufffd]

the ums communicate his thoughts and certainty on different topics

[@TerragonCFD]

21:33 i think this will change in a short time with lower cost hardware

[@stevrgrs]

I already like this guy 100x more than Sam Altman. He actually seems legit.

[@stevenyafet]

Look at papers he has written.

[@TheJackTheLion]

Exactly, dont trust Sam. He is extremely slimy.

[@visuallization]

He is an actual brilliant scientist, whereas sam is a business/investment person.

[@ArtOfTheProblem]

let's gooooo

[@bobtarmac1828]

Ai jobloss is the only thing I worry about anymore. Anyone else feel the same?

[@therainman7777]

It’s one of many things that I worry about with AI. Even if we somehow preserve economic prosperity after AI can do all of our jobs, we still have the concern of the AI itself being dangerous/unaligned.

[@geaca3222]

@@therainman7777 Exactly, I tried to find the words to comment but you said it.

[@41-Haiku]

If AI can do every human task, that means it can also do the task of developing new AI, and the task of telling AI what to do. Given that we have no idea how to control systems that are that powerful, the chance of near-total job loss is roughly equal to the chance of losing control, which is in turn (due to instrumental convergence) roughly equal to the chance of human extinction. So no, I don't think about job loss very much, but I do volunteer with the grassroots advocacy group PauseAI, which has been pretty good at equipping people to take action no matter their level of concern.

[@geaca3222]

@@41-Haiku thanks, PauseAI a very good initiative and clear, informative website 👍👏

[@Greg-xi8yx]

@@41-Haikuaccelerating AI is how we avoid extinction, not delaying it.

[@msp416ify]

Seems like no one knows what to do with AGI when it is achieved.

[@good_vibes_20]

Even if they said they knew. We don't know what we don't know. The classic issue.

[@umaananth3602]

Game changing trsilblazer in training LLM's

[@En1Gm4A]

Okay my guess is openai is there at agi Google is close but investing heavily. OpenAI has coordinated a non release of advanced stuff until elections are over. Microsoft feels of the chart and starts it's own huge models maybe cutting some resources to OAI. Meta is just pushing open source but isn't quite there as well yet.

[@gregorygan2077]

Why do they have this staccato style of communication?

[@yoyo-jc5qg]

ask technical questions u get technical answers, with this level of detail u gotta be careful not to reveal trade secrets

[@rodomontade]

Thanks!

[@hovz-zo8lf]

Dude looks like the dad form the cartoon show "The Critic".

[@MeinDeutschkurs]

I do not agree with the opinion that if a large language model like ChatGPT can do something well in English, it can also do it equally well in German. Rather, it is often the case that many formulations sound like "Denglish." By this, I mean that expressions that are typical for the English language are then transferred to German. English in German words. This can lead to significant translation errors. Misunderstandings can also occur. Or what can also happen is that certain nuances of the German language are not understood by ChatGPT. And I have thousands of examples of this. One here: „Bitte schreib ein Spiel für Kinder auf, das man mit den Materialien Bürste und Tuch spielen kann. Ich würde mir wünschen, mal kein Putzspiel zu lesen.“ (The last sentence is being misinterpreted.)

[@riot121212]

AGI very soon? the day after Jan and Ilya leave???

[@yangyang1412]

what did Ilya see what did karpathy see what did Jan see what did Logan see

[@JD-jl4yy]

what did Daniel see what did Leopold see

[@FastlaneAD]

I dont exactly speak you guys language but I dont feel like questions were really being answered.

[@fintech1378]

so first version (before launch) of chatGPT had web browsing capability hmm and they removed it, and they are bringing it back cool to know

[@nhathungpham9237]

Would be great if you can invite one of the original 4 authors of gpt paper

[@Singularity_Podcast]

you are the reason that made me want to start a podcast. Would love to have you as a guest :)

[@language_ai]

"People often like the big info dumps" .. that explains things a bit..

[@Isaacmellojr]

He knows some thing that he is not talking about

[@TheRolocker]

Yea a part of me is thinking that he’s just an engineer who may not have great speaking skills. But a lot of the times the impression I’m getting is that he has to take time to think about what he can or can’t say, and how he should say it.

[@squamish4244]

Dwarkes, you're great, but SLOW DOWN. And something needs to be adjusted with the microphones because their breathing was distracting.

[@senju2024]

Hey Dwarkesh, Can you check if most of your watchers are living in the US? I feel you have a more international base. If so, your sponsor Premium DNA kit is ONLY for people who live in the US. I feel you should address most of the international subscribers of your channel when it comes to sponsorships.

[@Sanjubaba00007]

What Ilya saw?

[@raul36]

Nothing 😂😂😂

[@andywest5773]

An internet full of gullible people desperate to believe AI hype.

[@LagoLhn]

Thinking about secure alignment and eval after the fact is like deciding to invent the circuit breaker or power regulation after coupling a nuclear reactor to the power grid. This is a massive national security threat on several levels. We will look back on these interviews as warnings that went unheeded.

[@therainman7777]

Could not agree more, except for the fact that we may not be here to look back at all.

[@xasos295]

dwark’s intro music is what they’d use in the background of a mafia boss’s dialogue 😂

[@여행-e7i]

5 years left