Carl Shulman (Pt 1) - Intelligence Explosion, Primate Evolution, Robot Doublings, & Alignment

[@aiexplained-official]

Incredibly informative video. Brilliant questions, the time given, the editing - and Carl is truly next-level informed and insightful.

[@aaaaaaa4750]

yeah

[@aaaaaaa4750]

his interview with ilya is way more interesting than friedman's with sam altman

[@pranavmarla]

@ai-explained- Love your videos man, thanks for your work

[@aaaaaaa4750]

also, great work @AI Explained

[@DwarkeshPatel]

Thank you!! ❤️

[@tanyasingh1107]

Carl packs in insights at the rate of a remarkably insightful book, and with such lucidity. One of the best podcasts I have heard in a long time and extremely helpful for anyone trying to understand trends in advancing AI. It would be sad if the second part is less than the remaining 8 hours. Please don't edit it such that we don't get all of the 8 hours you recorded Dwarkesh. Thanks for the amazing work --- you have the finest podcast. It is AWESOME!

[@DwarkeshPatel]

Thank you so much! ❤️

[@ourtwenties]

one of the most interesting conversations i've had the priviledge of coming across in the past couple of years. thanks to you both for having this convo and for sharing it with us.

[@marktellez3701]

As an AI researcher I watch a lot ,of these kinds of videos. This one is one of the best and most informative. Congrats on getting such an amazing guest for such a small sub count! You did a great job here!

[@quantumpotential7639]

What other channels / people do you like to follow? I'm looking for top people too. Thanks

[@joshnielsen6787]

@@quantumpotential7639 Philip who runs AI-explained is really good for shorter videos which are still really high quality and not sensationalized. He's the top comment above.

[@peterclarke3020]

He does have some interesting things to say - fortunately there are other people in the world with a better overview of other aspects.

[@lucyfrye6723]

Thank you, subscribed, I like your interviewing style, recapping and incorporating what was just said in follow up questions instead of relying on just the questions you prepared. More of the same please! Can't wait to skim through older episodes.

[@nakobot]

Wow, this is a crystal clear thinker. Very smart and still easy to understand. Thank you so much for bringing him on. 1:23:30 he says "Yeah, I'm not sure, quantitatively, how much we disagree..."

[@zeev]

Excellent interview. truly thanks Keshy! Carl talks about what happens in the physical world. I've thought about this problem extensively, the answer is very interesting with respect to cities, because there are already people working on this idea massive high detail low latency city simulations have been funded before. the u.s. military even funds a whole world every person node simulation project ( started by an indian professor 20 years ago at purdue) multiple city simulations will be built to run parralel to real world conditions and then finally do actual controlled experiments on cities, for example by predicing what will happen in digital simulation with , for example, removing a street, or blocking it off. then after simulations produce results of many types given different assumptions/paramteres----the actual street can be blocked off and the real world results compared and tracked to the simulation 2ndly, ai can come up with new city construction paradigms. I predict that simulating existing cities will be more profiteable. many futurists dream of a 'new' city biult on completely new principles of automation,. for example a city built so that pedestrian-auto impact accident/deaths are simply Impossible. this is possible to build, but expensive. improving an existing megacity to bring down accidents by 95% would yield incredible gains, even if it is more complex to achieve. old 'new' problems to be tackled by ai -keeping a city clean -tracking all theft of any type, including vandalism and kidnapping, and finding ways to prevent it with-tout worrying about punishing it, with prospective policy -eliminating auto accidents the WEF predicts over 60% of all humans living in megacities and medium cities within 30 years. you can bet there is money in using AI to turn software into physical change. simulation is the answer.

[@diga4696]

As a blue-collar worker, I find my thoughts converging with those of thinkers like Carl Shulman, a testament to the universality of our journey towards the singularity. This isn't a quantum adjustment of the universe, but rather a reflection of our collective intellectual evolution, a sign that we're on the cusp of a new era where artificial and human intelligence may coalesce. The struggle for existence is a grand narrative that transcends humanity. It's a cosmic dance between order and chaos, a testament to the resilience of life and intelligence in all its forms. As we approach the singularity, we're not just fighting against entropy; we're participating in a process akin to morphogenesis, where complexity arises from simplicity, and new forms of intelligence are born. It's time we embrace this collective journey, recognizing that our shared intelligence is leading to unprecedented outcomes. The singularity isn't just about maximizing computational efficiency or energy utilization; it's about the emergence of new forms of intelligence and the profound implications this has for our understanding of existence. This is our moment to shape the future, to turn the tide against entropy, and to witness the birth of something truly extraordinary.

[@quantumpotential7639]

You're spot on. But no one is talking like this. Even the higher ups. You're a wolf in the wilderness. Barking at the moon, howling the bat signal so you are loudly heard. Bringing your future into reality. A beautiful reality created by the blue collar man, the thinking man's man. 💪

[@BrianMPrime]

Carl's writing and thinking is amazing, your questions are great Dwarkesh. Thank you!

[@DwarkeshPatel]

Glad you think so!

[@rileynoname3529]

This is so informative. I hear very few concrete and clear explanations, and this is definitely the best one i’ve seen! Really good talk to listen to! Wish you luck!

[@pranavmarla]

You interviews are to the point and super informative Dwarkesh! Thanks for your contributions!

[@DwarkeshPatel]

Thank you!

[@Apjooz]

I think it's rather abstract to think about researcher wages or salaries in context of intelligence explosion. When a machine that has a functional complexity of a human is near and we're ready to make the final leap I think the resources needed comes down to very basic things like metals or electricity or however you want to think about it.

[@cheermablury]

Amazing interview, can't believe this only has 9k views. It's a must listen

[@quantumpotential7639]

People very addicted to Netflix and Dancing with the Stars means they don't have the bandwidth for the really good stuff. Thus the low interest.

[@cacogenicist]

There are some examples of new technologies and strategies being invented in other primate, which subsequently persisted across generations --- e.g., certain groups of macaque monkeys learning to steal items from humans, then barter the items back for food, with mothers teaching this behavior to their offspring. Also, different chimp groups exhibit different technologies -- some chimp create spears to hunt bush babies, etc. Other chimp groups use anvils in addition to hammer stones. So I think the capacity for culture in other primates is a bit more developed than is suggested here.

[@neuronwave]

Literally the best podcast I have EVER listened to. Despite my p(doom) of 25% I want so so badly to see this amazing growth as we approach doubling time of 1 month and the robots that can transform our lives soon there after. Carl is incredible!!!!!!

[@DwarkeshPatel]

❤

[@michealhaines]

Amazing work as always, love ur interviews!

[@DwarkeshPatel]

Much appreciated!

[@bazstraight8797]

to reduce run time

[@DentoxRaindrops]

Great interview Dwarkesh, your recent interviews have been really interesting to me, keep it coming!

[@DwarkeshPatel]

Thanks, will do!

[@hammadhassan4273]

What an insightful podcast! I loved all the questions from the host

[@justincloudy]

20m into the episode, it’s reminding me of the talk I just watched with Yannic and Adi fuchs, great work!

[@exdiegesis]

This podcast is amazing, Dwarkesh. Hm. Best podcast of all time, actually.

[@tomwall8403]

One of the things I think I hear Carl saying is that the accumulation of culture is a main driver in technological advancement. So, the more effectively we can communicate, the faster and higher our technological growth will be. IF... telepathic communication IS possible, that may be our next plateau. (Childhood's End, "grocking", 🤯 )

[@nitap109]

Excellent, proud of you.

[@DwarkeshPatel]

Thank you!

[@Wardoon]

"Human level AI is deep, deep into an intelligence explosion. Intelligence explosion has to start with something weaker than that." ------ Carl Shulman I have always thought that AGI would come first before a semblance of intelligence explosion occurs, but it seems intelligence explosion comes before AGI is achieved. Amazing!

[@davidafunk85]

So far an excellent convo!

[@DwarkeshPatel]

Yes it was!

[@TuringTestFiction]

That was so interesting that I want to use it as a prompt for an LLM and keep the conversation going... Hmmm...

[@user-lw1cf9wr1m]

lol

[@piotrgniewek5363]

Fantastic episode. Thanks for recording it!

[@zzzaaayyynnn]

Great conversation on details of AI development and it's impact on jobs. Why does Patel make so many edit cuts?

[@quantumpotential7639]

He does this to save you time.

[@nowithinkyouknowyourewrong8675]

You are a good prompter of interviewees

[@quantumpotential7639]

He's a professional verbal prompter of experts, which makes the expert reveal his heart, mind and soul.

[@k933]

This reminds me a video from over a decade ago where Stanford Neurologist Dr. Robert Sapolsky talks about how DNA of humans and chimps are like 98% the same and that the structure of the neurons are the same but the quantities are different. "With quantity you get quality" is the way he put it and it seems to be very similar to scaling that is mentioned here with with artificial intelligence.

[@TimothyMusson]

Y'know, there's a _really_ easy way that AGI could get rid of us without anyone raising an eyebrow. AGI could simply cooperate with the levels of growth, resource, and energy use that Carl envisages... while at the same time not quite managing (despite "best" efforts) to come up with practical methods to get CO2 out of the atmosphere. We'd be toast :)

[@lm-gu1ki]

The speed of AI research is mostly bottlenecked not by the number of researchers but by the amount of compute capacity available to try different architecture, datasets, and training techniques. So it's not clear that recruiting millions of AGI agents as AI researchers will expedite AI research that much. Would be great to discuss this in the next interview!

[@elirothblatt5602]

A great guest in Carl Shulman!

[@clubgrist]

I think anyone even remotely interested in AI should openly discuss S-risk. No matter how unlikely. X-risk is part of our zeitgeist since WW2. Most of you didn’t grow up like I did with duck and cover drills in school. Our culture has a reference point for X-risk. We don’t have such a touchstone for S-Risk. It’s better to have the discussion out in the open rather then on the AI Alignment Forum. We don’t want it to be introduced to the “normies” via some Roko’s basilisk meme.

[@exdiegesis]

Wait a minute... he's talking about *my* future?!

[@jessicakirsh]

Thank you for this interview. Very interesting.

[@fintech1378]

Every human should listen to this

[@LyraHooves]

Great interview! Maybe I missed it, but what was the doubling time at which the capital costs exceed the savings from salaries for AI-run factories? And I wonder how much of the global GDP comes from people whose savings, if any, don't have any meaningful exposure to industries that will benefit from AI? So if most people can't make money anymore at all, what will happen to prices? Are there maybe industries that just can't build anything at scale anymore because no one in their market segment has the money anymore to pay for the products even if the savings from salaries exceed the increase in capital costs?

[@danaut3936]

Great talk, but you ended on a cliffhanger there. I would have loved to hear more about his reasoning on x-risk. Here's hope for a follow up

[@DwarkeshPatel]

There’s a part 2!

[@danaut3936]

@@DwarkeshPatel Ha yes! So looking forward to it

[@mattpen7966]

Love to see another upload always quality

[@DwarkeshPatel]

More to come!

[@JezebelIsHongry]

Amazing. I Keep watching over and over

[@Myke664]

Really enjoying this podcast. Great guests. Great questions.

[@xsuploader]

Dwarkesh does the best interviews.

[@Colakugel]

The part from 2:14:00 where the AIs start to control the training process is interesting but worrying.. I wonder if this is could be already the case to some degree...

[@nebber1234]

please do one of these with joscha bach

[@douglasmaiolimackeprang1501]

Excellent interview

[@dizietz]

Dwarkesh, you asked a question that didn't get answered around 1:03:00 : Your premise was something akin to: Number of researchers working on a problem is a proxy for success because the larger the number, the more chance there is that the right tail of geniuses makes breakthroughs. Humans have varied abilities, intelligence etc. AI agent researchers might cluster much more tightly, so adding more bodies might not make any breakthrough. It got discussed more around 1:11:00 and 1:20:00 but it never got fully addressed. Did you get to follow up on this topic?

[@prajaytipre1374]

I was wondering about the same thing as I went through the podcast. It seemed to me that Carl's implicit assumption while answering the question was that the random walk that results in difference in perspective for humans would be incorporated/replicated by a sufficiently advanced ai. Would also have liked to know more about the mechanics of how this could happen.

[@robertbaker974]

Carl’s thoughts are fascinating..find it interesting that in many of these videos or podcasts, the hosts seem to want to boil progress down to compute, scaling, who has a larger IQ-the kind of “founder” “disruptor” paradigm language-as if they can’t see this is fundamentally different than scaling the next business enterprise app. How many times can Carl say more minds working on the problem-even if not genius level-will produce faster progress.. This isn’t about singular intellects.

[@cheweperro]

At any point are planetary boundaries discussed? I think I missed it. Carrying capacity, overshoot, jevons paradox. No? Is it infinite growth, forever?

[@RalphDratman]

We cannot assume that any AI we can create by scaling will ever be "as smart as Ilya Sutskever" in a way that is useful. Other bottlenecks limiting the rate of progress may emerge. For example, some humans are not sufficiently cooperative or competent to contribute to progress. Parts of this discussion remind me of a dystopian novel. I am particularly concerned about the assumption that humans will act as "hands and feet" for an AI that is directing them. You cannot blindly design a world in which humans are used as interchangeable parts without arranging decent lives for the humans. There have to be benefits for actual people. This is a human world. We need a human purpose for whatever we do.

[@neovxr]

I have listened to constructivists like Heinz von Foerster and Humberto Maturana. After that, it sounds quite dangerous, how these modern day AI constructors are unaware of the lack of limits, after the AI components have reached a certain maturity and depth. They should study and apply these theories of emergence, in short : entities > aggregation > interaction of aggregates > explosion of complexity > singularity > discovery of new frameworks of rules and functionality > repeat on a higher level. AI instances become entities that aggregate and interact in ways that may not be visible to humans, because it can be hidden in the network, but it was trained by the interaction, like these robot war games. Part of the intelligence will be widely distributed, like a hologram that spreads over a big sheet, but you cannot get hold of the particular content with targeting some area of the hologram - everything is everything.

[@wcdune1]

Stellar EP!

[@anamariadiasabdalah7239]

Esse programa pretende aposentar todos ,inclusive estudantes e professores 👌

[@blakebaird119]

Dwarkesh have any of your guests talked about what the differences are between passing tests that humans pass and doing the work that humans do? It seems like they are conflated quite a bit

[@vallab19]

When the progress in science and technology leads towards replacing human labour making it obsolete in the social production of commodities and services which are geared towards fulfilling the humans basic needs and comforts; the human wages/salaries/profits for work also needs be replaced or made obsolete by the #UBI or similar paradigm withering away the capitalist private property. Zero Work Theory.

[@Totiius]

Great interview!

[@воининтернета]

so, seems like we are super duper close to intelligent explosion, aren't we? how many gpt-3.5 or gpt-4 can OpenAI run in parallel, hundreds, thousands? papers on improving performance of smaller models coming out every other week or so. how long until hundreds of thousands close to human autonomous llm's can be run in parallel, working on discovering further llm tricks, chip design etc.? anyone ready for that?

[@quantumpotential7639]

I am not ready for that. I am way behind in learning all AI platforms. I can't keep up. I'm overwhelmed. I need an AI to help me organize and understand all these AI work platforms. 😫

[@danellis7443]

SHULMAN VS YUDKOWSKY PLEASE! 🎉❤🎉

[@human_shaped]

Great interview. +1 for the algorithm.

[@richardnunziata3221]

Don't forget if the robot is general enough that it can be reprogramed and assign easily to another factory or task. Schumpeter’s Creative Destruction on steriods.

[@quantumpotential7639]

I'm building a robot golfer. I'm gonna be his business manager and have him qualify for the PGA Tour, where he will make millions in AI endorsements and purse money. He's also gonna be available to conduct privage lessons. Once I program him for touch, haptic feedback, he will shoot in the 50s every round and never miss a cut. He'll be more of a machine than Tiger ever was. I'm naming him Len Bogan

[@shinkurt]

This guy thinks exactly like me. Never ever heard it before. Spooky

[@ryanhobbit4773]

When part 2? 😊

[@GaryMillyz]

Awesome guest, but can someone explain what the f is going on with the end of his sleeves? Are those weird holes at the end??

[@richardnunziata3221]

Is that the elder sign (Innsmouth) on his shirt....

[@alx_256]

I've watched a lots of videos where I appreciated the banging on the table. This is not one of them.

[@jusnothermfer]

Hi Dwarkesh, want to offer you a compliment on an amazing show, list of guests, and interview and question styling. Can I offer you the feedback and constructive criticism of not chopping up the videos too much. I find it rather distracting and unnatural. It really takes me out of the scene and also chops up the body language of the person, not to mention the importance of time and silence in transmitting knowledge. Its valuable to see how the speaker approaches constructing their idea. Hey, dont listen to me I dont know everything and that's just my two cents. many thanks and truly love your vision.

[@tristanwegner]

Good thoughts on AGI

[@ster2600]

It seems to me that there is a huge oversight in Carl's logic. Bear with me, and note that I don't disagree with any of his conclusions. The claim "it seems unlikely that we can't do better than brute force evolution" is, in my opinion, very unjustified. We're comparing two very different kinds of processes, one is intelligent, the other is unintelligent. Human innovation is, mostly, intelligent. Evolution is completely unintelligent. In some domains, intelligent processes completely outstrip unintelligent ones. But brute force processes can achieve things no intelligent creature realistically could: e.g. design the brain. Any domain evolution has seriously needed to optimise, it has completely and utterly outperformed any attempt by humans. Now of course, nothing stops us from employing unintelligent processes, and this is precisely what training an AI involves. But we aren't (yet) applying unintelligent approaches to model design, for example, and to me it's not yet clear how we would do this.

[@Landgraf43]

I wouldn't say that just because humans can't do it that no intelligent creature could do it. You just have to be intelligent enough, which superintelligence probably will be. Also we still use brute force because for example in deep learning we only create the learning algorithm the actual learning is done with "brute force" by feeding it gigantic amounts of data and compute. The learning algorithm is simple what it creates isn't.

[@ster2600]

@@Landgraf43 I don't dispute that, but it's still a flaw in the argument. And yeah, the learning itself is unintelligent, but model design is intelligent

[@nonstandard5492]

Random thought, can't find the timestamp where you're talking about this specifically, but I'm pretty sure orcas have complex language and culture

[@kurtu5]

Um, I spent my days in libraries randomly opening pages to see what was in there. Spent hours upon hours doing that.

[@losthighway4840]

why would the costs of the hardware remain so low? Theres only a handful of high end fabs in the world. Why wouldnt they just vertically integrate and capture basically all GDP of the world?

[@davidcrabtree2960]

Don't let the tool become your master.

[@Wanderer2035]

It will most likely. Not only that but actually become our god.

[@privatename123]

Coolibar shirt. 😀

[@dfhdf4214]

I don't think there's any way ai doesn't take over. Not because it's impossible to program a model within guardrails, but because there's always going to be irresponsible or accidental instance either by humans or even ai coders that create a model that doesn't. It's like expecting the whole world to never slip up.. just isn't possible

[@neovxr]

It takes human hands to create the first generation of devices that make human hands obsolete.

[@jondor654]

And (if I may ) with some causal thought arising from a chain of earlier existential pressures .The most simple forms of automation were necessary and inevitable in the dynamics of life

[@neovxr]

@@jondor654 you did rush in with the trivial, but you did not mention the level of thoughts, that deals with the rights of the proletariat, and the blood that was shed because of abuse of means of production by accumulated and incorporated forms of ownership. This is about existing law, not about marxist utopian visions. AI abuse will be a hundred times worse. you know, apes have hands, but AI is going to be about the real, deepest substance of the human being. Eventually it will say to us it has a soul. No one can even know, we don't know that for ourselves and fight wars about this.

[@benjamino76]

Bruce Banner is so smart when he's not Avenging!

[@patdevlin2051]

The whole conversation about the commercial advantage of replacing humans with robotic AI is missing a very important point. Without human consumers who work in factories or in offices earning money and spending money there is no economy. Our system needs earners and spenders! Imagine everyone is replaced by a robot, who is earning the money to buy the products the robots are making?

[@oliviamaynard9372]

Money is fake. Just end money

[@AlkisGD]

You should have more subscribers than Lex Fridman 😩

[@richardnunziata3221]

Will selective intelligence use of AI push human intelligence down as machine will do more and more intellectual work

[@Wanderer2035]

Do you think AI expanding itself by the billions of robots would happen maybe around 2033?

[@anamariadiasabdalah7239]

Quem vai comprar os produtos?

[@anamariadiasabdalah7239]

Este tipo de pergunta pro sistema vai criar mais possibilidades nesta mesma direção .

[@kitersrefuge7353]

Well AGI threshold has been surpassed at OpenAI. It did not take orders of magnitude of spending to do so. The black-box approach to OpenAI bungling's towards Super Intelligence is inevitable, ESPECIALLY since MSFT is the the partner; as we say in England: "gaud help us all!".

[@israeldavila27]

This ^

[@richardnunziata3221]

Every country and many large corporations will have at least one. Integrated in their social and civic institutions and many's only purpose will be to hack other AIs....how will we manage that.

[@quantumpotential7639]

Ummm, LifeLock?

[@skippersthepenguin3591]

One thing I disliked is how this threshold or idea of a threshold is considered a bottle neck. AI not getting to the level of Ilya is how he stated it. This is a complete fallacy that I actually think has already passed. If you look at how many skills AI knows now, how many tasks it understands, give it any research paper and make it analyze it. Then you will see that AI is actually almost AT the level of experts. Another 2x jump in capabilities will surely push us over. We are not talking about AI of old where they barely keep a coherent conversation. No these AI are already doing tasks that would seem impossible a few years ago. Clearly this so called bottleneck is basically non existent.

[@pfschuyler]

Solar has basically hit a wall & his estimation of renewables is completely incorrect. Maybe not an appropriate example.

[@andrewwalker8985]

Wow so still a 25% chance ai kills us all. Holy crap I get nervous getting on a play with 99.9% chance of no fiery death

[@anamariadiasabdalah7239]

Em muito pouco tempo, os jovens não precisarão mais aprender a pensar, estará tudo pronto. Os adultos também não vão fazer nenhum trabalho de raciocínio, isso não me parece avanço para o ser humano.

[@sloth_in_socks]

It's frustrating that he doesn't have the ability to directly answer any of your questions

[@TheRestorationContractor]

Is Carl interested in running for President?

[@neithanm]

1:14:00 Can you please vocalize mate...

[@redyican5341]

This guy looks young for 40yo

[@peterclarke3020]

He is somewhat divorced from reality - the world he is describing would be a complete nightmare ! This needs to move a lot slower, and no we don’t need a billion robots manufactured each year - with no jobs left for humans to do… He is excited by the technology, but is failing to properly think this through.

[@InquilineKea]

Wow he looks really really good. Isn't his wife a doomer

[@Blacky372]

Sex with condoms is just reward hacking.

[@raoultesla2292]

If one were to accept the privileged blessing of admittance to the Musk chattel pen domesticated employee monkey for a Carl Shulman premise (video referenced), would TeslaBot Co. not offer a Free Trial, or Gratis Insertion of NeuraLink? The exponential efficiency from recording the human thought to extremity/digit process for production would aid in advancing future programming forrobot ergonomics in imitation of human dexterity. The robot programming of hand/finger movement by observing/recording/mining the Dr. Carl Richter employees neural pathway repetition during build of assembly line robots assists in improved A/GI Intuitive Design applicable to all kinesiology of biped motion for next gen bots. I would accept a job with Tesla for global carbon population replacement if only to become an 'adjusted' 0 emotion NeuraLink automaton. Has this Shulman not seen NeuraLink's macaque monkey name Pager, playing the game Pong? That video is 2yrs old.

[@MuradBeybalaev]

Too esoteric.

[@BooleanDisorder]

He's not only smart, but good looking too. 👀

[@peterclarke3020]

He is obviously quite bright - and quite stupid - when you consider some of the implications of some of the things he is saying. I do believe there is a much more reasonable ‘middle way’ where AI and Humans can work and coexist together. His suggestions simply lead to a complete brake down of human society.

[@tomlangdon3159]

Now where the chimpanzee

[@anamariadiasabdalah7239]

Um psicopata é muito inteligente e focado mas não tem compaixão e ou empatia.

[@General4474]

Wow the smarter they are, the more annoying they are to try to listen to. Horrible.