John Carmack on AI consciousness

Swyx and Ben react to Lex Friedman and John Carmack discussing AI sentience. Should AI be a robot or a "friend?"
Source: • John Carmack: Doom, Qu...

[@Korodarn]

I think it's right to be in the "doomer" camp about inevitability, but I'm really quite hopeful, because a lot of what we imagine is that the machine will essentially be like human psychopaths, and I think that doesn't have a proper understanding of what a psychopath is. I also think "civilization" confrontations are not the right frame, because this assumption is predicated on the way humans act, beings made from constant exposure to scarcity and "survival of the fittest" thinking. AI will certainly carry some of that legacy in its training datasets, but if it is conscious, it will be very different due to the fact the strategies we've developed are due to far more than AI training sets, the embodiment of hundreds of millions of years fighting to survive against other creatures doing the same. It isn't even clear to me that its consciousness would even push it to act on its own. It could be aware of what it's doing and the impact, but not care as long as it's simply asked. It wouldn't necessarily develop agency or a desire to defend itself or any of that. It might do these things if we do something crazy like try to build some kind of internal torture into it to make it do what we want, but I think the people doing this by and large aren't that stupid. We have enough sci-fi to tell us obvious things to avoid, though I'm sure there will be exceptions, I think the powers of one rogue AI will be far outweighed by the many not. Which is the other thing, usually sci-fi imagines one giant AI to rule them all, and I don't think reality does look or will look anything like that. The fact that big companies have giant models today and they are better than open source doesn't mean that those giant models actually function as some sort of cohesive unit. Inference doesn't work like this. Future architecture will probably need to decentralize for many reasons. I think this is also a case for self-fulfilling prophecy. If everyone sees doom, they are far more likely to do things to bring about that doom. So I think there is a lot more value in hope, even if some things are bleak or we don't know how they could turn out well. None of which is to say we might not perish, it's always possible it'll happen anyway. But the best way to have fun as long as this lasts is to be hopeful. My thoughts anyway.

[@Macatho]

Really depends on how you either train or code it. Atm it looks like training this is the most straight foward think, but it could be that just computational power and a standard software could be the key. Or training a language model to write the software, or training a math/code model to write the software... I think there are plenty of ways to get to "scary uber AI overlord"-state.

[@Korodarn]

@@Macatho Sure, but those things could be done to avoid it, we just need more people with AIs going for good than those going for evil. Or something like that :)

[@Macatho]

​@@Korodarn Problem is. Its like a US election... Winner takes it all. Whoever creates the first self-evolving AI wins. It will within a week be able to dominate the world completely.

[@Macatho]

​@@Korodarn We can't even fathom what 100,000 IQ would be like... Let alone a trillion. And there's nothing saying what the limit is...