There's No Rule That Says We'll Make It

We're not doomed! But doom is definitely an actual possibility, and we need to act like it.
If you're thinking about working on AI Safety, check out AI Safety Support:
www.aisafetysupport.org/resou...
(Disclosure: I sit on the board of this organisation)
There are lots of jobs on the 80k Job Board:
80000hours.org/job-board/ai-s...

[@CorpsDiplomatique22]

“Extinction is the rule. Survival is the exception.” (Carl Sagan)

[@Mayrink.]

From the makers of "the universe is under no obligation to make sense to you", comes the spectacular: *Humanity's catastrophic problems are under no obligation to be solvable.*

[@existerequo5349]

Or survivable

[@jakebrowning2373]

​@@existerequo5349 that's implied

[@jamesboston]

"It is possible to commit no mistakes and still lose." - Captain Picard

[@theblinkingbrownie4654]

Anybody who played any random chance game should know that

[@garmrdmr]

and the Navy seals teach this... if you cannot accept that doing everything right can still lose.. you have no place as a seal

[@Biped]

I think we are more or less just hardwired to be mildly optimistic and to act as if we can deal with everything. That's because generally that is the best way to act. There is no evolutionary advantage to being doomed and admitting it.

[@demoniack81]

Agree. If you're doomed and recognize it, you die just as much as the guys who were doomed and didn't. But if you're _not_ doomed and (wrongly) recognize it, you might die because you stop caring, and the ones who didn't don't.

[@kurodashinkei]

The sensible middle ground would be "Possibly doomed, and realistically committed to finding the way out"

[@cod.liver.failure]

agree, however, A few hundred/thousand years ago a person or small group of people were not capable of taking an action that would introduce a global (runaway) state of doom. The effects would be local(ish). I think we need to learn to overcome some of our evolutionary instincts, now we've out grown them.

[@0MoTheG]

I think there is an entire group of such "it doesn't matter for what I have to do" heuristics. Many people are violently opposed to genetics and differences between individuals. I suspect it is because in the end it doesn't matter (you have to treat everyone nice and to their needs) and the details are just mental effort.

[@christopherg2347]

@@cod.liver.failure Well, we are still working on overcoming our genetic eating behaviors. Which were optimized for hunter/gatherer, not farmer/rancher food production. And those are seriously failing us right now.

[@NoNameAtAll2]

As futurama has nicely put it "Thanks to denial, I'm immortal!" I feel many people have that fundamentally. I might even want to say "evolutionary"

[@ekki1993]

We all have it from evolution. It's part of the hardware we come with.

[@tristanwegner]

I always wanted to write a shorty story about fighting an imminent fast existential threat, and the story would just randomly end somewhere mid sentence

[@unsane604]

That could be a whole series. One for every existential threat. :)

[@fordprefect859]

Add blank pages to the end of it so people don't know it's going to end.

[@harmless6813]

That reminds me of The Nine Billion Names of God by Arthur C. Clarke

[@oakvalepennybridge]

I love thi

[@AVERYhornyMrDinosaur]

@Tristan Wegner that would be a treasure for aliens to find. a book written and unfinished from the destruction of their world. like Anne Frank's diary.

[@shachna]

"We have never observed humanity being wiped out." Robert Miles 2022

[@jaylewis9876]

Yes this is the problem. Its an extension of how most people don’t change habits even after having a heart attack. Decades of days of not dying provide common sense confidence its “not going to happen to me”, despite seeing it happen to many orders. There are way more extinct animals than existing and all civilizations before ours collapsed and some had total loss

[@iestynne]

I think there's a simpler and more universal reason why people don't "take it seriously". It's because if they did, then... they simply wouldn't know how to act, and to most people that's an unbearable state of being. To truly take it seriously would mean immediately changing your entire life to focus around helping to tackle this issue. However, the psychological structure of a person's daily habits and thought patterns takes a lifetime to build - it is not generally a matter of sudden change, unless the issue is forced by external events (e.g breaking your back in an accident). I don't think most of us are capable of making that kind of change, so shrugging it off in some way and storing these thoughts in a tightly sealed box in the dark recesses of the mind is thus the more common response. I frequently see people doing this when faced with far smaller problems that call for a far smaller scale of change in their lives. I suspect that people only build the capacity for dramatic personal change if they have been forced to do so by prior unfortunate events in their lives. So perhaps people in the developing world, where life is generally less safe and comfortable than in the richer nations, may on average be more capable of taking these kinds of issues seriously enough to act upon them. So in a way... safety has turned out to be the most dangerous thing.

[@fjbz3737]

It’s pretty much what I observed around the start of COVID, and why I became immensely less optimistic toward the future, not because of COVID alone but because of the glimpse it gave me into how we might handle AI’s danger

[@2000YG]

@@fjbz3737 Or any danger

[@pas.]

it's enough to simply say that as an x-risk we need to allocate some time for it in our lives (to persuade others, to start the cooperative decision making process to move global systems toward decreasing this risk, etc) for example, even if we can't blow up an incoming asteroid against the smaller ones it might work to go underground for decades with AI of course the problem is that machine learning seems very useful (self-driving cars, real time any-language spoken translation, medical diagnosis, blablabla), so it's hard to burn it to the ground. yet, naturally, we can set up global systems to try to apply an opposite pressure. (would that work? who knows! yet it certainly starts to decrease the risk) it's enough to accept that it's a risk, accept that it's important, accept that we'll allocate resources toward "solving it", and that we need to accept trade offs. (sure, this is all easier said than done. most people are incapable of doing this with regards to climate change, which compared to a hostile/uncaring superintelligence is 0 risk) people don't take it seriously, because they don't really know what that means. they don't have the context, the lexical knowledge, the trust network (they don't even know that what you are telling them is credible, who to trust to solve this) ... I bet that when the first semi-intelligent AIs start to appear as threats "world leaders" will kickstart the waa waa machines and people will instantly start to care, and will instantly take it seriously. (just like some people instantly took DnD and satanism and rock and roll seriously as someone credible said to them; just as some people started taking AI seriously as Elon Musk said it to them, etc)

[@user-py7qh8bn8s]

hell, whats incredibly funny is im pretty sure someone with no knowledge of Robert miles, ignoring the two movie references, would presume this video would be about climate change. rewatch it with that view in mind, and tell me you dont see it. in general, its just dreadful. because this is a universal philosophical issue and there isnt any clear awnser to it other than to just tell people to start

[@Traumglanz]

The pandemic showed that some people are able to do it while most are utterly overwhelmed from even the little things that changed.

[@KrzysztofDerecki]

It's so calming, you talking about doomsday with those ocean waves and wind blowing palm trees in the background

[@wassollderscheiss33]

I was listening while sitting in the tropics and outside a thunderstorm developed. Took me quite some time that it wasn't coming from the video (which was shot on Scilly Islands?).

[@GlitchyRijndael]

I think you missed a third group of people (that I fall into). Where I am in my life means I have 0 chance of contributing positively or negatively to the field of work tackling these hard existential problems. So while I fully recognize that we can fail at these tasks, I don’t dedicate emotional weight to it because it’s meaningless to me. If we succeed it will have been through no credit or effort of mine. If we fail it will have been through no blame or effort of mine. I cannot sway the result. However I can sway existential threats to me personally like drunk drivers, crime and safety in my community, my own health and well-being, etc. all things that are more likely to kill me long before an asteroid or an AI squishes me or converts me to paperclips. So my emotional energy and bandwidth is better spent on subjects where I can actually affect the outcome, rather than living in a cold sweat about existential problems out of the scope of problems I’m able to tackle. If I’m to die by misaligned AI or asteroids, there’s absolutely nothing I personally can do about that, so I might as well enjoy the time I have left rather than agonize over an inevitability.

[@LevelUpLeo]

Yeah this is pretty much the view I have. I kind of liken it to a being a flight attendant on a plane; I have 0 control on whether the plane stays up in the air or not, but I can make sure others are happy until it all goes down. (I have no idea what being a flight attendant is like)

[@maxwellclarke1862]

I'd be surprised if anyone had literally nothing to contribute. For example just spreading these ideas to people you know is useful.

[@Zeuts85]

This pretty well expresses my own sentiments. The one point I'd add is that for normal folks like us, if we CAN make a positive contribution toward the problem, it will be indirectly--by managing our own lives well enough to be a positive and thoughtful presence in the communities from which the heroes arise. There's a lot I don't agree with Jordan Peterson about, but the whole "clean your room before you try to change the world" pitch rings pretty true to me.

[@Jack_Cats]

The people that would theoretically solve these problems are not born heroes. The people who have the most "direct" impact on that ultimate fate of humanity don't come into those positions of power purely by their own will. Those people are born into a society that raises them, educates them, provides opportunities for them to grow into "the smartest people" or "the most creative problem solvers" etc. Those people are informed about important problems by the rest of their community, people who came before them who maybe weren't as talented but were still knowledgeable to an extent about which things really needed solving. The people who will solve these problems will know that these problems exist to begin with because someone told them. This is something YOU can do. If anyone can solve these problems, the people that will do it will be people who had support from their communities to reach those levels. Right now, this typically consists of people born into more fortunate circumstances, people whose critical formative years are not degraded by poverty, starvation, homelessness. Right now, out of the entire pool of living humans to choose from, a small minority fraction will be reasonably provided these opportunities, and by luck. Imagine how many more of these hypothetical genius problem solvers we might have if it was possible to take every child who would otherwise die of starvation in a third world country and give them unlimited access to education and all of the other benefits of first world society. Imagine how many more inventors and researchers we might have if children whose families could not afford life saving treatment didn't have to consider the price tag of their child's life. Imagine how many more problems we could solve if normal people's day to day life didn't include a minimum wage 80 hour work week demanding the vast majority of their waking hours. It would be a lie to say that YOU could solve all of those problems on your own. But here's the thing: you're *not* on your own. You're a member of the society you live in. You might have the power to vote (making some assumptions here) to promote changes to your community that would be steps in the right direction for the points listed above. You have the power to talk to other members of the society you live in about all of these issues. You *definitely* have the power to spread these ideas and videos and comments around to people who are inexperienced, have never heard of the issue, don't have accurate information, and whatever else. I KNOW you have that power because you have an internet connection. At the very least, you can inform yourself well enough to know how to point any potential "heroes" to the right place to learn more. Your name most likely will not be on the research paper that proves the magic solution to the extinction level threats. Your name could be on the list of donors to the research institute that employed the authors of that paper. Your name *could* be on the list of people who voted in a lawmaker who gave everyone free healthcare, or made higher education free. Your name could be on the recent contacts list of one of those theoretical researchers, or in the back of their mind as the person who gave the initial nudge that eventually led to the solution to the worst problems we face. They say the best time to plant a tree is 20 years ago; AI researchers tend to estimate an AGI is at least 40 years away. How big could a tree you plant today become by then?

[@Dinamicoplus]

A very good remark

[@FunBotan]

Don't worry, we're the protagonist, we have plot armor

[@thomasvleminckx]

that's what they all think until they get hit by a random car in the street and die a senseless death

[@LuisAldamiz]

Until some upstart AI decides otherwise.

[@regalx1]

I'm a year late but I wanted to add that this exactly describes the dilemna that I'm facing. I just discovered a concept called survivor bias, and how people tend to only listen to the few survivors and ignore the majority of people that failed at a task. And most of the times the survivors can't tell you anything mroe than that their series of decisions worked out and others didn't. Basically what I'm trying to say is that humanity has just been lucky so far, but we don't realize it and assume that we know what we are doing solely due to survivor bias.

[@HauntedHarmonics]

Same dude. With the insanely rapid development of AI systems recently, it seems pretty apparent that safety research is going to continue to lag behind until it’s too late, unless the field grows very dramatically, very quickly. But how to get people to take the risks more seriously? They’re always shut down with claims of “being alarmist”, but we forget that sometimes people that ring the alarm are right lol.

[@dmwalker24]

This is very similar to the point that I raised. Many people are almost incapable of imagining that we won't 'figure something out', because that's what they've seen over and over. But with resource exhaustion, climate change, and now AI, we're facing three threats that all have the potential to wreck civilization, if not end humanity outright.

[@dmwalker24]

@@HauntedHarmonics There's at least some chance that perhaps the climate, and disruptions to the global economic system might derail AI before they make a mistake they can't take back.

[@nickcni]

Cool video . Here’s something you might want to consider: healthy individuals pretty much always overestimate their control over a given situation in most experiments when compared to depressed test subjects. In other words, slightly overestimating our ability to control our environment and overcome any challenge is considered a healthy trait in clinical psychology.

[@dr-maybe]

I find myself coming back to this video every once in a while. The emotional and psychological aspects of dealing with existential risk from AI are perhaps even more important right now than the technical arguments. It is just so difficult to make people emotionally understand how dangerous our current situation is.

[@slash196]

I'd add to this: there's no rule that says just because WE make it, that YOU make it too.

[@2000YG]

Everybody is the survivor in the zombie movie. Nobody is one of the dead walking and hunting for brains.

[@danielkokotajlo6096]

Are you imagining scenarios where most of humanity is wiped out or otherwise disempowered/imprisoned, but not all? I find those scenarios pretty unlikely compared to scenarios where >95% survive, which in turn seem pretty unlikely compared to scenarios where approximately no one does.

[@nihatakdamar7711]

This reminds me of a book I read a long time ago. A neurosurgeon says something like, You are not a real surgeon untill you pop your first aneurysm. Only after that, when the dread of making that same mistake again looms over you at every operation you do, are you a true surgeon. Only in this case, we don't have the luxury to live through such a mistake.

[@lioelbammalf7483]

Reminds me of a Terry Pratchett quote: “The thing is, I mean, there’s times when you look at the universe and you think, “What about me?” and you can just hear the universe replying, “Well, what *about* you?”". Its nice to think you're important in some way but the universe doesn't care. Things just happen, a sense of fairness only exists in our minds.

[@underrated1524]

Personally, I think AI safety is as fitting a "dramatic final test" for humanity as anything. It's a challenge that sums up humanity's defining traits: engineering, and arguing over values. Maybe that's just me though.

[@ClaySchubiner]

Good way to put it

[@Flackon]

Trust me, engineering is not one of humanity’s defining traits (source: I’ve talked to many non-engineers)

[@ekki1993]

That's kind of the problem, though. It's not a dramatic final test, but that's a very human way to conceptualise a very complex problem we might have in the future.

[@DrunkGeko]

Well the point is that perhaps there is just no practical solution to AI safety There is no reason why we should believe that we will eventually solve it, even if given enough time. Maybe it's just beyond our capabilities and we just don't get to have a fair attempt at survival

[@GAPIntoTheGame]

@@FlackonAs a species, it absolutely is. The human race would be very different without it.

[@Lufernaal]

One small scale example of that is when Alphago faced Lee sedol. Pretty much every serious go expert said Alphago had no chance. It'd do well, maybe, but not beat the best go player in the world. Lee is an incredibly strong player, one in a billion type of deal. Yet, when the game actually happened, Alphago played around with Lee and took him down a dark dark path. Lee managed to win one game simply because of the way the people who coded Alphago were not able to fix a very specific problem the algorithm had. Not only that but the version of the AI that learned to play by itself was much stronger than the version that learned from human games. Basically, having humans involved in it made it weaker. Once full AI is implemented in some significant capacity in our world, there's no way we'll be to compete with it. It'll be far beyond anything we can even comprehend, let alone anticipate and stop if necessary. The very tiny version of AI we have built so far have made us look stupid by comparison. I play chess very seriously and when I watched Alphazero play, I didn't even understand what was happening. Stockfish is just brute force all the way and it does things that seem fine, not really crazy, but then you realize that they're very strong later on. It's all about solid, somewhat predictable play. Its biggest advantage was solely the fact that we can't calculate that deep. Alphazero, on the other hand, plays like a maniac, it goes for these crazy sacrifices and it almost always focuses on neutralizing the opponent first by playing in a way that looks very risky, but that eventually gets into a position where its adversary can't really move well. It's a positional God to a degree that no human can challenge. Stockfish was programmed by humans and it was limited by what we thought made sense to characterize in the code. Alphazero learned chess in its essence and it shows that our conservative, appearance over substance style is flawed and that God level play looks insane because it is simply something we can't do, no matter how smart we think are. Even Stockfish uses deep learning now because of how strong it is, programmers now know that there's no point in trying to simply code the best chess engine. AI won.

[@Ceelvain]

The dichotomy is wrong. There is no old-style AI vs. new-style AI. No brutforce/heuristics/hand crafted algorithms vs statistical methods. No alpha-beta vs. neural networks. AlphaGo wouldn't be any good without its hand-optimized state-space exploration or its monte-carlo tree search. I'm pretty sure Stockfish has used for a long time some heuristics based on statistical data obtained from actual games. On one hand, you can write an AI from beginning to end the old fashion way. On the other hand, you can't just take a neural network, give it a chess board and say "master it". There is currently always a program around it that implements the rules. An architecture that arrange networks and subnetworks together to be a good fit for the problem you're trying to solve. There is always some form of inductive bias hard-coded by humans. The future of AI is not neural networks all the way. That's what we thought 15 years ago when they started getting traction. NN struggle *a lot* at extrapolation and at everything about logic or symbolic reasoning. That's why successful AIs always have some hand-crafted algorithms to carry out this part of the task, while the neural networks carry out the "intuition", the indescribable part. What we don't know yet, is how to mix those two approaches in a generic way. But it's, for sure, the future of AI.

[@Colopty]

Though funnily enough Stockfish was never really a representation of the best knowledge we had on the game, it just contained whatever heuristics we were able to define well enough to write into a computer program, such as piece values, while not containing heuristics we had an intuitive understanding of but couldn't really define mathematically, such as positional advantages. As such, high level chess players all noted that AlphaZero's understanding of positional advantages actually made it play more like a human than Stockfish, so if anything it's a better representation of human ingenuity than the hand coded version. The part about easily definable heuristics would be why it took us so long to make a good go AI too. That game has exactly 0 easily definable heuristics, as opposed to chess which could basically be simplified to "the pieces are worth this many points, check which move results in the ideal point difference in 10-15 moves, prune the search tree by checking out branches that involve a capture first", and you basically have a superhuman AI already. Meanwhile with go it takes like 20 moves before anyone even has any solid points you can tell the computer about (note that go has a branching factor of 250 as opposed to chess with a branching factor of 31-35, and that unlike chess there's no good heuristic for pruning branches so getting the same search depth is much harder), and getting those points might've involved giving your opponent ten times as many points 100 moves down the road.

[@HauntedHarmonics]

Born too late to explore the earth. Born too early to explore the galaxy. Born just in time to *be forced to try to save all of humanity*

[@Garbaz]

I get your point, but I'm not sure about the reasoning. At least for me, I think there are two parts to the reason it's difficult to take a threat seriously: 1) Reacting appropriately is (or feels like it is) beyond one's capability. So in trying to avoid the dread of that fact, we simply don't engage with it. It is very hard to accept the existence of a threat, if we have no way to react to it. The whole point of feeling threatened, after all, is to push us to act. 2) "The people in charge" always find a way. Sure, things might go up and down, but in the end, if things get really serious, a solution will be found. In a way, this is the adult version of the assumption children make, that the adults in their life have a plan. There always are people who know more or have more control. Though I do think there definitely is something to your point that we tend to reason about reality based on fiction, even if we are on a conscious level aware that it's entirely made up. Especially when it comes to topics we don't have much non-fictional experience with.

[@4rthurzz]

I love the philosophical part of your content, it expresses my "intellectual dread" with precision. If you feel this way in Britain you can only imagine how it'd be like in a country like Brazil, in which I live. Really excellent text! I wish you made more content like that.

[@rafabulsing]

As a fellow Brazilian watcher, I agree with Arthur! Love this kind of content. And yeah, I too think most people don't *really* get it, even when they say they do. Hell, I don't think *I* get it all the time. Most of the time it's a more intellectual, abstract thing. There are moments where I really feel it in my bones though, the incredible amounts of luck that got us here, and how easily it could all end in a way we could not feasibly or even theoretically avoid. And it's *terrifying*, when you catch a glimpse of that truth and really internalize it. So there's a bit of, I guess, self protection from the brain, to avoid really thinking about that on that deep level for longs (or, for most people, any) stretch of time.

[@tonykt]

as a Brazilian living abroad, having grown up in Brazil and now living in an extremely (physically, if anything), safe society as Korea is, for a couple of years now, I do think a lot about how scared of violence we tend to be back home. It's a fear we've all but sublimated, but once you take a step back and look at it from afar it becomes surprisingly salient. As predictable as it might sound as time went by here I realized a fair bit of my intellectual dread wasn't even "intellectual" - it was wholly footed in reality, that is, in my reality as a middle class Brazilian. One way or another you will be quick to recognize how comparatively easy it is to be shot any other day on your way back from home or work and die with no apparent reason in urban Brazil lol. We have an entire vernacular for that (I personally find the term "bala perdida" fascinating). But naturally it's a less romantic conclusion so I'm sure many of us deliberately shy away from it. Oh well this might as well not be representative of the wider population. But your somewhat defeatist tone, if you will, makes me think otherwise

[@elityre2710]

I think "an asteroid strike, 300 years ago" is a really strong example of the way the problem can just be beyond our abilities, and expect to use it in future.

[@buzz092]

I loved this video until it implied that I should have to do something about it 😰

[@wolframstahl1263]

Time to get your band of misfits together, space man!

[@amaarquadri]

I think that's the entire point of it.

[@axelperezmachado3500]

well, we could start by sharing it so people closer to make a significant difference can be influenced as well

[@jag4rok]

@@wolframstahl1263 let's be bad guys!

[@HauntedHarmonics]

@@axelperezmachado3500 I think that’s the point. There is no “somebody closer”. This is an insanely difficult impending problem that almost no one is working to solve. Either we step up to solve it, or we leave it up to fate. Born too late to explore the earth, born too early to explore the universe. Born just in time to be *forced to try & save all of humanity*

[@RobertMiles2]

If your first thought after watching this is to talk about a science fiction film, consider the possibility that you have not quite grasped my point. (I haven't seen Don't Look Up btw)

[@RolfSvenning]

You should, it's quite enjoyable :) Also I think a lot of people watching this video have that film in the back of their mind, I had at least! As always, thanks for an interesting video :-)

[@lindalinsefors6578]

Why not. One way to overcome a fictional induced bias that we will survive, is to consume fiction that ends with everyone dying. (I think that is why people are mentioning Dont Look Up.)

[@RodelIturalde]

Which is your favourite Science fiction movie, and why is it Terminator 2? :P Anyway. We can see same mentality when it comes to climate change. People say 'science/politicians etc will fix It.'. We can see it everywhere, and it is growing. People's ineptitude to take responsibility for their own action and often bad ways of thinking about risk. For quite some time now, the world have been seen as a machine, and a machine is predictable. The more we measure, the more we can predict and thus we don't get any surprises. We don't meet any risks. At some schools, kids are not allowed to climb in trees, because the risk is to high is one such example. We are slowly removing everything that could create a risk. And when this is done, the person have no fucking idea on how to behave or react when the unexpected happen. He can no longer take responsibility, because he have never needed to take responsibility. Because there had never been any risks involved.

[@quangho8120]

@@lindalinsefors6578 The people on that ship does survive though, so it's not a "all hope is lost" scenario

[@bryanburgess3950]

All the women on the ship are post menopausal

[@bramgeron7267]

We went sailing once, and the captain instructed us on safety protocols - wear your lifejacket, when we're doing this maneuver then a rope might swing, make sure it doesn't hit you or knock you off the ship. The captain previously hosted a religious group. After the safety instruction, their leader said: "Just ignore the captain, God is taking care of us."

[@pas.]

so in that case the captain didn't even unmoor the boat, right?

[@bramgeron7267]

@@pas. They were already on water 😁 And presumably the group had already paid…

[@Bootleg_Jones]

I would have told that religious leader that they were expressing a very slothful attitude towards taking care of themself

[@somedragontoslay2579]

Bootleg Jones When facing these kind of people, you need to speak in parables. This one would do the trick: "One day a good Christian faced a flood in his city. His home was underwater and he went to the roof. A couple on a canoe came and said him they could take him to safety. He answered 'No, thank you. God will take care of me' and he went on to pray. Then, lifeguards came on a boat and they asked to help him, he declined and went to pray. Then, a helicopter came and he did the same. When the flood grew, he drowned and went with God. He asked Him: 'Why didn't you come and save me?' 'I did! I sent the canoe, the lifeguards and the helicopter, and you decided to ignore them. So here you are!'"

[@beskamir5977]

Then in heaven: "God why didn't you take care of us!?" "Well... I made sure you had a very knowledgeable captain that gave you some excellent safety advice..."

[@fjbz3737]

3:31 I think an even better example to consider than asteroid is the implausible event of a supernova, which even our space-faring technology wouldn’t allow us to escape

[@thomasvleminckx]

"Even our"? Our spacefaring tech is a complete joke, we aren't anywhere near able to reach the nearest star. To say it's inadequate is an understatement of interstellar proportions. Luckily, the Sun is not a star capable of going supernova.

[@LuisAldamiz]

I believe "we" (i.e. astronomers) are reasonably sure that no such dangerous event will happen in our vicinity any time soon, lucky us!

[@robinhood5627]

I hear it ALL the time: "Humans are super adaptable and will easily survive XYZ" Meanwhile in realityland....Nope.

[@mjstow]

"They know it, but don't *feel* it." Excellently put.

[@bob3ironfist]

Living in the United States I feel this everyday. The overwhelming sense i get from people around me is that everything will just work itself out. Which it won't. Why should it. So when confronted with existential threats like climate change, ai, or nuclear war, people just don't take the possibility of a proper game over seriously. Or even with the fate if democracy in the United States. People don't take the threat of it ending seriously. They just can't bring themselves to actually consider it a possibility. This isn't just me lamenting that I don't think that my country will exist in 10 years, I've made my peace with that, (sucks for the people of the United States, but hardly a species wide threat), it's just that the way that it happens is really important. Whatever happens from the breaking up of the US there will be multiple powers where the old country used to be armed with nuclear weapons. All it takes is a failure of communication, or an itchy trigger finger, or a zealot in charge to start throwing nukes around. And that is most certainly a species wide threat.

[@the1exnay]

In the US i feel like the biggest problem is people have just come to believe that nothing can be done. And i think the reason for that is that they focus on federal politics. Like if you point out that part of the problem is how federal congressmen are elected and then ask them what it would take to fix that. A lot of people will say that it would take the federal congressmen choosing to upset the very systems that keep them in power. And that feels impossible, how would we convince them to give up their own power when the problem is how little control we currently have over them. But that's wrong, the states have the power to decide how the congressmen that represent them are elected. The states have the power to set rules that govern their representatives. Though, ofc, changing things at the state level isn't a trivial task. But there has already been some success. We can use the states to improve our democracy. And we can use that improved democracy to improve our nation. But... It's nice to feel helpless. If there's nothing you can do, then you don't have to do anything.

[@bob3ironfist]

​@@the1exnay The hard part is getting people to care less about federal politics and more about local. And for that big media needs to be refocused. Social media has to feed people less national and more local news and politics, local media (newspapers, local news stations, etc.) need to be rehabilitated/recreated since so many are either gone completely or under new ownership that only cares about profits (and national news sells the best currently), and generally get the 24/7 news outlets to be somehow less influential. Problem is that we don't have much of a voice in who buys out whatever local paper/station. And what control that we do have over how social media helps drive political conversations in the nation is filtered through the federal government, which isn't well equipped for the current challenge of what to do about it and how to do it. There's just too many things going wrong at once and it's not looking good from my perspective. I'm confident that given enough time we could solve these problems. But I don't know that we have time.

[@tolep]

Serious decisions are made anyway without asking the people's opinion. No one is even pretending anymore. No one is trying to maintain the façade. The very idea of democracy seems strange, artificial, unnatural to me. Democracy is a short episode in human history, an experiment that is about to end. By the way, the entire nuclear arsenal amassed at the height of the Cold War would not threaten the existence of the human species. Not even close.

[@TotalBedroom]

The thing is I’m not sure what ‘making it’ means either. What would humanity ‘making it’ look like? Just surviving into perpetuity? All my ideas for that more or less come from science fiction and seem extremely improbable if you actually stop and think about them for any length of time. Also, people have a hard time coming to terms with something as concrete as their own death, never mind dealing with something as abstract as the survival of the species. Thanks for making the video, it’s refreshing.

[@phoenixpinkmyn5535]

It doesn't really make sense. Humanity cannot "make it". There must be an end eventually. His proposition is faulty. Either we'll go extinct completely, evolve into a new form, heat death of the universe, etc.

[@41-Haiku]

I'd be satisfied that we "made it" if humans still exist 50 years from now.

[@aDifferentJT]

This is a good point not just for AI safety but also for climate change. We have a case of survivorship bias for the human race.

[@LuisAldamiz]

A much more urgent threat, basically unfolding as we speak, about which nearly nothing is done at all. There are powerful short-term reasons, notably the power of the oil-backed mighty US dollar, but still it's amazing that nobody in power has the statesmanship to actually do something at all.

[@wassollderscheiss33]

No, absolutely no. Climate change is no runaway problem. Maximally it will become a little inconvenient, that's all. The world will not turn into lava. Just take my word for it: I live on the equator and absolutely no one here is concerned about the environment becoming uninhabitable. That is a story told in the West, that doesn't coincide with reality.

[@wassollderscheiss33]

@@LuisAldamiz Actually, hyping climate change as an existential risk is diverting resources away from areas that are really dangerous. The AI problem is a gazilliontimes more risky than Climate Change will ever be. Climate Change is for stupid people.

[@Happypast]

On fiction: man, Game of thrones really dropped the ball. They had an existential threat looming on the horizon and all the sides of the conflict where fighting among themselves. An apocalyptic ending would have been such a rare and powerful message.

[@JonWeinand]

I want to say that I very much appreciate your videos. In this one I see a lot of frustration. It seems like you've been running up against a lot of people who don't understand. Most people never will. The whys and hows seem to be pretty well-covered by other commenters, so if you read this, I hope that you don't feel like you have to reach everyone. As you say, there's no rule that says the challenge of getting people to understand can be overcome.

[@NateJGardner]

I love the lines you added at the end. I often think about that. When thinking about how to live my life, I constantly consider that idea- it's up to me. I feel grateful there are others who think that way too. I just hope I select the right problems to work on.

[@nickm3694]

Gosh, the fiction-as-some-sort-of-evidence thing just really grinds my gears. Every time I hear someone pointing at some sort of AI or technological improvement and making some sort of comment like "clearly they've never seen " (I make an example out of AI, but I've seen this happen at least a few other times in other ways that I can't recall off the top of my head). It's just so mind boggling that people would use these movies as the best guess for what would happen if humans tried some particular thing. However you do bring up a good point about the flip side of that where people just assume we'll always win in the end because of those very same movies. That's something I'd not quite considered, but it makes sense. But there is one other point that I'm not sure has been considered: that we don't consider challenges we can't face because then there'd be no point. If there's a challenge, we might as well assume we can succeed, lest we demoralize ourselves past the point where, if success were possible, it no longer is. But on the other side of the equation is how much effort is required to overcome these challenges. For example, people may assume "ah, as long as we got the top scientists working on in we'll be fine", when in reality we would need hundreds or thousands more people getting involved in the field and actively working to fix the problem in order for it to be solved, thus requiring action from a substantial selection of the millions to billions of people that just assume that as long other people are working on it, we'll be fine.

[@oldvlognewtricks]

Doubly so when religion and fiction (tautology alarm) are constructed to satisfy human cognitive prejudices. The way humans think causes us to both assume we will survive, and to create fiction in which we will survive. Using the product or our thinking to justify our thinking is some miscausality soup if ever I saw it.

[@gasdive]

@@oldvlognewtricks spoiler alert for Don't Look Up. Really, read no further if you think you might one day see the movie. . . . . . . . . . . . . . I had a conversation the other day with someone who thought, even after the asteroid hit, that they'd figure out how to save everyone. They were completely shocked when the house started to disintegrate and all the viewpoint characters died.

[@oldvlognewtricks]

@@gasdive I guess if you didn’t watch Cabin in the Woods or any movies from the 70s it might be surprising that fiction sometimes works this way. Sometimes the bad guys win, y’all! The only other ending I would have accepted for Don’t Look Up would have been if *everyone* fucked up - the rich guys tank the economy and cause global unrest and the scientists were out by point zero zero zero two percent and it missed. To highlight the *utter* fallibility of the human condition… And then the sun suddenly goes nova and destroys everyone anyway.

[@josiah42]

I'm a genetic engineer and if I could go back in time I would kill Michael Crichton before he wrote Jurassic Park. Those scare tactics are responsible for tens of millions of cancer deaths by now.

[@taragnor]

I feel most of the AI fears are also based off science fiction though too. Thus far AI general intelligence just doesn't exist. Even relatively basic tasks that even your average stupid human can do, like driving a car, are still difficult for AI. Most of the times when you hear fears of AI, it's basically some kind of sci-fi AI superintelligence. We're so far away from Skynet at this point it's not really worrying about. When we have a general intelligence that's on par with a dumb human then we may have cause to start worrying. My view comes from the opposite direction when it comes to AI. There's no rule that says AI superintelligence will ever happen.

[@vanderkarl3927]

This reminds me of the parts of your past videos where you mention the absurd possibility that the human mind could be literal magic and therefore irreproducible, unimitable; while that is almost certainly not the case, there are many people who will vehemently and confidently disagree, no matter one's appeals to logic and physics. I think you put your finger on the reason for this in this video, the idea that humans are special. It's almost impossible to argue otherwise, given that every opponent you might face is inherently special, in the "special relativity" sense. It takes a fair bit of reason and perhaps a little bit of faith to voluntarily make the leap from one's own special perspective to a more general perspective. Excellent video as always. Looking forward to the next!

[@christopherg2347]

The ironic part is, that means they are comitting a deadly sin. At least one definition of pride is "the quality of having an excessively high opinion of oneself or one's own importance". The arrogance of people to think they would be special to a god. That in all the galaxies and galaxy group, our planet must be special. That god has to sent them a sign if they are wrong.

[@marin.aldimirov]

This is crazy, because I've been having this exact same thought for a couple of months now. Your title - word for word.

[@zanec14]

Another reason why most people can't understand this is because it is a colossal amount of psychological responsibility to constantly carry around.

[@Spiderboydk]

I think the relative stability and prosperity of the last many decades is a big reason we tend to feel invincible.

[@DamianReloaded]

On one hand is fear of death and on the other pragmatic cynicism: "What can **I** personally do about it? If I don't pay my 3 mortages next month I might just as well consider myself dead anyway"

[@taragnor]

@@DamianReloaded Yeah worry about the things you can change and don't concern yourself with the things you can't. Quite simply the average person isn't in a great position to influence global policy.

[@DamianReloaded]

@@taragnor In an ideal world average Joe would be well informed and would govern policy democratically through voting. In our world everything is a bit more difficult since we gotta wade through the sociopaths that are in charge of everything.

[@taragnor]

@@DamianReloaded Yeah. I mean at least in the United States, there's a lot of reasons why your vote doesn't really mean much: A) if your two candidates basically suck and neither addresses the issue in question. B) Because of the electoral college and the winner-take-all state mentality, if your state isn't a swing state, your vote is useless. C) Your candidate gets in, but takes bribes and backstabs you later.

[@DamianReloaded]

@@taragnor It still works tho. Mayors often get elected more finely according to the mentality (or state of mind) of the citizenry. In any case, it's still better than having a Pooh or a Putin permanently there telling you what to say or think. Could be better.

[@anonymous.youtuber]

“As far as we know, we never had an undetected error”

[@Noobinski]

Please do more. Your input is needed.

[@michaelhunter5510]

I don’t see the usefulness of thinking we might be fundamentally incapable of dealing with an existential threat. I can see the usefulness in thinking at we may need to throw everything we’ve got at a problem in order to succeed, but not the usefulness in thinking that we’re fundamentally incapable of dealing with a problem. It doesn’t improve our lives, it doesn’t make us happier, and it doesn’t make us capable of doing what we’re incapable of doing (by definition). Actually i suppose it might be useful to the extent you get enjoyment about thinking about it, or it makes you appreciate your current existence more.

[@TheMan83554]

My thoughts exactly, it's unactionable, we can't engage with it beyond acknowledging it. What more is there to *get*?

[@teapotexorcist]

Great video. Another possibility is that people subconsciously have the continuation of their existence rooted in part of their identities. So when they hear something like "there are problems we might not be able to survive" it conflicts with their sense of them having existed, and all their ancestors having existed despite all odds etc. So they see it as more unbeatable odds like have seen before, without recognizing the fact that we don't know what we don't know, and even what we do know shows some very real danger in some areas.

[@dhinas9444]

One of the best videos so far!

[@mathmagician5990]

Thought provoking stuff. Love your videos.

[@andreachiarello9001]

I was talking the other day with a colleague about the future of our home country. I was saying that economically and socially we are facing hard challenges and I'm worried, and thinking that as a country we are going to collapse. Her answer: we will overcome that, we always did, we will find a way. I did not answer, but I was thinking: there is no rule or law saying it will be ok in the end... Another scale, but same principle.

[@musicalBurr]

Great talk. Thanks!!

[@d007ization]

I think this also runs into our insistence that we are more free than other objects. And while technically, we are, we are those objects and as such have better control over ourselves than other objects.... the instinct is "since we are the ones who would set world domination by AGI into motion, we can always just choose not to switch them on." This presupposes that we are a monolith but that's another thing we (hah) don't often think about.

[@Iextrimator]

That’s a very thought provoking video, good job!

[@wolframstahl1263]

Let's hope it's an action provoking video as well. ;)

[@silkwesir1444]

@@wolframstahl1263 thought also is a kind of action. an action that is even usually required before you can do any 'actual' action.

[@Macieks300]

I think that way of thinking does not come from religion or even more definitely not from fiction. I think it's just part of human nature. We are equipped with survivorship bias and are meant to think only in terms of us and the people living immediately next to us, not in terms of the greater picture and humanity. Also, no living person ever experienced what is like to die and even though everyone knows they're going to die people ignore that fact in their day to day life. On top of that every living person's experience of life is that they always overcome all of their problems that life throws at them because if they didn't then they wouldn't be alive and wouldn't be able to think at all.

[@AxMi-24]

Problem is that this approach worked well on the savannah. As we advance, problems are becoming a lot more complex and threatening to humanity as a species. In the case of developing GAI, at least we can say that we created something more advanced before going extinct (it will in a way be our child), but it is a lot harder to be equally at peace with nuclear war (more likely every day) induced extinction, because then the only thing we achieved was to kill ourselves. Not exactly something to be overly proud of. We need to grow up as a species, away from egoism and glorifying psychopaths, and towards cooperation and trying to improve lives of everyone rather than the select few.

[@Macieks300]

​@@AxMi-24 "We need to grow up as a species [...]" Not really possible. What you're asking for is for human nature to change but the only thing that will change human nature is natural or artificial selection based on evolution. And evolution doesn't work on scales of hundreds of years but on hundreds of thousands of years.

[@LuisAldamiz]

@@Macieks300 - We can change our societies for the better. In fact it's the only thing we can do.

[@cpb42]

I'm so glad I subscribe to you, that was an excellent thought, and I am glad you shared. I believe we have maybe 10 years left, before consequences of climate change make this planet uninhabitable for the vast majority of us. I think the only solution is that, within the next 3 years, we wake up and realize we are all one, and we have to work together to do the research, science and development necessary to tackle extinction-level issues. We need to do away with laws and regulations that inhibit the sharing of information and applications of technologies; we have to get rid of intellectual property.

[@DavidSartor0]

Why do you believe these things?

[@cpb42]

@@DavidSartor0 given all of the things contributing to accelerated climate change, how do you believe we have more than 10 years? Ocean acidifcation and temperature increase is going to have a pretty dramatic effect on many things, include an acceleration of releases of greenhouse gasses that are trapped under the sea beds and under currently defrosting permafrost. I doubt if science were capable of proving catastrophic climate change in 10 years, we would hear about it. The governments don't like to panic the people. I suspect we will see famines of large scale within 5 years, unless we put our differences aside and work togther to solve these problems.

[@chawaphiri1196]

i haven't watched the video yet but i am here just to say that please start uploading more content about ai safety now that it has blown up. I love your channel and everything i learn from it

[@stumby1073]

This is a great video but WE ARE HUNGRY for more of your usual content. 🥺

[@paulthelibrarian7794]

Consistently. Great. Content.

[@biocode4478]

I mean none of us ever "makes it". Say we brought a disaster upon our entire civilization and we all vanished. A true tragedy, everybody died. What is the alternative? We prevent it we continue to just die till the heat death of the universe? The disaster kills less people than our continued existence. We can't even promise a healthy fulfilling life as we keep creating more and even best of those lives will end. Like I do believe in striving to make the world a better place to make life as good as possible for everyone around but honestly that's only necessary because people won't stop making more people no matter how bad it gets until it's simply impossible

[@fjbz3737]

The way in which fiction bears relevance to our reality is in it’s power to portray the human spirit, (usually in a good way that ultimately works out, because that’s what we want for ourselves in real life as well), but the issue is that being guided by correct principles in the abstract is only one part of the puzzle in real life- you have to actually take steps to realize these principles as well, and it just so happens that in fiction those are neatly presented as always being perfectly within reach.

[@crowlsyong]

Just discovered the second channel. Really liking it.

[@yesterday1396]

The amount of existential threats we are facing is hard to deal with. All it takes is one and that's it goodbye human race. How I try to cope is focus on living in the present and making the most of each and every day and breath it all in.

[@majkelmajkel5119]

A great video. Thank you! My reflection on it: Let’s assume it’s 200 years ago, and you, a clever astronomer, inform us about the incoming asteroid. How would this message change our lives? I presume you would get exactly the same reactions as you are getting now. It would be a threat beyond common comprehension - thus ignored. So maybe instead of expecting us to build a rocket/space missile, we should focus on creating the dream of flying?

[@Dante3085]

Just giving my thoughts to the first minute of the video. As in many other cases I think this is trying to work against some adaptation of the human mind to keep in the "It's alrighty state". Just think about it, the actual implications of what the AI-Safety community is saying are incredibly horrible. This goes for something like climate change as well (Maybe less, because its not literal extinction and uncertainty about everything). When I realized that AGI might be possible and that the AI-Safety community maybe has a valid point that was kinda terryfing and I am not shure I am better off now. Edit after 6:25 I am now more convinced that the answer to his problem is that humans tend to some sort of equilibrium mental state. It's hard to go on otherwise. Being emotionally hit by "AGI might create our worst nightmares if we dont do something about it." daily is not part of that. Especially because compared to the mainstream this is a quite esoteric idea. Edit after 9:05 I think I agree with the point that consuming fiction has an effect on evaluating real scenarios, even though we don't want it that way. This might be an unintended side-effect of mass entertainment XD

[@Vlad-jg2ku]

I love watching your videos not just because they are applicable to AI, but because as I see it, a lot of these issues we want to overcome with AI also have analogs in politics and social relations. There are a lot of instances throughout history, for an example, of exactly what you’re talking about here. People having a general sense that they will make it, and ending up being wrong. I’ve been reading about the history of Russia and Ukraine, and the recent history has been a chain of unfortunate turns. In the early 20th century for example, there were many within the elite of the Russian empire that were ok with undermining the Tsar and shaking the house a bit, because they thought that they were on the right side of history and would come out on top. Ultimately, Vladimir Lenin and his Bolsheviks were more clever and better organized and were able to take advantage of the instability to seize power, despite having less popular support than multiple other factions. (The Bolsheviks weren’t even in the picture in any significant way when some of the factions started undermining the Tsar) The communists, when they took power, then thought that they and their system were inevitable. (Of course the west also thought the same about capitalism and democracy) The communists turned out to be wrong and things did not turn out as they expected. All of the “sacrifices” they made to pursue their goal ended up just being costs with no payoff. Then once the Soviet Union collapsed, there was again a disappointment. The people of Ukraine and Russia thought that by opening up and becoming part of the global economy they would start improving at the rate that the west was. Instead, people found themselves jobless and in bread lines in the 90s, as the former government industries were raided and there were no internationally competitive private industries yet in place to take on excess labor. The inevitable prosperity people expected; it did not happen. In the early 20th century the Russian empire was large, powerful, beginning to industrialize, and on a general upward tragectory that had lasted a few centuries. If anyone tried to predict what would happen to it, they would have been wrong. Nobody predicted the Bolsheviks taking over, and nobody predicted the USSR collapsing. It seems that when things are going well, people have this general sense of “this is how it should be”. When things are going poorly and some shakeup happens, people tend to think “this is the turn to the upswing we’ve been waiting for”. Overall, it seems that people tend to be optimists. Personally, I don’t think that’s a bad thing, and it’s probably a healthy thing for the mind. It is good, however, to occasionally temper that optimism with a reminder such as this, that there is no rule that says we’ll make it. There is a quote/motto that I recently came across that I feel is apt. “Pessimism of the intellect, optimism of the will”

[@LuisAldamiz]

But communism is inevitable if we're going to survive, it's clearly capitalism what is failing at everything, every day more and more clearly so. The bloshevik regime was never "communist", just read what Engels has to say about the Blanquists, precursors of Leninism, extremely prophetic and nonchalant. They were a hybrid neo-Platonist (rule of the philosophers) implementation of "communist" or socialist ideas in underdeveloped semi-colonial nations, which served a developist (national-capitalist) purpose but were not at all "communist" and, as the wisest of them all (Trotsky) eventually acknowledged, they could only end into a restoration of capitalism, as it did happen. We can't face the ecocatastrophe with capitalist tools, because capitalism is all about worship of a god called Invisble Hand and that magic hand happens to be not just invisible but pretty much non-existant. We need some sort of real communism, i.e. total democracy, also over the economy (especially over the economy) if we are to fix our planetary problems running out of control.

[@dylanrichardson199]

@@LuisAldamiz it can also be said that we don't have real capitalism, let alone real democracy. Almost to the same extent that the soviets didn't have real communism. So let's not leap to conclusions about which ideal is superior.

[@LuisAldamiz]

@@dylanrichardson199 - That's what the far right likes to say. However the word "capitalism" was coined by 19th century socilists (Louis Blanc first of all) and means what it means: a social class system dominated by "capitalists", in turn seemingly a Dutch term for those who live off assets (capital) and not by means of work. Democracy on the other hand has a very clear meaning of "rule of the people" and its historical reference is the Athenian Demokratía (as well as others, from Argos to Samos, etc.) You may of course argue that not all people were allowed to partake in the Demokratía of Athens, but all the legal "demos" was, i.e. every male citizen (or just every citizen, because women were not citizens strictly speaking), regardless of wealth or education. The Athenian proletariat was proud of their Demokratía and gladly contributed to the military might of Athens as rowers in the triremes, giving the city a major advantage in naval battles (slaves don't row nearly as well as free motivated citizens). People who willingly disdained political affairs were called "idiotoi" (from which our "idiot").

[@dylanrichardson199]

@@LuisAldamiz I'm not of course referring to capitalism in any obscure perjorative sense, I mean free markets and free enterprise. Tariffs and anti-competitive subsides are pervasive and exact quite a cost. I suppose I'll take the history lesson as agreement with the democracy part? We fall significantly short of our own ideals. Assuming we are talking about the US, even something as simple as reforming the electoral college would be a massive benefit, as would ending gerrymandering, fixing the supreme court, fixing campaign finance laws and citizen united. There are numerous opinion polls out there that show a wide gap between what citizens want and what Congress does. I will also point out, in case this is the sticking point, there is not necessarily anything "socialist" about universal healthcare. Insofar as something is not subject to market forces, there is no reason to call public funding of it "socialist". For your own history lesson, I'll remind you that it only became known as such because the right wing found it an effective way to fear-monger and the far left co-opted the term in response.

[@LuisAldamiz]

@@dylanrichardson199 - It's not obscure nor pejorative: it's clear and descriptive: a social system dominated by the parasitic class of capitalists (those who become rich by having other people working for them). It's not Smith nor Ricardo who described Capitalism, it was the early socialists trying to make sense of their world and the struggles they lived through: why even if the aristocracy (landowners) had been deposed, inequality, injustice and class war were still major issues? While it was Blanc who coined the term, surely the deepest analysis was of course made by Karl Marx. It's a very extensive analysis but, for what matters here, even if you begin with perfect markets, they are bound to become concentrated monopolies (or olygopolies acting in cartel, same thing). That's because the markets, even if not rigged, exclude the weaker competitors one after the other BUT do not incorporate replacement competitors, hence market competition dooms itself to monopolism. Those very first capitalists of the Netherlands (soon to be imitated by those of Britain, etc.) created the first stock-owned corporation: the VOC (Dutch East India Company), removing themselves and their personal property from any risks and damages that the corporation could do. Like impune puppeeters they control the economy since those days, exempt of almost any responsibility and yet extracting nearly all social wealth (produced by workers).

[@crubs83]

Reminds me of COVID. We've all seen doomsday movies like "Contagion", but never really considered it happening within our lifetimes. We knew it was possible, but never took any serious measures to prepare for it.

[@Kannot2023]

Diseases don't wipe out big populations, even native American who 80% where wiped out, left 20% who survived

[@thomasvleminckx]

COVID is a joke compared to the outbreak from Contagion. A better comparison would be the plague (in a no-antibiotics scenario) or the Spanish flu.

[@Muskar2]

We were very lucky that the first modern pandemic was from SARS-CoV-2 and not something a lot more serious, indeed. There's a now infamous TED talk of Bill Gates warning about how unprepared the world is of pandemics in 2015 and how small things could help us prepare greatly. And nobody did anything as far as I'm aware. Although some countries like South Korea were more prepared because they've faced much more recent outbreaks. And this short-sighted view of existential risks is certainly an issue - but I don't think it's just about assuming we'll make it. I think it's also more about humans having a very bad sense of risks. Our intuition suggests something like 80+% = 100% and

[@yeetyeet7070]

this might just be a nurture thing, everyone lives their entire life as themselves, so it would be strange if one didn't feel like oneself is the protagonist in the universe. This would still be the case for anyone growing up without religion or fiction.

[@LKRaider]

There is only You. There is no “stepping outside” of You. But You can choose what is the highest value to worship.

[@sinity8068]

"It sort of happened somewhere (...) still feels real" Reminded me of Erogamer fic: ------------- > "Mister Charles, suppose I told you that you, right now, didn't exist. Say, you're only an animation in a Disney movie. How could you tell?" > "I… what?" said Charles. "If I know literally anything, I know that's false! I have to exist in order to ask whether or not I exist!" > "Aha!" said the grey-bearded preacher, holding up a finger. "But maybe you don't exist and therefore you're not asking whether you exist. How can you tell the difference? If you were an animated drawing, you'd be saying just the same thing." > "That… honestly sounds to me like nonsense," said Charles. > "Exactly, mister gentleman Charles sir! It is nonsense just as you say! Maybe somebody outside of existence thinks that we don't exist, but from the inside of reality we've got no way to know that and no reason to care. It's the same way with Disney princesses. You can say all you want that Snow White doesn't exist, but she can't hear you, so as far as she knows, she exists. She's got just as much proof of her own existence as you have." > "But…" Charles said. "But, accepting that for the sake of argument, how do we go from that to Snow White becoming real here?" > "Ah," said the preacher. "Well, that's where Bob comes in." ------------------ > Even so, in Charles's opinion, Sonia was trying too hard to optimize her build. For her first element she'd taken Flesh, to be the party's healer and general transhumanist. Sonia was still trying to explain exactly what her second element was, and it was delaying the whole campaign from getting started. > "Stephen Hawking," said Sonia, "A Brief History of Time. 'Even if there is only one possible unified theory, it is just a set of rules and equations.'" In the dream Sonia looked younger, but she was still taking her usual tone of a sixtysomething professor. "'What is it that breathes fire into the equations and makes a universe for them to describe? The usual approach of science of constructing a mathematical model cannot answer the questions of why there should be a universe for the model to describe.' Either this question has an answer, or it does not. Let us suppose that it has an answer, and designate as 'quintessence' whatever it is that actualizes a mathematical structure---whatever breathes fire into equations and makes them describe a universe. That's my second Element." > "You can't do that!" said Starry. "The whole premise behind my character is that everything already exists somewhere and that's why she can summon it! You can't introduce an Element into the campaign that makes things be real. That means other things wouldn't be real!" > Sonia sighed. "Can't your character just summon from among all possibilities rather than all actualities, and actualize possibilities only upon their summoning?" > "My character doesn't want that much responsibility," Starry said. > "I don't think it makes sense either," said Charles within the dream. "How could you tell from inside a universe whether it had quintessence or not? Even if you say Snow White's universe lacks quintessence, from the inside Snow White has no way of telling that." > "Hm," said Sonia. "I suppose that's a fair point. Well then, let us suppose instead that even though everything exists---or equivalently and more simply, nothing exists---there is nonetheless some mysterious factor by which certain things exist more." > "Bullshit," Charles said. > "I suppose we could call it that," said Sonia. > "No, I mean the whole idea is bullshit," Charles said. "It sounds like a metaphysics George Orwell would invent as a parody. 'All animals are real, but some animals are more real than others.' How could they tell?" > "You claimed yourself that Snow White exists as much as we do," said Sonia. "Then why don't we run into her at the corner shop? Why do I find chocolate biscuits there, instead of Snow White? There is a story in which the two of us meet, and the people within that story have no way of knowing themselves to be unreal. And yet I find myself discovering chocolate biscuits instead. Clearly, there is some factor that the possibility containing myself and chocolate biscuits possesses in greater quantity, compared to the less real possibility containing myself and Snow White. Even if we reply to the great question by answering that nothing exists, some zeroes are greater than other zeroes and quantitative ratios may be established between them. Like any other self-observing structure, Snow White finds herself to be exactly as real as herself, a ratio of one to one, and in this sense her existence is locally an absolute. But to say this does not say whether Snow White is more or less encounterable than other things. There is some quantitative degree to which our universe is looking more towards the chocolate biscuits. Some essence of how much something is observed or experienced, which the chocolate biscuits have more of than Snow White. We could call it quintessence, or propensity, or mana, or the blood of God." > "Or bullshit," Charles said. > "I suppose that term is as good as any other," Sonia said. "By whatever name, it is the single, sole, and only truly universal form of money. And we can extend the same logic further. Having postulated the notion of bullshit, it would follow that things are more real only to the extent that they are, in some sense, more bullshit. Or rather, by definition, anything that makes a possibility more encounterable is exactly what we are calling bullshit. Possibilities are experienced by conscious beings in exact proportion to the total bullshit breathing fire into those possibilities. Then to whatever extent a mathematical model is not describing bullshit, it is mere math. Only to the extent that a mathematical structure is about bullshit does that structure form an encounterable part of its universe. It follows that every sapient species, if they investigate the physics of their universe far enough, will eventually find a level at which physics seems solely to describe the arrangement of pure bullshit---some physical quantity whose apparent meaning is making possibilities more encounterable. By your own argument, the presence or absence of bullshit must be falsifiable for anyone inside the universe to notice a difference. Then the eventually-discovered laws will show that variations in bullshit are experimentally observable and cause other events to proceed differently. Indeed, variations in bullshit will be the only causally potent factor. Across every universe with conscious life, any inquisition into physical law, sufficiently advanced, will render everything into bullshit, which is and must be the sole constituent of experienceable causality." > "Can you simplify that?" said Charles. > "No, but I'll do it anyways," said Sonia. "Whatever it is that makes one possibility more encounterable than another, we are calling that bullshit by definition. Then whatever bullshit is, if something isn't made of it, we won't encounter it. And if we couldn't detect something experimentally, we'd have no way of encountering it. So bullshit has to appear to us as a quantity in our equations---in fact, the only thing structured by our equations, because we can't encounter anything else. 'By convention there is sweetness, by convention bitterness, by convention color, in reality only atoms and the void.' And underneath atoms and the void, it's all just bullshit." > "Can you simplify that some more?" said Charles. He felt ashamed, but he wasn't at his mental best while asleep. > "The only thing that truly basic physics equations can be about, is the stuff that breathes fire into equations, the stuff that makes things be more real," said Sonia. "That's what I'm taking as my character's second Element, if the DM lets me get away with that." > "Then what the hell is a reality anchor?" Charles said. "The whole reason I'm having this dream is that I'm nervous about being turned into one of those while I have no idea what it fundamentally is." > "A reality anchor is somebody who has, or rather is, a whole lot of bullshit," said Sonia. "In the quantum mechanics you knew, the density of bullshit was spread widely among many possibilities. Now suppose we focus, into a single possibility, an enormous amount of bullshit, structured by laws which say that all of the bullshit follows a single path into the future and doesn't spread out over time. Then from the standpoint of anybody looking at you, there is a single possibility for Charles Adan that they are far more likely to experience than any other." > "But what does that feel like?" Charles said. > "It feels like being the Charles who gets reality-anchored and not any of the other Charleses," said Sonia. "Your presence weighs heavier on future events. To use a literary metaphor, the larger story of your universe is more likely to be told from your viewpoint and center on what happens to you. Also, reality anchors automatically acquire superpowers." > "Yeah, I'm not buying this," said Charles. "Not. At. All." > "Well, it happens to be true anyways," said Sonia. "Not just here, but in any other universe you'll ever be able to enter. It's not a law you can ever escape. People with enough bullshit acquire bullshit powers, and the more bullshit they are, the more bullshit they are."

[@hammagamma3646]

holy shit, thats a good video. That example with the people from 200 years ago...i felt that.

[@OldFormat]

We live next door to two uninhabitable planets. Venus may have been able to support life in the past but a runaway greenhouse effect has made the planet too hot and dry to support complex life. Mars is similar but went the other way to airless, cold, and irradiated. We exist in a middle time on Earth. The planet was too hot in the past and will be again in the future. Even on a smaller time scale, the planet has been far less habitable with extreme swings to being mostly ice covered. What is rather new and unique is we seem to be generating a shift in climate much faster than natural forces have done in the past.

[@danielweber9414]

I can't speak for everyone, but I also think something like "With all those different possible extinction scenarios (+ Great Filter), we'll almost definitely not make it, and there's nothing I can do about that, so I might as well enjoy life in the meanwhile"

[@AxeSovax]

Nice video. Seems like the best starting point is to develop understanding of deduction vs induction vs inference. Should we try and take more of a bayesian approach when considering the future of AI? I assume so.

[@cyndicorinne]

This video was inspiring. I know that sounds weird, but it was 💜

[@anishupadhayay3917]

Brilliant

[@stevenbaumann8692]

People often view their own deaths the same way. You are correct. Your assessment as to why is pretty spot on IMO. Although I’m confident that as a species obsessed with a single apocalyptic event, that we won’t even see our own extinction coming. It’ll be slow. It’ll be painful. And 50,000 years from now someone will be hunting in a field with a rock, lol around, and have a heart attack. That’ll be the end.

[@nickalasmontano1496]

You're absolutely correct. But at the same time I'm not sure I see what's beneficial in keeping in mind that there might not be a solution. It feels like that just opens the door to give up in despair at a certain point. The only useful thing I could see from such a scenario is that if everybody thought this way an AGI would never be made.

[@theblinkingbrownie4654]

There's definitely a solution considering we exist, so i dont think robert said that

[@elijahclaude3413]

I think that IS the point. If we know that there is a massive risk of danger, and we also know that we currently have no solution for that danger, then it's best to avoid the risk of said danger as much as possible until we do have a viable solution. It's like if you're hiking and you see a cliff. You know that you will fall and die if you walk off that cliff, so better to turn around and find another way. Until/unless you or someone else is able to build a bridge or stairs or some other way to traverse the cliff. Unfortunately, many people today are busy trying to run full steam off the cliff even though we have no viable solution down the cliff! And they are taking all of us with them. We need more people to seriously realize that we don't have any viable solutions right now. Sure we can and should look for some, but we seriously should wait to try and build AGI until we have those solutions!

[@MrGreenAKAguci00]

Well, I'm willing to contemplate and experience the existential dread of the possibility that we will be wiped out by something we can't even comprehend not because it's so complex or difficult even, but because it may just be in our blind spot and will hit us randomly seemingly from nowhere. Plague, asteroid, comet, AI, nuclear war, major climate shifts and catastrophes, there are a lot of angles I can think of and even more that I can't think of. I am willing to really digest such a possibility but I can't be living with it present in my thoughts all the time. I would be paralyzed.

[@osakanone]

Came back to the video and rewatched a few times to think about this. Its the difference between understanding and comprehension. With exponential hyperobject problems (Timothy Morton has a fantastic explanation of this problem), humans suck at internalizing their nature because they are too abstract for us cuz we're great with linear increments and small to medium scale stuff, but god we suck at stuff that shoots off to bonkers places super fast like "how many football pitches is the sun" or "how many human lives fit into the history of earth stacked end to end?" since we use those big long things as ways of understanding those kinds of distance, since we didn't evolve to be around that stuff at all. I think part of the risk is people think fiction is the same thing as a thought experiment. What this tells me is that our representations of thought experiments to people need to change. It makes me wonder if the world needs a story that talks about the truths of these problems and explores them in a way that people can understand, like an emotional story as interface to learning? I remember that when people really started caring about the environment and companies got obsessed with the optics of doing so was after we got movies which talked about the issue. I guess art in that sense is like the central nervous-system of a society or something? Like, a way to help someone comprehend even without understanding on a gut level that thing dangerous. "If we can convince even the least interested person that this is important, then that has to be worth something, surely?" Like what's that old saying? You convince people with emotions, not with facts? Having to perform appeal to emotion does suck but when the stakes are this big, do we even have a choice if we wanna survive?

[@418teapot9]

Well there goes my good mood for the day. No but in all honesty good video!

[@shoug6555]

Death is certain. No matter what any lifeform has ever thought its fitness function was, every single one, historically, has eventually "failed."

[@WryAun]

You have a wonderfully expressive face, Robert, I think it's part of why you're a great storyteller and these straight to camera talks really highlight that skill! Thanks for putting this feeling into words. I think it made me reconsider my own complacency

[@leow.2162]

His facial expressions are probably more pronounced because this is a thing that really annoys him

[@WryAun]

@@leow.2162 The passion is wonderful

[@LKRaider]

Also because he is as thin as a sickly person, all the bones and thin muscles showing through. He should probably exercise, eat and sleep better.

[@nikczemna_symulakra]

After all, at the very bottom of Pandora's box.. after all the whole crappiness crawled out.. there was this one thing called hope. Hellenes already knew it's a double-edged sword

[@tatianabeastmode6573]

Think no more, lad; laugh, be jolly: Why should men make haste to die? Empty heads and tongues a-talking Make the rough road easy walking, And the feather pate of folly Bears the falling sky. Oh, 'tis jesting, dancing, drinking Spins the heavy world around. If young hearts were not so clever, Oh, they would be young for ever: Think no more; 'tis only thinking Lays lads underground. (A.E. Housman)

[@davidmays2660]

4:05 "Religions, they're often okay with the idea of an apocalypse (of humanity or the world ending), but they're not okay with an accidental apocalypse. They have a very hard time taking that kind of thing seriously." Honestly, this strikes me as an incredibly human psychological response to death, although just on a more massive scale. As individuals, I believe it's important to recognize this human tendency if one wants to live a life of intention and (hopefully) try to act in ways that make a better chance for a good and meaningful life for everyone. It reminds me of a story that I think everyone should read at least once: "The Death of Ivan Ilyich" by Tolstoy. It reveals a lot about how human beings, particularly those with an inflated ego or sense of self-importance, simply presume that everything that happens in their life is, as you say "part of the plan." In the story, Ivan Ilyich is a successful and respected judge. One day, while reaching awkwardly for curtains, he falls and hits his side on the windowsill. Although he initially dismisses the injury as a bruise, he soon begins to experience pain and other symptoms that ultimately lead to his death. Throughout the story, Ivan struggles to accept the reality of his impending death, which is caused by a seemingly insignificant injury. He believes that his life should have had more significance, and that his death should have been caused by something more important or meaningful. This struggle to accept the trivial nature of his death reveals our tendency as humans to look for meaning in our lives, especially in the critical moments such as death. We often feel that our lives must have a purpose or significance, and we may struggle to accept the idea that our existence could be meaningless or insignificant. Ivan's experience reflects this universal struggle to find meaning in our lives and to come to terms with the reality of our mortality. The story also suggests that our preoccupation with status and success can prevent us from living fully and appreciating the simple pleasures of life. So glad you made this video -- it's depressing as hell, and it's not fun to look at, but it's important. And it's important to continue making meaning in your life regardless of the objective absurdity we all face.

[@nickthurn6449]

After trying to debunk some AI boosters I was directed to your channel. You are a very smart, very fair, thoughtful guy. I'm still digging through your videos but I see some points I disagree with - I'll continue, perhaps you will educate me (perhaps not! 🤣). Hey I'm.old! Regarding this video - there is a level of existential terror many people refuse to face. That seems to be your position. We may go extinct suddenly despite being a unique chemical phenomenon. Or despite being surrounded by "friends" we can never meet because fusion is impossible at scales below stars. Don't give up - existential terror is the home of those who join the dots - pat the cat and be thankful for all the good things (where "good" may vary widely)

[@bernhardkrickl5197]

Funnily enough, just recently some fiction showed me how we might actually lose control of AI and how hard it could be to contain. I am just watching the series "Person of Interest" where a brilliant engineer creates a super-intelligent AI to do, essentially, predictive policing. In flashback scenes in the fourth season it was shown how he struggled hard to build the AI in a way that it wouldn't run loose and destroy everything. The AI eg. tricked another developer into giving it his password so it could take over the developers laptop for more processing power. Another version tried overheating a server so that the automatic fire-extinguishing system would suck the air out of the room killing the programmers in it that could have stopped the AI. Of course, in the series, the brilliant engineer solves the problem. But then again, a longer running story arc is about a rival Super-AI that is not bound "morally" the way the original one is. I have yet to see how that plays out in the series. ;)

[@maxmustermann5353]

10:02 It might be hard, but you make it easy to understand. If you answer a row of the "why not just" questions, it becomes very apparent what you mean.

[@karlhenke91]

Stories are how we explain the world. Sir Pratchett taught me a lot about that. Granny Weatherwax's sister was the "good" one, because she told the stories and made everyone's dream come true. But Granny was actually the good one, because she lived in actual reality. Life isn't a story. It can help point us in a direction we need to go. Stories can have truth, but they aren't true. Humanity, in nearly all stories, either wins or is still around to fight again, and it may just not happen that way. Another thing is, your average person reading KZbin comments hasn't lost. They've had setbacks, sure, but I'd bet a good chunk of people who do take this seriously have just...lost, at some point. Declared an unconditional rout, and retreated. I'm not going to say how (so as not to anonymously trauma dump or anything), but I feel like the fact that I have, in a part of my life, completely and utterly failed, lets me realize we can just...lose. It's a reality, and it's terrifying, but it can happen to anyone. It doesn't mean we will fail, of course, but these are threats we have to take seriously.

[@ixion2001kx76]

This is seen strongly in long timescale simulations of the habitability of Earth-like planets: that the Earth has been habitable for long time scales BY ACCIDENT, nothing really maintains that long term. Most planets won’t be habitable for that long.

[@HITABikes]

Buddy I'm working on it. It's just hard to know sometimes if you're doing the right things.

[@JM-us3fr]

Great video Rob! I have a video request: Are we misaligned intelligences? From climate change, to factory farms, to reward-hacking with drug addictions, we seem pretty similar to the pathological examples of AGI given in your videos. Just thought it might be interesting

[@vwabi]

Humans being misaligned is by definition not possible (unless you're talking about humans being misaligned to each other). What you're describing is humans acting irrationally. They still have their own interests, they just act in a way that doesn't always most effectively achieve those interests.

[@LuisAldamiz]

What do you mean by "misaligned"? We were never meant to live in a civilization, we evolved as hunter-gatherers and therefore we are "misaligned" re. civilization or even agriculture even. We created those things out of need or accident and we did so because we are intelligent tool-creators and social animals. They are somehow part of our evolution but they are cultural shells of evolution and not biological aspects, genetically speaking we remain hunter-gatherers and deep inside we yearn every day for a return to hunter-gatherer "primitive communism" and thus we value things proper of that unreturnable social stage: freedom/democracy, solidarity/socialism and truth/science. We cannot return but we can maybe fix techno-urban society so it fits our actual needs much much better than the disaster oligarchism has created.

[@JM-us3fr]

@@vwabi I was going to leave the details a bit ambiguous to encourage a rich discussion, but yes, obviously an agent cannot be misaligned when compared to their own goals; but they can be irrational. Also, obviously two agents with different terminal goals would be misaligned relative to each others’ terminal goal. But I think there is something to be said about how we generally don’t like unnecessary suffering, and we generally have a harder time empathizing with nonhuman animals, and the horrors of factory farms exists. This seems a lot like what Rob described as having the top 20 parameters being optimized and letting the 21st parameter get set to an extreme value.

[@JM-us3fr]

@@vwabi Oh also, there is still a way humans can be misaligned, and it’s in the exact same sense a machine might be. If humans have slightly different terminal goals, then what exactly does misaligned even mean? Do we take some sort of average of everyone’s terminal goal, and set that as the ideal terminal goal? If that’s the case, then most humans would be misaligned.

[@tim40gabby25]

@@JM-us3fr agreed. Checkout an apparently successful person sitting alone in their Bugatti in a smoggy traffic jam crawl - and step back.

[@dylanrichardson199]

We need more movies and literature where everyone dies or fails at the end. I'm not referring to tragi-comedies; there are plenty of those. I mean ordinary, mundane narratives where eventually, the "just barely make it" scenario turns into a "don't make it scenario". No holding onto the ledge by the finger tips, just straight up plummeting, roll the credits. I would like this to be completely serious too, no deeper lesson or meaning, no rationalization. That's just how things happen sometimes. Just think of how much more enjoyable and interesting films and books would become!

[@channel11121]

A good one is: To Build a Fire by Jack London

[@damian_smith]

"Knowing" was a very annoying movie, but satisfies that requirement.

[@PriitKallas]

Don't look up is a good movie about not surviving

[@LarsMacD]

Well, there is only way to actually find out. (or, as Oat Willy used to say, "Onward through the Fog.") Seriously all we can do is keep going, and seek the wisdom to deal with whatever comes along.

[@erickmagana353]

Though I agree in spirit, in practice reality in not as important as believing that you can handle it, that you can figure it out, that the problem can be fixed. Otherwise there's only paralyzing panic or worse, nihilism and cynicism. What I'm trying to say is, if you want to solve a Rubik's cube it's very useful to believe that the Rubik's cube is solvable and that you can solve it.

[@spa3093]

In the spirit of "...so we'd better get to work", I have started.

[@tronvillain]

The problem I see with this is if we're NOT capable of dealing with something or don't have time to deal with something, then you're in the "massive asteroid impact five hundred years ago" situation: do whatever you want, it makes no difference. What we HOPE is that we're in is the "massive asteroid we see coming fifty years from today" scenario, where we're doomed if we do nothing, might fail, but might be able to do something. You just want people to be more worried about failing and more invested in doing something, not to take the "it's inevitable, we can't do anything" position. Especially when what you're worried about is that we're possibly building the start of the asteroid ourselves and giving it a push.

[@Laborejo]

My bet is on a super high energy, tight-beam Gamma Ray burst. You see it the moment it is too late. Literally physically nothing you can do.

[@John_does]

Another reason, particularly for the AI problem, at least the reason I think I have, is that this is a complete human creation and until it's too late it is realtivly easy to stop.

[@alexandregermain8011]

You might not suspect it's too late until the last nail is hammered.

[@LuisAldamiz]

Robert has videos on that emergency stop button, watch them.

[@dgershko]

We're programmed to be optimistic (because otherwise the algorithm won't be admissible)

[@sirpoopface1181]

I have tried to get this across to people so many times. Like you said they understand it in an academic sense, but they don't feel it. It (most of the time for most people) takes deeper integration of a belief, and actual acceptance of it before that belief starts showing up in people's actions. This isn't meant to knock people down, or be superior(I am much the same way). But it is a problem, because there are so many problems that are not being taken seriously. Existential problems as well as quality of life/value problems. The human brain is not perfect, it is limited, and without recognizing this it seems unlikely that society can shift in the ways necessary to avert the danger. It's funny to me when people hold up stories, either scifi or religion, as evidence for how something is or is not a concern. One of, if not, the biggest problems is that people understand life, the world, and themselves as some sort of narrative. It's a way to contextualize and place yourself in the world so that you can map out actions, and maintain a model of the world. But that's not what this is. There is no story, we are not "meant" to do or be anything, and there is nothing in the universe that inherently pushes us towards our goals or away from our problems. We are entirely responsible for it, and the only way that it happens is if we do something about it. Hoping, wishing, wanting, nor disliking something have the power in themselves to affect an outcome. I desperately want to believe that we'll get past our problems, but my entire life has told me that it's not likely. Anyway, I sincerely enjoy you, your videos, the thoughts, and approach that you bring to things.

[@Traumglanz]

The real tragedy about is that most issues seem pretty "easy" to solve. Just mildly inconvenient.

[@sirpoopface1181]

@@Traumglanz Absolutely, it's not a technological problem at this point. It's a sociocultural problem. People are just reluctant or refusing to make the choices that we need to.

[@dariusduesentrieb]

Either we can do something about it, then I as an individual without special knowledge or skills in the relevant field won't make a difference (except maybe vote for a relevant party when having the luxury to live in a democracy), or we can't do anything about it, then I can neither. So I don't care about it, unless I find the topic interesting.

[@TheJamesM]

An important point implied by but not made explicit in the video is there no guarantee that the problems presented to us by the universe will be solvable; there is also no guarantee that _the problems we create for ourselves_ will be solvable. With the asteroid example, in a situation where we not only lacked the technology to do anything about it, but also lacked the scientific basis to even know how to work towards that kind of technology, in a sense knowing about our impending doom (or even its possibility) isn't tremendously valuable, unless you count allowing everyone to prepare for annihilation. But where I think it's really crucial is where we're creating a potentially intractible problem for ourselves, and pushing it more and more in the direction of intractibility with each passing year. Far too many people - many of them in positions of considerable power - would rather blithely pay for today's excesses with tomorrow's hypothetical technologies; technologies that might never come, or might arrive too late, or might not even be abstractly possible.