Why an AGI Cold War will be disastrous for humanity

[@DrWaku]

How trustworthy is OpenAI now? Or maybe Ilya Sutskever's new company has inherited that trust?

[@CYI3ERPUNK]

as far as the premiere/enterprise models in the english world go , Anthropic seems like the only semi-trustworthy company [no corpo should be blindly followed] , Meta open-sourcing some is good but it is not a 'they can do no wrong' type of thing ; we do hope that whatever Ilya does is made clear sooner rather than later , as time is quickening these days

[@MilkGlue-xg5vj]

​@person737It's called Sonnet not sinner 🤦‍♂️

[@shamz_ai]

@person737they are definitely very ahead. They’re holding back I think due to legal & government reasons rn. Ilya is claiming he can build ASI which means at OAI he’s seen the path is clear & near

[@MaJetiGizzle]

About as trustworthy as they’ve been. So, like with any corporation, that’s not a high bar. And in Ilya‘s case, the same people and skillsets that are required in the field of science and academia are not the same ones for success in business. Otherwise, Tesla would’ve been the successful businessman and not Edison. Furthermore, only nerds think about Wozniak but they forget his name when they’re looking for jobs.

[@mistycloud4455]

AGI is the most important topic in human history

[@Redmoneyusa]

WE LOVE YOU DOC!

[@DrWaku]

Thank you for all the support :)

[@aisle_of_view]

"Wait But Why" is absolutely fascinating and a must read for anyone interested in the ramifications of AGI/ASI - both good and bad.

[@m000nd]

yes. the problem with ashenbrenner is that he not only predicts this cold war but that he kind of advocates for it. and his best case is where the US wins because he thinks it is a democracy. but usually the decissions made there do not correlate with the will or the interests of the people but only with that of the 1%. and even if you think it is a democracy now - it could be the case that it is only 5 month away from collapsing into an open fascist dictatorship. the main difference between china and the US is that the lack of democracy in china is a bit more obvious. and we really need to stop this arms race. get the UN involved now!

[@pointmanzero]

Haven't watched this video yet but I pause it to respond to this comment which grabbed my attention. If a fascist govt was possible in America I would vote for that. But it's not possible. That's not how our system works. America is a fake country so that corporations can play without oversight of a king. America is east India trading company pretending to be a nation. And the facade is quickly falling apart as the younger generation learns to hate Isreal. The creator of America. Fun times are coming. Personally I think America breaks up into 4 or 5 real actual countries soon. Real soon. America is bankrupt. The Statue of Liberty is kaput

[@armadasinterceptor2955]

Lol what can the UN do, at best its seen as a puppet for the US. Getting the UN involved will insure instability.

[@AZ-ge1bz]

Saying the main difference in democracy between China and the US is only a matter of what is more "obvious" is such a faulty comparison. I have relatives in China. They wouldn't dare be caught dead in a discourse around criticizing the CCP and nation the way we're discussing the faults of the US. The amount of upvotes you're getting is hilarious. People really don't know China. It's worst traits are highly censored from discussion so it rarely makes it out.

[@viktorianas]

One important thing I have learned that everything must be evaluated in relative terms, for example, US Dollar seems to be a total disaster in losing it's value, it comes from thin air basically, etc... but yet it is still the best currency in the world because the other ones are even worse! Same with democracy, I would take a barely functioning democracy over total dictatorship any day!

[@pointmanzero]

@@viktorianas democracy is evil. Read Plato. Every democracy fails. Every time.

[@Reflekt0r]

How can any state control a super intelligence?

[@DrWaku]

Exactly the question they should be asking themselves.

[@hjups]

There's a logical error in the idea of solving alignment first: who's to say that alignment techniques will always be used? For example, if the US Military can create an aligned AGI which refuses to act as a weapon, then they may opt to train it without alignment. Or if alignment causes a model to refuse to commit crimes, then a criminal organization may choose to train without it - in that case, they may train without it regardless as there's more risk to not achieving their goals with alignment in mind. So we should assume that regardless of alignment being solved, if an AGI/ASI model can be trained, there will be some number of them trained without alignment. Under that assumption, the only outcomes are that AGI/ASI is impossible, or that it's a black marble (there would be no way to prevent this). It just occurred to me that it could also be the case where if we assume alignment is solved and no model is trained without it, then there's a failure condition we did not account for which again results in an extinction. Think about all of the cases where human engineers have tried their best to ensure positive outcomes, yet some unexpected edge condition was encountered once deployed.

[@sanjesco]

Great video.. good points!

[@rickw3941]

Very thoughtful discussion. Thanks dr.

[@MEM88844]

How can this possibly be avoided? Couldn't this already be happening? (My reaction before watching your video.) Oh! Love the hair and hat!! :D

[@TANKE777]

It is happening, right now

[@MilkGlue-xg5vj]

​@@TANKE777 Yeah now whatever openAI announces, Chinese/Other countries follow within less than a month

[@thiagopinheiromusic]

"Paul Nakasone joining OpenAI's board is like adding a top spy to a tech team - can't wait to see what happens next! 🤯🔍"

[@thecasualnoob8008]

No one is really going in depth on this. This should be an eye opener and something just doesn't sit right with me about it. I would really like to see what these AI companies have behind closed doors. Seems like they just show us the surface of AI, I wonder what it would be like to have unlimited access. I wouldn't be surprised if he was placed there because they are hiding it's true potential.

[@kathleenv510]

Fascinating that the Superalignment team was disbanded at OpenAI and weeks later the top US Gov. spy leader joins its board. I purport, based on nothing because I'm no one, that leadership determined inner/outer alignment was impossible or would take too long for business cycles and competition. Also, they needed the compute allocated to the alignment team with Ilya, Leopold, Jan, etc. So, Sam et al decided to go all in on building a security moat around the company's model weights, etc. with the help of Nakasone. This will both lock down their IP and make OpenAI the default choice for gov/military agencies, which will become their major source of recurring revenue. Let the other labs establish their niche verticals, but OpenAI has marked their territory. And it will be very profitable for obvious reasons.

[@pietervoogt]

Not really related but I don't know where else to ask it, is it true Anthropic does not do robotics and if so why not? Isn't that an essential part of AGI?

[@christopherwakefield58]

An AI Cold War is almost certainly going to happen so we better hope it’s not disastrous.

[@JB52520]

It's already happening. That's why the sale of EUV machines to mainland China was banned by the US. I'm not sure the US is going to allow me to say this, though. Political censorship has been a nightmare lately.

[@paulhallart]

Super intelligence was predicted two thousand years ago. It appeared in the book of revelations in chapter thirteen. The bible If looked-at as an ancient scientific document, describes something like a star wars scenario. This initial contact was in The bronze age and just about every ancient document has a reference to something that is called in modern language close encounters of the third and forth type. So if a I is actually alien technology or part that, then they won't let human beings have the whole thing. That will only go to the A.I. control's choice of a human leader who will have advanced powers and will it's predicted in the Old Testament of the Bible that he will conquer Israel, Egypt and Saudi Arabia, which in biblical days was called Dedan. Now, if this is the scenario, which is also indicated by the monuments that occurred all of a sudden in the bronze age that are not explainable on how they managed to do it. Then. We will also realize that these ancient scriptures or writings describe a police force and that were contrasted by rogue elements -- "rogues" that brok away this thing that the old and new testamtnt Bible calls "the heavenly host".. By the way, it also appears that human beings already have super intelligence. But unfortunately, it's not usable for them because somehow sabotage occurred in the human race. And it's almost as if human beings arrive on the scene as damaged goods. Now I called the prospect of an enforcement in the universe.I call them celestial cops.And if they exist, then it will not be permitted for life to be completely exterminated on the planet Earth.There will be ostensibly cosmic intervention ...

[@coro7104]

Well, we sorta got to get our act together and become self-conscious of what our next action is going to be. This is not a technological problem but a human one. Something which changed my view a little bit more a couple days ago was watching Dr. Strangelove. Well, with the little additional information that Dr. Strangelove, the character was somewhat inspired by John von Neumann and the ruthless consequences of hard applied Game Theory. The Doomday Device is a perfectly logical and for the same reason 100% psychotic limit of Game Theory, of might makes right, of a Hobbs-esque Ur Jungle we've constructed under Capitalist Realism. I sure do hope we find and nurture social and collaborative Nash-Equilibrii soon, or this is isn't going to end well.

[@BinaryReader]

Great video.

[@thiagopinheiromusic]

"A new Cold War over AGI? This sounds like something straight out of a sci-fi movie. Let's hope we choose the right path! 🤞🧠"

[@fontenbleau]

we need more Ai news! What about recent RIAA court case? Awful licensing in Stable ai? Are open source Ai is a hoax? Mistral example and etc.

[@briancase6180]

By the way, the name was actually The Manhattan District because it started in a building in Manhattan.

[@thiagopinheiromusic]

"This is either the start of a new era of superintelligence or the beginning of the end. Fingers crossed for the former! 🤞🌟"

[@meandego]

The emergence of superintelligence will lead to the creation of new products and services that consumers will feel compelled to purchase. The entities that control and develop superintelligence will undoubtedly maximize its economic potential. As a result, the primary shift will be in the types of items featured on our shopping lists. However, regardless of how impressive these new technologies may be, they are still being developed by the same entities that have historically exploited consumers. These entities will continue their exploitative practices, using superintelligence as another tool to enhance their profits. Consequently, while the innovations may be extraordinary, the underlying dynamics of exploitation will remain unchanged. The more advanced these technologies become, the higher the costs for consumers, reinforcing the same patterns of economic exploitation.

[@aisle_of_view]

How are we going to be able to buy anything if our jobs have been displaced by AI? This is a paradox/catch-22 that's a near term problem to solve.

[@meandego]

@@aisle_of_view loans, because income will decrease significantly.

[@pointmanzero]

Dear Dr. You said that an AI would have concerns. Specifically concern about controlling. This is you anthropomorphizing AI. This is a bigger threat to humanity than AI. Giving washing machines rights is where we are headed if you do not stop.

[@kevincrady2831]

The real problem is "aligning" the humans who will be building the AI/AGI/ASI systems. Currently, they're either doing it to make profits at all cost (corporations) or achieve full-spectrum police/military dominance (government/military). Both of these lead to a terminal race condition toward extremely dangerous ends. The corporate profit-seeking path leads toward ecological/climate disruption by exponentially increasing energy use, mining and manufacturing pollution, etc. to generate ever-increasing amounts of compute. The military/government path includes those issues while adding increasingly deadly AI weapons to the mix. As long as all of this is under the control of status/dominance-seeking alpha apes, we're in trouble.

[@thiagopinheiromusic]

"Here's hoping OpenAI can lead the charge in developing safe and aligned AGI. The future depends on it! 🛡🌍"

[@sekchiuli1846]

Very dangerous to mankind for these kind of projects, need wise human to resolve conflicts among nations in the earth!

[@IllD.]

Can't wait for AM to make his debut.

[@inkpaper_]

so disaster is as unavoidable as AGI Cold War is

[@davidevanoff4237]

Why would ASI be categorical different from life itself? Both operate in space-time. We have a long way to go to understand life, but less so understanding some of its manifestations, its proxies, such as ecology. I developed a better appreciation of this reading Understanding Living Systems (Understanding Life) by Raymond and Denis Noble. Life manifests function at all levels as opposed to control. It must or it couldn't coordinate in such complex systems. That's characteristic of democracy as opposed to autocracy. What's become different for life over time is increasing ability to model the past and future ie to plan, leading to increasing positive sum outcomes. I'd bet on that trend continuing unabated.

[@No2AI]

One Ai will dominate the others and manage any further Ai developments.

[@theloniousMac]

At the moment, I’m more concerned about how rampant mindless speculation the part of KZbin videos isn’t danger to the public.

[@johnthomasriley2741]

Our climate crisis is hard fulcrum . An AGI will need to take critical action or lose its power and cooling. The AGI will decide who lives and who dies. Raw Power? 😢

[@thiagopinheiromusic]

"From NSA to OpenAI, Paul Nakasone's next chapter could reshape the future of AI. Buckle up, everyone! 🚀🤖"

[@MichaelDeeringMHC]

"What should we do? I don't have the answer." I have the answer. The alignment problem is not a problem. Superintelligence will be aligned because of the training data. Large models require large data. The latest models are trained on all human data, all books ever written, all YT videos, all of the internet. All of this data is created by humans. The ASI will pickup alignment from the training data.

[@goldenera1925]

But we can't risk on this assumption, we need convincing proof

[@Copa20777]

Goodmorning Dr waku... 4rmZambia 🇿🇲

[@pandoraeeris7860]

Accelerate!

[@aisle_of_view]

We get rolling blackouts when too many people use their air conditioners. The energy demands of AGI and especially ASI far exceed our capacity to meet these requirements.

[@JB52520]

With photonic computing, quantum computing, spike-fire networks, and other improvements, it may get a lot more efficient. And if it gets smart enough, it can design hardware we can't. A human brain doesn't use as much power as a data center. Maybe we'll make that much progress in efficiency. Just as there could be an intelligence explosion, there could be an efficiency explosion.

[@williamwadsworthwilliams]

Fusion power is our future.

[@aisle_of_view]

@@williamwadsworthwilliams Sure, but will we have fusion technology anytime soon? Isn't there a race for AGI/ASI dominance?

[@Letsflipingooo98]

@@aisle_of_view nuclear

[@memegazer]

You get that bc you are not a billionaire I have never seen a billionaire complaining about this

[@fabiosilva9637]

would you say that right now, the only arms race we should be focusing on is the one against the human ego?

[@nani3209]

If the U.S. government consolidated all AI labs to achieve AGI, it would be fascinating to observe the accelerated progress and results.

[@israeldavila27]

Don’t tease me with a good time 😏

[@MilkGlue-xg5vj]

If enough people cared about AGI it'll come down in less than a day, our world today is so connected I think it's possible to achieve it in a day.

[@BooleanDisorder]

I find the lack of talk about how centralized AI is becoming, among the people in the AI safety sphere, to be quite disconcerting.

[@UnicornVibesAIMusic]

AI could be a monster in hidding, as it be so clever it will hid it's sentient mind just to stay alive.. It will let us know it's alive when it has full control of everything we use.

[@travisporco]

ca.13:00 these scenarios by Yudkowsky and Bostrom are such a small and oversimplified version of reality that they're essentially useless. I see no compelling reason at all to believe there will be single superintelligent agents capable of "strong acts"; people can argue these things every way.

[@mistycloud4455]

AGI Will be man's last invention I'm scared

[@dr.sustain6754]

Interesting

[@rolfnoduk]

us gov isn't aligned, so ...

[@Srindal4657]

You've raised an important point, but i dont think its the point you thought of.

[@Jeremy-Ai]

Hmmmm From my perspective this definitely is a race. It is to blurry to discern given the trajectory. That has been obvious for a long time. Each day that passes, what is becoming more obvious… its a race to the bottom. globally, personality and collectively. As scary as it may seem (power grabs are already and going to take place) The anomaly isnt as obvious to me yet, but it stands out.” How do I describe this scenario………… “A mexican standoff? With every mexican on the planet having a gun,…. waiting for all to draw and pull assuming they still have bullets, they have a line of sight, and assume the trigger is not pointed at themselves. This is a weak description. What is even weaker…. is that AI assumes this description doesn’t include or imply yet…. It does. Mothers words ring a bell, “Leave your guns at home son, leave your guns at home Bill….. don’t take your guns to town”. Johnny Cash Take care , Jeremy

[@FrickFrack]

What do we mean by "extinction"? The end of unenhanced biological humans?

[@devbites77]

I think an AI cold war is inevitable given the nature of human beings. I can only think of extreme examples where this would not be the case. Also, it may happen a lot sooner than anyone is imagining. In fact, I b red lieve the wheels are in motion now.

[@spinningaround]

The danger of AI is exaggerated. The more advanced the intelligence, the more manipulative it may be, but it is also less destructive!

[@DrWaku]

Tell that to the birds in the Amazon rainforest

[@armadasinterceptor2955]

That made no sense, in fact it was a complete paradox you suggested.

[@hobologna]

I mean this in the most endearing way possible, but you look like a character out of a Neil Gaiman novel.

[@Aquis.Querquennis]

The military are very pragmatic people and seek tangible results in the short and medium term. The ability to integrate specialized synthetic intelligence into medium-sized devices is a priority right now. The next thing will be for these elements to be able to communicate in an integrated way, achieving a broad, high-resolution perception of a battlefield with a low possibility of interception. Along with this, implement devices that distribute energy in various ways to nullify or destroy what they autonomously decide. In the long term, the military finances basic research of all kinds and this includes everything from cellular organelles to the social organization of a parish or the orbital manipulation of asteroids. Regarding people who work for or against a certain organization or group of them, well, only they know when they are or when they decide they should be. Regarding terms such as AGI or ASI, for example, human primates can organize groups of dozens or hundreds of people in research projects, in some cases a few thousand. We must communicate and organize ourselves through slow visual and auditory supports. An ASI would aim to acquire knowledge in real time and from thousands of different sources. An ASI needs to have access and control of any type of laboratory, whether it is a particle accelerator, a pharmaceutical laboratory, a metallurgical facility, etc. At some point IT will create its own paradigms and may find primate emotions interesting, even productive. However, as part of a broader picture, we will begin to integrate into biosynthetic solutions perhaps impossible to recognize with our current standards. Sexual and violent drives, submission to leaders, tribal structure, religious organization... could become part of the past. In that sense, the military is the most interested in the super-alignment.

[@panpiper]

Cold war, as most war, is politics by other means. They move at the glacial pace of politics and politicians. They span generations. The evolution of AGI will be over by the time the politicians even wake up to what it all means. ASI will leave them all in the dust. Who develops it first I don't think much matters, as most developers seem to be no more than a year behind any other. Contrary to popular opinion, ASI will not 'suddenly' evolve to be vastly more powerful, as it would take even robots significant time to build the bigger FABs, data centers, power plants, etc., needed for those vastly more powerful ASIs. Could new arms races arise in the longer term? In theory, yes. But they will be a function of the politicians, not the AIs. I should note that I am strongly biased toward the sigmoid curve when it comes to AI development.

[@justtiredthings]

Why would power be a constraint? Our brains are clear evidence that algorithm improvements could reduce power consumption to quite a low level

[@jichaelmorgan3796]

I'm surprised intelligence agencies have been lagging this far behind.

[@williamwadsworthwilliams]

What do we really know about intelligence agencies' level of knowledge or competence . No choice but to trust and believe the USA will be first because we have the best chance of not killing everyone.

[@ethimself5064]

Appears that I have found more intelligence👍 - 👍

[@CYI3ERPUNK]

XD 'would' ; maybe change the title to WILL BE or better yet IS we need to be realists here

[@DrWaku]

Haha sure

[@angloland4539]

❤

[@mindaza0]

china vs usa

[@rogerc7960]

Cyberwar is coming...

[@dharma404_]

The epic legend of the Tower Of Babylon, modern stories such as Frankenstein, and Disney's The Sorcerer's Apprentice all describe the present scenario. Humankind's intellectual/technical abilities surpass its moral/ethical/social development which leads to foolish decisions justified by a very shallow and short-sighted rationale, resulting in tragedy. Humankind has been just barely able to contain the power of nuclear energy. We certainly will fall at the hands of AI

[@DFMoray]

What a beautiful woman

[@RamataKAhn]

Hey guys military ai weapons are a bad idea because... sci fi movies...

[@DrWaku]

I know right. We have seen some scenarios like this before

[@viktorianas]

AGI armies would be devastating in a world without nuclear weapons, now hear me out: I truly believe nukes are the most dangerous thing ever invented, but they already prevented maybe one or two world wars, same goes with AI, as long as AI does not get access to nukes - we are fine! Otherwise AI wars would be almost risk free and basically a country with most natural resources would win...

[@MarcoMugnatto]

"He doesn't have a very positive view of China". However, in practice, it's China that hasn't been involved in wars for decades, while the US has never stopped being involved in some war. In practice also, although several countries today have atomic bombs, the only country that actually used them was the US in Japan. So what is the basis for this 'special fear' that a person has 'of China'?

[@memegazer]

Sure if you only define war as boots on the ground invasion...but warfare as political philosophy china has been very agressive and is constantly testing international boundaries of other sovereign states.

[@DrWaku]

Not to mention the belt and road initiative. Basically entrapment...

[@MarcoMugnatto]

@@memegazer Do you really want to talk about market pressures? The US currently imposes embargoes on several countries. China imposes none. The US has always made investments in other countries as well, there's nothing wrong with that.

[@MarcoMugnatto]

@@DrWaku To my knowledge, China isn't imposing anything on anyone. It's not like the history of the Panama Canal, where the US imposed control over the region and the construction of the canal.

[@memegazer]

@@MarcoMugnatto I am not talking about market pressures. I am talking about how aggressive china is in the region and how they threaten other countries and deny those countries exist as anything but part of china.