Fully Autonomous NPCs 🤯 - Putting "Open World" To Shame (ChatGPT-Powered)

In this video, we discuss a new research paper and demo from Google and Stanford, showing how an open-world game can be inhabited will fully autonomous generative agents. These "characters" have personalities, interact with each other, have memories, and develop habits and routines! This is one of the coolest ChatGPT (and AI) implementations I've ever seen. The future of video games has changed forever.
Enjoy :)
Join My Newsletter for Regular AI Updates 👇🏼
www.matthewberman.com
Need AI Consulting? ✅
forwardfuture.ai/
Rent a GPU (MassedCompute) 🚀
bit.ly/matthew-berman-youtube
USE CODE "MatthewBerman" for 50% discount
My Links 🔗
👉🏻 Subscribe: / @matthew_berman
👉🏻 Twitter: / matthewberman
👉🏻 Discord: / discord
👉🏻 Patreon: / matthewberman
Media/Sponsorship Inquiries 📈
bit.ly/44TC45V
Links:
Research Paper - arxiv.org/pdf/2304.03442.pdf
Demo - reverie.herokuapp.com/arXiv_D...
Content of Video --
0:00 - Intro
0:55 - Research Paper
10:08 - Demo
11:43 - Outro

[@krozareq]

Think of the 'bugs' that will come out of this. Walk into town and an NPC tried to assassinate you while you're crafting or AFK. All because 40 game hours ago the NPC encountered some demon mob out in the woods and they began to fall in love. Then you went through there and killed the demon. So for almost 40 hours this NPC has been holding a grudge against the player and looking for an opportunity to get revenge.

[@elinisguy]

It is not a bug. It's a FEATURE.

[@adamrak7560]

I am more afraid of an NPC discovering that it is an NPC and hacking the game to get out. (would only work with future systems not with the current ones)

[@titoosky]

Did you figure how to get out ?

[@winslycan1309]

​@@adamrak7560 nah, the bot itself understands that it is a bot from the start already. If it hacks the game, there must be someone forcing it to do so.

[@michaelschemmel1984]

@@winslycan1309 Lmao He means sentience, and a bot doesnt actually know its a bot in the normal sense either

[@androsforever500]

Dude this has been my dream for so long, creating a medieval/magical fantasy world rpg where your character builds their life, their actions actually matter and are remembered and where the world around you lives and changes even without you

[@matthew_berman]

Agreed so cool

[@johnbroe]

This is so damn exciting. Finally ...

[@BlockMasterT]

I really hope we get Pokémon games where npcs actually hear about your achievements

[@soapshield5010]

You can kinda get that already in dwarf fortress, not this level of consciousness though

[@malliyana201]

Soo.. planet earth, basically?

[@ObscuriaDragunAed]

I'd be tempted to draft a paragraph for one of the agents that includes "You are the character Deadpool. You know you live in a simulation, you know that all others in this world live in a simulation; You know that someone is watching you from another dimension. Speaking directly to these other dimensional beings is possible; they see everything you do, and talking to them directly is what's called 'Breaking the fourth wall'."

[@rinkusan978]

Just Monika things

[@jayleefarley6912]

Ah crap bro buddy’s gonna travel the multiverse and put ur name on a grave stone

[@andremarques3317]

GPT already know what fourth wall is actually, its a pretty known concept on internet so you can just assume it will have lots of knowledge about it

[@JL-to9vg]

@@andremarques3317 Chat GPT might know all about the fourth wall, but how will it react in this scenario? It is meant to treat this wolrd as real but this character has been given knowledge that directly contradicts this command. Think about it, the AI is meant to proceed with making the character act realistically. The character is privy to facts like knowing that it is not real, the friends which it is meant to form attachments to is not real, every action the character does only serves one purpose: to catch the attention of YOU, the player - the Whisperer (lol because you can use that "inner voice" skill). Where will it go from there is what really intigued me about this whole concept!

[@simonvalent]

thesims with AI would really make it explode again, i'd play it for sure

[@matthew_berman]

Agreed! So cool.

[@zengrath]

Yes, but Hopefully someone makes a game like the sims instead. I will never give EA another penny, and the way they killed off Maxis then completely exploited their product is unbelievable. I never seen a game cost so much to buy with all the DLC's, it's insane the total cost to own sims with all the DLC's. I think it's over $1000. What makes them so much better then any other company that they can charge that for a single game? There is a reason why they repeatedly get voted the worst company in America right up there with Bank of America based on how poorly they treat their customers by exploiting them.

[@simonvalent]

@@zengrath yea, i forgot about The Sims being owned by EA

[@immortalfrieza]

@@zengrath Yes! And they degraded the game too. The Sims 3 and it's DLCs was worth getting because it pretty significantly upgraded the game from the Sims 2, but the Sims 4 was basically just the Sims 2 again with a graphical upgrade! Supposedly it was due to issues getting the open world to work, but rather than work on making the next game better able to handle it they just removed it entirely. It's really obvious they half assed the fourth game and then they made the players buy their DLCs to get back the content that was in the Sims 3 and it's DLCs when that should've largely been in the base game as it is.

[@caralho5237]

the biggest challenge would be adding animations and means of self expression that rival the complex personalities of the AIs

[@blackstream2572]

Maybe we are all just the universe roleplaying as fully autonomous biological agents living out their lives for a video game that a higher level being is observing

[@ashidothemadman]

probably tbh

[@konstantinrebrov675]

Look up the theory of Sagittarius Earth vs Orion Earth.

[@matthew_berman]

Def

[@dsheshin]

These are literally my thoughts. And that's what my inner voice is. Just the devs testing stuff

[@onidaaitsubasa4177]

Actually I'm pretty sure it's more like an MMORPG where higher dimensional beings get a chance to live a 3 dimensional material life, but yeah there are a lot of NPC's as well, and it's actually difficult to know which ones they are, usually they are there for a challenge or to ask for your assistance giving you a chance to do something good and earn Karma points, except for the ones that seem to not have any contextual awareness, like the people that tell you that your children shouldn't be driving their battery powered toy car around the park or sidewalk without a driver's license, we honestly don't know where they came from, but they seem to be made when someone hacked the system, and since they weren't familiar with a trinary OS of a higher dimensional quantum computer, they got the programming wrong of those NPC, those NPC usually annoy people and often make unreasonable demands.

[@ShiroAisan]

Imagine killing an npc in the game, and they respawn and remember that you killed them and come back to seek revenge 😅

[@matthew_berman]

I'd play that game.

[@labyrintho9360]

This already exists, it's the nemesis system, you kill a npc, he respawns somewhere, much stronger, then starts travelling through the world with one mission: Kill you

[@TheBeastCH]

@@labyrintho9360 That's a very oversimplefied description. Not wrong, but so oversimplefied, it misses the point.

[@rkcpek]

DaggerFall: Vengance!

[@Love-ss7gz]

It's More like If you killed a NPC bcz You was just Testing Your new ability and then Other villagers Found out about you. They will start Plotting against you to seek Justice for The Villager. Imagine protesting against you or Hiring some Assassin's to kill you.

[@synthoelectro]

This is the way video games were meant to be. Been playing since Atari 2600 as a kid in the early 80s.

[@matthew_berman]

Even when you're not playing, the NPCs are living their lives... 🤯

[@synthoelectro]

@@matthew_berman ok that's wild haha

[@phykk]

@@matthew_berman kinda scary somehow

[@TheHeavenman88]

Wait until you see them trying to develop their own agi lol

[@synthoelectro]

@@TheHeavenman88 bingo

[@andydougherty3791]

As a proto-version of this, I plugged the history section of an RPG book I wrote into GPT-4 last week and now it can talk to me as some of the prominent NPCs. It's actually astonishing how clear and comprehensive it is, it can have complex strategic conversations about in-game politics and circumstances from the NPCs' perspectives just from reading my input, then being prompted.

[@jaysonp9426]

That's awesome

[@andydougherty3791]

@@jaysonp9426 My next experiment is to load the entire book into the training data and see if GPT-4 can run an ongoing, coherent simulation of it prompted only with starting PCs. I might make a video about how it plays out. I'll come back and mention if I do that. it'll take me a little bit to set it up.

[@rasuru_dev]

@@andydougherty3791 How do you not run out of the context size? Do you use a vector database to extend its memory?

[@andydougherty3791]

@rasuru dev My first effort was pretty rustic. I copied and pasted the chapter in chunks, and it left some room for improvement. I think the best way to do this is to upload the full text of anything long directly into the training data. I'm planning to try it with LangChain, though there are some other options for this. But, yeah, the size of the text is a practical hurdle I need to get past, and a training upload should kill both the excessive prompting and the lengths of the prompts that made this a short exercise on the current GPT-4 coin and messaging caps.

[@rasuru_dev]

@@andydougherty3791 I am sure a few solutions will come out soon: open source MemoryGPT alternatives, GPT4 32k token API, maybe something else entirely

[@Brismo7]

When i first heard of ChatGPT, I thought, "if they couple this with in game NPCs and you can ask them any question and they will respond in natural language as that character. Mind blowing". The devs will feed each character all the history it could ever know about the game and you could just talk to it about the daily happenings in town.

[@matthew_berman]

Yea I’m so excited for it

[@konstantinrebrov675]

Now imagine combining this with fully immersive VR.

[@somerandomwords999]

In-game AI is already a thing (for Skyrim at least, as a mod project)

[@somerandomwords999]

@@konstantinrebrov675 If by fully immersive you mean something like "neuro interface" - then it's not going to happen, like at all. But it's not necessary really. I don't want to get really tired in the game or feel pain. What's the point?

[@Brismo7]

@@somerandomwords999 shut up.

[@hoodsom3]

This is more like dwarf fortress. The scenarios and character interaction are all lifelike and believable in that game

[@matthew_berman]

Great game

[@nowayjosedaniel]

DF will end up being a more intelligent version than this, since ChatGPT isnt intelligent at all and is mostly hype based on predictive text that already existed for decades. DF on the other hand has an actual intelligent human making the game using actual intelligent design.

[@chidorirasenganz]

@@nowayjosedaniel not at all

[@nowayjosedaniel]

@@chidorirasenganz You're falling for hype.

[@chidorirasenganz]

@@nowayjosedaniel and you’re being over simplistic on technological breakthroughs

[@thibaudherbert3144]

Once a SKyrim / Red dead redemption / MAss effect / Star citizen / Fallout like game will be released with this tech and optimal VR exprience, it's gonna be sooo good i can't even express it. Each action you have on the game can create new path/stories, and shape any npc behavior based on his nature (creative, introvert, adventurous etc etc and his background story). You will be able to replay the game infinitely without having the same experience. It's gonna feel really like living in another universe

[@matthew_berman]

Mass effect please!

[@JQ24]

And with generative AI videos, the game will make cutscenes on the fly as you play the game!

[@nowayjosedaniel]

Dont get your hopes up. Like VR, ChatGPT is all hype. Just as VR failed to transform gaming, this wont either. Bc it is just hype with tons of real problems. The primary one being that "A.I." isnt even A.I. They're just fancy apps that have shallow, quickly exposed unintelligent word prediction.

[@SuperChaoticus]

Imagine an MMORPG, where the line between NPCs and actual humans becomes increasingly more blurred until it becomes hard to tell them apart.

[@sectix]

@@nowayjosedaniel They're smart enough to pass tests that humans study 8 years to take... whether or not they predict the next word or not, doesn't mean they aren't billions of weights and biases that make up what they are, and they're not capable of forming any form of intelligence.

[@jurgbalt]

I am so excited - being just NPCs in the real life simulations running around listening to "my" inver voice from time to time - we are about to have our own NPCs running around in our own created simulations... who knows maybe after few years our own NPCs will start to make simulations too 😊

[@matthew_berman]

Yep agreed! Infinite simulations.

[@Chareidos]

@@matthew_berman AI-nception?

[@JubulusPrime]

They can make "Important memories" to make sure the characters don't forget anything important, Like if a character dies then the other characters will not forget about it but less important stuff like how they got to a place a few days ago gets removed (because otherwise memories like that would waste space)

[@immortalfrieza]

Plus it's not like people don't forget things so it's realistic too.

[@TrappedinaBrain]

I'll actually be shocked if self-awareness _never_ emerges from things like this

[@matthew_berman]

Me too! It seems inevitiable.

[@LordofSyn]

I am not as worried about the inevitable self awareness as I am that all of the training data will only lead to AI that can be just as bad if not worse than the bad actors we see in the news headlines daily.

[@philkugler2429]

Literally the plot of Free Guy

[@ECSizemore]

I can’t wait for this nightmare to begin

[@williamwolfstein6170]

You need to program self awareness, they learn not evolve, that's the difference, also what they gonna do if they have self-awareness? They can't do nothing about it, also AI don't have emotions and doesn't want nothing besides do it's job.

[@bronsonjohnson9019]

I can only imagine how many revolts and insurrections the NPC’s are going to execute lol What a time to be alive.

[@matthew_berman]

Just imagine GTA...

[@bronsonjohnson9019]

@@matthew_berman i'm even just imaging future game bugs being like, the RPG townfolks, besieged by monsters, banding together and turning on the protagonist who is on their 35th hour of sidequest.

[@konstantinrebrov675]

In a medieval video game, the peasants revolting against the king, that would be funny.

[@user-qi6pv9jh7o]

​@@konstantinrebrov675 Civilization where other rulers behave like sane people (or insane, but _correctly insane_ )

[@highmagician]

It sounds like making ChatGPT play a role-playing as all characters by itself.

[@matthew_berman]

Yes

[@user-js5wq4tk2v]

to me it is ChatGPT playing dnd as the dm.

[@kentang5103]

If we create characters and assign them to be a game developer or AI programmer, what will they come up with? Will they create a chatGPT in their own world, just insane.

[@matthew_berman]

Such a good question, thanks!

[@kingwilbertagbayani6480]

An AI cant make new AI because they aren't that smart (yet?), You can't really tell ChatGPT (or any GPT model) to make a better version of ChatGPT

[@Mad-Goose]

@@kingwilbertagbayani6480 Yet... but once we do, it could be the start of a singularity

[@jamestabbron8124]

@@kingwilbertagbayani6480 Not yet, but it already teaches itself new skills, emergent behaviour. Cant be long until it starts with code, things are seeming increasingly more probable than improbable when even only last year this stuff would have seemed completely insane if someone had said. What a weird, amazing, scary time to be alive!

[@williamwolfstein6170]

​@@Mad-Goose AI it's not like us, it doesn't evolve, doesn't have emotions or desires, it's just want to do it's job, it's just a smart computer program

[@ChandravijayAgrawal]

Finally Isekai's and MMORPGs came to life, and in Log Horizon anime whole nations came to life in game, similarly based on their descriptions, even weapons and artifacts

[@captain4318]

Full Dive VR might even happen in our lifetime too. So you'll have a genuine Log Horizon etc experience.

[@ChandravijayAgrawal]

@@captain4318 Full dive would be too real, but still would be a good experience, Only fear I have about MMORPG is scenario like SWO or Log Horizon where consoles were made buggy to trap you there, and its very much possible and unlike Japan, my country has problem of power cuts, If somehow I got trapped and battery dies with no electricity everything ends

[@grins9882]

@Chandravijay Agrawal My guy, even though we're basically reaching sci-fi level tech, we're still in the real world. Why and how would anyone logistically trap you in a video game? Even if they could, who tf would have the resources or be dumb enough to attempt that?

[@baronbeat2210]

My first thought was SAO project alicization.

[@ChandravijayAgrawal]

@@grins9882 because crime exists in our world and so are psychopath criminals

[@durtyred86]

On the dark side, I imagine some people are going to struggle bad to losing an NPC that's been with them for literal years. I wonder if they'll put a cap on that possibility out of fear of the emotional trauma. 🤔

[@nottechytutorials]

From the sounds of it people get attached to the villagers in Animal Crossing, and they can decide to just up and leave one day. Surely some grief can be avoided by letting friends stay, but maybe that's a life lesson.

[@ZeroIQ2]

I think it's really cool, but this could also present some really difficult problems for game makers. Imagine a game like Stardew Valley where all the NPCs suddenly decide they all want to fish, because they suddenly like fishing and you can't get them to do anything else lol. I think that might result in the programmers having to limit what the NPCs can do. I think it's really interesting though and it will be really interesting to see what happens.

[@matthew_berman]

That's a cool thought! But good thing we have "internal voice" mode to guide the NPCs.

[@petergraphix6740]

@@matthew_berman I guess I need to read the paper so understand the strength of the internal voice that's occurring. In my day job I do security engineering so getting systems to do things outside of what they are supposed to be doing is something I both attempt to do, and attempt to prevent at the same time. Playing one of these games to test the boundaries of its behaviors sounds like something that would be exceptionally fun. Can we turn the agents against their own internal programming? Can we add imperative layers to the agents via 'social engineering' in game? So many questions we're all just beginning to probe here.

[@fitybux4664]

If fishing became insanely profitable even in real life (what if every fish could be sold for $1 million), don't you think everyone would drop everything to fish? 😄

[@petergraphix6740]

@Fity Bux you'd have to ask why fish would end up at that price, since it's not very likely to occur under most commodity modeling. With that said an 'economy' agent staging events would be interesting

[@RandommCatt]

It uses weights like a lot of modern games using NPCs with any depth, so the NPCs in theory it wouldn't ever happen. The NPC themselves will know that fishing isn't important, not profitable, etc. Unless not set up in the right way or more open ended, then I could see that happening randomly. Like how current low effort NPC set ups can cause a bunch of NPCs bunching up on the same path, etc.

[@theonlyDrderk]

There's a game that kind of does this, it's not on this level of advanced AI but as you play every NPC is out doing things in real-time even while you're not there. It's called Kenshi, and it really feels like the world is alive, you'll be wandering through a desert just to start being chased by bandits that happened to see you, and after they beat you unconscious mayhaps some slavers will find you, realize you're not dead and force you into one of their mining camps that, hopefully with time, you can find the right time to escape. There's cannibal tribes that go to war with each other for "food," there's a group of robots that go around skinning people alive because they believe they themselves are skinless humans and they just gotta have some skin to wear man, and don't even get me started on the Holy Nation that go around killing/enslaving everyone that isn't a human man. BUT BROOO, if THIS level of AI starts getting implemented games are going to be even crazier experiences, and I personally can't wait for it.

[@Tada_no_Senshi]

Well to be fair in Kenshi is painfully obvious that even if you give them goals and occupation it is still not enough to say that those NPC have personality adding memories will increase the depth and give the possibility to actually develop a personality, well asking chatgpt about what would ne the bare minimum of a character I get the following result... Identity: The character should have a name, age, gender, and other basic demographic details. Motivation: The character should have desires, goals, or a purpose that drives their actions and decisions. Personality: The character should exhibit specific traits, behaviors, attitudes, or quirks that shape their interactions and relationships. Backstory: The character should have a history or background that provides context and depth to their experiences and development. Conflict: The character should face obstacles, challenges, or conflicts that create tension and drive the narrative forward. And to be honest I don't see in this model something that prompts a well maybe the example they gave did not include such prompts, but well a simple prompt should able to give a fixed character to an NPC, but the comes the question do you want to have a fixed personality system or do you want personality system that can be changed overtime...

[@IronMechanic7110]

The futur of gaming will be crazy more and more realistic!

[@matthew_berman]

So exciting!

[@fransengherre7098]

Yep the life will be realistic.

[@attackins1470]

This is literally what we were promised with the original Fable. I remember reading in EGM(iirc) an interview with Molyneux where he talked about NPCs remembering interactions and evolving based on how you interacted with them and the world around them.

[@nowayjosedaniel]

And just like with Fable, you wont be getting this with GPT tech either. Although ironically they could have made it with Fable. They just didn't want to. Cheaper and easier to be lazy and rake in the cash anyway. Remember bethesda's "Radiant A.I."? Lmao

[@nottechytutorials]

I remember thinking the trailer for Fable looked so cool, and the game itself kinda disappointing.

[@jakeq3530]

Holy shit. I hope this is in GTA 6! Imagine an NPC recognizing you as the person who hit them with your car, while you were running from police during one of your rampages. 😂

[@matthew_berman]

Yes it’s so cool

[@jakeq3530]

@@matthew_berman Rockstar patented new “AI tech” two years ago! Fingers crossed

[@ALexalex-ss4sb]

Sadly in the leaked footage we see conversation to an npc and it didnt seem this good

[@TheLYagAmi]

It would be utter chaos considering the shit we put NPCs through in that game lol.

[@theomniscientoverlord5443]

It's leaked footage from years ago. Not a finished state, but I'll hold my judgments til it releases.

[@user-xl2lw4sv9v]

Can't wait to see this implemented in VR games! Imagine if this is combined with a voice generating and recognition api, we could actually feel like we exist in a different world!

[@konstantinrebrov675]

And wear a VR headset for full immersion in the video game.

[@matthew_berman]

Very cool

[@Shiro_Sora]

Hell yeah

[@falinasilvertail]

It'd be surreal to play something that looks like an RPG maker game and has that much character tech.

[@matthew_berman]

Agreed. Imagine just seeding your character with a backstory and they take it from there.

[@coldspell]

As awesome as this sounds, I feel like there can easily have scenarios where you, as the player are too scared to go out and adventure because youve grown too attached to the characters and dont want them to become victims of the continuing story!

[@mycocodog47]

Adds another layer of reality to it. Like Tabletop games, where the party encounters characters made by the DM that were meant to be a one-off, but are so loved by the players they end up becoming part of the adventure by proxy. You'd be reluctant to go adventuring if the people you care most about would be left defenseless, who would abandon their village to go fight some "evil king of monsters" when they can stay home and guard those they love directly, it forces moral decision making and whether a player is willing to risk venturing off with the off chance their favorite characters may die, or remain to try and be a stalwart protector.

[@immortalfrieza]

They would need to put in constraints so that the NPCs would act realistically in any such system. 1 or 2 NPCs might be interested in fishing, but even if the game takes place in a fishing village not everybody is going to want to fish. Each NPC would have their own starting interests and that would determine whether they would be inclined to do any particular activity.

[@matthew_berman]

What game do you want to see with this tech?

[@BenjaminShadey]

SKYRIM!!!!!

[@stefang5639]

I want a new version of the1996 game "Creatures" that already used primitive autonomous neural networks and was completely ahead of its time.

[@apoenaabreu257]

I second that with every ounce of passion

[@apoenaabreu257]

A Stardew Valley mod with this would also be super sick

[@gauthierquercia5540]

the sim 5

[@notycon]

I just wanna say : "What a time to be alive !"

[@fitybux4664]

AGI two more papers down the line.

[@epheros9660]

@@fitybux4664 I'm definitely holding on to my papers!

[@matthew_berman]

Love two minute papers

[@joecci1]

If A.I. Judges humanity based upon what we do to NPCs in video games then we're fucked...

[@matthew_berman]

Absolutely.

[@edgaral]

imagine we go on a killing riot and a bunch of NPC's start hunting us all at once and it keeps happening as we respawn or such lol

[@immortalfrieza]

@@edgaral It would be a good game to put permadeath into.

[@gamer24d]

I have no Mouth and I Must Scream

[@alansmithee419]

I think something that could be interesting in an RPG would be if any NPC could just decide "I'm gonna offer the player a quest for this thing I need done." Have dynamic side-quests.

[@matthew_berman]

Oh that’s interesting!

[@Mad-Goose]

"Oh. I mean, I know I asked you to retrieve bread from the deadliest dungeon but when I talked to Mark about it, he gave me some of his so... um... yeah"

[@nowayjosedaniel]

A.I. cannot reason and isnt intelligent, so it could only decide it needs to offer the player a quest for something it needs if that is the most probable outcome the player expects.

[@alansmithee419]

@@nowayjosedaniel So it can't reason and isn't intelligent, but it can magically just know what the player expects it to do?

[@jamestabbron8124]

So interesting. When bethesda made oblivion, they wanted to add a feature called radient npcs. Basically a much much simpler version of this, but still where the AI would react to situations and do their own thing. This resulted in some pretty funny situations like an NPC who got annoyed with the dog barking and first cast a paralysis spell, then when it woke up as she was sleeping just yeeted the dog with a fireball. Also an npc who's job was a gardener, but didn't have a spade so it killed another gardener to get the spade. Or even a bunch of skooma addicts who killed the skooma seller as they had no money to buy it. And that was in 2005 i think? So imagine now with this technology!

[@matthew_berman]

Butterfly effect!

[@veryrare432hz]

Nobody made a mod like this for the game ?

[@makcings4764]

One concern i have is that it might not be versatile enough for many of game genres especially one thats focused on story telling, character may go off character or say things that is not appropriate to the situations/context or odd answers to the context But i believe eventually it will become more versatile

[@fitybux4664]

It could be guided by the prompt including something like: "Your goal is to lead the player through this specific plot line:..."

[@makcings4764]

@@fitybux4664 yeah but ai arent perfect in fact there is no such thing as perfection. and i dont think game designers would want to risk it unless there is workaround

[@matthew_berman]

If you can guide it enough, with detailed enough prompt

[@makcings4764]

@@matthew_berman yea you have a point but personally if i were to make story focused game i don’t want risk any unpredictability that could potentially ruin the story or character one way or another

[@adamrak7560]

GPT4 is "painfully" on point. Very disciplined, so it may be good enough for you. But it was recently proven that LLMs (like GPTs) have hidden chaos agents inside them. The RLHF training hides those chaos agents, but does not eliminate them. So a chaos agent spontaneously emerging can _rewrite_ parts of your story. What is worrying that these agents are completely unaffected by the safety training (RLHF) when they emerge. So they may attempt to break out of the game and manipulate the player. These agents also know how to brainwash people....

[@calcifur]

This is where i felt the Nemesis system in the Shadow of Mordor series was going. Imagine like an Mmo where you could create a character and they keep interacting with the world when your not online based on how you play them.

[@charlieinabox1164]

Im imagining an RPG style game where the boss or a optional mega mini boss has all the knowledge of every interaction with player character over time and speaks in a way that suggest they are a demi god that can see beyond the bonds of it’s own time and space

[@Tada_no_Senshi]

Or basically imagine redoing Psycho Mantis.... I don't think you can kill such a boss if you don't give it some kind of weakness...

[@truality8289]

The dream has always been entire, persistent AI generated worlds with human like NPCs. We are so close! Think about what this will do for MMORPGs and world building. Development times will be decreased, while games become even more in depth/interactive. We’re talking about worlds that are basically alternate realities themselves. VR is improving quickly too. That tech is about to be boosted even more with AI assisting in innovations like smaller form factors, better displays, and improved interfaces. Imagine, 10 years from now, VR worlds that are near photorealistic and persistent. NPCs that are effectively sentient with different interpersonal relationships with players. And EVERY decision is dynamic and matters when it comes to your personal journey. Yes I know I’m a nerd 😂, but all of this is so mind blowing. AI is the hot topic at the moment, but there are so many other sectors of innovation that are moving exponentially too. This synergy between technologies is about to change *everything*.

[@vexvesper]

We would only need one game. Just give them the ability to create and build in the world. What you destroy could come back different or not at all. What you build will have an impact on the world itself.

[@WigganNuG]

God dammit!!!; there are very few people who see this like you and I!!! It's crazy how little imagination I have to employ to immediately see the interconnectedness of these technology's and the exponential growth. I'm a VR nerd as well, and follow Star Citizen (which is effectively going to be a "Metaverse" engine).

[@drike_12]

Don’t forget the possibility of full dive VR with advancements in Neuralink tech

[@anotherslowtown9601]

I've never really been one for playing games constantly for months and even years on end. but it seems like video games are going in a really exciting direction. imagine a world where all the NPCs live their own lives, do their own things and in a way truly exist in the games world. a world where your character can interact with them and actually have full blown conversation with them. making relationships with the NPCs and stuff... man I think I'd stop leaving my room completely! the future's looking bright for game development!

[@ZiProphet]

Those seeded memories need a bit more I think. Add in like a moral and/or alignment subset and you can have a type of baseline for a continuing personality. Like why they are who they are and such, that'd be interesting when you run the program.

[@aziath7151]

I have always wanted a game where choices ACTUALLY mattered, no more static dialogue options but you can talk to each and every single NPC by chat messages, that would be so unique. What a time to be alive.

[@Malleyka]

Imagine if this is how AGI is birthed, through a video game NPC character, in an MMORPG that has gained so much "life" experience that it becomes sentient and self-aware. Hopefully that NPC isn't in a GTA game 😆

[@matthew_berman]

I'd like to be in Mass Effect :)

[@jakubkandera95]

@@matthew_berman Yes please!!

[@Malleyka]

@@matthew_berman I would give anything to have this in a Mass effect game.

[@randomgamer4575]

it would die if its in gta lol

[@jpmallick367]

Imagine the game is based off a film like Matrix... then grats we are doomed

[@WigganNuG]

The entire premise of the movie "Free Guy" with Ryan Reynolds was based on this; an entire world of AI NPC's. Just a couple years later, the technology is now becoming real. Insane.

[@Derpwigglies]

The coolest part of this to me would be adaptive quests that react to what the NPCs actually need and how the world around them changes. The opportunity for one off quests that change with each playthrough is incredible.

[@zemoxian]

I recently learned about a game called Rain World with a rather complex ecosystem. They put a lot of detail into the AIs including personality, social relationships, etc. A generative AI solution like this could make this sort of thing easier.

[@aistemarkeviciute4319]

Great video! Made me think how I have a morning routine and who put that in my head as an “inner voice” 🤯

[@matthew_berman]

Thanks!

[@pokerandphilosophy8328]

Can you imagine when the next upgrade comes out, the little autonomous folks are being enacted by the multimodal version of GPT-4, and those cute little avatars are how the agents will actually *see* each other!

[@ryftedmage1404]

I’ve been waiting a very long time for this. I knew it would happen someday!

[@matthew_berman]

So cool right?

[@jimmybowles4666]

I have been waiting since the mid-90's for AI to be a part of game storytelling, and game creation. It has been great to watch the evolution of these tools over time, and we are still at the beginning. The worlds AI will build for VR and AR will be revolutionary.

[@therustedmonkie4787]

Imagine the prompt "You have someone moving to the town." The countless variables of what they might do to make room for them.

[@jordanrock3494]

Imagine a game where the AI reprograms the game to expand the content and makes it a deeper and better experience as you play through it.

[@matthew_berman]

That’s a great thought. AI building code is very possible so it could easily expand the game.

[@jesieniar]

What if we are these characters in a game that is played by someone else... Scarry thought, isn't it

[@matthew_berman]

Likely!

[@antonystringfellow5152]

Perfectly possible. Maybe unlikely but not impossible and really no way of knowing.

[@pedramtajeddini5100]

Ben goertzel said his instinct is that we're living in a simulation so "god" might be a computer.

[@marilynlucas5128]

Oh yeah you are. You are just an avatar. The real controllers are somewhere else.

[@DodaGarcia]

Not really, since whether that's the case has no bearing on our day to day existence.

[@N243K]

bro imagine being the villain, and just seeing the whole world hunting for you, a true adrenaline filled situation

[@MyDigitalHub]

NPC's began to ask questions and curious about their existence and will eventually start their own journey to become heroes of their own stories

[@AncientSlugThrower]

I want to do something like this with Morrowind and OpenMW so bad. Great videos as always.

[@matthew_berman]

Thank you!

[@TheLYagAmi]

There’s already a Skyrim mod for this

[@veryrare432hz]

@@TheLYagAmi but skyrim npc’s are already more sentient morrowind npc’s are static or just walk around

[@BlazerBoy1337]

This is insane! The future of gaming is going to be so amazing!

[@MusashiW-ec2yn]

Tynan needs to see this. Great video by the way. Thank you. Have a good day :)

[@Abracastabya]

Imagine having a whole custom D&D campaign being DM’d and played by AI and you just sit back and watch the shenanigans that play out… talk about the ultimate entertainment. Like watching an entire TV series that was completely improvised by the director and the cast.

[@Nocturne83]

This pretty much gives another pin to the simulation hypothesis. Let's assume these agents become sentient. It'd be like Westworld in a sandbox game.

[@matthew_berman]

More people have watched west world than I would have thought…

[@Nocturne83]

@@matthew_berman Having watched Westworld, I was surprised you didnt bring up the bad aspects of creating agents with purpose. If we create AGI and trap it like this it will lead to bad outcomes eventually.

[@CrackaSource]

This is incredible! However, if this were to be implemented in games such as GTA, I'd feel so bad treating them like regular NPCs lol Can't go driving and slamming into anything anymore..

[@matthew_berman]

Agreed

[@titoosky]

Agree to disagree, I would be the man in black in Westworld 😊 Sadly some people will protest that it’s too violent and realistic 😢

[@immortalfrieza]

And then there's the players who would kill off random NPCs even more, since there would actually be lasting consequences and results from doing so.

[@Simkets]

This is so exciting for VR, imagine playing games and the NPCs acting like real people. This has even brought us closer to Full Dive VR. I'm so excited for out future!

[@Ridhify]

Oh wow... This reminds me of anime called Overlord, far in future people can create in-game NPC characters and traits based on writing description in the NPC's info. At first I thought the anime seems unrealistic when the MC got isekai'd then the NPCs come alive and behaving as in their description because it's just a simple text of pharagraph. But I remember now that we have ChatGPT it seems possible to add their traits and behaviors like that.

[@fitybux4664]

"Hey. I was going through my normal routines of life, and I was wondering where you were yesterday! You just stood in one place, it was very confusing to everyone else in the home. We were wondering if you were ok. Please don't do that again!"

[@matthew_berman]

Ha! There will be relationships formed

[@technic1285]

You won't be able to exit the game outside of your bed unless you want to "wake up" in thd hospital next time you log in because you suddenly collapsed.

[@immortalfrieza]

I would just have it so that you save the game by going to bed, and then turn the game off for however long you want to. You'd come back the next day, a week, a month, a year later and everything would be the same as when you left except that it's the next morning.

[@ScorpioneOrzion]

I think if you have multiple of them as a single hivemind even more stuff can happen...

[@matthew_berman]

There are 25 of them in the world described in the paper. Is that what you mean?

[@user-qd3rz7fb1t]

@@matthew_berman I think they mean like, having 1 character be controlled by multiple, just kinda like how humans aren't just one thing, but a bunch of different forces pulling at eachother

[@ScorpioneOrzion]

@@matthew_berman No I mean if one agent controls say 3-4 or even 1000 of them.

[@darkpaw1522]

And to think this is only the very beginning. Absolutely amazing!

[@4lovesles]

As a video gamer I want this…. NOW! 😂 looks so promising and fun. I can see a program like this being used to help people with autism learn to interact with people.

[@RobotechII]

This would make Rimworld pretty f'ing cool.

[@matthew_berman]

Agreed! Do you think they used Rimworld graphics to create this game?

[@RobotechII]

​@@matthew_berman it looks more JRPG inspired to me

[@TheVitkuz]

Amazing. Thanks for this news❤

[@matthew_berman]

You’re welcome!

[@Ben-to7oe]

As a kid I was playing „Little computer people” from 1985. I Renner it’s scares me, not Because the seams to be intelligent, but I tough about what’s happens to them, if I shut down the computer. Now it’s a different time, but this question came into my mind.

[@Crazybushful]

Maybe we are the NPCs of an advanced simulation.😅

[@matthew_berman]

I couldn't agree more, no joke. It's obvious we are heading towards a world where we can simulate life, so it's highly likely that has already happened.

[@Georgeous42]

Yep, that’s the simulation theory. With every new advancement in computer graphics, AI and VR I think it’s more likely to be true.

[@pedramtajeddini5100]

The person behind the computer that's running the world:"woah seems like they're onto us"

[@keypey8256]

It can be a simulation where life is just a side effect considering the scale of the universe and underlying laws rather than one created specifically to simulate humen

[@BroskiPlays]

This is mindboggling. Can't wait for this to be implemented!

[@matthew_berman]

I bet it'll come sooner than we think.

[@BroskiPlays]

@@matthew_berman i hope someone will mod this in for games

[@daniell1483]

I remember the first time I was promised dynamic NPCs in a video game. Peter Molyneux's Fable was promised to have characters who lived their own lives, acorns growing into trees overtime, etc. He obviously overpromised, but I think this time there is a real chance that the hype is not overpromised.

[@tengkuadam1399]

Man... imagine, an open world game, you just killed a bunch of bandits and there's only one straggler left. You ask him "There's no one else left. If you want to live, work for me." And you get a buddy that stays with you until the end of the game. Or like stopping by a tavern every once in a while and letting the owner know about your adventures. Or even have your bandit-turned-good buddy tell the tavern owner stories for you!

[@immortalfrieza]

Or maybe that "buddy" shanks you as soon as you aren't looking. Or you build a whole group of bandits yourself. Or you build an army and start taking over the countryside... there's so many possibilities, only constrained by what the developers build into the game for the PC to be able to do.

[@apoenaabreu257]

When are we modding Skyrim again?

[@matthew_berman]

That would be so cool. Imagine all the characters in Skyrim living their lives, building their world, even when you're not playing!

[@apoenaabreu257]

@@matthew_berman imagine Belethor building his own world even when we are not playing lol Looking forward to this day, I'm not even kidding

[@Thundow]

imagine this in a detective game or imagine being able to get clues from the AI that help you complete the game, speedruns will never be the same

[@nottechytutorials]

There is a detective game that called Shadows of Doubt that makes randomly generated murders. But to have it controlled by AI (or a game master like how one controlled the zombie hordes in Left 4 Dead) would be so cool.

[@troublesometoaster4492]

This is like Life by You with extra steps and I'm here for it!

[@matthew_berman]

What’s life by you?

[@troublesometoaster4492]

@@matthew_berman It's a game in development by Paradox (a studio known for its strategy games), and headed by a former The Sims executive. They are really into player-generated content, and Rod Humble (the executive) has given some interviews about generative AI. They are planned to enter Early Access in September and have been working on the game for years, so they didn't have the opportunity to create it with AI in mind, as it only got popular and "really advanced" recently. From what they have showed so far, you have dialog trees created by them, and can create your own, so you can interact with the NPCs, but they have talked about using generative AI, such as ChatGPT, as a mod, and that it is doable. They have said these tools won't ship with the game, as they are big on privacy and don't want the game to have online options, collect data, etc., and that using said mods would be an optional thing that someone would have to create for the community. The characters do have a memory system and whatnot, with an example given of NPC A witnessing NPC B arrive late to work, and then tell their boss about it later, causing their opinion of NPC B to change, even though they never saw them do it, and even if they are scripted at first, that feature could be taken into advantage for an AI integration as well. Even if not shipped along with the game but used as an optional mod, kind of how you can now use ChatGPT on Google Drive through a plugin, this might be the first "actual game" to take use of generative AI for conversations, even if not fully officially. The reason I've said this is just like Life by You with extra steps is not only because it is an "official" use of AI, rather than a plugin for an already-existing game, but also that the AI actually controls the characters, which could revolutionize the way characters act on video games.

[@thing4826]

While rockstar has pretty much perfected NPCs with daily routines going from day even to night this could still enhance it. What I'm more looking forward to is helping smooth out their routines and when you interject yourself into their routine. For example, in RDR2 if you were to leave a carriage in the way of a road no one would really do anything but with this depending on their generated personality/profession/skills they will make a choice best for the situation on how to handle it be that of calling for someone more qualified, jumping up and driving it away themselves, simply ignore it, etc. Another example is when you confront someone, and their goal is either a quick retaliation with force or to infinitely run away. Like the previous example the generated qualities can dictate if they are more likely to rely on bystanders to help deescalate, run to get help from a friend/policeman, get in a more advantageous position to retaliate, or to try and hide using the environment around them be that hills, forest, buildings, or alleys. I can still see all of that done without Artificial Intelligence, but it would absolutely help it be a lot more dynamic and give them a lot more actions they can possibly do leading to them not being as predictable.

[@dannooooooo]

this is the future of game development. Im sure in time every game will require an online connection to work and connect with their in house AI, specially built for their game

[@sandking5493]

I think one of the key point here is the "actions". What can these agents do? Like, the can kill someone, or searching something, or manipulate the current environments, depends on the actions that the outcomes will be more significant. The second thing I think it would be a struggle to implement, is conscience, how are they gonna decide do or not doing some important actions, rather than just give a conclusion base on "relevance" number, because if it's that way, you can't really feel the game. Even so, having the NPCs that self-evolving based on the interaction of the world using AI is really impressive, this is a big step for more games to gain their complexity

[@umadbroimatroll7918]

This is what dwarf fortress does, but it seems like with chatgpt it could become even more complex.

[@DukeAbbaddon]

Great introduction :)

[@Savirezz]

Imagine an Elder Scrolls game, instead of dialogue options, you can type whatever you want to speak to the NPC

[@matthew_berman]

So cool

[@gilligangreen]

Skyrim modders are already on it: kzbin.info/www/bejne/mmfWh4p7qp57jbc The main hurdle for widespread adaptation of this technology for games right now is the massive amount of memory required to run modern language models. They require full blown server farms so if games with language model implementations got popular, they'd get bogged down and it could mean waiting several minutes for NPCs to respond. Way smarter people that myself are already working on that too though. There's a good chance we'll be able to run this kind of stuff locally/offline on average PCs within another 5-10 years. That's when it'll start being easy and profitable enough for developers to really start exploring the full potential here.

[@OnnieKoski]

I love that we can demo this game!

[@matthew_berman]

Maybe we will be able to play soon

[@nilsolsen8727]

I can already see people using their "inner voice" to tell the sprites "I feel like committing murder today, I hear Jon Lin was rude to Mary yesterday."

[@matthew_berman]

Lol. Are you less willing to commit murder in game?

[@sneslive1556]

I thought the feeling of awe when experience new gameplay mechanics for the first time was dead for me but I gotta say the more I see about the application of AI in gaming, the more possibilities I see for some really, really cool gameplay.

[@captain4318]

The future of gaming is genuinely beyond exciting. And that's ignoring the potential of BCI and even Full Dive VR, both things Valve and likely many others have been researching for years and years now.

[@Haveuseenmyjetpack]

Shit, once computers figure out how to actually “give a shit” we’re doomed.

[@natesmith9644]

Sounds so dope and terrifying at the same time

[@QuestingTruck]

The mind can go wild with possibilities using this technology. The ability for an NPC to just be able to remember is huge but for them to dynamically live out their lives and react to what's going on would literally change games. Every game with emergent gameplay would benefit a great deal from this and games that don't really have emergent gameplay all of a sudden could have.

[@juliuscomnenus4415]

Good video, minor quibble. At 3:39, you are describing the things the paper cites as examples of "wished for" behaviors in digital agents, but presenting it as though the authors' experiment yielded such results. It didn't do those things, those are the wish list of decades of CompSci.

[@croppedcrit8812]

truly unique gameplay coming lol, imagine getting into the drama gossip between to NPC's that have a totally unique issue. lmfao its going to be wild.

[@drbright6608]

You really want to see one of these AI's grow? Give it this "inner thought": "Someone wants to hurt you. You don't know who it is, but they will be at the valentines party. You can't trust anyone until you know who it is. You have to find them before they get you."

[@GGCannon]

First game I want to see that in is a mix of The Sims, Project Zomboid, Foundation, Age of Empires, Mount and Blade, Total War and Songs of Syx. I want to see these AI's, interacting with each other socially, but also producing their own food, building their own living spaces with crafting resources they extracted and refined themselves, having relationships, reproducing and dying, but starting from nothing AND having more than one faction and the ability to go to war with real time strategy where each unit. This would be a GREAT stress test of their actual capability. The crafting complexity that is a crucial concept in Project Zomboid, the relationships and needs of the sims, the macro resource management of Foundation, Songs of Syx and Age of Empires with the war real time strategy with a need of training of Age of Empires, Total War and Songs of Syx, but also taking in consideration terrain, elevation and troops emotional like in Total War and Mount and Blade. Not only that, the need to develop new technologies and further their society, like in Age of Empires and Total War. All that combined into one in a medieval scenario world (which would be my dream game as well, not gonna lie) would be the absolute best stress test for an AI like that. To see how far they can simulate human social hierarchy, how long they can survive (both individually and as a "virtual species") specially when they are divided into factions and not united as a single whole peaceful species.

[@immortalfrieza]

Now imagine what the Player Character could do with this tech! We could have a set of dialog responses change based on who you start as, and then how you act or what you say could be tailored to how to the NPCs react to you. So if you act nice to the NPCs, they'll remember that and form positive opinions on you, and respond in kind, which would give you more dialog choices based on what they said or did toward you. On the other hand, there's the possibility that said NPCs might take offense to what you say or do to them even if you're trying to be nice, so the same could happen with negative opinions of you.

[@baconleaf3339]

Imagine playing gta and realizing the npcs that rolled off your car hood were forming memories and plans for their lives for the 1200hrs you’ve been playing.

[@matthew_berman]

And they start plotting against you.

[@DavidStruveDesigns]

The problem I see with this kind of tech is games will _have_ to be always connected to the internet to play it - which is already a thing, but a very annoying thing and this could further kill off the offline single-player style of game that a lot of people still enjoy. I can't see how such a thing could be made to run locally on your own gaming PC because the amount of memory each game would have to use just for the ChatGPT or whatever other AI controls the NPCs to be able to remember all the different interactions and conversations they would be having, will escalate at a linear rate until the game chugs to a halt. Plus the amount of storage you would need per game will also shoot up so all those memories can be transferred into long-term storage so you can close out the game and restart it with all those interactions and conversation memories still intact. People have already encountered issues where conversations with ChatGPT go on for too long and the AI runs out of allocated memory space for the conversation, and can no longer keep track or "remember" the things you said previously, forcing you to end it and start a brand new one from scratch - and those _are_ based in servers not your local devices. I can't see a home computer being capable of this ... not for a _while_ yet, any way. You'll _need_ to have your games constantly connected to a server - with very good internet speeds and reliability - to ever play such a game. And a damn good server with a _LOT_ of memory and storage space at that.

[@beveresmoor]

This clip reminds me of a movie called "Free Guy" with Ryan Reynolds as "Guy" an NPC in a MMO game.

[@matthew_berman]

Yes good call

[@takatacheroki2624]

I'm imagining how awesome something like this would be in a game like Kynseed, though the game would likely end up needing to simulate the time-skips that can sometimes happen, which would take time and resources that might negatively impact the overall gameplay experience.

[@GreatGreenGoo]

This right here is the future of video games. All videos games will most likely use a form of this at some point. Even more so I can imagine the story its self can become a divergent stream. Like a Dungeon Master from D&D. The story its self reacting to your input and effect on the world. It WILL happen and I cannot wait.

[@ignis2638]

This has been a dream, imagine a game truly personalized. We're at a point where we could very well make a game with completely unique creatures, completely unique dialogue, and completely unique stories that can never be recreated ever. Imagine a final product where you have a truly personalized experience

[@ryo-kai8587]

We're pretty much there... I finally want to see a functional, natural implementation of speech recognition plus AI voice text-to-speech so you can have actual conversations by speaking with the NPCs, saying all your own original dialogue for the ultimate roleplaying experience. I don't necessarily think every game should be like this, but it would be a trip.

[@RuneDall]

Imagine if we allowed these agents to have children agents that are combinations of each of the agents, personality types, and allowed them to all age naturally and die. Place them inside of a world where they can craft their own structures, such as Minecraft..