Modbox OpenAI-GPT3 / Replica AI Scripting

[@MrBenderrrr]

Call me paranoid but i think Bob is not actually working at the city hall

[@mattizzle81]

Janitor man, I'm pretty sure. Don't be so paranoid.

[@Reido50]

He's so aggressive. I feel like he's attacking me with every word

[@galaxycoffee933]

What do you mean? he's right on his way to City Hall at City Hall Street Location!

[@egileigengrau6637]

Nah, pretty sure he's one of those IT- guy who can't get fired or else the whole place breaks down.

[@randyx007]

Are you the real Bender Bending Rodriguez?

[@CallmeCram]

Can't wait for this to be commonplace in open-world games

[@Freak_Gamer]

It will be a while. For starters, it does not have the level of polish that most AAA (and even AA) developers would find acceptable, but more importantly, the AI is actually closed-source (Its called OpenAI only in name lol), so developers cant actually use it except on rare cases.

[@nickfury1507]

@@Freak_Gamer Yes, "a while" meaning it could be 5 years before a major developer produces a game with major improvements on this demo the likes of which players will find sufficiently convincing. It might not take that long, there are plenty of big companies which see the value of this technology, and these guys aren't the only ones working on projects like this.

[@Kacky]

@@nickfury1507 they'd need to develop their own fork of the ai that adheres to their game's lore as well. lots of variables but it would be pretty badass!

[@jesselindsey9760]

Give it 20 or 25 years.

[@DeathSalsa]

Yes, I can't wait to be shouted at by autistic hippies with faux-British accents who somehow work in local government

[@Razumen]

This is the future of RPGs, it has to be. Can't wait until the processing power for this can be done locally.

[@jimj2683]

Better to process parts of it in the cloud.

[@HiddenExp]

What a time to be alive!

[@Ziggurat1]

Hold on to your papers!

[@HiddenExp]

@@Ziggurat1 😂

[@luiztomikawa]

Imagine how it will be two papers down the line!!

[@yosha_ykt]

haha nice

[@SamyasaSwi]

What about the fluid dynamics though?

[@rainbowtoyabc7155]

Please show more! I bought Modbox just now!

[@yosha_ykt]

+

[@dotails]

This was working nearly half a year ago so I bought it but we're still waiting on approval buy GT3 hosts, so he can release a version with GPT 3 last I knew.

[@wldorfman]

@@dotails where could we follow the progress?

[@not_x]

Yes! I've been waiting for someone to start using neural language models in games, a bright future is ahead :D

[@ddd99991]

This video is so cool because it showcases so many different applications of machine learning. Speech to Text, Natural Language Understanding and generation, Text to Speech, and coupled with futuristic VR goggles.

[@aidanwolf4756]

This is blowing my mind. Two questions: - Could you potentially set up a "general knowledge" prompt that would give all AIs knowledge of things like street names, currency, character names? - Could you potentially store every Q/A over the duration of testing, play testing, and production, to eventually get instant responses?

[@LeeVermeulen]

Hey Aiden! And yeah definitely. If you want it to know about the real world could just prompt it as a AI assistant. If you want to give it a lot of knowledge - just add it all the prompt, there isn't a limit really (tho each question would then send entire prompt to OpenAI, which would be costly)

[@Ziggurat1]

@@LeeVermeulen Would that require a transformer network?

[@lucascastro2802]

If you haven't heard of AIDungeon, look it up, it uses a GPT based AI that has advanced a lot since its start, and today has features to define world characteristics (races, important names, places) which the "characters" source from during dialogue. It's really neat!

[@GuillermoValleCosmos]

@@LeeVermeulen I think GPT-3 API limits the prompt length to 2048 tokens

[@RealCrate]

@@LeeVermeulen how about training the AI with the prompt once or using world info feature like in AI Dungeon?

[@nickfury1507]

Who here would donate just to expand these experiments? I vote we make a kickstarter for these guys so they can dedicate more time to exploring what's possible. I'll kick a little money in on my end to start with.

[@yosha_ykt]

Yeah agree

[@niikito8959]

Definitely agree with you. Hope he I'll read this and make a Kickstarter🙏

[@valberm]

How much? Like 10 bucks?

[@cfryantofficial]

I'd be up for that, especially if there's some decent documentation attached to this aspect of development.

[@zugbob]

I'm working on a VR game for the quest where you can talk to the NPC's

[@yosha_ykt]

Imagine GPT-3 NPCs on Skyrim. It will be a total immersion, the conversations will be very interesting.

[@crumbledsanity]

That'd remove the charm of Skyrim NPCs.

[@MaxLohMusic]

The problem is behavior; otherwise they're just walking roomba's who can hold a conversation but won't respond properly when you say "I will kill you now". GPT-3 power needs to be harnessed for scripting their actions, not just their words.

[@alexpapatonis978]

I do NOT want a semi sentient Nazeem. No way in hell

[@johnv605]

I want to see them have a mental breakdown after getting convinced they are artificial

[@johnv605]

@Solve Everything The only problem would be having several instances of it running and having to keep progress of all of them

[@ShaneoTheGreat]

Could you imagine asking a driving NPC why they drove right into you and getting into an argument with them after they cause you to lose your mission.

[@victoraurelius5334]

what a time to be alive

[@chocoais6601]

I expect a lot of KZbin videos based on that situation alone

[@AzMahgs]

Describe the npc next time as "pretends to work as a carwasher because he doesn't want anyone to know that he is actually making money by selling drugs"

[@alph4966]

It seems that he passed the Turing test. I can't wait to see how much GPT-4 will evolve.

[@exosproudmamabear558]

It is now at %81 previously it was in %71-75. If the next one is above %90 you can't even tell the difference from a human. But for it to become %90 it needs to have an exponential amount of training and data which will be cumbersome. I think Ai code needs to get smarter or it won't reach that level.

[@Consural]

No. GPT-3 is just good at being impressive at a superficial level. Once you spend enough time with it, you can tell that it's just "bs-ing" you, that it's not really forming actual "logical thoughts". It has no general knowledge whatsoever about the world. It has many potential use cases, but it's nowhere near being "actually intelligent".

[@arvideo2145]

Hi! Just wanted to say that it's super super impressive. I had a similar idea few months back, but since i know nothing about voice input to text stuff and AI's (i'm just a big fan of VR) it just stood as a dream, that maybe one day it's possible that we can talk to NPCs like this. And wow, i found someone who has a working prototype. Impressive!!! Already started reading about OpenAI now.. Wish i could try something like this one day too..

[@yosha_ykt]

Same, I do not understand anything in programming, but I am very interested in the topic of neural networks.

[@jpatt0n]

Incredible work Lee, you're always working on cool stuff - so excited for the future of this tech

[@BigBrezzy]

Mark my words: This tech (or something similar) is gonna be in GTA VII as a selling point.

[@valberm]

I have no doubt about it. And wisely said VII and not VI.

[@ionphaser7092]

@@valberm gta 6 had some sort of AI stuff planned many months ago in advance as they advertised, so I presume, how advanced they already were in 2013 with gta5 compared to many other games, gta 5 still rocking and winning over other games even in 2021, the truth can be they could have perhaps already much more advanced AI script than this one given how advanced and rich company they are, yet it might be also not the case, as they dont consider it being an asset and they could have invested into other thing or AI but differently , though if they really wanted they could perhaps match up, or even surpass this

[@Davids6994]

see ya in 2028

[@BigBrezzy]

@@Davids6994 That's quite optimistic, given that even GTA VI is nowhere in sight yet.

[@muhammadgheith2492]

Hello, Bobby! Bobby: ^-^ . . . . . . . . . . . . . " *WHAT* *DO* *YOU* *NEED* , *IM* *LATE* *FOR* *WORK* *AND* *I* *CANNOT* *TALK* *TO* *YOU* "

[@doyboy7377]

the potential of this game is absolutely insane, definitely buying it now

[@TommyLikeTom]

The future of opensource is amazing.

[@IliaSretenskii]

Okay, now it's time to find out what is a Modbox.

[@beattherapydotcom]

Bravo! Love seeing this tech in action! ... "City Hall is on City Hall Street Location" 😅😂🤣😂 that cracked me up. Have you tried playing with the mood variants on the Deckard voice? 🎧💙👍

[@eatonkuntz]

You're smart, I like that about you.

[@panteleymonschekochikhin-k1978]

Holy fucking shit, this is the future of interactive gaming i've been waiting for. I'm guessing we'll see it fully implemented in like TES 8 era-games, most probably.

[@exosproudmamabear558]

Tes 7,Ai is really fast growing. In one year things gets change completely and Tes 6 still is in progress. Tes 7 probably going to be released 6 years or so by that time who knows how things will be.

[@panteleymonschekochikhin-k1978]

@@exosproudmamabear558 in indie games, sure. In AAA games this should work flawlessly and seamlessly for it to be included in the final game. You can't have pauses, ai mishearing the player, giving weird answers, or really anything that breaks the immersion. For that level of polish we would have to wait another couple of years, that's why i think tes 8 is more likely.

[@exosproudmamabear558]

@@panteleymonschekochikhin-k1978 Umm did you see the AAA in the last five or so years. They are more shitty than some Indies.Not even considering they not even little "work flawlessly" or "seamlessly"

[@alexpapatonis978]

Damn so like 50 years at this rate 😂

[@KCGamingBuddies]

tes6 possibly, since thats like 5 years away and godd howard said they were waiting for better technology to do what they wanted for it

[@OmegaGlops]

I can't wait for modders to eventually implement this kind of stuff into older games utilizing the NPC's original dialogue as the basis for their prompts. Take the first Toad you meet in Super Mario 64, for example: "Am I glad to see you! The Princess...and I...and, well, everybody...we're all trapped inside the castle walls. Bowser has stolen the castle's Stars, and he's using their power to create his own world in the paintings and walls. Please recover the Power Stars! As you find them, you can use their power to open the doors that Bowser has sealed. There are four rooms on the first floor. Start in the one with the painting of Bob-omb inside. It's the only room that Bowser hasn't sealed. When you collect eight Power Stars, you'll be able to open the door with the big star. The Princess must be inside!" There's so much important exposition from that one paragraph alone to give him enough context of the environment that he's in, his current state of mind, and what the player should be doing right now, that I would imagine he'd be able to have a proper panicked conversation with the player explaining what's going on and what to do, instead of being an instructional booklet with the appearance of a Toad. Pairing the AI with the beefy lore-heavy NPCs from a game like Skyrim? That could be absolutely nuts in a few years. Really excited to see where this goes in the future!

[@kapsi]

Are we about to be singularized? All I know is that I'm definitely a real person.

[@ramdas363]

Just what a game character would say.

[@fitybux4664]

No. We are not about to be singularized. Google just made an AI chip using an AI algorithm. Do not turn off your TV sets. Do not look away. You are not being overtaken, tiny humans!

[@kras_mazov]

You should feed TTS with "hmmm" and "oh", "well" while the request is processed. It would make it much more believable.

[@RiccardoGrinover]

This is awesome and super impressive! Can't wait to see how the project progresses!

[@yosha_ykt]

This awesome, I was waiting for this. GPT-3 game

[@turbone6028]

This is absolutely mind-bogglingly amazing!

[@gygaxthegoat2491]

This is astounding, Lee!

[@ESTAMOSJUGANDOSP]

The next year we will start to see indie games using this technology. I can't F! wait.

[@Freak_Gamer]

This is quite rather fantastic, although it is quite a shame that it requires connection to an external server.

[@irecordwithaphone1856]

Hopefully one day we can all have the hardware capable of running this offline. I mean, wishful thinking but still

[@rackneh]

*Starts talking* (Oblivion Background music starts playing) Seriously though this would be amazing for a VR oblivion remake

[@journeywithoutdistance]

I find this to be one of the most fascinating game developments. I've searched "gpt-3 game dialogue" a few times since watching it in hopes to see more. Any chance you'll post some more of these in-game conversations?

[@LeeVermeulen]

Sure. Hopefully soon

[@MrGogoblaster]

I'd love to see more!

[@ireallydontknowwhatimdoing9508]

This is game changing! literally! and imagine what kind of awesome games this guy can make with enough funding

[@LightStreamer]

wish you made this longer it was very entertaining

[@KountKasper]

I just got modbox then realised this is for modbox 2.

[@LeeVermeulen]

It's for a upcoming update for Modbox - not a separate sequel/game

[@evashaevich5301]

Future is coming! Awesome!

[@Shadi2]

ooh. contextual aware AI. impressive.

[@jvoidhuey3088]

YES I FOUND A LARGER VIDEO!!!!!!! Only saw the clip on uploadvr. This is fascinating. Noticed as well I was already prescribed. Probably from a previous curiosity bender. I don't drink... I think!

[@yosha_ykt]

Same, i saw on Telegram news gaming

[@bravojr]

This looks amazing why do more people not know of this?

[@TheLethalDomain]

So... Bobby is a public servant that is late for work, therefore too busy to serve the public. What a fucking legend.

[@SirusStarTV]

Playing GtaSa for long time was interesting because you can do different situations with npc (like make them fight together) so if you would have been able to talk to them it would be even more fun

[@Igorious92]

Cool! Generate a history of NPC, generate a voice of NPC, generate a world of NPS, generate... our reality?

[@rcavicchijr]

When this type of adapting ai can be done locally; in real-time; without the latency and processing delay: it will Change story telling and video games forever. Right now games are presented to us like interactive, but fully scripted, movies. But when this tech, and tech like it advances: people will literally be able to craft their own narratives interactively, and in real time. Like a kid playing make-believe in the back yard, games can create their own plot around you based on your choices. You could literally put on a VR headset and narrate yourself into a lucid dream. Hell maybe with things like next gen broadband, wifi, and 6g wireless they can do it in real time over the internet, and people's stories can interact. A cyberpunk world can but up against a witcher world in real time and interplay with eachother. The Dragonborn could jump into the capitol wasteland and and take out the vault dweller... I don't want to hear the word "metaverse" until we can do that.

[@FEBC23]

This is insanely nice, can you make more (lengthy) videos where you interact with NPCs, and put them in various situations and making use of GPT3? Thanks.

[@yosha_ykt]

It reminds me of NPC from the movie Jumanji.

[@TannarThompson]

I'm hoping to one day build a game where you can use your voice to talk to an ai companion. kinda like a Google Home but it's a walking character who can take orders or give you orders/suggestions. my hope is to give the player a character to grow alongside during their journey and grow in a unique way depending on the player. The player would actually care for their ai friend because the character will be unique to that player. unlike games like The Last of Us where every player experiences the same sidekick

[@creator9071]

Imagine this in Skyrim

[@yosha_ykt]

Subscribed, I want more dialogue)

[@SirusStarTV]

4:42 его движения напоминают мне персонажей "уроки хороших манер" (и другие программы с этими персонажами)

[@WesleyAusten]

凄い！

[@xenowhite]

The future is amazing

[@Alexthatsnottaken]

CITY HALL IS LOCATED ON CITY HALL STREET LOCATION

[@yaroslavche]

Wow, that's awesome! Did you try to initiate talking between NPCs?

[@sabresound]

I’d actually play sims a lot if it took the right step in this direction or any game if this was integrated correctly like just imagine GTA with a crazy level of intelligence from npcs like wow just take my money! Keep the change you deserve it and Heres a extra tip for ya ££ they could probably rob me blind for something this cool if it works fluidly enough!

[@toddlertalk7922]

Dear Lee You are a pioneer of using GPT3 in gaming. It's great! But GPT is a solution for chatting, you cannot send Bob to action. For instance, if you ask "where is the bus stop here?", Bob cannot point and say "Yah, over there". It means that GPT doesn't understand a player I'm developing an API which aims to resolve the "understanding" problem. At the end it could be attached to Bob's abilities to interact vocaly with a player in a player's mother tongue, not only in English. Could we arrange a zoom meeting and talk about this? Sincerely, Chaim Ash

[@styxrakash4639]

You are doing gods work

[@DrFlashburn]

at 4:00 when you're showing the configuration for Bobby. Couldn't you use GPT-3 to generate configurations like that one?

[@dolcevitapizza1711]

Is this a feature that is available in Modbox currently, if not, is it something that is going to be introduced at some point, or is this just your personal api access to GPT-3 and not available to someone that buys Modbox?

[@LeeVermeulen]

The plan is absolutely to have it available in Modbox. It's up to OpenAI if we can do that (I still need their permission). We will likely have a few different language API options though

[@dolcevitapizza1711]

@@LeeVermeulen I appreciate the fast response and honest candor. I'm sold and look forward to seeing this implemented assuming OpenAi/Microsoft cooperate.

[@jacobdoesstuff6178]

Hi! I'm making a video on the future of storytelling and entertainment, and this very much fits the bill. Could I include a little clip to showcase what you have achieved here?

[@LeeVermeulen]

Sure, include whatever of this you want

[@jacobdoesstuff6178]

@@LeeVermeulen Thanks!

[@DarksiderDarmoset]

Really sick =o

[@morkovija]

So at wait point do we give them agency in a game like minecraft and see what they come up with?

[@nickfury1507]

I think as soon as it gets cost-effective enough to run them cheaply. Virtual evolution simulations, as many as we can generate, makes you wonder huh?

[@morkovija]

@@nickfury1507 indeed. much much cheaper than having them in physical form in our world. no need to build boston dynamics to get interesting results. Was thinking about it, was getting results suspiciously close to the plot of westworld x)

[@nickfury1507]

@@morkovija Well, if you keep up with Two Minute Papers, Curriculum Based learning is already moving things forward quite a bit, in terms of stuff like learning how to walk. But full fledged worlds with virtual beings developing over generations? The thing is we're building speed, the amount of knowledge gained from that would be tremendous, and we're building up to it. At this point, other than the cost, all you have to do is keep correcting this until humans give it a high rating for being convincing. The exciting thing is that we can see it's not strictly a matter of technology anymore, it's more a question of implementation.

[@morkovija]

@@nickfury1507 of recommend Yannic Kilcher channel even more than two minute papers. In case you'd want to go deeper into

[@johnkris8802]

Modbox beauty

[@Lao_lu]

It would be interesting if the GPT-3 technology was used to drive the behaviour of the AI, not just the speech.

[@luiztomikawa]

My dream is to have a Deadpool game that uses this kinda stuff to break the Forth wall and talk to the player XD

[@prof_pock1426]

Loved this vid!

[@cyag6864]

Cant believe my shower thought idea came to life dude!!!

[@douglaspriester]

gpt3 2 years ago?

[@ElectricSlevin]

is that update out yet? the steam news page doesn't have any 2021 updates listed? I'm buying this game as soon as this is added!

[@pinkmilkproductions6760]

Extremely impressive stuff! I've been meaning to develop something similar that uses the new RTX cards' tensor cores so it can run in real-time. I would absolutely love to hear about your experience chaining these modules together and your thoughts on the system. Do you have a Discord server or someplace where i can talk to you about this? Anyway, very cool to see that this concept works as well as i hoped it would work!

[@notanthing3343]

very creative streets name city hall streets

[@poopsikins3522]

Would it be possible to run this on GPT-2 Instead? I know it's a lot less incoherent but at least you can run it on your own computer locally (I have a Ryen 5 CPU and NVIDIA 1070) without needing to go through OpenAI's API.. The Delays should be a lot quicker too.

[@LeeVermeulen]

I should try that

[@gymboybtroy5127]

I've been struggling very hard to change player aspects like speed and others. I want to make a grapple hook and make the player move fast in low gravity and changed health ext. How do I change the player?

[@cameron-smith]

This is very cool!

[@colinharter4094]

Six people didn't have 10 coins for a sweet hot dog.

[@theRPGmaster]

I personally would like the very spicy hot dog that has tabasco sauce on it which makes it very hot.

[@thebrax]

I have access to the GPT-3 beta api and am interested in making a similar project, can you explain to me how exactly you implemented the GPT-3 api into the game? Im fairly new to programming and was just wondering how hard it would be

[@eternalreturnal]

is this out? also curious to where is GPT3 running, assuming it's not locally.

[@jaimdiojtar]

this AI is 10x times better than cyberpunk 2077 broken AI

[@pdbsstudios7137]

@Lee Vermeulen can you mod the millenarie mod so the npcs will be as smart as yours?

[@heatherperkins9737]

Wow the hotdog man.......pretty strange.

[@jayanimations494]

The real project milo

[@colinharter4094]

Think about how lit video games are going to be in 50 years. Right now, there are considerable advancements happening every year in the fields of computer vision, natural language processing, speech synthesis, photorealistic graphics, transcription, and translation. All of these advancements are going to contribute to some truly incredible video games.

[@josephlang2586]

I wonder if bobby would remember he likes you because you're smart if you interacted with him later

[@ionphaser7092]

which map is tis one, can I load it somehow? is iit now one month later available at least for some testing players?

[@ludwigvanbeethoven61]

THis is THE REVOLUTION for RPG and Pornindustry

[@javiell]

i hope cyberpunk uses this

[@TheNinjaOfTheGoldenTemple]

awesome!

[@fleshwere]

Bit scary. Kinda feels? like they're on a low conscious level.

[@eatonkuntz]

PhilosopherAI app feels like that too

[@staff97]

*VERY SPACEY HOT DOG*

[@holdthetruthhostage]

Wait until Ai can literally run the entire game engine oh just thinking about it drooling 🤤

[@ElCarnesX20]

What happened to this beautiful project?

[@thesral96]

And how would you finance this? I mean there's no way GPT-3 is running locally on your computer.

[@LeeVermeulen]

Requires a Modbox account to use

[@thesral96]

@@LeeVermeulen So we pay with our user data?

[@LeeVermeulen]

You pay by buying Modbox on Steam

[@thesral96]

@@LeeVermeulen Are you sure that this is sustainable? Lattitude with AI Dungeon is also using GPT-3 and they need got quite expensive monthly subscriptions. But i guess they also need to cover costs from free users.

[@LeeVermeulen]

@@thesral96 we haven't decided what what user quotas will be, or how much we will charge for more quota. We definitely will only do it in a profitable way

[@Roblox22r]

Cool feature

[@RoddKane]

Could it be possible that you sell this functionality as a standalone outside of modbox?

[@ArinBennett]

Wow, this is super impressive. Will this be implemented in Modbox at some point?

[@LeeVermeulen]

I am not sure I can release the OpenAI part yet - but the rest will be in the next update

[@KoolenDasheppi]

@@LeeVermeulen Maybe let the users bring their own OpenAI API keys? I have one and would love to use this in Modbox.

[@completechaos9988]

this game is great.

[@VaupellGaming]

Love it,,, dont have the game, but i would buy the assets if you made the asset.. ;)

[@Fangh44]

Which API do you use to send the text to OpenAI ? I thought GPT3 was in closed beta.

[@Freak_Gamer]

He must have gotten access to the closed beta.

[@lolplo6115]

Sweet....