GPT-4o & Unreal Engine 5

GPT4o can now see the world in Unreal Engine 5
made using some plugins... and lost some hair to get it to work :D

[@DirkTeucher]

This is really cool . That high res screenshot is definitely slowing this down. If you ran a python screen grabber in the background that just grabbed an image at 2fps i bet it would double the speed of the responses. GPT 4o is supposed to be able to respond in something like 300ms which is nuts. Its virtually realtime. At this rate we will have real time AI next year.

[@sunilnarine6313]

This gpt takes so much time and nothing anything like falsy advertised gpt 4o presentation. The same boring ai with some improvements.

[@Cusey]

@@sunilnarine6313 nothing anything like falsely advertised? Sick grammar. Do you think that because it's slow in this persons version of it running in UE5? You can't put together that maybe it's this particular implementation that's slow? It's literally describing in detail exactly what he's looking at. Calling it the "Same boring ai" is pretty fucked up

[@wolfgangsam4652]

Thats really pretty cool! Kudos!

[@ThiagoLionheart]

Incredible, very fantastic ❤

[@YahiyaJasem]

Great stuff!

[@imxtwinky]

That will change everything in the future.

[@Nkrasart]

Awesome man! 🔥

[@sheikhsadab]

This is the beginning of something cool. It still needs to be a lot faster to be actually used in shipped real time products.

[@FUNBBTV]

wow, amazing

[@tbarsnessvfx]

in a couple months we wont need to send over high res screenshots. It will be able to take audio directly as well as video. No translation to text needed as input transform.

[@aj08coder]

This is amazing! It would really help me if you could tell me how you get the 4o model in unreal? What plugin did you use? Thanks :) edit: Nvm i found your vid thanks!

[@aj08coder]

@@GES1985 I found it’s using the api and he uses a json plugin to do it. He has another whole tutorial

[@thronosstudios]

Hey, as long as this is turned into something practical where you can give unreal prompts for character creation, material making and animation, then I'm all for it.

[@whome.773]

cool plugin but i think it would be cooler if the camera would be added to her face, I think the problem is that there is not quite an illusion in this implementation. We have been using the photo analysis for a while now so its not super impressive.

[@OnionFestival-si7tr]

The "scene" should be what is visible from her head camera.

[@MarvelMaster]

i know but you can do that 😁

[@MaxStudioCG2023]

@@MarvelMaster so how can we make this ? implement this chat gpt ? is it works with chat gpt 3.5 ? (because i gues this 4 is payd version )

[@MarvelMaster]

@@MaxStudioCG2023 open ai api is always paid... per tokens

[@MarvelMaster]

used it 1 day about 50 cents

[@MaxStudioCG2023]

@@MarvelMaster yea well thar sucks :))

[@merljns]

Hi, how many time until AI game maker ?

[@rofethegeek5820]

one tutorial please is awesome

[@yanhu]

You make the robot look around animation but still upload the screenshot to the GPT.😂 If the character camera works like a cellphone camera in GPT4o that will be really Awesome.

[@LeonGustin]

Gosh. I really can't wait until npcs Gain the ability to observe its virtual surrounding and react to it. Naturally that's gonna revolutionize gaming, Please experiment more with this. Thank you for the video

[@MarvelMaster]

in theory i can at least make walk along a path an ut explains what it sees and on certain triggerwords icould make it do stuff

[@LeonGustin]

@MarvelMaster Is there a way to give the NPC a camera like the player sees through as we walk around in the virtual world? Cause, it seems like it's not a continuous feed that the AI uses. It always needs a screenshot. Am I correct about that?

[@MarvelMaster]

@@LeonGustin you can place a camera on the npc head and take screenshots from there when needed

[@emrehan328]

JUST TAKE MY MONEY 💵💵💵💵💵

[@nilox4037]

this is not the fast version

[@MarvelMaster]

thats at least the latest api I could get

[@nilox4037]

@@MarvelMaster In the video they are talking to GPT4-o and can interrupt it. This functionality isn't yet integrated into the API and when it will it will be many many time fasten than the current implementation.

[@MarvelMaster]

@@nilox4037 for conversation and image recognition I used gpt-4o and for text to speech tts-1, I also expected faster speeds but not sure if its because UE5 uses JSON for comunication.

[@Nereal_Dvigatel]

Когда нибудь NPC перестанут быть тупыми и начнут анализировать окружение в реальном времени, тогда в офлайн игра выйдет на другой уровень

[@artoriasdenostradamus3628]

Tutorial ?

[@MarvelMaster]

hm uses a little marketplace plugin that I needed to edit in c++ and also the api is pay per token

[@artoriasdenostradamus3628]

@@MarvelMaster what is the plugin name ?

[@OriginRow]

@@MarvelMaster Plugin name please

[@MarvelMaster]

@@OriginRow its just for uploading an image to google drive...I had to edit its c++ to get the link where it uploaded to, but also wrote a request for the creator www.unrealengine.com/marketplace/en-US/product/drive-messenger?sessionInvalidated=true