Yeah, its in that genre. Its actually meant to be quite similar to overcooked as I wanted a single screen multiplayer game to work with for an experiment.

I do consulting in the area of generative AI, cloud and creative technology. Riva is just one of the various tech stacks I try out. This particular one is for an AI virtual agent demo.

@@zoombapup I sent you an email. Might go to junk ;-)

Glad to hear it. You always have no idea how long any of these things will remain relevant. Good to know it still works.

Thanks Rich! Loads of interesting things going on in AI right now.

Honestly, its just as easy to ask the LLM to evaluate the emotion directly by asking it to return an emotional score value. A big part of the problem is that emotions are far more complex than most ML models treat them. I don't trust most of the classifiers because they're not really grounded in the psychology, but rather a shallow reading of it (I mean you can skim some work by Paul Ekman and write a classifier, but it doesn't make it good). There's a lot more nuance and things like micro-expressions that need to work better if this kind of technology is going to improve.

Unfortunately because it was part of a research project, I can't share it until a year has passed (due to the terms of the contract).

@@zoombapup Year up?

@@priyankt3i The project itself isn't really something I'd be too happy to share in its current form. But there is a similar one on the Unreal Engine marketplace that is reasonably similar. The trouble is that to make it more real-time you have to host it on pretty powerful local GPU's like my A6000's to make it really work unless you're happy for the round trip latency to be pretty bad as it is here.

@zoombapup I have been meaning to build something like this ever since aws made alexa api available, but now with LLM it makes more sense, also want to add features like emotion detection and eye contact, but not a lot of experience in 3d or game engine is holding me back... I have the backend built in both nodejs and python but.. needed some reference for the front end on webXR. well I guess it's gotta wait till real time LLM video is a thing, so building stuff would be like calling APIs only...

@@priyankt3i The main thing I learnt from this, is that the LLM's can add a good level of human feel, but latency is a big issue. Right now, you can reduce the speech-to-text latency using NVIDIA's nemo models (which are super fast). You can also reduce the latency of the LLM by using a service like Groq, but I prefer everything running locally in docker containers. The big issue is actually the expression part of things. This used Amazon's Polly model for the voice and that also gives you visemes you can use for the face. Ideally you'd have a voice model that was low latency and had not only visemes but also body expressions too. But I can't see that happening because none of the big tech companies care about that. NVIDIA has something useful in audio2face, but I don't actually like what it produces. Ubisoft have done some work on voice to body expression, but I've not come across anything with a convincing voice -> face + body model yet.

Unfortunately I can't open source it as it was done via a research grant and a partner owns the rights to it for a year.

Check all comments for this video, I have mentioned ways to achieve each component of this project, which might help.

I wrote it.. they have an example in their code repository to get you started.

Yes, its sampling the pixels just to the outside of the transition (i.e. where the alpha mask changes from off to on), so you can basically sample wherever you want by just changing the offset value. That said, I'm sure there are better ways to do this, I just didn't have a lot of time to experiment with it, nor anything to compare it against for what people were after. You can sample the background pixels (its one of the texture samplers you get from the engine) so you can do whatever maths you like really. Try it, its fun!

@@zoombapup no i totally get it I need to experiment and put some work in🤣 thanks so much for sharing it's really great info. I'm experimenting with virtual production and my pre keyed exr media always comes in with slightly jagged edges and just want to do anything I can to blend the layers better. So light wrap and edge blur would be awesome.

@@LiveUPMedia If you look at the composure plugin for UE you'll see some examples of materials that do the blur, which is basically a box filter that reads from a bunch of pixels to average over. All of this is pretty easy once you can look at the shader code in the materials. Good luck!

Hmm, to some extent I could see that happening. But I think its more likely that AI will simply improve past that being a reasonable position fairly quickly. There's a lot of people that think that "creativity" is uniquely human, but I disagree. I think we'll eventually have computational creativity that works and that will become the default mode of operation for the various language and image/video/3D models, working as a co-author (so that copyright can be assigned) or as a unique author.

Use the voicelab, they give you some drop downs for age, accent etc. I chose old, british, male. There's actually a couple more tweaks you can make that also change the voice tone quite a lot too.

Its a bit different. Some of these companies like MidJourney don't even say where they get their images from. Others have used the open source LAION dataset, which is just images scraped from the internet via a huge web crawl. Its more about the lack of attribution, given they have literally billions of images that they've downloaded from the internet and trained the models on. Adobe are trying to say that they're different by using their own paid-for stock content library, but even that has images generated with AI, which in turn were trained on LAION. Personally I think its quite a tricky issue, not as clear cut in either direction.

@@zoombapup What is your opinion on AI programming? I have been seeing people saying that the future for coders will be telling the AI what to do in terms of what code it writes. Meanwhile, I have also seen people like Jothan Blow laughing at the idea due to it not really having a true understanding of what it is writing.

@@sleepwalker6043 There's already plenty of people who use AI based models to help them code. I think eventually we'll have a lot of code written by AI, but its not really clear yet how it will pan out. I use it to help me remember python syntax as I'm mostly used to other languages like C++, so I know what I'm looking at and use it mainly as a memory aid. Bit like a spellchecker in Word. JB's right in that the model doesn't have an understanding, but that's not to say it will never have one. Code follows a rigid structure really, so I think it's inevitable that it will become more automated.

You can get viseme information from Amazon's Polly text to speech.

When you generate something, it does so using a number called a seed. Normally this is randomized, but if you use the same number again with the same prompt you get the same image. So what you do, is get the seed from what was randomly generated (the webui I use gives you the seed for each image it generates) and enter that again, but then change the text prompt, this is the result. Check out Lexica for more examples of image prompts and seed settings.

You can see the seed I used here.. the -S<number> value is the seed. Enter the same prompt text and seed and you should see the same images.

@@zoombapup thanks! i'll try xD

@@zoombapup Hi, Cool pics!!, what are the other parameters '-b' '-C' -S7044...'? and if you could share the class that corresponds to them. Thank you!

Yeah, you can change prompts to get different outputs. But the language you use in the prompts is very important.

@@zoombapup hey, I tried different way but couldn't get two people with details acting done. Also suppose I made 4 iteration with constant seed and I like the 2nd result, can you tell how do I get that 2nd iteration photo directly?

@@devajyotimazumdar3686 If you use Stable Diffusions DreamStudio application, you get the seed for each image. You simply need to give it the same seed and it will generate the same image again. Then you can alter the prompts while keeping the same seed and get slightly different results.

There's a bunch of them up on youtube. I'm not sure I'm the best person to do this as I've got a non-consumer card in my ML machine, so might not hit all the potential problems. Pretty much everyone can use this by looking for the man Stable Diffusion colab notebooks that are popping up, or using HuggingFace's stablediffusion demo. Or reward the makers of the model and use their DreamStudio web app.

Its a combination of disco diffusion via discoart and then using a neural painter to paint it.

@@zoombapup And the neural painter is written by you?

@@Seignil No, I'm currently rewriting it based on another one.. can't remember the source though..

An influence map is data "in the world", so think of it like "is this area near a wall". You might use EQS to query that data and choose a location because of it. But influence maps can provide useful world-space information and more importantly, have a mechanism to propagate it to other agents. So its kind of like a shared memory of the space. Things like "when was the last time this position was visited by a guard" might be used for patrolling. It allows us to add information into a this data, to change it over time, to find min/max/threshold values etc.

Yeah, I've got a few things to talk about where I don't want to spend forever editing content, but still want to have some visuals to keep youtube happy.

I did start on recording one, but I need to explore some particle setup stuff first so I know what I'm doing with that side of things. It is super fun though!

You know, that's a really good question! I guess the thing I see, is the incorporation of machine learning into the development process. There's an interesting swedish company called Embark Studios that is looking like they're thinking in that direction. My own work has moved in that direction, using computer vision to drive procedural content. I still think we'll have "traditional" AI in the mix, but moving towards data-driven AI content is pretty clear to me. Also, cloud-enabled UE5 is coming and that will likely move us forward in that direction too. Thing is, I'm seeing a lot of jobs relating to ML in games and it's a really nice overlap with your traditional game AI dev too. So if you're into Game AI, then I think ML will impact your role quite a lot. That gives me hope for progress. Thanks for the question!

@@zoombapup Very interesting! Glad to hear you're still optimistic and exploring the space! Your Ground Branch videos from a while back are excellent. From a layman's perspective, hearing you talk through your ideas is neat. > My own work has moved in that direction, using computer vision to drive procedural content. I wonder... How might that overlap with the vision tech behind autonomous vehicles?

@@spectator_6 Very similar in nature. There's a paper called Meta-Sim2 out of University of Toronto in Sanja Fidlers lab where they generate self-driving car training data in Unreal Engine using a probabilistic scene graph architecture learnt from a real dataset of self-driving car images (popular one called the KTTI dataset). I'm using the same approach (essentially doing probabilistic rule expansion on a grammar) to generate cinematography for Unreal Engine scenes. So there's lots of overlap. Computer vision is a really rich source of input data to PCG systems and given we have billions of hours of data here on KZbin to train models with, we can leverage that for lots of different purposes.

@@zoombapup Wow, that sounds AWESOME! It reminds me of a conversation I read a while back where "heat map" data from players (their paths and pause points, etc) could be used to "feed" AI path finding routes. All of that data is already fully stored and "deciphered", why not put it to good use? Leveraging that for cinematography scenes? So cool! Would be neat to see some of what you've been cooking up, if you're ever able to share.

@@spectator_6 I'll post some videos about it on this channel over summer most likely. Right now I'm working on the UE4 toolset UI part of that research.

Welcome the future from the past!