How I Made Lip Syncing Software In Godot

[@LucyLavend]

Note: when I poke fun at other engines I don't mean "Godot is superior". I use Unity too next to Godot and I definitely don't hate it. Thanks for watching! ❤️

[@lolcat69]

I am making an engine, and I was going to implementó texture atlas, and your video gave me the perfect idea of how to do it in my cpp code and my shader code... Thanks :)

[@godofdream9112]

Mam, can you shere the github link, please... I want to see it in deppth... Please .

[@kiryonnakira7566]

well, i use both, and i hate unity. LineRenderers and TileMaps just rub me the wrong way.

[@GameEndeavor]

That is very cool how you're able to sync your lips with your audio. I could see this being useful in a game too. Amazing work. :)

[@LucyLavend]

Thanks GE! A small game jam around microphone input would be pretty interesting 🤔 There would probably be a lot of "scream to move" games lol

[@GameEndeavor]

@@LucyLavend lol, I was thinking of lip syncing dialogue in like a farming simulator Stardew Valley kind of game. But that works too. xD

[@LucyLavend]

@@GameEndeavor Ahh as in useful for game development, that's a really interesting application too!

[@igorgiuseppe1862]

@@LucyLavend maybe an chat app!

[@mmmegaidiot]

idea: somehow use live feed from a microphone in real-time, you could potentially use it in an online game

[@duducorvao]

This is such a cool video. And we can hear the excitement in your voice, which makes it even better. Please, keep making more :3

[@elrelster]

This is incredibly awesome. Never though I'd see frequency analysis of voice spectrum happening in godot :D

[@Atropos148]

That shader for changing expressions is a cool idea :) also, it seems that if you add keyboard shortcuts it might actually be a faster than what you did before :)

[@igorgiuseppe1862]

you mean shortcuts for changing the pose?

[@Atropos148]

@@igorgiuseppe1862 yes the pose and expressions, the muscle memory of "what expression is where" would make this workflow even faster :)

[@BreezyDefrag]

WHOA! I didn't even know you could do this kind of thing in GODOT! You should set your buttons to a Stream Deck, or Keyboard controls. That would work out then you can talk while pressing the buttons on the Stream Deck/keyboard so you don't have to manually press the buttons in engine.

[@rubixkyoob2]

This is so clever! And thanks for breaking down the shader code. I've been thinking of using a similar effect for the face of the character in my own project

[@queenofsquiggles]

I have some linguistics background so I can corroborate what you found with vowels and "s". There's a class of 'segments' that you're describing. (segments are individual speech sounds, in English each letter can have multiple segments associated with it depending on an individual's accent). They are called "sonorants" because they are the segments that can be held and have a "periodic" frequency. Their frequencies stabilize out to whatever frequencies that is unique to the speaker's vocal tract, but there are some common ranges for them. Sonorants include vowel sounds, and [n], [m], [s]. The square brackets refer to the symbols being IPA (International Phonetic Alphabet), not necessarily English. Because sonorants are so clear and stable, they are super easy to identify with a little bit of spectrum information as you show in the video. Something really cool is based on the frequency of sonorants, you can determine the percentage of your lips that are rounded, so if you wanted to go more dynamic you could check for a frequency range particular to your voice, and use differently rounded lip textures. Specifically, it's the F3 formant that is linked to the roundedness. A low F3 means very rounded, and high F3 means flat/spread lips). You can also use F1 for mouth openness, where low F1 is very open and high F1 is very closed. Linguistics is super cool to me and it was really enjoyable to see you show some linguistics tidbits in this video. I'm not sure if you researched it before or stumbled upon it, but either way it is fun to look into. Best of luck with making your workflow more efficient! Your videos have always been very informative for me learning Godot Some easy resources to read more about the linguistics concepts I mentioned: en.wikipedia.org/wiki/Roundedness en.wikipedia.org/wiki/Sonorant

[@piousthepious]

This is super cool. I can see this being super useful in games that use mic input to talk to others. This can honestly be applied to a simple portrait, not even just a 3d model

[@Harotyx]

Amazing, would love to see some more Godot tutorials since your way of explaining and the production behind the tutorials are so much better than those unedited in-engine footage other tutorials have. Keep up the good work!

[@LucyLavend]

Thank you! More content is in the making 😄

[@Harotyx]

Im sorry for my comment, unedited video is ok too

[@benjattkk]

One of your best vids! The app looks great.

[@maranr]

I love the simple lip sync technique. Would love to see a detailed tutorial on how you did that. Or better yet a demo or plugin!

[@badmonkey91]

I'm tooling around with it. Using the frequency to identify vowels really isn't accurate. It half-ass works if you have someone with a flat, very consistent voice. But if it's someone that speaks with any inflection, it's not at all accurate. An example being someone that speaks with 'up-speech' (like Australians or valley girls), or when asking a question. If you've identified a particular vowel - let's use O as an example - as being at a certain set of frequencies. When the person's voice shifts pitch (during up-speech, or when asking a question), it will shift which frequencies you're testing, and the shape in general changes. So you can't consistently test for a set of particular frequencies, or for a particular wave shape. I looked into how professional voice dudes identify vowels in spectrums. They work on charts that have the chart set with time and frequency as the two axis'. Not frequency and volume, as this spectrum analyzer shows. It seems like speech recognition in realtime would be immensely difficult and complex. Here's one example: kzbin.info/www/bejne/o4jInWigYqtroqc As far as a program to make a v-tuber talk...you're probably better off just having it play the mouth animations randomly. That's what some cartoon makers do. Given more time, maybe you could come up with something slightly more accurate though. like the identification of important vowel sounds, like O, E, and A.

[@erwinbroekhuis]

A nice middle-ground solution would be random mouth movement but using the analysis to close the mouth at the right moments.

[@smolblockboi]

Really cool to see someone else using Godot this way! I made my stream avatar using a similar method via layering Animation Players since mine's a 2d sprite but totally didn't think of making the remote control UI on screen and then mask it. I've got way too many keyboard shortcuts that I have a sticky note on my desk for reference lol.

[@BlueCubeSociety]

I always thought these were already done in Godot :shrug: Definitely cool, especially the lip sync

[@monochroma3803]

this is so cool! I never imagined you cant achieve this in Godot! Pretty nice stuff! Good work!

[@thomasamathew4058]

Please don't stop making Godot tutorials. Subscriber from India ❤

[@sasaha8389]

tbh i think this is way faster and more comfortable to use rather than keyframing the stuff in hitfilm. great job well done !!! :D

[@krzyszt0fus]

OMG! You are alive 😉 Thanks again, and looking forward for more 😀

[@seraaron]

I'm surprised you didn't go full vTuber, but this is still really cool!

[@josephducreux4236]

love the low poly and pixel art combo aesthetic

[@soloshottie]

watching Lucy explode at the end cured my depression

[@decoy3641]

Been awhile, worth the wait! ❤️

[@ViviTheEnbyWitch]

this is how learned that you can animate properties in hitfilm without doing composite shots, oh my god

[@nyeeesss2424]

I had no idea Godot had such good mic support! You video's quality just got a whole lot better Lucy!

[@thedupy124]

Thank you so much for this video ! I'm working on a Voice tools (for voice working), and you save my life 💗

[@sherekhangamedev]

Really neat solution, and great use of skills to reach it out. Kudos!

[@Galomortalbr]

really cool video, it's good to see godot being used for non gaming applications, i also love Gonkee's content.

[@franknord4826]

How the hell did you never show up in my recommendations before!? Subbed.

[@LucyLavend]

Thank you 😆

[@MrEliptik]

Really cool video and awesome work as always!

[@LucyLavend]

Thank you MrEliptik!

[@Gbtx6]

You did save a crap ton of time by doing this. Fantastic work!

[@crowcoding699]

1:41 omg. Definitely can relate. I love the workflow you made. It looks so intuitive for a project you made for yourself. Didn't know phoneme recognition was possible to do with such a simple-sounding approach. Tho if you like the realtime approach I'd recommend binding each pose to a numpad key. It'll be hard at first, but just like a keyboard layout, with practice you can memorize it blindfolded.

[@bourbonbobo]

This video is so good Lucy! Also you should totally trey streaming with this haha.

[@LucyLavend]

Thank you Breogán! I could indeed just green screen key and crop in in OBS itself 🤔 I enjoy showing my irl face on streams, but cropping out the buttons directly in OBS is also really useful outside of streaming

[@AlfredBaudischCreations]

That's so cool - what a creative thing to do with Godot!

[@Asocial-Canine]

This is a really neat little tool! The only improvement I could see with it would be to let the lipsync take an audio file as input instead of the raw microphone stream; It'd solve the issue of needing to change the poses live as you're talking. Though I imagine you could kinda bodge that by playing the audio file to Voicemeeter or something

[@KillerGameDev]

Alright, you earned the subscribe. That was pretty cool. Been thinking of making something similar but mostly for work so I can pretend to be at my desk in meetings. lol. Thanks for sharing with the world!

[@gnastygames]

cool af and I might actually use something like this in my own game for basic cinematics

[@jotanart]

Really cool idea!

[@anemonemnemone3918]

blinking IS cool. real keen on blinking

[@NekotoArts]

LUCY'S EDITING +12 MILLION! At 4:15 you can sample the texture once and save it to a vec4, then use the appropriate channels for ALBEDO and ALPHA instead of re-looking up the texture. (Although, ig it doesn't really cost much performance since your model is like the only thing in the scene). 7:06 is every programmer, ever. Hehe.

[@Pefeper]

WOAH this is super cool, awesome work!

[@LucyLavend]

Thanks! 😄

[@jackfishthe6th373]

Really cool! I wonder if this could be used as a early precursor to making a VTuber model in Godot?

[@stevenrodkiss473]

Absolutely brilliant!

[@KingThrillgore]

This is very cool. I'm guessing the logic you have written for the lip sync works off a fourier of the mic input, and its modularized enough to work with any AudioStreamPlayer source. I'm wondering if its at all possible to put the changes in the Shader on the AnimationTree or as part of an Animation, then you could run the full gamut of emotions on top of lip sync, but that may be out of the realm of possible with how its implemented now (you would need to triple the size of the sprite sheet as well). As someone who has given thought to accomplishing lip sync in Godot (and the few solutions on the asset library) this is a relatively simple to follow implementation. Pretty cool stuff.

[@JTriggerVideos]

This is the coolest thing ever

[@Zwiebel4]

Can you maybe make a tutorial on how you analyzed the audio in real time and turned it into animation data? I'd like to learn about the coding side of your approach because currently, Godot is still lacking a real-time audio analyzer that selects animations for phonetics for lip-syncing character models to audio. I know I can use external tools to generate the keyframe data for audio files, but having it processed in real-time would make this so much more elegant.

[@pipelonsogames]

oh, that's so epic, i want to make some similar for me, i need a microphone documentation for my linux mint operative sistem

[@R3W3W]

Mapping the poses to keys or xbox controller buttons would be cool

[@graydwarf22]

Love your work. Always amazed by the quality and effort you put in. Walked away from this wondering why you don't just record the audio without the hassle of timely animations and then during post, replay/review with some form of tool (your animation UI) that can create entries on a corresponding timeline (animate/function timeline) which you can later play together and edit.

[@pixelettee]

This is so amazing.

[@luigisake]

that looks cool // would be cool if you made a tutorial on how to make one ! or if there was a build out there to Download!

[@akio1st]

You are a Life saver, next video I will be doing wont be dealing with hardcore stuff hahah! Thx

[@nachoabadie4707]

Hey really nice and smart implementation!

[@blablagame3914]

very cool work! Do you have any resources on how to make the model itself?

[@danisob3633]

this is sick!

[@damonmichiels]

Amazing, I love it

[@CaveJohnson376]

you could import face expression texture as 3d texture or 2d array texture and rework shader code a bit to make it simpler. this would allow for easy, practically hassle-free scaling of amount of face expressions

[@abderrahmanemouhsine9561]

welcome back

[@Matto_Sama]

thanks

[@chimeforest]

This is very cool ^^ Have you considered instead of using a mic, using a audio file instead? Then you wouldn't have to focus on talking and clicking at the same time. It could also open up the possibility of adding key frame functionality in the future.

[@igorgiuseppe1862]

i NEED this, ok, i dont have the characters designed yet, but i need it! i cant wait to see the future of vtubers with it, maybe they make something... good btw, you had the same idea of using chromakey that i used once xD except i was using: webcam as texture, to put my face on a 3D cube on blender, then capturing the window on obs, and croping it with chromakey

[@InfernalWAVE]

Is there any way you could give deatil about how exactly you did the vowel recognition from levels? I have a whole spectrogram dawn at this point, but am not sure how to go about processing the info

[@felusion6290]

man thats a good idea to record a video like you do WOW

[@dragoonduneman4161]

oh a next idea for you what about motion control with hands so that you can act out your own character ?

[@user-js7ud9du2y]

awesome project

[@LORDSofCHAOS333]

great now i want to make png / gif tubers great again and try to code my own

[@qubitx64]

How did i miss this

[@areftb752]

Hi, Thanks for this tutorial. But, How did you get lip sync out of the spectrum??

[@Chevifier]

This is freaking Cool ngl

[@caveirainvocada9438]

This is just what i was looking for! But is this a good technique for use in games? It's pretty amazing but would it be slow or anything? I don't know much about shaders

[@NeZversSounds]

You have beautiful poses, why you don't you blend between them?

[@LucyLavend]

That way it would look a lot less snappy, but I could give it a try to see how it looks, thanks for the tip 😄

[@liefhebber6277]

Cool stuff!

[@LucyLavend]

Thank you! 😄

[@parvjain21]

It's epic🔥

[@llcoolgames]

woohoo! new vtuber just dropped! 😄

[@J0linarTube]

so cool, any chance you could share how you made it work when the mic receives actual input by voice?

[@J0linarTube]

actualy how you made it detect the vows 😅

[@ShinSpiegel]

That's so cool!!!!!

[@einlandersvideos6213]

You're a Vtuber now!

[@parvjain21]

What's vtuber?

[@nicholaswilliamson1684]

could you attach a prerecorded audio file to do it as well?

[@LucyLavend]

As long as it runs through the audio bus with the spectrum analyzer effect on it, yes! Godot even has build in support for recording audio, so I might add that next

[@hidemat5141]

Any chance you would share the code you use to sample the peaks?

[@BHanif-zc2pr]

Can you make a video on how can i reduce noise from my audio in godot. I want to record audio using godot and tgen remove the noise

[@TsukiDrawing]

Is there a possibility that you would make tutorial how to do it? I see more flexibility in making my own Vtube'y thing this way than with provided online tools, especially while I'm Mac user.

[@SynthGirl64]

hi lucy, your lil lowpoly animated avatar is really cool, I wanted to do the exact same thing for my music production/speedruns stream but also put the avatar at a desk with a keyboard and mouse (use mouse position and keystrokes so that it knows how to move the hands/arms in real time). as a trans woman who doesn't have access to hormones, lil avatar thingies help with gender dysphoria cause then viewers don't just assume my gender based on my voice. only problem is I got about 100 hours clocked into blender and STILL haven't managed to model my character yet hahaha, modelling is hard

[@ChrisTay229]

kzbin.info/www/bejne/o2S6l2B7msx1nck Try this video. It goes quite fast but there's a full 2 hour stream linked in the description, and it goes over making a psx style character. Not an exact match for what you're looking for but I managed to adapt it for some nier characters, and you could probably just have higher res textures and get the effect you're going for :)

[@SynthGirl64]

@@ChrisTay229 thanks :D I'll check it out

[@SynthGirl64]

@@ChrisTay229 oh wait hahaha, I have a reply on this video from 2 years back, yeah I've seen this but I never checked the stream linked, I'll take a look

[@hidemat5141]

Nice new avatar!

[@arrangemonk]

whats hindering you from actually playing the movement between two poses now?

[@watergategaming5639]

Holy shit this is so cool

[@LabFind]

"Is it faster?..." Wellll, it's more of an...initial investment

[@liamkoehler8820]

Is the green-screen thing really required? I would think that you could simply export with a transparent background, right?

[@LucyLavend]

Yeah but then I'd have to write an exporter that exports render textures from the viewport. That's definitely an option.. but I'm lazy lol. Currently I'm not using this anymore and writing a Blender addon instead 😆

[@liamkoehler8820]

@@LucyLavend Thanks for the quick reply :) Blender *does* have the ability to export with a transparent background, so maybe you will make use of it there. Will you do all your editing in blender? I hear the video editor isn't shabby.

[@Paruthi.618]

Definitely cooler 😎

[@vishnuviswanathan5308]

Good video

[@Goldenfightinglink]

wow, it's basically vtubing, which can rack up to tens of thousands of dollars. I can also see using the same concept of your face "setting" in other static huds/menus. with the mouth animations being proof that one can go really indepth with it

[@MikeCore]

you have too much power!

[@MikeCore]

also, love seeing another hitfilm user ^-^

[@jkouss]

Nice

[@Matto_Sama]

finally, lucy 2.0

[@LucyLavend]

Lucy 2: Electric Boogaloo

[@luigisake]

are you willing to do this in Godot 4?

[@UODZU-P]

blud reinvented v-tubers

[@el_jim1479]

did she just explode at the end

[@Matto_Sama]

WHOA!😊

[@tarsyth3433]

I'm working with a friend on a game and we need a couple of soundtracks is anyone willing to work with us

[@LucyLavend]

oh no

[@hairiclilred]

I'm working with a friend on a game and we need a couple of soundtracks is anyone willing to work with us

[@LucyLavend]

@@hairiclilred noo

[@igorgiuseppe1862]

nice intro

[@Bravo_0-7]

Honestly, if im given the choice of spending 2 hours per video editing or spending 10 hours questioning life while i make a cool program that would have so many bugs during the process of making it to the point where i wanna kill myself, i think the 10 hours are worth it lol, so it is saving time

[@prerit6915]

omg

[@Matto_Sama]

😢

[@Sina_Qadri]

Dude Please share this code without the model its not that hard been hours here im trying to do the same but the coding is pain in hell i cant even get the mimic right at last share the code not the project????

[@FalconFetus8]

Does this technically make you a VTuber?