How to Use Your Voice as Input in Unity - Microphone and Audio Loudness Detection

[@someone2939]

we need the finger slap in Spotify LOL. THANK U SO MUCH your videos are literally the best and they're helping me with my graduation project !! THANK YOU!

[@projectafterworld2557]

This video killed 2 birds with 1 stone for me. 1: Is IS possible to just GET microphone input in unity without relying on some external library. 2: You can just MAKE audio clips from float arrays, and overwrite them as they loop in real time to get a continual stream of audio. (Which makes me suddenly understand one of the old VRchat voice bugs a LOT better) FINALLY I can comfortably start tinkering with real-time audio processing. Voice modification or even complete re-synthesis. Detecting phonemes, pitch, tone, all by evaluating that float array from the microphone samples. Then generating a new float array based on samples, filters and processing rules, packing those into a new audio sample and playing that instead of the direct mic input. Naturally there's a lot of other little details to work out, like using a Fourier transform to find the fundamental frequency of the voice sample for pitch detection, and basing all the other evaluations off that pitch so it's consistent no matter who's talking. This is mostly a pipe dream project.

[@jacobdean3045]

This is brilliant, a very straightforward way of integrating microphones as a mechanic in a VR game. Thanks so much!

[@rdqplaysro4875]

very easy to understand and straightforward. Thank you, internet stranger!

[@ValemTutorials]

Thanks for watching internet stranger !

[@colling46]

This is brilliant! could you also share how you made the mouth movement so smooth? It looks like they all want to know how the 'smoothness' variable is used in your script. 12:10 (lower right on the screen)

[@파우캣]

I'd really like to know this too please.

[@MetalMonstrumGear]

tell me, did you manage to deal with smoothness?

[@colling46]

@@MetalMonstrumGear Yes but with my own method which I'm about to share. I made the y scale of the mouth gets smaller little by little on every update. People, let me know if following method worked for you too. add this code in Update function. --> if (loudness < threshold) { loudness = 0; //to prevent jittering issue, slow down the speed when the mouth go back to default size. if (transform.localScale.y > 0.2f) { transform.localScale = new Vector3(transform.localScale.x, transform.localScale.y - 0.08f, transform.localScale.z); } }

[@Sp4iK]

I saw a similar video some time back and it took time to understand. Now I have just seen yours and it´s perfect! I´ll try to use it for a proyect that has been in my mind time around. Thanks!

[@BrandonNyman]

Works amazingly! Thank you. I wrote it to affect the emissive texture as well so that I can make things flash when talking or playing audio.

[@guckka94]

this is so cool!! thx mr. genius

[@mcleverm01]

Hi. Thanks for the video. @ 4:30 seconds into the video, you create a component called 'ScaleFromAudioClip' and have this code: 'public AudioLoudnessDetection detector;'. However, my c# says that the namespace cannot be found. What do I need to get this error fixed? Thanks!

[@AyadenAllen]

Are you using visual studios? If so, go to the error and it should give you the option to auto-fix the problem. In this case, it would add the appropriate namespace at the top of your script. I hope this helps. Last but not least, google the error number and you should get the correct namespace to type at the top of the script

[@pillemarisken]

I'm having the same error unfortunately. :(

[@khelogame7467]

I'm having the same error unfortunately. :(

[@ZacharyHelm]

This means your IDE isn't detecting the class you should have created at the start of the video. Check that you've created the AudioLoudnessDetection class with no typos, and that it's saved to a file called AudioLoudnessDetection.cs in your project.

[@GamerReality]

This came at the perfect time for my next project!

[@DoubledEthan]

Thanks Valem, i got several questions. 1. I don't understand how startposition = clipPosition - sampleWindow works. If you start from position 4 and remove 64 samplewindow aren't you in a negative position? 2. Why do you return an totalloudeness / samplewindow instead of just returning totalloudness? 3. Source.timesamples Does it give the specific time per frame of the sound? (If so it can miss several sample not? 2.

[@ValemTutorials]

Hi man thanks for the commment I'll try to answer these questions below. 1) Yes you can have a negative start position, to counter this issue you can return a loudness of 0 in that particular case. Thats what I did at 4:07 2) totalloudness / samplewindow gives a mean value around the data we collected. A mean value will give us an 'idea' of the loudness of the audio around the clip position. A mean value is computed by taking the sum of all element in the array and divide by the number of element in the array. That's why I divide by samplewindow which is in this case the number of element we have 3) I'm not sure to understand this question correctly but Source.timesamples gives the timesample the audiosource is currently at in the audioclip that it is playing. So using it in the update function I can get the loudness of the clip at the exact moment the audio is at.

[@NaviYT]

I saw the title, clicked, heard the voice, and thanked god Valem was the one to make this tutorial. You're going to be HUGE someday man! Like VR Brackey's

[@xanestudios]

For me my microphone works when testing connected to pc but when i build to quest 2 my microphone is not sending sound (and ive given the app permissions)

[@jiteshkumar208]

NOICE! Thanks for this!

[@projectafterworld2557]

Finally got around to setting this up and in typical fashion I ran into an error that was my own fault. Turns out if you try to make an audio clip with a length of 0 (Because you obsessively expose every value as a variable in the editor, then forget to actually SET said values) problems happen. xD

[@ValemTutorials]

This happens all the time for me too glad you fixed it

[@jelliottmason]

How did you smooth it out? You never told us how to do that, but we can see the smoothness factor in your video....

[@sataStrike]

It's probably using Lerp like he did with changing the cube size. The more smoothness, it just means the time to Lerp is longer.

[@amberzhang7926]

Thanks Valem, This is really helpful.But I don‘t know why should use" clipPosition - sampleWindow" to calculate a startPosition?

[@osmanDemir104]

Thanks Valem

[@SomeDudeWhoDoesStuff]

Is there a way to do something like this but to change an image? As audio scales the picture changes?

[@fabrihp]

Are there any scientific paper about the use of Unity for collecting sound triggers, and it's reliability?

[@mylogic156]

to make the mouth open without jitters, i used the vector 3 move towards function. EXAMPLE transform.lossyScale = Vector3.MoveTowards(transform.lossyScale, loudneess.position, scaleSpeed); prolly not exactly right but i had a different use case hope i helped!

[@rdqplaysro4875]

I know im late but is there any way to translate this for 2D?

[@유민영-d4c]

Thank you so much for the good tutorial. If Android allows microphone access, is this also applicable in Android environments? It works very well on Windows!

[@mcleverm01]

Thanks. I'll try it.

[@ponysonic]

fantastic! any chance you can give a hint on calculating the smoothness for the mouth animation?

[@SkeleTonHammer]

You can set a "goal" float for the mouth size, and then approach that goal over time using delta time instead of applying the loudness directly to the mouth size.

[@SwapnilLadkhedkar]

why did you used sample window as 64, i have copied the code word to word but my cube in scene isn't scaling. i am using audio from stream

[@Gamoplane]

Same

[@mrmony7069]

I have the error, "Microphone failed. result=25 (Unsupported file or audio format.)" any help?

[@viba_dev]

I have the same error. Did you find a solution?

[@TimebombSyaz]

hello valem i am a new game developer students so basically I have an idea to make 2d horror game sound detection where if player make sound through their mic the circle detection will get bigger and enemy can detect it.....so my question is can this tuto be used in 2d game or is it only specific on VR games....i saw that on the last part you make the mouth moving but im not quite sure.

[@Keiru]

Dang, I want to be as cool as you. This tutorial will really help me with one of my CompSci dissertation projects-- which is basically making a sound based platformer game. Is it possible to do something similar for pitch?

[@usernotfound.3837]

omg this accent is super sweet, i can not concentrate

[@tonywhite4476]

Can You elaborate more on the particle system? Great video

[@smoothiefemale]

well explained!!! thx

[@brunoseba15]

How can i get a string of what are you saing on the micro??

[@Kris4Infinity]

I wish i had any fuckin idea how to script in C# cuz Id want to modify this to be able to drive a parameter used in the animator from 0 to 1 depending on loudness- 1 being the loudest- but I cant figure out how to do that for the life of me TwT Any help w that please?

[@Gamoplane]

Which editor u r using and can i use it today also

[@MetalMonstrumGear]

There is no answer anywhere for smoothness. In discord too =(((

[@I3ra]

For anyone who hasn't been able to get their "loudness" variable: Microphone.GetPosition requires a string for the device name. I'm not sure why his script worked for him but I had to change it to Microphone.GetPosition(microphoneName)

[@petitwhito3196]

Hey, i have a small issue with sockets and Photon multiplayer. When i put an object in a socket, the synchronisation with photon works perfectly, the issue is that when i take back the object from the socket, the object stays in the socket for the other players and the photon synchronisation stopped working.

[@XanTheAwesomeGuy]

does anyone know how to do this but when you talk it changes a material?

[@MarkTreeNewBee]

It's an unbelievable turial, thank you very much, it help me a lot in unity.

[@jefflim69]

Thank for tutorial , then does it works on Android device ?

[@Sp4iK]

Hi Valem, I got it working, but only if I disable the first cube that has an AudioSource as input. Do you know why could this be happening?

[@killereks]

How do i know what mic sensitivity should be if everyone has different loudness

[@AmayukiShiina]

If I install the game as apk to my oculus quest 2, the microphone wont work, How should I deal with it(

[@bangojetty]

Does Unity have a way to save a recording to an audio file for use later?

[@ValemTutorials]

You mean record the player microphone to a wav file ? This project works well : github.com/ah1053/UnityRecorder/releases/tag/v1.0

[@FabiLous09]

does this work for multiplayer?

[@davidjunadi3924]

hi, is that support for unity webGL?

[@imnotcaseoh]

How would you get the average loudness of the clip?

[@utopiavr_iwg]

could you use it to save a voice command something. You save your voice and then compare the words on the mic with the clip so you can fire off say something like command open main menu?

[@ValemTutorials]

Good idea ! There is the speech recognition api from windows that you can use to recognize some specific word in unity : kzbin.info/www/bejne/aGrZqniFnNiKbtU But it does not work an all platform. Meta has enable voice recognition in the oculus quest lately but I'm not sure we can use it ourself with the Oculus sdk I'll have to look at it. :)

[@utopiavr_iwg]

@@ValemTutorials thanks let me know I am interested in that. Also have you been in NEOS vr app. in there world builder you can add code to 3d objects so players can build there own addons to things. Would have any idea how to start that. I would love to add this my app. If you think it is out of the scope of your tuts would you be interested in side work?

[@jcz8225]

la, do you have the project? to test it, it's not vr right? I'm from Argentina, I'm not a programmer

[@MinecraftGamerLR]

How can this be done with PlayOneShot audio clips, though? I want to add some simple lip sync to my avatar, but not using the microphone. Instead, I want to add voice acting.

[@MinecraftGamerLR]

Another question: How did you add the smoothness to the code?

[@thuctran453]

Does anyone get its work on Mac? Seem like the clip position always return 0 with Mac built-in microphone

[@Jaskier13]

Does this work for Android too?

[@zhenyuanwei-i6i]

cool!

[@dezjdidjdi]

What if i want it at android

[@riddhimusmade459]

does anyone have the code?

[@joelkanna3563]

how to make it 3d?

[@MPRGyanStudio888]

How to add alan voice unity lip sync

[@nicolasportu]

Outstanding! Hey, Doesn't work for webgl. But it works perfectly building for PC Windows ".exe"

[@HowIsYourPc.123]

can anyone paste the code he re

[@EriHenOficial]

Anyone can give me the script pls?

[@realmix3720]

Nice one! You use a Smoothness factor in your MOUTH example kzbin.info/www/bejne/mqunYaSGbducgtkm13s How do you calculate this factor?

[@jelliottmason]

I would also like to know this.

[@TheNoxide]

using UnityEngine; public class AudioLoudnessDetection : MonoBehaviour { public int sampleWindow = 64; public float GetLoudnessFromAudioClip(int clipPosition, AudioClip clip) { int startPosition = clipPosition - sampleWindow; float[] waveData = new float[sampleWindow]; clip.GetData(waveData, startPosition); float totalLoudness = 0f; for (int i = 0; i < sampleWindow; i++) { totalLoudness += Mathf.Abs(waveData[i]); } return totalLoudness / sampleWindow; } } public class ScaleFromAudioClip : MonoBehaviour { public AudioLoudnessDetection detector; public AudioSource source; public Vector3 minScale; public Vector3 maxScale; private void Update() { float loudness = detector.GetLoudnessFromAudioClip(source.timeSamples, source.clip); transform.localScale = Vector3.Lerp(minScale, maxScale, loudness); } }

[@bruheny]

legend

[@bushraalshamsi1329]

what about other scripts?

[@calalmuradov6162]

Is it same like my talking tom?

[@pertdiagram1051]

I'm gonna be the bigger dog here, since i have a dog in this fight, there is missing a code: void Start() { MicrophoneToAudioClip(); } i didnt saw it, so don't complain if it's already in the video

[@ValemTutorials]

What's your dog name ?