How to make an AI VTuber Using GPT 3 and Google Cloud TTS

Github: github.com/adi...
Twitch Oath Token: twitchtokengen...
OpenAI API: openai.com/api/
Google Cloud: cloud.google.com/
List of Supported TTS Voices: cloud.google.c...
VB Audio Cable: vb-audio.com/C...
OBS Script for Captions: gist.github.co...

[@adi-panda]

I FORGOT TO MENTION: MAKE SURE TO INSTALL VLC ON YOU'RE COMPUTER (64 Bit Version) on your computer. ALSO IF YOU RUN INTO ISSUES INSTALLING THE PACKAGES YOU WILL HAVE TO MANUALLY INSTALL PACKAGES. (pip install twitchio ... keep on doing until all packages are installed) Sorry if this tutorial isn't well made, I just wanted to share how I was able to make an AI Vtuber with all of you

[@Byron_Hill]

can you maker a follow up on how to set up vlc? i am having trouble with this

[@pterodactylw3899]

If you add voice mod to the mix it would be amazing and if you get a anti AI text rescripter then your making free money

[@justinhwang1160]

@adi-panda Hi, thanks for the awesome tutorial! I've made sure that all of the input/output settings are done correctly but I still can't seem to get the audio to work. I set the Vtube input to VoiceVolumePlusMouthOpen and output to ParamMouthOpenY, just like the tutorial. However, when I type in the chat, no sound comes from OBS studio and the Vtube model mouth does not move. (I also don't get how you would listen to the bot speaking if you set the speaker as the virtual aux) Also, I connected my Twitch account to Vtube Studio and OBS but nothing happened, so I guess I'll disconnect them again? Any help would be greatly appreciated!

[@MrRaja]

How do I feed llama outputs into my output folder..

[@bronxandbrenx]

This tutorial is so much :)

[@Nazralova]

19:15 For anyone wondering why you can't hear the sound output from OBS Studio, you need to change the Desktop Audio --> click the three dots --> Advanced Audio Properties --> change Audio Monitoring to other than Monitor Off. For example, if you record with the Monitor and Output on, both sounds (in this case, monitor and speaker) will be recorded.

[@Gr13fM4ch1n3]

As soon as I saw clips of Neuro, I immediately wanted to know more. This video popped up in my recommendations just hours after finding her, and I'm so grateful.

[@2DReanimation]

For a chatbot like Neuro though, you'd need a local large language model, as the responses need to be fast. And GPT3 and ChatGPT will go "I'm afraid I can't do that Dave" too often to have the interesting responses Neuro has. ^^

[@alexlarex7773]

@@2DReanimation to have a local large language model you need a local GPU farm with enough VRAM to fit it all, and stuff on the level of GPT-3 will require you like 300 gigs of it. Which is not feasible. When you use gpt3 or even gpt3.5 via it's api, it won't do "I'm afraid I can't do that Dave" whatsoever, it's your responsibility to filter the responses so they don't break openAi terms and conditions. Edit: not even just 300Gb, 700Gb of vram for gpt3-like nets considering it's 175B parameters...

[@alexlarex7773]

​@@2DReanimation Nope. for training you would need even more. 700 Gigs of vram is for inference (just running the model with some inputs just to get some outputs) on a GPT-3-like model. LLAMA models are smaller than GPT, yes, but even those are far from tiny. The smallest LLAMA model (7B params, which is considerably worse than gpt on this specific task) needs at least 14 gigs of ram, which is indeed possible for a consumer rig. The proper 65B param LLAMA model needs 130 gigs of ram, which is technically possible if you will run it on the cpu, but for gpu that's already like 10 GPUs worth of vram. Also have you actually tried doing any of that yourself? I'm speaking from experience saying that using the APIs is actually faster than running the net locally, i get 1-3 second responses with the API on average, which is perfectly fine for answering twitch comments. ChatGPT is GPT running with a specific prompt before each of your messages, and openAI's filters before and after. ChatGPT is not a good example of the limitations of GPT.

[@danielhughes3758]

​@@2DReanimation He did say GPT 3/3.5 API, not chat GPT. I haven't tried that yet so I can't confirm or deny, but I believe that if you skip the chat GPT added functionality and just go with underlying GPT then you have more freedom in those regards

[@Gr13fM4ch1n3]

@@alexlarex7773 I've been testing with Vicuna and it's only taken 20ish GB of space. It gets a little confused and starts writing fanfictions halfway through some conversations, but it's still funny.

[@Descalibrado_]

This is way more than what i expected, very cool to know that you can also refine the model so it learns

[@zapp3264]

Thx for sharing your knowledge on this topic! You ve done an amazing job in this tutorial :D

[@elzilcho222]

had to do some fancy audio stuff to make this work (for example, it doesnt matter on my computer is the output is the vb cable or not), and the vtube model talks for any sound coming out of my computer, but still super cool! I never would have gotten this far without this video, thank you!

[@MYPSYAI]

THIS is exactly what I wanted. Lets make a community around this "advanced beginner" level!

[@darkff9090]

If you tried it Bro can you telling me what's your Instagram account I want to ask something about this..? Please

[@ComputerSage-Dev]

Quick FYI, the git repostory uses text-davinci-003 model, very powerful but is the most expansive, gpt-3.5-turbo is a better option, both are GPT-3 (3.5 technically) but gpt-3.5-turbo is focus on chat like interactions and is way cheaper, it will require you to change the openai request tho.

[@endymion3213]

How can we do that? total noob here, i'm trying to use chatgpt to figure it out but it's still way over my head. ty

[@TehBrownie]

how Do you change it

[@gispry]

For an update on this, the git repository now uses gpt 3.5 turbo

[@DansuB4nsu03]

Why is this video in my recommended, I'm not even that much into coding, lol Thanks KZbin, you're the best for recommending me smaller creators :)

[@qmac9966]

Wow I feel like I should be paying money for this information

[@adi-panda]

thanks man

[@ViewOfAria]

Fr

[@zimizi]

there is a better thing now. but i struggle to find sovits voice for neurosama.

[@Homiloko2]

Great tutorial! The anime girl speaking with a male voice cracked me up though, gj

[@shonuff2216]

thank you dude, thank you very very much. you helped me fix the audio for my code.

[@jefftseng6939]

So dope, i create my own chat bot using python, but i dont know how to connect it with VST, thank you, now i think i can move on next step.

[@VRAnimeTed]

Mine is really coming along! I'm running 5+ hours a day on twitch while tested. Managed to get gpt3.5 working. I thought I'd posted the code changes in chat, but maybe I restarted the computer without hitting send, hmmm... main swapping engine for model, but more to it.

[@1doom1000]

Knowing your work, I'm curious and excited to see where this goes...

[@TehBrownie]

Ya want to share the changes to GPT3.5

[@FARSIDE0_0]

How'd you only have 360 Sub? You deserve more

[@ghostsheepy.]

grats on the 10/10 from ign

[@chromacobble]

This is just impressive, kudos to you!

[@bronxandbrenx]

this is awesome. Learned so much from the video.

[@replikvltyoutube3727]

Good video thanks for going over the process.

[@gispry]

Got this all set up and it works well. Though I want to have it pull the most recent message after it has stopped speaking as currently it seems to just go top to bottom regardless of how many people are chatting. I can manually reset it to force it to jump to the most recent message but it would be good to know how to do that automatically. I have been researching for a couple hours now and I cant find how to do that.

[@gispry]

I am so lost. How can something that seems so simple be so complicated. i just want it to read the most recent chat message instead of reading every chat message top to bottom like a list :(((((((( How is this not something someone has wanted before. Why cant I find anything online

[@gispry]

For anyone else wanting this I did finally find how to do it. Just needed to add if not message.content.startswith('-'): return under the async def event_message(self,message):

[@gispry]

Though doing this in notepad is very difficult as you have to indent manually

[@kuroraikiri6343]

This is amazing!!! please make a video explaining how to train the model ❤

[@Mamika_AFK]

Great info thanks! Just subbed 👍

[@ViajanteGaross]

this feels like an 5k dollars worth tutorial

[@fate2304]

Now I can make a coding buddy that can not only help me help with coding but also talk to me to keep myself sane

[@Overneed-Belkan-Witch]

*The Power Anime and ChatGPT in the palm of my hand*

[@LucyKosaki]

Is it possible to make a version that uses elevenlabs api instead of google tts? The google ones seem a bit restrictive in terms of voice selection and elevenlabs not only has very easy voice cloning features but it also gives you 10k free tokens a month for testing. I think this might be better for imitating anime voices. But I'm not a programmer, I don't know if this would be just some very minor adjustments to the code or if you need to rewrite the whole TTS section of the script with new api requests.

[@incongruous4]

Will there be an updated video re-explaining after all the changes to the GitHub files recently? Thanks

[@Morzeabyss]

Aaaa dios santo si tan solo supiera inglés, pero este videos es Genial y de gran calidad, Gracias broder

[@BanriFerdinand]

Hay mucho contenido en español solo que está todo por partes. No hay un tutorial en especifico w

[@ittz_apollo]

Hey, I don't suppose you could rework this to instead of being for twitch, it is integrated with Discord. As in.... It is linked to a bot that we can create in the Discord Dev Portal, it joins a channel when the Host joins and it uses Speech recognition to listen to what that person says and then responds text to speech. This would be extremely helpful if you have the time to look in to this. Thank you.

[@septumkmm6030]

in your bellow code, I have found that once the conversation limit is reached, it will remove the prompt which is found in the .txt file. if len(Bot.conversation) > CONVERSATION_LIMIT: Bot.conversation = Bot.conversation[1:] so I have fixed this issue by using if len(Bot.conversation) > CONVERSATION_LIMIT: Bot.conversation.pop(1) this way memory list is still preserved while prompt is still active which each new request sent.

[@basaran3d]

Bro single handedly made me learn coding

[@nightcorealg5765]

now all I need is a tutorial on how to run everything on your computer ;-;

[@adi-panda]

sorry lol, I tried my best, but this was only really a quick project lol

[@ViperousVT]

Thanks this would work for my Lyra Ophuchius model 🐍 😃

[@gouphee]

hey this is a fantastic video. i was wondering if you would be making a video on how to get started with fine-tuning or if you could point me in the best direction to learn more about it? thanks again for the great content!

[@deadmeme4058]

Is there any way for connecting this to Elevenlabs instead of google cloud tts?

[@MrErick1160]

awesome bro I need to do this. That will help HEAPS. I subscribed!!!

[@Kiwi_brand2004]

wow esto es increible, soy estudiante de programacion, asi que lo voy a recrear, muchas gracias por el tutorial

[@Bitscuits_La_Cielhare]

It very appreciated for sharing your knowledge for free!

[@flibflop8219]

Can you do the tts in a different way without cloud? Im just trying to cut costs. Im not going to be streaming with it so I was wondering if there was a way to do it without cloud and just running off your pc, Im writing this only like 5 mins in so Ill edit this if my question changes.

[@gab_campos]

I have the same problem as Video. The bot stops talking after a while. How to solve? And also in the prompt appears code snippets, which should not be shown, it should only have the bot's lines.

[@holyamvs4594]

can you make a google colab version, but stll this is some good work

[@breakyourlimit735]

thanks bro i will try it later for my adding project

[@twelvyvisionz7637]

Are we able to give the AI a personality? or lore?

[@jymcaballero5748]

que gran video amigo, felicitaciones, 1 suscripcion!

[@joshuamart3910]

This is incredible, thank you very much, but how is it possible that vedals can talk in real time with his AI and it responds in real time, even interrupting him like a normal conversation.

[@deepnightly3824]

damn brother i hope this still work, Thanks Man!

[@SeanKula]

Dude, your tutorial rocks! Thank you. Can we get a tutorial on an AI that plays the video games, kind of like neuro.

[@maublanch]

Hi, please I was wondering if I can lock the process so my avatar would only speak when asked mayube with some commands or words, I can't understand or find the solution for this, maybe is only adding an if, but I'm not good at it.

[@wikanz09]

Is there any way to do these with elevenlabs and youtube chat instead? Appreciate your efforts in making this video. This Ai vTuber thing could be a goldmine in the future

[@thomasfrigerio6147]

I was here (Sure u''ll gain a lot of subs)

[@youjinchoi3430]

Thank you so much for this. My only question is would it be possible to set up certain personality of an AI when it reads and responds on the chat? + due to some troll viewers, some things have to be filtered or ignored. Is there any way to do so? I am not sure if my question makes sense and is there any way to set this up in Korean?

[@hevilmateold]

FINILLYYYYY LETS GOO

[@slowmilkandcookies]

hi!! im currently stuck on~ 10:42, if i may ask, what did you do to fix this error on your end? i tried the twitchio command but still couldnt enter the -requirements.txt or python txt i keep getting an errno 2 error~

[@angelmontemayor7106]

Did you run pip install -r requirements.txt?

[@BenCatQ]

i had the same problem the fix i found is create a .bat file in the same folder with all the other parts and stick pip install -r requirements.txt :)

[@angelmontemayor7106]

@@BenCatQ Which parts?

[@bogartwilley]

I keep getting a Syntax error when running main.py It keeps pointing to my Token as the issue, but for the life of me I can't find the issue.

[@barrygovino3100]

Dude your awesome!! I'm gonna try it out soon. Btw is it possible to add different voices instead of the default one?

[@adi-panda]

yeah you have to change the code though, in the description I put a list of available voices

[@evieyorisou2586]

​​@@adi-panda does the language of the TTS voice use have to be the same as the language that's being used in chat such as if a chinese user is saying Chinese would you then have to use one of the Chinese voices or if English users are using all English would you therefore have to use only English voices or do all the voices basically translate to the same thing meaning it doesn't matter what language you're speaking the language of the voice will still be able to be used.

[@rioharta526]

thank you for sharing, will watch it later

[@anenfairy4996]

big question, how do I teach them right off the bat not to say anything bad? can I create an exclusionary dictionary or?

[@MsAnxiety]

in miniconda, I keep getting error messages whenever I try to put in the folder location. How would I fix this so that I could continue?

[@elfVex]

am at the part where he tells me to do the "pip install -r requirements.txt" but it says "ERROR: Could not open requirements file: [Errno 2] No such file or directory: 'requirements.txt'" anyone have any idea what i did wrong?

[@AalummiCh]

I'm having problems with registering the audio to make the avatar talk in relations to the audio, even though it's set the complete same way and I have the cable program installed

[@deusexmakima]

same dude, if u have the solution plz tell me ;)

[@Bettychan1]

is it possible to use elevenlabs tts instead of the google one?

[@annasbites]

Thank you for good information!

[@AraragiKunHiPach]

Hello, I really want to do this, but I can’t because Google won’t give me access, can you help me please? I really want to do this, really, please help me!

[@dylan-lp6km]

What is supposed to be running on port 7500? I get an error when connecting to the web socket there

[@tangerinacat2409]

I like your words, funny internet man

[@clipsofohnepixel]

I keep getting this: FileNotFoundError: Could not find module... Kuebiko-main\libvlc.dll' (or one of its dependencies). Try using the full path with constructor syntax. What's the fix for that?

[@dixce5248]

Do you fix it?

[@clipsofohnepixel]

Nope, I tried every alternative but it simply doesn’t want to work 🤷‍♂️

[@dixce5248]

@@clipsofohnepixel I already fix it, just install the correct vcl version for your windows, chek if you have installed the 64 or 32 bits

[@yedochen84]

Thanks for sharing!

[@paradoxgaming10]

Love the video man but I want to know if I can make my own custom responses to certain questions and keywords instead of using the GPT or any other AI service ? So like in this I want to make custom responses and the rest of the process is same as you showed.

[@zazukanal]

Wow you are best it is what i need

[@VegaLyrae]

So I’ll just drop this helpful tip here as someone who made an ai vtuber. There’s a version of vb audio cable that is $5 and you can have multiple inputs and outputs. It’ll make your stream audio 1000% better to not use the desktop audio for the vtuber if you intend to stream as well.

[@chinonoob3219]

Nice video, more tutos coming up?

[@holopekochan]

thank you. the video is very informative.

[@0verDDoS]

Do you have to have any programs open to use this? My cpu runs a bit high when streaming games and idk if this will affect it

[@livelearn5741]

How do you retrain the model? Thanks for the great tutorial!

[@madman7923]

i found an incredibly easy way to do this. its really freaking cool and uses vroid studio models. let me kno if you all are interested. i sorta want to share it because its so cool and simple and don't want to share because its so simple and cool (they could patch it). but let me kno

[@hananwarraich3822]

I’m interested 👍

[@Grib_bolotny_2000]

Really interested

[@joseduarte1240]

If i want some custom voice what i can do? dont want that voices

[@DrunkyBearAslaxVODs]

hi cool video bro, is it possibly to change the tts voice to an anime waifu voice? a costom one? thanks

[@programateiro]

Great tutorial sir! Just a little question can I run the chatbot only with vtube studio without connecting to twitch?

[@yokoandkamina4837]

My main.py is completely different than yours? Could somebody help Please im stuck on 8:02

[@tuoni410]

information goes into cred file.

[@mattiapugliese8777]

The Ai can also answer to voices? Like if I speak she will answer or is only a chatbot?

[@sweetfry]

Isn't there some kind of Free TTS programs that let you record your own voice and make a clone

[@TheBBWolf]

pretty cool friend. Question: can I use a totally free alternative to openai, if so how and which one?

[@Dev_jH]

My main.py is completely different frorm that, could u help?

[@evieyorisou2586]

Is it possible to use ElevenLabs instead of Google TTS or cloud services and if so how would I do that to make the necessary changes to the Python

[@saturnorbit6810]

I like your perspective on fine tuning, thanks!! Any discords you recommend???

[@limajgarcia]

I'm trying to create a AI vtuber but I'm not able to integrate the chat reading system that you used so that it can interact with my system that chats with a microphone, any tips? How can I learn to do this?

[@thatawesomekk]

How is the text getting auto resized on obs ?

[@procrastinationguru]

Thank you very much to the author for the video! You are truly a legend. Guys, is it possible to somehow change the TTS model to another neural network so that my neurotuber speaks more vividly, I know a couple of TTS neural models, but how do I Connect them

[@Tuti_v3]

Hi, great tutorial. Can it be set to only read cheering messages?

[@l0neZer0l]

How can I interact with the AI with microphone or just text without having to use twitch chat

[@bladi8588]

there's some way to change the voice provider? google cloud don't let me use the text-to-speech without pay

[@Fedz-il-Lupo]

Is it possible to do the same also for the KZbin Chat? Or even integrate both?

[@danielhughes3758]

Pretty dang sure it's possible, as you're just streaming from OBS which in turn captures all the input. You'd need to get a youtube API key and modify the code to also connect to youtube using the key. You could even modify the script to take an input parameter so you can default to both but specify if you want to when you run the file. Perhaps youtube has some crucial limitation that Twitch doesn't have that would prevent running this, but I highly doubt it

[@Xaldras]

How do you change the ai's voice? is it through the initial text document?

[@adi-panda]

In main.py at line 74-76

[@STE4LTHYP1CKL3]

Hey I know this video is quite old so not sure if I'll get a reply but were you using any plugins for OBS for the subtitles? I can't seem to get my subtitles like yours the words are just cropped out of the text box if they don't fit (this is using the lua script).

[@Real_Flutoni_Lyne]

Is there anyway to use VTube type systems with Linux 32 python

[@roseydeep4896]

Can I run this on colab/remote GPU somewhere & use a different model (Not GPT3/4, I wanted to use Vicuna)?

[@gosudrm]

Thank you very much!

[@asf1062]

Is it possible to just run this without the use of twitch?

[@realezda2088]

can we do something that it can read another platform chat in persian and with another voice?

[@SeigenGaming]

Do you have tutorial for KZbin?