3 Seconds of Audio Can Clone Any Voice - Speech Editting with VoiceCraft

[@giovannif2567]

Awesome quality for a zero-shoot! I'll give it a try. Also thanks for the update on the upcoming voice clone video! I'm glad it's coming, hopefully during this week!

[@pawelszpyt1640]

It is worth to note that both Voicecraft and Audiocraft cannot be used commercially (weights have non-commercial licenses). Most people probably will not care as it is hard to prove, but I guess some people would like to know that. AFAIK you can use Tortoise (and probably RVC) commercially. Thanks for the video though, the results from just 3s of audio are just stunning. I get it that you can copy the voice patterns pretty well if you get like a few minutes of speech, but what you can get here with just 3 seconds is crazy.

[@FenrirRobu]

Exactly,

[@hehe42069-k]

SWEET finally, we are so BACK!

[@classic_sci_fi]

VoiceCraft seems to be the best one I've heard so far -- at least on KZbin.

[@Ovyron]

Lyrebird was at this level long time ago, but it was commercial, yeah, finally open source caught up!

[@mygamecomputer1691]

Any chance you can do your magic and create a one click install that will work on windows if you have an Nvidia GPU? Pretty please?

[@JaysterJayster]

Just found a KZbinr who did it on his patreon :)

[@BeyondTheLastPage-zm4up]

Cool stuff. Appreciate the update on the next voice clone repository video

[@adamrastrand9409]

Hello, why do I need one voice sample in the tortoise auto aggressive fine-tuned model in the voices folder? Why can’t I select just none and will the voice sample affect the quality of the trained voice I mean when you put voice samples in the voices folder after training and selecting the model and wears the prepare a new language tab and how does it work?

[@tylerchambliss8379]

Hey Jarrod. Do you think this would be a viable replacement for tortoise for me to use to make audio books? I really need something soon.

[@FenrirRobu]

The license is "Attribution-NonCommercial-ShareAlike 4.0 International", whether or not that affects you is a complicated question. Tortoise is free to use for almost *anything*.

[@Jarods_Journey]

If the audiobooks are for personal usage, I don't see why not. But unfortunately, I won't be able to implement this into any of my projects anytime soon

[@dhrumil5977]

I think the voice engine by open ai is just hype now because its from open ai

[@hihohihoi1325]

Looks nice, but isnt pretrained on other lagnauges i believe? So if i wanted to ass german fe, i would have to trian myself?

[@psalmy26]

I can't get docker working, it's breaking on the first cell. Would LOVE a walkthrough.

[@few2012few]

Thank you for good content. I'm just curious about the commercial use . Since Coqui Public Model License is only for non-commercial use . Is it good to use output of this model in youtube ? I'm also planning to use tts in my content.

[@Jarods_Journey]

Probably not, I'm not a lawyer lol.

[@nicknightly336]

How does the cloning do on voices with special effects tied to them? I.e. SHODAN from System Shock? Will a large enough finetuning dataset accomplish good results, or will the inherent underlying training cause issue from not training on voices with special effects/pattern variances?

[@ukaszLiniewicz]

I made an API server for VoiceCraft as well as added it to my audiobook/dubbing generation app, Pandrator. Both run on Windows and Pandrator has a one-click installer. I'm not sure what I think about it yet, to be honest. I achieve very good results with XTTS, but I cannot experiment with VoiceCraft too much, because generation is very slow on my measly 4GB 3050 (laptop), slower than processing XTTS results with RVC, even. I have only tried the smaller model (though, according to the author, the difference in quality is negligible). Sometimes it drastically changes the pitch, it sounds as though a sentence or a part of one was generated using a different voice altogether. It can be mitigated by playing with the parameters a little, probably. The quality of voice cloning alone is much better than XTTS (I'm not sure about XTTS+RVC), but consistency seems to be worse.

[@Ravisidharthan]

Hi, man thanks for the work.. Does it support apple silicon? Can you pack one click installer for mac mps? is there any demo online that Mac users can test, Huggingface? And does it support silero Indic languages? ❤ Great work

[@vineyardworker]

Thanks for Pandrator, much appreciated.

[@manhattanmyaa-mufc5555]

Love your stuff. I am using this repo and was able to reproduce the output wav files. I need to be able to load the model and then make many inference calls without having to reload the model every time. Do you have insight on how to do this or can you share your visual studio code project on github? So far, I am loading the model using fastapi and unicorn to host it. I can do inference via a post call the first time, but when trying to run again, I get WARNING:phonemizer:words count mismatch I think I need to unload or re-execute something to reset it, without removing the entire model.

[@GraveUypo]

oh, awesome. progress. we need more of it!

[@nightknight8651]

very cool I have to say I learned a lot from you especially when it comes to RVC so thank you very much but even with all of that I could never get a voice model that is identical to the original voice any help with that?

[@geraldcortez826]

please do a video on windows setup. I really want something to clone voices from small datasets. I been replacing game audio of Castlevania SOTN. and the PSX voice for Dracula doesn't clone because of only one minute of audio. I really would like to use all the PSX voices but right now been using PSP Dracula. thank you for any help. and thank you for the video.

[@Interprestor]

Can this do things like laughing, sneezing, coughing or things like that?

[@jonathandaudin]

It works only in english?

[@clarkkent12880]

What's the lowest power Nvidia card needed to make this reasonably work

[@zachary3603]

if you don't know the answer to this, installing it is going to be a NIGHTMARE

[@CES-x5t]

have you tried like indian accent or singlish accent ? will it follow?

[@Digildon]

6:00 First he says "dislike the video" and then stops to think, "What the hell am I saying?"

[@Benbobr]

Can you share your build? This is the quickest way to use VoiceCraft

[@dthSinthoras]

Before I try to istall this: How does it perform with other languages?

[@TheSpartan-tu2fn]

What you think about using cloud gpu?

[@justriseandgrind6910]

it's getting scarier and scarier

[@wakandaPeter]

Thanks Man for all you do

[@joshuashepherd7189]

Jarod, How was the inference speed and GPU utilization on your 4090(I assume you're using one)?

[@Jarods_Journey]

Eh, about 5-15 seconds once alignment was finished using MFA. It is pretty fast generally, but I'm sure it's not optimized either

[@joshuashepherd7189]

@@Jarods_Journey That's not too bad at all! I'm trying to move away from freaking ElevenLabs for my project. So I'm about to get real heavy into RVC and Tortoise. Gonna hafta binge your videos XD

[@HarryClipzFilmz]

@@joshuashepherd7189 What is the difference from ElevenLabs and this? I am new to voice cloning and just want to understand the programs to use to do it

[@aziacomics]

Cool. Does it run on CPU or you must have a graphics card.

[@Jarods_Journey]

Need a GPU, will run on CPU, but way too slow.

[@احمدصبيح-خ7و]

Thank you for this useful and wonderful channel. I have been searching for a long time for a program that converts text to speech for Arabic. I have tried many programs and found that they do not reach the desired level regarding the Arabic language. The best one was eleven labs, but it's not free. Also, xtts coqui is good for the Arabic language, but it needs improvement. Currently I am looking for a program that converts text into speech. It can be trained to pronounce the Arabic language.. I do not know how to train xtts coqui in the Arabic language.

[@encapsulatio]

What about the best open source speech to text? Is there nothing better than Whisper in accuracy?

[@Jarods_Journey]

Whisper is the best afaik

[@MrcVicM]

thanks for all the insights !!

[@ryannpulido4004]

Do you have suggestions for real time TTS models that are open source?

[@pawelszpyt1640]

Tortoise with the right settings I guess. Select DeepSpeed and low samples (2) / iterations (

[@AEONIC_MUSIC]

Alltalk is the best I have seen better then tortoise and it has a streaming mode it does one word at a time and plays it do you could probably do real time. It's also faster

[@pawelszpyt1640]

​@@AEONIC_MUSIC It uses XTTSv2, which is very powerful model, but released under terrible Coqui license. Not open source at all. I considered paying Coqui for it, but before I pulled the trigger, they shut down their business and now you have no way to legally use it at all.

[@Jarods_Journey]

Tortoise TTS would be it or xtts if you don't care about licensing. Styletts2 is a good contender as well

[@tr1pod623]

could you make a one click installer for us? that would be greatly appreciated!

[@FernandoOliveira-jd5il]

Hi, it can talk another languages ?? By the way, tanks for all the work, nice channel to creators.

[@Jarods_Journey]

Just English I think for now

[@dougmaisner]

great stuff!

[@Jarods_Journey]

Thanks :)!

[@NaitorStudios]

Does it work well for other languages?

[@Jarods_Journey]

I think it only does English right now

[@warpsol]

How does this compare to my boy tort TTs RVC ?

[@AEONIC_MUSIC]

Alltalk is way better and faster at everything then tortoise but im curious to see what this one is like compared to allltalk

[@Jarods_Journey]

Haven't done any thourough comparisons, but, it's pretty darn good lol

[@Random_person_07]

Does this come with the webui?

[@greypsyche5255]

No, it comes with a jupyter note and I have no idea how to use this thing.

[@Jarods_Journey]

No, I built the webui for it to make it easier to use. Someone said they were going to PR their own version of one on the repo, so we'll have to wait and see

[@BobKane-g6x]

This license allows only non-commercial use of a machine learning model and its outputs. Why even bother...

[@FenrirRobu]

Did they specify it for outputs? Usually it's been vague but risky, did they finally admit that it's just not usable?

[@BobKane-g6x]

@@FenrirRobu Yup.. for output.

[@yuyutsurao]

You can use they are not going to re check you

[@Jarods_Journey]

It's cool tech, the abilities for prosody and pitch maintenance from 3 seconds of audio is pretty wild. To put it out in the open means that another project will build upon it, possible with more permissive licenses

[@FenrirRobu]

@@Jarods_Journey let's see if lucidrains takes up the task.

[@nanduofficial18]

How to run in Google colab

[@Jarods_Journey]

Since it was built from Linux, you can probably just follow the readme

[@yuyutsurao]

Can I run this without GPU any method 😢

[@Jarods_Journey]

I don't recommend it. It seems to run on CPU, but the outputs take much too long to generate.

[@Trungkhai-z1h]

Does it support CHinese and cantonese , vietnamese ?

[@Jarods_Journey]

Just English

[@MadeEasyTube]

Pleas can use with arabic voice

[@adamrastrand9409]

Hello, why do I need one voice sample in the tortoise auto aggressive fine-tuned model in the voices folder? Why can’t I select just none and will the voice sample affect the quality of the trained voice I mean when you put voice samples in the voices folder after training and selecting the model and wears the prepare a new language tab and how does it work?