Eleven Labs Dubbing + voice cloning AI | How does it compare to HeyGen AI?

Get on my daily AI newsletter 🔥
natural20.beehiiv.com/subscribe
[News, Research and Tutorials on AI]
See more at:
natural20.com/
My AI Playlist:
• AI Unleashed - The Com...

[@danquixote6072]

Haygen is amazing. The lip sync is so impressive. I did a Chinese video the other day and everyone I sent it to thought I had been learning Chinese - except my Chinese girlfriend who wondered why I had a Taiwan accent.

[@dv_interval42]

The way you hear yourself vs. the way we hear you is different. We have a biased sense of the sound of our own voice. When I hear the various cases, they do sound a lot like you. Of course all the intonations might not be perfect, but it generally sounds similar.

[@ChristianJarhult]

I was about to write exactly that. 🙂

[@daveinpublic]

Ya a lot of those options sounded exactly like him 😂

[@thespecialist3608]

I believe this is due to differing resonant frequencies! We often dislike the sound of our own voice when it is recorded and played back, as opposed to hearing it directly come from us as we are speaking.

[@huang47tw]

This is amazing I was using your another video "Dawn of LMMs" and tried English -> Chinese and it works pretty well

[@SteveRowe]

Hate to tell you, Wes, but those imitations of you sounded just like you.

[@Glowbox3D]

It’s crazy how so many different AI applications are coming out in two year. Nuts.

[@3dus]

Interesting comparison. Heygen uses elevenlabs tech. So maybe bc it’s lipsyncing it has more “freedom” thus resulting in better outputs. Part of the art of dubbing is trying to match and adapting the translation/expression to the original lip movement.

[@MrRecapFromYouTube]

But how did you make Russian translation with HeyGen? There's no Russian language

[@BrianMosleyUK]

Couple of comments... It can easily take up to TWO MONTHS to get a custom voice out of Eleven Labs - they process your voice samples in batches each month and the quality of your samples has a dramatic effect on the quality of output. Secondly, I called you out on one of your recent editions that your embedded narration video looked jerky and you didn't answer, but it looked exactly like the heygen output... I think it's great, but you should be proud of your workfow, I expect you to be using the best AI tools available in your work! 🙏👍

[@SophyYan]

The Chinese sounds amazing ❤ thanks for a great video as always 😊

[@Blocky007]

Think the big difference is the lip Synching. Which Heygen has due to creating from scratch, whereas Elevenlabs it doesnt appear to lipsynch (at all). For quick and easy dubbing probably suffienct and super good. For more involved: wonder if its feasible to extract the generated voice from the clip, and run it again through a wav2lip service to additionally do the lip synch based upon the eleven labs dub.

[@CM-zl2jw]

😂. Lots of fun. This will likely kill people’s desire to learn most languages. I liked your ai voices 😎.

[@connor883]

Doubtful because this doesn't mean now that you can communicate any better with foreign people. It's just a tool for outputting translated video

[@geomfilms]

can you put a source of your voice (English) and do the dubbing of it still in english but now you select one of your custom voices? I can't seem to find option to pick what type of voice you want the dubbed voice to be switched to. Just says select language.

[@neithanm]

That spanish is the classical mix of south american accents most americans associate it with. It has nothing to do with Spain's spanish, which is weird because it's called Spain/España 😂

[@Aaron-oe4yr]

The end of bad dubbing in movies

[@VIDEOAC3D]

Foreign movies might start competing with Hollywood now...

[@sinnwalker]

​@@VIDEOAC3DSoon Hollywood won't exist, so no more competition. When anyone can create any piece of content they want with AI, nobody will be paying for someone to make it for them, tho I think a small group will pay to see other works, it'll be nowhere near the level it is rn in scale.

[@VIDEOAC3D]

@@sinnwalker You're right. It might eventually become adapive to each user too, auto-generated content that's tailored to your individual tastes, or even your mood that day... What's scariest is that it might soon shape human perception that way, just like how most people are incapable of discerning fake news, but much worse. Because your thoughts are much less original than you think, and are shaped by the thousands of smaller moments and interactions throughout your day, and life. So an AI that creates all user content, suggestions, fills out all search results, teaches (with bias), automates tasks, plans, notifications, etc., could have an unfathomable influence over society's thoughts and perceptions... shifting beliefs in small "nudges," individualized, tailored specifically to each individual, iteratively reshaping opinions.

[@sinnwalker]

@@VIDEOAC3D I get your concern, but it's no different than what's been happening for ages.. lmao. Just with new tech. that's why open source is so important. Also FYI if we reach ASI we won't have control over the model, so our bias wouldn't exist. Also with countless AGIs in the wild, it diversifies the power. We basically WANT to get to that level, cus humans are easily corrupted, we want diversification. This is actually something the founder of Google's Deep mind said, there's a power shift coming. That's the difference this time.

[@bjg6056]

The German sounds great! The lyrics are a bit fast at the end but it sounds very impressive!

[@MetaverseAdventures]

How did you get Vietnamese as I am not seeing that as choice in Eleven Labs dubbing? Thanks for the video.

[@stianchrister]

Don't worry about it, it was really bad.

[@MetaverseAdventures]

@@stianchrister Sure, but I wish to still show my Vietnamese relatives for a laugh and as a point in time so we can track progress. Exciting to see how it improves over time.

[@Cordis2Die]

Daaaamn in Russian your pronunciation from HeyGen sounded absolutely perfect

[@WesRoth]

yeah, that was really impressive. I'm blown away by how good this tech is.

[@Cordis2Die]

Same. This is amazing!

[@MrRecapFromYouTube]

But how did he do Russian voice. HeyGen doesn't have Russian language.

[@Cordis2Die]

I'm also curious about that@@MrRecapFromKZbin

[@DefenderX]

would be fun to have a video about the uses.

[@spaceadv6060]

Heygen is surprisingly good!

[@ernesto.iglesias]

3:20 It's normal you don't recognize your own voice but ask others, it indeed sound a lot like you

[@WesRoth]

Interesting! thank you.

[@kettenotter]

Yeah I also think it sounds quite similar. Had to look at the video to make sure at which parts he was playing the ai audio. Spooky stuff

[@GCdevine1]

Can you use HeyGen and/or Eleven Labs do speech to speech conversion?

[@filthyjay7518]

Your eleven labs voice was identical to you 😂

[@s11-informationatyourservi44]

i lolled at the du hast reference

[@user-hm5rs4bp7r]

can you play out srt file and edit it if anything is spoken incorrectly?

[@thesadprofessor]

German in Eleven Labs sounds amazing (the first clip, not the 2nd one). HeyGen was a bit robotic, indeed…

[@jdray]

You commented on the translation quality for Russian. Based on that, I presume you speak Russian. Have you tried recording yourself in Russian or another language that you're fluent in and having it translate to English? English is notoriously difficult to get right (I'm only fluent in English, so can't say for sure, though).

[@ernesto.iglesias]

Aloud is the KZbin version, I've don't see anyone using it yet

[@Hisgreenhouse]

Ok my man, get us some quality audio on that phenomenon 200 mil views very very fast

[@R.E-O]

The Spanish, in general, is very good, but I have listened to several examples (not just yours), and I don't think it's 100% native accent. I think it sometimes sneaks in some degree of an English accent in a couple of words (not the whole audio) as if it was an English speaker who is almost perfect at Spanish, but not completely. But I'm from Spain, so judging that kind of Latin American accent (I don't even know if they are mimicking a specific one or if it is some kind of fabricated "neutral" accent) is not always easy for me, but I get the same feeling as watching Wagner Moura playing the role of Pablo Escobar in the Netflix show 'Narcos', i.e., even though I'm not familiar with the Colombian accent, I am able to recognize that he isn't a native Spanish-speaker (he is Brazilian). But overall, I'd say I'm nitpicking a little bit; it's pretty good. German on the other hand, I think it sounds even better, but I'm not a native speaker, although I've had a lot of conversations in German with native speakers for many years. The Asian languages... I can't be sure as I don't speak any of them, but Chinese and Vietnamese don't seem to sound good. Those are tonal languages, and I simply can't hear the same kind of tone variety that I hear when listening to a native speaker. I've listened to another KZbinr using Eleven Labs' dubbing from English into Japanese, and I think it was awful, with a very noticeable foreign accent. I don't speak Japanese either, but I've listened to a lot of Japanese on KZbin and some non-dubbed anime, and it simply doesn't sound good. I get the same sensation when listening to the Russian dubbing. In summary, it is a promising technology, already very good in some languages, but still has room for improvement in others. Also, the way it needs to fit the target language in the same timeframe as the original language creates some glitches that completely ruin the illusion of the person in the video actually speaking the language naturally.

[@Blocky007]

this was mentioned to me as well a few times when I used my own voice clone (enlgish/german original samples) and i.e. into portuguese. That it sounds very good and like me, but with a small accent. Which ironically actually makes it unintendedly more authentic to trust that its your original voice?:D

[@R.E-O]

@@Blocky007 Yes, I agree. When I've listened to a Spanish speaker clone their voice and use it to read a text in English, if they choose a native English accent, then I find it very difficult to recognize whether it is the same voice, but when the Spanish accent is kept in the English voice (I think even a little bit is enough, it doesn't need to be a thick one), then it is much easier to recognize that the cloned voice is from the same person as the original.

[@WesRoth]

Interesting! yes, multiple people mentioned that the Spanish isn't "from Spain". It's probably more similar to how Mexico and South America speak Spanish. For the Russian dubbing, the consensus seems to be 'excellent' for the HeyGen version and 'decent' for the Eleven Labs dubbing. A friend of mine says the Thai and Vietnamese dubs were poorly done :(

[@R.E-O]

​@@WesRoth In Latin America, they are used to having the same dubbing in movies for all the countries in the region. Even if the Spanish spoken in two South American countries can be as different as the English spoken in the US and Australia, they get the same version. But in Spain, we are used to having our own dubbing. All the big Hollywood studios and streaming services do that dual dubbing, one for Spain and one for Latin America. I hope Eleven Labs will add multiple accents/dialects for each supported language, although I think that the priority should be on improving the Asian languages, especially those with tonality (Thai, Vietnamese, Chinese, etc.) Also, the way HeyGen manipulates not only the audio but also the video, I think that it's completely necessary to eliminate the problem of different lengths in audio depending on the language, and Eleven Labs should look into that. When studios dub movies, they have to choose the speed of speech and the words so that they fit in the same timeframe, and at the same time more or less respect the lip movements. That sometimes causes the translation to be less accurate than desired. Using AI to dynamically adapt the video would be a great solution for having more accurate dubs that would still feel natural.

[@hyper_channel]

It sounds Latin American Spanish to me, something like the accent a Mexican person would have

[@fingerling613]

The voice for Vietnamese still has a long way to go tho. Nearly unrecognizable, really sounds like a foreigner who took like 2 classes of Vietnamese language.

[@WesRoth]

yup, I've been hearing more negative takes on the Vietnamese dubs. Same with Thai, it seems those aren't up to par yet.

[@joecavanagh1297]

So is this supposed to keep your accent but in a new language or try to make you sound native?

[@dogeelon9865]

Lol, these definitely sounded like you.

[@adrianfiedler3520]

Oh wow, the German in the beginning sounded so natural I thought: "oh cool, that guy is a German"

[@pq2667]

funny, but most options DID sound like you :)

[@r.m8146]

It sounded exactly like you; people don't usually recognize their own voice...

[@mosca204]

Any open source alternatives?

[@WesRoth]

nothing at this level right now, as far as I know. However give it a minute 😉 Open Source tends to catch up fast.

[@o_2731]

Bro got rejected from art school before saying 'Nein!'

[@clumsy_en]

Google might be working on real-time AI video translation for KZbin, but it won't use your actual voice for the dubbing.

[@CM-zl2jw]

What’s your native language?

[@s11-informationatyourservi44]

as a chinese speaker, the accent is impressive

[@TiagoTiagoT]

Your synthesized voice wasn't really that far from the real thing; it probably sounds more different to you than to other people because you're more used to your own voice, and specially because you usually hear it from inside your skull, while people hear it from the outside.

[@mvasa2582]

Is it me - or your mouth movement in HeyGen is much more in sync than Eleven Labs?

[@robbiero368]

My understanding was Heygen use Eleven labs to do the audio

[@WesRoth]

oh, no way! I did not know that... Thanks for adding that, I will have to look into that.

[@robbiero368]

@@WesRoth I thought I saw it on their website

[@user-pw7ij4ut3t]

heygen literally uses elevenlabs technology to power it's own AI dubbing and lipsync technology. so technically, heygen only offers lipsync

[@basspig]

I tried it on a portion of the first episode of Fumetsu no Anata and the results were quite hideous.

[@sitedev]

Yeah, mate. Can ya start translatin’ ya vidyo’s to Orstalyian? That’d be beaut!

[@macmanbd]

Bro 3:30 does sound like you!

[@TheSparkoi]

As a french , 1:35 its sound rly weird

[@ngamashaka4894]

All I can say, as a French speaker it is not as amazing as it sounds to you. I easily can see people making a great effort to not have to listen to the translation.... A good subtitle translation will still be better for now for me.

[@yarmand]

Ditto :)

[@WesRoth]

ok, thanks for that! yes, the eleven lab dubs, do seems like they are not top quality. But this was the first day they were released, maybe it will improve, maybe I just need to learn some tricks to do it better etc. But, I enjoyed hearing myself in French, but I'm sure the dubbing wasn't great :(

[@ngamashaka4894]

@@WesRoth Well it is as if Chat GPT was making a translation and the voice was in the uncanny valley. I'm sure with time it will get better. They wanted to be the first to present to possibility I guests...

[@bloodust7356]

Yeah French had a really strong accent, and at the end i could not understand what it says. I think here it's trying too much to keep the speaker accent. Text to speach felt way better (when it's not using the canadian accent)

[@AlastairGames]

It's funny how you don't think your own voice sounds similar, when it does to me. I think you'd probably need someone else that knows you to do it since it's probably hard to judge the sound of your own voice.

[@JackTheOrangePumpkin]

2:00 ther German is very good. It's sound like an American immigrant who lives like 15 years in Germany

[@JackTheOrangePumpkin]

Who is also kind of a machine... But maybe that's also a kind of your normal intonation idk

[@xerxos1980]

Well, still some errors in 15 seconds of video. Not as good as a human yet.

[@CodyRiverW]

It sounds like you but in different languages

[@connor883]

I dont think this is going to prevent people from learning other languages because this really doesnt help you communicate with foreign people any better. In fact ithink it will help people pick and aspire to learn the languages they think they sound good in. Remember its not impressive if you can communicate with technology but without

[@WesRoth]

true.

[@zyxwvutsrqponmlkh]

I only really care about open source developments. Commercial offerings are not worth a sack of beans if I cant run them locally why even bother.

[@MicahYaple]

Creator's make entire channels for other languages - it will kill their revenue. Other's will also start just copying their videos and uploading & profiting from other creator's work. KZbin will have to deal with this NOW not later

[@Meza201]

The creators will no longer need to manually create those other language channels. They can reach the same audiences and more with good dubbing without any extra effort. It expands their audience not reduced it. They'll make more money from more views in multiple languages.

[@Diginema]

Vietnamese is far from closed

[@thebluesclues2012]

KZbin will add this automatically and it will kill heygen (they deserve that) and eleven labs.

[@vu70]

The Vietnamese version sounds horrible 😂 It really sounds like a westerner trying to speak Vietnamese for the first time. I don’t understand anything.

[@Beauty.and.FashionPhotographer]

Elevenlabs renders bad results on cloning your own voices. the worst.

[@ismbeatz]

heygen is better

[@MRTree-zz3wf]

the axent is pretty bad

[@freecode.ai-]

For some reason no one is talking about these translations from other languages to English. There are some foreign movies I want to watch without subtitles.