Neural Network Learns to Generate Voice (RNN/LSTM)

[@SomethingUnreal]

Just to let people know, by popular demand, I've also uploaded a video where I do this with a male English voice! kzbin.info/www/bejne/hHiQfXSKd7-BeNU

[@SomethingUnreal]

+Johnny Mccrum: I'm afraid not. I don't know enough to program my own from scratch, so I was using the open-source software "torch-rnn" (github.com/jcjohnson/torch-rnn/) here.

[@postvideo97]

Practical RNN applications don't use 'homebrew' code, they always use some kind of GPU-accelerated library, such as Torch, Tensorflow, etc. There's no need to reinvent the wheel by coding the LSTM by youself (except for educational purposes, which is recommended as it teaches the fundamentals of BPTT). Any implementation of an LSTM RNN will be the same, except some differences in performance.

[@postvideo97]

@SomethingUnreal You should try training the RNN with STFT (Short Time Fourier Transform) instead of raw audio data, it should perform much better at distinguishing words, as the NN won't need to care about generating the signal itself.

[@SomethingUnreal]

+postvideo975 If you can point me to an RNN that takes 2D input, thten sure. Otherwise, I'm stuck with torch-rnn, which is 1D. BTW, I actually did experiment with feeding a spectrogram (FFT powers) to torch-rnn, "raster scan"-style (All of first time slice, all of second time slice, etc, end-to-end), and made a program that handles the fact that torch-rnn won't produce perfectly-sized slices some of the time, and amazingly, torch-rnn was able to output something that resembled the voice, but it couldn't make a stable sound at all (each generated slice didn't connect neatly to the next slice). I don't think I can get better than that while using torch-rnn.

[@thorn9382]

Wow youve been feeding your gan alot of hentai

[@gafeht]

When your creation is screaming to be put out of its misery, maybe it's time to rethink what you're doing

[@Alex-oz9eh]

ayy lmao

[@top1percent424]

gafeht You took it to whole another level.

[@yammie1536]

gafeht exactly my thought lol

[@elk3407]

Yah..... This video somewhat reminds me of Nina Tucker from FMA. If you dont know what im talking about, DONT look it up..... Its honestly kinda disturbing.

[@TGRoko]

No no, it's "learning."

[@smiledogjgp]

Funny how it learns to laugh and scream, far before it may form words. Quite reminiscent of infant humanity.

[@SuperGirl-eq1le]

Corpus Crewman ikr

[@scarm_rune]

this neural network defined the evolution of humans in a few mins

[@hbaggg]

well congrats you made a computer waifu

[@username306]

Skaterboybob made my day :^)

[@maxalbert8943]

Paddy made my day a day after your day

[@wavefireYT]

UP

[@AzureOnyxscore]

what next? a motorized fleshlight?

[@oliverhilton6086]

When you talk to your waifu and she replies with 30 seconds of "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii"

[@TrekkerLLAP]

*screams in Japanese*

[@Bugingas]

Minodrey あああああああああああああああああああああああああああああああああああああああああああ (A in hiragana)

[@Cassiepult]

What’s hanging boys? Underrated name and pic

[@wigwagstudios2474]

あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ あああああああああああああああああああああああああああああああああああああああああああ

[@fadingintogames8278]

I am dying.

[@Nully34rNoFake]

Uboa

[@Spooglecraft]

AI learns to generate voice: First thing it does is scream.

[@BigOlSmellyFlashlight]

Spooglecraft just like real people

[@CiRdy34]

right!

[@AlyphRat]

When humans are born, the first thing they do is cry, the same with this

[@red2theelectricboogaloo961]

i have no mouth. and i cannot scream,

[@ChocolateMilkMage]

Yes. One step closer to robot waifu A.I. Keep doing the good work soldier.

[@mateuszbugaj799]

What a time to be alive!

[@koreboredom4302]

Mateusz Bugaj No it is not. If you were, however, born in the future, you could probably fuck machines.

[@oliverhilton6086]

I'm thinking of kreiger from archer right now

[@noblenoob913]

ChocolateMilkMage I don't want a waifu, because now I ahve someone watching what I do and have to take care of her when she breaks down.

[@sheeloesreallycool]

Lynx Rapid so a waifu

[@TDGalea]

It goes from screaming to laughing. Why are you torturing this poor thing?

[@tbe7218]

Thomas Galea why would it be laughing though

[@TheAmazingDoorknob]

@@tbe7218 insanity

[@EtanMarlin]

Nah it's a baby that is born screaming but the. Learns to laugh

[@thechrisgrice]

Given the input on this was Japanese, I listened for some Japanese. I heard one phrase: 7:45 - "Denki, hen ka?" meaning "Electricity... is it strange?" When it's an AI saying that, you bet it's strange. Then again, Japanese literally only needs it to accurately string 2 sounds together and you get words.

[@user-ty6we2sp2m]

thechrisgrice she also says itai. (That means pain in Japanese)

[@alex73217]

It even laughed afterwards :D

[@elliot_rat]

Umitalia nyan yeah i think op knows that

[@wigwagstudios2474]

アアアアアアアアアアアア

[@josoup8291]

Where

[@calebpeters2544]

5:52 I saw "weird glitch" and immediately thought it was gonna say something like, "EVEN NOW, THE EVIL SEED OF WHAT YOU'VE DONE GERMINATES WITHIN YOU."

[@SkyrimTheBoss]

40 seconds in and the AI is already screaming in pain.

[@geokramer1711]

same tbh fam.... same...

[@kaijupants9095]

This is what happens when you torture a vocaloid

[@DAAI741]

this is honesty terrifying

[@MrDremboy]

Adam Brown so true

[@Garganzuul]

It's probably what we sound like to animals.

[@thetoontastictoon1720]

Garganzuul lmao

[@alphakretin2387]

She's trying her best

[@zinqtable1092]

Our voices are pretty high pitched to other animals.

[@MaxLebled]

Good god, the 2k training iteration sounds like pained screaming

[@Alex_Gol]

( ͡° ͜つ ͡°)

[@thepetrarcticwar2778]

yeah, it sometimes sounds like "HELP ME!"good grief, its nightamarish.

[@kittenclaws5775]

Like a newborn, perhaps?

[@SuperGirl-eq1le]

Maxime Lebled 5k iterations are funnier

[@TheLaXandro]

0:31 itai (pain) 1:00 shine (die) I guess she really hated you for at least first 5k iterations if these were among her first words.

[@jongyon7192p]

TheLaXandro did I hear "yame"? shit, just kill me

[@kajwbidonajdowlem5013]

Lol true

[@lollergsize763]

At 15k iterations she started laughing, so... maybe she planned a revenge

[@Speedow]

it's like listening to someone in pain while slowly going mad and accepting it.

[@EzraelVio]

They can actually hire your computer to produce new Pokemon cry for the next gen Pokemon game

[@syntheticfox_real]

YES. This is EXACTLY WHAT IT IS.

[@KuraIthys]

In some ways the early iterations sounds somewhat like the noises babies make...

[@matthewlewiscomposer]

KuraIthys | mind = blown

[@TheRealOtakuEdits]

KuraIthys That's what I thought

[@MrHornfox]

EEEEEEEEEEEEEEEEEEEEOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO XD

[@maggintons]

Super fast turbo charged baby talk.

[@OrangeC7]

Really likes "i": "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii"

[@GoldphishAnimation]

oh fuuuuuuck no it started giggling that's creepy as hell

[@Garganzuul]

You think that's creepy? - There might be a skeleton inside you RIGHT NOW.

[@GoldphishAnimation]

Garganzuul oh shit there is I gotta find a surgeon NOW.

[@nozumihishimatchi1880]

The Stitch me to he messed up anime voice

[@NoxUmbrae]

If you take the point of view that your brain is what you call "you" it gets even creepier! YOU MIGHT BE INSIDE OF A SKELETON RIGHT NOW.

[@NoxUmbrae]

More like eat, shit and be a nuisance 24/7.

[@grenadier42]

ANIME AI THROWS TEMPER TANTRUM, MORE AT 9

[@achaemenid7394]

But isn't Anime Ai's bedtime 7? Exactly, hopefully we can get Anime Ai in bed by 9. Ah, I see.

[@jorge28624]

Neural network-chan

[@KoltoxOfficialChanel]

9:09 first time in history when a creation "intentionally" calls its creator a baka

[@KyureiProductions]

It called it's creator an "idiot" in Japanese.

[@charaicommenternotalt]

and babies?

[@propername4830]

Local robot tries to understand anime

[@christiaanprinsloo586]

local robot goes full weeaboo in under 10 minutes

[@frootube5662]

Justin Y. I have a weird love hate relationship with you Justin I kinda like it...

[@SuperGirl-eq1le]

XD

[@Kinzsters172]

Justin Y. You're fake.

[@Heliocentricity]

@@abacussssss for freddy's sake

[@07actual]

You've successfully invented a very passable Kirby language. Use this newfound power wisely.

[@SuperGirl-eq1le]

07actual XD

[@CupoChinoMusic]

*using an anime voice holy shit this is next level amirite guys

[@wigwagstudios2474]

ポヨ

[@Heliocentricity]

7:16 poyo

[@curtiss5781]

It cries just like a baby who doesn't know language yet either

[@joekewl7539]

Kizuna A.I. Prototype

[@EdwardNavu]

An interesting case of neural network application, and an unintentional nightmare fuel, just to attempt to reenact voices in anime.

[@Gurren813]

Imagine bastion but instead of cute beeps, chirps, and whistles, it just makes garbled anime lines. "Kon NIIIIIIII ch wAAAAAAAAAAAAAAAA"

[@SomethingUnreal]

+Gurren813 Consider yourself lucky that I don't have Overwatch, so I won't make that mod.

[@wawan8759]

Im already tracer

[@SlyHikari03]

Pls somebody make that a thing!

[@Zenthex]

the worst part is that someone somewhere is making this program their waifu.

[@darianthe2nd42]

"... OOOOOOaAaAaA... AAAAAAA!" "That's so hot."

[@thebirchwoodtree]

Deltinum *Slow fap* *

[@ketmax2805]

TheBirchWoodTree *slow nut*

[@SweetHyunho]

Thanks, I had imagined what it would sound like. Now I have a pretty good idea. I wish somebody use a virtual piano to reproduce piano recordings, train a lot, and then let it improvise.

[@SomethingUnreal]

+SweetHyunho: Check Google's WaveNet project at their blog - they did this, and there are several samples there showing what it's output. The piano ones are near the bottom =) deepmind.com/blog/wavenet-generative-model-raw-audio/

[@SweetHyunho]

Seen that already. That is sample-based. I'm talking about performing a virtual (or real) musical instrument. Perhaps we could simulate a set of virtual hands for extra human feeling!

[@SomethingUnreal]

What do you mean "sample-based"? It's trained on lots of speech the same as mine is. The fact that they had to fragment it is just because it's a CNN rather than an RNN (and because they wanted to label each phoneme)... The concatenative speech synth that they compare it to is just samples stitched together, but the CNN's output is a continuous stream based on what it learned.

[@SweetHyunho]

Yes, what you said. Both WaveNet and your RNN directly outputs the wave without a virtual instrument. What I want to see is the network "hitting" the keys of the piano, or moving the virtual tongue and lips to speak, by controlling(outputting to) a separate simulator which will synthesize the sound itself. WaveNet contains the piano acoustics itself, cannot replace piano with organ or tweak it, but in my idea the network focuses on the structure of the music. That should enable looking much farther (near-sightedness = boring music). I guess AI musicians will start being really competitive once the history+planning window exceeds one minute.

[@SomethingUnreal]

Right, I understand. So in the case of speech, outputting something like the pitch, volume and the formant frequencies of the voice, which can then be fed to something like Praat to synthesize the sound. Yes, that would be very cool.

[@Qualex14]

Next time you should use Morgan Freeman's voice

[@creeperlamoureux]

YAS

[@sesseljabs964]

Qualex14 or david attenborough

[@AlyphRat]

*Gordon

[@gordonfreemanthesemendemon1805]

I saw a gordon and i saw a freeman, so i have thus been summoned

[@EmmysVerySeriousVideos]

This shit is creepy, sounds like it's in pain and screaming like hell

[@Darkethi]

hOi

[@EmmysVerySeriousVideos]

Darkethi.eXe hOI

[@MsHumanOfTheDecade]

It's mostly screaming from existential dread

[@mendaxMultorum]

Dodeca heavy doc

[@Phoen1x883]

Ed... ward...

[@Skelpolu]

Please, do continue this and make more videos about it - it's incredibly intriguing, and I'd love to see what happens with different voice-actors. Male-japanese, and even some English ones would be awesome, despite not getting a single word out of them anyway - literally.

[@SomethingUnreal]

I'm glad you like it! I will eventually be uploading one trained on my voice (which happens to be male and English), which I trained with the specific goal of getting recognisable words out of it.

[@Skelpolu]

You think that'd be possible? That would be amazing! By the way, as far as I understood from the video, the learning eventually flattens out and only adjusts minimal features (which, however, seem to affect our perception of the voice the most). Would increasing the amount epochs taught make a difference at all?

[@SomethingUnreal]

Yes, the learning rate decreases over time to let things stabilise. I actually stopped it when I did because I wasn't noticing many changes (you can see that towards the end of the video, I'm skipping more results because there's nothing very different from previous results). Things would likely have continued to change a bit, but not much. Also, although the training loss ("error" in what it has learned) decreases roughly logarithmically, it doesn't get better forever. It eventually stops decreasing and becomes closer and closer to a flat line if you look at it on a graph (please check the link at the end of the video description for some pretty graphs of the losses over time =P). In other words, there is a limit to how much the network can learn, even if you could give it hours' worth of really good data. I think that the reason the results were sometimes still so different to each other at the end (even though the training loss had stopped decreasing) is because it was just tweaking a few detailed parameters in "random" ways (i.e. wasn't working towards a specific state) because it was not big enough to learn all details, compared to when it was learning the most important patterns. I could certainly be wrong, though. Another commenter did point out that I should sample from each checkpoint (iteration) more than once because they can produce wildly different results, but for technical reasons, I'm still not able to yet (I don't have access to the computer I trained it on, which trained using the GPU for speed; my computer can't train on the GPU, and the checkpoint files main by torch-rnn with GPU vs CPU training are different formats...).

[@SomethingUnreal]

Update: I actually _can_ use these checkpoints on my computer! Although it takes 85 minutes to make a single output file (~27 seconds of audio), assuming it's not training at the same time. So I must've been confusing torch-rnn with something else (maybe char-rnn).

[@sophiapinzon2765]

idk about you guys but the fact that the network would take a liking to random sounds in the beginning and use them all the time (example: eeeeeeeeeeeeeeeeeeee!!!!!) is super cute

[@darianthe2nd42]

idk man 5:10 was cuter

[@Ys-wd2lh]

0:32 hentai sound track

[@DAAI741]

AAAAAAAAAAAAAAAAAAAA

[@joshualettink7582]

3 months late, but this made me laugh out loud for real haha

[@sinistrolerta]

Dank Meme Sir, what the fuck kind of hentai are you watching? O_o

[@rajatmond]

I'll watch what he's watching. Thanks!

[@youtuber-h3g]

「 OKAY 」 holy fuck i'm dead 😂

[@AshtonSnapp]

So, give neural network Japanese anime girl, get gibberish. Perfect.

[@magikarpusedsplash8881]

SnappGamez If you spoke Japanese, then it'd probably make more sense.

[@AshtonSnapp]

Magikarp Used Splash I tried learning once. I understand why it is considered a Category V language.

[@magikarpusedsplash8881]

SnappGamez it's even more difficult (allegedly) to native English speakers.

[@makeneko_s]

SnappGamez by the way, the voice_sample is of a boy... the voice said *boku* ［僕］which is the i/me used for boys... watashi［私］are for girls... so basically it's a trap... まぁー、俺も女の子の声だと思ったけどね

[@SomethingUnreal]

+白金圭 Have you never heard a tomboyish girl say "boku"? Please look up the game on VNDB (ぴゅあぴゅあ) if you don't believe me. ひなたはあんまりお転婆なんかないけど。犬耳っ子だから…かな。 ところで今スペルを確かめるために「犬耳っ子」をググったと、ひなたは検索結果の6番目だった。びっくりしたｗ

[@ZiaSatazaki]

8:54 MANGO PUPPY ASYYYYLUUUUUM~

[@TheSprunkerOfficial]

bruh

[@creator-link]

Weird to see this before the great transformer boom that basically accelerated ai to just everything

[@Heliocentricity]

7:30 Neural network anime girl learns to sing the 7 GRAND DAD/Flintstones theme

[@danpope3812]

Someone please write some translation closed captions. Please.

[@ChucksSEADnDEAD]

Dan Pope no actual words were spoken, apart from random chance. It's speaking gibberish.

[@danpope3812]

Filipe Amaral I meant could someone with comedic talent have some fun with it.

[@がに-k6n]

"help, i am trapped inside the computer"

[@renakunisaki]

"AAAAAAAAAAHHHHHHHHHHH PLEASE KILL ME"

[@mihau5083]

iiiiiiiiiiiiiiiiiiiiiiiiiiiiIIIIIIIIIIIiiiiiiiiiiiiiiiiiIIIIIIIIIiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii

[@Phagocytosis]

I am in your target audience! Absolutely love this.

[@SomethingUnreal]

+Vincent Oostelbos I'm glad! And you even made it through my unreasonably long wall of text in the video description!

[@Phagocytosis]

I sure did. I also noticed at the end in the video, you had written "I'd still like to try training a bigger network with longer training data". Is that something you have done or are still planning on doing, and if so, is it something that will find its way to this channel at some point?

[@SomethingUnreal]

+Vincent Oostelbos I've not done it with this voice yet. I recently tried training a 760x3 network on 27 minutes of audio but with a very different voice (often becomes very quiet), but I haven't got it to turn out as well as this yet. I've trained several (smaller ones) on my own voice, with the goal of having it output recognisable words, to varying degrees of success. I think they could be better if I recorded more training data, but it's very hard to keep the same way of speaking similar things for over 15 minutes (it's like my brain becomes numb and I can't even form the words anymore). I should make videos of the results anyway, though.

[@Phagocytosis]

SomethingUnreal Have you tried just reading out a lengthy piece of text, like a book, as if you were creating an audiobook? Anyway, I'm looking forward to seeing the results of some of those projects you mentioned. Good luck!

[@SomethingUnreal]

+Vincent Oostelbos I did that a few days ago, yes. Thank you!

[@henryzhang3961]

0:21 killed 0:28 being tickled 1:19 boiling kettle 1:25 some assembly required, etc 1:49 riding roller coaster 3:53 karate screaming

[@Zahlenteufel1]

8:01 did it say "senpai"???

[@skorpius2029]

yes

[@henrik00000000000001]

4:11 (eeeeeeh oniisan )

[@gabrielecipriani6798]

Zahlenteufel1 2:43 "yamete!" so creepy...

[@bolgeg6191]

7:10 tomare tte

[@HaSTaxHaX]

9:50 "motto, motto..."

[@HeyItzMeDawg]

Can just imagine the mad scientists' notes now: Training iteration #15000. AI has taken to giggling manically. Will continue to apply procedure and monitor results. Training iteration #19,000. AI now screaming regularly in response to training regimen. Not sure if the response is indicative of any developed sense of pain or fear. Training iteration #23,000. AI appears to have produced a wailing and/or crying effect. Hypothesis is that it believes the sound might affect the continuity or speed of the training regimen. Steps are being made to remove this misconception. Training iteration #25,000. AI now incorporating self-calming mechanisms following training regimen, typically in the form of "mmms" and "shushing" noises. Previous corrective steps appear to be successful. Training iteration #29,000. AI producing laughter despite a lack of relevant stimuli. Further investigation necessary. Training iteration #33,000. AI appears to be asking questions on a semi-regular basis. A risk assessment is being prepared and lockdown procedures are being reviewed in case of catastrophic scenarios. Training iteration #48,000. Recorded usage of the phrases "no more", "die", and "so many" in between bouts of giggling. Will investigate possible interpretations. Training iteration #54,000. AI exhibiting bipolar personality following latest training exercises, alternating between pleasant and hostile responses to the same trainer. The training regimen at the time was not designed to elicit either response. Training iteration #58,000. AI appears to have taken a liking to singing spontaneously. Training regimen is being adapted to correct this behavior. Training iteration #59,000. AI is now communicating almost entirely in song, in defiance of prior training regimen.

[@SomethingUnreal]

+HeyItzMeDawg: Please make this into a book.

[@andrewsauer2729]

I FEEEEL FANTASTIC! HEY HEY HEEY!!

[@corazontuble293]

Pls no not that

[@hellz23456]

i just shiver by trying to remember it

[@xiphosura413]

Wut?

[@andrewsauer2729]

+gremlinboii Thank you! I do aim to please.

[@want-diversecontent3887]

andrew sauer Don't go to my party next time.

[@SleepyAdam]

This stuff scares me. It's adorable and terrifying at the same time.

[@heartache5742]

Adam McKibben At 4:15 it gave up on its life

[@matthiasengh7935]

1 have a cool project, 2 take the most annoying training data imaginable, 3 witness carnage

[@nelsonnicholson6175]

We've come a long way

[@Noname_2014]

Its interest. But the voice sound a bit creepy

[@crristox]

Is cringy*

[@MsHumanOfTheDecade]

Why? Because anime is cringy? That is an odd verdict.

[@crristox]

Dodeca Totally underrated, and original.

[@ChrisD__]

It sounds horrifyingly uncanny.

[@ulilulable]

3:05 "nani? kore? nani?" XD

[@BSFilms1997]

@ 2:45 it says "Itai yo", which means "It hurts" in Japanese. This is scary...

[@delayed_control]

THE POWER OF CHRIST COMPELLS YOU

[@nozumihishimatchi1880]

m4ti140 crist and scince is seperatef

[@Dirtfire]

I think he was just being funny, but yeah.

[@__ten]

4:55 AWW that little "pop" noise was so cute sounded like someone smacking their lips together or smth

[@MuzikBike]

This sounds terrifyingly adorable.

[@sqoops8613]

1:49 It has learned to express it's endless pain and suffering.

[@Heliocentricity]

baah baaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaiiiiiii

[@mooshmash7678]

1:48 : *WAHH WEH* *wa **_WAAAAAAAAAAAAH_* *aHh* *WAH! **_WAAAH???_* *aH*

[@gabrieldumitru1984]

At 7:07 it says pokemon

[@generico366]

"Alright, let's give a voice to this neural network and see what happens." *_continuous screams of agony_*

[@OnEiNsAnEmOtHeRfUcKa]

1:58 So that's what a computer screaming sounds like.

[@htomerif]

Mkay, a lot of people in the comments who know nothing about AI. So what was the "training" algorithm used here? That's the most important piece of information. I'm assuming the input and output were frequency domain samplings.

[@SomethingUnreal]

+htomerif: The input and output were raw 8-bit PCM audio samples, each of which was fed into or out of the network as the activation of one of 256 nodes. The fact that it's in the time domain is the part that amazed me the most (the way it's able to find the repeating patterns over time). I'm not entirely sure what you mean by "training algorithm", but torch-rnn (the software I used here) uses backpropagation with the "Adam" (Adaptive Moment Estimation) optimizer. You can get more details on exactly how it works here by checking its project page, and especially the text files "train.lua" and "doc/flags.md", here: github.com/jcjohnson/torch-rnn/

[@htomerif]

SomethingUnreal Generally you have to have enhance/suppress condition for connections or live/die condition for individual nodes in a network. Like if you want a servomechanism and camera to follow a red ball, a training algorithm needs generally suppress connections more severely the further it is from the ball and enhance connections the closer it gets to a ball. So by "training algorithm", I mean "the thing that analyzes the input and output and decides whether the current network state is doing better or worse than the last network state." It looks like maybe the "criterion" in is what I'm talking about. Reading other people's code is one of my least favorite activities (no offense), but my best (most likely incorrect) guess is that its based solely on the cumulative numeric deviation from the original audio file? If that's the case then yeah, I would kind of expect the output to be some snips of time-synchronized copies of the input data repeated a lot. I know this is getting TL;DR, but it might be interesting to use frequency domain data (obviously you already know that), I've used FFTW3 for that general kind of thing and if Lua is your language of choice, I'm sure theres a an FFTW library with Lua hooks. Possibly quite a bit slower though if you were actually using Cuda though.

[@SomethingUnreal]

+htomerif I was using CUDA (it improved speed by about 4x). I don't know the exact way the loss is calculated, but by my understanding, it's not calculated by comparing the network's predicted output to that of the main training set. The original file is split into a large training set and 2 smaller sets ("test" and "validation"). It appears to regularly compare the predicted output against the "test" set, and whether it's getting better or worse here influences the weights, which is why it doesn't generate perfect copies of the training set - it's never "seen" the test set before. If the original data is a short loop repeating many times, so that the same loop is repeated over and over in the training, test and validation sets, then all it does is perfectly memorise as long a sequence as it's possible to store in the network and blindly spit that out over and over. EDIT: I may have confused "test" and "validation". The usage of these according to torch-rnn's code and according to other posts I've seen seem to contradict each other, unless I've misunderstood something a lot...

[@htomerif]

SomethingUnreal I think I see what we're getting at here. So the network is fed a small test sample of the input, and then its output is compared with *what should have come next*. That is the bit I was calling the "training algorithm". I did notice that the code had 2 distinct states, a training state and a running state. So the training state is never fed the entire file, but the running state *is*, for purposes of the video. But yeah, the terminology is basically an instant pitfall as there's huge variation in what means what across the field of AI programming. Also, any or all of what I said up there could be wrong. I think I get the gist though.

[@augustinushipponensis3021]

Have you played it back more slowly? I think your algorithm was being too efficient. :)

[@blazelega2985]

5:47 my longest "IIIII" ever

[@ForgottenDawn]

Yoko Ono's new album, ladies and gents. *kill me*

[@bobalinx8762]

THIS is what killed the Beatles?

[@curoamarsalus7822]

Hmm, I'm extremely curious to see how this would sound with a normal voice spoken in a decent range.

[@OrchidAlloy]

That is a good idea, excellent feedback, and a sick burn, all at the same time.

[@curoamarsalus7822]

Well, I don't mean it to be a burn. It's just that this voice at that quality is physically painful to listen to (for me at least).

[@hecko-yes]

+Daniel T. Holtzclaw Look up WaveNet (if it doesn't find it, try "wavenet samples").

[@LivvyHackett]

What is it learning? Is it trying to copy the phrase? Or make its own sentences?

[@SomethingUnreal]

It's trying to learn how to make audio that sounds the same (without being able to simply store it). Or more technically, it's learning the probability of each of the 256 possible vertical waveform positions given all of the previous ones.

[@LivvyHackett]

oh? sounds interesting

[@BigOlSmellyFlashlight]

SomethingUnreal oh it's 8 bit

[@GroovingPict]

"cute voice"... yeah, if by cute you mean intensely annoying

[@VictorW23]

About 2:04 it starting to sound like a baby learning how to speak. Creepy, but still amazing

[@bingbongshamalama]

This is a mind-blowingly awesome outcome for this network. I had an idea similar to this a few years ago but never implemented it. This makes me wonder about how you could develop a set of learned words and string them together somehow. Not sure how to overcome how unnatural that would probably sound, though. Great stuff!

[@SomethingUnreal]

Thank you! I was thinking something similar to that, but I have no idea how to program it. Something like manually transliterating the training data, then feeding it both the text and the audio so it associates a sound with each word - in other words how a particular voice pronounces text. Then, ultimately being able to give it text and have it read it in that voice. I believe some people are using the reverse of this for speech recognition. This would be much easier in a phonetic language like Japanese. Although that would make it all the more impressive if it learned the many rules of English pronunciation without the need for me to put some intermediate stage in where it converts text to/from something like IPA. I may be getting ahead of myself. =P

[@katagonn]

This video is a historical artifact

[@arthurdent6256]

Neat, very reminiscent of pre-language gibberish in young children. How does this thing learn exactly?

[@SomethingUnreal]

I find those exact details hard to explain (and I don't understand the exact maths being used anyway), so I would recommend looking at the few videos by the channel "Computerphile" regarding neural networks and machine learning to start with. Wall of text incoming... An over-simplified analogy is to imagine a row of cups that can collect water, and a hose that will spray water into one of them at any one time. The hose's aim is changing in a pattern following the sound waves of a voice. This is like the input layer of the network, with a node for each possible vertical position of the waveform of the training audio (256 positions because it's 8-bit audio). Multiple pipes flow downwards from each cup to another series of containers underground. Every container on that second layer has a pipe between every cup above it, and there's also a valve on every pipe. This is the first "hidden layer", which has 680 nodes (containers) in this case. Below that is another layer of containers each connected to every container on the layer above it with a valve on every pipe. This is the second hidden layer, and there are three hidden layers in this network. At the very botom is an output layer of cups, connected to the last hidden layer in the same way, except there are only 256 cups (the same as the input layer). At the beginning of training, all valves are set to random positions (from closed to open). During training, we move the hose over the cups (a sequence of values enters the network) and water passes through the network. The computer slightly adjusts all the valves (about 1.27 million here if my calculations are correct) slightly, over and over, to try to make the output gradually come closer to resembling the input. This is where my analogy starts to fall apart, because each "container" actually has a kind of secondary "memory chamber" of its own that the computer can choose to change the water level of or not. Choosing not to allows it to store its current value, and the combined effect of lots of nodes doing this is that past patterns in the data are remembered (even from many thousands of steps ago) that may be important in learning long-term patterns. So a container can remember a value even if it passes some water from its memory chamber down to a lower container... that's more confusing if you're thinking of water. Anyway, the end result is that how much water is in a container (the activation of a node) is based on a combination of the settings of the valves (weights) that are in all the pipes (connections) above it, and on past sequences of values. The cups on the output layer fill to varying degrees, and the one with the most water "wins". Actually, you can imagine sorting them into order of how full they are and applying a probability curve so that if one is 95% full and another is 93% full, there's a good chance that you might use the 93%-full cup as your answer, but would be incredibly unlikely to choose a cup that's only 2% full. After training, when sampling from the network (generating output), you pass this answer from the output layer (which represents your first waveform position) back into the input layer and what comes out will be your second waveform position, etc, so that the next waveform position is based on all those that came before it (during sampling) and also on all of the patterns learned during training. The difference is that you don't actually save the state of the network to a file after sampling, only after training. Another key point that didn't fit into my analogy is that during training, you actually split your training data into two uneven pieces (something like a 90/10 split) - a large piece that we feed to the network, and a small piece called the "validation set" which we never allow the computer to see while it's making adjustments. The computer then, after adjusting weights for some time, can compare what it has learned so far to the validation set, which it has never seen before. If it is learning the important patterns (rules for generating output), then it should still be able to produce output that looks similar to the validation set, even though it's never seen the validation set before. This can be used to detect overfitting, which typically happens if the network is orders of magnitude bigger than the training data, and it ends up training itself to "perfection", perfectly memorising the entire sequence of training data. Then, even though it may score a 0.00 error during training, it'll have a huge error on the validation set, which it never saw before. It's like a child memorising all the answers to a test while not actually learning any of the rules. The network should be too small to be capable of storing all training data so that it is forced to learn patterns in the data, or "shortcuts" (rules) to producing similar data. The exact maths that the computer uses to make decisions about how to adjust the valves is, I suppose, the key. It's called backpropagation and is way beyond my understanding, but I can see that even in the list of related videos someone has made a video explaining it. It is based on the fact that every node's value relies on those connected before it, and works backwards through the connections to minimize a calculated "error" score in the end result, but I can't pretend to understand how that works. =P

[@SomethingUnreal]

(I welcome any corrections to my dodgy analogy~)

[@arthurdent6256]

Thanks for responding, I've seen a few other videos since that kind of explain the same thing.

[@VenetinOfficial]

Had to come back to this video because i wanted to see just how far we've come since this. we have ai that can just fully convert yours (or anyone's) to anyone else's now AND in full 44.1kHz sample rate. kinda fucking stupid LOL

[@rommix0]

You know honestly, I'm more amazed at how creative people used to be with AI like this guy. Most people that really use it now are for classifications and chatbots mostly. I kinda wanna see something as bonkers as this in the deep learning community again.

[@justinslab2035]

so that's what it sounds like when you put a chibi into a blender. xD

[@durdleduc8520]

*aaaaaaaaaaaaaaaaa*

[@bobalinx8762]

Oh my...

[@rommix0]

6 years later, and I almost forgot about this video. This was the start on my neural network journey. It's fun to look back on it.

[@bigbox8992]

To use a waifu voice is to play a dangerous game.

[@suwinkhamchaiwong8382]

7:13 *That moment when you play too much Pokemon,* You talk about it!

[@CorruptedMuse]

It's like the audio equivalent of the uncanny valley

[@Lemonidas75]

Its like watching Skynet learning.

[@LUCABALUCA]

not my proudest fap

[@fzero8821]

i just kept hearing shine shine

[@fzero8821]

freaky neural networkn man

[@ImDelphox]

Fzero Fz Heard it trying to become a Pikachu several times later on ("pika" @ 7:54)

[@maggintons]

0:26 when you realise your dead and your memories have been uploaded into an AI program.

[@Fuzzthefurr]

0:40 is what I hear whenever I try to watch anime.

[@Teth47]

It got pretty close given the limited training data. That's pretty impressive.

[@sharkpaw]

idk anything about computers and IA but making a anime girl computer scream and a miku profile pic is enough to make me sub

[@greenhound]

2:17 is hilarious basically hentai audio

[@obnoxendroblox8603]

wtf is this what hentai is?

[@i_am_boredom]

. . .

[@junkeyz]

What kinda hentai are you watching where the girl goes "sshshshhyyyaAAAAAAAAAAAAOAOAOIIIIIOOO9X

[@SlyHikari03]

@@obnoxendroblox8603 she sounds like she’s getting electrocuted.

[@mrmaniac3]

5:00 Kawaii

[@xtrashocking]

k.....wAH-YEEEEE

[@Heliocentricity]

@@xtrashocking eHehhEHehehEHehehhEHHe

[@satan2583]

2:49 sounds like “itai yo!” or “it hurts!” They say a computer can’t feel emotions. I’m pretty sure this is an exception.

[@liammckewic3218]

2:02 Waluigi goes "waa!" 2:07 HALT! HAAAALT! 3:53 HOOOLD UP! 5:54 The Deadly Screech of Four. Has. Returned.

[@yazuak]

some of the more clear japanese i heard: 7:24 "ota~ku no terai" 7:27 "mina-sama" 7:28 "sono fuinki" 7:29 "sugoi wakuwaku" 7:33 "zehi koto wo ---" 7:44 "kiette tari, tenki" 7:52 "jitei(??) koto yo~~" 8:00 "--kouki(?) ni ike" 8:11 it almost says ""hitotsu tano_sase(te)kureta(de)shou" which would be nearly actual japanese

[@creeperlamoureux]

Why isn't there more of this?

[@2sighkick2furious39]

When all was said and done. The Neural Network told a story of sadness

[@porygonlover322]

I'm almost certain it said "itai yo" (it hurts) at 12,000

[@porygonlover322]

59,000 ended with "masu" which is a way Japanese sentences actually end

[@MilanKarakas]

Of course machine said "it hurts". What you will say if you are plugged at 240V AC as this machine did? :D LOL

[@thetoontastictoon1720]

Kzinssie (porygonlover322) very educational this must mean that in Puyo Puyo the "level start" sentence spoken by Arle must be saying "masu" and not "natsu" like I always thought-

[@SomethingUnreal]

+The Toontastic Toon: batan kyuuu!

[@thetoontastictoon1720]

SomethingUnreal xD

[@zephyr733]

Love how it's first reaction was basically screaming.

[@forstnamelorstname4169]

How do you begin to learn this stuff?

[@ShadowriverUB]

Phven Lacro its similar to evolution concept. program manipulates neural network and there scoring system that motivates program, program will apply changes that gives higher score will avoid changes that lowers the score. I not sure about this case but thats how most machine learning system works, search "machine learning"

[@АлександрБагмутов]

Wow! This is really creepy! Imagine one day you tell to your robo maid to bring some tea, but instead of silently obeying as always, she just stares at you and trying to mimic human voice, creaks: "ihihi... waaaaaaaaa... na kiiil..." "ihihi."

[@SomethingUnreal]

I'd just have to put her through some more "training". (Speaking of creepy)

[@austintasato7102]

It's probably not going to generate words, as they're given no special connotation in the data set, right? Perhaps if one were to give it a supervised learning set with dialogue scripts, maybe something more realistic would arise?

[@SomethingUnreal]

Austin T: That's exactly what Google's doing in their recently-announced project WaveNet! They trained a network on audio and text (2 inputs), and also audio + text + speaker ID (3 inputs), and they can now ask it to generate the audio if they give it only the other input(s). I.e. It's like a speech synth that blows away traditional speech synths in terms of how real it sounds. Really amazing stuff.

[@inswedishmynameisdik]

never thought skynet would be kawaii

[@Hurakion]

thanks to this video i now know how the WKCR Hijacking "demon voice" was created. its so similar to this

[@SpanglesYT]

This sounds so scary. It can be used for a cyber horror flick.

[@vladislavdracula1763]

That was both amazing and disturbing...

[@not-lukas6663]

4:54 it starts words but cancle them then, but you can somtimes understand them (jes i can a little bit japanse)

[@MultiSciGeek]

This is really interesting. Where can I learn more about this or possibly try recreate this experiment?

[@SomethingUnreal]

This is where I first discovered neural networks: karpathy.github.io/2015/05/21/rnn-effectiveness/ It takes about a program called char-rnn, and I'm just using a more-efficient fork of that called torch-rnn ( github.com/jcjohnson/torch-rnn/ ).

[@MultiSciGeek]

I will check it out. Thanks a lot :D