Word Embedding and Word2Vec, Clearly Explained!!!

[@statquest]

To learn more about Lightning: lightning.ai/ NOTE: A lot of people ask for the math at 13:16 to be clarified. In that example we have 3,000,000 inputs, each connected to 100 activation functions, for a total of 300,000,000 weights on the connections from the inputs to the activation functions. We then have another 300,000,000 weights on the connections from activations functions to the outputs. 300,000,000 + 300,000,000 = 2 * 300,000,000 Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/

[@karanacharya18]

In simple words, word embeddings is the by-product of training a neural network to predict the next word. By focusing on that single objective, the weights themselves (embeddings) can be used to understand the relationships between the words. This is actually quite fantastic! As always, great video @statquest!

[@statquest]

bam! :)

[@joeybasile545]

Not necessarily just the next word. Your statement is specific.

[@NoNonsense_01]

Probably the most important concept in NLP. Thank you explaining it so simply and rigorously. Your videos are a thing of beauty!

[@statquest]

Wow, thank you!

[@chad5615]

Keep up the amazing work (especially the songs) Josh, you're making live easy for thousands of people !

[@statquest]

Wow! Thank you so much for supporting StatQuest! TRIPLE BAM!!!! :)

[@exxzxxe]

Josh; this is the absolutely clearest and most concise explanation of embeddings on KZbin!

[@statquest]

Thank you very much!

[@davins90]

totally agree

[@rachit7185]

This channel is literally the best thing happened to me on youtube! Way too excited for your upcoming video on transformers, attention and LLMs. You're the best Josh ❤

[@statquest]

Wow, thanks!

[@MiloLabradoodle]

Yes, please do a video on transformers. Great channel.

[@statquest]

@@MiloLabradoodle I'm working on the transformers video right now.

[@liuzeyu3125]

@@statquest Can't wait to see it!

[@SergioPolimante]

Statquest is by far the best machine learning Chanel on KZbin to learn the basic concepts. Nice job

[@statquest]

Thank you!

[@pushkar260]

That was quite informative

[@statquest]

BAM! Thank you so much for supporting StatQuest!!! :)

[@harin01737]

I was struggling to understand NLP and DL concepts, thinking of dropping my classes, and BAM!!! I found you, and now I'm writing a paper on neural program repair using DL techniques.

[@statquest]

BAM! :)

[@aoliveiragomes]

Thanks!

[@statquest]

BAM!!! Thank you so much for supporting StatQuest!!! :)

[@mannemsaisivadurgaprasad8987]

On of the best videos I've seen till now regarding Embeddings.

[@statquest]

Thank you!

[@JawadAhmadCodes]

Oh my Gosh, StatQuest is surely the greatest channel I found to learn the whole universe in simple way. WOW!

[@statquest]

Thank you! :)

[@ashmitgupta8039]

Was literally struggling to understand this concept, and then I found this goldmine.

[@statquest]

Bam! :)

[@yuxiangzhang2343]

So good!!! This is literally the best deep learning tutorial series I find… after a very long search on the web!

[@statquest]

Thank you! :)

[@pichazai]

this channel is the best resource of ML in the entire internet

[@statquest]

Thank you!

[@manuelamankwatia6556]

This is by far the best video on embeddings. A while university corse is broken down in 15minutes

[@statquest]

Thanks!

[@myyoutubechannel2858]

In the first 19 seconds my mans explains Word Embedding more simply and elegantly than anything else out there on the internet.

[@statquest]

Thanks!

[@mycotina6438]

BAM!! StatQuest never lie, it is indeed super clear!

[@statquest]

Thank you! :)

[@tanbui7569]

Damn, when I first learned about this 4 years ago, it took me two days to wrap my head around to understand these weights and embeddings to implement in codes. Just now, I need to refreshe myself the concepts since I have not worked with it in a while and your videos illustrated what I learned (whole 2 days in the past) in just 16 minutes !! I wished this video existed earlier !!

[@statquest]

Thanks!

[@TropicalCoder]

That was the first time I actually understood embeddings - thanks!

[@statquest]

bam! :)

[@acandmishra]

your work is extremely amazing and so helpful for new learns who want to go into detail of working of Deep Learning models , instead of just knowing what they do!! Keep it up!

[@statquest]

Thanks!

[@noadsensehere9195]

This is the only video I was finding to understand this basic concept for NLP! tHANKS

[@statquest]

Thanks!

[@user-wr4yl7tx3w]

This is the best explanation of word embedding I have come across.

[@statquest]

Thank you very much! :)

[@awaredz007]

Wow!! This is the best definition I have ever heard or seen, of word embedding. Right at 09:35. Thanks for the clear and awesome video. You guy rock!!

[@statquest]

Thanks! :)

[@haj5776]

The phrase "similar words will have similar numbers" in the song will stick with me for a long time, thank you!

[@statquest]

bam!

[@DanielDias-vl2js]

Thank goodness I found this channel! You've got great content and an excellent teaching methodology here!

[@statquest]

Thanks!

[@wizenith]

haha I love your opening and your teaching style! when we think something is extremely difficult to learn, everything should begin with singing a song, that make a day more beautiful to begin with ( heheh actually I am not just teasing lol, I really like that ) thanks for sharing your thoughts with us

[@statquest]

Thanks!

[@ananpinya835]

StatQuest is great! I learn a lot from your channel. Thank you very much!

[@statquest]

Glad you enjoy it!

[@rathinarajajeyaraj1502]

This is one of the best sources of information.... I always find videos a great source of visual stimulation... thank you.... infinite baaaam

[@statquest]

BAM! :)

[@gustavow5746]

the best video I saw about this topic so far. Great Content! Congrats!!

[@statquest]

Wow, thanks!

[@FullStackAmigo]

Absolutely the best explanation that I've found so far! Thanks!

[@statquest]

Thank you! :)

[@dreamdrifter]

Thank you Josh, this is something I've been meaning to wrap my head around for a while and you explained it so clearly!

[@statquest]

Glad it was helpful!

[@flow-saf]

This video explains the source of the multiple dimensions in a word embedding, in the most simple way. Awesome. :)

[@statquest]

Thanks!

[@EZZAHIRREDOUANE]

Great presentation, You saved my day after watching several videos, thank you!

[@statquest]

Glad it helped!

[@muthuaiswaryaaswaminathan4079]

Thank you so much for this playlist! Got to learn a lot of things in a very clear manner. TRIPLE BAM!!!

[@statquest]

Thank you! :)

[@mazensaaed8635]

I promise I'll be member in your channel when I get my first data science job

[@statquest]

BAM! Thank you very much! :)

[@RaynerGS]

I admire your work a lot. Salute from Brazil.

[@statquest]

Muito obrigado! :)

[@exxzxxe]

Hopefully everyone following this channel has Josh's book. It is quite excellent!

[@statquest]

Thanks for that!

[@wellwell8025]

Way better than my University slides. Thanks

[@statquest]

Thanks!

[@michaelcheung6290]

Thank you statquest!!! Finally I started to understand LSTM

[@statquest]

Hooray! BAM!

[@lfalfa8460]

I love all of your songs. You should record a CD!!! 🤣 Thank you very much again and again for the elucidating videos.

[@statquest]

Thanks!

[@alexdamado]

Thanks for posting. It is indeed a clear explanation and helped me move forward with my studies.

[@statquest]

Glad it was helpful!

[@ajd3fjf4hsjd3]

Fantasticly simple, and complete!

[@statquest]

Thanks!

[@mahdi132]

Thank you sir. Your explanation is great and your work is much appreciated.

[@statquest]

Thanks!

[@MarvinMendesCabral]

Hey Josh, i'm a brazilian student and i love to see your videos, it's such a good and fun to watch explanation of every one of the concepts, i just wanted to say thank you, cause in the last few months you made me smile beautiful in the middle of studying, so, thank you!!! (sorry for the bad english hahaha)

[@statquest]

Muito obrigado!!! :)

[@mamdouhdabjan9292]

Hey Josh. A great new series that I, and many others, would be excited to see is bayesian statistics. Would love to watch you explain the intricacies of that branch of stats. Thanks as always for the great content and keep up with the neural-network related videos. They are especially helpful.

[@statquest]

That's definitely on the to-do list.

[@mamdouhdabjan9292]

@@statquest looking forward to it.

[@ah89971]

When I watched this,I have only one question which is why all the others failed to explain this if they are fully understood the concept?

[@statquest]

bam!

[@rudrOwO]

@@statquest Double Bam!

[@meow-mi333]

Bam the bam!

[@eqe-kui-nei]

@@ah89971 A lot people in this industry (even with a phd) actually dont.

[@channel_SV]

It's so nice to google and realize that there is a StatQuest about your question, when you are certain of that there hadn't been one some time before

[@statquest]

BAM! :)

[@fouadboutaleb4157]

Bro , i have my master degree in ML but trust me you explain it better than my teachers ❤❤❤ Big thanks

[@statquest]

Thank you very much! :)

[@mykolalebid6279]

Thank you for your excellent work. A video on negative sampling would be a valuable addition.

[@statquest]

I'll keep that in mind.

[@m3ow21]

I love the way you teach!

[@statquest]

Thanks!

[@alfredoderodt6519]

You are a beautiful human! Thank you so much for this video! I was finally able to understand this concept! Thanks so much again!!!!!!!!!!!!! :)

[@statquest]

Glad it was helpful!

[@amirhasansafizade2128]

Great as always. Thanks Josh

[@statquest]

Thanks again!

[@familywu3869]

Thank you very much for your excellent tutorials! Josh. Here I have a question, at around 13:30 of this video tutorial, you mentioned to multiply by 2. I am not sure why 2? I mean if there are more than 2 outputs, will we multiply the number of output nodes, instead of 2? Thank you for your clarification in advance.

[@statquest]

If we have 3,000,000 words and phrases as inputs, and each input is connected to 100 activation functions, then we have 300,000,000 weights going from the inputs to the activation function. Then from those 100 activation function, we have 3,000,000 outputs (one per word or phrase), each with a weight. So we have 300,000,000 weights on the input side, and 300,000,000 weights on the output side, or a total of 600,000,000 weights. However, since we always have the same number of weights on the input and output sides, we only need to calculate the number of weights on one side and then just multiply that number by 2.

[@surojit9625]

@@statquest Thanks for explaining! I also had the same question.

[@jwilliams8210]

Ohhhhhhhhh! I missed that the first time around! BTW: (Stat)Squatch and Norm are right: StatQuest is awesome!!

[@natuchips98]

You literally saved my life

[@statquest]

bam! :)

[@saisrisai9649]

Thank you Statquest!!!!

[@statquest]

Any time!

[@vpnserver407]

highly valuable video and book tutorial, thanks for putting this kind of special tuts out here .

[@statquest]

Glad you liked it!

[@danish5326]

Thanks for enlightening us Master.

[@statquest]

Any time!

[@eamonnik]

Hey Josh! Loved seeing your talk at BU! Appreciate your videos :)

[@statquest]

Thanks so much! :)

[@周子懿-y5r]

Thank you Josh for this great video. I have a quick question about the Negative Sampling: If we only want to predict A, why do we need to keep the weights for "abandon" instead of just ignoring all the weights except for "A"?

[@statquest]

If we only focused on the weights for "A" and nothing else, then training would cause all of the weights to make every output = 1. In contrast, by adding some outputs that we want to be 0, training is forced to make sure that not every single output gets a 1.

[@bancolin1005]

BAM! Thanks for your video, I finally realize what the negative sampling means ~

[@statquest]

Happy to help!

[@ramzirebai3661]

Thank you so much Mr.Josh Starmer, you are the only one that makes ML concepts easy to understand Can you , please , explain Glove ?

[@statquest]

I'll keep that in mind.

[@ColinTimmins]

Thank you so much for these videos. It really helps with the visuals because I am dyslexic… Quadruple BAM!!!! lol 😊

[@statquest]

Happy to help!

[@LakshyaGupta-ge3wj]

Absolutely mind blowing and amazing presentation! For the Word2Vec's strategy for increasing context, does it employ the 2 strategies in "addition" to the 1-Output-For-1-Input basic method we talked about in the whole video or are they replacements? Basically, are we still training the model on predicting "is" for "Gymkata" in the same neural network along with predicting "is" for a combination of "Gymkata" and "great"?

[@statquest]

Word2Vec uses one of the two strategies presented at the end of the video.

[@ericvaish8841]

Great explanation my man!!

[@statquest]

Thank you!

[@denismarcio]

Extremamente didático! Parabéns.

[@statquest]

Muito obrigado! :)

[@wenqiangli7544]

Great video for explaining word2vec!

[@statquest]

Thanks!

[@AliShafiei-ui8tn]

the best channel ever.

[@statquest]

Double bam! :)

[@ar_frz]

This was lovely! thank you.

[@statquest]

Thanks!

[@MannyBernabe]

Great work. Thank you.

[@statquest]

Thanks!

[@nouraboub4805]

‏‪goood, thenk you so much for this playlist is the best ❤️😍

[@statquest]

Glad you enjoy it!

[@vicadegboye684]

Thanks sooooo much for your videos. Let me not belabor the praise as it's been established that you are triple bam! 🙂 Meanwhilt, I've understood every single thing in your deep learning series up till this video. I'm still a bit confused about the negative sampling thing. I don't understand the idea of how using "aardvark" to predict "a" and "abandon" somehow means we are excluding "abandon". The concept is the only thing I've not understood in the 17 videos of this neural network/deep learning playlist. I would appreciate your help.

[@statquest]

The idea is that there is one word for which we want the final output value to be 1 and everything else needs to be 0s. However, rather than focusing on every single output, we just focus on the one word that we want the output to be 1 and just a handful of words that we want the output to be 0, rather than all of them.

[@oliverlee2819]

@@statquest So does this mean this negative sampling is implemented in each round of backpropagation optimization? I am also not quite sure about this part either. I guess a more detailed (but simplified) demo clarify this concept better. Or maybe some articles to reference?

[@statquest]

@@oliverlee2819 Yes, you do negative sampling every single time.

[@oliverlee2819]

@@statquest So the word that "we don't want to predict", means the words that we just want their predicted output value (prob) to be zero right? Is this done via teacher forcing method to force the output of one word to be 1, and the words that we don't want to predict to be zero?

[@statquest]

@@oliverlee2819 The first part is correct. The second part is a little off. This isn't technically teacher forcing. We're just focusing on the 1 word we want the output to be 1 and a handful of words we want the output to be 0.

[@pedropaixaob]

This is an amazing video. Thank you!

[@statquest]

Thanks!

[@avishkaravishkar1451]

For those of you who find it hard to understand this video, my recommendation is to watch it at a slower pace and make notes of the same. It will really make things much more clear.

[@statquest]

0.5 speed bam!!! :)

[@janapalaswathi4262]

Awesome explanation..

[@statquest]

Thanks!

[@auslei]

Love this channel.

[@statquest]

Glad to hear it!

[@tomoki-v6o]

My favourite topic its magic. Bam!!

[@statquest]

:)

[@pakaponwiwat2405]

Wow, Awesome. Thank you so much!

[@statquest]

You're very welcome!

[@ishaqpaktinyar7766]

you da bessssst, saved me alota time and confusion :..)

[@statquest]

Thanks!

[@minhmark.01]

thanks for your tutorial!!!

[@statquest]

You're welcome!

[@yasminemohamed5157]

Awesome as always. Thank you!!

[@statquest]

Thank you! :)

[@MadeyeMoody492]

Great video! Was just wondering why the output of the softmax activation at 10:10 are just 1 and 0s. Wouldn't that only be the case if we applied ArgMax here not SoftMax?

[@statquest]

In this example the data set is very small and, for example, the word "is" is always followed by "great", every single time. In contrast, if we had a much larger dataset, then the word "is" would be followed by a bunch of words (like "great", or "awesome" or "horrible", etc) and not followed by a bunch of other words (like "ate", or "stand", etc). In that case, the soft max would tells which words had the highest probability of following is and we wouldn't just get 1.0 for a single word that could follow the word 'is'.

[@MadeyeMoody492]

@@statquest Ohh ok, that clears it up. Thanks!!

[@study-tp4ts]

Great video as always!

[@statquest]

Thanks again!

[@jingzhouzhao8609]

Great Video in high quality!! Just wondering "times 2" at 13:27, because I saw 4 neurons in output layers, so not "times 4"?

[@statquest]

A lot of people ask for the math at 13:16 to be clarified. In that example we have 3,000,000 inputs (only the first 4 are shown...), each connected to 100 activation functions, for a total of 300,000,000 weights on the connections from the inputs to the activation functions. We then have another 300,000,000 weights on the connections from activations functions to the outputs (only 4 outputs are show, but there are 3,000,000). 300,000,000 + 300,000,000 = 2 * 300,000,000

[@gabrielrochasantana]

Amazing lecture, congrats. The audio was also made from an NPL (Natural Language Processing), right?

[@statquest]

The translated overdubs were.

[@MaskedEngineerYH]

Keep going statquest!!

[@statquest]

That's the plan!

[@c.nbhaskar4718]

great stuff as usual ..BAM * 600 million

[@statquest]

Thank you so much! :)

[@lexxynubbers]

Machine learning explained like Sesame Street is exactly what I need right now.

[@statquest]

bam!

[@The-Martian73]

mr.Starmer I think you really loved Troll 2 😅

[@statquest]

:)

[@akashbarik5806]

@statquest "5:30" I'm not sure if I'm right, but after researching a bit I found out that the number of activation functions have nothing to do with the number of associations with each word, The number of activation functions depend upon the structure of your neural network, and the number of vector representations solely depend upon how you want to embed the words. In simple terms, you can have a 3 vector representation of a word and use only 2 activation functions. I may be wrong but thats what I found out.

[@statquest]

To create 3 embedding values per input with only 2 activation functions, you could connect all of the inputs to 1 activation function and put 1 weight on each input, but then you'd need to connect all of the inputs to the other activation function and use 2 weights for each input. The problem with that second activation function is that input * w1 * w2 = input * (w1 * w2) = input * w3. so I believe you'd end up with the equivalent of just 2 embedding values per input in the end. I believe this is why neural networks are always designed to have one weight per input per activation function.

[@akashbarik5806]

@@statquest Thanks alot for the clarification

[@aniketsakpal4969]

Just incredible!

[@statquest]

Thank you!

[@tupaiadhikari]

Great Explanation. Please make a video on how do we connect the output of an Embedding Layer to an LSTM/GRU for doing classification for say Sentiment Analysis

[@statquest]

I show how to connect it to an LSTM for language translation here: kzbin.info/www/bejne/gmmrfKqbj66Co8k

[@tupaiadhikari]

@@statquest Thank You Professor Josh !

[@mgeich5]

Thanks for the great video as always. One question about the vocabulary size, 3M sounds like a huge vocabulary for today standards, is that the actual number?

[@statquest]

These days vocabularies are in the 30,000 range.

[@shamshersingh9680]

Hi Josh, again the best explanation for the concept. However, I have a doubt. As per the explanation, word-embeddings are the weights associated with each word between the input and activation function layer. These weights are obtained after training on large text corpus like wikipedia. When I train another model using these embeddings on another set of data, the weights (embeddings) will change during back-propagation while training. So the embeddings will not remain same and change with every model we train. Is it correct interpretation or I am missing something here.

[@statquest]

When you build a neural network, you can specify which weights are trainable and which should be left as is. This is the basis of "fine-tuning" a model - just training specific weights rather than all of them. So, you can do that. Or you, you can just start from scratch - don't pre-train the word embeddings, but train them when you train everything else. This is what most large language models, like ChatGPT, do.

[@neemo8089]

Thank you so much for the video! I have one question, at 15:09, why we only need to optimize 300 steps? For one word with 100 * 2 weights? not sure how to understand the '2' as well.

[@statquest]

At 15:09 there are 100 weights going from the word "aardvark" to the 100 activation functions in the hidden layer. There are then 100 weights going from the activation functions to the sum for the word "A" and 100 weights going from the activation functions to the sum for the word "abandon". Thus, 100 + 100 + 100 = 300.

[@neemo8089]

Thank you!@@statquest

[@hasansoufan]

Thanks ❤

[@statquest]

:)

[@NewMateo]

Great vid. So your going to do a vid on transformer architectures? That would be incredible if so. Btw bought your book. Finished it in like 2 weeks. Great work on it!

[@statquest]

Thank you! My video on Encoder-Decoders will come out soon, then Attention, then Transformers.

[@thomasstern6814]

@@statquest When the universe needs you most, you provide

[@ang3dang2]

Can you do one for wav2vec? It seemingly taps on the same concept as word2vec but the equations are so much more complex.

[@statquest]

I'll keep that in mind.

[@stevesamson-p2k]

funny and very nicely explained.

[@statquest]

Thanks! 😃

[@josephorero4552]

Great 👍

[@statquest]

Thanks!