torch.nn.Embedding explained (+ Character-level language model)

[@yuehpo-peng]

It's so satisfying that you type anything.

[@the_osbm]

I love this man, and your keyboard sounds are the cherry on top.

[@mildlyoverfitted]

Hehe:) Thank you!

[@blaine_stl]

@@mildlyoverfitted what kind of keyboard are you using?

[@DikkeHamster]

Cool to see the embedding evolution at the end!

[@mildlyoverfitted]

Heh:) Glad you find it cool! Thank you!

[@TheRovardotter]

At first, I did not understand the typing as sped up. Was very impressed by your typing skills.

[@mildlyoverfitted]

Haha:D Just for the record I am not trying to make it look as if I was a fast typist. I speed up the typing parts because otherwise the videos would get ridiculously long:)

[@TheRovardotter]

@@mildlyoverfitted nice vid by the way!

[@navins2246]

Doing ML in vim is absolutely gigachad

[@mildlyoverfitted]

Hahaha:D

[@vijayabhaskar-j]

Please tell me you're speeding up the portion of the video where you type, if not you're making others look like peasants in typing.

[@mildlyoverfitted]

LOL! Of course I am speeding them up! ~3x when writing code, ~6x when writing docstrings. Otherwise the videos would get too long!

[@namanjaswani8925]

What an explaination !

[@mildlyoverfitted]

Glad you found it useful!

[@culturosityofficial]

Seeing the steps to how the embeddings worked under the hood got me through my assignment, thank you!

[@kajalchaurasia950]

Just a thought Create a model for your videos. Input : Raw video Output : Normal speed where you are speaking and fast forward where you're typing.

[@mildlyoverfitted]

Not a bad idea actually!

[@fuat7775]

Thank you for this video, very clear. BTW, you can use higer dimensions for embedding and use PCA to reduce into two dimensions when you need to plot

[@mildlyoverfitted]

Thank you:) Yeh, great point:) I think I just wanted to make the video as short as possible:)

[@IqweoR]

Very informational, definitely looking forward seeing your next videos!

[@mildlyoverfitted]

Thank you! If you have any topic suggestions I would love to hear them!

[@azmyin]

Thank you very much sir. Deeply appreciate your hands on approach to teaching how to use embedding layer with LSTMs

[@mildlyoverfitted]

You are most welcome

[@peter62709]

Actually found this very helpful! The sped up typing sound was a bit annoying, but the content was great and it was well paced.

[@mildlyoverfitted]

Thanks a lot for your feedback! Really appreciate it:) Yeh, the keyboard sound is something I also worried about. I will try to turn down the volume in my next videos, however, if you have other suggestions I would be more than happy to hear them:)

[@randomforrest9251]

@@mildlyoverfitted I actually enjoyed it a lot. Gave some hhkb asmr vibes

[@connorsheehan4598]

I think the keyboard sound is great, probably just a preference thing

[@gnorts_mr_alien]

@@mildlyoverfitted yah came here for the embeddings, stayed for the nice keyboard sounds

[@LyurGG]

Okay but what keyboard setup are you using. Or is it a sound effect?

[@mildlyoverfitted]

Actually, this is a result of speeding up the footage of me typing. I am using Adobe Premiere Pro.

[@thepresistence5935]

bro, what keyboard are you using. sound was amazing.

[@mildlyoverfitted]

Logitec MX keys. Not a mechanical keyboard. The sound is augmented by the speedup effect.

[@thepresistence5935]

@@mildlyoverfitted Thanks a lot

[@redone9553]

Amazing quality! I'd love to know how you approach learning ML? Do you "just" visit courses at university and read math books? And what book/course would you recommend? Really appreciate an answer. Thanks!

[@mildlyoverfitted]

It is a tough question because IMO everybody is different and has a different background. With that being said I learned some math, stats and basic ML+DL at my university. Since leaving the university, I have been working in the industry doing ML on a daily basis. Additionally, I have been working on hobby projects in my free time (e.g. this KZbin channel). So I would definitely recommend learning some maths first (calculus + linear algebra) and then learning ML+DL basics. Everything from then on should be as practical (=coding) and as regular (=getting an ML day job or coming up with ML projects) as possible. So IMO rather than studying an entire book or a going through an entire course it is better to work on a specific project and focus only on the theory/technologies that are needed for this project. I hope it helps but again everybody is different:)

[@redone9553]

@@mildlyoverfitted amazing! Really appreciate your long answer! Thanks, you helped me! Right now I study Bishop's famous book but your right, it's becoming boring and tedious and practical projects keep oneself motivated. Best greetings (:

[@user-wb7mc7ip3z]

What's the keyboard? Sounds great when typing.

[@mildlyoverfitted]

Logitech MX Keys. I think it is mostly the fact that my mic is right next to it. It's not a mechanical keyboard and it is not that "loud" in real life.

[@mrigankanath7337]

Hey great video!! liked and subscribed, Hey I really wanted if you could make a video on how to work with attention layers in Pytorch? AS you know they are very cool but there is lack of resources on how to use it in your custom model. It will be really great! Thanks!!

[@mildlyoverfitted]

Thank you for the comment! That is a good idea:)))

[@Danny-we4vz]

Very thocky! What switches are you using?

[@dddaaa931]

Thanks for your vedio i search a long time for this step and can not find a good answer

[@mildlyoverfitted]

Thank you for your comment!

[@locorocozhong3941]

dude your keyboard as weird typing sound, what keyboard are you using? Btw the video is nice

[@mildlyoverfitted]

Logitech MX Keys:) Thank you!

[@trenvert123]

I liked the typing sound. I feel like the video is very information dense. I like it, but I'm having a very hard time parsing through everything. I think it's great for condensing this complex problem into a 20 minute video, when in reality, this video would have to be +1 1/2 hours to cover everything on word embedding alone. I think I'll come back to it at a later date when I have studied more. I'm still trying to understand how word embeddings work, and how word2vec like skip-gram allows them to represent similar words closer within the vector space.

[@coc2912]

excellent

[@ofeknourian8913]

At 11:30 shouldn't you take the last layer of h instead of averaging over all of h? from my understanding the last layer of h represent the last prediction of the sequence.

[@mildlyoverfitted]

Hey there! Great question. I guess taking the last layer of the hidden state is definitely a possibility. However, if the `n_layers=1` it does not make a difference.

[@mikhaeldito]

Thanks so much for the video. Could you explain again how Embedding can accommodate unseen vocab? Thanks in advance.

[@mildlyoverfitted]

Thank you. I guess the standard approach is to have a special entry in the vocabulary (like UNKNOWN) and whenever you stumble upon an unseen word/token/category you just use this special entry. The above strategy is exactly what BERT tokenizer does: github.com/huggingface/transformers/blob/6e535425feae20ca61a8b10ae5e8a7fab4d394ba/src/transformers/models/bert/tokenization_bert.py#L134-L136

[@harshmankodiya9397]

bro is rolling a dice at 10x speed

[@mildlyoverfitted]

hahaha♥️

[@beizhou2488]

Are you interested in implementing reinforcement learning papers?

[@mildlyoverfitted]

Sure, are there any that you find particularly interesting?

[@ifthikhanmohamed1681]

text.txt what contains i got key error

[@mildlyoverfitted]

Hello! Can you maybe describe your problem in more detail?

[@popamaji]

I didn't watch the video carefully but does it explain how nn.embedding code is been written or it's just the behaviour of nn.embeddings is explained in the first of video?

[@mildlyoverfitted]

Great question:) This video shows the behavior and usage of `torch.nn.Embedding`. Its actual implementation is not discussed.

[@popamaji]

@@mildlyoverfitted thanx for your awesome videos. can you make one for implementation, it would be great

[@mildlyoverfitted]

That is an interesting suggestion! Thanks!

[@The8thJester]

Thank you so much for this video! Do you perhaps have any suggestion on where my model might be going wrong? It seems that no matter what, my model always converges to outputting only space characters. I guess that's the most common character after all, but it's not quite what we're looking for is it? Thanks again!

[@mildlyoverfitted]

Interesting! I would try to make sure that you are able to overfit on a small training set/a single batch of samples. This way you can be sure that the model has "enough power" to do the task. Then I would move on to training on all your data while monitoring your train/test loss and metrics! Hope that helps!

[@iamragulsurya]

Name of the font?

[@mildlyoverfitted]

So the theme I am using is here: github.com/morhetz/gruvbox . The README talks about the fonts I believe.

[@saodeyipi9609]

how can you type so fast, jz??

[@mildlyoverfitted]

Heh, it is not my real typing speed. The video is sped up so that it is shorter.

[@djfl58mdlwqlf]

are you coding with vim...? how is that even possible lol how can i get those codes beautifully painted

[@mildlyoverfitted]

Yes. I am using this plugin: github.com/morhetz/gruvbox

[@koiRitwikHai]

Very informative video. Applause. I am just wondering ... why BERT does not use any norm in its embeddings? 1:40

[@sumithhh9379]

How the embedding weights are getting updated with a forward and backard pass. I understand the backprop but couldn't understand this. Please also reduce your typing sound. It feels like you are firing bullets instead or key strokes :).

[@mildlyoverfitted]

Thank you for your comment! The weights are updated when the `optimizer.step()` is called and PyTorch takes care of the technical details. Regarding the typing sound, I actually turned it down in all of my recent videos!

[@iraklimachabeli6659]

First of all thank you , very good intro combining embedding and LSTM. I run example and curious why in generate_text you use new_ch=np.random.choice( ..., p=probs) ,probs is logits[0] which is probability of next char being first element in vocab (which is ' ' blank char). I'd expect something like new_ch =dataset.ix2ch[ logits.argmax().item() ]

[@mildlyoverfitted]

Thank you for your comment! If I am not mistaken, `logits[0]` actually extracts the first and only sample from the batch. Or in other words, it is there to get rid of the batch dimension. Hope that helps:)

[@AlokMalik18]

I came from an internet search, without seeing like to dislike ratio I have no idea what's the quality of this video is. Please bring back dislike button youtube.

[@mildlyoverfitted]

Totally agree! I hate the fact that they removed it.

[@sahhaf1234]

impossible to understand.. as clear as mud..