BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

Full explanation of the BERT model, including a comparison with other language models like LLaMA and GPT. I cover topics like: training, inference, fine tuning, Masked Language Models (MLM), Next Sentence Prediction (NSP), [CLS] token, sentence embedding, text classification, question answering, self-attention mechanism. Everything is visually explained step by step.
I also review the background knowledge in order to understand BERT, by starting from an introduction to large language models (LLM) and the attention mechanism.
Slides PDF: github.com/hkproj/bert-from-s...
BERT paper: arxiv.org/abs/1810.04805
Chapters
00:00 - Introduction
02:00 - Language Models
03:10 - Training (Language Models)
07:23 - Inference (Language Models)
09:15 - Transformer architecture (Encoder)
10:28 - Input Embeddings
14:17 - Positional Encoding
17:14 - Self-Attention and causal mask
29:14 - BERT (overview)
32:08 - BERT vs GPT/LLaMA
34:25 - Left context and right context
36:36 - BERT pre-training
37:05 - Masked Language Model
45:01 - [CLS] token
48:26 - BERT fine-tuning
49:00 - Text classification
50:50 - Question answering

[@sandipanpaul1994]

One of the best videos for BERT in youtube

[@JRB463]

Thanks for breaking this down so well. As a privacy lawyer, this is the first time someone has managed to explain this to me in a way that I can actually understand!

[@prashlovessamosa]

Hello you are one of the best teacher I have found in my life.

[@meili-ai]

Love the way you explain things, so clear. Excellent Work!

[@greyxray]

i was trying to get the full picture of this for so long… seeing this felt just like getting a days long headache gone. thank you!

[@1tahirrauf]

Thanks Umar. I really appreciate your time and effort you put in creating these videos. I am anxiously waiting for the implementation video.

[@hengtaoguo7274]

This video is awesome! Worth watching multiple times as refresher. Please keep up the good work!🎉🎉🎉

[@user-xk7dy4nb7w]

Great primer or BERT. Excellent illustrations, and you explained the concepts very well.

[@LongLeNgoc-qq5qn]

Excellent video sir! Can't wait to see coding BERT from scratch.

[@hessame9496]

Man! You know how to explain these topics! Please continue uploading these great videos for the good of the community! Really appreciate it!

[@Daily_language]

explained much clearer than my prof. Great!

[@Nereus22]

I'm coming here from the trasformers video, and again really really good and detailed explanations! Keep up the good work and thank you!

[@rjkunal]

Man, you are an excellent teacher. I must say.

[@advaitpatole8988]

Thank you sir you explain these topics in great detail.Please keep uploading , excellent work.

[@mmaxpo9852]

Thanks Umar, I really appreciate your time and effort you put in creating these videos, very appreciate to create video coding BERT from scratch with PyTorch.

[@CandiceWinfield]

As a student, I do really appreciate your vedios. On the Internet, there's barely the guide about how to code a complete model, most of the vedio is talking about the theory. But if we don't walk through the code, we couldn't understand and use the model well. So I subscribe your channel to see your vedios about how to code a model from scratch, it's really awesome! Very clear and comprehensible. Can't wait to see Coding a BERT from scratch!

[@hoi5771]

We need more videos from you sir.. Like explaining more papers and LLMs

[@ansonlau7040]

Thank you so much Jamil, it's really helps a lot!!😁

[@nahidzeinali1991]

thanks a lot, I love all your presentations, please talk about GPT and other models as well.

[@MagusArtStudios]

I've been using a Bert models summary endpoint and it's pretty good! Bert is underrated AF.

[@linuxmanju]

Brilliant video, thank you for sharing.

[@rameshundralla6001]

Wow , what an explanation!!Thanks a lot

[@charlesity]

Amazing explanations!

[@xuanloc5111]

Excellent work!

[@enggm.alimirzashortclipswh6010]

If you could make a video on one of the finetuning tasks with example dataset and finetuning BERT on it, that basically would complete this lecture as one video contains every single information.

[@nilamkale5263]

Thank you for explaining bert in layman's language 👍

[@mahmoudghareeb7124]

Great as usual ❤❤

[@xuehanjiang3474]

very clear! thank you!

[@Hima_inshorts]

Thanku for information video..... after this video , I got a idea for my work❤...,

[@akshayshelke5779]

Please create more content it is helping us lot.....thanks for video

[@user-wr4yl7tx3w]

your Chanel is pretty awesome

[@bishwadeepsikder3018]

Great video explanation, could you please explain how the embeddings are generated from the token ids for the positional encodings are added?

[@user-wr4yl7tx3w]

best video explanation by far

[@umarjamilai]

Thank you for your feedback. Make sure to like and subscribe! Hopefully you'll love my future as much as this one.

[@user-yf7qv8zj6y]

Thanks for providing a good quality of video as always. As a newbie of computer vision tasks, I still have found myself struggling with training and inference of source code with dataset. Would you please upload an informative video to show and explain the entire process of how we can do this with an open source code? Thanks.

[@amitshukla1495]

Waiting for the BERT implementation from scratch !

[@mokira3d48]

Very good guys!

[@MW-ez1mw]

Thank you Umar for this great video! One confusion I have is about CLS token at 47:49, you mentioned we should use it because it can attend to all other words, since the first row don't have any 0 values(in the orange matrix). But wouldn't it also hold true for other tokens/rows in this matrix? Since this organge matrix is derived from softmax(QKt/sqrt(d)), while calculating this orange matrix, each row(word) in Q will multiple with every column of Kt matrix. Wouldn't this process enable each word to interact with all the rest words of the input? Sorry if I missed anything.

[@RayGuo-bo6nr]

谢谢你! Hope you enjoy the life in China.

[@arpanswain4303]

Could you please make a video on Complete Gpt architecture, different versions....your explanations are really good!!!!

[@li-pingho1441]

what a awesome video!!!!!!!

[@umarjamilai]

谢谢你！

[@RahulPrajapati-jg4dg]

Best Explained, can you please add some more videos regarding different different architecture related bert and transformers

[@christianespana1]

Hey bro, great and amazing work. Please do a video about coding BERT from scratch

[@1tahirrauf]

Thank you, Umar, for the video. I genuinely appreciate and enjoy your content. I'm looking forward to your implementation video. I have a question. At 6:14, you mentioned that the output of the Encoder would be a sequence of 10 tokens. In the case of BERT, wouldn't the output be simply the contextualized embeddings of the input token (rather than next token of sequence)? Thank you

[@umarjamilai]

Hi! It depends on which task BERT has been fine tuned for. If you fine tune it for the next token prediction task, it will return 10 tokens, of which the last one is the next token. If you fine tune it for another task, then the output will still be 10 tokens (since a transformer is a sequence-to-sequence network), but you need to interpret the output differently based on the task.

[@modaya3382]

Thanks for your efforts, I would love if you can make a tutorial on how to code OCR from scratch. Thanks

[@ariouathanane]

Awesome explanation. Cls token is important just because there is no zero values with others token?

[@VishalSingh-wt9yj]

thanks sir

[@PP-qi9vn]

Thanks!

[@user-kg9zs1xh3u]

Thanks Umar Jamil,

[@umarjamilai]

不客气🤓

[@thecraftssmith1599]

In 12:56 you mentioned the idea of Cosine function to denote similarities between words. Can you tell me in which research paper this idea is most skill-fully mentioned.

[@Lilina3456]

Thank you that was really helpful, can you olease do vision language models.

[@kingsleysoo8307]

Isnt the value vector for each Q,K,V should be different? Or say it is not necessary to be equal? And the product of the softmax function should be dot product again with Value Tensor V?

[@Koi-vv8cy]

I like it

[@yonistoller1]

Fantastic content, like always. Just worth noting that Q and K (and V) don't necessarily have to be the same in self attention

[@umarjamilai]

In Self-Attention the Q, K and V are always the same, that's why it's called self-attention. When K, V are coming from somewhere else (and are different from Q), in that case we talk about cross-attention.

[@Vignesh-ho2dn]

Great video. Thank you. Could you please do coding BERT from scratch? I'm very curious to learn how to implement MLM and NSP tasks in PyTorch

[@brajeshmohapatra9614]

Hello Umar. Could you please make a video coding BERT, MLM and NSP from scratch like the previous video on Transformer? That would be very helpful to us.

[@user-wr4yl7tx3w]

I think I followed your explanation of how BERT can be fine-tuned for Q&A, but still I am amazed how that can work. For example, fine-tuning, so that it knows the financial capital of China is Shanghai doesn't mean that it knows the financial capital of Indonesia.

[@umarjamilai]

Of course. The LLM will only know the concepts it is taught. If you never mention Indonesia in the training or fine-tune dataset, the LLM will never know what is Indonesia or its capital Jakarta.

[@Dad-rk8pi]

​@umarjamilai sir, I have some confusion, as you mentioned about MLM and NSP, how does that work in code? Does it pass one sentence (A) with some masking (and loss be called L) and simultaneously guess sentence B from sentence A (and loss for that be called L') and then train itself minimising L+L'? Like does it work on one loop like: for i in sthg: Fill MLM Evaluate loss in guessing word Guess sentence B from current filled sentence Evaluate loss in guessing sentence Minimise the total loss? Is there something that I can learn (as tutorials) for understanding how filling in the blanks and guessing next sentence works?

[@Dad-rk8pi]

​@@umarjamilaior is it calculated as MLM first and then NSP (like two different loops?)

[@tubercn]

@@Dad-rk8pi Thanks for your question, i am too, waiting someone can answer this

[@umarjamilai]

@@Dad-rk8pi Hi! It is calculated as two separate task, for which the loss is summed up as L1 + L2. I saw many different implementations of BERT online and so far the most trustworthy is the Hugging Face one.

[@tubercn]

Thanks for your wonderful tutorial, waiting for the code part👀👀

[@sathish331977]

excellent explanation of BERT , Thank you Umar. Can you suggest how to implement this for a NER type of task

[@umarjamilai]

In the future I plan to make a video on how to code BERT from scratch, but it's gonna take some time :D

[@srikanthganta7626]

Thanks, most of it great! But the Q&A fine-tuning is confusing.

[@learnwithaali]

In the framework of scaled dot-product attention, particularly within the context of Transformer architectures, a key consideration is the interaction between the query (q), key (k), and value (v) matrices. This mechanism typically involves scaling the product of q and k by the square root of the dimensionality (d), followed by the application of the softmax function, and subsequently multiplying this result with the v matrix. A critical aspect of this process is the role of the attention heads in processing tokens. If we consider a scenario where each attention head is exposed to the entirety of the token set throughout the training process, how might this influence the effectiveness and efficiency of the model? This question assumes particular relevance in a setting where the embedding dimension is fixed at 512, and the model is dealing with a substantial number of tokens, potentially in the range of 10,000. Could such a comprehensive exposure of all heads to all tokens potentially compromise the model's performance, or are there mechanisms within the architecture that mitigate this concern?

[@umarjamilai]

To be honest, within the context of multihead attention, all the tokens are exposed to each head, but each head is only working with a different part of the embedding of each token. This is exactly what happens in the vanilla Transformer and also in BERT and works wonderfully. For example, if the embedding size is 512 and we have 4 attention heads, the first attention head will see the first 128 dimensions of each token, that is, the range [0...127], the second head will see the range [128...255], the third the next 128 dimensions and so on... So I don't understand your question: the transformer is already working like this, and so does BERT. In BERT, particularly, each token attends also to tokens coming after it in the sentence, which is called its "right context". Hope my explanation clarifies your doubts

[@learnwithaali]

@@umarjamilai First of all, I'd like to express my gratitude for your insightful videos. They've been a great resource as I embark on my PhD journey in generative AI in Spain. Though I'm originally from Pakistan, I've been living in Spain since I was nine years old. Now, regarding my query about the Transformer model: I appreciate your explanation of the multihead attention mechanism and how each attention head interacts with a different segment of the token's embedding. However, my question specifically focuses on the logic behind multiplying the attention weights by the value matrix (V). While I understand that the product of the key (K) and query (Q) matrices indicates the relationship between tokens, the rationale behind subsequently multiplying this result by V isn't clear to me. Could you please clarify the purpose or reasoning behind this step in the attention mechanism?

[@InquilineKea]

What's the dimension in most LLMs like mixtral

[@KaushikJaiswal-eh4zt]

Loved your explaination of Transformers, BERT and all other videos around the same. Btw I have query for above video. The part where you are explaining training and inference of language model there you mentioned Transformer Encoder architecture. Shouldn't that be decoder based architecture as the inference is generative type. Please help me understand the same if it is Encoder type only @Umar Jamil

[@user-qr2wv8gb7v]

Please tell us about AliBi using the example of embedding in Llama 🙃 Instead of rotary coding 🙂

[@Tiger-Tippu]

Hi Umar ,please let me know the diff between Language models vs foundation models

[@umarjamilai]

A Foundation Model is a large language model, just trained on a massive amount of data that only big tech can afford (Meta, Google, etc.). Foundation Models have been pre-trained on a variety of data, so they can easily be fine tuned for a specific task or even used with zero shot prompting on unseen tasks.

[@dineshrajant4000]

Please do a video on GPT and LLaMA too....!!

[@umarjamilai]

I have two videos on LLaMA, check them out ;-)

[@syedabdul8509]

Shouldn't for Q&A tesk, linear layer should be 3 classes for each token, because for all other tokens apart from T10 and T27, there should be another class which will say NOT START/END.

[@umarjamilai]

Nope. Since we have two linear layers, each token will either be "Start/NOT_Start" or "End/Not_End". This means each token can be in 4 possible states: "Start - End", "Not_Start - End", "Start - Not_End" and "Not_Start - Not_End". This also covers the case in which the same token is the start and the end token at the same time, because the softmax score for that token will be the highest for both linear layers for the "Start" and the "End" class.

[@syedabdul8509]

@@umarjamilai Oh here by 2 linear layers you mean parallel layers. I assumed 2 linear layers as in series like an MLP. Also, if I think now, we don't require two separate layers right, we can use a single layer with output as 2 neurons, where each neuron will be binary START/NOT_START and other will be END/NOT_END.

[@Vignesh-ho2dn]

@umarjamilai Then in this case, how do we ensure end_token is always start_token?

[@MariemStudiesWithMe]

Hit the like button if you are impatiently waiting for " code bert from scratch" video🎉

[@s8x.]

is BERT outdated now?

[@Udayanverma]

25:00 your input sequence is different and you are explaining matrix for different sequence!! isnt it a mistake ?

[@umarjamilai]

Hi! I didn't understand what is a mistake... The whole video I have referenced the first line of the Chinese poem. Sometimes, even for different inputs, I reference the same matrix so that people know we are talking about the same concept.

[@Udayanverma]

you were using seq of China.... but table was not referring to that. in fact you said relation of EOS but table was confusing as it didnt refer to the same seq you were basing on. All in all i loved all your vidoes its just this one appeared out of sync in that period rest its cool. already watching your 2 hr video :)

[@swiftmindai]

At the point 25:00, he was just trying to explain causal mask concept with regards to the self attention in case of usual vanila transformer where the tokens doesn't interact with token which comes after it and hence made them -inf before applying softmax and ultimately become 0 after softmax is applied to them. But, It doesn't apply in case of BERT which particularly use MLM concept where by masking certain tokens at either side. Again, this is my understanding from his explanation. Once @Umar does coding, it would be more clearer I believe.

[@mdriad4521]

I would better if you cover coding part as well..

[@souronion3822]

I’ll use the restroom beef or Ethel come

[@souronion3822]

Tomorrow

[@haohuangzhang6917]

You are a legend

[@xugefu]

Thanks!