What are Transformer Models and how do they work?

This is the last of a series of 3 videos where we demystify Transformer models and explain them with visuals and friendly examples.
Video 1: The attention mechanism in high level • The Attention Mechanis...
Video 2: The attention mechanism with math • The math behind Attent...
Video 3 (This one): Transformer models
If you like this material, check out LLM University from Cohere!
llm.university
Get the Grokking Machine Learning book!
manning.com/books/grokking-ma...
Discount code (40%): serranoyt
(Use the discount code on checkout)
00:00 Introduction
01:50 What is a transformer?
04:35 Generating one word at a time
08:59 Sentiment Analysis
13:05 Neural Networks
18:18 Tokenization
19:12 Embeddings
25:06 Positional encoding
27:54 Attention
32:29 Softmax
35:48 Architecture of a Transformer
39:00 Fine-tuning
42:20 Conclusion

[@zafersahinoglu5913]

Luis Serrano, this set of 3 videos to explain how LLMs and transformers works is truly the best explanation available. Appreciate your contribution to the literature.

[@luizabarguil5214]

A clear, concise and conclusive way of explaining Transformers! Congrats and Thank you so much for sharing it!

[@anipacify1163]

Best playlist on transformers and attention. Period . There is nothing better on KZbin. Goated playlist. Thank you soo much !

[@YannickBurky]

This video is incredible. I've been looking for material to help me understand this mess for quite some time now, but everything about this video is perfect: the tone, the speed of speech, the explanations, the hierarchy of knowledge... I'm screaming with joy!

[@lightninghell4]

I've seen many videos on transformers but this series os the first where I understood the topic at a deep enough level to appreciate it.

[@sandhiyar8763]

Absolutely worth the watch! The clarity in Luis's explanation truly reflects his solid grasp on the content.👍

[@RuliManurung]

What an awesome series of lectures. I spent 10 years teaching undergraduate-level Artificial Intelligence and NLP courses, so I can really appreciate the skill in breaking down and demystifying these concepts. Great job! I would say the only thing missing from these videos is that you don't really cover how the learning/training process works in detail, but presumably that would detract from the focus of these videos, and you cover it elsewhere.

[@alidanish6303]

Finally the 3rd video of the series and as usual with the same clarity of concepts as expected from Serrano. The way you have perceived these esoteric concepts have produced pure gold. I am following you and jay from Udacity and you guys have made real contribution in explaining a lot of black magic. Any plans to update grokking series...?

[@MohamedHassan-pv1xl]

I feel super lucky to come across your videos, normally I don't comment, but I saw the 3 videos of the series and I'm amazed on how you explain complicated topics. You're efforts are highly appreciated.

[@jazznomad]

Very clear. You are a natural teacher

[@amiralioghli8622]

There is a plethora of videos available on the theoretical aspects of transformers, but there is a noticeable scarcity of content when it comes to their practical implementation. Furthermore, there is a notable absence of videos demonstrating how to implement transformers specifically for time series data. In light of this, it would be highly appreciated if you could devote some attention to the practical implementation of transformers, with a particular emphasis on their application to time series data.

[@william_8844]

I concur, I would really love to watch the vid

[@calebadobah641]

Yhh don't know why people don't really post on such stuff

[@usefbob]

This series was great! Appreciate all the time and effort you've put into them, and laid out the concepts so clearly 🙏🙏

[@patriciasilvaoliveira6130]

Amazingly clear and encouraging to learn more. Thanks, maestro!

[@johnschut164]

Your explanations are truly great! You have even understood that you sometimes have to ‘lie’ first to be able to explain things better. My sincere compliments! 👊

[@karunamudliyar5625]

The best video, that I watched on Transformer. Very clear explanation

[@Tuscani2005GT]

Seriously some of the best videos on the topic. Thank you!

[@SkyRiderJavelin]

What an excellent series on Transformers, really did the trick !!! The penny has finally dropped. Thanks very much for posting this is very useful content. I wish I came across this channel before spending 8 hours doing a course and still not understanding what happens under the hood.

[@shaktisd]

Amazing series on Transformer. Never ever imagined the true rationale behind Q,K,V ... it is actually clear after watching your video. Thanks a lot.

[@utkarshkapil]

Beautifully explained. Loved how you went ahead to also teach a bit of the pre-requisites!

[@jorovifi89]

Great work as always, thank you keep them coming

[@nazmulhaque8533]

Excellent presentation. Waiting to see more videos like this. I would request you to make a series about aspect based sentiment analysis. Best wishes...

[@faraazmohammed3693]

Liked the video while watching at 7:15. crystal clear explanation. Good job, thank you Serrano and I appreciate your work.

[@blueberryml]

excellent -- clear and concise explanations

[@nileshkikle8112]

Dr. Luis - Thank you for taking all the effort for creating these #3 videos. Explaining complex things in the simplest way is an art! And you have that knack! Great job! Been following you ML videos for years now and I always enjoy them. PS- Funny enough, for typing these comments, I'm being prompted for selecting the next prediction word! 🙂

[@timothyjoubert8543]

thank you for this series - wonderfully explained. 💯

[@johnny1966m]

Thank you Mr. Serrano, it was very educated and lectured in very good way. In relation to Positioning, the example with arrows gave me the idea that the purpose of this stage is to make sure that only the correct positions of words in the sentence cluster together, while the incorrect ones diverge them, thus the neural network distinguishes their position in the sentence during training.

[@poussinet2]

Thank you for these really high quality videos and explanations.

[@aminemharzi7222]

The best explanation I found

[@harithummaluru3343]

great explanation. perhaps one of the best videos

[@AboutOliver]

You have a skill for teaching! Thanks so much for this series.

[@dragolov]

Deep respect, Luis Serrano! Thank you so much!

[@AmanBansil]

Incredible - just found this channel and I am about to pour over all videos. Thank you so much for your effort.

[@chrisogonas]

Well illustrated! Thanks for sharing.

[@GiovanneAfonso]

The best explanation available

[@danieltiema]

Thank you for explaining this so well.

[@abdelrhmanshoeeb7159]

Finally ,i am waiting it from a month. Thank you alot.

[@markuskaukonen3903]

Very nice stuff!, This first time somebody explained clearly what large language models are. Especially the second video was very valuable for me!

[@anuragdh]

Thanks.. for the first time (not that I've gone through a lot of them :)), I was able to appreciate how the different layers of a Neural Network fit together with their weights. Thanks for making this video with the example used

[@sohamlakhote9822]

Thanks a lot man!!! You did a fantastic job explaining these concepts 🙂

[@silvera1109]

Great video, hugely appreciated, thank you Luis! 🙏

[@Glomly]

This is the BEST explanation ever you can find on the internet. I'm serious

[@analagunapradas3840]

Agree, definitly the BEST, as always ;) Luis Serrano

[@lakshminarayanan5486]

Hi Luis, Excellent material and you know how to deliver it to perfection. Thanks a lot. Could you please explain a bit more on positional encoding and how the residual connections & layer normalization, encoder-decoder components fit into the very same example.

[@skbHerath]

finally, i managed to understand the concept clearly, Thanks

[@edwinma9933]

this is amazing, it deserves 10M views.

[@incognito3k]

As always amazing content. We need a book from Luis on GenAI!

[@sathyanukala3409]

Excellent explanation. Thanks you.

[@anupamjain345]

Thanks! Never came across anyone explaining anything in such a great detail, you are amazing !!!

[@SerranoAcademy]

@anupamjain345, thank you so much for your really kind contribution, and for your nice words!

[@leenabhandari5949]

Your videos are gold.

[@vasimshaikh9857]

Finally 3rd video is here 😮😅 thank you sir I have been waiting for this video since last month , everyday I check your channel for the 3rd video sir , thank you so much sir You're doing great work 👍

[@ColinTimmins]

Very nice! Love your work and visuals. =]

[@samirelzein1095]

the great Luis! i am recommending you in my job posts, your content is a prerequisite before working for us

[@SerranoAcademy]

Wow thanks Samir, what an honor! And great to hear from you! I hope all is well on your end!

[@samirelzein1095]

@@SerranoAcademy honor is mine! you are the artist of going inside it, seeing the wiring and connections and delivering them as seen to all people. that s the job of prophets and saints. bless you. i am doing great, plenty of text and image processing currently :) digitizing the undigitized!

[@sriharsha580]

Thanks for the wonderful presentation. In the previous video of the series while discussing the relationship between query and key(building the context relationship between words), it is mentioned the relationship QK and v(predicting next word) will be covered in this video, may I know whether it will be covered in another video or not.

[@panoskolyvakis4075]

you re an absolute legend

[@Omar-bi9zn]

fantastic videos !!

[@jukebox419]

You're the greatest teacher ever lived in the history of mankind. Can you please do more videos regularly?

[@SerranoAcademy]

Thank you so much! Yes I'm definitely working hard at it. Some videos take me quite a while to make, but I really enjoy the process. :) If you have suggestions for topics, please let me know!

[@andreibuldakov2641]

Man, you are the best!

[@ikheiri]

Best video i've come across that explains concepts simply. Helped tremendously in my learning endeavor to create a mental model for neural networks (there's a joke there somewhere)

[@SerranoAcademy]

Thanks! Lol, I see what you did there! :)

[@joaomontenegro]

These videos are great!! I would love to see one about the intuition of cross attention in, for example, the context of translation between two languages.

[@SerranoAcademy]

Thanks, great suggestion!

[@maethu]

I am happy like a zygote about this video! Great work, thanks a lot!

[@EugenBurianov]

Great great video! Thank you

[@BasmaSAYAH-vp7vo]

Thank you 🙏🏻

[@william_8844]

I like the attention explanation

[@emvdl]

well done 👍

[@htchtc203]

Sir, thank you for very clear and informative series of presentations. Excellent job? May I ask something about embedding or word2vectors. How is a NN trained for words in order to cluster words for some kind of similarity grouos in multidimensional vector space? Is this training proceas guided or is it like self organizing map or process?

[@wanggogo1979]

Finally, I waited until this video was released.

[@karstenhannes9628]

Thanks for the video! I particularly liked the previous video about attention, super nice explanation! However, I thought most transformers simply use a linear layer that is also trained to create the embedding instead of using a pre-trained network like word2vec.

[@dayobanjo3870]

Great video, speaking from Abuja capital of Nigeria

[@SerranoAcademy]

ohhhh greetings to Abuja!!! Nigerians are the kindest people, I hope to visit sometime!

[@craigwood6561]

Amazing

[@alexanderzikal7244]

Thank You very much for all your videos! Whats Software do You use for your presentations? All looks really nice, all pictures…

[@aspaksharif4376]

Thank you..

[@jayanthkothapalli9.2]

Great Videos sir, Thank you for helping us to increase India's GDP!! Sir can you make videos on Fine-Tuning?

[@TemporaryForstudy]

Your videos are rocking as always. Hey, do you have any remote internship opportunities in your team or in your organisation? I would love to learn and work with you guys.

[@SerranoAcademy]

Thank you so much! Yes we have internships, check them out here! jobs.lever.co/cohere

[@vankram1552]

This is a fantasitc video, by far the best on youtube. My only feedback would be the guitar music you use between chapters is a little abrasive and can you take you out of the learning process. Maybe some calmer more thought provocing music along with more interesting title cards would be better.

[@Aleks-ng3pp]

In the previous video you said you would explain how to compute Q, K, and V matrices in this one. But I don't see.

[@ifeanyiidiaye1889]

Very awesome video. Thanks a lot! However, can you please suggest the Q and A dataset format for finetuning an LLM to answer questions? Is there a very specific format, especially if you are using domain-specific dataset or can a regular CSV file with columns ["Questions", "Answers"] be used for that purpose? Please, I will appreciate any advice or recommendations you make 🙏

[@SerranoAcademy]

Thank you so much! That's a great question! Normally companies that train LLMs curate their own datasets, and I'm not sure exactly how they look. But here is one built publicly, that looks pretty good! paperswithcode.com/paper/toolqa-a-dataset-for-llm-question-answering

[@ifeanyiidiaye1889]

@@SerranoAcademy Thanks a lot

[@ohhbatu1303]

good video

[@Nerdimo]

8:53 do you think it’s a plain feed forward neural network, or something like an RNN, LSTM to be specific? Just a thought.

[@maneeshajainz]

I like your videos. Can you post a quiz after each of your videos?

[@arsalanzabeeb6467]

and this 3rd video summed up so nicely that i am just in a denial ........is that it ? Thank you so much

[@user-mh1ip4er7x]

Thanks!

[@SerranoAcademy]

Thank you so much for your kindness! Very appreciated. :)

[@atriplehero]

Does the whole "Once upon a time" already built gets fed again into the whole process again as a 'input' in order to get attached its next word/token? In order words, is it like a cycling again and again untill a "seemengly complete" answer is generated? If this is the case, it would be a whole lot of inefficiency and explains why so much electricity is consumed!! Please answer this crucial detail.

[@techgayi]

Excellent video, will be good to indicate what encoder - decoder model is in transformers. Couldnt figure that out here.

[@SerranoAcademy]

Thanks! Yes that's something I'm trying to make sense of, perhaps in a future video. In the meantime, this blog post is the best place to go for that: jalammar.github.io/illustrated-transformer/

[@CreatingUtopia]

Thanx

[@AI-Kawser]

awesome

[@rasmusnordstrom9947]

Is there any good explanation out there for the "second" input into the transformer structure: Outputs (shifted right) in the original paper?

[@SerranoAcademy]

Thanks for the question! Not sure exactly what second input. The one coming out of the attention mechanism and into the transformer? I would say that that's a 'enhanced' vector for the input text. Namely, one that carries context on it. Lemme know if that's what you meant, or if it was a different one.

[@rasmusnordstrom9947]

​@@SerranoAcademy If I understand correctly when looking at Figure 1 in the original paper (attention is all you need), the initial prompt is first fed into the encoder and then inserted halfway into the decoder, which finally yields the first token. As far as I understand, to generate the next token, we don't simply append the first token to the initial prompt and run it through both the encoder and decoder. Instead, we insert the newly generated token directly into the decoder (or runs it through a different encoder?). I'm somewhat confused about this part.

[@SatyaRao-fh4ny]

This is a great video, clarifying a number of concepts. However, I am still not finding any answer to some of my questions. e.g in this video, when the user enters "Write a story.", these are 4 tokens. But the "model" spits out a NEW word "Once". Where is this NEW word coming from? How does the "model" even "KNOW" about such a word? Is it saved in some database/file? Is there a dictionary of ALL the words (or tokens) that the "model" has access to? And I guess the other question what does "training a model" actually mean- on the ground- not just conceptually? After training, is the end result some data/words/tokens/embeddings that are save in some file that the "model" "reads/processes" when it is used later on? What are parameters? I have watched several hours of videos, but have not found answers to these questions! Thanks for any help for experts!

[@SerranoAcademy]

Thanks, great questions! Yes, there is a database of tokens, and what the model does is output a list of probabilities, for each token. The ones with high probability are the ones that are very likely to be the next in the sentence. So then one can pick a token at random based on this probabilities, and very likely you'll pick one that has a high probability (and that way, the model will not always answer the questions in the exact same way, but it'll have variety). The training part is very similar to a neural network. It consists on updating the weights so that the model does a better job. So for example, if the next word in a sentence should be "apple", and the model gives "apple" a very low probability, then the backpropagation process updates the weights so that the probability of "apple" increases, and all the other ones decrease. The parameters are the parameters of the neural network + the parameters of the attention matrices. If you'd like to learn more about neural networks and the training process, check out this video: kzbin.info/www/bejne/eIOcmWdtf9mkr9k

[@romanemul1]

Thank you Louis G. Any future courses in plan with Udacity ?

[@SerranoAcademy]

Thank you, glad you liked it! Nothing with Udacity recently, but I did make this one with Coursera: www.deeplearning.ai/short-courses/large-language-models-semantic-search/

[@romanemul1]

@@SerranoAcademyThanks . Enrolled

[@laavispamaya]

Ooh thanxs for mentioning Emmy Noether

[@SerranoAcademy]

Yayy!!! Huge fan of Emmy! :)

[@udaya2008]

One confusion, I had was to comprehend the seperation of model development vs the scoring/inferencing phases from your explanation. I am understanding that your explaining lot of inference parts and some development parts.

[@nafisanawrin2901]

While training model if it shows wrong answer how it is corrected?

[@mrcool4172]

Very well explained !! I want to understand if a trained transformer model can be further trained with a curated dataset to specialise in say chatbot?

[@SerranoAcademy]

Thank you! Absolutely, what you would do in that case is fine tuning. That is, you take a trained model, and then post-train it with the data that you want, then it becomes better at answering from that dataset.

[@khaledbouzaiene3959]

nice wow but please i still have a question, you didn’t mentioned how the words with similarities are placed close in embedding, i know after we assign the mechanism attention score but don’t get do the embedding is a separate neural network as in video

[@tantzer6113]

That closeness is achieved automatically in the end result because it’s more efficient. It isn’t something that the human designer plans for.

[@SerranoAcademy]

Yes, great question! The idea is to train a neural network to learn the neighboring words to a particular word. So in principle, words with similar neighbors will be close in the embedding, because the neural network sees them similarly. Then the embedding comes from looking at the penultimate layer in the neural network, which has a pretty good description of the words. So for example, the word 'apple' and the word 'pear' have similar neighboring words, so the neural network would output similar things. Therefore, at the penultimate layer, we'd imagine that the neural network must be carrying similar numbers for each of the words. The embeddings come out of here, so that's why the embeddings for 'apple' and 'pear' would be similar.

[@khaledbouzaiene3959]

@@SerranoAcademythanks for clarifying i got confused coz i just want this huge neural network composed of multiple layers of smaller neural networks where the first one is the embedding layer not separate one , but generally everything now make sense now no matter the design

[@VerdonTrigance]

24:59 - again... who and how defines these layers and network to set words vectors? All it comes to it. How do we know that cherry and apple has a similar 'properties' ?

[@user-sm1re8xm5p]

I'm a bit confused : to create (train) a neural net you need to give it input : embeddings. and then whatever the penultimate layer shows is the embedding. isn't that pulling yourself out of the swamp by your own bootlaces ?

[@SerranoAcademy]

Great question! Yes, it does look like a cycle the way it's shown, but these are trained at different times. Normally the embedding gets trained first, with a neural network that is trained to learn neighboring words. This is not a transformer, so it doesn't have attention layers. This NN is not good at generating text, but it gives a strong embedding. Once you have the embedding, then you train the transformer to generate sentences, with attention and all the other parts.

[@user-sm1re8xm5p]

@@SerranoAcademy Clear ! Thanks !

[@anthonymalkoun6188]

What is the capital of Lebanon? Beirut! Cheers from Beirut, and thanks for the great video series :)

[@SerranoAcademy]

Thank you!!! Lots of greetings to Beirut!!! :)

[@hansu7474]

40:55 Who discovered abstract algebra? Galois

[@tantzer6113]

Given that the positional encoding is added to the word embeddings, how does the transformer learn to separate the combined positional and embedding signals?

[@SerranoAcademy]

Great question! That's a bit of a mystery. All that is clear to me is that positional encoding changes each word based on the ordering. The model is trained to learn these small disruptions in order to pick up the order of the words. How it combines it exactly with the embeddings is not very clear (at least to me), but positional encoding has worked well in practice, especially with lots of data and very large models.

[@tantzer6113]

Thank you. It occurred to me that the question I asked, “how does it do it?” is the wrong one. The question should be, why might one expect it to be *possible* to do? The creators of the positional embedding must have had a rationale or intuition for the possibility. This is suggested by the specificity of the form the positional encoding takes.

[@genkidama7385]

when you give a command most of the time it says "too complicated" and replaces code with comments.

[@aidanthompson5053]

31:08

[@vil9386]

Wow, the three videos you have put in this playlist - kzbin.info/www/bejne/hammoYqteah3fLM - makes it possible to understand Attention, otherwise looked impossible. Thank you!