Encoder-Only Transformers (like BERT) for RAG, Clearly Explained!!!

[@statquest]

Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/

[@tuananhvo9506]

Complete this new one I may have been roughly watching all of the videos of StatQuest already. Deeply invested in the channel for the last few months, I feel much more confident in my quest to get the first AI related job. Massive thanks Josh for relentlessly bringing the right intuition for the mass of us!!

[@statquest]

Good luck with that first job!

[@PradeepKumar-hi8mr]

Wowww! Glad to have you back, Sir. Awesome videos 🎉

[@statquest]

Thank you!

[@NottoriousGG]

Such a cleverly disguised master of the craft. 🙇

[@statquest]

bam! :)

[@Kimgeem]

So excited to watch this later 🤩✨

[@statquest]

future bam! :)

[@mbeugelisall]

Just the thing I’m learning about right now!

[@statquest]

bam! :)

[@nossonweissman]

Yay!!! ❤❤ I'm starting it now and saving to remember to finish later. Also, I'm requesting a video on Sparse AutoEncoders (used in Anthropic's recent research). They seem super cool and I have a basic idea on how they work, but I'd to see a "simply explained" version of them.

[@statquest]

Thanks Nosson! I'll keep that topic in mimd.

[@free_thinker4958]

You're the man ❤️💯👏 thanks for everything you do here to spread that precious knowledge 🌹 we hope if you could possibly dedicate a future video to talk about multimodal models (text to speech, speech to speech etc...) ✨

[@statquest]

I'll keep that in mind!

[@davidlu1003]

And thx for the courses. They are great!!!!😁😁😁

[@statquest]

Glad you like them!

[@tcsi_]

100th Machine Learning Video 🎉🎉🎉

[@statquest]

Yes! :)

[@THEMATT222]

Noice 👍 Doice 👍Ice 👍

[@davidlu1003]

I love you, I will keep going and learn the other courses of yours if they are always free. keep them free please, I will always be your fan.😁😁😁

[@statquest]

Thank you, I will!

[@kamal9294]

Nice explanation, if the next topic is about rag or reinforcement learning , i will be happier (or even object detection, object tracking).

[@statquest]

I guess you didn't get to 16:19 where I explain how RAG works...

[@kamal9294]

@statquest bro but in LinkedIn I saw many rag types and some retrieval techniques using advanced dsa(like HNSW). That's why I asked.

[@statquest]

@@kamal9294 Those are just optimizations, which will change every month. However, the fundamental concepts will stay the same and are described in this video.

[@kamal9294]

@@statquest Now I am clear, thank you!.

[@thegimel]

Great instructional video, as always, StatQuest! You mentioned in the video that the training task for these networks is next word prediction, however, models like BERT have only self-attention layers so they have "bidirectional awareness". They are usually trained on masked language modeling and next sentence prediction, if I recall correctly?

[@statquest]

I cover how a very basic word embedding model might be trained in order to illustrate its limitations - that it doesn't take position into account. However, the video does not discuss how an encoder-only transformer is trained. That said, you are correct, an encoder-only transformer uses masked language modeling.

[@etgaming6063]

This video came just in time, trying to make my own RoBERTa model and have been struggling understanding how they work under the hood. Not anymore!

[@statquest]

BAM!

[@swarupdas8043]

What can be better to learn ML when we have a teacher like you. Thanks for all the effort you have put into. I would buy if you have any Udemy courses covering ML stuff. Please let me know

[@statquest]

I have a book coming out in the next few weeks about all these neural network videos with Pytorch tutorials

[@iamumairjaffer]

Well explained ❤❤❤

[@statquest]

Thanks!

[@rishidixit7939]

Very Beautifully Explained as Always. It takes a great amount of intuitive understanding and talent to explain a relatively tougher topic in such an easy way. I just had some doubts - 1. In case of context aware embeddings of a Sentence of a Doc are the individual Embeddings of the tokens averaged. Does this have something to do with the CLS token ? 2. Like a Variational Autoencoder helps in understanding the intricate patterns of images and then creates its own latent space , can BERT (or any similar model) do that for Vision task (or are they only suitable for NLP Tasks) 3. Are Knowledge Graphs made using BERT ? Any help on these will be appreciated . Thank You again for the Awesome Explanation

[@statquest]

1. The CLS token is specifically used for classification problems and I talk about how it works in my upcoming book. That said, if you embed a whole sentence, then you can average the output values. 2. Transformers work great with images and image classificaiton. 3. I don't know.

[@tonym4926]

Are you planning to add this video to neutral network/ deep learning playlist?

[@statquest]

yes! Just did.

[@aryasunil9041]

Great Video, When is the Neural Networks book coming out? Very eager for it

[@statquest]

Early january. Bam! :)

[@barackobama7757]

Hello StatQuest. I was hoping if you could make a video on PSO (Particle Swarm Optimisation) Will really help! Thank you, amazing videos as always!

[@statquest]

I'll keep that in mind.

[@draziraphale]

Great explanation

[@statquest]

Thanks!

[@alecollins01]

THANK YOU

[@statquest]

double bam! :)

[@nathannguyen2041]

Did math always come easy to you? Also how did you study? Do math topics stay in your mind e.g., fancy integral tricks in probability theory, or dominated convergence, etc?

[@statquest]

Math was never easy for me and it's still hard. I just try to break big equations down into small bits that I can plug numbers into and see what happens to them. And I quickly forget most math topics unless I can come up with a little song that will help me remember.

[@benjaminlucas9080]

Have you done anything on vision tranformers? or can you?

[@statquest]

I'll keep that in mind. They are not as fancy as you might guess.

[@noadsensehere9195]

good

[@statquest]

Thanks!

[@aihsdiaushfiuhidnva]

not many people outside the know knows about bert it seems

[@statquest]

yep.

[@SuperRobieboy]

Great video, encoders are very interesting in applications like vector search or down-stream prediction tasks (my thesis!). I'd love to see a quest on positional encoding, but perhaps generalised to not just word positions in sentences but also pixel positions in an image or graph connectivity. Image and graph transformers are very cool and positional encoding is too often only discussed for the text-modality. Would be a great addition to educational ML content on KZbin ❤

[@statquest]

Thanks! I'll keep that in mind.

[@dailygrowth7967]

PIZZA GREAT!❤

[@statquest]

:)

[@epberdugoc]

Actually is, LA PIZZA ES MAGNÍFICA!! ha ha

[@statquest]

:)

[@Apeiron242]

Thumbs down for using the robot voice.

[@statquest]

Noted

[@ChargedPulsar]

Another bad video, promises simplicity dives right into graphs with no background or explanation.

[@statquest]

Noted

[@Austinlorenzmccoy]

@@ChargedPulsar the video is great, visualization helps people capture context more Maybe cause i have read about it before but it sure explains better But if you feel you do better, create the content and share so we dive in too