Encoder-Only Transformers (like BERT), Clearly Explained!!!

[@statquest]

Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@NottoriousGG]

Such a cleverly disguised master of the craft. 🙇

[@statquest]

bam! :)

[@PradeepKumar-hi8mr]

Wowww! Glad to have you back, Sir. Awesome videos 🎉

[@statquest]

Thank you!

[@tcsi_]

100th Machine Learning Video 🎉🎉🎉

[@statquest]

Yes! :)

[@THEMATT222]

Noice 👍 Doice 👍Ice 👍

[@nossonweissman]

Yay!!! ❤❤ I'm starting it now and saving to remember to finish later. Also, I'm requesting a video on Sparse AutoEncoders (used in Anthropic's recent research). They seem super cool and I have a basic idea on how they work, but I'd to see a "simply explained" version of them.

[@statquest]

Thanks Nosson! I'll keep that topic in mimd.

[@kamal9294]

Nice explanation, if the next topic is about rag or reinforcement learning , i will be happier (or even object detection, object tracking).

[@statquest]

I guess you didn't get to 16:19 where I explain how RAG works...

[@kamal9294]

@statquest bro but in LinkedIn I saw many rag types and some retrieval techniques using advanced dsa(like HNSW). That's why I asked.

[@statquest]

@@kamal9294 Those are just optimizations, which will change every month. However, the fundamental concepts will stay the same and are described in this video.

[@kamal9294]

@@statquest Now I am clear, thank you!.

[@free_thinker4958]

You're the man ❤️💯👏 thanks for everything you do here to spread that precious knowledge 🌹 we hope if you could possibly dedicate a future video to talk about multimodal models (text to speech, speech to speech etc...) ✨

[@statquest]

I'll keep that in mind!

[@etgaming6063]

This video came just in time, trying to make my own RoBERTa model and have been struggling understanding how they work under the hood. Not anymore!

[@statquest]

BAM!

[@barackobama7757]

Hello StatQuest. I was hoping if you could make a video on PSO (Particle Swarm Optimisation) Will really help! Thank you, amazing videos as always!

[@statquest]

I'll keep that in mind.

[@davidlu1003]

And thx for the courses. They are great!!!!😁😁😁

[@statquest]

Glad you like them!

[@thegimel]

Great instructional video, as always, StatQuest! You mentioned in the video that the training task for these networks is next word prediction, however, models like BERT have only self-attention layers so they have "bidirectional awareness". They are usually trained on masked language modeling and next sentence prediction, if I recall correctly?

[@statquest]

I cover how a very basic word embedding model might be trained in order to illustrate its limitations - that it doesn't take position into account. However, the video does not discuss how an encoder-only transformer is trained. That said, you are correct, an encoder-only transformer uses masked language modeling.

[@Kimgeem]

So excited to watch this later 🤩✨

[@statquest]

future bam! :)

[@tonym4926]

Are you planning to add this video to neutral network/ deep learning playlist?

[@statquest]

yes! Just did.

[@aryasunil9041]

Great Video, When is the Neural Networks book coming out? Very eager for it

[@statquest]

Early january. Bam! :)

[@davidlu1003]

I love you, I will keep going and learn the other courses of yours if they are always free. keep them free please, I will always be your fan.😁😁😁

[@statquest]

Thank you, I will!

[@mbeugelisall]

Just the thing I’m learning about right now!

[@statquest]

bam! :)

[@nathannguyen2041]

Did math always come easy to you? Also how did you study? Do math topics stay in your mind e.g., fancy integral tricks in probability theory, or dominated convergence, etc?

[@statquest]

Math was never easy for me and it's still hard. I just try to break big equations down into small bits that I can plug numbers into and see what happens to them. And I quickly forget most math topics unless I can come up with a little song that will help me remember.

[@benjaminlucas9080]

Have you done anything on vision tranformers? or can you?

[@statquest]

I'll keep that in mind. They are not as fancy as you might guess.

[@draziraphale]

Great explanation

[@statquest]

Thanks!

[@iamumairjaffer]

Well explained ❤❤❤

[@statquest]

Thanks!

[@aihsdiaushfiuhidnva]

not many people outside the know knows about bert it seems

[@statquest]

yep.

[@epberdugoc]

Actually is, LA PIZZA ES MAGNÍFICA!! ha ha

[@statquest]

:)

[@noadsensehere9195]

good

[@statquest]

Thanks!

[@SuperRobieboy]

Great video, encoders are very interesting in applications like vector search or down-stream prediction tasks (my thesis!). I'd love to see a quest on positional encoding, but perhaps generalised to not just word positions in sentences but also pixel positions in an image or graph connectivity. Image and graph transformers are very cool and positional encoding is too often only discussed for the text-modality. Would be a great addition to educational ML content on KZbin ❤

[@statquest]

Thanks! I'll keep that in mind.

[@Apeiron242]

Thumbs down for using the robot voice.

[@statquest]

Noted

[@ChargedPulsar]

Another bad video, promises simplicity dives right into graphs with no background or explanation.

[@statquest]

Noted

[@Austinlorenzmccoy]

@@ChargedPulsar the video is great, visualization helps people capture context more Maybe cause i have read about it before but it sure explains better But if you feel you do better, create the content and share so we dive in too