L19.5.2.3 BERT: Bidirectional Encoder Representations from Transformers

Slides: sebastianraschka.com/pdf/lect...
0:00 Introduction
0:29 BERT (Bidirectional Encoder Representations from Transformers)
1:44 BERT Inputs
4:18 BERT Pre-Training Task #1
9:47 BERT Pre-Training & Downstream Tasks
11:51 Transformer Training Approach
12:16 BERT Pre-Training & Fine-Tuning Approach
13:49 BERT vs GPT-v1 Performance
14:59 BERT Pre-Training & Feature-based Training
-------
This video is part of my Introduction of Deep Learning course.
Next video: • L19.5.2.4 GPT-v2: Lang...
The complete playlist: • Intro to Deep Learning...
A handy overview page with links to the materials: sebastianraschka.com/blog/202...
-------
If you want to be notified about future videos, please consider subscribing to my channel: / sebastianraschka

[@lily-qs7cr]

thanks for all of your transformers viedos :)

[@SebastianRaschka]

you are welcome! glad to hear you like them!

[@billykotsos4642]

Quick question… the output of the Bert is several word embeddings right? So does that mean they need to be concatenated/added/averaged to create the feature to be passed to the MLP classifier? What is the most standard method these days?

[@payam-bagheri]

I have been trying to understand how sentence embedding (one embedding vector for the whole sentence) is generated by BERT. What I've understood so far is as follows: - The [CLS] token that is added to the beginning of the sentence (assuming you give the model a sentence and want the embedding vector as the output) evolves in the same way as any other token evolves as it passes through the network (BERT) and what results from that process for the [CLS] token is used as a representative embedding for the whole sentence. - Or, the max/mean along each dimension of the embedding vectors for each token is used as the sentence embeddings.

[@peregudovoleg]

17:55, 2 years ago, 300kk parameters were "quite large". Look at us now - 405kkk llama 3. 1000+ grow. What about next 2 years...