... which gives you **exactly** the same result (just plug in 0 to see why)

This is a "standard supervised machine learning" terminology - we have a bunch of IID "examples" in the training data in the form of x = feature vector, y = the label (vector). An example in NLP could be sentiment classification: A single example is a movie review, one possible baseline "featurization" is a bag-of-word (I believe we cover it in the lectures), and the label would be one of the two values (0 = negative review, 1 = positive review).

@@anonymous71208 A follow up question about the example you gave. A review reads: I loved the movie Shawshank Redemption. It's a modern classic. Inference - this is a positive review. Task - Classify this as positive (1) or negative (0), so binary classification. In this case, the text "I loved the movie Shawshank Redemption. It's a modern classic" in its vectorized form is xi the feature vector? and the classification - positive = 1 or negative = 0 (only one of them either 0 or 1) that comes out as a result of the processing of the text by the model, in its vectorized (closer to 0 or 1) form, is the gold label? Sorry to ask a silly question. This is very important for me to clear my concept.

@@harshvardhanagrawal "In this case, the text "I loved the movie Shawshank Redemption. It's a modern classic" in its vectorized form is xi the feature vector?" -> yes "positive = 1 or negative = 0 (only one of them either 0 or 1)" -> this is the so-called "gold label" which is **known** in advance. This example will be used for training the model. The model we discussed here will output a number between zero and one (ie. the conditional probability of y=1 given the input feature vector). For an example from the training data, the closer the prediction is to the gold label, the "less error" the model makes.

@@anonymous71208 Thank you so much Prof."

Have a look at slide 14 onwards in the following lecture (kzbin.info/www/bejne/qaLYZYGDZdh6irs) - it shows that under the hoods there are three inputs (query, key, value) in a general transformer but these are all taken identical in BERT

Thank you! Yes, I was teaching this course (almost identical) last winter term, here's the GitHub page github.com/trusthlt/nlp-with-deep-learning-lectures/ and here's all the recorded lectures kzbin.info/aero/PL6WLGVNe6ZcB00apoxMtj7WSUOlpm2Xvl Enjoy :)

@anonymous71208 thank you so much!

Thank you! :)

Thanks! So if you look at slide 18, DTransformer gives you the entire list of prob. distributions over vocabulary for each token at once (internally it's using masking so that the attention looks only "back"). And then at slide 21, line 6: the loss is computed for each token and summed up. This is done for each training example, and can be paralellized in batches (which consumer much more memory). Hope it helps!

"embedding vector for each word is part of data to be trained" - I believe you think "embedding vector of each word is to be learned from training data", right? Because "embedding is part of trained data" does not make sense to me, I'm sorry

Clarified in github.com/trusthlt/nlp-with-deep-learning-lectures/issues/8

I mean the slide at this point kzbin.info/www/bejne/hJmvamRsps2XjdU

Which slide number? Appreciate your feedback - can you please open an issue on GitHub instead so we can fix it there? (the same for your other comment)

Fixed in github.com/trusthlt/nlp-with-deep-learning-lectures/issues/9

Thank you, I fixed DAG on GitHub. Regarding the first comment: Why do you think it is not correct? Maybe open an issue on GitHub, where it's easy to write math in markdown, thanks!

@@anonymous71208 you are correct regarding my first comment. I was thinking about something else. I am sorry. The Github info is helpful!

Thanks, hope you find it helpful!

Thanks for commenting! Hope you're doing well :)

The speaker is waiting for the students to come in and sit down

That's right, $V^{D_{[i]}}$ would be even "more correct"... but I hope you get the gist: it's a one-hot vector with 1 at position of i-th word in the document.

Thanks for pointing this out! In my opinion, it's not that badly out-of-sync, you can still follow, I hope

@@anonymous71208 Yes of course! I am just kind of sad that my subtitles were worse than I had hoped for. Working on making them better..

:) Courtesy of Monty Python and the Holy Grail... a hilarious piece, indeed kzbin.info/www/bejne/gX-clGWKdryAosk

@@anonymous71208 Haha awesome. I haven't seen this one from Monty Python, yet and I could totally picture it being hilarious, too.

Thanks a lot! You know the rules - if you like it, share it :)

That's a good question, which I'm afraid only the original author of this particular slide could answer... I don't see any direct connection myself, but there's some research combining regex and fractals out there ( www.sciencedirect.com/science/article/abs/pii/S0096300312007291 )

@@anonymous71208 Hey thank you for the link and your fast reply! That looks very interesting.

A few of the lectures are given by Mohsen Mesgar instead. You should find all of the 'missing' lectures here: kzbin.info/door/4VNXEo_buXs4v1dFfIuaew

Thanks! We split the lectures with my colleague Mohsen Mesgar this year, for the full overview see our Github pages: github.com/dl4nlp-tuda2021/deep-learning-for-nlp-lectures (there's slides and LaTeX source codes too)

Thanks, Yamen! :)

The "blue" vector is (0.5, 0.2, -0.2, -0.9), so the second term is 0.5. Maybe the minus sign is not that visible at 33:30 but if you go back to 32:40, you'll notice it.

@@anonymous71208 true, my mistake🤦🏻‍♂️

Good catch! :) Thx

Thanks Manu! Fixed in the slides on github

lol

Thank you! That's a fair point, but for now we're keeping them private as these are (mostly) graded... But I re-think this next year. Step by step :)

Thanks, much appreciated!

Good catch! Thanks for sharing your experience!

Oh, you're right! >>> SUBSCRIBE <<< and stuff like that... :)) Next time, I promise!