C5W3L07 Attention Model Intuition

Take the Deep Learning Specialization: bit.ly/2TF1B06
Check out all our courses: www.deeplearning.ai
Subscribe to The Batch, our weekly newsletter: www.deeplearning.ai/thebatch
Follow us:
Twitter: / deeplearningai_
Facebook: / deeplearninghq
Linkedin: / deeplearningai

[@TheBlackhawk2011]

"if you can't explain it simply you don't understand it well enough" Andrew is one of the best instructors in the world. No wonder he teaches at Standord.

[@grownupgaming]

as someone from berkeley, this is a good comment!

[@Mesenqe]

I can't believe that, Siraj Raval has around 700K subscribers and this most valuable channel only 76K.

[@AIPlayerrrr]

the worst channel on youtube...notorious...

[@Mesenqe]

@@AIPlayerrrr ok, if #Deeplearning_ai is the worst channel, where almost everyone become the master of Machine learning, and Deep learning...plz recommend me which channel you use? N.B. Not in Chinese, in English.

[@AIPlayerrrr]

Gery W. Adhane I am talking about Siraj Raval....

[@AIPlayerrrr]

Gery W. Adhane I remember Lex’s interview on Siraj Raval which Siraj said he only knew 50% of the materials lolll

[@Mesenqe]

@@AIPlayerrrr Oh sorry, I thought you are referring to #deeplearning_ai. In that case I agree. He manipulates us all.

[@mehmetcelepkolu7660]

3:24 - Woah, the legend is speaking French!

[@frankie59er]

These explanations are top-notch, definitely deserving of way more views

[@MasterofPlay7]

you know the BERT model is the best nlp model right?

[@PookyCodes]

Thank you so much for this valuable video!

[@aryanyekrangi7093]

Great video series!

[@arborymastersllc.9368]

Would definitely have to shift the location context weights for different languages as expressions shift word order across languages. Example: 3 left 2 right for language A. And 2 left 4 right Language B, with language specific variance to weight distribution themselves

[@bionhoward3159]

thank you!

[@anggipermanaharianja6122]

really clear explanation

[@arthurswanson3285]

Well explained.

[@theSpicyHam]

I learned an lot, thank you

[@youssefdirani]

Magical voice

[@yeming3777]

helps me a lot

[@zhenpingli2313]

real great

[@marinzhao3513]

解析的很清晰

[@fackarov9412]

to me seems like applying kernels(1D CNN) to RNN and call it "attention"

[@sandipansarkar9211]

good expalantion but need to see again

[@taku8751]

I hope the cursor can be bigger, I can not see it.

[@mohsenboughriou9846]

man u're the best

[@MrAkhilanil]

Sounds cool, someone should build a chat bot with this tech!

[@abail7010]

The only thing I am not understanding is why its harder to translate shorter sentences? :)

[@namHoang-lb6jp]

Some segments in the video are stamped not adjacent to each other

[@arborymastersllc.9368]

Recursive context checking every so many words? Like after every noun verb combo identified, recheck context appropriatness.

[@saeedullah5365]

Why LSTM have more accuracy than Bi directional LSTM though is new approach

[@thepresistence5935]

Attention model invented by "avengers" 😆😆 2:55

[@beniev1]

How can one know how many words should be in the output?

[@beniev1]

I mean when you translate new sentence, not in the training...

[@coralbow]

@@beniev1 There's a separate RNN for both encoder and decoder. Encoder RNN takes in a fixed length sequence as input and decoder RNN outputs a fixed length sequence. To make every sentence the same length (for encoder RNN), they usually add 0s to the start of sentences which are shorter than required length. For decoder RNN the idea is similar: they output a fixed length sequence of hidden states (word distributions) and for each state in that sequence they choose the most probable word. In that word distribution there is a word (EOS which basically stands for 0) they don't show this in visualisation because it adds no information.

[@abbashoseini9344]

@@beniev1 I think according to this video there is no need to know number of words for your translation and you can't know this until you translate it . your translation will be finished when your network generate EOS word. and then you can count how many words network has generated for translation.

[@louisraison]

@@abbashoseini9344 Exactly ! What is a bit misleading in the example here is that each word in the French sentence is translated by exactly one in English, but any word could actually be predicted instead, and the translation of this word could come later.

[@abbashoseini9344]

it is important to note that it depend on you application. for example in sequence tagging problems you need to force model to make output and input have equal length. attention mechanism has this power to generalize all kind of problems with various constraints on input and output length.

[@danielcai1017]

3:29 it seems he knows French well

[@bhimireddyananthreddy1487]

What does some set of features at 3:36 mean?

[@ThePritt12]

an encoding = features of a sentence

[@Utbdankar]

To determine each feature vector(the set of features) you use the input of the word itself and the previous feature vector, which outputs a new feature vector. You can think of the feature vector as "everything needed to translate the current word that came before the word in the sentence".

[@2005sty]

It is not correct that a human translator carries out translation part by part especially in the case of translation of two different languages with different grammatic rules.

[@isodoubIet]

The "parts" don't have to be in the same order.

[@moodiali7324]

very good content with very bad audio quality, hope that improves one day

[@nichosetiawan1377]

the video image is too poor, you need to fix it more

[@fahds2583]

video starts here 3:11

[@ananthakrishnank3208]

3:00

[@procrastipractice]

Good luck with German where one verb can be split over a huge distance

[@heejuneAhn]

Oh, I see that is why Google translation is still bad with Engish-Japanese or Korean!

[@socratic-programmer]

Is that because the sentences read in an unusual order? Incidentally, recent models have become a lot better at multi-translation, so maybe they are better now than before.

[@peterfireflylund]

@@socratic-programmer No, it's not about an unusual order. That's actually quite easy to handle. It's because Korean and Japanese have completely different grammar from English. They are agglutinating, have pretty free word order because they both "tag" their words with short sounds that tell what role they play in the sentence, and they both allow most of the "real" sentence to be left out if it can be inferred from context. There are also problems with the semantic mapping between J/K and E where context is needed to figure out how to translate words/idioms. To top it all off, Japanese and Korean both have really complicated systems of honorifics. Oh, and copula is handled in *completely* different ways in J/K and E. Current translation systems are really bad at handling context above the sentence level, so... you can see the problems. Wikipedia has pretty good articles on Japanese, Korean, and English grammar.

[@peterfireflylund]

Forgot to add that tokenization is another issue. There are translating neural networks that are completely end-to-end: they take characters/punctuation as input and produce characters/punctuation as output. Most deep learning systems use a tokenizer before the input and a "detokenizer" after the output. Such a tokenizer may give all common words their own token number and it may split rarer words into smaller components, often using simple rules based on tables and regular expressions. It may also turn things like "isn't" into "is not" for English and "du" into "de le" for French. How to properly tokenize ideographic scripts like Chinese hanzi and Japanese kanji (and Korean Hanja) is still a research subject. Actually, even tokenization for *English* is still a research subject!

[@socratic-programmer]

@@peterfireflylund Some of those linguistic terms elude me, but that makes sense. Thanks for that explanation.

[@dnaphysics]

'Attention' should have been called 'context'.

[@francois-xaviermenage4531]

The French sentences are written on a very wrong way..

[@jozefkoslinski4111]

「動画の音が良くない」、

[@umairgillani699]

the audio quality from all professor NG's videos sucks!!