NLP Demystified 14: Machine Translation With Sequence-to-Sequence and Attention

[@BuddingAstroPhysicist]

First of all thanks a lot for these videos , I think they are one of the best on the internet. I have one doubt at 22:15 isn't the input to the scoring function should be h1,s1 instead of h1,s0 for calculating attention for the second output?

[@futuremojo]

Yep! Nice catch. That's a mistake in the diagram. It should be s1.

[@BuddingAstroPhysicist]

@@futuremojo Ok thought so , Thanks a lot again :)

[@byotikram4495]

@futuremojo In the similar context I want to ask a question. So in this period, you are showing how model will generate outputs in the inference time right ? Since from earlier examples we know that we have to apply teacher forcing while training. Am I correct ?

[@ueihgnurt]

My god this video is genius.

[@futuremojo]

Timestamps 00:00:00 Seq2Seq and Attention 00:00:37 Seq2Seq as a general problem-solving approach 00:02:17 Translating language with a seq2seq model 00:05:53 Machine translation challenges 00:09:07 Effective decoding with Beam Search 00:13:04 Evaluating translation models with BLEU 00:16:23 The information bottleneck 00:17:56 Overcoming the bottleneck with Attention 00:22:39 Additive vs Multiplicative Attention 00:26:47 [DEMO] Neural Machine Translation WITHOUT Attention 00:50:59 [DEMO] Neural Machine Translation WITH Attention 01:04:53 Attention as information retrieval

[@klausschmidt982]

I love your clear and succinct explanations. I really appreciate the effort you put in these videos. Thank you.

[@futuremojo]

Thank you, Klaus. I'm glad you're getting value from it.

[@vipulmaheshwari2321]

In the future, when I launch my own company, your series will be an essential foundation for NLP. Your teaching is top-notch, presentations are engaging, and your exceptional clarity in explanations truly stands out! YOU ARE BEST! Long Live Brother

[@futuremojo]

Thank you!

[@ungminhhoai4510]

khóa học của bạn đúng là khai sáng cho những người muốn bắt đầu học ML

[@MadhukaraPhatak-xb4op]

Really love this series. Thank you for sharing theses videos and notebook.

[@cheridhanlissassi8716]

Thank you a lot for sharing. Words can't express my gratitude now. The explanation, the illustration are very good. Wish you all the best and thanks once again.

[@johnbarnes7485]

Loving this series.

[@futuremojo]

Thanks, John. Working hard on the last module on transformers.

[@johnbarnes7485]

@@futuremojo Great!

[@mahmoudreda1083]

You are the BEST, Thank you.

[@horoshuhin]

I can't express how great this series on NLP is. Every video is like a Christmas present. I'm really interested in how you approached your learning about NLP. What have you found helped you along the way? thank you Nitin

[@futuremojo]

Thanks for the kind, motivating message! My approach to NLP is similar to most technical things. The top things: 1) Use multiple resources (books, videos, blog posts, etc). We're in a golden age of autodidactism and we can blend multiple resources to get a cohesive picture of a subject. And we don't even need to go through every resource in its entirety. Perhaps this book explains one concept more intuitively than the other, but the other one fleshes out the math better, and this other resource shows one approach to implementing the concept. Another benefit is that multiple resources act like a voting mechanism. A *lot* of materials out there, particularly the blog posts, have conflicting facts. Trying to get at what's true and what's wrong really forces one to dig deep at the fundamentals. 2) Start at the right level of difficulty, then expand outwards. In my case, it was important to cover both theory and practice, so I'd start with a resource that I could understand. If it was too difficult, I looked for something more practical. Once I was comfortable with that, I revisited the more theoretical stuff, but just enough to serve the goal of helping others gain a solid grounding in the subject. And the more I learned, the further I could expand. 3) Lean into the pain. Learning this stuff was often frustrating because so much out there is hand-wavy or opaque or doesn't answer "why?". Implementing this stuff was even more painful and frustrating. But I kept going because I knew I would eventually get it. And it's important that one believes one can learn anything if one persists long enough. 4) Few things solidify your understanding and act as a reality check than teaching it to others. 5) And of course, actually putting things into practice through code.

[@rabailkamboh8857]

thats a brilliant way to explain these difficult topics. Thankyou so much . Also please make a video on transformer models for neural machine translation. thats quite a hot topic .much needed also

[@futuremojo]

Thank you, Rabail. Transformers (along with pretraining and fine-tuning) is coming up next. Sign up at the course site for updates.

[@exxzxxe]

Very well done! Thanks.

[@anujsolanki5588]

Best channel

[@CSKdataLab]

Your videos are one of the best. Flow of topics and concise language. But, when explaining attention theory and code, two terms are beings used interchangeably --> "encoder output sequence" (y1,y2,y3 and so on) and "encoder hidden states" (h1, h2, h3 and so on). This is creating lot of confusion, making it difficult to follow along the tutorial .... maybe something wrong with my understanding. Can you please make a similar series explaining "Generative Models for Images" and "Reinforcement Learning".

[@aliabasnezhad7872]

great playlist! are you planning to add more videos to this playlist? thanks!

[@futuremojo]

Thanks, Ali. Yep, there is one more module that's going to be released this month. It's going to cover transformers, pre-training, and transfer learning. We'll go over transformers in depth, code one from scratch, and then learn how to use pre-trained transformers for our own projects. Sign up for updates on the course homepage at nlpdemystified.org.

[@aliabasnezhad7872]

@@futuremojo Great, looking forward to it!

[@SaikatDas-jd9xd]

Hi there! Loved the series on NLP. Can you please share any link or resource on how to code up the accuracy function like you did with loss? I would like to calculate accuracy of the epochs.

[@curdyco]

Why the padding (0) is converted to at 34:57 ???? The index for is 1 when i print source_tonkenizer.word_index so why 0 is converted to ? Doea this mean that 0 and 1 index both are reserved for ???

[@jenibites]

How does it can translate “my name is udi” if it never saw the word Udi”

[@futuremojo]

If it's a word-level model, it can't. If your model uses characters or subwords, *maybe* it can translate the individual components of the word to get something sensible, but there needs to be enough data for that to happen. Names, especially from low-resource languages, are a special case which are hard to translate without prior exposure.

[@jenibites]

@@futuremojo what about CopyNet? Could it help?

[@futuremojo]

@@jenibites I don't know. I don't know what CopyNet does and haven't looked at it. How would a system translate the Hungarian name "Andras" to its English-equivalent of "Andrew" without data? And in practical cases, I imagine one wouldn't want to translate the name at all but keep it the same.

[@ateyashuborna1554]

hey, the video is really amazing, however, I was hoping if you could share how to implement and use the BLEU score in your model?

[@amparoconsuelo9451]

Where, when and how can I download the corresponding source code your clear explanation of NLP together with the libraries? I am halfway towards watching all your videos I have downloaded in my cell phone. I will watch them again with the source code.

[@futuremojo]

The notebooks are here: github.com/nitinpunjabi/nlp-demystified

[@amparoconsuelo9451]

@@futuremojo Thanks.

[@sebastianbejarano350]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 4706: character maps to :(

[@sebastianbejarano350]

Adding encoding='utf-8' to open solved it

[@ristaaryantiwi3795]

when I try to execute method translator_trainer.fit() NameError: name 'encoder' is not defined

[@onlysainaa5764]

what is this model accuracy? or bleu score? how to solve it brother?