MIT 6.S191 (2022): Recurrent Neural Networks and Transformers

MIT Introduction to Deep Learning 6.S191: Lecture 2
Recurrent Neural Networks
Lecturer: Ava Soleimany
January 2022
For all lectures, slides, and lab materials: introtodeeplearning.com
Lecture Outline
0:00​ - Introduction
1:59​ - Sequence modeling
4:16​ - Neurons with recurrence
10:09 - Recurrent neural networks
11:42​ - RNN intuition
14:44​ - Unfolding RNNs
16:43 - RNNs from scratch
19:49 - Design criteria for sequential modeling
21:00 - Word prediction example
27:49​ - Backpropagation through time
30:02 - Gradient issues
33:53​ - Long short term memory (LSTM)
35:35​ - RNN applications
40:22 - Attention fundamentals
43:12 - Intuition of attention
44:53 - Attention and search relationship
47:16 - Learning attention with neural networks
54:52 - Scaling attention and applications
56:09 - Summary
Subscribe to stay up to date with new deep learning lectures at MIT, or follow us @MITDeepLearning on Twitter and Instagram to stay fully-connected!!

[@andreas.karatzas]

I'm still trying to figure out how did you manage to perfectly describe the logic behind attention mechanisms in 10 minutes ...

[@MahJohn]

It was absolutely amazing, especially how she connect the notion of search to attention. Thanks for the intuitive connection.

[@ramlalramlal8072]

@@MahJohn .aaaaaaakaaaaaaa

[@jacktrainer4387]

Everything that comes out of MIT is pure gold. You'd think that the concepts would be described at a high, inaccessible level, but that's not the case. The lectures are student friendly & homeworks are challenging and doable.

[@kevinbissinger]

Yeah unlike the Harvards of the world MIT always seemed to focus on having the richest education, not the richest students

[@usrehman5046]

just wanted to know is this just theoretical stuff or they also walk through any code for its implementation ??

[@chiranjeevisagi]

Thank you to Alexander Amini and Ava soleimany for making this course accessible to everyone, which otherwise is a distant dream for many people like myself to learn such high quality content.

[@mohammadalaaelghamry8010]

Just amazing how well those two lectures are layed out, structured and explained, nothing comes close to them in my experience so far, thank you so much Alexander and Ava, heading for the first lab now.

[@SteveSperandeo]

Who needs GPT-3 when we have Ava? Amazingly clear, succinct, and enjoyable presentation. Thank you Ava!

[@robertoooooooooo]

this is genius. This lecture is pure gold. Such difficult concepts like transformers explained in a 15 minutes seems to be impossible but she did it. Thank you MIT!

[@laminsesay1435]

if you are watching, learning and practicing this video, you have be granted a visa to the future. Alexander Amini, Ava Solemany and the rest of the team thanks. you guy are amazing

[@mtamjidhossain]

Set reminder, patiently waiting. It's a great initiative, cant thank the organizers and instructors enough!

[@argc]

This is by far the best explanation of Transformers that I have ever seen. It all makes more sense now. Thanks

[@tantzer6113]

Excellent lecture. Very well designed, clear, intuitive, well balanced. A lot was accomplished in one hour! I learned a lot.

[@ImtithalSaeed]

unable to describe how amazing is this ... thank you Ava

[@user-th7jy6zy1f]

The best explanation of attention mechanism I have ever seen. It is very intuitive and easy to understand.

[@alekseimikhalev1807]

This is just the best, brilliant explanation of RNN and attention mechanism I've ever seen! Thank you guys for such a hard work!

[@arnavraina2615]

Feels like I'm waiting for a much awaited movie trailer! This is quality.

[@bobbymajor8122]

God bless MIT

[@vantagepointmoon]

Takes a really bright mind and a lot of practice together with hands-on experience to pack so much information about a complex subject so briefly and elegantly. I bet it took some back propagation through time to arrive to that.

[@HarshithBachimanchi]

Excellent explanation! This is perhaps the best description about the roots of the attention mechanism, and the intuition behind it. People who follow the route of CNNs -> GANs -> ViTs in their deep learning journey have trouble in understanding the self-attention (without having much knowledge about RNNs). This is like an excellent "bridge" video that fills all the gaps! Great effort by Ava!

[@hyewoncho4797]

This is definitely the best video for describing attention mechanisms and the logic behind them. Many videos only try to review as it is written in the paper. Thank you so much! It really helped me a lot to get the attention even more clearly!

[@AAmini]

Thank you very much!

[@asokakarunananda]

Excellent presentation on the transition from RNN to Attention-based Transformer networks. Thank you

[@281019641]

Precise and very well explained. Thank you for making this course accessible.

[@dianakapralova6025]

Thanks for detailed explanations. Especially, attention!And finally attention all that we need and additionally understand thanks to you:-)

[@jovijsdsilva]

Really intuitive way of teaching. The concepts are explained really well.

[@TheKuhlschrank]

Amazing intuition behind Transformers, thank you!

[@doctorrobot9757]

I am happy to be able to access this course. Your job is much appreciated. THANKS

[@livesinging3924]

Figuring out attention mechanism in minutes is super cool and intuitive. Thanks ava for such a clarity of your train of thoughts throughout the lecture. God bless.

[@SinkingPoint]

This is by far the best explanation of attention that I've seen. It definitely deserves its own video. Maybe a video on transformers that covers attention and some more detail on the other components of the architecture?

[@tantzer6113]

Yes, I too would love to see another lecture on attention and transformers, with a more detailed treatment of the Q, K, V matrices and how they function. I understand what was said but can’t derive it in my own.

[@siak2910]

God bless you, Alexander and co

[@tricialobo9233]

Ava is a very talented lecturer - thanks for the cogent explanation of RNNs.

[@abdullahalsefat9439]

Yes I really appreciate the intuition of attention very much.

[@ShaidaMuhammad]

Finally, I understood the self attention mechanism completely.

[@Fordance100]

Good explanation on self-attention. It gives me better intuitions on the topic.

[@XBJASHITHABATULA]

this is one of the best lectures ever

[@tantzer6113]

Ava Khanum, you put together an excellent lecture. Thank you very much!

[@soheilparsa1]

Such a great explanation! Thanks for sharing!!!

[@kenedymatiassoportella808]

What a lecture! Thanks a lot and keep up the great work.

[@shubham-pp4cw]

nice and clear video explantion on RNNs and Attention

[@user-tc7gu8yo2e]

Thank you for this lecture!

[@abdjahdoiahdoai]

This is very well made. Thanks a lot!

[@arielcarvalhodados]

Thanks for the content, all the details were amazing!

[@Freddychao]

Oh Ava became your wife! congrats!

[@mihaitudor8924]

Nicee... Glad to see you guys are back.

[@seth.photongraphy2673]

The "deep learning couple" is at it again! and congrats!

[@anikettayade6000]

Very great explanation, completely understood. thanks to team MIT 😁

[@jzhuo0119]

This is so well explained - thanks a lot

[@Gerin-Benoit]

Impressive presentation, thanks a lot for sharing !

[@ZorroNumber1]

Passion for science and technology is just oozing out of Ava !! Persian Passion.

[@user-vq8bk1qi5z]

the single most fascinating and entertaining video for me.... let me just express my appreciation YYDS (means best of ever)

[@ansariyusuf4774]

I struggled to wrap my head around neural networks for sequential data. The intuition provided for each concept is perfect, it was so engaging that the timeline of this lecture is now embedded in my head (pun intended ;))

[@josedemetriosanchezchamorr8247]

excellent lecture, I'm excited!

[@dougb70]

43:26 - The Transformers: More than Meets the Eye. Missed it on the news, but saw it on Netflix.

[@waibeeYB]

Thanks a lot for uploading these videos. I have already done the amazing DLS by Andrew Ng, but this video is a perfect summary and a revision. By the way, at 32:30, it's the derivative that is assigned the value 1 whenever the argument is greater than 0. I guess that's a speaking error, but just wanted to put the forward.

[@floopybits8037]

Loved the explanation of Transformers you made such a complicated topic so much elegant and easy to understand

[@style24_7]

Reminder on,

[@subashchandrapakhrin3537]

Wonderful !!! Ava

[@carlotonydaristotile7420]

Very cool lecture.

[@yegounkim1840]

Great lecture

[@jijie133]

Great video!

[@naveenrs7742]

Awesome loved this knowledge shower from India

[@poojadurgi5650]

This is goldddd

[@dariovicenzo8139]

Ava forces herself to be clear splitting in little piece complex concepts to be understandable by us.

[@masoudakhgar8544]

well done. proud of you when I see there are such persian speaking people.

[@ahmedchaoukichami9345]

good work thanks a lot but how we can get the linear layer or how we calculate it to get attention

[@ibrahimadiallo1020]

thank you

[@MarcJSchmidt]

43:00 "long memory" is a bit misleading as transformer has O(n^2) in its attention layer, which is not scalable either (there are variations with better O, but they are less poweful). so it might work better than rnn/lstm, it is not a silver bullet and only slighty increases the context size (to few thousand tokens). we still can't have "long memory" in the sense of hundreds of thousands or millions of tokens.

[@caiomar]

I love these series! Thank you for sharing the knowledge! I am listening to very word! Now I am getting Instagram ads for MIT Full AI course for the hefty price of $3300 USD , I wish I could afford it ;/

[@thosedreams]

Which course is this?

[@shashankrao8791]

The attention mechanism is being used widely in computer vision (vit). In an example of image classification, what type of image data is treated as query, key and value ?

[@leonardooliveirapaschoal552]

Awesome explanation about attention mechenism in a very shot time hahhahaha!

[@mechanicsmechatronicsandro3514]

Bravo!

[@ummantekutti7479]

I'm waiting ....

[@Jkauppa]

you can directly use FIR and IIR filter methods to neural networks, and laplace transforms

[@Jkauppa]

you just have multiple output and inputs, maybe like image filtering

[@Jkauppa]

or video filtering, with the time data with frames included

[@andrewlane7233]

This is absolutely amazing! Does anyone know of any programs that can be used to generate such great looking neural network/code slides? Or the one that was used to create these?

[@AAmini]

Thanks! We use Powerpoint for the slides.

[@aidanmclaughlin5279]

Why does the naive concatenated approach lose a notion of sequence? (I'm confused about the red X next to No Order at 42:40)

[@jingxinzhao878]

Anyone tried Lab1 Part2? I wonder whether anyone get any songs in the last cell? Even the sample_song does not work.

[@mahdica5879]

Is it possible to get a certificate from the related website? I'd be grateful if you guide me.

[@vent_srikar7360]

at time step 42:00 is she referring to encoder and decoder model

[@thosedreams]

What's really "self" in self-attention? If it's the "input" as mentioned in this video, how is it different from attention?

[@newbie8051]

30:28 What exactly is gradient clipping ?

[@allenhsu2033]

HI. i stll can not understand the encoding bottleneck about limitations of RNN, what exactly means about encoding bottleneck? can anyone explain it more clearly?

[@MahJohn]

Since you get some intuition here, consider this lecture here kzbin.info/www/bejne/iWOaoXuBd6qjaZI I found his explanation about one of the shortcomings "encoding bottleneck" of the lstms that motivated the need for the attention mechanism useful. I hope you will find it helpful.

[@OBHACKER]

Where i can find the link of attention mechanisme lab? i did check the link..

[@forheuristiclifeksh7836]

57:51

[@satviksrivastava6632]

Why we have same weights for different inputs??

[@hamidkhan1433]

I need to get some help regarding the process of getting admission in MIT EECS for under graduation that how can I apply and what are the requirements?? If someone has any idea kindly help me out with that I will really appreciate your effort...

[@avoidprogress6002]

I dont really understand the „many to many“ example in the sequence modeling applications. It looks to me that the output sequence has always the same length as the input sequence, which is rare for language translation. Also the first element of the output sequence is produced without any feedback from later timestamps and is not reevaluated later on.

[@avoidprogress6002]

Maybe as an example: the network translates english to german and the input sequence starts with „The“. The Network might then have a first output sequence element of „Das“, which is fine. Then the second element of the Input sequence is „Human“, the network gets „Mensch“ as second output element. Now it would have to change the first output from „Das“ to „Der“, which i dont think would ever happen with the presented Modeling.

[@saurabhchopra]

16:19 Shouldn't the last loss be Lt, rather than L3?

[@MahJohn]

I suppose it's just some typo.

[@AAmini]

Yes, thanks for pointing that out

[@brown8350]

Too good to be true! But why? Thanks a ton!

[@mertbozkurt329]

💯

[@JPwyckoff]

10:46

[@helloansuman]

Very fast videos. Need to slow down and explain key concepts clearly. Otherwise it's like a sweet story.

[@adarshdayalan9056]

How is this available for free

[@abdikafihashihersi4728]

What is linear layer at 50:03?

[@AAmini]

A dense (i.e., fully connected) layer with a linear activation function (i.e., no activation function).

[@abdikafihashihersi4728]

@@AAmini Got it. By the way, I was not expecting this urgent response. Thank you sir Mr. Amini.

[@AAmini]

Thanks for the question!

[@mosca204]

Can I give this video more than 1 like?

[@amielwexler1165]

Comment for algorithm

[@ajaytaneja111]

"Attention Is All you Need" - The intuition of Query, Key and Value is one of the best from what I've read or watched (in other courses) until now....Excellent job Ava Soleimany, thank you

[@keshavmaheshwari521]

I'm waiting ....