MIT 6.S191 (2022): Recurrent Neural Networks and Transformers

[@andreas.karatzas]

I'm still trying to figure out how did you manage to perfectly describe the logic behind attention mechanisms in 10 minutes ...

[@MahJohn]

It was absolutely amazing, especially how she connect the notion of search to attention. Thanks for the intuitive connection.

[@ramlalramlal8072]

@@MahJohn .aaaaaaakaaaaaaa

[@jacktrainer4387]

Everything that comes out of MIT is pure gold. You'd think that the concepts would be described at a high, inaccessible level, but that's not the case. The lectures are student friendly & homeworks are challenging and doable.

[@kevinbissinger]

Yeah unlike the Harvards of the world MIT always seemed to focus on having the richest education, not the richest students

[@usrehman5046]

just wanted to know is this just theoretical stuff or they also walk through any code for its implementation ??

[@mohammadalaaelghamry8010]

Just amazing how well those two lectures are layed out, structured and explained, nothing comes close to them in my experience so far, thank you so much Alexander and Ava, heading for the first lab now.

[@chiranjeevisagi]

Thank you to Alexander Amini and Ava soleimany for making this course accessible to everyone, which otherwise is a distant dream for many people like myself to learn such high quality content.

[@ajaytaneja111]

"Attention Is All you Need" - The intuition of Query, Key and Value is one of the best from what I've read or watched (in other courses) until now....Excellent job Ava Soleimany, thank you

[@robertoooooooooo]

this is genius. This lecture is pure gold. Such difficult concepts like transformers explained in a 15 minutes seems to be impossible but she did it. Thank you MIT!

[@SteveSperandeo]

Who needs GPT-3 when we have Ava? Amazingly clear, succinct, and enjoyable presentation. Thank you Ava!

[@argc]

This is by far the best explanation of Transformers that I have ever seen. It all makes more sense now. Thanks

[@laminsesay1435]

if you are watching, learning and practicing this video, you have be granted a visa to the future. Alexander Amini, Ava Solemany and the rest of the team thanks. you guy are amazing

[@SeshaB]

Excellent explanation! This is perhaps the best description about the roots of the attention mechanism, and the intuition behind it. People who follow the route of CNNs -> GANs -> ViTs in their deep learning journey have trouble in understanding the self-attention (without having much knowledge about RNNs). This is like an excellent "bridge" video that fills all the gaps! Great effort by Ava!

[@hyewoncho4797]

This is definitely the best video for describing attention mechanisms and the logic behind them. Many videos only try to review as it is written in the paper. Thank you so much! It really helped me a lot to get the attention even more clearly!

[@AAmini]

Thank you very much!

[@SinkingPoint]

This is by far the best explanation of attention that I've seen. It definitely deserves its own video. Maybe a video on transformers that covers attention and some more detail on the other components of the architecture?

[@tantzer6113]

Yes, I too would love to see another lecture on attention and transformers, with a more detailed treatment of the Q, K, V matrices and how they function. I understand what was said but can’t derive it in my own.

[@tantzer6113]

Excellent lecture. Very well designed, clear, intuitive, well balanced. A lot was accomplished in one hour! I learned a lot.

[@alekseimikhalev1807]

This is just the best, brilliant explanation of RNN and attention mechanism I've ever seen! Thank you guys for such a hard work!

[@ImtithalSaeed]

unable to describe how amazing is this ... thank you Ava

[@asokakarunananda]

Excellent presentation on the transition from RNN to Attention-based Transformer networks. Thank you

[@arnavraina2615]

Feels like I'm waiting for a much awaited movie trailer! This is quality.

[@bobbymajor8122]

God bless MIT

[@ภาณุปโยราศิสกุล]

The best explanation of attention mechanism I have ever seen. It is very intuitive and easy to understand.

[@mtamjidhossain]

Set reminder, patiently waiting. It's a great initiative, cant thank the organizers and instructors enough!

[@vantagepointmoon]

Takes a really bright mind and a lot of practice together with hands-on experience to pack so much information about a complex subject so briefly and elegantly. I bet it took some back propagation through time to arrive to that.

[@tricialobo9233]

Ava is a very talented lecturer - thanks for the cogent explanation of RNNs.

[@livesinging3924]

Figuring out attention mechanism in minutes is super cool and intuitive. Thanks ava for such a clarity of your train of thoughts throughout the lecture. God bless.

[@seth.photongraphy2673]

The "deep learning couple" is at it again! and congrats!

[@XBJASHITHABATULA]

this is one of the best lectures ever

[@281019641]

Precise and very well explained. Thank you for making this course accessible.

[@TheKuhlschrank]

Amazing intuition behind Transformers, thank you!

[@dianakapralova6025]

Thanks for detailed explanations. Especially, attention!And finally attention all that we need and additionally understand thanks to you:-)

[@ShaidaMuhammad]

Finally, I understood the self attention mechanism completely.

[@dougb70]

43:26 - The Transformers: More than Meets the Eye. Missed it on the news, but saw it on Netflix.

[@siak2910]

God bless you, Alexander and co

[@MarcJSchmidt]

43:00 "long memory" is a bit misleading as transformer has O(n^2) in its attention layer, which is not scalable either (there are variations with better O, but they are less poweful). so it might work better than rnn/lstm, it is not a silver bullet and only slighty increases the context size (to few thousand tokens). we still can't have "long memory" in the sense of hundreds of thousands or millions of tokens.

[@jovijsdsilva]

Really intuitive way of teaching. The concepts are explained really well.

[@Fordance100]

Good explanation on self-attention. It gives me better intuitions on the topic.

[@shubham-pp4cw]

nice and clear video explantion on RNNs and Attention

[@abdullahalsefat9439]

Yes I really appreciate the intuition of attention very much.

[@ansariyusuf4774]

I struggled to wrap my head around neural networks for sequential data. The intuition provided for each concept is perfect, it was so engaging that the timeline of this lecture is now embedded in my head (pun intended ;))

[@sarahsalt-l8n]

Thank you for this lecture!

[@jzhuo0119]

This is so well explained - thanks a lot

[@刘天阳-d9v]

the single most fascinating and entertaining video for me.... let me just express my appreciation YYDS (means best of ever)

[@ZorroNumber1]

Passion for science and technology is just oozing out of Ava !! Persian Passion.

[@soheilparsa1]

Such a great explanation! Thanks for sharing!!!

[@doctorrobot9757]

I am happy to be able to access this course. Your job is much appreciated. THANKS

[@mihaitudor8924]

Nicee... Glad to see you guys are back.

[@tantzer6113]

Ava Khanum, you put together an excellent lecture. Thank you very much!

[@yhbarve]

Thanks a lot for uploading these videos. I have already done the amazing DLS by Andrew Ng, but this video is a perfect summary and a revision. By the way, at 32:30, it's the derivative that is assigned the value 1 whenever the argument is greater than 0. I guess that's a speaking error, but just wanted to put the forward.

[@SphereofTime]

57:51

[@Freddychao]

Oh Ava became your wife! congrats!

[@abdjahdoiahdoai]

This is very well made. Thanks a lot!

[@arielcarvalhodados]

Thanks for the content, all the details were amazing!

[@kenedymatiassoportella808]

What a lecture! Thanks a lot and keep up the great work.

[@masoudakhgar8544]

well done. proud of you when I see there are such persian speaking people.

[@Jkauppa]

you can directly use FIR and IIR filter methods to neural networks, and laplace transforms

[@Jkauppa]

you just have multiple output and inputs, maybe like image filtering

[@Jkauppa]

or video filtering, with the time data with frames included

[@aidanmclaughlin5279]

Why does the naive concatenated approach lose a notion of sequence? (I'm confused about the red X next to No Order at 42:40)

[@vent_srikar7360]

at time step 42:00 is she referring to encoder and decoder model

[@josedemetriosanchezchamorr8247]

excellent lecture, I'm excited!

[@dariovicenzo8139]

Ava forces herself to be clear splitting in little piece complex concepts to be understandable by us.

[@saurabhchopra]

16:19 Shouldn't the last loss be Lt, rather than L3?

[@MahJohn]

I suppose it's just some typo.

[@AAmini]

Yes, thanks for pointing that out

[@Gerin-Benoit]

Impressive presentation, thanks a lot for sharing !

[@caiomar]

I love these series! Thank you for sharing the knowledge! I am listening to very word! Now I am getting Instagram ads for MIT Full AI course for the hefty price of $3300 USD , I wish I could afford it ;/

[@thosedreams]

Which course is this?

[@shashankrao8791]

The attention mechanism is being used widely in computer vision (vit). In an example of image classification, what type of image data is treated as query, key and value ?

[@naveenrs7742]

Awesome loved this knowledge shower from India

[@anikettayade6000]

Very great explanation, completely understood. thanks to team MIT 😁

[@carlotonydaristotile7420]

Very cool lecture.

[@floopybits8037]

Loved the explanation of Transformers you made such a complicated topic so much elegant and easy to understand

[@thosedreams]

What's really "self" in self-attention? If it's the "input" as mentioned in this video, how is it different from attention?

[@andrewlane7233]

This is absolutely amazing! Does anyone know of any programs that can be used to generate such great looking neural network/code slides? Or the one that was used to create these?

[@AAmini]

Thanks! We use Powerpoint for the slides.

[@subashchandrapakhrin3537]

Wonderful !!! Ava

[@jijie133]

Great video!

[@yegounkim1840]

Great lecture

[@poojadurgi5650]

This is goldddd

[@leonardooliveirapaschoal552]

Awesome explanation about attention mechenism in a very shot time hahhahaha!

[@ahmedchaoukichami9345]

good work thanks a lot but how we can get the linear layer or how we calculate it to get attention

[@mahdica5879]

Is it possible to get a certificate from the related website? I'd be grateful if you guide me.

[@avoidprogress6002]

I dont really understand the „many to many“ example in the sequence modeling applications. It looks to me that the output sequence has always the same length as the input sequence, which is rare for language translation. Also the first element of the output sequence is produced without any feedback from later timestamps and is not reevaluated later on.

[@avoidprogress6002]

Maybe as an example: the network translates english to german and the input sequence starts with „The“. The Network might then have a first output sequence element of „Das“, which is fine. Then the second element of the Input sequence is „Human“, the network gets „Mensch“ as second output element. Now it would have to change the first output from „Das“ to „Der“, which i dont think would ever happen with the presented Modeling.

[@allenhsu2033]

HI. i stll can not understand the encoding bottleneck about limitations of RNN, what exactly means about encoding bottleneck? can anyone explain it more clearly?

[@MahJohn]

Since you get some intuition here, consider this lecture here kzbin.info/www/bejne/iWOaoXuBd6qjaZI I found his explanation about one of the shortcomings "encoding bottleneck" of the lstms that motivated the need for the attention mechanism useful. I hope you will find it helpful.

[@OBHACKER]

Where i can find the link of attention mechanisme lab? i did check the link..

[@abdikafihashihersi4728]

What is linear layer at 50:03?

[@AAmini]

A dense (i.e., fully connected) layer with a linear activation function (i.e., no activation function).

[@abdikafihashihersi4728]

@@AAmini Got it. By the way, I was not expecting this urgent response. Thank you sir Mr. Amini.

[@AAmini]

Thanks for the question!

[@jingxinzhao878]

Anyone tried Lab1 Part2? I wonder whether anyone get any songs in the last cell? Even the sample_song does not work.

[@satviksrivastava6632]

Why we have same weights for different inputs??

[@Qurat4k]

Reminder on,

[@helloansuman]

Very fast videos. Need to slow down and explain key concepts clearly. Otherwise it's like a sweet story.

[@hamidkhan1433]

I need to get some help regarding the process of getting admission in MIT EECS for under graduation that how can I apply and what are the requirements?? If someone has any idea kindly help me out with that I will really appreciate your effort...

[@adarshdayalan9056]

How is this available for free

[@ummantekutti7479]

I'm waiting ....

[@mechanicsmechatronicsandro3514]

Bravo!

[@brown8350]

Too good to be true! But why? Thanks a ton!

[@ibrahimadiallo1020]

thank you

[@JPwyckoff]

10:46

[@mosca204]

Can I give this video more than 1 like?

[@mertbozkurt329]

💯

[@amielwexler1165]

Comment for algorithm

[@newbie8051]

30:28 What exactly is gradient clipping ?

[@keshavmaheshwari521]

I'm waiting ....