Sequence-to-Sequence (seq2seq) Encoder-Decoder Neural Networks, Clearly Explained!!!

[@statquest]

To learn more about Lightning: lightning.ai/ Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@graedy2]

One of the best channels on youtube! Wanted to provide some constructive criticism: Either I am blind or you have forgotten to link the og paper you show in the video in the video description.

[@statquest]

@@graedy2 Here it is: arxiv.org/abs/1409.3215

[@tornadospin9]

This channel is like the Khan Academy of neural networks, machine learning, and statistics. Truly remarkable explanations

[@statquest]

Thank you!

[@eliaborras9834]

it's way better :) khan Academy does not have such cool songs =:)

[@reinerheiner1148]

This channel is gold. I remember how, for my first coding job, where I had no programming knowledge (lol) but had no choice than to take it anyways, I quickly had to learn php and mysql. To get myself started, I searched for the simplest php coding books and then got myself two books from the php & mysql for kids series, even though I was already in my mid twenties. Long story short, I quickly learned the basics, and did code for a living. Complex topics don't have to be complex, in fact they are always built on building blocks of simple concepts and can be explained and taught as such IMHO. Thank you so much for explaining it KISS style. Because once again, I have to learn machine learning more or less from scratch, but this time for my own personal projects.

[@statquest]

BAM! I'm glad my videos are helpful. :)

[@cat-a-lyst]

I literally searched everywhere and finally came across your channel. seems like gradient descent worked fine .

[@statquest]

:)

[@mohammadrezababaei8871]

I am nearly finished with the ML playlist, and I must say that I am extremely satisfied with the content and I do regret spending so much time on other videos and courses before finding your channel. Thank you for creating this invaluable resource. I hope you continue to create videos on more complex subjects, particularly in the medical field. Once again, thank you very much.

[@statquest]

Thank you very much!

[@gabip265]

I can't thank you enough for these tutorials on NLP. From the first tutorial related to RNNs to this tutorial, you explained so concisely and clearly notions that I have struggled and was scared to tackle for couple of weeks, due to the amount of papers/tutorials someone should read/watch in order to be up to date with the most recent advancement in NLP/ASR. You jump-started my journey and made it much more pleasant! Thank you so much!

[@statquest]

Glad I could help!

[@juliali3081]

It took me more than 16 minutes (the length of the video) to get what happens since I have to pause the video to think, but I should say it is very clearly explained! Love your video!!

[@statquest]

Hooray! I'm glad the video was helpful. Now that you understand Seq2Seq, I bet you could understand Transformers relatively easily: kzbin.info/www/bejne/sKm0qoeBbdaor7s

[@gilao]

Another great explanation! It is so comforting to know that whatever I don't understand in class, I can always find a video in your channel and be confident that I will understand by the end. Thank you!

[@statquest]

Glad it was helpful!

[@m.taufiqaffandi]

This is amazing. Can't wait for the Transormers tutorial to be released.

[@statquest]

Thanks!

[@rachit7185]

An awesome video as always! Super excited for videos on attention, transformers and LLM. In the era of AI and ChatGPT, these are going to go viral, making this knowledge accessible to more people, explained in a much simpler manner.

[@statquest]

Thanks!

[@shafiullm]

I got my finals of my final course in my final day tomorrow of my undergraduate journey and you posted this exactly few hours ago.. thats a triple final bam for me

[@statquest]

Good luck! :)

[@paulaoges5525]

exact same situation bro

[@ligezhang4735]

Wonderful tutorial! Studying on Statquest is really like a recursive process. I first search for transformers, then follow the links below all the way to RNN, and finally study backward all the way to the top! That is a really good learning experience thanks!

[@statquest]

Hooray! I'm glad these videos are helpful. By the way, here's the link to the transformers video: kzbin.info/www/bejne/sKm0qoeBbdaor7s

[@paulk6900]

I just wanted to mention that I really love and appreciate you as well as your content. You have been an incredible inspiration for me and my friends to found our own start up im the realm of AI without any prior knowledge. Through your videos I was capable to get a basic overview about most of the important topics and to do my own research according to those outlines. So without taking into consideration if the start up fails or not, I am still great full for you and I guess the implications that I got out of your videos led to a path that will forever change my life. So thanks❤

[@statquest]

BAM! And good luck with the start up!!!

[@mateuszsmendowski2677]

Coming from video about LSTMs. Again, the explanation is so smooth. Everything is perfectly discussed. I find it immersively useful to refresh my knowledge base. Respect!

[@statquest]

Glad it was helpful!

[@serkanbesim131]

I genuinely love you for these videos holy smokes

[@statquest]

BAM! :)

[@diamondstep3957]

Love your videos Josh! Thanks for sharing all your knowledge in such a concise way.

[@statquest]

Thank you! :)

[@ZinzinsIA]

Absolutely amazing as always, thank you so much. Can't wait for attention and transformers lessons, it will again help me so much for my current internship !

[@statquest]

bam!

[@shahrozansari672]

I like how your videos backpropogate so I have to watch all of them if I want to understand one.

[@statquest]

Ideally it would be nice to just have one video that explains all the details, but I think for this specific topic it was pretty important to split the individual bits into separate videos since they can all stand on their own.

[@shahrozansari672]

@@statquest Yeah it's a fair point and thankyou so much for making these videos.

[@MCMelonslice]

Incredible, Josh. This is exactly what I needed right now!

[@statquest]

BAM! :)

[@AI_ML_DL_LLM]

Great video! thanks for producing such a high quality, clear and yet simple tutorial

[@statquest]

Thank you!

[@sheiphanshaijan1249]

Been waiting for this for so long. ❤. Thank you Josh.

[@statquest]

Hooray! :)

[@fancytoadette]

Omg I’m sooooooo happy that you are making videos on this!!! Have been heard it a lot but never figured it out until today 😂 cannot wait for the ones on attention and transformers ❤ Again thank you for making these awesome videos they really helped me A LOT

[@statquest]

Thank you very much! :)

[@Foba_Bett]

These videos are doing god's work. Nothing even comes close.

[@statquest]

Thank you!

[@GenesisChat]

14:34 seems like a painful training, but one that, added to great compassion for other students, led you to produce those marvels of good education materials!

[@statquest]

Thank you!

[@Er1kth3b00s]

Amazing! Can't wait to check out the Self-Attention and Transformers 'Quests!

[@statquest]

Thanks! :)

[@53_ritamghosh63]

Wow man, triple bam indeed, the concept is crystal clear to me now !

[@statquest]

Thanks!

[@민서김-f2d]

Thank you, so I now can have intuition of why the name is encoder and decoder, that I've curious for full 1 years.

[@statquest]

bam! :)

[@ygbr2997]

using as the first input in the decoder to start the whole translation does appear to be magical

[@statquest]

It's essentially a placeholder to get the translation started. You could probably start with anything, as long as you were consistent.

[@timmygilbert4102]

Can't wait to see the stanford parser head structure explained as a step towards attention!

[@statquest]

I'll keep that in mind.

[@mattymatics22]

BAM!! Loved the quest, as always!

[@statquest]

Thank you so much! :)

[@sheldonsebastian7232]

Yaas more on Transformers! Waiting for statquest illustrated book on those topics!

[@statquest]

I'm working on it! :)

[@KR-fy3ls]

Been waiting for this from you. Love it.

[@statquest]

Thanks!

[@bibhutibaibhavbora8770]

See this is the kind of explanation I was waiting for❤

[@statquest]

bam!

[@Sarifmen]

We are getting to Transformers. LEETS GOOO

[@statquest]

:)

[@Truth_had_to_comeout]

Vamosssss. 😂

[@roczhang2009]

Hey, thanks for your awesome work in explaining these complex concepts concisely and clearly! However, I did have some confusion after watching this video for the first time (I cleared them by watching it several times) and wanted to share these notes with you since I think they could potentially make the video even better: 1. The "ir vamos y " tokens in the decoding layer are a bit misleading in two ways: a. I thought "ir" and "y" stood for the "¡" and "!" in "¡Vamos!" Thus, I was expecting the first output from the decoding layer to be "ir" instead of "vamos." b. The position of the "" token is also a bit misleading because I thought it was the end-of-sentence token for "¡Vamos!" and wondered why we would start from the end of the sentence. I think " ir vamos y" would have been easier to follow and would cause less confusion. 2. [6:20] One silly question I had at this part was, "Is each value of the 2-D embedding used as an input for each LSTM cell, or are the two values used twice as inputs for two cells?" Since 2 and 2 are such a great match, lol. 3. One important aspect that is missing, IMO, in several videos is how the training stage is done. Based on my understanding, what's explained in this video is the inference stage. I think training is also very worth explaining (basically how the networks learn the weights and biases in a certain model structure design). 4. Another tip is that I felt as the topic gets more complicated, it's worth making the video longer too. 16 minutes for this topic felt a little short for me. Anyways, this is still one of the best tutorial videos I've watched. Thank you for your effort!!

[@statquest]

Sorry you had trouble with this video, but I'm glad you were able to finally figure things out. To answer your question, the 2 embedding values are used for both LSTMs in the first layer. (in other words, both LSTMs in the first layers get the exact same input values). If you understand the basics of backpropagation ( kzbin.info/www/bejne/f3-ViaB4na5_qpY ), then really all you need to know about how this model is trained is how "teacher-forcing" is used. Other than that, there's no difference from a normal Neural Network. That said, I also plan on creating a video where we code this exact network in PyTorch and in that video I'll show how this specific model is trained.

[@roczhang2009]

Can't wait to learn the coding part from you too. And thanks for your patient reply to every comment. It's amazing. @@statquest

[@mitchynz]

Hi Josh - this one didn't really click for me. There's no 'aha' moment that I get with almost all your videos. I think we need to walk through the maths - or have a a follow up - even if it takes an hour. Perhaps a guest lecturer or willing student (happy to offer my time) ... alas I guess as the algorithms become more complex the less reasonable this becomes, however you did a masterful job simplifying CNN's that I've never seen elsewhere so I'm sure if anyone can do it, you can! Thanks regardless - there's a lot of joy in this community thanks to your teaching.

[@statquest]

Yeah - it was a little bit of a bummer that I couldn't do the math all the way through. I'm working on something like that for Transformers and we'll see if I can pull it off. The math might have to be a separate video.

[@harshmittal63]

Hi Josh, I have a question at time stamp 11:54. Why are we feeding the token to the decoder, shouldn't we feed the (start of sequence) token to initiate the translation? Thank you for sharing these world-class tutorials for free :) Cheers!

[@statquest]

You can feed whatever you want into the decoder to get it initialized. I use because that is what they used in the original manuscript. But we could have used .

[@BHAVYAJAIN-lw1fo]

cant wait for the tranformers video

[@statquest]

Me too. I'm working on it right now.

[@enestemel9490]

Thank you Joshhh !!! I really love the way you teach everything

[@statquest]

Thank you!

[@hawawaa1168]

yoooo Lets goooooo , Josh posted !

[@statquest]

bam! :)

[@khaikit1232]

Hi Josh, Thanks for the much-needed content on encoder-decoder! :) However, I had a few questions/clarifications in mind: 1) Do the number of cells between each layer within the Encoder or Decoder be the same? 2) From the illustration of the model, the information from the second layer of the encoder will only flow to the second layer of the decoder. Is this understanding correct? 3) Building off from 2), does the number of cells from each layer of the Encoder have to be equal to the number of cells from each corresponding layer of the Decoder? 4) Do the number of layers between the decoder & encoder have to be the same? I think my main problem is trying to visualise the model architecture and how the information flows if there are different numbers of cells/layers. Like how would an encoder with 3 layers and 2 cells per layer connect to the decoder that perhaps have only 1 layer but 3 cells.

[@statquest]

First, the important thing is that there are no rules in neural networks, just conventions. That said, in the original manuscript (and in pretty much every implementation), the number of LSTMs per layer and the number of layers are always equal in the Encoder and the Decoder - this makes it easy for the context vector to connect the two sets of LSTMs. However, if you want to come up with a different strategy, there are no rules that say you can't do it that way - you just have to figure out how to make it work.

[@juaneshberger9567]

Best ML vids out there, thanks!

[@statquest]

Wow, thanks!

[@willw4096]

1:57 5:25 5:48 5:56❗ 6:07 6:12 6:41 8:08 8:20 8:30

[@statquest]

bam!

[@alecrodrigue]

awesome vid as always Josh :)

[@statquest]

Thank you!

[@tupaiadhikari]

Thank you Professor Josh, now I understand the working of Se2Seq models completely. If possible can you make a python based coding video either in Keras or Pytorch so that we can follow it completely through code? Thanks once again Professor Josh !

[@statquest]

I'm working on the PyTorch Lightning videos right now.

[@arshdeepkaur8842]

Thanks@@statquest

[@101alexmartin]

Thanks for the video Josh, it’s very clearly explained. I have a technical question about the Decoder, that I might have missed during the video. How can you dynamically change the sequence lenght fed to the Decoder? In other words, how can you unroll the decoder’s lstms? For instance, when you feed the token to the (let’s say, already trained) Decoder, and then you get and feed it together with the token, the length of the input sequence to the decoder dynamically grows from 1 () to 2 (+). The architecture of the NN cannot change, so I’m unsure on how to implement this. Cheers! 👍🏻👍🏻

[@statquest]

When using the Encoder-Decoder for translation, you pass the tokens (or words) to the decoder one at a time. So we start by passing to the decoder and it predicts "vamos". So then we pass "vamos" (not + vamos) to the same decoder and repeat, passing one token to the decoder at a time until we get .

[@101alexmartin]

@@statquest Thanks for the reply. I see your point. Do you iterate then on the whole Encoder-Decoder model or just on the Decoder? In other words, is the input to the model Let’s + go + in the first iteration? Or do we just run the Encoder once to get the context vector and iterate over the Decoder, so that the input is just one word at a time (starting with )? In this last case, I assume we have to update the cell and hidden states for each new word we input to the Decoder

[@statquest]

@@101alexmartin In this case, we have to calculate the values for input one word at a time, just like for the output - this is because the Long and Short Term memories have to be updated by each word sequentially. As you might imagine, this is a little bit of a computational bottleneck. And this bottleneck was one of the motivations for Transformers, which you can learn about here: kzbin.info/www/bejne/sKm0qoeBbdaor7s and here: kzbin.info/www/bejne/mIKYc6Klob1sd8k (NOTE: you might also want to watch this video on attention first: kzbin.info/www/bejne/hoTWZ6Guo8x_bM0 )

[@101alexmartin]

@@statquest thanks for your reply. What do you mean by calculating the values for the input one word at a time? Do you mean that the input to the model in the first iteration would be [Let’s, go, EOS] and for the second iteration it would be [Let’s, go, vamos]? Or do you mean that you only use the Encoder once, to get the context vector output when you input [Let’s, go], and then you just focus on the Decoder, initializing it with the Encoder context vector in the first iteration, and then iterating over the Decoder (i.e over a LSTM architecture built for an input sequence length of 1), using the cell and hidden states of previous iterations to initialize the LSTM, until you get [EOS] as output?

[@statquest]

@@101alexmartin What I mean is that we start by calculating the context vector (the long and short term memories) for "let's". Then we plug those values into the unrolled LSTMs that we use for "go", and keep doing that, calculating the context vector one word at a time, until we get to the end up of the input. Watching the video on Transformers may help you understand the distinction that I'm making here between doing things sequentially vs. in parallel.

[@张超-o2z]

Another amazing video and I cannot thank you enough to help us understand neural network in a such friendly way! At 4:48, you mentioned "because the vocabulary contains a mix of words and symbols, we refer to the individual elements in a vocabulary as tokens" . I wonder if this applies to models like GPT when it's about "limits of the context length (e.g., GPT3.5, 4096 tokens) or control the output token size.

[@statquest]

Yes, GPT models are based on tokens, however, tokens are usually word fragments, rather than whole words. That's why each word counts as more than one token.

[@vicadegboye684]

Once again, we can't appreciate you enough for the fantastic videos! I'd love a clarification if you don't mind. At 8:44 - 8:48, you mentioned that the decoder has LSTMs which have 2 layers and each layer has 2 cells. But, in the image on the screen, I can only see 1 cell per layer. Is there something I'm missing? Meanwhile, thanks a lot for replying on your videos. I was honored when you replied promptly to comments on your previous video. Looking forward to your response on this one.

[@statquest]

The other LSTMs are there, just hard to see.

[@anupmandal5396]

Awesome Video. Please make a video on GAN and BPTT. Request.....

[@statquest]

I'll keep those topics in mind.

[@anupmandal5396]

@@statquest Thank you sir.

[@dsagman]

this is my homework assignment today. how did youtube know to put this in my feed? maybe the next statquest will explain. 😂

[@statquest]

bam! :)

[@Priya_dancelover]

you are my NEW GOD 😇😶‍🌫

[@statquest]

:)

[@cat-a-lyst]

you are an excellent teacher

[@statquest]

Thank you! 😃

[@WunrryWu]

This video is awesome,they helped me a lot.Thank you very much.

[@statquest]

Thanks!

[@kmc1741]

I'm a student who studies in Korea. I love your video and I appreciate that you made these videos. Can I ask you when does the video about 'Transformers' upload? It'll be big help for me to study NLP. Thank you.

[@statquest]

I'm working on it right now, so it will, hopefully, be out sometime in June.

[@jamesmina7258]

thank you so much, I learn from this vedio a lot about LLM

[@statquest]

Glad to hear that! I also have videos on transformers (which are the foundation of LLMs) here: kzbin.info/www/bejne/sKm0qoeBbdaor7s and kzbin.info/www/bejne/mIKYc6Klob1sd8k

[@jakemitchell6552]

Please do a series on time series forecasting with fourier components (short-time fourier transform) and how to combine multiple frame-length stft outputs into a single inversion call (wavelets?)

[@statquest]

I'll keep that in mind, but I might not be able to get to it soon.

[@omarmohamed-hc5uf]

can someone explain to me more thoroughly what is the purpose of the multiple layers with multiple LSTM cells of the encoder-decoder model for seq2seq problems because i didn't understand it too well from the video as the explanation was too vague. but still it's a great video 👍

[@statquest]

We use multiple layers and multiple LSTMs so that we can have more parameters to fit the model to the data. The more parameters we have, the more complicated a dataset we can train the model on.

[@benetramioicomas3785]

Hello! Awesome video as everything from this channel, but I have a question: how do you calculate the amount of weights and biases of both your network and the original one? If you could break down how you did it, it would be very useful! Thanks!

[@statquest]

I'm not sure I understand your question. Are you asking how the weights and biases are trained?

[@benetramioicomas3785]

No, in the video, in the minute 15:48, you say that your model has 220 weights and biases. How do you calculaamte this number?

[@statquest]

@@benetramioicomas3785 I wrote the model in PyTorch and then printed out all trainable parameters with a "for" loop that also counted the number of trainable parameters. Specifically, I wrote this loop to print out all of the weights and biases: for name, param in model.named_parameters(): print(name, param.data) To count the number of weights and biases, I used this loop: total_trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)

[@ririnch7408]

Hello, thank you for the wonderful tutorial once again. Just a question about word2vec output of embedding values, I'm a bit confused as to how we can input multiple embedding values from one word input into LSTM input. Unrolling it doesn't seem to make sense since its based on one word, if so, do we sum up all these embedding values into another layer of y=x and with weights associated them in order to get a single value for a single word input?

[@ririnch7408]

Or do we use each individual embedding value as input for different LSTM cell? (Which would mean that we can have 100-1000+ LSTM cells per word)

[@statquest]

When we have multiple inputs to a single LSTM cell, extra connections to each subunit are created with additional weights for the new inputs. So, instead of just one connection from the input to the subunit that controls how much of the long-term memory to remember, we have one connection per input to that same subunit, each with its own weight. Likewise, extra connections are added from the inputs to all of the other subunits.

[@ilirhajrullahu4083]

This channel is great. I have loved the series so far, thank you very much! I have a question: Why do we need a second layer for the encoder and decoder? Could I have achieved the same result using only 1 layer?

[@statquest]

Yes. I just wanted to show how the layers worked.

[@baocaohoang3444]

Best channel ever ❤

[@statquest]

Thank you! :)

[@chrischauhan1649]

This is what the internet is made for, world class education at home for free.

[@statquest]

Thanks!

[@datajake2742]

I'm confused why we're providing the decoder with the token as input yet waiting until the token is produced by the decoder to exit the recursive loop.

[@yasharzargari4360]

This channel is awesome. Thank you

[@statquest]

Thanks!

[@oliverlee2819]

Hey Josh, thanks for another great video! I am not quite sure how the second cell works within one layer though. Is it similar to adding another node within the same layer as in the vanilla neural network model and then the two cells output will be weighted and summed up? Or it's a different concept?

[@statquest]

The second cell works independently, and it's outputs are also independent. All they share are the same inputs. The fully connected layer at the end merges everything together.

[@oliverlee2819]

@@statquest Thanks Josh. I have another question: what is the benefit of using the LSTM for the encoder? My understanding of LSTM is that it can predict a value based a series of historical values that are related to each, such that the long/short term memories keep being refreshed. However, in seq2seq case, let's and go don't seem to be sequentially related. So why still add them as input in a sequential way (via unrolling network), rather than input these two words' embedding into two parallel networks?

[@advaithsahasranamam6170]

Great explanation, love it! PS do you have a suggestion for where I can learn to work with seq2seq with tensorflow?

[@statquest]

Unfortunately I don't. :(

[@codinghighlightswithsadra7343]

can you share the code if you find how to work with seq2seq with tensorflow Please?

[@TonnyPodiyan]

Hello Sir, I was going through your stats videos (qq plot, distribution etc)and loved your content. I would be really grateful, if you can make something regarding a worm plot. Nothing comes up on youtube when I search it.

[@statquest]

I'll keep that in mind.

[@GenesisChat]

As other people say, these lessons are gold. Le'ts say SOTA. There's a very little detail I don't understand though. Why using the words let's to go in the example, when what we want to translate let's go? It kinds of make things somewhat confusing to me...

[@statquest]

I'm not sure I understand your question. Can you clarify it?

[@mangokavun]

@@statquest The question is why did you use "Let's *to* go" instead of "Let's go" starting 4:18. Where's that "to" coming from that's fed into the network?

[@statquest]

@@mangokavun To make the example mildly interesting, I wanted to be able use at least 2 different input phrases: "let's go", which translates to "vamos", and "to go", which translates to "ir". So the input vocabulary has the tokens "lets", "to", "go", and "" so that I can create different input phrases. Likewise, the output vocabulary has "ir", "vamos", "y", and "" so that we can correctly translate the input phrases. NOTE: I included "y" (which translates to "and") in the output just to show that the transformer could learn not to use it.

[@ankushpandit7708]

What is the logic behind using multiple layers and multiple cells in each layers?

[@statquest]

The more layers and cells we use, the more weights and biases we have to fit to the model to the data.

[@nimitnag6497]

Hey Josh thank you so much for this video. It really helps in understanding the concept behind the working of encoder and decoder. I have a question though. Here we translated Let's go into Vamos which came as a result of the softmax function in the last dense layer. What if the phrase we want to translate is of less length than in Spanish. What I mean is if we HAVE 1 word in english which translates to more than 1 word in spanish. How will the softmax function give us the result then. It might be a silly question but as you say Always be Curious(ABC)

[@statquest]

You just keep unrolling the decoder to any length needed.

[@coolrohitjha2008]

Great lecture Josh!!! What is the significance of using multiple LSTM cells since we already have multiple embeddings for each word? TIA

[@statquest]

The word embeddings tell us about the individual words. The LSTM cells tell us how the words are related to each other - they capture the context.

[@Andreatuzze]

You are amazing TRIPLEBAAAAAMMMM

[@statquest]

Thanks!

[@rrrprogram8667]

Hey... Hope u r doing good..... So u are about to reach MEGA BAMMMMM

[@statquest]

Yes! I can't wait! :)

[@bobuilder4444]

Do you need the same number of lstm cells as there are embedding values?

[@statquest]

Technically no. If you have more embedding values, you can add weights to the connections to an LSTM unit and then sum those products to get the desired number of input values. If you have fewer embedding values,, you can use extra weights to expand their number.

[@ayeshashakeel]

This feels like Dora for adults and I love it 😂

[@statquest]

bam! :)

[@ayeshashakeel]

@@statquest Your videos are really helping me a lot! Could you please make a video on Gated Recurrent Units? And possibly bidirectional GRUs

[@statquest]

@@ayeshashakeel I'll keep that in mind.

[@CelinePhan]

love your songs so much

[@statquest]

Thank you! :)

[@Rumit_Pathare]

you posted this video when I needed the most Thanks man and really awesome 👍🏻

[@statquest]

HOORAY!!! BAM! :)

[@pranaymandadapu9666]

First of all, thank you so much for the clear explanation! I was confused when you said in the decoder during training that the next word we will give to the LSTM is not the predicted word, but we will use the word in training data. How will you let the network know whether the predicted token is correct?

[@statquest]

I'm working on a video on how to code and train these networks that will help make this clear. In the mean time, know that we just compare all of the predicted output values to what we know should be the output values.

[@pranaymandadapu9666]

@@statquest thank you so much!

[@yangminqi839]

Hi Josh! Your video is amazing! But I have one question: When building the Encoder, you mentioned that 2 LSTM cells and 2 LSTM layer are used, I think one LSTM layer has only 1 LSTM cell (in terms of Pytorch's nn.LSTM) if we don't unroll, isn't it? So is there two different LSTM neural networks (nn.LSTM) are used, each one has two layers, and each layer has 1 LSTM cell? Or there is just one LSTM neural network with 2 layers, and 2 LSTM cells in one layer (this means nn.LSTM can have multiple LSTM cells) ? Which one is correct? I think is the former, please correct me if I'm wrong! Many Thanks!!

[@statquest]

For nn.LSTM(), the "num_layers" parameter determines how many layers you have, and the "hidden_size" parameter controls how many cells are in each layer. Due to how the math is done, it may seem that changing "hidden_size" just makes a larger or smaller cell, but it's the equivalent of changing the number of cells. So, when I coded this, set "input_size=2", "hidden_size=2" and "num_layers=2". This is the equivalent of having 2 cells per layer and 2 layers.

[@yangminqi839]

@@statquest Thanks for your sincere reply! I think I have got your idea. You say that "hidden_size" parameter controls how many cells are in each layer, I think it's true under the situation that each cell generates a scalar output. But for Pytorch's nn.LSTMCell(input_size, output_size), only 1 nn.LSTMCell can transform the input of "input_size" to output of "output_size", which will involve some matrix multiplication not only scalar multiplication, isn't it? So even set "hidden_size=2" and "num_layers=2", I think the LSTM neural network has 2 layers and each layer have just 1 cell (nn.LSTMCell). Is my understanding right? Please correct me if I'm wrong. Thanks again!!!

[@statquest]

@@yangminqi839 nn.LSTMCell() creates the equivalent of a stack of "cells" when you set hidden size > 1. This is "equivalent" because of how the math is implemented.

[@szymonkaczmarski8477]

Great video! Finally some good explanation! I have a question regarding SOS and EOS tokens, sometimes it is mentioned that the decoder start the process of decoding by taking the SOS token, how does the whole picture differ then, for the both input sentences we always have then SOS and EOS tokens?

[@statquest]

It really doesn't change anything since the embeddings and everything are learned based on what you use. If you use EOS to start things in the decoder, then the embeddings and weights in the decoder learn that EOS is what is used at the start. If you use SOS at the start in the decoder, then the decoder and weights in the decoder learn that SOS is what is used. It really doesn't matter.

[@szymonkaczmarski8477]

@@statquest thank you! cannot wait for the transformers video!

[@SherkoAbdullahi-c3o]

Thank you, Josh. You are amazing. Would you please teach Graph Neural Networks?

[@statquest]

I'll keep that in mind.

[@weipenghu4463]

谢谢！

[@statquest]

TRIPLE BAM!!! Thank you for supporting StatQuest!!! :)

[@kadirkaandurmaz4391]

Wow. Splendid!..

[@statquest]

Thank you! :)

[@BooleanDisorder]

300 million bams! ❤

[@statquest]

Thank you!

[@WeightsByDev]

This video is very helpful... BAM!

[@statquest]

Thank you!

[@hannahnelson4569]

This is pretty cool!

[@statquest]

Thanks!

[@MariaHendrikx]

Really well explained! Thnx! :D

[@statquest]

Thank you!

[@siddharthadevanv8256]

You're videos are really amazing... ❤ Can you make a video on boltzmann machines?

[@statquest]

I'll keep that in mind.

[@xxxiu13]

Great explanation!

[@statquest]

Thanks!

[@Maxwellpaulwall]

Awesome videos! I was wondering how do people training larger models, know "im ready to press train" on the big version? Because if some of their assumptions were wrong they wasted all that time training. Is there some smaller version they can create to verify theyre getting good results, and theyre ready to train the big one?

[@statquest]

Usually you start with a smaller training dataset and see how it works first.

[@IshanGarg-y1u]

Didn't understand the part that when we are using 2 LSTM cells per layer, Since the input to these states is the same and we are training it the same way why would the weight parameters be any different. Pls correct me if I'm wrong.

[@statquest]

The parameters would be different because they started with different random initial values.

[@ishangarg2227]

Great thanks for the reply, means a lot.

[@harshilsajan4397]

Hi great video! Just a question, to give the input to lstm, the input length will be constrained by lstm length right? For example 'let's' in first one and 'go' in second one.

[@statquest]

I'm not sure what you mean by "lstm length". The idea here is that we can just copy the same sets of LMTMs as many times as we need to hand inputs of different lengths.

[@spp626]

Hello Josh! I really like your videos and explanation. Here I m with a doubt. Can we use the data of last 150 years for stock price prediction like crude oil etc in time series using garch? I have done the analysis by garch model but does it seem an over large data? Or should I use data of last 50 or 60 years only? Could you please help me out? Thank you in advance.

[@statquest]

Unfortunately I don't know much about GARCH.

[@spp626]

@@statquest oh OK..thank you for your reply! 🤗

[@zhangeluo3947]

Thank you so much sir for your clear explanation! But I have a question is that if you do word embedding for all tokens in d (let's say >2) dimensions, is that mean we can use the number of LSTM cells as d rather than just 2 cells for each layer? Or even more deep layers not just 2? Thank you!

[@zhangeluo3947]

Sorry, pardon my impatience, that's solved haha: 14:41

[@statquest]

BAM! However, it's worth noting that an LSTM can also be configured to accept multiple inputs. So you could have a single LSTM layer that takes more than a single input.

[@ereh3423]

Thank you for the content. I have three questions: 1) I've studied bentrevett github implementation and I've noticed that the size of LSTM hidden layers are 512. But the input for LSTM is 256(size of embeddings). The hidden layer output from LSTM shouldn't be 256? I understood the layers, for example, when I printed the shapes: hidden shape: torch.Size([2, 1, 512]) cell shape: torch.Size([2, 1, 512]) , I know I have size 2 because the LSTM have 2 layers. But the number 512 crash my head. 2) Cells are long short memory and hidden layers are short memory? 3) How batch size affects the model? If my batch size is 1, my sentence will be encoded in context vector and decoded in second LSTM. But if I pass 2 or more sentences, my encoder will handle it?

[@statquest]

1) I'll be able to give you more details when I create my video on how to code LSTM seq2seq models 2) Yes 3) See the answer to #1.

[@ereh3423]

@@statquest thank you very much!

[@avishkaravishkar1451]

Hi Josh. Are the 2 embeddings added up before it goes as an input to lstm?

[@statquest]

They are multiplied by individual weights then summed and then a bias is added. The weights and bias are trained with backpropagation.

[@Luxcium]

Oups 🙊 What is « *Seq2Seq* » I must go watch *Long Short Term-Memory* I think I will have to check out the quest also *Word Embedding and Word2Vec…* and then I will be happy to come back to learn with Josh 😅 I am impatient to learn *Attention for Neural Networks* _Clearly Explained_

[@bfc7649]

Love your vids

[@statquest]

Thanks!

[@sukhsehajkaur1731]

I loved the video. However, I have a few questions. The paper says "Finally, we found that reversing the order of the words in all source sentences (but not target sentences) improved the LSTM’s performancemarkedly". But in this video you have reversed the order of the target sequence, Right? Also, how can the outputs from the first LSTM layer in the encoder be directly connected to the first LSTM layer of the decoder if we have a stack of two LSTM layers in the encoder part?

[@statquest]

In this video I didn't reverse anything - it might look like the decoder is doing things in reverse because the decoder is initialized with the token, but if you look at the outputs, you'll see that the output is not reversed. And, in this example, both the encoder and the decoder have the same number of LSTMs and stacks of LSTMs, so things can be connected directly. If we have a different number, we can use a simple "fully connected layer" to change the number of outputs from the encoder LSTMs to match the inputs to the decoder LSTMs.

[@sukhsehajkaur1731]

@@statquest Thank you so much for replying. I would really appreciate it if you could confirm if the context vector of this simplified model uses 8 numbers? because we have 2 layers with 2 LSTM cells and each cell would have 2 final states (a final long-term memory (c) and a final short-term memory (h))? also, is the final state of first LSTM in the first layer of encoder used to initialize c and h of the first LSTM in the first layer of the decoder? I mean each LSTM cell transfers its c and h to the corresponding cell in the decoder?

[@statquest]

@@sukhsehajkaur1731 Yes. And, if you look at the illustration in the video, you'll see 8 lines going from the encoder to the decoder. And yes. This is also illustrated in the video.

[@sukhsehajkaur1731]

@@statquest Thanks a lot for the confirmation. Kudos to your hard work!

[@prashlovessamosa]

Damm again awesome stuff.

[@statquest]

Thanks!