181 - Multivariate time series forecasting using LSTM

[@DigitalSreeni]

I should apologize for misspeaking towards the end when it comes to forecasting. I said you can forecast the future 90 days when you do "model.predict(trainX[-n_future])". This is not true; you'd be predicting the past and not the future. This is a multivariate time series which means you need multiple variables as input to predict a single output. The input size shall be (n, 14, 5) where 14 was the number of days we look back and 5 was the number of variables. We only have multiple variables available until the last day in our input. So we can only predict one day into the future. In a single variate time series, you can predict multiple days into the future as you can use your predicted values as input to the model. You cannot do that here as we are not predicting multiple variables. Sorry again for making a rookie mistake. The code on github will be updated accordingly. Thanks to all who alerted me about the mistake.

[@kirannbhavaraju5978]

I have a question to ask... You trained the model on windows of 14 days so (14 * 5) was each window and the window was slid several times predicting one "open" price in the future (w.r.t the window). The model described only knows how to process 14 days on inputs.. i.e. (14*5) and give out 1 output. Given this context can you please explain why you are passing the last 90 days to the model when forecasting to predict one value? I would imagine that training and testing/forecasting instance shapes should be the same while training and inferring. I am unable to grasp the concept at the end. :)

[@dutchy5752]

yes exactly, you need predictions of all features to predict the next bar. But I would think in theory this would be possible. So not only predict the Open, but also the other 5 features. I did not do this yet myself but it should be possible, right?

[@sebastianandrescajasordone8501]

Why isn't it splited into test and train data as in the case of single variate time series? Wouldn't it be the same exercise of predicting a 1 day?

[@LordUche1]

Thanks for the great video! My question is how can I do multi-time-step predictions with multivariate time series using LSTM? E.g, How can I predict 10 days into the future with the LSTM model

[@augustoandre5668]

Looks like the code wasn't update yet, right?

[@Wysumay]

I am french and I understand this video better than any papers or videos that are actually in my own language. Thanks for this amazing explanation !

[@ivoriankoua3916]

J'espère que tes etudes avancent bien ou que tu es déja en fonction

[@lemeuneugeuceu4626]

@@ivoriankoua3916 @Wysumay j'ai besoin d'aide j'essaie de faire la m^me chose mais j'ia toujours une sortie exponentielle

[@chenqu773]

Figuring out the data input is indeed a crutial part of the whole training pipeline. Thank you very much for such a clear explanation and a great tutorial !

[@joelteixeira3557]

I have been struggling with the input-shape for multivariate time series for such a long time. And finally I found a great explanation for how to prepare my data. Thanks for this.

[@gustavopergola2019]

Wow... you are a FANTASTIC teacher! I wouldn't worry too much about when you go into detail for some stuff, it really helped everytime you did. Thank you very much!

[@raphaelguay9342]

You explained it really well. I was stuck in the datashape things. Now it's crystal clear. Thanks

[@iamharshvashisht]

same for me

[@dhruvshettty]

Thanks a lot for this video. Super helpful to understand how the conversion of timeseries data to a supervised ML problem takes place

[@DigitalSreeni]

Thank you very much.

[@salvinprasad8592]

Sir, I am PhD student - I have seen many videos but I love to see your video. When I was doing image processing part, I learnt the most from your videos. Now I am learning multi-step forecast horizon. Again, I am learning the data reshaping part from you. I have not met you but apart from my supervisors and co-supervisors you are one of my guru. Thank you so much sir.

[@cristhianfernandez5883]

Remember in this example, the first column is the predictable variable. Thank you very much for the knowledge you share, greetings from Colombia.

[@DigitalSreeni]

Thank you :)

[@creativeclash6989]

how to predict last column?

[@arhamtaseer1837]

This has been a huge help for a new project i am working on. I still need to optimize a lot of inputs to the model and implement feature importance methods but this basic framework has been an excellent start!

[@TheHarpanOnly]

I just started a project similar to this, and every single second in this video is an answer of of my questions during this project. Thank you and GBU.

[@DigitalSreeni]

Great to hear!

[@adityabik9669]

I am an indian. Thank you very much for such a clear explanation and a great tutorial !

[@DigitalSreeni]

Glad it was helpful!

[@l0ltaha]

Your explanations are absolutely spot on! Please don't think "dwelling" on a topic is a waste; it really helps with understanding! I would have preferred if you could also explain (or trace) the somewhat head scratching logic in those for loops for trainX and trainY but after starring at it for quite some time I was able to understand. Nonetheless, 11/10!! Thank you for these videos.

[@DigitalSreeni]

I am glad you find these videos useful. Keep watching :)

[@l0ltaha]

@@DigitalSreeni I have one question Mr. Sreeni. I was looking for your video #164b becuase in your GitHub I found something I was very interested in, which is the same LSTM forecasting but using the Timeseriesgenerator(). If you can link me the video or any resources that helped you I would be very grateful. Thank you.

[@myMathBanks]

Thank you for taking the time to explain the input shape! You explained it in a way that helps us develop intuition.

[@HaleemaRajpoot-i1b]

This video helped me alot to understand the basic concepts although I need to learn more advance optimization techniques..Thank you so much Sir!

[@nazaramin8635]

Many thanks, this is the first time I understand multivariate time series forecasting using lstm. You cannt imagine how much I need it, appreciated..

[@AzzozAlHasani]

Thank you very much for the trick about reshape predicted data to inverse scale. Lot of videos I watched before just plot the scaled data with predicted data without de-scaling, I was wondering how actually use the model instead of comparing performance. this video is very useful to me.

[@rudela9900]

Hi Sreeni, In this lesson you covered the most important concepts in time series forecasting I can think of. I would very much appreciate if you would expand some of the ideas you mentioned, such as including volume in the forecast, decouple the forecast from pd.to_datetime (use time of day or index as well), further coding ideas, etc. Once again, this is by far the most useful, applicable and compelling explanation I've attended. Thanks a lot.

[@DigitalSreeni]

Noted

[@davidperry8634]

The video was very clear and easy to follow, certainly in comparison to your peers. I like how you zoomed in to focus on the important part that you were currently talking about.

[@pravdomirdobrev4850]

As a person who spent hours on the inversion broadcasting error; I feel very appreciative of the 1 minute spent explaining why the shape of scaling is important.

[@DigitalSreeni]

Thanks :)

[@reidgwn7508]

Fantastic content! The part that you believed you dwelled on was actually the best.

[@DigitalSreeni]

Thank you :)

[@cineart5918]

The BEST explanation as I saw!!

[@VINCENTUMEH-p3j]

You're simply the best. This is torn into pieces

[@qugen_ai]

Very nice explanation Sir, I always get confused with shape of the data which you explained very well. I had one question, when we predict future values, lets suppose daily prediction then the value of future day 1 should be one input for future day 2 and so on. Please let me know if I am wrong. If my thinking is correct then how this prediction is working. If possible please explain.

[@aliardestani1225]

I really appreciate the method of your teaching, I've learned a lot from you.

[@DigitalSreeni]

Glad to hear that!

[@81rashmi]

Thank you so much for great explanation! I have couple of questions: 1. If I have a monthly data and I would like to forecast next 5 years, then what would be the best size for n_past. Would it be 60 (5 years) or just 12 (1 year). How do we gauge this size. 2. How do I compare validation loss with training loss and what is the good range to consider the validation loss is considerable. Any help in understanding these 2 questions would be appreciated. Thanks!

[@emiliovasquez594]

The scaling explanation was worth it, great video.

[@nguyenhaituan9838]

Thank you for very clear tutorial, I deeply appreciated your support.

[@DigitalSreeni]

Glad it was helpful!

[@vijilakshmi2575]

The concept I am looking for was clearly explained in this video. Thank you so much for sharing your knowledge. even small small things you explain in your videos even though you had already explained in previous videos. thanks a lot.

[@DigitalSreeni]

You're most welcome

[@NguyenHuynh-zm7tn]

Xin cảm ơn tác giả. Một bài hướng dẫn rất hay.

[@DigitalSreeni]

You are welcome.

[@GeorgeSserwadda]

Thank you for the detailed explanation on multivariable input

[@HamidSi-n4d]

Plainly explained. Enjoyed. Thank you

[@vaibhavshankhwar7016]

Thank You , Sir Thank You It helps a lot in my hachathon problem, to get optimized solution.

[@Dynamics_Structures]

Thank you, @DigitalSreeni. Please, can forecast LSTM be applied to degradation modelling of concrete, particularly? If yes kindly share a video to help, please.

[@gemechuabdisa1763]

Thank you very much. I found this tutorial very interesting. Can you provide a tutorial on how to forecast multiple responses (like VAR)? May be integrating VAR-LSTM.

[@chymoney1]

this is done by simply adding more output nodes.

[@abdessalamyacine]

Thank you for your video it's excellent, but I want to know why you used n_futur=90 when the model is trained to predict one day (1 days) in advance? THANKS

[@prakashd842]

Hi Sir , i am big fan of you . Learnt lot of stuff from your videos . My sincere thanks for preparing such a awesome practical content .

[@DigitalSreeni]

You are most welcome

[@russwedemeyer5602]

Thanks for this great video. It has been extremely helpful. At the end of the video you asked for ideas for future videos that would help people get through a difficult subject. I have been struggling with implementing Feature Importance for Multivariate Time Series prediction using LSTM. I believe this is very valuable for better understanding the predicting model and helping give it credibility. Please consider for a future video. Thanks again!

[@farshidk4127]

Hello, thank you for your amazing video. can we not only predict the Open, but also the other 5 features by this code? considering there are correlations between these 5 feature. Thanks in advance for your guidance.

[@corneliumaftuleac891]

Hi @digitalSreeni, While the overall video is very good and well explained, there are several critical flaws in the way how you split the training and validation data. I think this part is covered very poorly. Frist flaw is that you perform scaling on training and validation data which is wrong, in the real world you wont have validation data, and the real-time results when trying to forecast will be out of the scaler range. Therefore you need to split first, the only fit_transform on training data, then only transform alone on the validation data. Another flaw is the way you do the split itself. You rely on keras to do this automaticaly using validation_split paramater, but you dont take into account that actually some of the values (exactly 14 values) at the end of the training set actualy takes values from the future which are not avaliable when doing real-time forecasting. Therefore you need to drop the last 14 sequences from the end of the training set. Otherwise training on history and in real life will be very, very differant with desastrous consequences. Would be really great if you can make those corrections and apply an updated version. Otherwise great video!

[@danyalkhan2264]

Hi Sreeni, Thank you so much for the fantastic video on multivariate forecasting using LSTM. It is indeed a really informative lecture. I am currently working on a multivariate forecasting problem using LSTM and am wondering if there is any possibility of using difference sequence sizes for different input features. The number of input features in this case are 7. Your guidance will be appreciated. Thanks in advance!

[@damonlin3495]

Thank you so much for sharing such a wonderful video! I have a question...In this case, each observation in the dataset has included the past 14 days' stock data, does the sequential time order between the observations still matter when training the classification model? Can random shuffling be used when splitting training and testing data?

[@ubaobasi8766]

I have a question the piece of code model.predict(trainX[-n_future]) are we predicting the last 90 days in the train data or are we predicting the next 90 days using the last 90 training data? please respond thanks

[@skibielec]

I'm confused too. It's seems to me that in shown scenario, the neural network was predicting values for the past 90 with target to predict the next day. Eg for day -90, price for day -89 had been predicted. Later on the predictions were shifted on the graph + 90 days so it looks like it's predicting furute. I think i am missing something.

[@skibielec]

I think there is good explenation how to predict into the furure. kzbin.info/www/bejne/fmfHppKmm9qtpKc

[@wisdomogala9426]

Hi Uba, I support you on this. The model has seen the future 90 days already, so technically, it is predicting nothing. It is just replicating what it has seen already. I think the best way is to replace -90 with -1 and define an iteration that appends the output of the network into the first position of the sequence and eliminate the last one, a kind of a moving windows

[@DigitalSreeni]

I made a mistake when mentioning about predicting future values. Please remember that this is a multivariate time series, which means you need multiple variables as input to predict a single output. But, to predict future values we can only predict 1 day as we only have all the required input data until the last day. In a normal single variate time series, you can use the predicted value as input to predict the next day and then the next day, and so on. But here, we cannot do that as we are not predicting multiple variables. We are only using multiple variables as input. Sorry that misspoke in the video, should have paid more attention. In summary, when you select trainX[-n_future], you will just be predicting the past and not the future. This is fine as we need to verify the accuracy of our model. But, we can only predict one day.

[@user-nu5ww1el4q]

@@DigitalSreeni so what is the code to predict the next day? Thanks you.

[@CodeCult413]

After searching many places, finally I got it. Very well explained by you sir. Please make a video on Object localization of medical x-ray of chest and brain.

[@teklehaimanotaman3150]

Thank you very much for your excellent tutorials. While getting your forecast sequence, you used i+n_future-1:i+n_future,0. This is not so far clear for me. I would expect to be df_training_scaled[i:i+n_future,0]. Because in the X part, you take the sequence up to i but not included. Thank you for your response.

[@ptcita16]

Thanks for that amazing video. Help me a lot with my current work.

[@simmonds788]

Thank you, has been very useful and clearly explained.

[@Jason-wx1tv]

Thank you so much for taking the time to put this together. This has been very helpful.

[@DigitalSreeni]

You are welcome.

[@alvinjamur1]

very good educational. however, serious methodological errors - must split data and apply scaling to only train set. then apply fitted to test set.

[@soumyabrata111]

After doodling for past two weeks, finally got some relevant info. Thank you so much Sir

[@faez8220]

thank you for this tutorial, great and clear explanation!

[@kentsmith6918]

Great explanation - very thorough & clear! Thank you!

[@DigitalSreeni]

Glad it was helpful!

[@chiwenliao2237]

This video really helps me a lot! Thank you so much!🎉🎉

[@ramdhanandtara]

Thank you for the explanation, really help me for complete my thesis

[@avcstore9600]

great! awesome for your explain about LSTM , thank you sir

[@tanmoybhattacharya2761]

hi @DigitalSreeni for i in range(n_past, len(df_for_training_scaled) - n_future +1): trainX.append(df_for_training_scaled[i - n_past:i, 0:df_for_training.shape[1]]) trainY.append(df_for_training_scaled[i + n_future - 1:i + n_future, 0]) in these above lines where have you mentioned the predicted variable? 0 means your predicted variable?? can you please help me on that?

[@folashadeolaitan6222]

I would appreciate a response to this question as well. Where have you mentioned you are predicting the "Open", i would like to change mine to the close and see the out put. And how can we also predict multiple variables so we can then predict more than 3-5days into the future.

[@JonCianci12]

This was excellent! Great work with explaining the input shapes! Your predictions look pretty good also… and we got through Covid… 👍

[@rashaaw278]

Hi, Thank you for this amazing video. but please, I have a question, where you identify that you want to predict "open" feature ???

[@nehavasudev3549]

Hi Sreeni amazing videos on time series. I learned a lot about data prep from your video. Do you have any tutorials on Time series Multivariate with categorical variables?

[@robbert-janspernaweiland7786]

@DigitalSreeni Your neural network is now predicting the n+1 day open.. but you are feeding it the n day close for every single prediction. Wouldn’t the open for day n+1 almost always practically match the close on day n?

[@kickthecheater1479]

Hi, great tutorial! I did not understand the forecast part, maybe you could clarify: n_future=90 forecast_period_dates = pd.date_range(list(train_dates)[-1], periods=n_future, freq='1d').tolist() forecast = model.predict(trainX[-n_future:]) #forecast As I understand this takes : model.predict(trainX[-90:]) or last 90 entries in train_x and makes prediction for next one day as it was trained. I dont understand how can it be 90days to future with new dates and not just 89days of train_y and one day predicted. Also I have seen in other tutorials it usually : loop: predict next day, append train_x, predict next Is this some kind of keras feature?

[@AlexAuTung]

same question here, thanks

[@pfannenstill]

answer is he didnt predict the future. The only way to predict further in the future with this model is iteratively, ie. first predict one day in the future, then make a new sample with the last 29 days + the future day you just predicted, and predict the next day again, and so on. Obviously, small errors would quickly grow doing this so it's kind of pointless with financial data.

[@jenishmonpara]

Just to make draw the attention that your channel deserves, if you add ...LSTM ( Stock Price Prediction ) in title, people will come rushing to this video :P

[@DisguisedAsMohit]

You made it. As usual, awesome !!

[@DigitalSreeni]

Yes! Thank you!

[@الحمدللهعلىكلحال-م7ص]

Few times I comment. But I do when really deserved. This is exactly what I am looking for ;easy and simple on any level .Thank you so much I have one question: do we use forward walking validation or random validation trsting

[@joshuadawson5850]

Is there a correct way for building LSTM models for time series forecasting? Is there a sweet spot on features? Currently I have 138 features, getting a RMSE of 9 on a regression problem. Using Hyperband i have been unable to get a better RMSE then the base model. And I am wondering if LSTM features matter like Random Forest, where around 30 features is the sweet spot. Any help or guidence is greatly appreciated.

[@roxennh]

Good day, Sir.! I'm here to express my gratitude. This video has really saved my life. I can't express how grateful I am! I wish you the best of luck in the future.

[@Wissam-rk7tv]

awesomeee, please, how to prepare our data, in the case of a multivariate analysis but with redundant dates, for example the prediction of temperature in several regions ? (so we don't have a unique key)

[@jolittevillaruz5234]

Thank you very much. You are a very good teacher.

[@DigitalSreeni]

Glad you think so!

[@arundattagupta2833]

Hi , I hope you are doing well. I am new to Deep Learning. This video was very helpful to understand the LSTM implementation.

[@larrylot758]

Thank you very much, I recently got into LSTM to try to predict river levels based on the historical time series data but somehow I struggle when it comes to selecting and putting the data into the right format. like if I want to predict two days ahead, what is my best n_past? I have daily records for the past 40 years of the same river.

[@sudhanshusoni1524]

you got the solution ?

[@lawrencenehemiah5945]

@@sudhanshusoni1524 not yet

[@sudhanshusoni1524]

@@lawrencenehemiah5945 i beleive Google has already developed one algorithm of similar nature. Predicting the flood near rivers and which areas they will affect.

[@lawrencenehemiah5945]

@@sudhanshusoni1524 yes I have seen it before, but the details of implementation are not public. Thank you anyway

[@jaxlone997]

I'm glad I found your channel, that's exactly what I was looking for. It's important to spend time on the details, and the video was perfect!

[@DigitalSreeni]

Awesome, thank you!

[@fatihaelagri7753]

Thank you so much for your effort. I want to know if this method and the code are the same if I have 5 variables as inputs and one output.

[@avinashchavan3113]

@DigitalSreeni Hi Sreeni, Thank you for the video. I have my output as a Binary classification . Can you please help me what to do in that case for input data into a shape for the Train Y ?

[@jacknaneek1681]

Great video. Great explanation. Very clear and very well done!

[@DigitalSreeni]

Thank you!

[@dongjunlee5945]

Thank you for your vedio. That is so helpful. I have a question about forcast. If I assume that if we have just one Y value at initial time step and have other variables for forcast period, can I use the predicted values as my new input?

[@chiennguyen-zz1du]

thank you for video of you. i have a question that. when the vector input very large (about 20-30 para) and out put only 1 para. I dont know select the model ( how many layer LSTM, hidden of layer ... ) you can suggest about the problem? and when I run model of this video the result is diffirent between each of time. thanks.

[@disasterhunter3813]

sns.lineplot(original['Date'], original['Open']) sns.lineplot(df_forecast['Date'], df_forecast['Open']) Exception has occurred: TypeError lineplot() takes from 0 to 1 positional arguments but 2 were given I tried to run above codes but pop out the error, how do I fix it?

[@escandalomusical]

I got the same error. Did you find a solution???

[@edu1113]

nice.. do you have any video on using LSTM to gap fill missing data.. my interest is in ecohydological data such as water level logged every 30 mins.. how can we fill the missing data from a monitoring station for a long periods due to equipment malfunction.. assuming I have similar data from neighbouring monitoring stations

[@abinashr751]

I am creating a State of health estimation model for a Li-ion battery using LSTM RNN model. I am able to predict the values for test data using the trained model with train data. My doubt is now what is the use here like i am giving actual SOH , voltage and current as input to predict the SOH and i evaluate that prediction to see how good the accuracy is ,but what if i want to predict the future values of SOH with help of previous known data and we dont have any current input voltage or current.. or what if i want to find the SOH when voltage and current alone is given as input. In my code am using timeseriesgenerator for datastructuring if i use manual python code for datastructuring the loss and validation error is too high.please help me

[@w157-p5x]

I think what confused me the most is that 'number_of_samples' is the first and not the third index of the input shape.

[@enriquecatala]

Very well explained! thank you so much for sharing your knowledge!

[@DigitalSreeni]

Glad it was helpful!

[@zhihuauk5796]

Thanks so much Sreeni. It helps me a lot !

[@ahmadk5844]

first thank you for the video, im new to machine learning and im a little bit confused, why is the target label (Open column) included in the input data (trainX) while training the model? should we not split the data into features (input) and target labels?

[@skiraf]

There appears to be an error in column#5. The adjusted-close price should equal the close price, unless there is a dividend or stock-split. So the magnituded of the value after the scale operation should be approximately the same. In your table, the adjusted close is only a fraction of the actual close. This is not correct.

[@33d6]

the 5th second of this video should go viral

[@HiltonFernandes]

Great explanation and great technical skills. Congratulations !

[@HiltonFernandes]

Could you please be as kind as to share the code that you used in a git repository or a gist ?

[@DigitalSreeni]

Please look at the description under the video for the link to my github repo.

[@HiltonFernandes]

Therefore, your video is complete, professor @@DigitalSreeni !

[@HiltonFernandes]

I've seen so many videos that don't share their code that I forgot to find your sharing information in the video description.

[@HiltonFernandes]

Thanks a lot for that one too, kind sir.

[@danielesantiago]

Amazing content, thank you. It helped me a lot.

[@lossmoss]

thanks for the nice video. You asked for requests. Can you explain Multivariate ARIMA-LSTM please.?

[@juanete69]

In this example would it be useful to use .batch(), .prefetch() or cache()?

[@chiwenliao2237]

Thank you! more model demonstration for financial forecasts plz!

[@DigitalSreeni]

Thank you very much. Noted your point.

[@Absfor30]

I love the "I'm done with Covid" tone... again great content! Keep it up!

[@AKSHETTY1982]

hi , i am wondering why you keep the column that you want to predict as one of the input. shouldnt we remove the column being predicted out of trainX , as we are providing that column in trainY ?

[@dbgm12]

Hi Dr Sreeni "Open" is our dependent variable. Hence, we can only use 4 independent variables to predict "Open". Pls correct me if I am wrong. Thks

[@shashankkulshrestha2346]

HI I am working on automotive sensor modeling in exhaust after system , and here i have data in form of different small trips so please tell me how to train this LSTM multivariate model with no. of separate series data.

[@mryellowboots]

Thanks for the video. This is what I looking for

[@davidw.9711]

great could you make a video on tumor infiltrating lymphocytes and deep learning; here is one recent paper regarding this topic.

[@mrjaekwan73]

simple question for 21:30, when you visualised the forecast to the original data. I understood the methodology, and the video itself was very clear. two thing though! 1. I did not see the comparison between the the original data and the forecasted data from the graph. Also what is the accuracy measure of that forecast? 2. you showed the decline of the training error in the video. I would like to hear your opinion on the overfitting issue for the model. thanks

[@DigitalSreeni]

How can you measure accuracy of a forecast unless you live through the future and compare forecast against it. This is why it is important to hold out the last few data points from the original data to be used for validation. Also, please do not try to forecast too far into the future. Please remember that these approaches are more of modeling rather than forecasting. Overfitting is an issue which is why it is important to holdout about 20% of last data points for validation.

[@deaw3612]

Cool, this video is easy to understand lstm things. And I should using spyder tool for some project it look better than notebook to monitor the parameters

[@PrasannaRoutray97]

Nice tutorial. What if I want to predict for the past 14 days and not the future? Input is 14 sample 5 column and output is 14 sample 2 column?

[@chandrasekaranramachandran8766]

Thanks for your great explanation Sir! In this example, you forecast open feature. If I want to forecast open and close* (two variables), what are all the changes I have to make ?

[@rashaaw278]

are there any answer for this question?