LSTM Time Series Forecasting Tutorial in Python

[@GregHogg]

I offer 1 on 1 tutoring for Data Structures & Algos, and Analytics / ML! Book a free consultation here: calendly.com/greghogg/30min

[@kobiyobi629]

hey link doesn't work. i copied the code exactly and not working please help

[@YIYANG-p3d]

you are my god

[@PandemicGameplay]

Buddy this is the best explanation I have seen on this topic. You are an excellent communicator, if you're not a professor or instructor you should be.

[@GregHogg]

Glad to hear it!

[@mandem2735]

great video! Thanks for sharing. One comment, hard coding `i+5` in the `df_to_X_f` function will lead to some very unexpected results if the `window_size` is not set to 5. Better would be to use the `window_size` variable here to ensure the slice is always the same as the window. Cheers :)

[@GregHogg]

Oops, good catch yes I definitely meant window size, thanks!

[@freddiesimmons1394]

@@GregHogg I'm a complete newbie trying to complete his capstone project in a bootcamp. Knowing that you could make that kind of mistake that even I saw really is going to help me with my imposter syndrome moving forward. Also, 11/10 video

[@muhammadyousifjamali3491]

​@@GregHogg You have used train set instead of test set during prediction

[@kirwakelvinkering3122]

There was not even a single reason why I should leave without subscribing , leaving this comment and liking the video , what an explanation , what a model 🤗

[@GregHogg]

Very nice of you to say! Thank you greatly for the support :)

[@marlhex6280]

Wow the amount of flexibility on the communication is huge, great skill and of course the LSTM skills and python are perfect, can’t say enough thanks Greg.

[@GregHogg]

Glad to hear it!!

[@nirmagor5390]

Beautiful tutorial. I would kindly like to get your attention towards some bad practice when using the Numpy skills. You can use np.fromfunction( lambda x, y: x+y, shape=(n_windows, window_size) ) to create a matrix of indices and then slicing the df_as_np with the indices matrix to create X. Then you can use np.reshape to convert it to (n_windows, window_size, 1) shape. That avoids any explicit iterations (for loops) and allows numpy's backend to perform parallelism if possible. Overall very good and clear tutorial!

[@ramanamachireddy]

great video. It was explained in a very simple terms and precise. I watched few others videos from university professors but this one stands out. Great job Greg.

[@rohanjoseph1531]

Good explanation overall for the concept. However, some quick suggestions to make the videos a bit more comprehensive is to explain the key details like what is a tensor and why did you design a 3-D matrix for the model, or the part where you flatten the predictions; instead of just saying that the output will have extra dimensions run the code and exhibit it and then apply flatten. Given this, really like your work to help people like me understand Machine Learning from the very basics. 😊

[@GregHogg]

I really appreciate feedback like this. Very specific on important ideas like the tensor. I will keep this in mind for sure, and I appreciate your kind words!

[@TheMilkManHasSomeBeans]

Greg, this has been a great help. Even after 1 year, this remains relevant and super useful. Love it ✅

[@GregHogg]

Glad to hear it! :)

[@TheMilkManHasSomeBeans]

You know @@GregHogg I saw this and right now I'm trying to apply this in different scenarios. I'm failing to get it right and I am hoping to reach you. Do you have an instagram? LinkedIn?

[@ricgondo]

Thanks!

[@GregHogg]

You're super super nice and I really appreciate it 😊

[@ricgondo]

@@GregHogg Not at all sir! Great content!

[@GregHogg]

@@ricgondo Thanks so much.

[@kevinigweh8076]

Your video is amazing, man. Thank you so much. I was having problems understanding time series forecasting but you just made everything so clear and easy to understand. Again, thank you😁😁

[@teklehaimanotaman3150]

Thank you Greg, you cleared my fear of deep learning.

[@hamzaehsankhan]

ValueError: The filepath provided must end in .keras (Keras model format). Received: filepath=model1/ Solved this by providing model name with extension .keras

[@kyawhan3690]

For absolute beginners like myself, this is how that line looks like. cp = ModelCheckpoint('model1.keras', save_best_only=True)

[@cristobalgarces1675]

@@kyawhan3690 This was an annoying issue I was trying to fix for like 20 minutes. Thank you!

[@ericdudgeon1648]

Probably the best video on youtube for LSTM. Question which I cant find anywhere, if I want to apply this and show what my next forecast would be, would I just use model.predict() and pass in the last 5 intervals to get my next hour prediction? I cannot find a good example anywhere of someone actually using the model to forecast.

[@GregHogg]

That's very nice of you to say, Eric! Yeah, that's what you would do.

[@capsizabidin31]

Thank you Greg, perfect explanation and presentation!

[@adithyaprasadpandelu8032]

Thanks Greg, for the very informative content on LSTM. It's been great help!

[@GregHogg]

You're very welcome Adithya!

[@khalidrafiq8080]

Thank you so much for the great lecture. Just a quick question, when we use the test data finally, we are not using the rolling prediction(basically the output predicted by the lstm is appended as the last input and then this sequence is the new input and so on ...). Basically in the absence of the true test data itself. Thank you.

[@drperfectfeast]

Thank you Greg very much for the course! I have one question: if I want to forecast the temperature at 06:00 based on not one X(parameter) but 10 Xs (parameters) from 00:00-05:00(sliding window of 5) how should I tweak my df_to_X_y function?

[@vskraiml2032]

excellent video and covered all my doubts, thanks Greg...

[@moondevonyt]

really awesome video, learned a lot about LSTMs. thank you Greg!

[@GregHogg]

Great to hear!!

[@moondevonyt]

@@GregHogg do you have anything to predict future stock price? :)

[@GregHogg]

@@moondevonyt Yes check my recent Videos :)

[@moondevonyt]

@@GregHogg youre the man, thanks!

[@natalianunez1817]

Hi! I have a question with respect to the 20:07 part where you define the 'linear' function, in case I'm using a variable that doesn't take negatives like the price of an action which one would you recommend? Thank you :)

[@peachythegreat8945]

Wow...the greatest tutorial ever...

[@SamuelPhilip-bv3ij]

Idk why but I got an error "No file or directory found at model1/" when trying to save the best model

[@davidenzler1691]

This video was really helpful with implementing an LSTM in tensorflow! A lot of sources talk about it as either theoretical or building a toy one from scratch. Nice to actually see a tensorflow workflow used. When you are dealing with n-featured examples in the time series, how would you set up the model layers? For example: lets say you used barometric pressure in addition to temperature so your training matrix now looks like : [[ [x11, x12], [x21, x22], ... [xn1, xn2] ]] where x1 is the first entry in the series, x11 is the first feature of the first entry, and n is the window size. How would you set up the model layers? you would be dealing with a 3D matrix, where the third dimension is another feature matrix of window size n. Would the initial input layer just be a tuple of (5, 2)? edit: I just realized you have a whole other video on this so I will watch that lol

[@aliosman0]

This video helped me greatly. Amazing explanation. Thank you.

[@GregHogg]

Super glad to hear it :)

[@ahmedismailbinamrai1080]

This is amazing, thank you for this great video.

[@蔡远航-v8q]

Great video! I have a question, do you have to do normalization in time series forecasting, I see it was not presented in the video.Thanks！

[@rodrigorezende2332]

Hello, and thanks for the fantastic video. If I may ask, what would you recommend if I look for a specific kind of time series forecasting in which I don't want to give the model the real values x_test? I am pretending to use the model's prediction to create an x_test_model and then use the model on this set to achieve a y_test_model that I can compare with the real y_test.

[@TomVI1316]

Nice tutorial! I'm struggling to conceptually understand what happens when you call predict. I understand the training portion, but how do I get values for say 30 days out? It seems I need to feed it 30 values that I already know the answer for? What is it actually returning when you call predict?! For example, if I have a dataset of 100 values with a look back/ window of 5. I want to predict values 100 to 105. I train the model on 95 values and call model.predict on the remaining 5, do I get the predictions for 95-100 or 100-105?

[@ddddddddee]

Great VDO mate!!! Keep it up!!

[@GregHogg]

Thanks so much!

[@andrewdornack3373]

Great video. Do you know where I can find more information on how to determine the number of layers when performing the model.add() commands. For example, I believe you used 16 for your LSTM.

[@manuelnovella39]

Thanks for the video! Exactly what I needed

[@GregHogg]

Really glad to hear that!

[@valdompinga]

Hello, amazing video, may I ask something? So lets say we were to predict a single value, we don't need to use the time somehow or not? Does the model kinda learns to which time the data sequence is relationed? like: "this values n sequence... must be winter so the prediction is: x!" right???

[@ButchCassidyAndSundanceKid]

Don't you need to scale the data using MinMaxScale or StandardScale between 0 and 1 ? And then once you've done gone thru' the LSTM training, you scale the data back to original ?

[@InamKhan-kg8wx]

Very well explained! Thanks for sharing

[@mikekertser5384]

Thank you. What shall be modified in the code in case the X is not a single value (like temperature), but many features vector and y is a single value?

[@GregHogg]

I will cover this in the future. But essentially, we'd probably want the single items in each list (the third dimension) to be more values. Then you'd have to change the input shape of the model accordingly

[@muhammedcihadakkaya7140]

Hello Greg, first of all, thank you very much for the very lovely and exciting video. I do realize that in the test part, you are not updating the window with the new predictions. So mainly, you are doing predictions for every five timesteps, but then in the window, you are using the actual test values to predict the next value. So my question is did you try in this way as well, and if so, how was your results? Thanks again!

[@SAM-t5s]

I have the same question!!

[@TheGibberingGoblin]

Thanks a bunch! Do you have any advice on how to create confidence intervals for predictions?

[@SeamusHarper1234]

I'd like to know that too =)

[@choaibchafik3937]

Thanks for the video, I'm wondering how to use the first model to predict future values.

[@kavinyudhitia]

Great tutorial, thanks!!🎉

[@yousif_alyousifi]

Many thanks for this video. Can we use this method for forecasting univariate time series that involve outliers? If so, how to treat the outliers first?

[@elshadpiroghlanov7149]

Hi Greg, Does your model predict different numbers each time you run it? Could you make it give range instead of one number? Kind of Monte Carlo simulation.

[@yarasd2749]

thank you for the very good video ...but isnt working with np Arrays exhust the computer memory during fitting ? can we find another way to prepare the data before training... i am working on a multivariate multistep LSTM model ... i have 90 features to predict ine target value ... with many zeros in the target and also in the input features... it is very hallanging specially for multi step forcasting... any advice!?

[@TheySudi]

hello i'm a beginner of machine learning is it important that the index on the dataframe must use a timestamp before being put into the train model? is there any effect if the index still uses auto increment? sorry for bad english

[@GregHogg]

It doesn't do anything once it reaches the NumPy stage. It just allows the use of pandas functions to manipulate the dataframe in pandas

[@TheySudi]

@@GregHogg Thank you so much for your attention I have 1 more question how to determine a good number of hidden layers and good activation function ? I have tried the lstm model with different hidden layer configurations and activation functions several times with my data, but the training results are always not good

[@robmeyer7985]

Greg, Thx for great video. I'm a complete noobie, just starting to learn Python. But I'm confused. I've seen a number of videos with output plots for RNNs where, like this one, the prediction line seems to mirror the actual line, often AFTER (to the right of) the actual. Of course, I could take a stubby pencil and look at the thermometer outside my window and write the temp down and say it was my prediction five minutes ago. Voila! One line of code. But the prediction line should be BEFORE the actual line, no? What am I missing? Am I just not seeing it due to the scale on your graph? Rob

[@MeredithSargent]

Hi there! Found this really helpful, but I've managed to confuse myself about the data windowing and the test set. Since every target in y_test (except the last one) is somewhere in X_test, how are we not assuming our solution? I had a look at your stock price notebook and there's the bit about recursive predictions, but you don't have that here. Any clarification?

[@majamuster2470]

Great video! Question: I don’t understand the difference between validation data and test data. You say that test data has not seen the model before, but so did the validation set? I thought we only divide into train and test data

[@miguelpereira9095]

Hi, great video! Is it possible to run the data in a sliding window way and update the model as it runs? Thank you

[@arcsaber1127]

instead of creating a window function, you can just use the pandas "shift" function to create columns that are shifted by 1 each, using a for loop

[@andrewdornack3373]

This also seems only helpful if you want to predict a singular value in the future. What if you wanted a very large number of predictions?

[@SlfgjkAldfjgf]

Amazing explanation. Next show how to apply attention mechanism/transformers to time series.

[@afonsolenzi7590]

Hello congrats amazing video!! What should we do differently to consider more variables as X?

[@chaoukimachreki6422]

Bro you sound like Robert Greene and you are just as awesome as he is or even more awesome !

[@Wissam-rk7tv]

great, please how should we prepare our data if we want to predict the temperature in different cities, (redundant dates)??

[@AnandPrakash-lc9nr]

hi Greg, what if my target variable is not normally distributed but positively skewed? what should I do in such case? and also, is it recommended to datetime function as cos and sin of hour, weeks n all to provide it in the x variable?

[@alperyldrm4788]

Great video! Thanks for the great content!

[@mp3311]

Great! What should I do to improve the accuracy on my trained LSTM model?

[@GregHogg]

Dropout, regularization, adjust the model in any way you see fit!

[@mateoconcha]

Hi! Excelent video! How you can apply the model with new data, i mean when you have new variables without the temperature values. You would like to predict the new (future) temperature values? I hope you can help me with this. Thanks!

[@AGnanaprakasam]

Can I use the R2 value to measure the accuracy of the LSTM model in time series prediction?

[@JoshuaBradshaw-j2e]

If I wanted to forecast 18-24 months into the future, do you have a video on this for a supply chain context? Thanks so much Greg, this video was really well put together for understanding the LSTM Time Series. Also, do you have a video showing how you started off with a baseline model (i.e., Linear Regression or Seasonal Moving Average), then iterated to better models with regression metrics shown?

[@GregHogg]

Thank you! I probably do, although I don't remember sorry

[@denishaarmstrong8383]

Hi Gregg , when running the predictions :train_results = pd.DataFrame(data={'Train Predictions':train_predictions, 'Actuals':y_train1}) i have noticed that you've used data but no prior reference could you state why?

[@hemantnyadav]

This video at 13.26 shows a time series function, but it is single-step forecasting only... can you show how to modify it for multistep. I have tried to add multiple labels in the loop but it is not working with LSTM. Thanks, Greg for this video also.

[@SeamusHarper1234]

Would this approach work if you wanted to predict the next say 24 or 96 values, instead of just the next 1?

[@alemazzuca]

When I try to predict a temperature, the imput should be the prior 5 temp records, right? But the model accept only one value in order to give a prediction, how is that possible?

[@ahmedelsayedabdelnabyrefae1365]

Hi Greg, that was a lovely and informative presentation. but you are using the training data again in prediction. so why did we split the data into testing, training, and validation?

[@skyinca812]

Great Video Gregg!

[@GregHogg]

Thanks Akash!

[@LL-wx1yn]

Nice Video ! Just think about mention François Chollet and Keras for their tutorial

[@VijaySuryaVempati]

Can you giv e an example of multivariate forecasting?Also if i have temperature datas in two different files and i want to train the LSTM on both the datasets, how will i include that ? Example: first dataset is say temperature on antartica (-10 to 5 degree celsius) and other file has temperature of south africa(30-50 degree celsius). I cant just simply append both data frames. So do i need to train it twice on both datasets?

[@TheSerbes]

Why doesn't it show the same performance when I put P bar instead of temperature? The temperature values are small and have negative values, so I don't understand, did you make an arrangement?

[@SSJwalker]

Why do you use rmse over r2 score? I would like to know thanks!

[@osmansafi4517]

Do you consider the prediction of first predicted hour in second input to do further prediction?

[@Aman-yu4re]

i have a data set of about 4500 rows with daily prices and its dates, I am always getting a loss of around 1.2, and after that it keeps oscillating around this number . How do i resolve this issue ?

[@zerihunchere1036]

Amazing! How can i use this for raster data? Thank you!

[@ebenemmanuel9482]

Thank you so much. It definitely was much needed.😇😇

[@GregHogg]

You're very welcome and happy it helped!

[@JustAdrianYT_MTB]

in df_to_X_y() shouldn't row= [[a] for a in df_as_np[i:i+window_size]] and not i+5? I'm asking because i am new to python and i don't fully understand it yet but if feel to generalize this funtion it should be i+window_size and not i+5

[@JustAdrianYT_MTB]

Oh, sorry, ignore this. Found the answer in @mandem2735 comment from a year ago. Many thanks for the video. works great for temperature dataset. Struggling with stock prices but getting there...

[@smvnt3803]

Great video, really helped a lot!

[@GregHogg]

Glad to hear it :)

[@Algardraug]

The df_to_X_y function doesn't actually work for the df, just for the temp array. Confusing naming

[@fernandobarajas1665]

awesome tutorial bro!!!,any thanks for this video, to confirm, could you say that the 5 data above predict 1 value?

[@alekseyvladimirovich398]

very well explained video, but in terms of the results you show, it looks like your predictions are lagging behind the actual values. Meaning the model doesn't generalize well enough and relay in its prediction on the last known value. It's seen there the trend of the blue line(prediction) always one step behind the trend of the orange line(actual values).

[@mohamedabouelkhir1221]

Great explanation, it really helped me , but i have one question, how can we forecast on the future dates? Knowing that we dont have the X_future?

[@GregHogg]

Thank you! We can recursively feed our output back in as input to the model

[@gajshello2004]

@@GregHoggHi, can you give me a hint on how to do this? Please help...

[@giridharjadala2182]

wow ,you're a GOD!!

[@GregHogg]

Thank you haha

[@diegomq5169]

How is it possible that my model performs well when training and predicting with a dataset with many examples, but when I use only one example (to make the prediction) it performs really bad? Is it a problem of how the data is structured or do I have to make adjustments to the model?

[@luquinhassvc]

Amazing! Congrats for great explanation, my friend. I have a question. If I wanted to forecast future temperature regarding date in the future, how could I do that? Regards, Lucas from Brazil.

[@jackpickford4896]

I have tried creating a similar forecasting model but have to normalise the data, whereas you did not in your code. Is there another reason I'm not understanding that allowed your model to work with the real values?

[@kenhan168]

awesome, great explanation!

[@GregHogg]

Thanks Ken, I really appreciate that!

[@JesusRomero-zw8yw]

Excellent video Greg, I had two questions: 1. Why do you indicate a window size of 5, could I have chosen another value, maybe 10 or 20? 2. If after running the LSTM model I get a (loss: Nan) and the (MAE: Nan), what could I be doing wrong? Thanks for your help. Regards

[@GregHogg]

Thank you! 1. Pick any window you want. 2. Not sure unfortunately!

[@alperenkoza9624]

Hello Jesus. Your dataset might have some Nan or 0 values. Please check it.

[@JesusRomero-zw8yw]

@@alperenkoza9624 thanks Aperen Exactly I had those values ​​in my dataset

[@MarkkuPilarinen]

Thank you. Great video

[@vexlimits1506]

how would the df_to_X_y function change if instead of only having a date and temperature column you also have a location column because now X would have the format (n_samples, window_size, 2)

[@laharib7795]

Which data you as been taken in this video can you please show us or update a dataset your

[@deepfxd8]

does the lstm train only on 1 input ? i see this ((70086, 5, 1), for X.shape

[@sonthai3118]

Great video Greg. Thank you for sharing. I have a question. Is it better to perform data normalization/standardization before we start the process? Thank you

[@GregHogg]

Thank you. As long as you've preprocessed inputs in some form, you probably can't go too wrong

[@dewman7236]

I have a question. How do I line up the Dates with the corresponding actual values and predicted values? It seems like the lengths don't matchup

[@MikroAaltoUuni7]

Thank you so much, I followed the tutorial and got my first LSTM network working. I am however having trouble with generating out of sample predictions, I feel like I have tried everything and just can’t crack it. I am using the last batch of the test set to predict the first out of sample data point, but it ends in an error. I feel like it’s a dimension thing. Any help, anyone?

[@hnclienteshn4501]

fantastic, thank you very much for sharing

[@GregHogg]

You're very welcome!

[@TheTedyoo]

Thank you so much, it's very helpful

[@GregHogg]

You're very welcome and I'm super glad to hear that :)

[@kartik_exe_]

Love those voice cracks ❤

[@GregHogg]

Pfffft idk what you're talking about :/

[@Zerpyderp]

which direction are you feeding the data in the lstm?

[@vikramm4967]

Will removing the trend and seasonality from the data improve the accuracy?

[@GregHogg]

Not sure, go ahead and try it!

[@JuanManuelBerros]

Great stuff Greg! Thanks to this vid I finally got the input dimensions of the LSTM right. One quick question: why is the extra `Dense(8, "relu")` layer necessary, and not just the final Dense(1)?

[@GregHogg]

Thank you, and great to hear! I forget the final model but most likely this is for extra complexity

[@RedRose-ll4tb]

@@GregHogg Hey so, is it okay to put fully connected layer after LSTM Layer? Thank you

[@iramparvez-zp6qq]

Hi! i used the same code but shape of X,y is not right. they give(1,20,1),(1,20) they are not giving correct n_samples, total number of samples are 365557, and after minus 20 as window size it coulde be 365537.where is the problem