Time Series Data Preparation for Deep Learning (LSTM, RNN) models

[@himanshuverma8648]

Most of the people out there use a function to prepare the data for LSTM but here with the help of this generator it becomes really easy. Thank You.

[@story_teller_1987]

Most of the time series videos I have seen are done by so called experts , but themselves don’t know what they are teaching. Here at AI Engineering Srinivasan is doing a really great job. He knows to explain it to us by covering each and every corner of the topic. “If you can't explain it simply, you don't understand it well enough.” Albert Einstein

[@AIEngineeringLife]

Thanks for such a wonderful note 🙏

[@Induraj11]

If I need to learn some concept, I know "AI engineering" is the first place I search for. Thanks for the time and efforts you and your team put sir.

[@AIEngineeringLife]

Thanks for this note Induraj :)

[@arashchitgar7445]

Finally.... someone explained this stuff clearly. Thank you sooo much!!! 🙏🙏

[@tejask5417]

Unlike everyother idiot out there who predict single input LSTM's, you did a fantastic job in explaining how to create a samples for multiple inputs and multiple outputs. Just subscribed to your video, thank you!

[@waqitshatasheel5875]

This is very detailed explanation of windows, shifting and labeling in time series. I have came across Tensoflow Documentation, but I didn't understand at all. Thanks for sharing.

[@shwetabhat9981]

Great content sir . Hope you are doing well, kindly continue sharing your knowledge on ML DL as it greatly helps 🙂

[@alliwant8383]

Great video my friend!

[@moustafa_shomer]

Another great video, really great job man, hope your videos reach more people

[@renemiche735]

Great thanks, perfect explanation.

[@carrocesta]

very well explained, thank you sir!

[@leonandorfi5191]

Thank you for the video, by far the best explanation I could find. Just one question, why do we add the column we are trying to predict in the data parameter of TimeseriesGenerator if we already have it in the target? I guess I'm having trouble understanding what those parameters are, but isn't data the set of features we want to use to predict the target?

[@AIEngineeringLife]

Leon.. In time series future value is typically a functions of past lag of predictor. This is account for trend, seasonality and others. Reason in time series analysis we use lags of target. In traditional classification we typically do not use target as the output is just function of input feature alone

[@thebiggerpicture__]

@@AIEngineeringLife Sorry, I have the same question but didn't get the answer. Could you please develop? Thanks! great video.

[@MaximGehlmann]

Why do you give the appliance power also in the feature? Doesn't the LSTM then learn that the label is purely a copy of the appliance column?

[@treasuremshololo7938]

Welldone Sir.

[@ajithshenoy5566]

you're amazing . Thanks a lot

[@AIEngineeringLife]

👍

[@edwardchamberlain4910]

Excellent.

[@isururajapaksha6700]

great explanation bro

[@slash7954]

Your video is very good. When is the next video we can watch? I want you to use an autoencoder. Thanks a lot!

[@AIEngineeringLife]

Thank you.. LSTM should be coming in less than a week and later followed by autoencoders

[@darshanayenkar5833]

one more time great thanks

[@vijaychakole5929]

nice explanation ...thank you..it will be more beneficial for beginners like us if you can share a notebook for learning purposes for us...thanks again...

[@AIEngineeringLife]

It is all here Vijay - github.com/srivatsan88/End-to-End-Time-Series

[@vijaychakole5929]

@@AIEngineeringLife thank you so much, sir...

[@nibinjoshy4218]

which version of tensorflow are you are using

[@soumitramehrotra5547]

Hi Srivatsan, thank you for the video. I have one question regarding Time Series using LSTM. I am working on a project where I have 3000 users, and corresponding to each user I have Time Series data. One naive thing would be to train a model for each user independently, but is there any other way I can train my LSTM to address this case? I would appreciate any input you can provide. Thanks

[@AIEngineeringLife]

Best way is to train individual model for each time series but if you see the distribution is not different then you can check my multi time series DeepAR video where you can train a global model as well as individual models on each time series

[@AveRegina_]

I'm using RNN for my PG thesis work. I've a query. Do we have to run stationarity test for our time series data before feeding it in the neural network model... or this step is only required in traditional time series models like ARIMA?

[@elispot17]

Same question

[@AMVSAGOs]

Hi... This is really a great content, at 20.28 min you mentioned features. I think they are samples. because we have None values in the last 2 rows of target , we have to skip their corresponding input samples. That is why we are considering all the samples except last 2. could you please check and confirm whether I am right or not ?.

[@anangsuwasto7660]

Thanks a lot for the video. I have a question. if we have daily data and want to predict the next 30 days so we shift the multi_target by -30?

[@AIEngineeringLife]

Anand.. Nope.. shift is how long of data you want to use to predict the future. So if length is 30 it will take past 30 days of data to predict 1 future. If you want future 30 days then it is like looping day by day with current prediction to get into future. Have you seen my LSTM time series video. I cover in that

[@anangsuwasto7660]

@@AIEngineeringLife Thank you. So we also need to predict the other features ex: T_out, TH_1, Visibility. since we don't have it in the next days after the first prediction right?

[@anirbanmukherjee7547]

Normally, a model takes six consecutive time steps as input and predicts on 7th time step...now, I want the model to take six consecutive timesteps from t=0 to t=5 as input and t=12 as label, instead of t=7...what will be the arguments of TimeSeriesGenerator( ) in this case? Is it possible?

[@AIEngineeringLife]

Anirban.. To my knowledge not possible out of the box. You can write a custom function and achieve it or I would say create a multi step output as shown in my example above that will predict all t-6 to t-12 and then pick t-12 as output

[@anirbanmukherjee7547]

@@AIEngineeringLife Thank you sir...

[@vashistnarayansingh5995]

What should be the value of stride if i want a tumbling window. Is it equal to window size ?

[@abidabdulazeez3930]

Yes

[@mp3311]

Hi. Can you help me improve the accuracy of an Lstm model?

[@qiguosun129]

Thank you for sharing this data preprocessing technique. I have a question: what if I am dealing with a CNN-LSTM case? In other words, my input features are in t*n*H*W shape. Can I still use this method? Hope for your answer, thanks!

[@AIEngineeringLife]

I doubt this can be used. This is mostly for sequence structured data and might not work for image data where we have H and W. I maybe wrong as well in this case

[@qiguosun129]

@@AIEngineeringLife Thanks for your reply, I found that using Einsum may be better for my specific problem.

[@dilankawijesena5382]

Sir could you please tell how to input data to this model for getting more future predictions please help me

[@aninsignificantman001]

Thanks for the video . It was very helpful . Still I have a doubt and a request . doubt :: When using lstm model at the time of prediction, to generate output of the second and subsequent timesteps , we will also need the other features at respective timesteps . So are you missing something in the video or is it my understanding that is wrong . Request :: when working with data of cryptocurrency price prediction , the range of values for price is very high . Using an ordinary scaler does not work and I suspect using log transform will induce a lot of error . Is there a way that can help . Can you make a video on it as it is on a related topic with a different set of problem . Thanks in Advance .

[@rafainfernal]

You rock!!

[@riswandaayu5930]

how to predict per hour with lstm ?

[@pyclassy]

hello sir can we get any tutorial on Conv-LSTM with complete explanation?

[@edgaraskryzevicius9369]

Your one of the features is appliance and your target is also appliance?

[@AIEngineeringLife]

This is time series model and in this case future forecasts are dependent on past forecast in many cases by way of trend and seasonality. In time series typically target and feature might be similar but it takes lag of that variable rather learning one to one mapping

[@elizabethmj6506]

Sir, Could pls explain how to treat time series classification if I have a feature with multiple values? The structure of the data is also given below label: 1 cl_data:10,10,2,12,12,12,12,12,1,1,1,1,1,3,3,3,3,3,1,1,1,1,1,1,1,1,1,1,3,3,1,1,1,1,1,3,3,3,3,3,3,1,1,1,1,1,1,3,3,3,1,1,1,1,1,1,1,1,1,1,3,3,3,3,3,3,1,1,1,1,1,1,1,4,4,4,4,2,2,1,1,1,1,1,1,2,1,1,1,3,3,3,3,1,1,1,1,10,10 label: 0 cl_Data:3,3,3,6,2,1,1,1,1,2,1,8,8,8,8,8,8,1,1,1,1,1,1,4,2,1,1,1,1,1,7,7,7,7,7,28,28,28,28,28,2,1,2,1,1,1,1,1,3,3,3,1,1,1,1,1,4,4,2,1,2,1,1,1,1,3,3,3,3,1,1,1,2,1,1,1,1,1

[@matts.4937]

how can we get the code for this? its very helpful

[@AIEngineeringLife]

github.com/srivatsan88

[@mohammadrahmaty521]

Thank you!

[@gurjeet333]

Can u pls provide the link of the notebook

[@khanduom9341]

Sir, Would you mind to share some suggestion how to predict the following using LSTM please... 0, 1 and 2 are labels for the size of the network traffic. I want to use frame 1 and frame 2 to predict frame 0 This is the sequence of the frame 1 0 0 0 2 0 0 0 2 0 0 0 1....... Thank You

[@AIEngineeringLife]

Did you try it as time series classification problem?.. basically create LSTM and then last layer define for multi class classification. Passing a sequence it must predict if it is 0,1 or 2

[@khanduom9341]

Sir, thank you so much for the valuable insight. I will try to frame it as a time series classification problem and will see. I have one question regarding time series prediction in the above case. For example, If i take 13 previous observation, the model can definitely predict the next point observation i.e 14th index as mentioned in your 1st Use Case. As far as the 2nd Use Case is concerned, Is it possible to use 13 previous observation to predict the data with ONLY label 0 ......OR do i need to take 13 previous observation and set offset as 13. But then it will predict the whole 13 sequence including 0, 1 and 2 data labels and looks like a multiple input and multiple output prediction problem. Therefore, I will appreciate your advice whether it is not possible to use data labels 1 and 2 to predict data label 0 as a time series prediction problem. If not, the only option is to use multi class time series classification. Thank You

[@darshanayenkar5833]

THANK YOU SIR

[@statsnow3354]

can i have the notebook file?

[@AIEngineeringLife]

It is here - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Time_Series_Functions_for_Sequencing.ipynb

[@Wanderlust1342]

Thankyou

[@ashwin_raikar]

Here is the dataset link: archive.ics.uci.edu/ml/machine-learning-databases/00374/

[@UllasKC]

Please Share the code

[@brucewillis8780]

This is a ripoff from the Official Tensorflow Tutorials on Time series forecating.

[@AIEngineeringLife]

No it is not.. The images are taken from TF website and not the code. If you see the same code in TF website then paste the link here. If you see my notebook on this the first 3 lines are images i download from TF website - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Time_Series_Functions_for_Sequencing.ipynb Moreover it is official TF documentation and examples are given to use or teach. In work do we not use documentation?