End to End Time Series Modeling using Auto ARIMA

[@mkeremyucedag]

Wow great explanation man. Nobody explains time series as clear as this tutorial. Very good work!

[@deeptigoyal4342]

Apart from the algo and concept explanation. I love the visualizations and their explanations too. :) keep posting. It is really helpful. I have been doing time series analysis from past 2 years but now it makes more sense to me.. Love the visualizations

[@AIEngineeringLife]

Thanks Deepti 👍

[@dataflex4440]

Sir you saved my Job

[@jongcheulkim7284]

Thank you so much. This is very helpful.

[@jagdishjazzy]

Arent we creating here the model for the whole data and then doing test and train split. Shouldnt we first split and then create the model? Please clarify this.

[@AIEngineeringLife]

Jagdish.. Model is created on train only here - model.fit(train['demand']) But i used entire dataset to find the order of ARIMA model. Now this can be train as well. I assume train and test are same distribution and from same overall population. Top part just highlights the parameter and not really fits a model. Fit is done below

[@pragyasingh8887]

Hi, Thanks a lot for the nice explanation. I am using the same pm version, dataset and code, and still cannot reproduce the forecast plots. The values in the confidence interval is insanely high. Do you have any idea what could lead to such an issue? Thanking you in advance.

[@muhammedimdaad]

will you do a session for multivariate time series forecasting using SARIMA? If not what will be the changes in the above program to use it for multivariate time series? thank you. enjoying the playlist.

[@AIEngineeringLife]

I will be covering VAR models for multivariate. It must be live before this week. Thank you

[@muhammedimdaad]

@@AIEngineeringLife thank you. waiting for it. we always deal with IoT sensor data. your tutorials are the best learning resources. keep up the work. thank you

[@navomichacko2589]

Thanks for the clear explanation of ARIMA models. But why we check the Stationarity in ARIMA when we use a combination of EMD-ARIMA model??EMD will already give the IMFs that are stationary right??

[@ansylpinto2301]

Nice Video

[@anthonykangethe3040]

Thank you very much for this educative content. I request you to do a video on SARIMAX with multiple seasonality

[@aishwaryanarkar2954]

hello thank you :) do you also have multivariate auto arima ?

[@amanarora7591]

Sir I have a doubt here- When we are creating the 'model' object (code block where we are importing pmdarima), we are telling the model that our target variable is demand. Isn't this part exposing the whole data to the model? Though, later, we are splitting the dataset into train and test and using model.fit just on train dataset but have we not already exposed the entire data while initializing the model object? Please help!

[@chanchalshukla683]

Hi Sir, I have a doubt. Please have a look. You have taken data Monthly instead of daily for prediction like below: d3_monthly=d3.resample('M').mean() and while creating Auto Arima object, pass parameter m=12, as monthly data is our input data, But What if we want prediction on daily data?

[@AIEngineeringLife]

First Auto Arima is very slow if you are going to run on lot of data so better to aggregate it. As per documentation m=7 is for daily 7 - daily 12 - monthly 52 - weekly Better go for prophet in case if you have lot of data

[@mithzeesh4806]

Hi Sir, Would have been really helpful if there was in-depth explaination (or seperate video) on AR, ARMA , PACF, ACF like you have done for MA and then ARIMA.

[@AIEngineeringLife]

Chaitanya.. I am going to cover it as I have mentioned in this video.. I will cover ARIMA in detail. I will come back to it once I cover prophet. At that time I will be covering ACF/PACF and other statistical techniques

[@mithzeesh4806]

Thank you very much.. This series will soon be Best Time Series Content on KZbin.. Thank you so much..

[@ARJUN-op2dh]

Where can we find this jupyter notebook?

[@kapilkhond6339]

When you are running Auto ARIMA, for (0, 0, 1)(0, 0, 1, 12) system threw: "Near non-invertible roots for order; setting score to inf (at least one inverse root too close to the border of the unit circle: 0.998)" warning. What does it mean, will it affect our results?

[@AIEngineeringLife]

That is fine as Auto Arima rums for different combination of p,d,q parameter. Wherever you see above prints those model will not be selected and ignored by Auto ARIMA

[@samiullahkhan6594]

How can solve that problem..find MMS MMS forcast for the ARIMA (3,0,0)

[@rajarajeshwaripremkumar2842]

Thank you so much for you videos and help sir, continuing on below comment, I cloned you repo and ran the Time_Series_Auto_ARIMA notebook on my laptop , it throws this error the first time I run "MissingDataError: exog contains inf or nans" in cell 19 i.e. where we run the auto_arima, second time it runs fine. Can you please advise on this?

[@kapilkhond6339]

Is it necessary to remove seasonality from time-series data before applying models like SARIMA, ARIMA?

[@AIEngineeringLife]

Not required for SARIMA.. But if the time series is not stationary it is good to remove some variance in the data

[@neelakashchatterjee1725]

Is it possible to train multi-variate model using auto arima ?

[@TheNishi42]

when we are using auto arima, calculating ACF and PACF not necessary ?

[@aditinagar6688]

The video was very helpful! Sir, I am getting AIC around 143695 which is pretty high and also the colab is crashing due to the full consumption of RAM. Could you please guide me regarding the AIC problem and any other way to fix it or the possible mistakes that might have happened. Thank you.

[@AIEngineeringLife]

Aditi, Without knowledge of model it is difficult to comment but AIC should be just used to compare models and high AIC might not mean anything. On RAM if your data is huge try rolling up data say from sec to minutes or hours so it can easily fit in RAM

[@aditinagar6688]

Thank you for the response Sir. Sir I tried rolling up my DateTime column to 'D' but all the results are in a single range (flat). Also, is there any other way to use auto_arima without rolling up the data as I've to predict the hourly transactions. Kindly shed some light on this problem. Thank you!

[@sriadityab4794]

Can auto ARIMA do all the required data transformation techniques to make the data stationary before feeding to the model?

[@AIEngineeringLife]

It can set the d parameter of ARIMA which is nothing but difference to make it stationary. Said that other analysis has to be outside of it

[@sriadityab4794]

@@AIEngineeringLife Thanks. Suppose if we directly apply Auto ARIMA to the raw data and the final output obtained from it is final one irrespective of the model performance?

[@AIEngineeringLife]

The way to use it is to run auto ARIMA to get p d and q parameter and then iterate it manually to get better model

[@sriadityab4794]

@@AIEngineeringLife Thanks 😊

[@sriadityab4794]

@@AIEngineeringLife Are there any specific techniques for feature selection to build a time series model or can we use general use case FS techniques like correlation, RFE, Extra Tree Regressor, Wrapper, RF feature importance scores, etc?

[@kunalpandya8468]

great video. you forgot to evaluate model using RMSE, MSE OR MAPE. can you help me with that?

[@rajarajeshwaripremkumar2842]

Have you used python auto_arima in production? it gives error the first time i run it saying "MissingDataError: exog contains inf or nans ", but is okay once i rerun , my results are really good with arima, would be really helpful if someone could advise on how to avoid/fix this issue..

[@AIEngineeringLife]

Auto arima using statsmodel package which expects data to be filled in for all sequences. Now this is surprising it runs fine again on same data. Frankly I am not sure. Maybe something to do with dataframe initialization that model is not able to pick it up first time. Difficult to tell without seeing the code

[@abhishekarundash1798]

I am running an Auto-ARIMA model, but of the output of the model is a straight line i.e. a constant forecast is output from the model for the whole period, @AIEngineering can you suggest how to handle that.

[@AIEngineeringLife]

Can you elaborate please. If it is constant forecast then why ARIMA?. Did you check simple techniques like moving averages . Is there any trend in the data?

[@abhishekarundash1798]

@@AIEngineeringLife the data is stationary, and has no trend, the input has variations, but the prediction has a single output

[@truptikachare6906]

Sir, do you know how to fit this model into tableau using TabPy? What changes do we need to make in the code?

[@AIEngineeringLife]

I have not tried it but 95% of code should work and 5% mostly in restructuring it so we can accept and return value from tableau. Just the integration part has to be structured

[@truptikachare6906]

@@AIEngineeringLife Sir can you please check your mail?

[@AIEngineeringLife]

@@truptikachare6906 I did not get any email. Better to reach me on LinkedIN message

[@jaspreet0305]

GitHub Repo for the code files: github.com/srivatsan88/End-to-End-Time-Series

[@soumyabrata111]

Thanks Brother

[@anonymously809]

When I run the last forecast plot, it throws error: tzinfo argument must be None or of a tzinfo subclass, not type 'unitdata'. Could you pls help?

[@maneeshmarine]

Hi Srivatsan Sir, Can we train auto_arima time series model by passing train data in batches. I'm able to use complete data to train auto_arima model due to memory error. Would you please suggest the best way to use auto_arima model for huge data. Thanks in Advance.

[@AIEngineeringLife]

Not to my knowledge. If your data is huge see if you can resample data to higher granularity. Like is data is in millisecond then make it to second or minute. Batching I doubt unless you move to LSTM. Try prophet maybe and see

[@maneeshmarine]

@@AIEngineeringLife Thanks Sir...I will try Prophet

[@jiyasworld5839]

Why we do not apply any hetero, normality and other test on arima model?

[@AIEngineeringLife]

Jiya.. This was just to show Auto Arima functionality. I have still not started ARIMA in my time series videos. If you have seen my time series playlist I have cover stationary test and will be covering remaining in days to come

[@jiyasworld5839]

If u have any idea plz explain it little bit. I am confused why we dnt apply any other test on arima series and what is the meaning of theory free arima series. Tomorrow my defense i applied arima but not clear about these question

[@AIEngineeringLife]

The video you saw is part of my playlist where I have covered and explained some of the test for testing stationary in data or trend in data that are key requirement in selecting the type of ARIMA model and p,d,q parameter . the playlist is here - kzbin.info/aero/PL3N9eeOlCrP5cK0QRQxeJd6GrQvhAtpBK What I am yet to cover is concepts such as de-trending, ACF/PACF and then build ARIMA model after validating all assumptions. Sorry if I confused you here

[@jiyasworld5839]

Thnk u for this kindness sir

[@nehasrivastava9684]

Hello sir..I am final year student of M.tech in Data Science. I needed some guidance regarding dissertation topic. I request you to kindly advise.

[@AIEngineeringLife]

Neha.. I am not sure of topics for dissertation but if u have to discuss on this and it is related t time series please message me on linkedin and I can discuss

[@ainafaqihah8328]

HI Sir. How to find the MAPE, RMSE? I have tried but this error has come out ValueError: Found input variables with inconsistent numbers of samples. Can anyone help

[@sahishnutaat2803]

Sir, I am getting LinAlgError: LU decomposition error. in model.fit(train['demand']) line..Could you tell me what could be the reason and how to fix it?

[@AIEngineeringLife]

Is it different dataset than that used in video?.. Have you checked to see if you have any missing info in data?. Here is my notebook and see if any step is missing - github.com/srivatsan88/End-to-End-Time-Series/blob/master/Time_Series_Auto_ARIMA.ipynb

[@sahishnutaat2803]

@@AIEngineeringLife There were missing data ...I imputed it by using forward fill..😊..I took the same link for data

[@AIEngineeringLife]

@@sahishnutaat2803 Sorry but is it working or you still have the issue after ffill?

[@bharatthakur3642]

Sir while applying sarimax on my data. Its giving all prediction as null. How to deal

[@AIEngineeringLife]

Bharat.. Might be some issue with how data is being passed during inference.. I need to see the code to see what issue it can be

[@satraj201]

what about RMSE and MAPE

[@diyyalasravs9508]

How to get the trace result from auto arima, I need to get second best arima order

[@AIEngineeringLife]

I need to check it out as it has been long I tried it out. There must be a way where trace can be routed to dictionary or something similar and then can be ordered by metrics. Are you not able to find any?

[@diyyalasravs9508]

@@AIEngineeringLife I tried when I converting to dictionary it is considering the parameters of best model, I need to get solution, can u please help me. Thank you.

[@AIEngineeringLife]

@@diyyalasravs9508 .. If you have dictionary object you can rank order by AIC or your metric. It is kind of leaderboard and you can pick p,d,q parameter for second and then apply it on arima model using statsmodel package to get output. I will see if I find sometime over weekend to show how to do it

[@diyyalasravs9508]

@@AIEngineeringLife ok thank you, I will be waiting for solution.

[@diyyalasravs9508]

Did u find any way for getting trace?

[@ramanjaneyuluthanniru1428]

Hi sir, Have you uploaded GitHub file on this time series video's

[@AIEngineeringLife]

Not yet Ram.. Will do it in couple of weeks when I get to clean the notebook and upload it

[@Cricketpracticevideoarchive]

you can access the code for this series on github.com/priyadarshi0007/TimeSeries-AIEngineering until sir uploads his notebooks

[@valerysalov8208]

@@Cricketpracticevideoarchive That AQI project is your's or Krish Naik's cause he has done a video on the same project.

[@Cricketpracticevideoarchive]

ifjo239r023fjj I was in the members when he started it and I have done the project side by side, so what are you trying to say

[@Cricketpracticevideoarchive]

ifjo239r023fjj no no, it’s not Like that, he is a good guy bro, I learned a lot in this project, no one stole no ones project, you can drop a message linkedin.com/in/pskumar89 we can cha there, we should not crowd the comment section

[@simple_sci]

kzbin.info/www/bejne/an7On5hrdtKfnbs