Detrending and deseasonalizing data with fourier series

This is Part 3 of a multi-part series on Pricing Weather Derivatives. In this video we take Daily Average Temperature (DAT) series from Sydney Observatory Hill starting from 1-Jan 1859 and attempt to de-trend and remove seasonal variation using fourier series. In timeseries this is also known as time series decomposition, where the terms are detrending and deseasonalizing data.
The denoised temperature time series reveals that temperatures have somewhat uniform peaks. This implies that we could use a first order fourier series model to estimate the seasonal variation in Daily Average Temperature.
For parameter estimation of the first order fourier series model, here we use xscipy.optimize.curve_fit which implements the Levenberg-Marquardt algorithm (LMA). This is a method of non-linear least squares and combines both the Gauss-Newton algorithm (GNA) and gradient descent methods.
Online written tutorial: quantpy.com.au/weather-deriva...
In this series we take a deep dive into a type of exotic financial products weather derivatives. Weather derivatives are financial instruments that can be used to reduce risk associated with adverse weather conditions like temperature, rainfall, frost, snow, and wind speeds.
Historical Data, Weather Observations for Sydney, Australia - Observatory Hill: www.bom.gov.au/climate/data/st...
★ ★ Code Available on GitHub ★ ★
GitHub: github.com/TheQuantPy
Specific Tutorial Link: github.com/TheQuantPy/youtube...
★ ★ QuantPy GitHub ★ ★
Collection of resources used on QuantPy KZbin channel. github.com/thequantpy
★ ★ Discord Community ★ ★
Join a small niche community of like-minded quants on discord. / discord
★ ★ Support our Patreon Community ★ ★
Get access to Jupyter Notebooks that can run in the browser without downloading python.
/ quantpy
★ ★ ThetaData API ★ ★
ThetaData's API provides both realtime and historical options data for end-of-day, and intraday trades and quotes. Use coupon 'QPY1' to receive 20% off on your first month.
www.thetadata.net/
★ ★ Online Quant Tutorials ★ ★
WEBSITE: quantpy.com.au
★ ★ Contact Us ★ ★
EMAIL: [e-mail]
Disclaimer: All ideas, opinions, recommendations and/or forecasts, expressed or implied in this content, are for informational and educational purposes only and should not be construed as financial product advice or an inducement or instruction to invest, trade, and/or speculate in the markets. Any action or refraining from action; investments, trades, and/or speculations made in light of the ideas, opinions, and/or forecasts, expressed or implied in this content, are committed at your own risk an consequence, financial or otherwise. As an affiliate of ThetaData, QuantPy Pty Ltd is compensated for any purchases made through the link provided in this description.

[@mohammedboussardi7718]

I like watching even if i do not understand the concept yet

[@binli307]

great content!

[@laukittom9789]

it is amazing, please do more analysis

[@alexgregory8453]

Great stuff! How would you approach the same problem with hourly data with diurnal and annual seasonality?

[@denisplotnikov6875]

Need videos about how to start (maybe step-by-step plan) the journey of quantitative finance until get a position of quantitative trader. Could you create such videos?

[@rawsaucerobert]

Hi there, I tried to apply this to my own weather data, and while the model_fit and model_fit_general fit perfectly to the seasonality, I needed to phase shift the "model" function by 180 degrees to get the correct curve. I would like to attribute it to my data being in the northern hemisphere but why would a sine wave care about the seasons? Do you have any thoughts on this? TLDR: Had to phase shift the "model" function by 180 degrees to get a proper fit on my data, everything else worked as expected.

[@graymars1097]

I’m half way through, given that I’m familiar with the math: this is awesome. Thanks for uploading

[@husseinnasser4428]

Great stuff! Can you please make a video on Sticky Delta/Sticky Strike and how that relates to fixed strike vols and Implied/Realized Skew on the vol surface?

[@QuantPy]

Yes, will work on making this after weather derivative series

[@husseinnasser4428]

@@QuantPy Awesome! Can't wait to watch

[@husseinnasser4428]

@@QuantPy Idea: when you make this video you should add in how to back out the implied spot/vol covariance from the strike prices. (e.g. a $1 move down in spot implies vol increasing by 5pts) Peter Carr's paper (2020) on PnL attribution has a way to do this, you can also do this by creating a portfolio where all Greeks are neutral and except Vanna and Theta, solve using linear algebra.

[@gambu4810]

Wow, please show your analysis as well. If you can take us through these steps in R-Studio or Python it would really solidify our knowledge.

[@QuantPy]

No worries, link to the code is listed in the description. Enjoy

[@stretch8390]

Adjusting omega for speed of processes is where I'm getting stuck at the moment!

[@arinpacuretu4202]

Hello! Can you please tell me which source have you used for the simplified Fourier series? I can't find them in the bio, thanks!

[@giovannipaolo2137]

Great video. Btw. have you thought about making a newsletter with new articles on your website?

[@QuantPy]

Thanks, yes I have, would you like having the tutorials in article format? Or something different?

[@giovannipaolo2137]

@@QuantPy yeah, the article format with coding examples is great!

[@phsopher]

Interesting video. Would have loved to see more of the code. Not sure what parsonomy means. Is it like parsimony?

[@QuantPy]

You’re in luck, link to the code is in the description 👍

[@phsopher]

@@QuantPy Cool, thanks!

[@_AbUser]

Well i think non normal error distribution caused by non monotonous growth the chart in a real.. You trying to multiply perfect line and perfect sin() and the modeling chart will growths monotonous while the real one - not. And of course You will get some excesses thant moment the real chart will start to grow like parabolic in average.. As for me thats pretty obviously.. I think the real chart should be additionally averaged, thats makes the average error bigger but makes it close to normal... (There is why You started to talk to try ARIMA like model i think..) Or to try the rose noise, i heard someone using to modeling the stock prices.. But the season decomposition with Fourier i think should be awesome to timing the stock market..

[@CTT36544]

2:30. You said you used a square function convoluted with the time series data and the got the plot. I got pretty confused here. First, why did you choose the square function (btw, wat exactly it is?) instead of sth else? Second, how did you calculate the convolution of a function with a discrete data, by substituting the time t into the square function and then computing the convolution of two data series?

[@QuantPy]

Convolutions can be difficult to understand, I recommend googling animation of convolution operation. It’s more important to understand why you would use it- i.e. denoise a time series

[@CTT36544]

@@QuantPy yes, I know wat a convolution is, although the understanding may not that deep. But I’m still confused on the two questions I asked from your video. You said you convoluted a square function with the data series. Then why not explicitly show the square function that you use and tell the reason that you choose this function?

[@QuantPy]

No problem, I choose the square function because it is simple (generic) - should be first choice for denoising. As for how I did it, all code is on my website in this tutorial. Hopefully that helps. quantpy.com.au/weather-derivatives/de-trending-and-modelling-seasonal-variation-with-fourier-series/

[@CTT36544]

@@QuantPy Thank you very much! I believe it will be much better to include the coding part in the video which can make it much more clear.

[@tsehayenegash8394]

wawwwwwwwww I am lucy when I get you. I have 15 years temperature data upto 36km. I have some questions 1. how can I calculate the amplitude and phase of the annual and semi annual oscillation? 2. how can I determine the trend of the given temperature data? 3. How did you determine a,b,alpha and theta in your equation? please send me the code. I need to your email

[@drewwilson1430]

Hi QuantPy, i've used your code and found the curve generated devolves into a fuzzy straight line if we remove the line omega = 2*np.pi/365.25. I've tried this because i'm interested in a series without a regular period. Do you know why this is?

[@QuantPy]

Seasonality is by definition- periodic, but you don’t have to define it (initialise it). If your data doesn’t display seasonality, fine. Don’t model it