Using a KALMAN FILTER to BOOST Returns

Having looked at a heap of backtests, it is clear that a Kalman Filter is not to be ignored and is indeed of value. If you trade pairs via statistical arbitrage, then you do not want to miss this one covering how to use a Kalman Filter to get an optimal hedge ratio for your spread calculation. In fact, when I came across this and worked through the detail, I spent the last 6 to 8 months completely rewriting the statistical arbitrage tool and all applications on the platform.
The Kalman Filter is known as an optimal estimator. It can estimate the true value of something which is unknown at the time, like the position of a car when it enters a tunnel from other input sensors prone to error. I like to think of it in financial trading as a way to extract signal from noise.
Here, we use it to calculate a dynamic hedge ratio for pairs trading in cryptocurrency and forex.
Will talk more on the platform changes in the next one.
Take care and talk soon,
Shaun
website: cryptowizards.net

[@Raekonn]

The problem with the static hedge ratio was that you were calculating the linear regression slope on all the data in the backtesting period, and using this one value for all the trades you were doing in the backtest. Therefore, you introduce lookahead bias, a super common oversight in all financial data science: when you are supposed to execute a trade in the backtest, if you were to calculate the slope at that specific point in time as if it was a real world trade scenario, you wouldn't have the data from that point onwards available, and the hedge ratio at that specific point in time would look different, and so would other critical parameters that depend on that initial linear regression you are doing. As you pointed out, we need rolling (everything) to get any meaningful backtest result. You really gotta think about possible ways you could be leaking information at any step of the development, otherwise we're just backtesting a scenario that in reality never happened, or that we had no possible way of knowing it would eventually happen. Really looking forward to your future videos about all these amazing things you've been crafting, great job!

[@CryptoWizards]

Excellent point on lookahead bias. This comment addresses the elephant in the room nicely.

[@markgamache6377]

nice summary

[@ruicosta3855]

​@@CryptoWizardsbut how do we trade with that info, don't care how it works, I want to make money lol

[@haedike]

Brilliant, thanks! Possible to add sample code (or notebook resource) to one of your courses? I am a CW subscriber and love all your courses.

[@dochaincryptoacademy5317]

Thank you my professor any time I see that you made an upload I run without looking back just to be the first to watch it. Great value delivered looking forward to this being integrated into the wizard platform

[@CryptoWizards]

Really appreciate such positive comments. Uplifting and motivating

[@nirglazer5962]

So the kalman filter in this case is used to determine the hedge ratio? Everything else is basically the same right? the trading is done via Z-score deviation still

[@markgamache6377]

Nice video, you should make sure to detail what TF you are trading and time period (like last X years). Would love to see the HFT version and if it’s profitable.

[@CryptoWizards]

Glad you brought up HFT. Something that we will for sure cover.

[@paolog2524]

Great video! As always Do you think this is applicable to the Indexes futures contract also?

[@CryptoWizards]

I haven't traded that myself but I dont see why not

[@proo3968]

Incredible! Thank you mate

[@CryptoWizards]

Always brother

[@GrizzCodes]

Thank you for this! Very valuable information. Is there a way you can trace those crypto pairs sharpe ratio back in 2022? Was there more opportunity during that time? Thanks

[@CryptoWizards]

Very good question. I haven’t run that analysis but could be worth a future video

[@polmaksim]

Hello Wizards Grandmaster. Is there any code examples for the video to understand more deeply about Intercept, Slope and Kalman Filter in Common... Thank you.

[@CryptoWizards]

Hi there. Not yet. Will prepare some tutorials once the upgrades are launched.

[@polmaksim]

@@CryptoWizards Thank you 🙏🏼

[@s1dev]

would you share on which language/stack was the new platform build for more performance?

[@markgamache6377]

he used Rust

[@markgamache6377]

So you are using the Kalman filter to make up for lack of data? For example if you have two or three years of previous data for your hedge calculation each subsequent data point won't change the line fit much therefore the kalman won't be that useful.

[@MrJambanana]

Actually Kalman filters do not blindly apply heavy weight to historical data; this is one of their most important characteristics. Kalman filters are extremely responsive, generally much more so than a simple moving average. Being "weighed down" by data from a long historical window is not a thing with Kalman filters. Further, they account for instantaneous changes in noise and adjust accordingly (self tuning).

[@juergenx298]

Have you are using the Kalmann filter? I haven't seen it.

[@alexeygrom1834]

wow you finnally discovered a kalman filter ! :) i remember i told you more then a year ago to try to use to divide one security for another u said ok i'll take a look .... dynamic relation ratio is almost the same as kalman but way simpler

[@MrJambanana]

Be careful about broad brush dismissal of simple yet powerful techniques. Kalman filter is far more capable than simply dividing the price of one security by the other, which should probably NEVER be your basis for trading (because that would never work). Hopefully, what you meant to propose is a moving average of the ratio of prices? The Kalman filter is not much more "complicated" than that, but it is hugely more powerful. It is actually very similar (both conceptually and in mathematical form) to a recursive exponential filter. For the small price of a *slightly* more involved calculation, it essentially "self tunes" and yields a remarkably responsive (e.g., low 'lag') yet low-noise estimate. This is accomplished by factoring in the covariance of the model, and the covariance of the observer(s). Big words, but not big concepts - look around on youtube for conceptual overviews. Another reason to learn this tool - you can use it to combine multiple indirect indicators to refine your estimate of just about anything. Keep in mind that the real time trajectory computations for Apollo were performed on an extremely primitive computer (far less compute power than a 1990s flip phone) by employing the Kalman filter algo.

[@ArunSharma-ek9tl]

Cwn this be implemented on equities. On say quantconnect or similar.

[@CryptoWizards]

We have it working for equities. Just filter on Forex, but enter stock tickers and it will work (same data provider)

[@dontbeafool]

What do you use to build the dashboard

[@CryptoWizards]

Javascript for frontend and Rust for backend

[@ministryNoiz]

Thanks advance

[@CryptoWizards]

Always friend

[@barbeywhetzell1983]

promo sm