Kalman Filter - VISUALLY EXPLAINED!

This tutorial explains the Kalman Filter from Bayesian Probabilistic View and as a special case of Bayesian Filtering.
As part of this tutorial, some of the essential foundational concepts such as the notion of Latent Variable Models, Bayes' Theorem, Markov Chain, State Space Modeling, and LG-SSM are also explained.
Here is a notebook showing the application of Kalman Filter on a Synthetic Dataset. The notebook uses Tensorflow Probability. What makes TFP interesting is that it can work in batch mode and make use of accelerators like GPU & TPU.
colab.research.google.com/dri...
The link to the paper which shows full derivation of Kalman Filter equations
arxiv.org/pdf/1910.03558.pdf
Look at Section 8 (the last section). Also, pay attention to the symbols/notations (x vs z).
#kalmanfilter
#BayesFilter
#StateSpaceModel

[@user-or7ji5hv8y]

So much grief could have been saved if your explanation was available in my university days. Really clearly explained.

[@shaswatgupta6590]

Sir, this is beautifully explained. Loved the clarity. The sequence of building blocks used to teach was so intuitive. Thank you!

[@KapilSachdeva]

🙏

[@100rabhkr_India]

Hands down. The best lecture on Kalman Filter!!!!!

[@KapilSachdeva]

🙏

[@abhinav_mzn]

Your presentation makes it very simple to understand...please keep making videos like this, there is a shortage of content like this on KZbin.

[@KapilSachdeva]

🙏

[@zlatasmolyaninova1140]

this is the most clear explanation among all i have read/watched! thank you very much

[@KapilSachdeva]

*Key insights/takeaways that I had wished to convey using this tutorial* _Failure to convey those insights properly and intuitively is a function of my limitations_ * Appreciate the difference between Model & Algorithm. Our goal is *state estimation* using the State Space model. There *could be many algorithms* that could operate to find the unknown parameters of our model. * Start first by understanding the model and then worry about the algorithms to use. Appreciate *why states form a Markov chain* and *why we can consider the observations as independent* of each other. * Instead of starting with Kalman Filter equations and/or trying to justify its components (Gain, matrix, etc), a more intuitive & scalable approach is to first understand the model, then *understand how and why the Bayes Rule would help with state estimation* * Kalman Filter is an *efficient algorithm* to find the parameters of the State Space model provided the model is *constrained/limited* to have "Gaussian" Random variables for state and observations and linear functions for transition & observation functions. * Efficiency of Kalman Filter is a consequence of the fact that when the components (likelihood and prior) in Bayes Rule are Gaussian then we do not need to compute the integral numerically (the normalization constant). We have an *analytical solution i.e. a closed-form solution* or perhaps in simple English, we can say we have a formula. * As mentioned above if every component in Bayes Rule is Gaussian it is helpful for computation. However, if your transition & observation functions are *non-linear* then the output (or the next state in our case) would not be Gaussian and then Bayes Rule will haunt us. This is *why linear functions are required*. In a way, appreciate that the requirement of linear function has more to do with the requirement to use Gaussian Random variables in Bayes Rule. * Many algorithms in statistics follow this 2-step pattern of "predict" and "update" and so does Bayesian Filtering (... and by extension Kalman Filtering) * Prediction at the very first step in the state-space model is your "intelligent" guess. In the subsequent steps, it's even more intelligent as it has included some observations from the previous time steps * Is it really fair to have assumptions of Gaussian Random Variables and linear functions for transition & observation? * Appreciate the delicate dance/balance between the accuracy of inference and the computational cost. If your product is extremely accurate but computationally so heavy that it does not provide desired user experience it is of no use to anyone! * Control theory people use *x* for state and *z* for observations. Statisticians and ML folks use *z* for state and *x* for observations. And people who suffer because of all this notational inconsistency are us!

[@michaelwirtzfeld7847]

Thank you. Your framing of the linear Kalman filter is very well done. I look forward to your future videos of the Kalman filter under non-linear conditions.

[@KapilSachdeva]

🙏

[@Omeomeom]

you ate this video up no lie your presentation and scope level was perfect thank you for this amazing information

[@KapilSachdeva]

🙏

[@lakdafari]

Outstanding explanation. Thank you so much for making this so clear.

[@KapilSachdeva]

🙏

[@amiyabanerjee2443]

Loved your way of explanation. Amazing! Keep adding more such educational videos.

[@KapilSachdeva]

🙏

[@yaoismyhero]

Kapil, your videos are amazing, including this one! Your teaching and presentation skills are top-notch.

[@KapilSachdeva]

🙏

[@rahulbirari401]

Just Mind-blowing loved the way you explained concepts and slowly built on it, that's how inventions and human mind works and mathematics is just a tool to realize/record complex ideas. Other lectures directly jump into maths without explaining the idea.

[@KapilSachdeva]

🙏

[@pranabislam7875]

This lecture was incredibly clear and informative wow 10/10

[@KapilSachdeva]

🙏

[@ayushichaturvedi3854]

Wish I came across this video wayyy back! Such a saviour, thanks Kapil sir!

[@KapilSachdeva]

🙏

[@violinplayer7201]

Best explanation on kalman filter! Thanks so much!

[@KapilSachdeva]

🙏

[@aruntakhur]

Beautifully Explained. Really seeking some other topics to be covered by you like Deep unsupervised learning (Pieter Abbeel) in your way :).

[@gastonparra2971]

Excellent presentation; the video is so interesting.

[@arunabhbora5610]

Best video I have ever watched in AI domain, Thank you Sir for the precisely detailed concept.

[@KapilSachdeva]

🙏

[@annawilson3824]

Kapil, this was nothing short of a brilliant video, 11/10!

[@KapilSachdeva]

🙏

[@vi5hnupradeep]

Thankyou so much sir ! You make things look so simple 💯

[@KapilSachdeva]

Thanks Vishnu for your appreciation 🙏 . It makes me happy to know that this is helpful.

[@ritishmaram3326]

Excellent explanation sir, truly motivating and inspiring sir. Thank you so much !

[@KapilSachdeva]

🙏

[@gvpathi]

Clear and information flow is natural.

[@KapilSachdeva]

🙏

[@rossijuan9548]

Excellent explanation, many thanks!

[@KapilSachdeva]

🙏

[@SaurabhKataria10]

Thanks so much for this. Terrific!

[@KapilSachdeva]

🙏

[@luizemediato6987]

Very nice explanation. Thank you very much.

[@KapilSachdeva]

🙏

[@raulpalmaa]

Thanks for this tutorial.

[@KapilSachdeva]

🙏

[@tuna5287]

The best! Thank you sir

[@szb120]

Well explained, thank you!

[@KapilSachdeva]

🙏

[@gentle-g]

Nice explain!! Thank you prof.

[@KapilSachdeva]

🙏

[@sun1908]

Thank you.. very nicely explained.

[@KapilSachdeva]

🙏

[@ruslansergeev4061]

This is an absolute internet gem !

[@KapilSachdeva]

🙏

[@MLDawn]

I really wish you would put an equally amazing tutorial on Kalman Bucy filters (continuous state space models). Thanks

[@Manishkumar-ww4gm]

Very Nice explanation Sir.Than you!

[@KapilSachdeva]

🙏

[@alexandervocaet566]

Your video is gold, good sir.

[@KapilSachdeva]

🙏

[@ssshukla26]

I just completed your video. Nice one. Subscribed.

[@KapilSachdeva]

🙏

[@user-or7ji5hv8y]

A possible suggestion. I think a Tensorflow probability example of this would be really nice.

[@AI_Financier]

A superb video, thanks. a simple numerical example at the end would be great.

[@KapilSachdeva]

There is a link to the notebook in the description of the video.

[@xuyang2776]

Thanks a lot ! This video is easiest to understand that I have ever seen. But I have a question. If the parameters A(k-1) in transition function and H(k-1) in the emission function are unknown, how to handle?

[@FGundogan]

An application would help great, many thx Kapil!

[@KapilSachdeva]

🙏

[@deepikabhoominathan4969]

Explained neat and clean...thanks a lot 👏 keep explaining the topics in artificial intelligence sir

[@KapilSachdeva]

🙏

[@jameschen2308]

Top notch!

[@KapilSachdeva]

🙏

[@naifalkhunaizi4372]

Art at its best

[@KapilSachdeva]

🙏🙏

[@Vidit2011998]

Professors in Germany couldn't teach a shit about this. Meanwhile you explained wonderfully!

[@KapilSachdeva]

🙏

[@vijaymaraviya9443]

Thank you 👌🏼❤️

[@KapilSachdeva]

🙏

[@kentoshintani3020]

Which presentation software are you using? Microsoft ppt?

[@KapilSachdeva]

Yes MS Powerpoint!

[@user-or7ji5hv8y]

Do you have examples of where such Kalman filtering can be useful.

[@KapilSachdeva]

I am assuming that you are asking about why Bayesian Perspective of Kalman Filter is important & useful. Kalman Filter, the algorithm, assumes "Linear Gaussian" State Space Model whereas most of the real world phenomenon do not have linear transition & observation functions and assumption of Gaussian distribution for State & Measurement Random variables is not always appropriate. Note - In the above paragraph I have distinguished between "algorithm" (i.e. Kalman Filter) and "model" (i.e. State Space). It is important to appreciate this difference. Seeing Kalman Filter as a special case of Bayes' Filter helps you appreciate that you can apply other algorithms to find your latent/state variables. Essentially, you now have a "general framework" of Bayesian "predict" and "update" equations where distributions could be other than Gaussian provided you have a way (algorithm) to solve these equations. For e.g. you could use Variational Inference or MCMC methods to approximate the posterior state. And in the modern machine learning realm, you can think of the transition & observation functions being modeled by neural networks. See for example this paper - arxiv.org/abs/1609.09869 (Structured Inference Networks for Nonlinear State Space Models) As you can appreciate that this way of thinking in terms of a general framework helps you find solutions for more sophisticated problems where Kalman Filter's requirements (i.e. linear functions & gaussian random variables) are not met. Hope this makes sense; let me know if you need more clarifications.

[@alnar447]

Excellent. At around 13 min mark the measurements could potentially indicate any state with certain probability? Emission function can take measure ment to any given state? So multiple arrows potentially

[@KapilSachdeva]

Not sure if I understand the question. Could you please elaborate?

[@pratikanand2200]

you really explained the concept very well but I think that if you could give an example first then understanding this, was fun. Wikipedia has a good example of this.

[@KapilSachdeva]

Thanks Pratik. Point noted. Good feedback; though it does not compensate for a concrete example I could have provided, there is a link to a notebook in the description. It shows a modern implementation of KF using tensorflow probability. Check it out if interested. 🙏

[@100rabhkr_India]

Sir, do we have lectures on particle filter. An solved example for Kalman Filter will be a "cherry on the cake"

[@KapilSachdeva]

Thanks Saurabh. Nothing on particle filter yet but I have thought about making one. Hopefully soon 🙏 Look in the description of the video. You will find the link to the example.

[@MLDawn]

I really wished you had also explained the meaning of Kalman gain at 28:45.

[@partha95123]

Wow!

[@KapilSachdeva]

🙏

[@roy1660]

Please upload video on particle filters and feedback particle filters

[@KapilSachdeva]

Sorry for the late reply; will make some tutorials on them as they are indeed very interesting. Working on some other tutorials that would be prerequisites for particle filters.

[@roy1660]

@@KapilSachdeva Thanx for the consideration.

[@KapilSachdeva]

🙏

[@aishahalbarakati7942]

Would you please share the paper of the derivation that you mentioned at the end of the video??

[@KapilSachdeva]

Here it is - arxiv.org/pdf/1910.03558.pdf Look at Section 8 (the last section). Also pay attention to the symbols/notations (x vs z).

[@aishahalbarakati7942]

@@KapilSachdeva Thank you

[@KapilSachdeva]

🙏

[@100rabhkr_India]

Time 28:58, Can you please explain what will be the value of m_k in the 1st equation of Update? How is the calculated and updated? Is it a constant?

[@KapilSachdeva]

m_k is the observation (eg the value that your sensor will give) at step k.

[@adarshsdinopark8251]

Can you please share the link for particle filter

[@KapilSachdeva]

Haven’t made one yet :(

[@mmattb]

Dumb question: Why are the transition (A) and emission (H) matrices time indexed (referring to the slide at ~26:30)? Is that intended to imply that a Kalman Filter can be applied to a nonstationary process? Second dumb question: why is the time index on H k-1 instead of k? The conditional probability distribution on the right side of the slide shows k as the time index, which makes sense to me, but I don't understand why k-1 is shown on the left side. Anyway, thanks a bunch for the video. I love you build up the basics first!

[@KapilSachdeva]

Both of your questions are valid (and definitely not dumb)- See below resource: arxiv.org/pdf/1910.03558.pdf Look at Section 8 (the last section). Also, pay attention to the symbols/notations (x vs z) (a) yes it can be applied to non stationary process and some literature puts the time step as you will see in above paper (b) I made the typo 😒it should be H_k and not H_{k-1}

[@mmattb]

Ah okay, that clears it up. Thanks a bunch Kapil.

[@KapilSachdeva]

🙏

[@user-wr4yl7tx3w]

Be great to see a python coding example.

[@KapilSachdeva]

There is a link to an example notebook in the description

[@PrashantBatule]

A good explanation, much appreciative of the content! But I think there is no extraordinary "visualization" here compared to a standard textbook. I would suggest a more intuitive explanation through an example, which is nowhere to be found on the internet.

[@KapilSachdeva]

🙏 thanks for the feedback. Yes I have been thinking about re-explaining it with an example. Regarding “extraordinary visualization”. This could be subjective and I understand that it is not a 3blue1brown level visualization 😊 but it does not mean it is not good enough. As a matter of fact some of the “extraordinary visualizations” could hide the necessary underlying conceptual details. They impress people but not necessarily help them understand the details.

[@jameschen2308]

Are you a professor?

[@KapilSachdeva]

No I am not :) ... but any concern?

[@jameschen2308]

@@KapilSachdeva no concern, sir, just very appreciative

[@KapilSachdeva]

@@jameschen2308 thanks. Very kind of you.