The Fisher Information

[@HowardMullings]

I highly recommend jonathanpober's video on "Intro to Fisher Matrices" as a compliment to this one. I feel you need this video and jonathan's to make sense of this topic. The visuals of this video are intuitive but jonathan explains why the log of the likelihood is used and how the Taylor expansion of the log likelihood relates to the hessian.

[@Mutual_Information]

Yea I've seen that video - it covers the topic quite well. I agree it's also worth checking out. As much as I like my video, *really* understanding FI requires seeing it from a few angles. Also, this video is not comprehensive. So +1 to the recommendation

[@Diego-nw4rt]

This is one of the best math/statistics videos that I have ever watched so far, if not the best. I don't have a background in statistics, however I understood the intuition behind, since your explanation and the tools that you used make the topic easier to understand.

[@Mutual_Information]

Wow man that’s so nice! I’ll try to keep the food stuff coming!

[@Boringpenguin]

After all these years I have finally understood the intuition behind fisher information, thank you so much!

[@sawmill035]

I had to pause the video every 5 seconds to re-listen to every phrase because it was just so dense with information (no pun intended). Thanks!

[@howtoeasy5882]

Indeed, the fisher information information can tell us what the Cramer Rao bound is. Researchers, like Dr. Ahmad Bazzi, use this to benchmark interesting signal processing estimators.

[@gordongoodwin6279]

This is by far the best video on Fisher Information and its not even close. Hope you put out more videos

[@faronray5903]

This is hands down one of the best math videos I've ever watched on KZbin. Thank you so much.

[@Mutual_Information]

What a compliment! Thank you, my intention is to keep it up.

[@anonymousalligator7500]

Bro, this is the top notch quality of education. You are an Educator, man.

[@Mutual_Information]

Thanks brotha

[@yessirge]

Shout out to you man, I can really tell how much thought went into the didactic decisions of this video. Thank you so much!

[@Mutual_Information]

And thank you for watching!

[@definesigint2823]

I feel like...I almost grasp this / like I need to study more. The discomfort's just about right (i.e., not intimidating) and is a nice reminder to keep working.

[@Mutual_Information]

It’s one of the trickier topics I cover. I remember not getting this for the longest time, but eventually I had this visual in my head which helped a lot. I think my tripping block was the two roles of theta.. both an evaluation point and to represent the “true” data generating value. It’s tricky! But if there’s something specific you aren’t sure of, feel free to ask.

[@Stopinvadingmyhardware]

MAKE tHINGS for people to steal and make money off of you!!!

[@DataTranslator]

I admire your approach. Understanding discomfort is needed to grow. Keep going, you’ll do great things

[@alexmonfis9305]

Thanks man!! I'm doing a master on Data science and you just save me for my test :) Great animations and clarity!

[@karanshah1698]

The vagueness of the goal. Finally. Someone I can relate to.

[@karanshah1698]

Was tired of seeing many takes on this topic, and randomly decided to give this video a shot. Just as #define SIGINT 2 mentioned, this is on the verge of comfort-discomfort! Neither crossed the brain fully nor did it intimidate... I'll watch this on repeat to digest bit by bit. Great work!

[@Mutual_Information]

Happy to hear it! If there’s something specific you don’t quite understand, feel free to ask.

[@karanshah1698]

@@Mutual_Information Really appreciate the content quality. Can you help relate the relationship b/w three concepts: Fisher, Hessian and KL Divergence with visuals like these? Edit: There also happens to be a misplaced usage of empirical vs non-empirical Fisher. Can you touch upon that as well?

[@DuaneRich321]

@@karanshah1698 These concepts come together nicely with an explanation of natural gradient methods. I can try to cover all those in that video.

[@ashwinkotgire2303]

Man, you just paved a concrete road to my future Thanks

[@abhishek.goudar]

The plot at 3:30 nailed the idea for me! Thanks!

[@mrx42]

Best teacher ever ! Keep up the good work ! You 've just turned my day brighter.

[@littlebigphil]

This video combined with thinking about performing gradient ascent was helpful. Our objective is to maximize the likelihood of our current parameterization is given our samples. Maximizing the log-likelihood is similar to maximizing the likelihood but with harsh loss for outliers. The score uses this loss to perform gradient ascent. Larger scores gives larger step sizes. Because the score at the optimal value is 0, for any score to be large, there must have been an interval where the slope of the score was also large. All of that gives the average (negative) hessian of the log-likelihood.

[@Mutual_Information]

Interesting stuff! This reminds me of natural gradient methods, which I’ll be covering later on.

[@user-wr4yl7tx3w]

regardless of the number of views, given the subject nature, such content is such a great service and will be relevant for years to come.

[@kimchi_taco]

You are the best DJ I've ever listened

[@semduvida3243]

Don't stop doing videos, your work is amazing!

[@Mutual_Information]

Ha don't worry, I have no plans on stopping

[@benvonhunerbein1865]

Really amazing video! Great step by step introduction of concepts. I also really like these movements across curves to give a better intutition before revealing the solution. Thank you!

[@spitfirerulz]

Hey, thanks! I came here from MITx 18.6501x. That bit in the middle about highly correlated 2-D case filled in the missing intuitive link for (sort of) grasping why Fisher Information matters. And I can now also see why it is used in the Jeffrey's Prior. The water-bending hand gestures are a bonus. Cheers.

[@Mutual_Information]

lol I may chill out the hand gestures. I'm still getting my KZbin legs. And I'm going to do a separate video on the Jeffrey's prior. That's a tricky one to understand.

[@CristalMediumBlue]

Thank you very much for sharing this valuable information. I am planning a binge watch on your channel in the next months.

[@isobarkley]

you're so passionate, engaging, and a talented educator. thanks for all of your content, new and old :)

[@Mutual_Information]

Thank you, much appreciated :)

[@lonamoch971]

When I saw the video cover, I was pumped for it. As expected, this was a fantastic intuitive explanation! Thank you

[@Mutual_Information]

Thank you very much!

[@Manu-gy6tq]

As a stats student: Thank you so much - amazing explanation!

[@lichungtsai]

Hey, guy. It’s the best video about fisher information ever!

[@javierferrer450]

Great summary and well explained with motivational dynamic graphs. Thanks!

[@alanamerkhanov6040]

Best video on Fisher Information on the web! Thanks, thanks a lot.

[@Mutual_Information]

Appreciate it Alan!

[@xingyanglan6836]

sometimes i wonder if my professors during zoom get curious and go see what video presentations on youtube look like and feel a lil sad deep down

[@rohansinghthelord]

going to grad school for ML and realize I needa brush up on stats, this helps a lot!

[@Mutual_Information]

Nice! Excellent choice in grad ;)

[@tchedoumenou1165]

I'm constantly feeling like you're going to announce some exiting news man! Great content btw.

[@xLyndo]

You can really tell a lot of time and effort went into this. Thanks a lot. Definitely subscribing and am looking forward to more videos.

[@Mutual_Information]

Thank you! Comments like these means a lot

[@toobabb3613]

Thank you, I wish I watched this video before searching lots of articles.

[@jpap676]

An impressive video. The quality of your visualizations is very high. Thank you for the insights.

[@junninghuang4343]

Nice video, haven't thought about likelihood, score function, fisher information matrix in this way, very intuitive and straightforward. The nicest part is the visualization is based on the evaluation on the true parameter, which explains the tricky identity of the expected likelihood gradient. One minor suggestion, because I learned FIM before, from my knowledge and wiki, FIM is the variance of log-likelihood gradient evaluated on any parameter theta, but in your video, you misstated that FIM is evaluated on the true parameter.

[@Mutual_Information]

Yea the FIM can refer to the function as you say. But in that case, the inputted parameter still acts like the true parameter. Because it’s only at the true parameter that the expected gradient is zero, and that’s always true of the FIM, regardless of the given theta . Still, I see your point - the terminology is better applied to the function than the matrix of values.

[@junninghuang4343]

@@Mutual_Information Exactly, from my site, the statement of FIM is evaluated on the true parameter would be misleading to someone, that's why I suggest keeping the function form of FIM in mind which is more mathematically rigorous. And yes, you are right, the fact that the expected gradient is zero indicates the true parameter and true FIM. Anyway, thanks for the nice visualization, making such a nice video takes a great effort more than writing a blog I guess. Salute!!!

[@Mutual_Information]

Glad to have your comments Junning here. Please stick around! :)

[@TuemmlerTanne11]

Impressive work! You should have way more subscribers. Because you deserve it and for the peoples sake ;) I feel quite privileged to have found your channel so early. Keep up the good work, arleady looking forward to the next video!

[@Mutual_Information]

Thank you very much! I’m getting a good response, so I think the growth is on its way. I appreciate the support!

[@zactron1997]

Came from mCoding's shout-out. Nice video!

[@rockapedra1130]

Fantastic educator! I've been avoiding learning this for quite a long time! Thx thx thx

[@Mutual_Information]

Happy to!

[@heinrichvandeventer357]

I paused at 25 seconds in and nearly choked to death on my coffee. I like the content. Keep it up! :)

[@Mutual_Information]

Lol as long as you’re not in fact dead, I’ll take it as a compliment!

[@heitorcarvalhopinheiro608]

Awesome work, DJ! Loved it. As some have said you fill, in a superb way, the statistics gap from 3b1b. I'm currently enrolled at the Bsc in Statistics and Data Science in Brazil and would love to hear from you what books, in your opinion, were essential for you to build that knowledge. That would be a great video, btw. "Essential books for those who aspire a career in Statistics and Data Science"

[@Mutual_Information]

Hey Heitor, appreciate the comment - glad you’re enjoy the channel. Regarding that vid, I probably will not make it, just because it’s not in line with my style of vid, which are all mathematic concepts. But! That doesn’t mean I won’t provide that info. I can tell you directly that my absolute most favorite, most influential books are : 1) Machine Learning : A probabilistic perspective, by Kevin murphy. This was a huge book for me. Super important. It covers so much in real good depth. There is a second edition draft free online too. It’s my absolute fav book 2) the elements of statistical learning. This is a classic, written by some titans within the field. Every page read is a worthwhile investment 3) Deep Learning by Goodfellow, bengio and Courville. Excellent book for navigating the Wild West of deep learning. Great intuition and very well written. Those are the big ones for me.

[@nabibunbillah1839]

​@@Mutual_Information thanks so much

[@piergiorgiolanza4572]

Excellent video that helped me to grasp this concept quickly and in neat way. Thank you

[@tejasvichannagiri2490]

Great video, very clear and easy to follow, but precise also. Thanks!

[@SO-wg4yb]

How great is this video! Thank you for making this great content. Please continue to do your great jobs:)

[@Mutual_Information]

No plans on stopping :)

[@sdsa007]

Wow! the visuals are even better than on Ian Explains...

[@smartboyvijey]

Your videos are amazing. Looking forward to more of your videos in Information Theory.

[@dayibey9700]

woow! very good explanation with useful, spot-on visuals. will surely help developing intuition about this sophisticated concept. subscribed to see you keep up with such a good work.

[@Mutual_Information]

Fortunately I have zero plans of slowing down

[@alexanderk5835]

thanks, such a great explanation video with an amazing visualisation

[@guillermosainzzarate5110]

Wow you really should make a video about statistical manifold. Thanks for your videos, they are really amazing!!

[@Mutual_Information]

Not a bad idea..

[@aiart3453]

Fantastic work mate. I added you on linkedin to get your help one to one. Thank you for the video. Cant get better.

[@psl_schaefer]

Amazing Video! Thanks for taking the time to produce such awesome content :)

[@nikoskonstantinou3681]

Keep up this great job! One day your channel will be big... I can sense it from the amazingquality of your videos and your passion on the subject!

[@Mutual_Information]

Thank you! That means a lot. These early days will be a bit of a slog, but I’m confident there’s an appetite for this level of details.

[@HelloWorlds__JTS]

(8:12) I think you neglected to change the plot labels, since they are no longer for normal distributions. Thanks for this video, is a great effort!

[@Mutual_Information]

Ahhhh yes, good point. Oh well, sounds like you knew what I was going for

[@6DAMMK9]

Come from... AI art community. Msc of CS here, but not a math pro. I was stunned by "fisher merging" was just a single line of equation. Now I know what is the "fisher" inside the hood 😂

[@marceloenciso6665]

Fucking genius! keep going this way, this kind of unique materials focus on intuition helps more than you can think of.

[@Mutual_Information]

Thank ya - more coming!

[@pengbo87]

thanks for making the world better

[@PiyushVerma-em6wq]

Question: can I consider two normal distributions as distributions from two different ML models (as if we are trying to compare which mode l has highest fisher information)?

[@Mutual_Information]

Yea, that's the idea here.

[@kirankulkarni2396]

Excellent explanation. Thank you very much!!

[@mohitwankhede9372]

You are fire..🔥 You explained this much easier way

[@abdullahsheriff_]

This. Is. Beautiful. Intuition 100.

[@alessiotonello9666]

Amazing video! Thank you so much!

[@visualish]

That was fantastic, thank you. keep up the good work

[@BiologyIsHot]

Can you do a video on canonical correlation analysis (CCA)? I get PCA but can't wrap my head around CCA and there aren't any great videos on it.

[@Mutual_Information]

Maybe one day, but for now I have no concrete plans for it. Do you know of any cases where it is used in real applications? I've only come across it in textbooks.

[@shskwkfvekqlevjwkwv]

so awesome, thanks for the effort!

[@stathius]

First of all, thanks a lot for taking the time to create such a great visual explanation, very refreshing way of presenting things! I was wondering if we are not at the true μ then the variance of the scores is not called Fisher Information anymore? Because irl we are most of the times not aware of the true μ anyways.

[@Mutual_Information]

Yes! Frequentist statistics has this radical.. irrelevance for that reason. Yet it doesn't stop people from using the MLE as a plug-in for the "true parameter" and charging forward as though there's no issue :)

[@njitnom]

hello at 6:03 when you start your intuition, you zoom in on a value of a single score function right. So when there is only 1 observation, a positive value recommends shifting mu to the right, a negative value recommends shifting my to the left. So in high variance case, more scores are closer to zero, but isnt it also the case that the low variance case recommends more extreme different shiftings? Because some of those score function are much more negative and some other score functions are much more positive, therefore recommending a huge shift to the right and to the left in contrast to the high variance score functions. If this is correct, how come that then still the high variance, and not the low variance, provide a bigger set of possible mu values?

[@Mutual_Information]

Hm, let me try to clarify. In the high variance case, the scores would have large magnitudes… so if you wanted to increase the log lik by 1, you wouldn’t have to move far at all (in either the left or right direction) If it’s the low variance case, then you get the “wildly different recommendations” as to where mu is. I think you might be getting a smidge confused on low variance / high variance. Low variance means scores are like -.001, .002, -.001, .0005. In the high variance case, the numbers would be like 10.2, -12.4, 8.7, … Hopefully that helps

[@njitnom]

@@Mutual_Information oohhh oops with high variance i meant low variance yeah, sorry about that. I try to rephrase my question :D, its very visual in which i formulate my question i hope u understand. The idea is to draw for each 2 variance cases, the true CDF of it on a (-inf, inf)x[0,1] plane. Then a random sample of n is created by taking a random sample of n of unif(0,1) and looking at their image. Then make a third axis (dlog p) that shows the score functions of each of these data points. Then if im correct the distributions of score functions can be derived by finding the intersect of these score functions on the [0,1]x[dlogp] plane, evaluated at a certain mu in (-inf,inf). And in this case this distribution approximates a normal distribution when the random sample tends to infinity right. Is the reason that in the low variance case, the variance of the distribution of scores evaluated at the true parameter value is higher than that of the high variance case, because: when we take one data point from the UNIF, and look at the corresponding high variance data point and low variance data point, and fix the plane at the true parameter value, the intersection point of the low variance data point is guaranteed to be closer to zero than that of the high variance case. And because this holds for all data points, the distribution of score functions of low variance, has a higher variance. If so, do you know why this is guaranteed to happen? Why are the slopes of the low variance score functions sufficiently small to guarantee this. Sorry for long text :D

[@marcuschiu8615]

This video helped alot!!! What software did you use to create those visuals at 6:14? My previous comments were deleted, prob since I wanted to share a website that recreated an interactive visualization of Fisher-Information WIP

[@Mutual_Information]

Hey Marcus, glad it helped. The visuals are created with Altair, which creates static plots (like Matplotlib). Then I use a personal library to stitch them together into vids.

[@scar6073]

Bro literally has Jaynes's book on the desk and talking about frequentist ideas lol

[@kartal1903-u4y]

Amazing job, DJ! It is very intuitive and the visualizations are on 🔥, can I kindly ask which visualization tool do you use? Thank you.

[@Mutual_Information]

Thanks Hidir! I use a plotting library called Altair (altair-viz.github.io/getting_started/overview.html), which is a Python plotting library similar to matplotlib. Then I have a personal library I use to stitch the pictures into videos

[@douglasmason6067]

Amazing work!

[@brandomiranda6703]

One thing I noticed is that the fisher information being high could be used to select the true parameter (or between different models, NNs, architectures, functions, etc)...but it must be super easy to construct artificially a function such that for a given data set the fisher information is extremely high (and the gradient wrt w is zero of course)...but will that work well on the test set? It seems in the end fisher information is a nice heuristic (if it's easy to compute which I doubt it is since it depends on the hessian, the variance of the scores should be fine to compute I hope) to choose a model - but the validation set (and test set without cheating) are the "ultimate truth".

[@AliRaza786]

Man you are amazing. Keep doing the good work.

[@Mutual_Information]

Thanks, I will!

[@ZarakJamalMirdadKhan]

Cant your explain further the hessian metrics and multidimensional expression of the FI in detail? Please. P.s: I'm saving the playlist. Your visuals makes the econometrics concepts so easy. Thanks a lot.

[@KaalvoetNinja]

mCoding sent me. Really glad he did ☺

[@Mutual_Information]

Welcome!

[@GumRamm]

Great video and explanation! One thing that wasn’t clear to me was that we take the expectation over theta*, where throughout the video we treated it as an unknown but fixed variable. How would one take the expectation when theta* is fixed or has an unknown distribution?

[@Mutual_Information]

In practice, you can't. That's why this is a little weird. In practice, you have to substitute in some estimate for the true parameter, and that's where a lot of the nice properties fall away. But, when we're speaking theoretically, we can do whatever we want! Like talk about a fixed, true parameter and derive results using it. Think of this video as making this statement: If you knew the true parameter value, you'd get this nice thing (the FI matrix) which tells you how certain you should be about estimates of the true parameter. That's a weird statement to make! But, it's a mathematical fact. People will utilize it in practice by substituting estimates in and hoping the math still holds.. well enough.

[@jadecheng7483]

Hi, beautiful video! I wonder if I could ask what tools you used to plot the first PDF plots, where you compared log(N(x|mu, 25)) to log(N(x|mu, 1))? they looked so pretty, as the intensity of the colour also indicates density of lines.

[@Mutual_Information]

Thanks! I use Altair, the python plotting library. It's for static plots and I use a personal library to convert them into short videos.

[@jadecheng7483]

@@Mutual_Information thanks!!

[@brandomiranda6703]

Btw, why do you say frequentist is a bad term...isn't that what nearly 100% of deep learning is now days?! Thanks for the video! Seems you have legit channel. :)

[@Mutual_Information]

Thank you! Very happy to have you as a viewer To answer your question.. from my very narrow view of the whole DL space.. no I don't think it relies heavily on freq statistics. Sure, p-values are reported sometimes (though, I can't recall seeing them recently) in some statistical analysis of performance on DL models.. but the models themselves don't share the most defining assumptions of frequentist statistics. I don't see anyone speculating there is some 'true' parameters of the DL architecture. One reason in particular is because we know we almost always arrive at some local optimum.. so we could never arrive at those 'true' parameter values. I think ideas from freq stats are treated more like a buffet. Some things get borrowed (the Fisher Information), but no one is subscribing to the whole of the freq stats.. and that's b/c it wouldn't be effective.

[@waterseethrow9481]

Great video!! Amazing animations! Is there any way to quantify the Fisher information? Is there any rule of thumb?

[@Mutual_Information]

The best we can do is to substitute the MLE for the true parameter estimate.. and then we can start working with numbers. But that version of the FI can disappointment. Not being at the true parameter estimate means several of the properties we like so much.. don't technically apply.

[@a_alex_l2041]

Wow, great, it really helped !

[@brandomiranda6703]

I think a good (foundational) question is if the expected gradient is zero over our data, then what is the difference of the fisher information concept with maximum likelihood? Will leave question to someone to answer or myself after I finish the video or think more about it myself. I am just at 5:32 !

[@Mutual_Information]

Glad to see you are taking your time with it Brando!

[@brandomiranda6703]

@@Mutual_Information I'm the type of person the doesn't let a subject go until I have completely surrounded it! It has its drawbacks but it's fun most of the time :)

[@tvvt005]

So is score function considered similar to an error term, showing how close or far we are from the true value generated by the parameter for the data and at score=0, we’ve reached the right point?

[@kalebbennaveed3704]

I have a slightly different question. What software tool do you use to create animated plots?

[@Mutual_Information]

Altair to create images of static plots, and then I paste them together with a little library i've written.

[@marcinelantkowski662]

As all the other videos, this one provides a great explanation, but tbh a key piece is missing: why would we ever care about the FI? When is it useful? Why is it popular? What problem can it solve for me? E.g. I already knew that if I want to measure some quantity, it's better if the underlying random variable has low stdev, instead of a large one :D

[@Mutual_Information]

Good point!

[@pluviophilexing2580]

Thank you so much 😘very intuitive

[@des6309]

amazing stuff thanks!

[@outtaspacetime]

I struggle a bit with the part on the covariance matrix, but I feel like I could get it if would do some hard math on it with some numerical examples with the intuition of this video in my mind! Thanks was really helpful

[@Mutual_Information]

The covariance matrix is a tricky concept. Took me awhile to get use to.

[@HuyNguyen-fp7oz]

Great! I hope you will keep high standard for your videos like your great answers on Quora.

[@Mutual_Information]

Ah a Quora reader! Glad to see you made it over here. And will do!

[@xandermasotto7541]

excellent video. My only complaint is when it turns black for several seconds and I think my laptop went to sleep lol

[@eduardodiaz5459]

Good video. Which software do you use to make the math animations???

[@dixztube]

This was good video. Kinda gotta slow it down but I followed lol

[@Mutual_Information]

Yea this is an earlier video, but I got that feedback. Newer videos are a bit better paced

[@zoesoohyunlee7209]

Love the visualization and clear explanation!! Finally, I can understand intuitively the log-likelihood function and Fisher information matrix. Thank you so much for creating this video!! One small thing I'd like to mention: I really enjoyed the liveliness of your explanation but found the hand gestures a bit eye-catching while I was trying to concentrate on the written information on the left. Maybe a slower movement could help?

[@Mutual_Information]

I haven’t heard feedback like this before - very useful. Did not think of that but totally makes sense. I’ll try to chill the hands out next time. I’ve already recorded a few vids without this feedback, but the ones beyond that should reflect that. Thanks for the advice!

[@orsike192]

@Mutual Information in case this lecture is difficupt for understand which books and/or videos would you suggest me to read/watch before rewatching this video? Thanks

[@Mutual_Information]

The Elements of Statistical Learning covers this topic well. I forget which chapter exactly but it should be easy to find. If you’re interested in learning about the whole field and you’re relatively new, they have a related book called “An Introduction to Statistical Learning”, which is from a related group of authors.

[@chen-yuwei8793]

Thanks for the great video! I wonder why the "log" in front of the density function? I mean, if I replace all log P by just P, does the quantity still make sense?

[@iitvlogwale]

Great video

[@sdsa007]

Interesting .... you mentioned Hessian.... but I don't think you mentioned Jacobian.... is there a Jacobian connection?

[@Mutual_Information]

Yep. Say g(x) is the gradient of the function f(x). Then the jacobian of g(x) is the hessian of f(x).

[@BillHaug]

thank you

[@amirmahmoodrafiey3496]

Really perfect 🙏

[@lukasstein6231]

Is there a paper that goes into more depth on those beautiful illustrations? (for example at 8:11)

[@Mutual_Information]

Thank! And, to answer your q, no, not that I'm aware of. When I first learned them, this is what I had in my head. Only way for me to make sense of it.

[@brandomiranda6703]

what textbook you have there in the beginning of the video?

[@Mutual_Information]

Probability Theory by Edwin Jaynes - a classic!

[@brandomiranda6703]

@@Mutual_Information Thnx! Cool channel btw! It's appreciated. Hope to see more of your stuff! I like your conceptual approach. To many ppl do either to informal or too rigurous. Thanks!

[@AlisonStuff]

I understand all of this!

[@Mutual_Information]

Haha good! You can explain all back to me on Christmas.