Expectation Maximization: how it works

Full lecture: bit.ly/EM-alg
We run through a couple of iterations of the EM algorithm for a mixture model with two univariate Gaussians. We initialise the Gaussian means and variances with random values, then compute the posterior probabilities for each data point, and use the posteriors to re-estimate the means and variances.

[@Ewerlopes]

Man, this guy is unbelievable! I wish I had professors like him! Great explanation, thanks!

[@salmankhalifa2867]

God, this is so interesting and now makes sense. I wish you were my data-mining lecturer! You should look into getting with khan academy!

[@MrReierz]

You have no idea how good this presentation was. Ive searched the web for hours. Nobody could explain this, except your video! Thankyou

[@Murphyalex]

What an excellent explanation! As soon as I pulled a face trying to figure out what the new mean estimation was doing, you stopped and explained it and realised how unusual it might look at first. So many teachers lack this ability to go beyond what they know and imagine how certain formulas/concepts might need an extra minute or two of explanation for people who haven't seen it before. Subscribed!

[@vlavrenko]

Thank you! Very happy you find my videos helpful.

[@NickLilovich]

This video has (by far) the highest knowledge/time of any other video on this topic on KZbin. Clear explanation of the math and the iterative method, along with analogy to the simpler algorithm (k-means). Thanks Victor!

[@xinpang9611]

This explanation is fantastic! I have been studying machine learning courses in my master's but always found it difficult to understand. Now I finally understand EM. Thank you Prof. Lavrenko.

[@archismanghosh7283]

You just cleared every doubts on this topic, it's 10 days before my exam watching your video and getting everything cleared

[@xisnuutube]

Very nice explanation. But need to see your pointing..

[@sanjaykrish8719]

One lecture like this can uncomplicate things so much to so many people around the world. After understanding this 1D it is so much easy to get a grasp of higher dimensions.

[@roshinishake9400]

I want professors like you in my college. You're so great sir thank you so much ❤️ for your great explanation ☺️. Your video's making easy machine learning easy

[@drianhoward]

Dear Victor Your machine learning tutorial videos are really great!

[@joelharsten2408]

That was very good explained! It was nice that you often referred and compared to K-means. That made it easy to understand this algorithm! Thank you!

[@dhineshkumarr3182]

you really mastered the technique of explaining big things to a beginer.it was very helpful.I am definitely going to follow your lectures in future.Thank you so much for the knowledge.

[@erfandejband8945]

so simple explanation and at the same time so comprehensive. this video really help me to understanding this algorithem

[@azuriste8856]

Great Explanation Sir. I don't know why it motivated me to appreciate and comment on the video.

[@InvictusForever]

So helpful. Really lucky to have found this goldmine!!

[@ashokharnal]

So easily explained with superb clarity.

[@JD-rx8vq]

Wow, you explain very well, thank you! I was having a hard time understanding my professor's explanation in our class.

[@rayanaay5905]

Incredible, I read a lot of article and papers around EM. but this video gave me everything I need to know !

[@deepakjoshi7730]

Splendid. Example very well portrays the algorithm stepwise!

[@orresearch007]

Lucid explanation on EM. Mr Lavrenko is a superb teacher, explaining concepts using understandable language and helps learner to make sense of the equations. Keep up the good work.

[@theOceanMoon]

annotations really helped understand better. Thanx man

[@tchen8124]

You are the best teacher I ever had. Thank you so much.

[@westsidde]

I have tried to find good explanation on GMM for beginners (me) new to this topic and your explenation was the best i found. Very clear and good explanation, im very thankful sir! Keep up the good work!

[@Luckymator]

I can't thank you enough! You explain it so well, that i can now understand what the formuals in the script mean, i have to learn.

[@vitid1]

best explanation I've ever seen

[@ghilesdjebara8066]

OMG this is such simple and intuitive explanation. THANKS !

[@rishabhchopra6418]

Amazing! Was stuck with the Udacity Course! Now, its all clear! :)

[@lima073]

Best explanation i've seen about this subject. Thank you very much, your videos are really awesome !

[@mohamadkoohi-moghadam7657]

Awesome!! Obviously you are a teacher who knows the art of teaching! Thank you!

[@phuongdinh5836]

I wish you were my professor! I keep having to go to your channel after the expensive in-class lectures.

[@MrYoyo2345]

God bless you sir, and your excellent explanations. You are a life saver

[@rohanshah9593]

Amazing explanation! I was struggling to understand this concept. Thank you so much!

[@SyedArefinulHaque]

This is a perfect practical example, helped clear my concept about EM algorithm!!

[@SandraMenesesBarroso]

You are very good at teaching, you make it looks easy.

[@muhammadfaizanasghar77]

Brilliantly done, hats off.

[@playerseazay]

You explains really clear. Thanks for saving me from struggling in EM algorithm.

[@luisfernandocamarillo9071]

Congratulations Sir, magistral EM lecture.

[@sairaamvenkatraman5998]

You explain this so well! Awesome!!

[@ShahzadHassanBangash]

Beautifully explained. Keep it up professor

[@user-sl4sj2cs6w]

may I ask what is the value for P(b) in Estep, if I have two clusters, does that mean I can use 0.5 as Initial P(b) in estep?

[@fengjeremy7878]

Students that have such a great teacher make me jealous.

[@pudinda]

great explanation! the animations and the equations on the side, coming at the right time, really helped :)

[@uniqguy111]

This is gold standard

[@Artaxerxes.]

what a brilliant explanation

[@gregoire33]

hello, thanks for this video at the first iteration what value of p (b) do you use?

[@hasnainvohra7754]

Excellent explanation. Thank you.

[@vlavrenko]

Thanks!

[@m07hcn62]

This is awesome explanation. Thanks !

[@sagarmeisheri2361]

Thank you very much. Excellent Example !!

[@vlavrenko]

Thanks!

[@andriananicolaou4087]

Thank you very much for the presentation. I have one stupid question though. During the EM algorithm, when we apply the bayes rule to calculate the posterior p(b|x), how do we know the prior probabilities p(b) and p(a) (which are supposed to be equal)?

[@raymendoza8157]

I had the same question. Fishing through the comments it seems some have used 0.5 and 0.5 in this two-class example. So I guess our null assumptions would just be that they are equally likely for all classes. Not sure what happens after we go past the initial step. Did you figure out this? Hope this helps!

[@jamesschoi87]

so what are P(a) and P(b) initially?

[@schummelhase6055]

Thanks from Germany! Very very helpfull.

[@MrMopuri]

great work Mr. lavrenko!

[@muhmazabd]

Thanks a lot, Very well explained,

[@vlavrenko]

Thank you!

[@TheDionator]

Best explanation I've seen on the topic! Please make more!!

[@wadewang574]

I have a question : At 4:16, the formula of calcualting μ_b has the division by (b_1 + b_2 + ... + b_n), but my first idea is division by the number of data points, i.e. n, so why not divided by n ?

[@JiangXiang]

Excellent! Thanks for the effort to make it easy to understand!

[@vlavrenko]

Thanks! Glad to know these videos are useful.

[@DM-py7pj]

Probability and likelihood are used interchangeably here. When looking at P(xi | b), given the notation I assume the calculation is indeed a probability. I enjoyed this. I would like to have seen an explanation of stopping at a threshold for convergence.

[@mikelmenaba]

Great explanation mate, thanks!

[@raymendoza8157]

Got a question for anyone who can answer! I follow most of the calculations but I'm not sure how we get p(b) in the second equation (application of Bayes' Theorem). Or, for that matter how we calculate p(a) for the denominator. We can't estimate them from prior class membership samples since we haven't labeled anything. Any idea? Am I misunderstanding it?

[@ismailatadinc816]

Amazing explanation!

[@babyroo555]

incredible explanation!

[@UC46Vf8SyesF0MGwcZDowszA]

Excellent! Clean notation, very clear explanation. One question, are the priors the mean of the Gaussians?

[@TheChavakula]

Wow! the best explanation ever!

[@harshtrivedi6605]

+1. Thanks a lot .. I finally understood EM. And now it makes complete sense :) Would you mind citing a resource which proves that initial choice of gaussians don't make any difference. And the proof that these gaussians when keep shifting iterationwise how they always converge?. I guess it would be same as K-means, isn't it? Thanks.

[@pperez1224]

Excellent , the only one that i understood :-) Question : Why does it converge ? Does it has something to do with the central limit theorem?

[@mahoneyj2]

Great explanation and example! Many thanks!

[@stochasticneuron]

at [7:34] instead of mu_1 to mu_n it should be mu_a and mu_b for variance calculation if I am not wrong?

[@arnabsen8633]

yes. and he mentioned the mistake in the video.

[@sreichli]

Thank you for creating this video -- very helpful!

[@_Junkers]

Thank you for this, very valuable.

[@vlavrenko]

Thank you!

[@Luca-yy4zh]

Very good explanation. Thanks a lot!

[@HongjoKim]

Thank you very much. This video helps me a lot.

[@ahmedibrahim-lw1ut]

Great explanation. I have a question tho, about the values of xi. Why do we need to multiply those values to the posterior probability? What does the values of x really represent, if they are just points on the line why do we need to multiply them in the equation of recalculating the means?

[@brynkimura6738]

Thank you so much for posting this explanation!

[@DesmondCaulley]

Thanks man. Really really great explanations.

[@virgenalosveinte5915]

Amazing, thank you!

[@ventezcamilo]

Thanks! it is great to finally understand

[@ajayram198]

In 2:04 why is the posterior calcluated ? and what is the difference between p(x | a) , p(x |b ) and p( a | x) , p(b |x) ? which of these correponds to the probability that the data point belongs to cluster a /b ?

[@cumaliturkmenoglu2240]

wonderful lecture. But how to determine the number of gaussians(clusters) are behind the data?

[@NuclearSpinach]

I'm trying to fully connect the theoretical pieces that justify the weighted average of mu_a, mu_b, sigma_a, and sigma_b estimates. I.e., where is the E step and the M step and how do they connect to what's written? *Great* explanation. Thank you for uploading.

[@RyeCA]

excellent, thank you

[@charlibravo2578]

Thanks a lot for this

[@welovemusic9056]

Thank you very much. Well explained !!

[@Dominoanty]

Great Explanation ! Thank you !

[@G1aVoL]

Thank you for the amazing video.

[@shaimaamohamed8797]

wonderful explanation ! thanks a lot .

[@oTwOrDsNe]

Thank you for making this so clear and intuitive! Love your lectures

[@sameerjain6086]

When you refer to a point saying 'that' point there I have no visual indication on the screen.

[@vlavrenko]

Added the annotations, hope this helps.

[@thestuarts3815]

Hi Victor, Thank you a lot for these interesting videos. do you post your slides anywhere ? id love to keep a copy of everything i watch here.

[@tonyngo9400]

Great explanation, thank you!

[@maryamomrani155]

Thanks for such a good explanation!

[@vlavrenko]

Thank you!

[@CHANTI8947]

Video states that we are aware of the data points coming from 2 gaussians..GMM being unsupervised model, would we have that kind of information(# of Gaussians) beforehand..

[@aakankshachoudhary8532]

Thank you so much...This helped me a lot!!! Thanks a lot, again!

[@Chr0nalis]

It sounded as if you said: "we will discover the Gaussians automagically" :)

[@hoovie3000]

Great explanation!!

[@dionwang]

Hi, when you calculate the variance of a, does the nominator is a1(x1-ua)^2+...+an(xn-ua)^2?

[@conradsnowman]

I cant help but notice the middle dotted line looks like a logistic regression curve. I should know this.. But is there any relation?

[@TankNSSpank]

You are the best!

[@vlavrenko]

Thank you!

[@lorrainewang9929]

Such a nice video! Thanks a lot!

[@vlavrenko]

Thank you for the kind words. Good to know you find it useful.

[@8147333930]

thank you so much

[@damianoazzalini8573]

Very clear video. Thank you.

[@vlavrenko]

Thanks!