Gibbs sampling

[@jacobschultz7201]

Seeing the algorithm "walk" around the plane really made it click for me. Also, this made clear why we need to find every single conditional first. Thank you for the great work!

[@TheDebidatta]

This is the clearest explanation of Gibbs sampling out there. Thanks.

[@TheAIEpiphany]

The example with the 2D Gaussian was invaluable to ground my understanding - thank you!

[@jaradniemi]

Glad it was helpful!

[@tg6452]

Clear explanation, clear voice, clear slides. Good Job! Thanks!

[@jessicas2978]

Amazing video! I was struggling with it and now I understand. Thank you so much!

[@jiagengliu]

Just wondering if randomizing the order of sampling (like instead of going \theta_1, ... \theta_K, you do a random permutation of K) will help here? Is there a particular reason why we sample in this order?

[@atjebb]

Thanks for the great and clear video! One question though: starting at 3:11, why do the normal conditional distributions have mean of rho*theta and variance of [1-rho^2]? Where is this parameterization coming from?

[@hussein-m6x]

Explanation can be found from the conditional distributions section under the multivariate normal article on Wikipedia.

[@atjebb]

Thanks for the tip!

[@bunnysm]

Thank you very much! This is nicely and simply explained.

[@ccuuttww]

I think the vector theta N contains Theta 1 and Theta 2 is a marginal distribution of Bi variate Normal when we plot a histogram with Theta 1 data what it tell us ? The mean of Theta 1 is the value we want to estimate?

[@jaradniemi]

The histogram of theta1 draws (not data) is an approximation to the marginal distribution for theta1. The mean of theta1 draws are an estimate of the mean of the distribution for theta1.

[@ccuuttww]

@@jaradniemi Mean (theta1,theta2) gives us center? I still not really sure that what marginal distribution try to tell us In Normal distribution case we can calculate its mean of marginal density to estimate mu and std but in beta binomial cases the Y parameter is absent only given N=20 a=5 b=5 p=0.5 the mean of marginal distribution of Y =10 P =0.5 is it mean that the beta binomial posterior distribution equal to Beta(10+5,,20-10+5)?

[@jaradniemi]

@@ccuuttww Gibbs sampling is a methodology to sample from arbitrary target distributions by sampling from conditional distributions. When the target distribution is a known distribution, e.g. beta, then you do not need Gibbs sampling. Similarly Gibbs sampling is not necessary for a bivariate normal since we know everything about the distribution given the mean vector and covariance matrix.

[@jacobmoore8734]

Btw, how did you derive the conditional distributions from the joint? When I wrote out the full analytic form of the joint PDF divided by one of the single variable PDFs, the equation did not simplify easily.

[@jaradniemi]

You certainly can do it, but it is a bit of work. Fortunately many people have already done it and the formula is one of those formulas you are expected to memorize in a STAT PhD program. The formulas can be found on wikipedia under "conditional distributions of a multivariate normal": en.wikipedia.org/wiki/Multivariate_normal_distribution#Bivariate_case_2

[@jacobmoore8734]

​@@jaradniemi Oh I see, thanks! I've been trying to implement various MCMC samplers from scratch to get a better feel for their inner-workings. I find Metropolis and MH are a bit more accessible as all you need are ~proposal and target distributions. Gibbs seems fairly simple once you have the conditional distributions, but automating this process seems like a serious challenge, from an implementation perspective.

[@jaradniemi]

@@jacobmoore8734 A number of software packages have already done this, e.g. BUGS, JAGS. They basically have implemented all the known conditional distributions. For distributions that are not known, they generally use slice sampling. So that might be another MCMC sampler that you want to look into.

[@ZbiggySmall]

Thanks for the video. I understood the concept but I am not an expert in probability. I know what conditional probability is. I am still struggling to figure out what it means when you sample Theta_1 given Theta_2, ..., Theta_K etc. and would not be able to explain why it's working if someone asked me. In the example of p(Theta) ~ N(0, Sigma), Theta_1 ~ N(ro x Theta_2(0), [1 - ro^2]). Here the covariance matrix became variance [1 - ro^2] and mean became ro x Theta_2(0). Where does it comes from?

[@jaradniemi]

This requires knowledge of the multivariate normal distribution and condition distributions from this multivariate normal. See here: en.wikipedia.org/wiki/Multivariate_normal_distribution#Bivariate_case_2

[@GooseGood2024]

Gibbs sampling is just one special example of MH

[@UlrichArmel]

Are all the samples accepted contrary to metropolis Hastings algorithm? Cool video

[@jaradniemi]

Yes. Although Gibbs sampling can be written as a Metropolis-Hastings algorithm with an acceptance probability of 1.

[@Xnaarkhoo]

I have seen some paper and codes where they have used more complicated conditional probability. e.g sometimes, some pieces of their conditional probability contain gamma distribution. can you make some comment on how to advance a gibbs sampling ?

[@jaradniemi]

In the context used here, the Gibbs sampler is determined from the target joint distribution. Here the joint distribution is a multivariate normal and thus its conditional distributiosn are normal. For another joint distribution, it is entirely possible that the conditional distributions are gamma distributions.

[@philipruijten2639]

Great explenation! Not to nitpick but from 2.14 the f(\theta) should be p(\theta) or am I looking at it the wrong way ? If not, what's the f ? Nonetheless really awesome explenation! Thanks!

[@jaradniemi]

Philip Ruijten You are correct, it should be p(\theta).

[@shubhamtoshniwal2221]

Thanks for such a nice video !!

[@mingyanchu6308]

very good explanation!! really helpful, thanks:)

[@SrikantGadicherla]

If possible can you please share the link for ppts. Thanks!

[@shyamumich]

Very useful video. Thanks a lot.

[@weixuanli163]

Nice lecture. Pretty clear explanation

[@Carutsu]

I'm a bit too green on this so sorry, but how did you construct the conditioning? ie how did you go from theta_1|theta_2 to N(pTheta_1, i=p2) EDIT: should've researched before, this seems like a solved problem, just plug the numbers.

[@somalichaterji]

+Carutsu The more general derivation of Y|X=x is given in the following article: onlinecourses.science.psu.edu/stat414/node/118 Here \mu's are both 0 and \sigma's are both 1.

[@boshranabaei8283]

Thanks a lot. Very good explanation.

[@jacobmoore8734]

Anyone got a quick python function for this?

[@jaradniemi]

You might need to be more specific about what "this" is.

[@jacobmoore8734]

@@jaradniemi Sorry, I see how that could've been a bit too vague. Trying to recreate your work using python: stackoverflow.com/questions/62348946/gibbs-sampler-fails-to-converge

[@supermarcio_]

Thank you for this (:

[@Clover-ft4vx]

awesome :)

[@tadmeri]

very helpful!

[@sa1dana]

Thanks!

[@pooya97]

thanks

[@daryoushmehrtash7601]

thanks. q