Drawing and Interpreting Heatmaps

[@statquest]

Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/

[@syednajeebashraf4101]

These are one of best Statistics presentations I have ever seen. Too much helpful.

[@KlizmaTime]

This is so good. I am not a statistician, but needed to understand what a heat map for RNAseq results meant in one of the papers I was reading. I have a better understanding now. Thank you!

[@statquest]

Glad it was helpful!

[@nikitrianta9896]

Thank you for making statistics look so simple!!! These are the best videos I've ever seen !

[@statquest]

Thank you! :)

[@zi3417]

I am going to watch all videos in this playlist. Very useful for a new grad student(also neural research). Thank you ;)

[@statquest]

Glad you like them!

[@felipegutierre7037]

I love when your videos results on my youtube searches! *.*

[@statquest]

BAM! :)

[@PriscilaSantos-vs2po]

I only discovered this channel yesterday. Thank you very much!!!!!!!!!!!

[@statquest]

Hooray! I'm glad you like the videos. :)

[@pedrovallim9170]

Nice presentation. Particularly because it shows how the choice of the parameter of evaluation of distances of a point to distinct groups affects the visualization of the maps of heat. I suggest exploring in another presentation what is the most appropriate selection method.

[@himalayanplanespotter2021]

Thanks Josh, helped me a lot in understanding heat map and hierarchical clustering. Greetings from Japan.

[@statquest]

Hooray!!! I'm glad to hear that the video was helpful. I will be visiting Japan in 2 months! I can't wait.

[@himalayanplanespotter2021]

That's awesome Josh! Hope you have a great time in Japan.

[@kwang-hwicho1298]

At. 1:23, you said the relative abundances are scaled on per gene basis. if so, should all genes have dark red and dark blue in their row?

[@statquest]

We "z-scale" all of these genes by subtracting the mean value (per gene) and dividing by the standard deviation (per gene). That said, this does not mean that all genes will have the same maximum and minimum values. For example, if gene "a" has these values: 100, 10, -10, -100, then the scaled values = 1.2, 0.1, -0.1, -1.2. In contrast, if gene 'b' has these values: 1, 1, -1, -1, then the scaled values = 0.9, 0.9, -0.9, -0.9. In this case, gene 'a' will have a darker blue and a darker red than gene 'b'.

[@kwang-hwicho1298]

@@statquest Thank you very much for your kind answer.

[@Mipetz38]

15:10 Is it just me or does comparing points to the cluster average results in a more neatly colored heatmap?

[@statquest]

I'm partial to the furthest.

[@mea97905]

Great effort was put in this! Thank you so much for sharing.

[@behnamassadi5885]

Thank you statquest, it was extremely helpful

[@statquest]

Glad it helped!

[@bizikur]

your videos are so clear and helpful! Thank you!

[@statquest]

Thanks for the complement! They are fun to make and it's nice to know people like them. :)

[@stillsummer]

This video is great! One teeny tiny part that can be improved is the name of genes and samples in 8:15. Using "gene #1" instead of "gene 1" could be misleading at the first glance since you call sample #1, #2, #3. However, it's just a tedious point for your future reference. Other than that, your video is flawless.

[@MegaGravitas]

it will be really great to get a dedicated episode for scaling data the way it is described in this video.

[@statquest]

I'll keep that in mind.

[@sumitpaliwal1540]

Thanks. You make these things look very simple.

[@songthanh896]

Thank you very much for your wonderful explanation

[@statquest]

Thanks!

[@sagek7949]

Thank you for such a lucid explanation!

[@statquest]

Glad it was helpful!

[@apulunuj]

The values used in clustering by the eucleadian distance are scaled read counts right

[@statquest]

Yep! :)

[@juanete69]

In another video you spoke about dividing by the lenght of the sequence and the dividing by the sum of the column counts. Is it better to do it like that or to normalize like you explain in this video?

[@statquest]

It really depends on your data.

[@debleenamukherjee2410]

This was so helpful! Thank you so much!!!❤

[@statquest]

Thanks!

[@zudma]

The start was really clear and helpful, but then on the second heatmap i instantly got lost, couldn't even tell what each row was, i wish you'd given some more references for it, and made it smaller, and maybe gave one of two examples of interpreting. That said, thanks for the video, it was helpful!

[@statquest]

I'm sorry you had trouble following the concepts in this video.

[@snehavalabailu]

Burst out at 12.30 😂😂😂 Can't believe stats made me laugh, I used to oly cry bcz of stats before 😂 you are goooood

[@statquest]

Hooray!!! :)

[@akhilmahajan1417]

Thanks Josh! I love your videos.

[@statquest]

Thank you so much!!!! :)

[@addisonmcghee9190]

Hi Josh, Just a quick clarifying question: When you say "sample", do you mean a collection of observations? I'm trying to conceptualize how the data is stored. Is each sample a single participant or is it a vector?

[@statquest]

Unfortunately I was relatively sloppy when I used the word "sample" in this video. At 0:32 I say that a "sample" refers to a column of measurements (the rows are individual genes, or measurements takin within each sample.). To clarify, imagine a "sample" being a single person and on each person we measure weight, height and age. That said, usually the word "sample" refers to a bunch of people or a bunch of whatever.

[@CWunderA]

Fantastic one-stop-shop for clustering basics!

[@statquest]

Hooray!!!

[@lkhmaj]

Hey Josh, I started to follow you from France and I find your videos really top. However, I have a small question: when you say for each sample, do you mean each row? thanks :)

[@statquest]

Unfortunately I am very sloppy when I say "sample" - in earlier videos I use the word to indicate a row of data, but it also means a collection of rows... In newer videos I call a row of data an "observation" and a collection of observations a "dataset". I wish I did that from the beginning... oh well. :(

[@andrzejzielezinski1489]

OMG!!:D In my 300 years+ as a vampire I've never seen such an excellent and clean explanation of heatmaps and hierarchical clustering. Thank you - you are a great teacher! Is there any chance you could share the PowerPoint presentation? (I'm okay if you say no - it's a lot of hard work)

[@andrzejzielezinski1489]

Thank you very much, Joshua. I'll just use the slides to explain heatmaps to a small group of my non-English speaking friends. Of course I will recommend your channel and the website. Please keep up this great work!

[@sandeshacharya553]

Hi! Quick Question: When we are scaling data, do we scale them by samples or by genes? Or do we find mean and standard deviation of all the genes (expressed in all samples) and scale them all at once? The second makes more sense to me. Would be glad if you could answer it. :)

[@sandeshacharya553]

If we scale the data per gene, can it be used for the comparison of expression among different genes in a single sample? Similarly, if we scale the data per sample, can we use to compare expression patterns in different samples?

[@statquest]

The scaling can go either way. The goal is to use something that makes the relationships clear. Sometimes that means scaling by gene, other times that means scaling by sample, and you can also do a global scaling.

[@ComputerScienceMaster]

Great tutorial... Love it

[@statquest]

Thank you! :)

[@257Silvia]

Such a great explanation!

[@statquest]

Thank you! :)

[@leixiao169]

Thanks for the great lecture! For hierarchical clustering in R, do we only need to provide a count matrix of the log transformed normalized counts of all samples as input (generated from Rsubread or DESeq)? The hierarchical clustering script will calculate the Z-scaling based on the input count matrix, am I right?

[@statquest]

I believe so.

[@danieldimitrov3146]

Great video! Thank you so much!

[@rebeccaeliscu3460]

Does applying a log transformation to the count data accomplish the same thing as normalizing the data across genes?

[@statquest]

Great question! The answer is that although the log transformation can make the data easier to look at (by preventing the highly expressed genes from making it impossible to see differences in moderately expressed genes), it's not the same as normalizing - it doesn't center the mean of the measurements on 0 and it doesn't standardize the range of values.

[@rebeccaeliscu3460]

@@statquest Is it then appropriate to normalize the rlog data? Thanks for your response!

[@statquest]

@@rebeccaeliscu3460 Yes. However, there are no fixed rules for how to transform the data - the goal is to have something that is relatively easy to interpret, and that means there is a lot of flexibility. Try stuff and see what works best.

[@mohammedmonzurmorshed2896]

thanks so much..you are making my life easier

[@statquest]

Hooray!!!

[@worldofinformation815]

Thank you Sir 🌹✨

[@statquest]

:)

[@factsfigures2740]

Sir , i understood why local scaling is helpful in case of an outlier in data. But can you please tell me how its done or any popular name of local scaling method which i can google and understand. Nonetheless , Thanks alot.

[@statquest]

Z scaling

[@ScubaTanksMusic]

Hi Josh, huge subscriber, Love you bro, Trying to clarify what kind of data are these numbers. Is it like, the "presence" of each gene in a cell? And what does that tell us? What are we looking for? I know these are big questions. You can just provide an example. My stats background is based mostly from linear regression, where samples are rows and variables are columns, but it always seems like you're rows and columns are reversed.

[@statquest]

The data in these examples are the number of mRNA transcripts per gene in a pool of cells. This number roughly correlates with how "active" a gene is in a cell. For example, in liver cells, genes associated with liver functions have relatively large numbers of mRNA transcripts associated with them. In contrast, in skin cells, genes associated skin function (like pigments etc.) have a relatively large number mRNA transcripts associated with them.

[@Tiago211287]

Can I make heat-map of only differential expressed genes? Or the right thing to do is doing of all expressed genes?. I did a heat-map, and, the differential expressed genes are mix with not differential expressed genes. Some rows and columns are bad to look at because some outliers.

[@Tiago211287]

+Joshua Starmer Thanks.

[@stillsummer]

One more follow-up question about heatmaps. If I have multiple groups of samples. Let's say Treatment 1, Treatment 2, and Control. Would you suggest us to make a heatmap based on differentially expressed genes in "Treatment 1 vs Control" or "Treatment 2 vs Control"? I am asking because those DE genes in the first comparison will not be the same as those in the second comparison. In this case, if you have to make a heatmap to describe your results, would you pick the top 100 genes with highest reads instead of top 100 DE genes? Does that make any sense?

[@stillsummer]

Hi Joshua, Thank you for your detail explanation! I aimed to discover the differentially expressed genes between different treatment groups so I think the second option would be more suitable for that purpose. Now I just have to figure out hoe to color code the heatmap...which would take a while. Thanks again :)

[@fidelialau]

Thank you so much! Your video helped a lot.

[@quickbiology5159]

hey can you make a video on how would i arrange data in a way that i would prepare heatmap of genes which are mostly expressed in all samples

[@statquest]

I'll keep that in mind.

[@bukhari3836]

it was really informative.

[@statquest]

Thanks! :)

[@uzmazehra9372]

Can you please do a video on PERMANOVA. I find your videos easy to understand

[@lifeandbeyond7279]

very nice video...thanks

[@statquest]

:)

[@ISK_VAGR]

That is crazy good. Now I have to mention. That I don't believe that one should go with the one that looks good. There is always a reason to choose one or other method. Rationalizing why using centroids for example will provide more robustness to the result because outliers is a better argument if you want to keep a conservative interpretation. Another is using different clustering methods and see those that converge in similar results would be advisable. At least that is my opinion. In any case super good statquest

[@statquest]

Glad you like the video. However, regardless of how you justify your method, you will always end up with some relatively arbitrary cut off because there is no way to prove that you've made the correct decision.

[@jonathanz4322]

nobody: StatQuest: statquest, statqueeessst .............. STATQUEST!

[@statquest]

:)

[@munaalhammadi4237]

Thank you for the nice presentation and great explanation. Is global scaling similar to scaling by column?

[@statquest]

No, it uses all of the columns to find a single scaling factor for all of the data instead of finding one scaling factor per column.

[@munaalhammadi4237]

@@statquest What is the function that performs global scaling in R?

[@statquest]

@@munaalhammadi4237 x

[@aftabnadim]

Thanks, Josh, could you please share with us a tutorial about how to draw heatmap in R-Studio or any other online database for the RNA Seq data.

[@statquest]

I'll keep that in mind.

[@marble2725]

made me say "OHHHHHH!!" out loud, 10/10

[@statquest]

BAM! :)

[@upalabdhadey6156]

@@statquest Double BAM :> :>

[@debajitbhowmick7079]

Will it be better if we use median than mean? 4 min 45 sec.

[@statquest]

The decision is arbitrary. Try them both and see what looks better.

[@debajitbhowmick7079]

@@statquest well I love to see but do not know R or any comp language. But it is good to know.

[@mikhailalexeyev636]

Thank you!

[@statquest]

Thanks!

[@quickbiology5159]

You are amazing ,,,,,,how you do it???? THANK YOU

[@statquest]

Thanks!

[@zaheral-masqari4827]

thx a lot for this rich explain and I wish if you have videos for PCOA, LDA, alpha diversity and R language

[@statquest]

I have videos on PCoA and LDA. Here's a link to all of my videos: statquest.org/video-index/

[@elaherezaee5672]

thank you very helpful

[@statquest]

Thanks!

[@창튜브-h3d]

thank you so much

[@statquest]

Welcome 😊!

[@drajaygupta7908]

wonderful

[@statquest]

Thank you!

[@PraveenKumar-pd9sx]

Can you pls do a video on genetic algorithm

[@苏峰-p6v]

a great vedio,help me lot.

[@statquest]

Hooray! :)

[@steveb.s.baleba5944]

Can you share with us the R code ???

[@adelutzaification]

Kick it up a notch! :)

[@adelutzaification]

I am somewhat surprised and a tad vexed that, in doing clustering, there are so many choices in terms of calculating the distance and choosing the method of clustering. It's very... empirical... I somehow expected that certain parameters would fit best certain scenarios/ types of problems . Has anybody checked systematically the various permutations of parameters for "best fitting" of a problem (say gene expression). Perhaps, the scenarios need to be better defined in order to find this correspondence. It seems that there are cases in which a certain combination of parameters fit best the essence of the system , according to the researcher's experience. Maybe scrutinizing such cases can reveal a certain feature of the system that is not encountered in seemingly similar cases for which the combination doesn't work so well...? Maybe some empirical "constant" a la theorems encountered in physics can do the trick? ;)

[@adelutzaification]

Gotcha. Each of these techniques in just another weapon in our arsenal. Hypothesis testing through experimentation is the best. But there is also model testing in statistics, isn't it. Can you make a video about that? Methods, parameters, best practices Thank you.

[@adelutzaification]

Thanks. I also meant to reply to your message on Statquest.org but I can't find it any more to save my life... Today I learned about about "lasso regression" . Apparently, it can "do variable selection and prediction" which seems like the bee's knees. Can you expand on that in a video, por favor? Here's something you might enjoy: another "Bam" aficionado ( kzbin.info/www/bejne/jqfEq4SLjqxkoac)

[@zeebazahrasultana9368]

Great for a beginner 😁

[@statquest]

Thanks! :)

[@danielaquijano5872]

I love you thank you

[@statquest]

:)

[@debajitbhowmick7079]

Greatdiscussion

[@statquest]

Thanks!

[@RyonBang]

its hard to be a scientist fck!! i was thinking what if i shouldve gone to medical school instead

[@majs7037]

strijderssss

[@statquest]

?

[@taotaotan5671]

Josh runs out songs...

[@statquest]

a sad day. :(

[@chad_the_stud]

horrible video quality, irreleavant choice of example

[@statquest]

Sorry about that.