When should I use a "groupby" in pandas?

The pandas "groupby" method allows you to split a DataFrame into groups, apply a function to each group independently, and then combine the results back together. This is called the "split-apply-combine" pattern, and is a powerful tool for analyzing data across different categories. In this video, I'll explain when you should use a groupby and then demonstrate its flexibility using four different examples.
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/jus...
"groupby" documentation: pandas.pydata.o...
"agg" documentation: pandas.pydata.o...
"plot" documentation: pandas.pydata.o...
== LET'S CONNECT! ==
Newsletter: www.dataschool...
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@coolfooly]

Seriously this was an amazing explanation. I cant like it enough.

[@dataschool]

Thanks! :)

[@MagnusAnand]

It is indeed

[@soumenhalder4831]

Me a Ph.D. student in experimental high energy physics getting spirit from all these tricks, and avoiding some boring tasks. Thank you very much, sir.

[@dataschool]

You're very welcome!

[@krishnendusaha5940]

Great video. It's better than many of the so- called great courses on the internet! Cheers!

[@dataschool]

Awesome, thank you! :)

[@mkaberli]

You’re my go to person for informative videos on pandas, Numpy, and Matplotlib.

[@dataschool]

Thanks!

[@mitchellyula4447]

Thank you so much! I never used the groupby function until now and this helped me complete my pandas assignment! very clear explanation and your videos have helped me a ton!

[@dataschool]

Glad it helped! 🙌

[@shahzan525]

Your content is amazing , I visited some others channel , they are not taught like you, I see this video in suggestions & I learn perfectly how to use groupsby and where to use ... Thanks man........

[@diegodesouza302]

ok...i just left my afforded course from edx to watch only Data School videos...no regreted!!!! The way this guy explain should be a pattern to all teaches in the world! Love from Brazil!!!

[@dataschool]

Thank you so much!

[@marcinzaremba1811]

That is exactly what I was looking for and couldn't find at the same time. Thank you!!!

[@dataschool]

You are welcome!

[@uguree]

I will start your Sci-kit series too as I enjoyed so much watching all these pandas series as very clear.

[@arashchitgar7445]

WOW... I wish all the tutorials were that clear. Thanks man!

[@dataschool]

You are too kind! 😄

[@uguree]

Corey and Kevin you are brilliant. Thanks Corey as he advertised your channel, so pleased. It's like here go how to make a you tube content to teach

[@robhubert8350]

Nice job with the explanation. I combed through dozens of videos and articles online specific to leveraging groupby and your's by far is the best.

[@dataschool]

Thanks very much for your kind words!

[@fezulhasan3647]

Man you are amazing there is no one who can explain better than you. Love from India buddy.

[@dataschool]

Thank you so much!

[@muyideenjimoh6672]

First video I clicked on and it answered all my confusion. Many thanks.

[@dataschool]

Great to hear!

[@robertobarriosduran2622]

Even for my bad english level, this explanation was so clear. Big Thanks!

[@dataschool]

Great to hear!

[@BigAC7]

This is great! I'm taking an absolutely hateful pandas course that's basically a needle-in-a-haystack scavenger hunt. Faced with a .groupby() problem where the course materials in no way logically lead to the answer, I wound up here. The problem doesn't precisely conform to your tutorial, but just three of your lines were enough to convey the basic mechanism of action that led to an answer. Thanks!

[@dataschool]

Awesome to hear!

[@nitishkumar-bk8kd]

the best way of explaining pandas ,big fan for your teaching

[@asadtanvir4065]

Thank you Man! You are doing a great job here by showing Pandas tutorials in such easy way. Pray that you can carry out this benevolent work more

[@dataschool]

Thanks very much for your kind words!

[@dipakshah1008]

You explain pandas very well! Pls keep posting ...would love to see something on how to apply custom functions to tables (avoiding iteration use)

[@ChartExplorers]

Hi, it sounds like what you are looking for is vectorization. I would recommend reading up on numpy where() and numpy select(). I looked through Kevin's (data school's) videos and didn't see anything on this topic. He is amazing at explaining things, in the mean time you many be interested in this video where I talk about vectorizing a little bit at the end of the video here: kzbin.info/www/bejne/eXiWdoltiqd4gqM

[@ruslanzakharov4218]

It was really useful for me. I don't know english well, but I understood almost all words in this video

[@dataschool]

That's awesome to hear! 🙌

[@redcat7467]

Man, you got one hell of a channel. Thanks!

[@dataschool]

I appreciate that!

[@TexaSol2020]

excellent description. Much better than many on LinkedIn.

[@dataschool]

Thanks!

[@janaal-sokhon2462]

Thank you so much for all of your videos that explain pandas. You seriously explain better than my teachers. THANK YOU

[@nelsonkayode106]

This is the best groupby video I've seen. Thank you for your commitment. You just gained a subscriber.

[@dataschool]

Thank you so much!

[@fawadkhan8905]

How nice of you to come up with such great videos. Thanks

[@dataschool]

Thanks for your kind words!

[@josephvargas6756]

You sir, are a Gentleman and a Scholar. Thanks for your videos.

[@dataschool]

Thanks!

[@cyruswpl]

This is really helpful and easy to understand! Looking forward to your future great works, keep it up.

[@dataschool]

Great to hear!

[@daudujonnie]

You are simply fabulous. I think you deserve a grand master award.

[@dataschool]

Wow, thank you!

[@Kristina_Tsoy]

Amazing video as always, Kevin! Thank you for great tip at 03:48

[@dataschool]

My pleasure!

[@julieye2260]

Thank you so much! This video is very helpful and great quality! I appreciate your speaking speed; it is definitely friendly for foreigners to follow.

[@sulaimankhan8033]

Your'e rate of speech is inline with your coding..., I appreciate it.. thanks

[@dataschool]

You're welcome!

[@angelfernandez4235]

Thanks!

[@dataschool]

Thank you so much! That is really kind of you! 🙏

[@rafaelzanfolin7745]

Thank you sir! Eu really saved my life, hugs from Brazil!!

[@dataschool]

Glad it helped!

[@davebeckham5429]

Excellent tutorials as always Kevin. Thanks for sharing.

[@dataschool]

Thank you!

[@fadilyassin4597]

the way you teach is unique and good

[@dataschool]

Thank you!

[@astronauta8132]

man, your explanations are so good. keep it straight, keep it simple, keep it upPPP!!!!

[@dataschool]

Thanks very much for your kind words! :)

[@sarandam5125]

Thank you very much for an amazing explanation. I doubted .groupby method for a long time and finally found this video, This video totally clear my doubt out :D

[@dataschool]

You are very welcome! So glad it was helpful to you :) You might be interested in my latest video, which has some usage examples of groupby: kzbin.info/www/bejne/moHafZ-AoLh7gtE

[@panchopaulo111]

This video was just amazing!! You just got a new subscriber.

[@dataschool]

Thank you!

[@karmakast806]

oh my go.. such clarity in explanation.

[@dataschool]

Thanks! :)

[@weskitlasten417]

Thanks! I realize this is years old, but worth a shot... One thing I would love to see clearly explained is how to change values in one column based on a condition on values in another column. For example, if drinks.spirits > 100 and drinks.total_liters_of pure_alcohol >10, change drinks.continent (or other column) to "lush". As an addition, if values in one column (say drinks.name) contain the word "dirty" change drinks.briny to True. Etc

[@dataschool]

For your first example, the code would look something like this: drinks.loc[(drinks.spirits > 100) & (drinks.total_liters_of pure_alcohol >10), 'continent'] = 'lush' loc is super useful! Here's more information on loc: kzbin.info/www/bejne/rqfTf3Rtl6hrmdU

[@brandonwarfield5611]

Wow! You explained that so well that it sticks. Thank you!!

[@dataschool]

You're very welcome!

[@praveenudayakumar5128]

Good explanation, audio clarity and good English too thank u bro.

[@dataschool]

You're welcome!

[@majidm4215]

you are awesome, I have never watched that many videos in one day Thank you so much

[@scorpsallday33]

Are you fucking kidding me.... you sir are the man. These tutorials have been blessing me. Where have you been all my life 😂

[@dataschool]

Awesome! Thanks so much for your very kind comments :)

[@abhishek-hb1vg]

wooooooowowowowowoow this is what i felt when u used the matplotlib function. U r amazing.

[@dataschool]

Thanks!

[@parthraghuwanshi2929]

Man you really bring great content

[@dataschool]

Thank you!

[@ramakanthrayanchi8888]

Awesome. The tip at last would be very much helpful if we want to apply a function on all the numeric columns at once for each group independently.

[@СтепанЦыбин-ю9д]

I have already written before and want to say it again. Great lessons, thanks teacher!

[@dataschool]

You are so very welcome - thanks for your kind comment! 🙌

[@scottlucas3710]

Excellent job of explaining pretty complicated subject matter !

[@dataschool]

Thanks so much!

[@Mark-sf4kh]

Data School you guys have a slack channel or blog where one can post questions?

[@zohaibhasan5061]

I am a beginner to pandas. Amazing videos that were suggested by a friend. I have a doubt where I am stuck. For me in Python 3 drinks.groupby('continent').mean() does not work. min, max and count works but this does not (not without specifying 1 or more column names)

[@dataschool]

Excellent question! In the current version of pandas, if a DataFrame contains non-numeric data and you want to calculate the mean of all numeric columns after a groupby, you have to include the argument numeric_only=True. Thus: drinks.groupby('continent').mean(numeric_only=True). Hope that helps!

[@zohaibhasan5061]

@@dataschool Yes it sure helps!!! Thanks for pointing out this argument

[@rajpaul1501]

You are simply awesome. Need more and more of your tutorials.

[@dataschool]

Thank you!

[@fet1612]

Q&A Series: Video #14 _04(TC01:14) ========================================== drinks.beer_servings.mean() What if you wanted to know beer_servings across continents? You wanted to know, how it varied from Europe to Asia to Africa. So this is where a GROUPBY (groupby) is USEFUL. (TC01:34)

[@raveeshmalhotra7347]

greetings BROTHER, at 1:47 you have written command drinks.groupby('continent').beer_servings.mean() , is there another way of writing which includes df['beer_servings'] instead of writing in attribute format (.beer_servings)??

[@iampujan]

Easily understandable. Thanks for the video and keep making such videos.

[@dataschool]

You're very welcome!

[@johnnyrotten43]

You made that so easy. Thank you.

[@dataschool]

You're very welcome!

[@honprarules]

Earned a sub in 3 mins. Amazing work!

[@dataschool]

Thank you!

[@robindong3802]

Very clear and simple. great demo.

[@dataschool]

Thanks!

[@hakunamatata-qu7ft]

please do a video on all possible string operations possible in data preprocessing

[@dataschool]

Does this help? kzbin.info/www/bejne/mKDJknZmfsieftE

[@Jettiesburg]

Thanks, top video! Only suggestion would be to expand on this with some strategies for creating a new DF with the grouped/agg values. Thanks again

[@dataschool]

Thanks for your suggestion!

[@law5]

Fantastic explanation :D

[@dataschool]

Glad it was helpful!

[@andreazecchi812]

that's a fantastic explaination! Thanks so much!

[@dataschool]

You're very welcome!

[@hariaiyar1119]

Amazing explanation bro.

[@dataschool]

Thanks!

[@freeandeasy9795]

Nailed it! Exactly what I was looking for. Thanks

[@dataschool]

You're welcome!

[@kwphysics]

This was a useful video and I appreciate your explanation!

[@dataschool]

Glad it was helpful!

[@terrancedejesus625]

Fantastic video. Using query() is another great way to filter through data and add multiple filters. I have often found it to be less typing and alot more simple. Example: drinks.query("continent == 'Europe'").beer_servings.mean(). Thanks for the video!!

[@dataschool]

Thanks for your kind words! I personally don't use "query" because the pandas documentation lists it as "experimental", which tells me that they might change the API or eventually remove it: pandas.pydata.org/pandas-docs/stable/indexing.html#the-query-method-experimental However, functions can always change regardless of whether they are listed as "experimental", so I guess there's no harm in using it! :)

[@kleczekr]

You can also try to run the code: import seaborn as sb before running the code plotting the graph. The difference is really amazing (and you can see the length of bars for Europe and North America, which in the standard graph are behind the legend).

[@dataschool]

Great tip! Yes, the default plot aesthetics will change simply by importing Seaborn.

[@theodoretourneux5662]

fantastic! exactly what I needed clearly explained and organized. thank you!

[@dataschool]

You're welcome!

[@rohitjoshi1777]

Clear explanation please make video on enumerate function in pandas

[@dataschool]

Thanks for your suggestion!

[@magelauditore333]

U r awesome man. Pls Pls make numpy series for data science, stats

[@tarekkhalefa1836]

thanks for your simple explanation

[@dataschool]

You're welcome!

[@carolinegh601]

I wanna give you a hug! Thank you so much!

[@dataschool]

You are so kind! 😊

[@Dmitrycheburirtrie]

Best explanation !

[@dataschool]

Thank you!

[@khalidhassani6173]

agg is the best. thanks a lot

[@dataschool]

You're welcome!

[@brianbii1104]

I really enjoy ur video. i was wondering if you had any videos on how to do a report and if not then can u point to some resources that may have a way to explain it to me.

[@ranger52289]

Great explanation! How could you show the information for the country in each continent with, say, the highest beer servings?

[@tayebzlezel9222]

u r a life saver my friend

[@SarmitaMajumdar]

Amazing handy guide!!! I must thank you for the same! Wish to receive more in near future!!!

[@dataschool]

Glad it was helpful!

[@Stella-rs5qf]

Very great explanation

[@raghavendra4955]

Y there are less views🙃🙃.. Its an amazing explanation..

[@dataschool]

Thank you!

[@339059331]

Awesome as usual! thanks very much!

[@dataschool]

Thank you!

[@ManojSharmaX]

2:58 filter and group bu 5:35 agg

[@rin_645]

4:49 and dot aggggggggg~ hahaha 😂 But seriously tho, what an absurdly good explanation of groupby function. I wish you could be a professor in my university🤪

[@dataschool]

Thank you! 😄

[@Tony-rx2zj]

i recently noticed someone use this approach, drinks.groupby("continent").agg({"beer_servings":"mean"}) instead of drinks.groupby("continent").beer_servings.agg("mean") . Is it more preferable to use the first solution as it returns "beer_serving" label on column?

[@jakeb2553]

What do you recommend if I want to group by conditionals? Let's say I have a column "Blood Pressure" and I want to group by values 120.

[@harijayaram]

very clear example of groupby ...thanks

[@dataschool]

You're very welcome!

[@edilsonsousa3743]

Thank you for making this video, really helped!

[@dataschool]

Glad it helped!

[@RandomStuff-dw2bj]

how can we do different functions for multiple columns - mean for 2 columns and sum for other three columns? 2 question: how can we do linear regression for each group after group by ?

[@All_Things_Trucking]

Hey, thanks for the video on GroupBy. I am curious how you can use GroupBy in the context of quicker data processing of slicing and dicing. Rather than use a full dataset, use first groupby function to condense dataset and then work with that condensed dataset. Can you think about that scenario and elaborate how to do it in the most efficient way?

[@dataschool]

Sure, you can save the groupby object as then use that for future calculations!

[@RichardGreco]

Excellent tutorial! If possible maybe another video to refresh and/or to address Pandas 1.0.x?

[@dataschool]

Thanks for your suggestion!

[@AnantMr]

Thank you the detailed tutorial. However, I had one question @4:34, where we are finding the avg,max, min beer consumption continent wise, could we find found per continent which country has the MAX and MIN beer consumption. Sample output : Africa Angola 50.0

[@dataschool]

I can't think of how to do that off-hand, I'm sorry!

[@AnantMr]

@@dataschool Thank you for getting back. You're interested in viewing the process, I'll send you my repository link.

[@ricardooberdanpereira793]

I made using idxmax() to get a index from a row with greatest consumption and use it to filter the DataFrame with iloc: drinks.iloc[drinks.groupby('continent').beer_servings.idxmax()] In SQL is easy to do that. But starting in pandas can be painful

[@russnagel1]

Like many others have said. More great content, thank you!

[@tomrhee1]

Unfortunately, some of the values in the drinks data frame turned out to be NaN. I wonder if you could show us how to handle missing values to this. Thanks.

[@syntel13764]

i am a great fan of you .. you rocks !!

[@dataschool]

Thank you! 🙌

[@MrCraptakular]

Hello. Firstly, thank you for this magnificent video series. I really mean that! I have a question, continuing with your beer example. If you didn't have a continent column in the source data how would you accomplish the same output?

[@dataschool]

Glad you like the series! Regarding your question, the continent column has to be part of the DataFrame, otherwise you can't group by it. Hope that answers your question!

[@luisfernandoperezarmas7198]

Thanks for this video bro, great explanations.

[@dataschool]

Thanks!

[@mustafaadelamer2789]

Thank you so much You really made it very simple

[@dataschool]

That's great to hear! :)

[@RayedWahed]

Great Series! Can you please upload a couple of detailed videos on plotting graphs and graphically visualizing data to spot noise and to determine which features may be less important? Thank you

[@dataschool]

Thanks for the suggestion! I will eventually cover visualization in this series. Regarding feature selection, I'm unlikely to cover it in this video series, though I will cover it in an upcoming course. Make sure you are subscribed to my newsletter to hear about upcoming courses: www.dataschool.io/subscribe/

[@bobvance9519]

Could you use the "for each" formulation when you're grouping by multiple categories?

[@dataschool]

Great question! Yes, you would say "for each combination of these categories".

[@shiyunjiang8085]

Can I ask how to calculate the group by aggregation and only count unique number?Thanks ( BTW YOU ARE AMAZING)

[@dataschool]

Thanks for your kind words! As for your question, could you clarify? I don't quite understand. Thanks!

[@terngun]

This was helpful! Thanks for sharing.

[@dataschool]

Great to hear!

[@garywasherefirst]

Great explanation. Thank you!