How do I use the MultiIndex in pandas?

One of the most powerful features in pandas is multi-level indexing (or "hierarchical indexing"), which allows you to add extra dimensions to your Series or DataFrame objects. But when should you use a MultiIndex, and how do you create, slice, and merge MultiIndexed objects?
In this video, I'll demonstrate:
- How to create a Series with a MultiIndex, and how to convert it to a DataFrame
- How to select from a Series with a MultiIndex
- How to create a DataFrame with a MultiIndex
- How to select from a DataFrame with a MultiIndex
- How to merge two DataFrames with MultiIndexes
WANT TO JOIN MY NEXT WEBCAST? Become a member ($5/month):
/ dataschool
=== RELATED RESOURCES ===
Download the lesson notebook: nbviewer.jupyter.org/github/ju...
Using the pandas index (Part 1): • What do I need to know...
Using the pandas index (Part 2): • What do I need to know...
Analyzing groups with groupby: • When should I use a "g...
Selection and slicing with loc: • How do I select multip...
My full pandas video series: • Data analysis in Pytho...
DataCamp course on MultiIndex: www.datacamp.com/courses/mani...
DataCamp course on merging: www.datacamp.com/courses/merg...
Tidy data: r4ds.had.co.nz/tidy-data.html
== LET'S CONNECT! ==
Newsletter: www.dataschool.io/subscribe/
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@infiltr80r]

It's rare to find exactly what you're looking for when it comes to Pandas but this was it.

[@dataschool]

Great to hear! 😄

[@rezas2626]

Slice(None) trick was what I looked everywhere to find and found it here. Thank you so much!

[@dataschool]

You're very welcome! :)

[@joaoostrowski]

Hello from Hungary Kevin! After a period away from the technicalities of python working mostly with Google Analytics and CRO techniques I'm glad to be back to your videos playing around Jupyter. I appreciate a lot the work you do on Data School, so here is a bit of my honest appreciation, thank you and keep it up! John Ostrowski :)

[@dataschool]

Thanks so much John! :)

[@khardu]

Thank you for sharing your knowledge and experience beautifully. I am new to Python and Pandas, was struggling to understand how to select a subset of the data in a MultiIndex scenario. You made it a piece of cake by putting it all in one video, and by comparing and contrasting between Series and Data Frame.

[@dataschool]

That's awesome to hear! 🙌

[@annieyang4420]

Life saver video! i have been panic for 2 days , read many pieces and still confused. this helped me out! thank you!

[@dataschool]

You're very welcome!

[@carolinefawer-wasser2570]

As always great explaination. You are a gifted teacher, so surely well worth the Patreon membership for you.

[@dataschool]

Thank you, Caroline, both for your kind words and for your support through Data School Insiders! :)

[@stephenmaddock2863]

Brilliant tutorial. This is the 2nd of yours that I have watched and it covered what I need in my new job. Thank you very much.

[@dataschool]

You're very welcome!

[@searchingsoul5910]

Extremely informative, thanks for making it so simple - really good job

[@dataschool]

Thank you! 🙏

[@emuratspahic]

Great explanation! Even though I used pandas a lot, I never really understood how multi-indexing worked. Thank you so much for this tutorial, this helped me out a lot!

[@dataschool]

Glad it was helpful!

[@HD_157]

The most helpful video I've ever seen about Python. Thanks a lot!

[@dataschool]

Awesome! Glad it was helpful to you :)

[@carlomision9546]

Clear explanation with examples. This is exactly what I needed to know. You have yourself a new subscriber.

[@dataschool]

Awesome! Thanks :)

[@brockobama257]

I couldn't find the Corey Schafer video on MultiIndex so I'm here. And I'm not disappointed... subscribed!

[@dataschool]

Welcome to the channel! 👋

[@billy818]

ive been stuck trying to understand this reading tru the documentation for ages. 10 mins and i get it this vid is great thanks

[@dataschool]

Awesome to hear!

[@ssankarnarayan]

slice(None) instead of : was a very important input. I struggled with it for quite a while before watching this video. Thank you.

[@dataschool]

You're very welcome!

[@adamyork8407]

This was so cool! I am already thinking of a dozen ways to use multi-indexing in my newbie projects and adding them to my little presentation next week. I owe a lot of my comfort with pandas to you. I hope your holidays went well, Kevin!

[@dataschool]

Awesome, thanks so much for your kind comments, Adam! 👌

[@mahyarazad]

Thank you gazillion, you are the best! I finally understand the functionality of the Groupby method!

[@dataschool]

Great to hear!

[@shadrinan90]

Your lesson is sooo simple and clear. I found everything I wanted to know. Thank you

[@dataschool]

Great to hear!

[@Brocollipy]

Thanks so much for this video. Really great to see a range of options with such clear advice.

[@dataschool]

You're very welcome!

[@wiresandwavessolutions4723]

Thanks for making this multi-index topic so easy for all of us. I understood, pivot table perfectly after this video. I am having difficulty in the following : I want to select all Symbols and calculate the difference between the closing price of all SYMBOLs from 4th to 05th .

[@markcampanelli]

Thanks a lot. Including slice(None) was very helpful.

[@dataschool]

Great to hear - you're very welcome!

[@shabdarana2848]

Thanks. This topic had been bugging me but you make it seem really simple.

[@dataschool]

It is complex, but glad you are feeling good about it now!

[@Mo3azSolomon]

Thanks a lot for this simplified Explanation 😍

[@dataschool]

You're very welcome!

[@adarshrai22]

This is really a value adding video for data specialists. Thanks a lot for this brother.

[@dataschool]

Glad it was helpful!

[@shrishailgajbhar3191]

Thanks for helping us out with multiindexing in pandas...Cleared my daunting confusions about it.

[@dataschool]

Happy to help!

[@owenlie]

24 hours struggling to get the answer, and the answer just single line *unstack()* method. Thank you so much, I'll subscribe immediately!

[@dataschool]

Great to hear!

[@amitsardanayt]

You made this complicated taboo topic so simple to understand. Worth half an hour i spent

[@dataschool]

Thanks for your kind words!

[@mdfaiz4583]

u make things very simple....easy to understand ....thanks man

[@dataschool]

You're welcome!

[@mkoc2071]

Thanks for additional topics. Thanks for your time.

[@dataschool]

You're welcome!

[@raminsadeghnasab9310]

Finallllly, I understand how the multi-index works. Thanks man.

[@dataschool]

You're welcome!

[@minglee5164]

Thanks for the excellent tutorial, I have been confused by this issue for a long time.

[@dataschool]

Glad it helped!

[@MagicmathmandarinOrg]

Thanks for the thorough explanation. Very helpful.

[@dataschool]

Great to hear!

[@robertodilisio7535]

Hi Mark! Great video on a very Syntax confusing matter such as multiindex. Thank you very much!

[@dataschool]

You're very welcome!

[@mohammadkaifi]

This is the solution I was looking for. Thanks!

[@dataschool]

You're welcome!

[@kilocesar]

Great content man! Exactly what I needed!

[@dataschool]

Great to hear! 🙌

[@wildlife_and_drones1320]

Very helpful. I will be back with questions about machine learning.

[@dataschool]

Great! I've got a free course: courses.dataschool.io/introduction-to-machine-learning-with-scikit-learn

[@akashpoudel571]

u r the bestest ever teacher to me.......damn lucid ua guideliness......thanks a tonne sir

[@dataschool]

Thank you! :)

[@jsr7599]

Perfect video for what I needed. Thanks!

[@dataschool]

You're welcome!

[@jerryetc]

Extremely clear, thank you

[@dataschool]

You're welcome!

[@spiderschwein8g]

Saved me hours. Thanks!

[@dataschool]

Great to hear!

[@donaldbebel7979]

Very good explanation of a confusing topic. Thanks you.

[@dataschool]

Glad it was helpful to you!

[@pankajthakkar311]

This is awesome, thanks for making this Video Kevin.

[@dataschool]

You are very welcome!

[@ankitsajwan7520]

Exactly what I was looking for. Thanks :)

[@dataschool]

You're welcome!

[@DeebzFromThe90s]

Thank you sir. I will shoot you a donation if/when my quant journey presents some alpha. You are a lifesaver.

[@dataschool]

Thank you so much!

[@rubencrespo1720]

thank you so much, finally I could change from a Dataframe to a multiindex

[@dataschool]

You're very welcome!

[@iloveno3]

Thank you. I have learned so much from you already...

[@dataschool]

Great to hear!

[@ocamlmail]

Great explanation, very clear, thank you!

[@dataschool]

Thanks!

[@ahmadalrashedi5718]

man thank you soooooooooooooo much, I had been stuck on one thing for hours and finally figured it out!! !

[@dataschool]

You're so very welcome!

[@5a035]

19:50 "slice(None)", I never would have figured that out..... thank you!

[@dataschool]

You're welcome!

[@amitgaj1]

Thanks Kevin. That's brillant explanation.

[@dataschool]

Awesome! Thanks for your kind words :)

[@ramvpage]

Excellent Tutorial ! Thank You!

[@dataschool]

You're welcome!

[@mxmmalinin]

Thank you for sharing your knowledge!

[@dataschool]

My pleasure!

[@SFayed-lg5sp]

it's so helpful!!! thank you so much!

[@dataschool]

You're so welcome!

[@ItsRainingSteak]

Top shelf content right here

[@dataschool]

Thank you!

[@reibalachandran4775]

Thanks for the video, I found it very useful and enjoyable to watch :)

[@dataschool]

Great to hear!

[@CNW21]

As clear as day now , thank you

[@dataschool]

You're welcome!

[@willd0g]

Excellent - yes a few 💡 moments indeed. Very good

[@dataschool]

Great to hear!

[@florianrudowable]

pandas is awesome and you are too! I cant belive that pandas is free and this video is free. Thank you.

[@dataschool]

Thank you so much! 😄

[@olenamelnyk2427]

You saved my life with slice(None) (19:45). Thank you so much.

[@dataschool]

You're welcome!

[@dmitku]

Very useful as always. Many thanks!

[@dataschool]

My pleasure!

[@jhonatangarcia3826]

Thanks man! You helped me a lot!

[@dataschool]

Great to hear!

[@krishj8011]

Awesome tutorial...thanks

[@dataschool]

You're welcome!

[@user-dp1fz8gr8c]

Thank you ! Your video is very helpful!

[@dataschool]

You're welcome!

[@szymonk.7237]

Thank you very much for this explanation ❤️

[@dataschool]

You're welcome 😊

[@ebena.k.8032]

Thank you so much. 'unstack()' has alleviated my headache.

[@dataschool]

You're welcome!

[@chancemeeting9756]

Super helpful, thanks!!

[@dataschool]

You're welcome!

[@uncommon_common_man]

great tutorial

[@dataschool]

Thank you!

[@marvinjoyceburgstededijk9749]

Great video and many thanks. Greetings from Amsterdam

[@dataschool]

Thanks for watching!

[@BookOfMorman]

This was great! Thank you!!!

[@dataschool]

You're so welcome!

[@KarlosEPM]

Great video! Thanks :)

[@dataschool]

You're welcome!

[@liamkholmes]

Amazing! Thank you

[@dataschool]

You're welcome!

[@pankushkukreja3101]

Thanks Alot, really a good series for pandas and easy to learn. I would request if you can make a video on Iteration in pandas like iterrows, itertuples and iteritems. Many Thanks In advance.

[@dataschool]

I think I cover it in this video: kzbin.info/www/bejne/eF7VaomrgJ1jms0

[@prathameshmahankal4180]

Thanks a lot for this video!

[@dataschool]

You're very welcome!

[@semaphoreLock]

this is excellent. thanks

[@dataschool]

You're welcome!

[@abcdefghi2650]

THANK YOU BILLION TIMES!!!!

[@dataschool]

You're very welcome!

[@dangantman5474]

unstack() + reset_index() for the win!

[@lathaprabakarg]

very nice.. so useful

[@dataschool]

Thanks!

[@zhenghaizhou6085]

Great video!

[@dataschool]

Thanks!

[@umeshdev6403]

Thank you. Nice explanation. how to get max values with the dates

[@srosalia]

Thank you!

[@gggganzo]

Well explined. Would be great a video about data visualization with multiindex data structures also

[@agaby18]

Thank you! You saved me...

[@dataschool]

Great to hear!

[@niva090]

Thank you, this helped me out so much! I have one question however, I hope you can answer for me here: I have a multi index with ID as the first and Day as the second. Now I want to select all ID's that has more than 6 days of data. How can I pass this logical statement to the selector?

[@philtoa334]

Very nice.

[@dataschool]

Thank you!

[@zennicliffzennicliff]

Thank you so much. You are such a talented teacher. Where can I find out how I can structurally learn courses from you? Thank you and greetings!

[@dataschool]

Thanks! This post might be helpful to you: www.dataschool.io/launch-your-data-science-career-with-python/

[@gowthamkumar6232]

Thanks!

[@dataschool]

You're welcome!

[@fredsarti1]

Why slice(None).... omg. Thank you for your amazing video! Substribed

[@dataschool]

Thanks!

[@alvaro8603]

GREAT GREAT GREAT VIDEO!!!

[@dataschool]

Thank you!

[@ramutillu5005]

Thanks alot. Very useful. Can u teach how to use multiple groupby in single expression.

[@adelsayadkooh8870]

thank you

[@dataschool]

You're welcome!

[@asdkop7818]

Nice video, anyway I would have liked a final section in which you could have explained the usage of logic over the selection on multi-indexes. Eg: what if I wanted to select data for each symbol and for each date except one? Is it possible to use the same logic as .loc with simple indexes? Thanks :)

[@dataschool]

Thanks for your suggestion!

[@coolzkabhijit]

Thanks for the video! I have a question. How to access a particular index without naming the index, like say without naming 'AAPL'

[@rayli2221]

Thank you for sharing this information. I would like to ask if I wanted to get this to do portfolio optimization so calculating expected returns and covariance on multi-index dataframe how would you set this up?

[@bhanuyadavyadav8870]

Great videos may I request to show how to use .iloc with multi-index

[@dataschool]

Thanks for your suggestion!

[@mountainscott5274]

Would you normally go through all of this? It seems like one can just stick to groupby() without complicating the syntax for selection or changing the shape of the dataframe. In other words, have you encountered multi-indexing often in the field?

[@dataschool]

Great question! Personally, I try not to use the MultiIndex very much. That being said, it is still useful to be able to work with it when you have to.

[@marklittlewood2418]

Nice presentation Kevin well done. How about accessing a range of dates for say apple, date:date ?

[@dataschool]

I'm not sure if that can be done using the colon, I'd have to check...

[@mponcardas94]

Wonderful!

[@dataschool]

Thanks!

[@mounah2182]

Thanks for the video but i have a question how to merge columns and indexes in order to not have blank spaces in headers in my html view ?

[@thimirabandara679]

Great video! is there a way to must index the column so there is an outer column and inner column? And if so how do you sort that? Because I know sort_column is not a callable function on the datafram

[@surenderp2863]

very good explanation in this vedio on multi-indexes, could you please explain, why do we need multi levels of columns and multilevels of row labels and again there is droplevel option, could you please explain on this.

[@dataschool]

The short answer is that you should use a MultiIndex if it helps you to represent the structure of the data.

[@ridthrahman6637]

How to use range selection (:) while inner and outer row labels are inside tuples? Use slice which probably did not allow range selection. Thank you!

[@dataschool]

Sorry, it's hard for me to say off-hand, good luck!