What do I need to know about the pandas index? (Part 1)

The DataFrame index is core to the functionality of pandas, yet it's confusing to many users. In this video, I'll explain what the index is used for and why you might want to store your data in the index. I'll also demonstrate how to set and reset the index, and show how that affects the DataFrame's shape and contents.
SUBSCRIBE to learn data science with Python:
kzbin.info?su...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/justmarkham/pandas...
"loc" documentation: pandas.pydata.org/pandas-docs/...
"set_index" documentation: pandas.pydata.org/pandas-docs/...
"reset_index" documentation: pandas.pydata.org/pandas-docs/...
Indexing and selecting data: pandas.pydata.org/pandas-docs/...
== LET'S CONNECT! ==
Newsletter: www.dataschool.io/subscribe/
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@avantikasingh4576]

Thanks for being so effortless and keeping it to the base level .... it is more understandable!

[@MaryMacor]

You are so articulate and never give me a headache like most of the others

[@WanderPalateAdventure]

You made my life easy. I am new to this language word and was struggling to understand the concepts. Your videos just saved me.

[@BiranchiNarayanNayak]

Data School is the ultimate place to learn about Pandas. Thanks for the excellent tutorials on Pandas.

[@nataliaagudelo8635]

I am so glad I found your channel, always very thankful for you and your capacity to share your knowledge with us!

[@dataschool]

Thanks, Natalia! :)

[@nathanzorndorf8214]

Dude your videos are so f&*@n clear, they put the MOOCs I'm in to shame.

[@dataschool]

Ha! Thanks so much, I'm glad to hear!

[@anabelsa]

I totally agree with Nathan. I am following a MOOC from Harvard, and I have to come here to understand many things that remain totally obscure (at least for me) there. Thanks so much!!!

[@AsadAli-ck6fv]

I'm following an online data science course - and your material is far clearer - keep up your good work!

[@marinajin8797]

Best of the best, I can follow you step by step, which makes the learning process so efficient and fun. Thanks a MILL

[@dataschool]

Thank you! 😊

[@sararabiee8610]

You are an amazing teacher. I am new to this field and your videos are great help. Thanks a million 🙏🏽

[@dataschool]

You're very welcome! Thanks very much for your kind words! :)

[@tomasfanta9768]

Nice, clear and very helpful, especially "pd.DataFrame.reset_index" usage, thank you!

[@dataschool]

You're very welcome!

[@khanyasser]

wooow so glad I chose this video over others! So thorough and clear! Easy to follow! I was struggling and now my code is working. thank you.

[@dataschool]

Glad it helped!

[@adventuresunlimited4483]

I really appreciate the hard work you put in in your videos. I am taking a class in Python and your videos really helped me develop a finer understanding of what is being taught in my classes. These are the best videos for Pandas, I totally love them. I have subscribed and visit your channel almost everyday and each day I get what I was looking for. All the best to you. Keep up the good work.

[@dataschool]

Awesome! Thanks so much for your kind words! :)

[@armenlevonian5204]

Super clear and helpful!! I so agree with others who commented as such. Thank you!!

[@dataschool]

You're very welcome! It's exactly the kind of video I wish was available when I was learning pandas, and that's why I made it :)

[@niteshmaruthi8784]

Brilliant- This made the exact function of indexes very clear - I am binge watching these videos now!

[@dataschool]

Awesome! 🙌

[@sep69]

Thanks so for this video. It made me understand something I was struggling with. I'm going to watch you other video's right away :)

[@dataschool]

Awesome! Great to hear :)

[@zennologyofeverything7265]

THANK YOU! I just started with python datascience projects. My first project is the famous iris dataset. this helps a lot

[@dataschool]

You're very welcome!

[@akashpoudel571]

i dont understand who is that super human who dislikes your videos Sir.....i thinnk we are blessed u are giving your time to teach

[@dataschool]

Thanks very much for your kind words! 😊

[@aqilraza]

Excellent! Excellent! Excellent! way of explaining the concept of index - you rock!

[@dataschool]

Wow, thank you! :)

[@bingochipspass08]

Perfect explanation! Even the documentation doesn't do such a good job!

[@dataschool]

Thank you! 🙏

[@leonardoferigutti9039]

Excelent job!! It's amazing the quality of your job!! Thanks

[@dataschool]

Thank you very much!

[@aqilraza]

Now I understand how to use set_index(). Excellent way of explaining key concepts.

[@dataschool]

I'm glad to hear it was helpful to you!

[@susansilver1334]

Very clear. So good. Subscribed! I will gladly give you my time. You are a good teacher.

[@dataschool]

Awesome, thanks!

[@Honest_Reply900]

Thanks a lot for the detailed explanation. I appreciate you hard work and sharing your knowledge

[@dataschool]

Thanks!

[@rvma77]

Your videos are great! Thank you for sharing your knowledge.

[@dataschool]

Thanks so much!

[@farooq36901]

You are a fantastic teacher.

[@dataschool]

Thank you!

[@christat13]

Many, many thanks for all of your awesome videos. Super helpful, quick and easy to follow! Wish I found your site sooner, it would have saved me a lot of aggravation!

[@dataschool]

You're very welcome! Thanks for your kind words :)

[@reshaknarayan3944]

Excellent bro! Keep up the good work.

[@dataschool]

Thanks!

[@ossenaar]

Great videos series in pandas. For scalar selections (individual elements) in a large data frame, there is a faster alternative for using loc (labels based) and iloc (integer position based): at and iat.

[@dataschool]

I did not know that... thanks for sharing!

[@sunilarora1750]

Dude, you have best communication skills among Data Science Coaches

[@dataschool]

Thank you!

[@lukeemrose]

Great videos mate! Keep up the good work

[@dataschool]

Thanks! :)

[@ganxe]

Very nice, very clear, thank you bro. I subscribe!

[@dataschool]

Thanks for subscribing!

[@zes7215]

no such thing as cx or nx or not, any can be perfx

[@akramsystems]

reset_index() exactly what I was looking for to make a new set of indices in order of your current DataFrame :D thanks!

[@dataschool]

You're welcome!

[@datascienceds7965]

You deserve million subscription. I got an answer that I was looking for ages!!! reseting index to column. You saved my time. Thank you so much. I was wondering if you have done any percentage plotting. Sorry if it is a wrong way of asking !!

[@dataschool]

Thanks! I'm not sure what you mean by percentage plotting, sorry!

[@datascienceds7965]

No worries. I got answer that I was looking. Thanks for taking time to reply.

[@husseinshuriye6380]

Excellent Explanations, Thank bro

[@dataschool]

You're welcome!

[@rsiqueir2]

Thank you! Your video helped me so much!!

[@dataschool]

Awesome!

[@mahmudhasanmunna9314]

your video help me a lot. thank you

[@dataschool]

You're very welcome!

[@peterfrantz477]

What a great tutorial!

[@dataschool]

Thank you!

[@kanokpornpoomsirivilai7018]

thanks a lot. very thorough.

[@dataschool]

You're welcome!

[@slavrine]

It must be python 3, but drink.index shows up differently: RangeIndex(start=0, stop=193, step=1). As for the quality of your presentation, this should be a paid course! You're presenting the material so well! Would love to see you do more python courses - Matplotlib, etc

[@dataschool]

The index difference is due to the pandas version. Thanks for your very kind comment! I appreciate it!

[@BadriNathJK]

Please do videos on merge concatenate and split of data frames. That would be extremely helpful.

[@dataschool]

Thanks for the suggestion!

[@rajeshd3574]

@@dataschool Eagerly waiting for this video

[@nabazgharib]

Thank you for your videos. You are amazingly explaining the commands in pandas, this is what I was looking for. I hope you give a tutor on datetime datatype, and how to parse string to datetime and vice versa in pandas. and how to create a time series.

[@dataschool]

Thank you so much for your kind words! I have a video coming out on July 19 discussing the datetime format... I'll let you know when it comes out!

[@dataschool]

My video on the datetime format has been released, and should be helpful to you! kzbin.info/www/bejne/r3TKe3qpnJWLl5Y

[@redcat7467]

The tip is amazing.

[@dataschool]

Thanks!

[@rachellelancer3577]

Super helpful. Thanks so much

[@dataschool]

You're welcome!

[@PankajKumar-ot3mg]

super amazing teacher

[@dataschool]

Thanks so much!

[@vishalaggarwal8783]

When you are making the Data Visualisation series.Please do that ,it is highly required.

[@dataschool]

Thanks for the suggestion! I'm not sure when I'll be creating it.

[@dulangikanchana8237]

Hello awesome videos, can you kindly make some videos for numpy, and sklearn libraries, and k-means clustering, and supervised learning algorithms,R, and maybe Tensor Flow you are one of the best data science teachers at KZbin, Best of Luck, don't stop making videos.

[@dataschool]

Here are my scikit-learn videos: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@user-co8ki3ce5n]

Hello Dear, Respect for the the elegance , respect for the simplicity I am Egyptian arabic speaker with good english competence,I 've started to teach some pandas videos in arabic (learn data analysis from a beginner:), You are my role model , wonder if dataset links here is open source? can i use them in my videos? Regards

[@ramakanthrayanchi8888]

Excellent video !! Cheers

[@dataschool]

Thanks!

[@albertteplitsky7629]

Hello Kevin, I love your videos and they have helped me improve my programming skills to an entirely new level. I was wondering if you would be able to provide links or availability on how to access more complicated science-based data-sets because often the hardest thing to do is just find data sets with the necessary complexity to practice on. Continue the good work!!!

[@dataschool]

Glad I have been helpful! Regarding datasets for practice, I like this source: www.kaggle.com/datasets

[@albertteplitsky7629]

Data School thank you very much Kevin!! I greatly appreciate this I will be doing some fun projects with these 👍

[@dataschool]

You are very welcome! :)

[@yucelozyazgan]

Simply life saver videos for new programmers like me! How can i and others like me support you to keep you on this way ? i mean is there any way to make simple supports to you ? ( like tip box )

[@dataschool]

Thanks for your kind words, and for your generous offer! On my channel page, there is a blue button on the right side that says "Support", which allows you to make a donation of any size: kzbin.info Also, I have a PayPal link at the bottom of this page: www.dataschool.io/about/ Thanks again! :)

[@DimasAnggaFM]

this video is great great!!

[@dataschool]

Thanks!

[@Martin-dw4eo]

Thank you!

[@dataschool]

You're welcome!

[@MagnusAnand]

Great video

[@dataschool]

Thanks!

[@asneogy]

Hey Kevin! I think it would be important to emphasize here, when you converted the country to an Index, that the values in the index need to be unique (primary keys). Otherwise (does it give an error?) it totally ruins the purpose of identification / selection / alignment / merging. That way you could also reconnect to the unique() and nunique() methods as pre-requisites before setting the index. Awesome video as always.

[@dataschool]

Actually, index values do not need to be unique! In fact, there may be cases in which you desire a non-unique index, and you can still use it for identification etc.

[@asneogy]

thats very interesting! could you give an example where that would be useful?

[@dataschool]

Sure. Imagine a dataset of web server logs. Perhaps you want to use the IP address of the requesting computer as the index, in order to make it most convenient to filter the dataset by IP address. Or perhaps you want to use the datetime of requests as the index, even though it's possible for multiple requests to come in at the same time.

[@fet1612]

4:31 "Why does the index exist? And there are 3 main reasons." 1) identification 2) selection 3) alignment

[@karthikudupa5475]

you are awesome bro!!

[@dataschool]

Thanks!

[@baixartv]

Hello from Brazil :)

[@dataschool]

Hello from the USA! :)

[@waela3637]

thank you, I was stuck how to change the header of my dataset

[@dataschool]

You're welcome!

[@nimesharya909]

amazing, best explantion. do u have tutorials for video analytics, machine learning, tableau

[@dataschool]

Thanks! Here's my video series on machine learning with scikit-learn: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@dawjid6092]

Thanks for your video its really help, please i want to divide a huge data-set base on date snapshot for time-series neural network

[@dataschool]

I won't be able to help... good luck!

[@vipinmadathil]

Can you post a video on creating dataframe from dictionaries nested at multiple levels

[@TadeuszKurpiel]

time 11:10 - resetting the index IS POSSIBLE even when the index does not have the name - in that case the column created from index will be named 'level_0' - at least it works that way in pandas ver.2.0.3

[@dataschool]

Got it, thanks for letting me know!

[@s2kcolin780]

thank you

[@dataschool]

You're welcome!

[@sivakavin2872]

Addicted to your way of teaching brother #kevin

[@dataschool]

Thank you! 🙏

[@LiveCustoms]

How do you get values in the data frame when the index values are always changing? I.e every time I update my csv file the dates will be updated?

[@danielebaldoni2181]

Thank you for the great work. How coukd do if i have an array (suppose the score of a pca with 3 components) and i would select just the index of the elememts who respect a certain condition?

[@karimmanseri8619]

Hi, Thanks for your videos, Could you make a video how to sort values from two different data frames, then store them in new data frames. For example data frame 1 has 2 columns name of product and column 2 the country where is made data frame 2 has two columns which are as well product and country, but in this second data frame we will include similar few similar products and country as first data frame. could you write a script to iterate through dataframe 1 and 2 then find the identical product , then append it to data frame 3 along with country column. Thank you

[@haisonduong4726]

@,@ The Jupyter Notebook lesson is excellent. They're free and better than my not-free books. @,@

[@dataschool]

Thanks so much for your kind comment!

[@nagarjunanaidupayyavula4256]

Great stuff Dude...where can I get your's full vedios on python programming?

[@dataschool]

I don't have a video series on basic Python programming - is that what you are looking for?

[@Andorich77]

Let’s say you have column headers that contain dates and the actual data frame rows contains values specific to each date and you wish to plot each value for each date. Ultimately creating a line graph from all the values. X axis being the dates and y axis being the values. How would approach this. I’ve looked at variety of your videos but can’t seem to find a definite way of doing this.

[@dataschool]

Sorry, it's hard for me to understand precisely what you are trying to do... good luck!

[@sanmaymishra204]

can you share an example of importing nested json (as complicated as you can think of) into a pandas dataframe

[@dataschool]

I don't have an example handy, sorry!

[@vikaskulshreshtha]

Can you please tell me how to convert strings into numbers. Word to number system.

[@chengqian4350]

good vedio! just one suggestion. either remove the subtitle or make the code above the subtitle.

[@dataschool]

Thanks for your suggestion!

[@christopherlyons7613]

Thanks for the vids. What if I want to use a column as an index (like country) but I also want it to be a column in the dataset so the values can be accessed using standard access techniques (i.e. df[column], df.column, getting subsets of the data with [:], data use with .loc & .iloc, etc.). Is there a way to tell pandas to use a column as an index AND leave it as a column? Or would you have to copy the column to a new name to keep the data acessible?

[@dataschool]

Great question! A column of data is either a column (in the DataFrame) or its the index, but not both. Definitely don't make a copy and include it twice. It sounds to me like you should leave it in the DataFrame as a column... is there a particular reason that you need it to be the index?

[@diegrootam]

Hello from Brazil 8:56 :D

[@dataschool]

Hello to you as well!

[@amitunemelody5612]

sir how can i use the default index values(i.e.0,1,2,3,4......) as one my column values. For example: I have columns such as ID, Name,Faculty. And here i want, ID column to be filled with the values of default index values(i.e 0,1,2,3,4.....).Hope for the postive response.

[@kadiryasar2225]

Hi, thank you so much for this very informative video series. My question is: suppose that we re-index the dataframe but the new index column contains same values for some rows. I tried and pandas allowed to do it, but it does not make sense for me. If smthng is index for my data, it shall be unique. Thanks.

[@dataschool]

It is true that pandas does not require that the index is unique. I realize that might not make sense if you are using to using the index as a unique identifier. But in pandas, the index is just supposed to be an identifier, and the user can decide if they want it to be unique. In other words, pandas is providing the user with flexibility. Hope that explanation helps!

[@_proton5]

Silly question: How do I get the neat separation black lines in my table of the df? I only get one line below the column headers.

[@dataschool]

Depends on the version of Jupyter Notebook

[@saradag6939]

Hlo... Sr. How to convert rows into colomns nd colomns into rows in dataset

[@fizzakhan9497]

Sir kindly tell me [corrmat=dataset.corr() top_corr_features=corrmat.index] What does .index function works and how this code working

[@adritaanwar2279]

If I have a 3000×563 raw dataset without any column names and a list of 563 features. How can I name my columns using dataframe?

[@DavitBarbakadze]

Are you on Patreon? I'd personally pledge for such detailed explanations :)

[@dataschool]

Thanks so much for offering! No, I'm not currently on Patreon, though I will consider it for the future! However, I'd love it if you would join my email newsletter: www.dataschool.io/subscribe/

[@dataschool]

Hi Davit, I finally launched a Patreon campaign! I would love your support! www.patreon.com/dataschool/overview

[@Druzian]

How you can found the index number of a row?

[@bsemailaccount615]

HI, I have a question please: what is the difference between using delimiter and sep? Are they interchangeable? Thank you.

[@dataschool]

They are equivalent.

[@imapublicagent]

Great video, but I don't think you fully answered the question. The user specifically asked about multi-indexing. Is there any advantage to including "continent" as an additional index? More than one index seems to add confusion.

[@dataschool]

You're right, I chose not to cover MultiIndexes in order to limit the scope of the video. The short answer is that how you choose what to use as an index depends on how you expect to use the DataFrame. Most of the time, one index is best.

[@dataschool]

You might be interested in my new video about the pandas MultiIndex: kzbin.info/www/bejne/qpS1eJRoqNSWY8U

[@brendafosmire6519]

What happens when the column you set as the index has >1 instance of that value? Like what if there was more than one row for Country?

[@dataschool]

The pandas index does not have to be unique. Does that help?

[@pratimtalukdar618]

why do the drinks.columns return the column names appended with the character 'u'?

[@dataschool]

That indicates a Unicode string in Python 2.

[@anugantisuresh5935]

Am getting output like below: drinks.index RangeIndex(start=0, stop=193, step=1) how to get like your output?

[@hassanalkotob390]

Thanks a lot for this helpful series of videos if i want to query from drinks_df the top 5 countries with maximum wine_servings for each continent any helpful tips make me thankful

[@dataschool]

I can't think of how to do this off-hand, sorry!

[@hardikvegad3508]

Sir if we have hundreds of columns without the name. Then how can we name them using pandas and a for loop or lambda function because if we try to name them using names=[] it will be a very time-consuming process. The name of the columns can be col1, col2 , col3...etc.

[@Theasifaliamiri]

How do I compare a mean value for two values in a column, for example, check if the mean value of 'beer servings' for Europe and Asia are same or not?

[@dataschool]

I'm not sure I understand your question - Are you asking how to write an equality statement?

[@Theasifaliamiri]

Data School sir thank you for replying. Now I got the answer from one of your other videos. Love from Pakistan.

[@dataschool]

Great to hear!

[@anugantisuresh5935]

Have a question, drinks.head( ) getting nice table format but am not getting. How to get with table?

[@patelhastik7887]

hello, how can I fetch column name and column index form excel table using panda package??

[@dataschool]

This should help: pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html

[@ghanemimehdi1063]

Hello, i would like to know how to convert a string value in dataframe in float or int (labelisation) in order to visualize my data and can do classification Thanks in advance !

[@dataschool]

If you're asking about how to convert data types in pandas, I recommend this video: kzbin.info/www/bejne/jGGkiKywi7KZa5Y If you're asking how to convert non-numeric data into numeric data for use with a machine learning model, I recommend part 5 of this notebook: github.com/justmarkham/DAT8/blob/master/notebooks/10_linear_regression.ipynb Hope that helps!

[@ghanemimehdi1063]

Thank you very much !! :-) it helps me a lot

[@sayakpaul3152]

Could you explain ix operator?

[@dataschool]

I discuss it in this video: kzbin.info/www/bejne/rqfTf3Rtl6hrmdU Hope that helps!

[@sayakpaul3152]

Extremely sorry I commented without watching. Thanks.

[@dataschool]

Since you asked previously about ix, I just wanted to let you know that it has been deprecated... which I discuss in my newest video: kzbin.info/www/bejne/qpaYe6WJeLxggrs Enjoy!

[@FabioRBelotto]

Can I have a non unique values for index? What would happen?

[@dataschool]

Great question! That's usually not what you want to do, but a non-unique index is currently allowed.

[@zapy422]

What about multi-indexing?

[@dataschool]

I'll have a video about that coming out soon... make sure you are subscribed :)

[@vman049]

🔥

[@dataschool]

Thanks!

[@nandinivetal8569]

Need help for adding new row in dataframe with next character index. I want to add row in dataframe which have character index like a,b,c .Now I want to add next row in dataframe with index 'd'.I want append new row with append() having index 'd'.

[@dataschool]

I'm not sure the best way to do this, sorry!

[@nandinivetal8569]

@@dataschool i got the answer..Thanks for reply

[@peterabdou6169]

is it a must to make sure that pandas index has unique values?

[@dataschool]

Actually, the pandas index doesn't have to use unique values.

[@kostasnikoloutsos5172]

Can we change the location of a column?

[@dataschool]

Sure, there are many ways: stackoverflow.com/questions/13148429/how-to-change-the-order-of-dataframe-columns