What do I need to know about the pandas index? (Part 1)

[@RikkaTravelAdvisor]

You are so articulate and never give me a headache like most of the others

[@akashpoudel571]

i dont understand who is that super human who dislikes your videos Sir.....i thinnk we are blessed u are giving your time to teach

[@dataschool]

Thanks very much for your kind words! 😊

[@adventuresunlimited4483]

I really appreciate the hard work you put in in your videos. I am taking a class in Python and your videos really helped me develop a finer understanding of what is being taught in my classes. These are the best videos for Pandas, I totally love them. I have subscribed and visit your channel almost everyday and each day I get what I was looking for. All the best to you. Keep up the good work.

[@dataschool]

Awesome! Thanks so much for your kind words! :)

[@BiranchiNarayanNayak]

Data School is the ultimate place to learn about Pandas. Thanks for the excellent tutorials on Pandas.

[@nathanzorndorf8214]

Dude your videos are so f&*@n clear, they put the MOOCs I'm in to shame.

[@dataschool]

Ha! Thanks so much, I'm glad to hear!

[@anabelsa]

I totally agree with Nathan. I am following a MOOC from Harvard, and I have to come here to understand many things that remain totally obscure (at least for me) there. Thanks so much!!!

[@AsadAli-ck6fv]

I'm following an online data science course - and your material is far clearer - keep up your good work!

[@sunilarora1750]

Dude, you have best communication skills among Data Science Coaches

[@dataschool]

Thank you!

[@nataliaagudelo8635]

I am so glad I found your channel, always very thankful for you and your capacity to share your knowledge with us!

[@dataschool]

Thanks, Natalia! :)

[@avantikasingh4576]

Thanks for being so effortless and keeping it to the base level .... it is more understandable!

[@slavrine]

It must be python 3, but drink.index shows up differently: RangeIndex(start=0, stop=193, step=1). As for the quality of your presentation, this should be a paid course! You're presenting the material so well! Would love to see you do more python courses - Matplotlib, etc

[@dataschool]

The index difference is due to the pandas version. Thanks for your very kind comment! I appreciate it!

[@khanyasser]

wooow so glad I chose this video over others! So thorough and clear! Easy to follow! I was struggling and now my code is working. thank you.

[@dataschool]

Glad it helped!

[@WanderPalateAdventure]

You made my life easy. I am new to this language word and was struggling to understand the concepts. Your videos just saved me.

[@fet1612]

4:31 "Why does the index exist? And there are 3 main reasons." 1) identification 2) selection 3) alignment

[@BadriNathJK]

Please do videos on merge concatenate and split of data frames. That would be extremely helpful.

[@dataschool]

Thanks for the suggestion!

[@rajeshd3574]

@@dataschool Eagerly waiting for this video

[@marinajin8797]

Best of the best, I can follow you step by step, which makes the learning process so efficient and fun. Thanks a MILL

[@dataschool]

Thank you! 😊

[@niteshmaruthi8784]

Brilliant- This made the exact function of indexes very clear - I am binge watching these videos now!

[@dataschool]

Awesome! 🙌

[@sararabiee8610]

You are an amazing teacher. I am new to this field and your videos are great help. Thanks a million 🙏🏽

[@dataschool]

You're very welcome! Thanks very much for your kind words! :)

[@AhmedHamada-f5e]

Hello Dear, Respect for the the elegance , respect for the simplicity I am Egyptian arabic speaker with good english competence,I 've started to teach some pandas videos in arabic (learn data analysis from a beginner:), You are my role model , wonder if dataset links here is open source? can i use them in my videos? Regards

[@ossenaar]

Great videos series in pandas. For scalar selections (individual elements) in a large data frame, there is a faster alternative for using loc (labels based) and iloc (integer position based): at and iat.

[@dataschool]

I did not know that... thanks for sharing!

[@bingochipspass08]

Perfect explanation! Even the documentation doesn't do such a good job!

[@dataschool]

Thank you! 🙏

[@aqilraza]

Now I understand how to use set_index(). Excellent way of explaining key concepts.

[@dataschool]

I'm glad to hear it was helpful to you!

[@tomasfanta9768]

Nice, clear and very helpful, especially "pd.DataFrame.reset_index" usage, thank you!

[@dataschool]

You're very welcome!

[@Honest_Reply900]

Thanks a lot for the detailed explanation. I appreciate you hard work and sharing your knowledge

[@dataschool]

Thanks!

[@zennologyofeverything7265]

THANK YOU! I just started with python datascience projects. My first project is the famous iris dataset. this helps a lot

[@dataschool]

You're very welcome!

[@datascienceds7965]

You deserve million subscription. I got an answer that I was looking for ages!!! reseting index to column. You saved my time. Thank you so much. I was wondering if you have done any percentage plotting. Sorry if it is a wrong way of asking !!

[@dataschool]

Thanks! I'm not sure what you mean by percentage plotting, sorry!

[@datascienceds7965]

No worries. I got answer that I was looking. Thanks for taking time to reply.

[@vishalaggarwal8783]

When you are making the Data Visualisation series.Please do that ,it is highly required.

[@dataschool]

Thanks for the suggestion! I'm not sure when I'll be creating it.

[@aqilraza]

Excellent! Excellent! Excellent! way of explaining the concept of index - you rock!

[@dataschool]

Wow, thank you! :)

[@armenlevonian5204]

Super clear and helpful!! I so agree with others who commented as such. Thank you!!

[@dataschool]

You're very welcome! It's exactly the kind of video I wish was available when I was learning pandas, and that's why I made it :)

[@leonardoferigutti9039]

Excelent job!! It's amazing the quality of your job!! Thanks

[@dataschool]

Thank you very much!

[@susansilver1334]

Very clear. So good. Subscribed! I will gladly give you my time. You are a good teacher.

[@dataschool]

Awesome, thanks!

[@TadeuszKurpiel]

time 11:10 - resetting the index IS POSSIBLE even when the index does not have the name - in that case the column created from index will be named 'level_0' - at least it works that way in pandas ver.2.0.3

[@dataschool]

Got it, thanks for letting me know!

[@sep69]

Thanks so for this video. It made me understand something I was struggling with. I'm going to watch you other video's right away :)

[@dataschool]

Awesome! Great to hear :)

[@farooq36901]

You are a fantastic teacher.

[@dataschool]

Thank you!

[@christat13]

Many, many thanks for all of your awesome videos. Super helpful, quick and easy to follow! Wish I found your site sooner, it would have saved me a lot of aggravation!

[@dataschool]

You're very welcome! Thanks for your kind words :)

[@dulangikanchana8237]

Hello awesome videos, can you kindly make some videos for numpy, and sklearn libraries, and k-means clustering, and supervised learning algorithms,R, and maybe Tensor Flow you are one of the best data science teachers at KZbin, Best of Luck, don't stop making videos.

[@dataschool]

Here are my scikit-learn videos: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@nabazgharib]

Thank you for your videos. You are amazingly explaining the commands in pandas, this is what I was looking for. I hope you give a tutor on datetime datatype, and how to parse string to datetime and vice versa in pandas. and how to create a time series.

[@dataschool]

Thank you so much for your kind words! I have a video coming out on July 19 discussing the datetime format... I'll let you know when it comes out!

[@dataschool]

My video on the datetime format has been released, and should be helpful to you! kzbin.info/www/bejne/r3TKe3qpnJWLl5Y

[@akramsystems]

reset_index() exactly what I was looking for to make a new set of indices in order of your current DataFrame :D thanks!

[@dataschool]

You're welcome!

[@rvma77]

Your videos are great! Thank you for sharing your knowledge.

[@dataschool]

Thanks so much!

[@albertteplitsky7629]

Hello Kevin, I love your videos and they have helped me improve my programming skills to an entirely new level. I was wondering if you would be able to provide links or availability on how to access more complicated science-based data-sets because often the hardest thing to do is just find data sets with the necessary complexity to practice on. Continue the good work!!!

[@dataschool]

Glad I have been helpful! Regarding datasets for practice, I like this source: www.kaggle.com/datasets

[@albertteplitsky7629]

Data School thank you very much Kevin!! I greatly appreciate this I will be doing some fun projects with these 👍

[@dataschool]

You are very welcome! :)

[@asneogy]

Hey Kevin! I think it would be important to emphasize here, when you converted the country to an Index, that the values in the index need to be unique (primary keys). Otherwise (does it give an error?) it totally ruins the purpose of identification / selection / alignment / merging. That way you could also reconnect to the unique() and nunique() methods as pre-requisites before setting the index. Awesome video as always.

[@dataschool]

Actually, index values do not need to be unique! In fact, there may be cases in which you desire a non-unique index, and you can still use it for identification etc.

[@asneogy]

thats very interesting! could you give an example where that would be useful?

[@dataschool]

Sure. Imagine a dataset of web server logs. Perhaps you want to use the IP address of the requesting computer as the index, in order to make it most convenient to filter the dataset by IP address. Or perhaps you want to use the datetime of requests as the index, even though it's possible for multiple requests to come in at the same time.

[@LiveCustoms]

How do you get values in the data frame when the index values are always changing? I.e every time I update my csv file the dates will be updated?

[@peterfrantz477]

What a great tutorial!

[@dataschool]

Thank you!

[@ganxe]

Very nice, very clear, thank you bro. I subscribe!

[@dataschool]

Thanks for subscribing!

[@zes7215]

no such thing as cx or nx or not, any can be perfx

[@vipinmadathil]

Can you post a video on creating dataframe from dictionaries nested at multiple levels

[@waela3637]

thank you, I was stuck how to change the header of my dataset

[@dataschool]

You're welcome!

[@danielebaldoni2181]

Thank you for the great work. How coukd do if i have an array (suppose the score of a pca with 3 components) and i would select just the index of the elememts who respect a certain condition?

[@reshaknarayan3944]

Excellent bro! Keep up the good work.

[@dataschool]

Thanks!

[@mahmudhasanmunna9314]

your video help me a lot. thank you

[@dataschool]

You're very welcome!

[@christopherlyons7613]

Thanks for the vids. What if I want to use a column as an index (like country) but I also want it to be a column in the dataset so the values can be accessed using standard access techniques (i.e. df[column], df.column, getting subsets of the data with [:], data use with .loc & .iloc, etc.). Is there a way to tell pandas to use a column as an index AND leave it as a column? Or would you have to copy the column to a new name to keep the data acessible?

[@dataschool]

Great question! A column of data is either a column (in the DataFrame) or its the index, but not both. Definitely don't make a copy and include it twice. It sounds to me like you should leave it in the DataFrame as a column... is there a particular reason that you need it to be the index?

[@Andorich77]

Let’s say you have column headers that contain dates and the actual data frame rows contains values specific to each date and you wish to plot each value for each date. Ultimately creating a line graph from all the values. X axis being the dates and y axis being the values. How would approach this. I’ve looked at variety of your videos but can’t seem to find a definite way of doing this.

[@dataschool]

Sorry, it's hard for me to understand precisely what you are trying to do... good luck!

[@PankajKumar-ot3mg]

super amazing teacher

[@dataschool]

Thanks so much!

[@karimmanseri8619]

Hi, Thanks for your videos, Could you make a video how to sort values from two different data frames, then store them in new data frames. For example data frame 1 has 2 columns name of product and column 2 the country where is made data frame 2 has two columns which are as well product and country, but in this second data frame we will include similar few similar products and country as first data frame. could you write a script to iterate through dataframe 1 and 2 then find the identical product , then append it to data frame 3 along with country column. Thank you

[@husseinshuriye6380]

Excellent Explanations, Thank bro

[@dataschool]

You're welcome!

[@imapublicagent]

Great video, but I don't think you fully answered the question. The user specifically asked about multi-indexing. Is there any advantage to including "continent" as an additional index? More than one index seems to add confusion.

[@dataschool]

You're right, I chose not to cover MultiIndexes in order to limit the scope of the video. The short answer is that how you choose what to use as an index depends on how you expect to use the DataFrame. Most of the time, one index is best.

[@dataschool]

You might be interested in my new video about the pandas MultiIndex: kzbin.info/www/bejne/qpS1eJRoqNSWY8U

[@nagarjunanaidupayyavula4256]

Great stuff Dude...where can I get your's full vedios on python programming?

[@dataschool]

I don't have a video series on basic Python programming - is that what you are looking for?

[@lukeemrose]

Great videos mate! Keep up the good work

[@dataschool]

Thanks! :)

[@vikaskulshreshtha]

Can you please tell me how to convert strings into numbers. Word to number system.

[@kanokpornpoomsirivilai7018]

thanks a lot. very thorough.

[@dataschool]

You're welcome!

[@amitunemelody5612]

sir how can i use the default index values(i.e.0,1,2,3,4......) as one my column values. For example: I have columns such as ID, Name,Faculty. And here i want, ID column to be filled with the values of default index values(i.e 0,1,2,3,4.....).Hope for the postive response.

[@sanmaymishra204]

can you share an example of importing nested json (as complicated as you can think of) into a pandas dataframe

[@dataschool]

I don't have an example handy, sorry!

[@rsiqueir2]

Thank you! Your video helped me so much!!

[@dataschool]

Awesome!

[@yucelozyazgan]

Simply life saver videos for new programmers like me! How can i and others like me support you to keep you on this way ? i mean is there any way to make simple supports to you ? ( like tip box )

[@dataschool]

Thanks for your kind words, and for your generous offer! On my channel page, there is a blue button on the right side that says "Support", which allows you to make a donation of any size: kzbin.info Also, I have a PayPal link at the bottom of this page: www.dataschool.io/about/ Thanks again! :)

[@nimesharya909]

amazing, best explantion. do u have tutorials for video analytics, machine learning, tableau

[@dataschool]

Thanks! Here's my video series on machine learning with scikit-learn: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@baixartv]

Hello from Brazil :)

[@dataschool]

Hello from the USA! :)

[@rachellelancer3577]

Super helpful. Thanks so much

[@dataschool]

You're welcome!

[@redcat7467]

The tip is amazing.

[@dataschool]

Thanks!

[@Druzian]

How you can found the index number of a row?

[@fizzakhan9497]

Sir kindly tell me [corrmat=dataset.corr() top_corr_features=corrmat.index] What does .index function works and how this code working

[@hardikvegad3508]

Sir if we have hundreds of columns without the name. Then how can we name them using pandas and a for loop or lambda function because if we try to name them using names=[] it will be a very time-consuming process. The name of the columns can be col1, col2 , col3...etc.

[@adritaanwar2279]

If I have a 3000×563 raw dataset without any column names and a list of 563 features. How can I name my columns using dataframe?

[@peterabdou6169]

is it a must to make sure that pandas index has unique values?

[@dataschool]

Actually, the pandas index doesn't have to use unique values.

[@ramakanthrayanchi8888]

Excellent video !! Cheers

[@dataschool]

Thanks!

[@chengqian4350]

good vedio! just one suggestion. either remove the subtitle or make the code above the subtitle.

[@dataschool]

Thanks for your suggestion!

[@kadiryasar2225]

Hi, thank you so much for this very informative video series. My question is: suppose that we re-index the dataframe but the new index column contains same values for some rows. I tried and pandas allowed to do it, but it does not make sense for me. If smthng is index for my data, it shall be unique. Thanks.

[@dataschool]

It is true that pandas does not require that the index is unique. I realize that might not make sense if you are using to using the index as a unique identifier. But in pandas, the index is just supposed to be an identifier, and the user can decide if they want it to be unique. In other words, pandas is providing the user with flexibility. Hope that explanation helps!

[@_proton5]

Silly question: How do I get the neat separation black lines in my table of the df? I only get one line below the column headers.

[@dataschool]

Depends on the version of Jupyter Notebook

[@haisonduong4726]

@,@ The Jupyter Notebook lesson is excellent. They're free and better than my not-free books. @,@

[@dataschool]

Thanks so much for your kind comment!

[@saradag6939]

Hlo... Sr. How to convert rows into colomns nd colomns into rows in dataset

[@mohammedsakaria5324]

Hello, Why is it that when I try to make 'continent' as the index, it gives me the KeyError?

[@dataschool]

What code did you write that generated the error? This should work just fine: drinks.set_index('continent') Let me know!

[@acezhang2414]

Maybe you have already run it before without attention, and when you run it again consciously, it gives the KeyError.

[@bsemailaccount615]

HI, I have a question please: what is the difference between using delimiter and sep? Are they interchangeable? Thank you.

[@dataschool]

They are equivalent.

[@ASNPersonal]

Am getting output like below: drinks.index RangeIndex(start=0, stop=193, step=1) how to get like your output?

[@diegrootam]

Hello from Brazil 8:56 :D

[@dataschool]

Hello to you as well!

[@dawjid6092]

Thanks for your video its really help, please i want to divide a huge data-set base on date snapshot for time-series neural network

[@dataschool]

I won't be able to help... good luck!

[@ASNPersonal]

Have a question, drinks.head( ) getting nice table format but am not getting. How to get with table?

[@brendafosmire6519]

What happens when the column you set as the index has >1 instance of that value? Like what if there was more than one row for Country?

[@dataschool]

The pandas index does not have to be unique. Does that help?

[@sivakavin2872]

Addicted to your way of teaching brother #kevin

[@dataschool]

Thank you! 🙏

[@MagnusAnand]

Great video

[@dataschool]

Thanks!

[@Martin-dw4eo]

Thank you!

[@dataschool]

You're welcome!

[@karthikudupa5475]

you are awesome bro!!

[@dataschool]

Thanks!

[@pratimtalukdar618]

why do the drinks.columns return the column names appended with the character 'u'?

[@dataschool]

That indicates a Unicode string in Python 2.

[@DavitBarbakadze]

Are you on Patreon? I'd personally pledge for such detailed explanations :)

[@dataschool]

Thanks so much for offering! No, I'm not currently on Patreon, though I will consider it for the future! However, I'd love it if you would join my email newsletter: www.dataschool.io/subscribe/

[@dataschool]

Hi Davit, I finally launched a Patreon campaign! I would love your support! www.patreon.com/dataschool/overview

[@NipunRawat08]

How do I learn all these methods man they are so many and really hard to remember

[@dataschool]

Practice, practice, practice! 😄

[@DimasAnggaFM]

this video is great great!!

[@dataschool]

Thanks!

[@Theasifaliamiri]

How do I compare a mean value for two values in a column, for example, check if the mean value of 'beer servings' for Europe and Asia are same or not?

[@dataschool]

I'm not sure I understand your question - Are you asking how to write an equality statement?

[@Theasifaliamiri]

Data School sir thank you for replying. Now I got the answer from one of your other videos. Love from Pakistan.

[@dataschool]

Great to hear!

[@sayakpaul3152]

Could you explain ix operator?

[@dataschool]

I discuss it in this video: kzbin.info/www/bejne/rqfTf3Rtl6hrmdU Hope that helps!

[@sayakpaul3152]

Extremely sorry I commented without watching. Thanks.

[@dataschool]

Since you asked previously about ix, I just wanted to let you know that it has been deprecated... which I discuss in my newest video: kzbin.info/www/bejne/qpaYe6WJeLxggrs Enjoy!

[@s2kcolin780]

thank you

[@dataschool]

You're welcome!

[@KimmoHintikka]

drinks.index is giving me a different output. Is this a change in Pandas? I get > drinks.index Out[2]: RangeIndex(start=0, stop=193, step=1)

[@KimmoHintikka]

Btw. Really great pandas series. Lot of topics really clearly explained.

[@dataschool]

Yes, RangeIndex was introduced in pandas 0.18: pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0180-enhancements-rangeindex Glad you like the series!

[@patelhastik7887]

hello, how can I fetch column name and column index form excel table using panda package??

[@dataschool]

This should help: pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html

[@yucelozyazgan]

question : iloc and ix produce full-NaN rows when they have values are out of the index of the data frame. what should i do to prevent this misleading empty rows ? ex : 1 - drinks.set_index('country').ix['Afghanistan'] # as expected --> the row of afghanistan 2- drinks.set_index('country').ix['x'] # as expected --> a key error 3- drinks.set_index('country').ix[ ['Afghanistan', 'x'] ] # unexpectedly Afg. + full-NaN row

[@yucelozyazgan]

isn't it really weird ? :(

[@dataschool]

It's not clear to me whether this is the "right" or "wrong" behavior - it just is. So, I guess the solution is to just avoid passing values that are not in the index! :)

[@yucelozyazgan]

:) it's just miss leading if you are looking for just the amount of records :D it took little time to realize this feature and want to share it. As you said, i must use if condition to check whether any of your element in the index or not. i'm trying to gain experience with simple course like udacity - intro to data analysis(free). however, it takes much more time if you do not have any assist, anyone to ask your questions .. i'm not sure when i'm ready for your any courses .. i wish to work with you .. did i miss the bus ?

[@dataschool]

I have a course on Machine Learning with Text in Python: www.dataschool.io/learn/ The Master Course started last week, but it's not too late to join! I'll also be announcing new courses in the future to my newsletter subscribers: www.dataschool.io/subscribe/

[@jainamshah28]

"boolean arrays use the same shortcut as slices to select along the index axis" can you please justify what the above statement means?

[@dataschool]

I don't remember that particular sentence, sorry!

[@biddyasdiddy]

Could someone explain the difference between having the duplicate index values vs. unique index values (as has been mentioned in the video, we can replace the index with any column which means that we can set the index value with column that has duplicates)?

[@dataschool]

I'm not sure I understand your question. Maybe you could clarify what you would like to know? Thanks!

[@biddyasdiddy]

Thank you for the reply, I tried to set one column with duplicates from dataframe as its index. Since we have duplicated values as an index, how would that affect slicing and selecting? I understand that in general, when searching for rows with index, it might return two rows that have the same index, except that what are the disadvantages of having duplicate indexes?

[@dataschool]

I think that is the main disadvantage.

[@FabioRBelotto]

Can I have a non unique values for index? What would happen?

[@dataschool]

Great question! That's usually not what you want to do, but a non-unique index is currently allowed.