How do I filter rows of a pandas DataFrame by column value?

Let's say that you only want to display the rows of a DataFrame which have a certain column value. How would you do it? pandas makes it easy, but the notation can be confusing and thus difficult to remember. In this video, I'll work up to the solution step-by-step using regular Python code so that you can truly understand the logic behind pandas filtering notation.
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/jus...
"loc" documentation: pandas.pydata.o...
== LET'S CONNECT! ==
Newsletter: www.dataschool...
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@Diachron]

Most people would have simply demonstrated the short way, but by taking the time to elucidate what's happening behind the scenes, you're performing a far greater service. Great job.

[@dataschool]

Thanks so much for your thoughtful comment! I really appreciate your support.

[@trs4184]

Him: So here's how you filter out by certain values. Me: *Copies code* Gotacha, thanks. Him: But that's the long way and you'd never actually do that Me: Oh haha of course. *deletes code*

[@guptaachin]

You say just the right words to get the concept and everything behind with lucidity. Thank you. i am a beginner but do not feel like so. :)

[@dataschool]

Wow, what a nice thing to say! :)

[@samuelgeorge6756]

I watch you at 1.5x :-D but like he said even a year later. Nice way to teach. Thank you.

[@alexcipriani6003]

you deserve my tuition money for that

[@saimaacademy5537]

kzbin.info/www/bejne/rJ3JZYOkbtOAq8k

[@guidogaggl4020]

Achin Gupta: my words. This guy is awesome

[@ThePRINCEBARPAGA]

Your voice is so soothing and makes the understanding of concepts much easier and enjoyable. Thank You

[@dataschool]

Thank you!

[@seangriffin3601]

3 years later and this dude is still a legend and still saving my ass on basic stuff for work!

[@dataschool]

Ha! That's great to hear 😎

[@zeeshantaj55]

You are the only man I can understand on KZbin about Data Science.

[@dataschool]

Ha! Thank you :)

[@leonardoferigutti9039]

Everything you're doing here for us, it's gonna go back to you in a certain way in the future! Thank You a lot for this master piece!!

[@dataschool]

You are too kind, thank you! 🙏

[@user-oj8nq4wn8n]

Despite the fact that English is not my native language, these are the best lessons about the Pandas that I saw! Thank you!

[@dataschool]

It's great to hear that my videos have been helpful to you! :)

[@LyyrRATM]

There is something charming in how you say "Okay" when you finish a thought :) The video is very helpful. Thanks for putting this up here.

[@dataschool]

Ha! Glad my verbal tics are charming :)

[@ludovicolaci9582]

@@dataschool Kevin.verbal_function.finish_thought is this right?

[@FergusMcGee]

Thank you so much. I'm new to pandas as this was very helpful, I couldn't understand why it wasn't obvious how to filter rows. You explained it so clearly.

[@lexboegen]

I like your in-depth explanation of _how_ it works, rather than a "recipe" of "just do it this way" without explaining _why_ it works. Now I'm off to see what other videos on pandas you have!

[@dataschool]

Excellent! I'm glad the depth of the explanation was helpful to you! My complete pandas playlist is here: kzbin.info/aero/PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@ASpace25]

It's imperative that you know the logic behind these codes. Thank you Mark for this amazing playlist.

[@dataschool]

My pleasure!

[@adityaladia3293]

Awesome explanation! I have referred to some resources on pandas earlier and felt like i could easily skip your content to get through faster and grab only the unknown piece of code in the process . But the way you explain things it shows your hold on pandas and i am highly motivated to go through the full video. Cheers great job.

[@sajeendrann]

It's was THAT simple, yeah? I wasted two days trying to figure it out myself. Life saver you are. Thanks for the wonderful vdo.

[@dataschool]

You're welcome!

[@guillermobarrio55]

When great teaching meets simplicity. Thanks so much!

[@dataschool]

Thank you so much! 🙏

[@abdulaleemseyed7043]

I am in heaven right now. "You get to understand what things are when you know what it is they do." I really love the fact that you provide the "nuts and bolts" it helps me better understand the "how and the why."

[@dataschool]

It's really nice to hear that I've helped you gain some insight on this!

[@hoomandehghani2035]

Thank you. I believe by showing us the theory behind it, you're helping us remember the method for a far longer period

[@dataschool]

You’re very welcome!

[@KingLiongWong]

Very clear explanation and make me understand how the logic works behind the filter. Great thanks.

[@dataschool]

You're very welcome!

[@rohitekka2674]

I loved how you showcased every possible way, from the heavy technique of using a for loop and then right to using operators. It made things absolutely clear. Thanks!!

[@dataschool]

You're very welcome!

[@sanaullahkhanhassanzai8432]

You are the best Pandas teacher on KZbin. I saw your previous videos as well but this is by far the best. You made the whole concept so simple. Thank you very much

[@dataschool]

Wow! Thank you so much for your kind words! :)

[@rishisagar4695]

Love these, helped with something I was stuck on for 2 days. I'm a beginner, loads of love

[@dataschool]

Great to hear!

[@yuriwrlk]

The explanations are really well conceived and you speak so clearly that even the youtube captioner can capture it, subscribed!

[@dataschool]

Awesome! Thanks for subscribing, and thanks for your kind comment!

[@iccherherbarium9688]

a big thanks for putting an end to my last 3 hours' confused state.

[@dataschool]

Great to hear!! You are very welcome!

[@lucasfrancab]

Such a pretty explanation. Now I discovered the 'Why' i use the bracket. Thanks!

[@dataschool]

You're welcome!

[@bitmagazin303]

Man. You helped million times more than pandas documentary..

[@dataschool]

Great to hear! :)

[@kingsleyobodomechine933]

you are a great tutor, taking the long road and then giving the short answer really teaches the concept very well. Thank you sooo much

[@dataschool]

You're very welcome!

[@RufusVidS]

Your series is terrific, only one issue, as has been mentioned before in these comments: "boolean" has 3 syllables.

[@dataschool]

Glad you like the videos!

[@GurunathHari]

You are an amazing presenter. Knowledgeable, fluent, crystal clear. Much respect. Let me know how i can give something back to you.

[@dataschool]

Thank you so much! If you want to support me, I'd love for you to join my community of "Data School Insiders" on Patreon: www.patreon.com/dataschool

[@srijitapal8100]

you are really a great teacher. It was so easy for me to understand pandas. Thank you so very much.

[@dataschool]

You're very welcome!

[@RAWRCoding]

Can't believe you do the entire video with so little edits. Amazing!

[@dataschool]

It requires a lot of planning... plus it helps that I love teaching this subject and I know it well!

[@MichaelCrossUK]

Your videos really are terrifically helpful, explaining things thoroughly from base concepts upwards as you do is just brilliant and your explanations are always extremely clear and precise. Thank you so much for all the time you spend on this - I'm tremendously grateful as I'm sure are many others.

[@dataschool]

Wow, thank you so much for your incredibly kind words! I'm very glad to hear that you have gotten a lot out of the series.

[@russnagel1]

Thank you. Great content is still great, even 5 years later!

[@dataschool]

Thanks very much for your kind words!

[@sapnapatil27]

Very nice tutorials. Easy to understand and to the point. Please prepare more for Data analysis.

[@dataschool]

Thanks for your kind words, and for the suggestion!

[@gcm4312]

Worth watching the whole video but if you are just looking for the shortcut: 9:21

[@dataschool]

+Gian Carlo Martinelli Great idea to link directly to it... thanks!

[@fupopanda]

Thanks. I was hoping he would first go straight to the point and then explain afterwords.

[@saimaacademy5537]

kzbin.info/www/bejne/rJ3JZYOkbtOAq8k

[@jillandersen4912]

This video was great! Thanks. I have a dataframe that I need to pair 2 rows of data, each have there own timestamp, drop the second timestamp and then filter the data with these pairs. I have alot of data to sort through, any help would be great

[@ajmolinaro]

Good afternoon. "Long time listener, first time caller." I really like your videos and your teaching style. As others have noted, you explain things clearly and in digestible and understandable chunks, which I appreciate. I have a question and/or a request for Q&A video. I am comparing dataframes. I am using a merge statement and a "fulll outer join". This works well for identifying records from both dataframes that do not match each other. Going a step further, I'd like to identify the individual attributes (i.e., columns) that do not match. My use case is that I often compare extremely wide datasets (200+ columns) and it is sometimes difficult to find the "offending/differing" column(s). I have researched at various places online, and have yet to find a solution that truly fits my needs.

[@dataschool]

Thanks for your kind words! That's an interesting question, I'm not sure if I 100% understand. It would be super helpful if you could code up a simple example (just a few columns) of what you are currently doing and explain exactly what your goal is. Thanks!

[@avallette51]

I have learned much from your channel. You're a natural for teaching.

[@dataschool]

Wow, thank you so much! I really appreciate your kind words and your contribution!! 🙏

[@vivekprajapati489]

thanks. was great help 1 points if u can tell: how to use multiple condition in the the data frame and select multiple columns of the same date frame

[@rahulduvedi3341]

Sir,your content is the best for pandas. Thanks a Lot

[@stoyanradev2943]

Hey man, you are doing a great work! I have the following question, though: What if I want to filter the data frame by a column that contains a list (actors_list) but only if an exact element (string) is present in that list? Let's say "I want every movie (data frame row) in which Al Pacino plays" ? Thank you in advance and keep up the good work!

[@dataschool]

Thanks! I think this will help you: kzbin.info/www/bejne/mKDJknZmfsieftE

[@GrumpyDerg]

Many thanks for revealing the mystery behind the weirdness of the filtering syntax. I've been using it already, but now I realize the reason behind it, which is pretty cool!

[@dataschool]

You're very welcome! I find that understanding filtering helps you to better understand how to use pandas as a whole, which is why I decided to explain it. Glad it was helpful to you!

[@renevillalta27]

Best way to explain, simple and accurate. Thanks

[@dataschool]

Thank you!

[@romanhar4151]

Best explanation I found so far. Thank you very much! But there is a tiny but very important bit missing: How can I filter by MORE then one criteria?????????

[@romanhar4151]

have found it: www.dataschool.io/python-pandas-tips-and-tricks/#filteringrowsbycondition Thanks :-)

[@dataschool]

See this video: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@erikahuerta2140]

Great videos! I have been struggling with the ideas and concepts behind decision trees and ensembles. I hope these are topics that you will cover in the future and if not, I would really appreciate any resources to gain a deeper understanding of this topic. Thank you!

[@dataschool]

Thanks! Regarding decision trees and ensembles, I highly recommend chapter 8 of this book for a conceptual understanding: www-bcf.usc.edu/~gareth/ISL/ Here are videos related to that book: www.dataschool.io/15-hours-of-expert-machine-learning-videos/ For Python code and more resources, see classes 17 and 18 of my data science course: github.com/justmarkham/DAT8 Hope that helps!

[@flexfpv1707]

This is very helpful! Your accent is so clear and it's very easy to understand you. Thanks!

[@dataschool]

Thanks!

[@mayanknaithani5412]

thanks , I learnt more from your videos than other online pandas resources combine

[@dataschool]

You're very welcome!

[@alessandrosarretta11]

Thanks, great step by step learning resource! I already knew the quick answer, but not the reason why it was working :-) Very informative!

[@dataschool]

+Alessandro Sarretta You're welcome! I think you will find that understanding how it works in this case will help you to better understand lots of other pandas functionality!

[@handet.6235]

Bro, you are just great teacher! Please keep going what you are already doing. I'll buy whatever you produce! God bless you! :)

[@dataschool]

Thanks so much, I appreciate it! :) Right now, I only sell one course, called Machine Learning with Text in Python: www.dataschool.io/learn/ But if you want to hear about new courses that I release, just subscribe to my newsletter: www.dataschool.io/subscribe/

[@chinmoly6428]

movies[movies.duration>=200]

[@mattymallz4207]

Thanks Molly!

[@asgaraliyev8740]

how can i do this by removing NaN

[@tomhas4442]

Very nice thurrough explanation! 🙏 thanks

[@dataschool]

Thank you!

[@ravivashatkar5585]

Love this video !! Exactly the one I was looking for. THANK YOU !... well explained..

[@dataschool]

You're very welcome! :)

[@GiggsBeckham]

I've learnt so much through watching this one video. Thank you. Now, I may need to re-write some my existing codes to make it more efficient and simple to comprehend. :-)

[@dataschool]

Great to hear!

[@yucelozyazgan]

he put step by step all the stones in my brain and set the wall! :D Simply great !

[@dataschool]

I love the metaphor... thanks for your kind words! :)

[@yucelozyazgan]

;)

[@VRandAI]

All the concepts very well explained. Your videos on pandas tutorial are just great !!

[@dataschool]

Thanks! :)

[@willykitheka7618]

You're an EXCELLENT TEACHER. Thanks a heap!

[@dataschool]

Thank you! 😃

[@subbuktek]

Beautiful teaching technique. You purposefully did it the long way first. Wow!. So the condition that I always used in [ ] in pandas is actually just a Boolean Series whose length matches the length of the dataframe.

[@dataschool]

Exactly! :)

[@TR3NDSETR]

Another fantastic video, Im so used to the 8-9 min vidoes now , 13+mins was a bit intense :)

[@dataschool]

Thank you so much!

[@venkateshkonangi2353]

Bro... Thanks for your great tutorials. I am beginner. From your videos i could understand the correct concept. Thank you very much.

[@dataschool]

Great to hear!

[@abigaileileenmarx6761]

Now that you have the rows selected that you want, can you then groupby a different condition. For instance, I am working on a project where I would need the average star rating for movies rated R and greater than 200 minutes. I feel like you could use this to determine the rows of the movies that are of that duration and then groupby the rating and average the star rating but I'm not quite sure how.

[@paperoga32]

WHY the entirely independently generated boolean list - which then you converted into a series and then applied to the dataset; it filters the "duration" series and not just another one. I mean How does pandas know which series to apply the boolean list to? Should you create a list manually adding a bunch of True and False up until 979 times, and then applied to the dataset... would Pandas apply to the same series "duration" ? please let me understand the logic behind. Thank you very much for your videos which help me a lot

[@dataschool]

Great question! The boolean list doesn't filter the duration Series. Rather, it filters the entire DataFrame. However, the effect of this is to filter rows based on values in the duration Series, since that's the Series we examined in order to generate the boolean list. Hope that helps!

[@brijkishortiwari2077]

Thank you very much sir for all your videos. its really a great experience and help to learn many things.

[@dataschool]

Thanks!

[@shtarkbsrb8793]

Very well explained! I was able to filter some data from a huge CSV file with your help! My first python program! =) import pandas as pd df = pd.read_csv('huge.csv',delimiter=";") chosen_ncm = 27129000 booleans = [] for ncm in df. CO_NCM: if ncm == chosen_ncm: booleans.append(True) else: booleans.append(False) filtered_ncm = pd.Series(booleans) out_csv = 'filter_result_' + str(chosen_ncm) + '.csv' df[filtered_ncm].to_csv(out_csv)

[@traveltripfamily7486]

You are very good teacher! Congrats!

[@dataschool]

Thanks!

[@immanuelsuleiman7550]

Expertly explained. Well done

[@dataschool]

Thank you!

[@mylovelykids5795]

amazing explanation. thanks for your time and patience to educate all.

[@dataschool]

Glad it was helpful to you!

[@thomasquer452]

I felt like my dog could understand this. Thank you

[@devtoshdubey2680]

How did Pandas know to filter on movie duration column why not on other columns? Is it because the for loop was used on duration columns? Thanks

[@OxanaGafaitiGO]

Many thanks for your tutorials. They are powerful. You make complex things easier. Thank you for your work.

[@dataschool]

Thank so much for your kind comment! I'm glad they are helpful to you!

[@shobharoy2033]

You just saved me a few hours reading a boring(and expensive) book on Python

[@dataschool]

Ha! That's great to hear!

[@hkrocksta]

Saved me so much time. Thanks for this !

[@dataschool]

Great to hear!

[@lonathompson1635]

pandasDoc < dataSchool = True

[@dataschool]

Ha! Thanks :)

[@mohammadfahim8870]

Thanks !! I was looking for something very specific and 2 of your videos was just right on point ... just subscribed !!

[@dataschool]

Awesome! Thanks for subscribing :)

[@ahmetkoklu5475]

Thanks for the video it helps a lot. I have a question: what if i want to list 2 (or more) columns with the same criteria (duration>=200)?

[@batuhanayyldz6147]

df.loc[df.duration >=200, ["col1","col2"]]

[@wampepatrick]

Great job ! I like the sensation when you feel more smarter because you understand something clearly ;-) #FromFrance

[@dataschool]

Ha! Great to hear :)

[@yashraut9977]

Lots of love from India 🙏🙏 you have done a great job sir thank you.

[@dataschool]

You're welcome!

[@waynewatson7970]

Confused (newbie) - how did the for loop list converted to pandas Series(is_long) associate the True/False values to the proper field, duration? Is it because the for loop used movie.duration? Why didn't the Series apply to any numeric field. Please explain and thanks for sharing.

[@imaansarwar2314]

I made a column with "0" and "1" values now I want to plot a graph if column values = 0 then corresponding record column should be taken )example: a nd b are two cols B has 0 then 1 so A column whose corresponding B is 0 plot it then B with 1 corresponding A should be ploted in the same graph so that I can analyse the change that I got in parameter A if b 0 shows before maintanence and B 1 shows after maintanence

[@jd5787]

I had to put Python on the back-burner for too long, back in the game now ;) Great series to get up to speed quickly on some key topics.. Quick question: how do we sort the rows in an ascending / descending way after applying the "movies.duration >= 200" ?

[@jd5787]

Answering my own question: just add a ".sort_values('duration')" after the code and boom, it works ;) Got to love Pandas.

[@dataschool]

Glad you figured it out! Those should be parentheses not brackets, just an FYI for anyone else who comes across this.

[@jd5787]

@@dataschool correct, edited for accuracy. Thanks!

[@jaywiji]

Your explanations are so clear. Thanks

[@muradjahangir]

Thanks so much! Excellent step by step explanation of the concept and method! Your clear explanation shows how well you have those concepts and logic embedded in you brain 🙂 Must subscribe!

[@dataschool]

Thank you so much, Murad! I really appreciate your kind words, and also for joining as a channel member! 🙏

[@hungminhdong3513]

Thank you Kevin! Excellent explanation. I have a question : movies.loc[movies.duration >= 200] will return a dataframe but movies.loc[movies.duration >= 200, 'genre'] will return a panda Series. Why is that?

[@dataschool]

Thanks for your kind words! The first code implies "I want all columns", whereas the second specifies a single column. Hope that helps!

[@grchoudhary5259]

your way of teaching is awesome!!!!! Thanks bro

[@dataschool]

Thank you so much! 🙌

[@alabtv4879]

Explained with brilliance! Congrats.

[@dataschool]

Thank you!

[@ahmadzaimhilmi]

This example is good for single or few columns. What if I have hundreds of columns and I want to query a particular value in all of the columns? The code will be very long since if I have to write similar conditon in all of the columns.

[@Jonas89offsuit]

Boolean is three syllables. Thanks for your videos!

[@dataschool]

You are very welcome!

[@forweb2411]

Wow, that's such a lucid explanation of things. Thank you so much for this fantastic effort👍👍👍. Can you please point towards a resource or let me know how to add multiple conditions like AND / OR etc?

[@dataschool]

Thanks for your kind words! See here: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@forweb2411]

@@dataschool Thanks a lot for your prompt reply.

[@rprana12777]

Great teacher, practically my mentor haha

[@dataschool]

Ha! :)

[@sdoken]

9:25, you also can just directly go to movies[booleans] because it turns out you can filter with a list of booleans. It gives the same answer. Any reason you would recommend against it? Ultimately, I would use movies[movies.duration >= 200 ] anyway so it does not matter but just saying.

[@dataschool]

I created the booleans object for educational value only, so no, I would never recommend actually using it.

[@justink7600]

How would I sort the filtered rows? ie) if i want the durations to go in descending order, but I only want to see the filtered movies and not the whole dataframe

[@rockspeed2010]

Awesome !! Your explanation is very clear and easy to follow. Keep up the good work !

[@dataschool]

Thanks for your kind words!

[@sdoken]

At 12:36, you can also do movies.loc[movies.duration >= 200]['genre'] because loc seems to allow you to select rows only without specifying columns and then you can select a column separately at the end. However, I will do movies.loc[movies.duration >= 200, 'genre'] because I think it's what @DataSchool showed. What about when you were getting all columns : Do we still prefer to use loc movies.loc[movies.duration >= 200] ? or is movies[movies.duration >= 200] just as good with no issues?

[@dataschool]

Either option is fine!

[@aegystierone8505]

Never knew loc is a powerful filter tool!

[@roozbehyousefnejad7234]

how can I use "or" statements in the filtering? imagine I want to filter values more than 200 and less than 20.

[@gauravburaniya9690]

in the bonus tip,if we dont want redundant data what should we do?

[@alexandercrump2298]

@AchinGupta I completely agree. You break down the methods so they are very easy to understand. This is definitely helped by the fact that you have a solid grasp of pandas! Thanks again

[@dataschool]

Thanks for your kind comment! :)

[@leandrorberto]

You're an expert!!! Thank you so much!

[@dataschool]

Thank you!

[@zeeshantaj55]

At 5:30, what's the need for converting the booleans list to pandas series? Because we can get the results even without doing that by using this command > movies[booleans] . It gives the same result.

[@dataschool]

I can't remember... maybe when I recorded the video, the filtering required a boolean Series rather than a list of booleans.

[@farzanarahmanchowdhury644]

I have a question. I have a data set where I am using the Dkjestra algorithm.in the problem, after getting the shortest path, I want to use and specific value whenever the route is changing. Say for example from if I want to go from node 1 to 5, 1st set of the node is node 1 and 2 has in red route values like 100 and the next shortest path is node 2 to 5 in blue route value as 200. since value us changing in column route, (100 to 200) additional travel time is added. So each time it finds the shortest path its checks the route value as well. How can I do that?

[@dkfaust]

BOO-lee-an These are great though.

[@dataschool]

Glad you like the series!

[@scodaddyyy]

This was driving me a bit nuts too, but you're not quite right either. Named after George Boole, it's pronounced "Boole - ee - an"

[@heathergray4880]

Maybe I am insane because the mispronunciation of Boolean (which should be "BOOL-EE-UN") made this video very hard to watch. :( Information is great. Too bad I'm such a freak. lol

[@swatyrao3591]

what happened to the false values in the command movies['is_long'] if we are only seeing the true values after the series is_long is passed to dataframe movies? The series is_long had false values also, how did they get filtered out?

[@seshu124]

The best ever Python site! Many thanks for time and effort !!

[@dataschool]

You're very welcome!