How do I filter rows of a pandas DataFrame by column value?

[@Diachron]

Most people would have simply demonstrated the short way, but by taking the time to elucidate what's happening behind the scenes, you're performing a far greater service. Great job.

[@dataschool]

Thanks so much for your thoughtful comment! I really appreciate your support.

[@trs4184]

Him: So here's how you filter out by certain values. Me: *Copies code* Gotacha, thanks. Him: But that's the long way and you'd never actually do that Me: Oh haha of course. *deletes code*

[@guptaachin]

You say just the right words to get the concept and everything behind with lucidity. Thank you. i am a beginner but do not feel like so. :)

[@dataschool]

Wow, what a nice thing to say! :)

[@samuelgeorge6756]

I watch you at 1.5x :-D but like he said even a year later. Nice way to teach. Thank you.

[@alexcipriani6003]

you deserve my tuition money for that

[@saimaacademy5537]

kzbin.info/www/bejne/rJ3JZYOkbtOAq8k

[@guidogaggl4020]

Achin Gupta: my words. This guy is awesome

[@avallette51]

I have learned much from your channel. You're a natural for teaching.

[@dataschool]

Wow, thank you so much! I really appreciate your kind words and your contribution!! 🙏

[@ThePRINCEBARPAGA]

Your voice is so soothing and makes the understanding of concepts much easier and enjoyable. Thank You

[@dataschool]

Thank you!

[@zeeshantaj55]

You are the only man I can understand on KZbin about Data Science.

[@dataschool]

Ha! Thank you :)

[@leonardoferigutti9039]

Everything you're doing here for us, it's gonna go back to you in a certain way in the future! Thank You a lot for this master piece!!

[@dataschool]

You are too kind, thank you! 🙏

[@seangriffin3601]

3 years later and this dude is still a legend and still saving my ass on basic stuff for work!

[@dataschool]

Ha! That's great to hear 😎

[@adityaladia3293]

Awesome explanation! I have referred to some resources on pandas earlier and felt like i could easily skip your content to get through faster and grab only the unknown piece of code in the process . But the way you explain things it shows your hold on pandas and i am highly motivated to go through the full video. Cheers great job.

[@LyyrRATM]

There is something charming in how you say "Okay" when you finish a thought :) The video is very helpful. Thanks for putting this up here.

[@dataschool]

Ha! Glad my verbal tics are charming :)

[@ludovicolaci9582]

@@dataschool Kevin.verbal_function.finish_thought is this right?

[@yuriwrlk]

The explanations are really well conceived and you speak so clearly that even the youtube captioner can capture it, subscribed!

[@dataschool]

Awesome! Thanks for subscribing, and thanks for your kind comment!

[@FergusMcGee]

Thank you so much. I'm new to pandas as this was very helpful, I couldn't understand why it wasn't obvious how to filter rows. You explained it so clearly.

[@РусланМиров-е1ч]

Despite the fact that English is not my native language, these are the best lessons about the Pandas that I saw! Thank you!

[@dataschool]

It's great to hear that my videos have been helpful to you! :)

[@ASpace25]

It's imperative that you know the logic behind these codes. Thank you Mark for this amazing playlist.

[@dataschool]

My pleasure!

[@lexboegen]

I like your in-depth explanation of _how_ it works, rather than a "recipe" of "just do it this way" without explaining _why_ it works. Now I'm off to see what other videos on pandas you have!

[@dataschool]

Excellent! I'm glad the depth of the explanation was helpful to you! My complete pandas playlist is here: kzbin.info/aero/PL5-da3qGB5ICCsgW1MxlZ0Hq8LL5U3u9y

[@zeeshantaj55]

At 5:30, what's the need for converting the booleans list to pandas series? Because we can get the results even without doing that by using this command > movies[booleans] . It gives the same result.

[@dataschool]

I can't remember... maybe when I recorded the video, the filtering required a boolean Series rather than a list of booleans.

[@guillermobarrio55]

When great teaching meets simplicity. Thanks so much!

[@dataschool]

Thank you so much! 🙏

[@sajeendrann]

It's was THAT simple, yeah? I wasted two days trying to figure it out myself. Life saver you are. Thanks for the wonderful vdo.

[@dataschool]

You're welcome!

[@sanaullahkhanhassanzai8432]

You are the best Pandas teacher on KZbin. I saw your previous videos as well but this is by far the best. You made the whole concept so simple. Thank you very much

[@dataschool]

Wow! Thank you so much for your kind words! :)

[@kingsleyobodomechine933]

you are a great tutor, taking the long road and then giving the short answer really teaches the concept very well. Thank you sooo much

[@dataschool]

You're very welcome!

[@grchoudhary5259]

your way of teaching is awesome!!!!! Thanks bro

[@dataschool]

Thank you so much! 🙌

[@RAWRCoding]

Can't believe you do the entire video with so little edits. Amazing!

[@dataschool]

It requires a lot of planning... plus it helps that I love teaching this subject and I know it well!

[@rohitekka2674]

I loved how you showcased every possible way, from the heavy technique of using a for loop and then right to using operators. It made things absolutely clear. Thanks!!

[@dataschool]

You're very welcome!

[@KingLiongWong]

Very clear explanation and make me understand how the logic works behind the filter. Great thanks.

[@dataschool]

You're very welcome!

[@moonchilled1439]

Thank you. I believe by showing us the theory behind it, you're helping us remember the method for a far longer period

[@dataschool]

You’re very welcome!

[@rishisagar4695]

Love these, helped with something I was stuck on for 2 days. I'm a beginner, loads of love

[@dataschool]

Great to hear!

[@MichaelCrossUK]

Your videos really are terrifically helpful, explaining things thoroughly from base concepts upwards as you do is just brilliant and your explanations are always extremely clear and precise. Thank you so much for all the time you spend on this - I'm tremendously grateful as I'm sure are many others.

[@dataschool]

Wow, thank you so much for your incredibly kind words! I'm very glad to hear that you have gotten a lot out of the series.

[@mayanknaithani5412]

thanks , I learnt more from your videos than other online pandas resources combine

[@dataschool]

You're very welcome!

[@lucasfrancab]

Such a pretty explanation. Now I discovered the 'Why' i use the bracket. Thanks!

[@dataschool]

You're welcome!

[@sapnapatil27]

Very nice tutorials. Easy to understand and to the point. Please prepare more for Data analysis.

[@dataschool]

Thanks for your kind words, and for the suggestion!

[@GurunathHari]

You are an amazing presenter. Knowledgeable, fluent, crystal clear. Much respect. Let me know how i can give something back to you.

[@dataschool]

Thank you so much! If you want to support me, I'd love for you to join my community of "Data School Insiders" on Patreon: www.patreon.com/dataschool

[@erikahuerta2140]

Great videos! I have been struggling with the ideas and concepts behind decision trees and ensembles. I hope these are topics that you will cover in the future and if not, I would really appreciate any resources to gain a deeper understanding of this topic. Thank you!

[@dataschool]

Thanks! Regarding decision trees and ensembles, I highly recommend chapter 8 of this book for a conceptual understanding: www-bcf.usc.edu/~gareth/ISL/ Here are videos related to that book: www.dataschool.io/15-hours-of-expert-machine-learning-videos/ For Python code and more resources, see classes 17 and 18 of my data science course: github.com/justmarkham/DAT8 Hope that helps!

[@yucelozyazgan]

he put step by step all the stones in my brain and set the wall! :D Simply great !

[@dataschool]

I love the metaphor... thanks for your kind words! :)

[@yucelozyazgan]

;)

[@srijitapal8100]

you are really a great teacher. It was so easy for me to understand pandas. Thank you so very much.

[@dataschool]

You're very welcome!

[@u0000-u2x]

Worth watching the whole video but if you are just looking for the shortcut: 9:21

[@dataschool]

+Gian Carlo Martinelli Great idea to link directly to it... thanks!

[@fupopanda]

Thanks. I was hoping he would first go straight to the point and then explain afterwords.

[@saimaacademy5537]

kzbin.info/www/bejne/rJ3JZYOkbtOAq8k

[@vasileioskolias4728]

These videos are pure gold! I have a question though: The final point at 11:55 is that column selection after filtering is not the best thing and it may cause strange behavior in some cases. I was wondering what is that strange behavior and what are those cases? Thanks!

[@dataschool]

Glad the videos are helpful to you! Regarding your question, I demonstrate the problem (and the solution) in this video: kzbin.info/www/bejne/aoOXiKZ6f5J-jMU

[@TechMoney33]

All the concepts very well explained. Your videos on pandas tutorial are just great !!

[@dataschool]

Thanks! :)

[@renevillalta27]

Best way to explain, simple and accurate. Thanks

[@dataschool]

Thank you!

[@alessandrosarretta11]

Thanks, great step by step learning resource! I already knew the quick answer, but not the reason why it was working :-) Very informative!

[@dataschool]

+Alessandro Sarretta You're welcome! I think you will find that understanding how it works in this case will help you to better understand lots of other pandas functionality!

[@russnagel1]

Thank you. Great content is still great, even 5 years later!

[@dataschool]

Thanks very much for your kind words!

[@handet.6235]

Bro, you are just great teacher! Please keep going what you are already doing. I'll buy whatever you produce! God bless you! :)

[@dataschool]

Thanks so much, I appreciate it! :) Right now, I only sell one course, called Machine Learning with Text in Python: www.dataschool.io/learn/ But if you want to hear about new courses that I release, just subscribe to my newsletter: www.dataschool.io/subscribe/

[@rahulduvedi3341]

Sir,your content is the best for pandas. Thanks a Lot

[@bitmagazin303]

Man. You helped million times more than pandas documentary..

[@dataschool]

Great to hear! :)

[@RufusVidS]

Your series is terrific, only one issue, as has been mentioned before in these comments: "boolean" has 3 syllables.

[@dataschool]

Glad you like the videos!

[@OxanaGafaitiGO]

Many thanks for your tutorials. They are powerful. You make complex things easier. Thank you for your work.

[@dataschool]

Thank so much for your kind comment! I'm glad they are helpful to you!

[@willykitheka7618]

You're an EXCELLENT TEACHER. Thanks a heap!

[@dataschool]

Thank you! 😃

[@tomhas4442]

Very nice thurrough explanation! 🙏 thanks

[@dataschool]

Thank you!

[@sdoken]

At 12:36, you can also do movies.loc[movies.duration >= 200]['genre'] because loc seems to allow you to select rows only without specifying columns and then you can select a column separately at the end. However, I will do movies.loc[movies.duration >= 200, 'genre'] because I think it's what @DataSchool showed. What about when you were getting all columns : Do we still prefer to use loc movies.loc[movies.duration >= 200] ? or is movies[movies.duration >= 200] just as good with no issues?

[@dataschool]

Either option is fine!

[@mistercakes]

you can use a list comprehension instead of the traditional for loop at 4:00 booleans = [length>199 for length in movies.duration]

[@dataschool]

Yes, that's another option.

[@iccherherbarium9688]

a big thanks for putting an end to my last 3 hours' confused state.

[@dataschool]

Great to hear!! You are very welcome!

[@ajmolinaro]

Good afternoon. "Long time listener, first time caller." I really like your videos and your teaching style. As others have noted, you explain things clearly and in digestible and understandable chunks, which I appreciate. I have a question and/or a request for Q&A video. I am comparing dataframes. I am using a merge statement and a "fulll outer join". This works well for identifying records from both dataframes that do not match each other. Going a step further, I'd like to identify the individual attributes (i.e., columns) that do not match. My use case is that I often compare extremely wide datasets (200+ columns) and it is sometimes difficult to find the "offending/differing" column(s). I have researched at various places online, and have yet to find a solution that truly fits my needs.

[@dataschool]

Thanks for your kind words! That's an interesting question, I'm not sure if I 100% understand. It would be super helpful if you could code up a simple example (just a few columns) of what you are currently doing and explain exactly what your goal is. Thanks!

[@TR3NDSETR]

Another fantastic video, Im so used to the 8-9 min vidoes now , 13+mins was a bit intense :)

[@dataschool]

Thank you so much!

[@traveltripfamily7486]

You are very good teacher! Congrats!

[@dataschool]

Thanks!

[@mohammadfahim8870]

Thanks !! I was looking for something very specific and 2 of your videos was just right on point ... just subscribed !!

[@dataschool]

Awesome! Thanks for subscribing :)

[@ravivashatkar5585]

Love this video !! Exactly the one I was looking for. THANK YOU !... well explained..

[@dataschool]

You're very welcome! :)

[@rockspeed2010]

Awesome !! Your explanation is very clear and easy to follow. Keep up the good work !

[@dataschool]

Thanks for your kind words!

[@venkateshkonangi2353]

Bro... Thanks for your great tutorials. I am beginner. From your videos i could understand the correct concept. Thank you very much.

[@dataschool]

Great to hear!

[@immanuelsuleiman7550]

Expertly explained. Well done

[@dataschool]

Thank you!

[@GiggsBeckham]

I've learnt so much through watching this one video. Thank you. Now, I may need to re-write some my existing codes to make it more efficient and simple to comprehend. :-)

[@dataschool]

Great to hear!

[@jaywiji]

Your explanations are so clear. Thanks

[@hkrocksta]

Saved me so much time. Thanks for this !

[@dataschool]

Great to hear!

[@flexfpv1707]

This is very helpful! Your accent is so clear and it's very easy to understand you. Thanks!

[@dataschool]

Thanks!

[@rprana12777]

Great teacher, practically my mentor haha

[@dataschool]

Ha! :)

[@jillandersen4912]

This video was great! Thanks. I have a dataframe that I need to pair 2 rows of data, each have there own timestamp, drop the second timestamp and then filter the data with these pairs. I have alot of data to sort through, any help would be great

[@mylovelykids5795]

amazing explanation. thanks for your time and patience to educate all.

[@dataschool]

Glad it was helpful to you!

[@alabtv4879]

Explained with brilliance! Congrats.

[@dataschool]

Thank you!

[@muradjahangir]

Thanks so much! Excellent step by step explanation of the concept and method! Your clear explanation shows how well you have those concepts and logic embedded in you brain 🙂 Must subscribe!

[@dataschool]

Thank you so much, Murad! I really appreciate your kind words, and also for joining as a channel member! 🙏

[@subbuktek]

Beautiful teaching technique. You purposefully did it the long way first. Wow!. So the condition that I always used in [ ] in pandas is actually just a Boolean Series whose length matches the length of the dataframe.

[@dataschool]

Exactly! :)

[@razahaider6173]

Can anybody tell me what happened at 7:10 when we pass a True/False series to df[bool_series] here? i don't get it? how does pandas automatically knows that the true/false is for duration?

[@immanuelsuleiman7550]

I will attempt to answer your question. When you pass a boolean into a dataframe, if the boolean is true, it returns the record at that row index if the boolean is false, it does not return that record this videos is some years old and he explains the steps in a long way what I tend to do is this: movies=pd.read_csv(somefilepath) lets say Duration is the field you are looking for i would write this: movies[movies["Duration"] > 100] This returns all records in the dataframe that return a true value is the Duration field is > 100 I hope this helps Example: ibb.co/d4Ct5SX

[@immanuelsuleiman7550]

timestamp where he shows code you need: kzbin.info/www/bejne/aHKpeIOag9NnfK8

[@dataschool]

Thanks Immanuel!

[@immanuelsuleiman7550]

@@dataschool thank you for all of your efforts. I work as a data analyst and learning python and pandas has helped my life a lot. I don't think my life would be the same if it were not for your tutorials I really appreciate your efforts

[@sdoken]

9:25, you also can just directly go to movies[booleans] because it turns out you can filter with a list of booleans. It gives the same answer. Any reason you would recommend against it? Ultimately, I would use movies[movies.duration >= 200 ] anyway so it does not matter but just saying.

[@dataschool]

I created the booleans object for educational value only, so no, I would never recommend actually using it.

[@seshu124]

The best ever Python site! Many thanks for time and effort !!

[@dataschool]

You're very welcome!

[@CivilModules]

movies[movies.genre=='Crime'] suppose I want to see both Crime and Action genre, then how to write ?

[@dataschool]

See this video: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@yashraut9977]

Lots of love from India 🙏🙏 you have done a great job sir thank you.

[@dataschool]

You're welcome!

[@romanhar4151]

Best explanation I found so far. Thank you very much! But there is a tiny but very important bit missing: How can I filter by MORE then one criteria?????????

[@romanhar4151]

have found it: www.dataschool.io/python-pandas-tips-and-tricks/#filteringrowsbycondition Thanks :-)

[@dataschool]

See this video: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@brijkishortiwari2077]

Thank you very much sir for all your videos. its really a great experience and help to learn many things.

[@dataschool]

Thanks!

[@xiao2634]

Like your video. Very detailed and clear step-by-step explanation!

[@dataschool]

Thanks!

[@roozbehyousefnejad7234]

how can I use "or" statements in the filtering? imagine I want to filter values more than 200 and less than 20.

[@wampepatrick]

Great job ! I like the sensation when you feel more smarter because you understand something clearly ;-) #FromFrance

[@dataschool]

Ha! Great to hear :)

[@aungthuhein007]

So the DataFrame class has overloaded the [] operator to accept a Series object? Is that correct? Can't help comparing Python to C++.

[@stoyanradev2943]

Hey man, you are doing a great work! I have the following question, though: What if I want to filter the data frame by a column that contains a list (actors_list) but only if an exact element (string) is present in that list? Let's say "I want every movie (data frame row) in which Al Pacino plays" ? Thank you in advance and keep up the good work!

[@dataschool]

Thanks! I think this will help you: kzbin.info/www/bejne/mKDJknZmfsieftE

[@abdulaleemseyed7043]

I am in heaven right now. "You get to understand what things are when you know what it is they do." I really love the fact that you provide the "nuts and bolts" it helps me better understand the "how and the why."

[@dataschool]

It's really nice to hear that I've helped you gain some insight on this!

[@sarikadatta3706]

Just like you added genre can I add more columns?? like filtering with movies>=200 and we get genre and movie name?

[@farzanarahmanchowdhury644]

I have a question. I have a data set where I am using the Dkjestra algorithm.in the problem, after getting the shortest path, I want to use and specific value whenever the route is changing. Say for example from if I want to go from node 1 to 5, 1st set of the node is node 1 and 2 has in red route values like 100 and the next shortest path is node 2 to 5 in blue route value as 200. since value us changing in column route, (100 to 200) additional travel time is added. So each time it finds the shortest path its checks the route value as well. How can I do that?

[@swatyrao3591]

what happened to the false values in the command movies['is_long'] if we are only seeing the true values after the series is_long is passed to dataframe movies? The series is_long had false values also, how did they get filtered out?

[@leandrorberto]

You're an expert!!! Thank you so much!

[@dataschool]

Thank you!

[@reneebutouy]

You have outstanding skills to teach. Thank you.

[@dataschool]

You're welcome!

[@harshinigowda1962]

If the table has numbers as labels I mean in the place of "duration" which u have mentioned in video.... if we have set of dates how I use in the for loop condition?

[@leythecg]

A really understandable and good explanation!

[@dataschool]

Glad it was helpful!

[@aegystierone8505]

Never knew loc is a powerful filter tool!

[@waynewatson7970]

Confused (newbie) - how did the for loop list converted to pandas Series(is_long) associate the True/False values to the proper field, duration? Is it because the for loop used movie.duration? Why didn't the Series apply to any numeric field. Please explain and thanks for sharing.

[@vivekprajapati489]

thanks. was great help 1 points if u can tell: how to use multiple condition in the the data frame and select multiple columns of the same date frame

[@thomasquer452]

I felt like my dog could understand this. Thank you

[@lonathompson1635]

pandasDoc < dataSchool = True

[@dataschool]

Ha! Thanks :)

[@rusty4104]

How would you further filter the df? I have two values I want to filter by but I want two different values based on the two values filtered by. So basically I have a data set that has x amount of rows by 26 columns. I only want row data of columns C and D but filtered from the values of columns A and B.

[@forweb2411]

Wow, that's such a lucid explanation of things. Thank you so much for this fantastic effort👍👍👍. Can you please point towards a resource or let me know how to add multiple conditions like AND / OR etc?

[@dataschool]

Thanks for your kind words! See here: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@forweb2411]

@@dataschool Thanks a lot for your prompt reply.

[@Jonas89offsuit]

Boolean is three syllables. Thanks for your videos!

[@dataschool]

You are very welcome!

[@adnanparvez8264]

can you please tell me why doesn't this line work, movies[movies['title'] == 'inception'].year . I want to see the values where title matches with the year,. I am a newbie so if you can help that will be much appreciated.

[@Dharshika_Diaries]

Hi , its great to get your class. I am struck with one question , From the above table , how can i find the movies with most duration difference ? and other one is from the rows can we explicitly call one particular movie name ? Thank you,

[@dataschool]

Thanks for your kind words! I don't understand your first question, perhaps you could clarify? Regarding your second question, I think this is the code you are looking for: movies[movies.title == 'Seven Samurai']

[@tasneemhegde9588]

You are an amazing teacher.... So clear and so succinct

[@dataschool]

Thank you!

[@abigaileileenmarx6761]

Now that you have the rows selected that you want, can you then groupby a different condition. For instance, I am working on a project where I would need the average star rating for movies rated R and greater than 200 minutes. I feel like you could use this to determine the rows of the movies that are of that duration and then groupby the rating and average the star rating but I'm not quite sure how.

[@jd5787]

I had to put Python on the back-burner for too long, back in the game now ;) Great series to get up to speed quickly on some key topics.. Quick question: how do we sort the rows in an ascending / descending way after applying the "movies.duration >= 200" ?

[@jd5787]

Answering my own question: just add a ".sort_values('duration')" after the code and boom, it works ;) Got to love Pandas.

[@dataschool]

Glad you figured it out! Those should be parentheses not brackets, just an FYI for anyone else who comes across this.

[@jd5787]

@@dataschool correct, edited for accuracy. Thanks!