Your pandas questions answered!

In this video, I'm answering a few of the pandas questions I've received in the KZbin comments:
0:18 When reading from a file, how do I read in only a subset of the columns or rows?
2:53 How do I iterate through a Series or a DataFrame?
4:24 How do I drop all non-numeric columns from a DataFrame?
6:03 How do I know whether I should pass an argument as a string or a list?
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/jus...
"read_csv" documentation: pandas.pydata.o...
"iterrows" documentation: pandas.pydata.o...
"select_dtypes" documentation: pandas.pydata.o...
"describe" documentation: pandas.pydata.o...
== LET'S CONNECT! ==
Newsletter: www.dataschool...
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@sinjini3189]

I have watched several data science tutorial videos, but none of them explained the concepts as clearly as you did. Thank you so much!

[@scottlucas3710]

You always come up with solid, well explained answers before I even ask the question

[@dataschool]

Thanks! I enjoy answering these questions.

[@hemantsah8567]

I would blame youtube algos for not suggesting your channel when I needed the most. Now, I can understand these things better . Thanks Kevin........

[@dataschool]

You're very welcome!

[@kissnicky7001]

the best teacher in Data Science

[@dataschool]

Thank you!

[@dineshsingh9309]

Hello, Nice work done by Data School. I have few questions on working with files. This time let us consider a CSV file 1.How can I read a csv file of size more than 300 mb to 10 gb file? 2. How many different ways avaliable to work with very large files ? 3. Is using chunk size is nessary ?? I would request you to make a detail video on working with very large size file and tag me in so i can gain indeep understanding. Hats off to Data School, Thank You again.

[@dataschool]

Great questions! I will certainly consider them for a future video. I will tag you in the comments if and when I release it.

[@dwikisetyawan1848]

i never skip the ad, just for you my man

[@dataschool]

Ha! 😄

[@balajirajaram9512]

Excellent video as always ❤❤ For the question, how to drop every non numeric columns in pandas? We can also use the dataframe._get_numeric_data() to select only the numeric columns from the dataset.

[@awaraamin6850]

You are great man! Thank you

[@dataschool]

Thanks!

[@NicatBehbudov]

You are amazing sir! One question. How can I keep practicing everything you taught? Do you have like 'challenges' or 'small projects' where you ask the question and we try to solve the problem? Any suggestions? I keep doing what you teach after watching the video but still, it would be great to have a practice ground on pandas. Thank you!

[@dataschool]

That is an excellent question! I am working on a pandas course now that includes additional exercises... stay tuned! One idea for practice in the meantime is to pick a dataset on Kaggle Datasets, and either download it and use it for practice, or create a Kaggle Kernel to practice it online. Hope that helps!

[@lucassantana7511]

First I click on the "Like" button, then I watch the video. U r great, man. Thanks a lot for these videos!!

[@dataschool]

Ha! Thanks so much :)

[@nikhilj.206]

That's exactly what I was about to type

[@MrPaglynn]

Excellent channel. Your work is very good. Used some of your tips in my code and getting a deeper understanding of it. Thank you.

[@dataschool]

Great to hear! Thanks for your kind comments!

[@pgupta24]

Excellent ! good explanations.

[@dataschool]

Many thanks!

[@surajkhanna1129]

Hey! awesome Tutorial. one doubt though, If I have a column which is supposed to be numeric in nature and by-mistake if there is a string value in one of the rows in that column, it treats the column as non-numeric, so while using the select_dtypes(include=[np.number]), there might be data loss why does python considers the column which has maximum numeric values and very less non-numeric values , as non-numeric?

[@lonewolf2547]

Really gr8 video....So informative.....DUDE.....u are really good

[@dataschool]

Thanks very much for your kind words!

[@shankarkr1603]

Only fectch the required columns command is not working for me

[@mazkaibil9108]

Hello, i love your videos! I was wondering whether there is a series of videos for plotting? Also, how to display the data values for each data plot on a line plot? Thank you! 😃

[@dataschool]

Thanks for your suggestion! I don't have a series on plotting right now.

[@MrDavisv]

I love your tutorials! Can you make a video on joins (inner,left, full) in Pandas? Also, could you provide an example of lookup dataframes as an alternative to joins as well?

[@dataschool]

Thanks for your suggestion! And thanks for your kind words :)

[@rexster_v5624]

Hi ! i just have this one small question, how do you check your pandas version ? i have tried a lot but I could n''t find an answer.

[@ashishgavit1661]

1.Find the version of the Pandas running on any system. # importing pandas as pd import pandas as pd # Check the version print(pd.__version__) 2. Find the version of the dependencies for the given version of the Pandas running on any system. # importing pandas as pd import pandas as pd # Check the version of the dependencies pd.show_versions()

[@akhilarayapati5292]

Helloo...thank you for sharing your knowledge.... I had a question...when i filtered my excel into only 2columns(say salary and country) from many then i want to iterate through salary with different conditions like inbetween 2000 like that for particular countries say India,US lika that..... And should have particular count for each country can you please help me with this problem..how to get this output ?

[@suren6885]

Question: 1. How to choose between Pandas and Spark 2. Why Spark doesn't have these cool features of Pandas. Request: Can you please do similar tutorial videos for PySpark

[@namratapatil5181]

Hii...I am here to learn pandas.u teach amazing , I am watching Ur videos but now I am confuse as u make two playlist on pandas , so could you please tell me which one that I should to follow to learn pandas fully?

[@yuliagorodetskaya3855]

Hello, first, thank you very much for your great classes!! I would like to ask some questions about sorting and filtering the Date type data (like 10/07/2006). How do I sort all columns by date column? and how do I apply filter, for example, by month 02? Thank you, Julia

[@dataschool]

You're very welcome! To answer your question, the first step would be to convert the date column to datetime type which I explain in this video: kzbin.info/www/bejne/r3TKe3qpnJWLl5Y Once that step is done, you can sort the DataFrame using the 'sort_values' method, which I explain here: kzbin.info/www/bejne/sIqXlaJ8a92Grrs And, you can filter using the dt.month attribute of the date column. Filtering is explained here: kzbin.info/www/bejne/aHKpeIOag9NnfK8 Hope that helps!

[@yuliagorodetskaya3855]

Thank you very much!!! Yes, its helped!!! :))

[@deep6858]

Hi I am new to python and for the first time deploying python code to AWS lambda. I zipping dependencies on windows 10 machine and uploading dependencies for pyodbc and pandas but the code is not recognizing both modules on AWS, though working on local windows machine. Does installing and zipping dependencies MUST be done on Linux . Thanks

[@dataschool]

I'm not sure, sorry!

[@gabrieladias799]

Hi teacher kevin, I would like to know why my output for the code is not in a tuple like yours. Code --> In[ 1 ] for index, row in iterrows: print(index, row.City, row.State) Out[ 1 ] 0 Ithaca NY 1 Willingboro NJ 2 Holyoke CO 3 Abilene KS

[@dataschool]

It's a difference in Python 2 vs 3.

[@KhalilYasser]

You can make it as tuple using `print((index, row.City, row.State))`

[@michaelangellotti5741]

Excellent explanations. Simple and short.

[@dataschool]

Thanks!

[@truthseekerbeast4188]

For anyone who wants to practice pandas, here is everything you need github.com/guipsamora/pandas_exercises

[@dataschool]

Thanks for sharing!

[@bhavanishanker5884]

At the outset, I would like to offer my encomiums to your illustrative lectures. I have a question - How do I Match Merge two data frames in Python (Not Inner, outer, left or right), The merge should resemble like Match Merge in SAS , Please explain

[@dataschool]

I'm not familiar with "match merge", sorry!

[@navalnaware5098]

how can we script to create a single excel master workbook xlsx file if we receiving data from different source in different format .it will be great if you can help me on this .... if you want i can share the files to get an idea on it .

[@dataschool]

I'm sorry, I won't be able to help you with this - good luck!

[@autotestweb7724]

Hey...Great work and effort put in this tutorial..Weldone..keep it up.. please I would like to know if data analysis could be done with .log file using the read_csv in pandas or pandas only works with .txt and .csv files. Most of the files I have to analyse are printer log .log files and I think pandas would make my life a lot easier.

[@dataschool]

pandas works with tabular data files. If your log files are tabular (meaning that each row includes the same set of columns), then you can work with them in pandas. How exactly you read them into pandas will depend on their formatting. Hope that helps!

[@burcakotlu7858]

Thank you very much for these well prepared videos. I have a question what if we have varying number of columns and column names that we need to consider in criteria. For example, in run1 we have only there columns to consider with names: sig1. sig2 and sig3 df = df[(df['sig1']==1) | (df['sig2']==1) | (df['sig3']==1)] In run2 we have four columns to consider with names: sig1. sig2, sig3 and sig4 df = df[(df['sig1']==1) | (df['sig2']==1) | (df['sig3']==1) | (df['sig4']==1] Column names and column numbers are resolved during runtime and condition is always the same (==1). Is there way to handle this case? Thanks in advance.

[@dataschool]

I think what you need is the "isin" method, shown in this video: kzbin.info/www/bejne/j4GspZmHbZykoK8 Just update the list that you pass to "isin" as needed. Does that help?

[@Apollo77392]

Hi ,is there any ml teaching videos in your blog?

[@dataschool]

I have a free ML course! courses.dataschool.io/introduction-to-machine-learning-with-scikit-learn

[@hitsviralonly2215]

Hi Friend, Thanks for you video first...i want to print only first 10 columns for index,row in ufo.iterrows(): print(index, row.genre, row.duration).head(10) gives me error

[@dataschool]

I'm not sure if it's possible with iterrows, I'm sorry!

[@kashishjain78]

can you tell about iterators and generators

[@dataschool]

That's a bit outside the scope of my teaching, sorry!

[@nikolaygerashchenko4873]

Thank you for finally making it all clear for me. I have to say that your explanation is really easy to understand if I compare it to machine learning courses I saw and passed before. It would be great if you have some time to answer my question. Is there a way to import an html to pandas dataframe when it has different separators - one space between 1st and 2nd column, 2 spaces between 2nd and 3rd and so on? Thank you in advance.

[@dataschool]

+Nikolay Gerashchenko Glad you like the series! Regarding your question, HTML is not a tabular format, and so it's not suitable for directly reading into pandas. But if you did have tabular data in which the separator character varied, you could potentially use regular expressions in the "sep" argument to the "read_table" function to match a variable pattern.

[@elilavi7514]

Thank you for answering questions ! Very helpful . I have a question about iteration of pandas dataframe : I have a filling that .apply() method with lambda function in it is faster than iterating over panda Series or dataframe with `for` . I think there are some factorization methods to work with pandas . Can you please add on this ? Thanks again !

[@dataschool]

+Eli Lavi I would agree with you, that using the vectorized operations that are built into pandas (such as apply) are faster than using 'for' loops. However, there are cases in which 'for' loops are the only option, which is why it's worth knowing about. Thanks for the question!

[@abhaygupta7931]

Sir, amazing content....

[@sanjeev4u15]

awesome explanation...(y)

[@dataschool]

Thanks!

[@pebre79]

Thanks!

[@dataschool]

You are so very welcome, and thank you yet again!! 🙌

[@SpokenStuff]

How do I read tabular data via API service call into pandas?

[@dataschool]

I would make the API call, save the result, and then read that result into pandas. If the result is JSON, for example, I think pandas has a read_json function.

[@awaisahmad5301]

i love your tutorials sir.i want to ask in data science what skills we need for data cleaning in python ?

[@dataschool]

That's a great question, though a bit broad... my best suggestion is just to find case studies of data cleaning and examine them for insights. Hope that helps!

[@harshinde]

Hi kevin, Great video series. In relation to the first question, I was wondering if there was a preferred or a neat way to sample a huge file into a dataframe? The first nrows you choose might not be representative of the entire file contents and how could one sample a file into a dataframe to get a sense of the type of data.

[@dataschool]

Great question! I'll try to cover that in detail in a future video :)

[@nikhilj.206]

Great work!

[@dataschool]

Thanks!

[@spurthishetty6834]

Awesome!

[@dataschool]

Thanks!

[@townheadbluesboy]

Your videos are great clear and easy to understand. How can you select random rows from a dataframe?

[@dataschool]

Glad you like the videos! There is a DataFrame sample method for this purpose: pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.sample.html

[@ahowl7mx]

Learned python pandas with your tutorials over the weekend. Thanks so much! The lesson about the and/or logic with the booleans was quite helpful but I'm looking to spin it the following way: How do I "print the filename" of certain files in a directory if they contain a string? By that I mean, I want to run code that only impacts some of the csv files in my directory :). import glob import pandas as pd path = r'C:\Users\Desktop\Experiment\' allFiles = glob.glob(path + "/*.csv") & (search filename 'AMX_error' = true) #

[@dataschool]

Glad the videos have been helpful to you! I think I understand what you are trying to accomplish. It seems like all you need to do is open each file within your for loop, and then perform whatever check/operation you desire within the for loop. Does that help? Or maybe I'm misunderstanding what exactly you are struggling with?

[@ahowl7mx]

Specifically for each filename in the folder I have to search for a couple strings. Like the filename will have an "AMX" or "AR" in it, and it will also have a date (..Daily_AMX_2016-08-07_...). Tricky because I have to convert that string into a date format. Then for every file add a column with "AMX" or "AR" depending on the filename plus add a column with the converted date. Would be a great trick to know :) I think it would use the re.search.xxx function but not sure.

[@dataschool]

Yes, regular expressions would be useful for that task!

[@PradeepKumar6]

Great !!! Awesome.Thanks for this. One question though, Is it possible to select(or drop) a numeric( or character) columns in a range of columns. For example , If I have a dataframe with a,b,c,d,e,f,g,h,i columns in order and let us assume a,d,e and i are numeric columns. Now i want to drop only numeric columns exist between c and g (drop d & e). One way that by using usecol approach and typing the names of variables which i want to keep but imagine if this list is large then it becomes cumbersome and lot to type. Is there any efficient way to do it in pandas. Thanks

[@dataschool]

+Pradeep Kumar You're very welcome! To answer your question, you can select a range of columns by name using the "loc" method. For example: df.loc[:, 'c':'f'] You can select a range of columns by position using the "iloc" method. For example: df.iloc[:, 2:6] For any given scenario in which you want to drop (or keep) certain columns, you can chain together a combination of loc, iloc, drop, select_dtypes, and perhaps other methods to achieve the desired result. The optimal method depends on the exact scenario. I'll cover loc and iloc in an upcoming video :) Hope that helps!

[@PradeepKumar6]

+Data School Thank You !!!

[@dataschool]

FYI, I just posted a video about loc and iloc, if you want to learn more: kzbin.info/www/bejne/rqfTf3Rtl6hrmdU Enjoy! :)

[@NishantKumar-cg6tw]

How to compare common series in two data frame in Pandas?

[@dataschool]

If you want to check for equality, you can use the Series method 'equals': pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.equals.html

[@naveenbandi9193]

Sir can u plz upload videos on matplotlib

[@dataschool]

Thanks for your suggestion!

[@pacrii]

Have you shown the different ways to make a DataFrame? Also what are Panels? I don't know of any reason to use them, so I'd appreciate your input.

[@dataschool]

+pacrii I haven't yet covered different ways to create DataFrames, but that sounds like a great idea. Thanks! I have heard of Panels but have never used them... I'll check into it...

[@pacrii]

Many thanks.

[@dataschool]

I know you asked this question about creating DataFrames months ago, but I finally made a video to answer it: kzbin.info/www/bejne/Y4DZYoFnlKuVhpo Hope that helps! :)

[@pacrii]

Thanks for responding. Brilliant :)

[@vishenmaharaj251]

Great videos!! thank you

[@dataschool]

You're welcome!

[@husseinsaad7969]

You are awesome man!

[@dataschool]

Thanks!

[@scottlucas3710]

GREAT JOB !!

[@dataschool]

Thanks!

[@hadesishykaru]

Thanks a lot!

[@dataschool]

You're welcome!

[@zjzhuang2981]

hi! how can i add series to an empty dataframe as rows?

[@dataschool]

I'm sorry, I don't understand your question. Could you clarify what you are trying to accomplish? Thanks!

[@zjzhuang2981]

I had solved my problem, thanks though!

[@pacrii]

Really sleek :)

[@dataschool]

+pacrii Thanks!

[@gcm4312]

Great format!

[@dataschool]

+Gian Carlo Martinelli Glad you like it! I'll plan to do more like this :)

[@peacekeepermoe]

1:52 I think you meant usecols=[0, 3] I thought I did something wrong haha. Great tutorials, very clear and concise. I am very grateful to have come across your channel and videos. God bless you!

[@dataschool]

Thanks for your kind words!

[@im4485]

Lol. Podliza must be Russian or Russian speaking....funny word