How do I use pandas with scikit-learn to create Kaggle submissions?

Have you been using scikit-learn for machine learning, and wondering whether pandas could help you to prepare your data and export your predictions? In this video, I'll demonstrate the simplest way to integrate pandas into your machine learning workflow, and will create a submission for Kaggle's Titanic competition in just a few lines of code!
VIDEO: What is machine learning, and how does it work? • What is machine learni...
VIDEO SERIES: Introduction to machine learning with scikit-learn: • Machine learning in Py...
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/jus...
Kaggle's Titanic competition: www.kaggle.com...
"loc" documentation: pandas.pydata.o...
"DataFrame" constructor documentation: pandas.pydata.o...
"to_csv" documentation: pandas.pydata.o...
"to_pickle" documentation: pandas.pydata.o...
"read_pickle" documentation: pandas.pydata.o...
== LET'S CONNECT! ==
Newsletter: www.dataschool...
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@aleksandramazurek1364]

I'm going through all of your pandas videos, and I can't stress enough what a wonderful job you've done here. Just wanted to say thank you

[@dataschool]

Thanks very much for your kind words! 😄

[@roccococolombo2044]

Your pronunciation is exemplary, nice for us non native english speaker.

[@dataschool]

Excellent! I tried very hard to be understandable.

[@mratanusarkar]

exactly!! I don't even need subtitles!! his voice, as well as the audio and recording devices are perfect!!

[@jiangweiguo1150]

i'm really appreciate this series of tutorial, thank you very much.

[@dataschool]

You're very welcome!

[@urwellwisher4ui]

You are just the best as being a beginner on polishing level your teachings are so much helpful....Gratitude!

[@dataschool]

You are so welcome!

[@salehahmed6806]

Your teaching is really awesome, I stopped seeing videos of other educational sites .

[@dataschool]

😊

[@Pepper-li3fl]

I was not aware that pandas data frames can be used in scikit-learn and I used to convert them to numpy arrays. Now my code is leaner. Also the bonus tip on object persistence is very helpful. Keep up the great work, many thanks!

[@dataschool]

Glad that the video is helpful to you! You're very welcome!

[@hyakushiki23]

Another excellent guide Kevin. I know the Titanic case is a popular one and I am glad I finally understand it. I also loved your Scikit-learn series. Lastly, thank you for explaining pickling. I read the MS Python book, but now I finally understand it thanks to you.

[@dataschool]

Thanks very much for your kind words! :)

[@RayedWahed]

Awesome!!! Didn't realize writing to a csv file was that easy. When you set the index to PassengerId and printed it, it seemed the formatting was a bit off but then I wrote the code, and viewed the generated csv file in Excel and it was perfect. Thanks :)

[@dataschool]

You're very welcome! I know what you mean about the formatting... that is just how pandas lets you know that a particular column is the index (rather than a "regular" DataFrame column).

[@nagnathsatav9978]

Thanks for ur Crystal clear explanation seems very easy after hearing our voice behind it. I think u r made only for data science community.

[@dataschool]

You're welcome!

[@MrKishorebabukotha]

Thank you very much in providing short and simple learning videos with practicals

[@MrDavisv]

Thank you so much for this video! I’ve been struggling with coding a LogisticRegression classifier in sklearn until I watched this video. Thank you!!

[@dataschool]

You're very welcome! Glad it was helpful to you!

[@Davidemmanuelkatz]

Needed this for so long!!! Big thanks from a helpless econ student

[@dataschool]

You're very welcome!! :)

[@shobhitsrivastava4496]

This you so much sir,for this video series coz this series solves much of my confusion about pandas.Greate Job!

[@dataschool]

Great to hear!

[@adriandeveraaa]

You are absolutely awesome, thank you for running down the simple structure.

[@dataschool]

Thanks very much for your kind words! I appreciate it :)

[@adriandeveraaa]

Do you have any videos on space transformation? Pre-processing tips such as space transformation? Btw Subscribed (:

[@dataschool]

Not yet, thanks for the suggestion!

[@kushagrasaxena5202]

I am having trouble in predicting a csv file in which the model predicts several labels, can u help me out with it, i have trained a good model but i dont know how to predict that on a test file given by kaggle

[@johnfreitas250]

Great tutorial. Thank you so much.

[@dataschool]

You're very welcome!

[@Shkvarka]

Great!! Great job!!! Thank you!!!

[@dataschool]

Thank you!

[@hridayborah9750]

your tutorial are very helpful , thnx lot , please keep helping us

[@dataschool]

Thanks!

[@sherlocksu1131]

Hello, Kevin. In the video of the bonus part , you mention the method ".to_pickle",since I used "to_csv" more usually, what is the advantage of ".to_pickle" ? and what the difference between them?

[@dataschool]

Great question! 'to_csv' writes the data out to a CSV file, which contains the data from the DataFrame in a structured format. You could read that CSV file into any other program, or even examine it in a text editor. 'to_pickle' writes the DataFrame object to a file. That file contains all of the information about the DataFrame itself, including the data. Only pandas can read that file. 'to_csv' is generally more useful since it is more flexible. However, 'to_pickle' allows you to preserve the exact DataFrame, so that you are able to reconstruct the identical DataFrame later. If you use 'to_csv', that will not preserve the properties of the DataFrame, just the data itself. Hope that helps!

[@pratikmehta1152]

@Sherlock Su: You may scroll below. User "Nikita Zakharov" asked the same query where Kevin has answered.

[@davidborger1487]

I'd like to sign up for your email newsletter, but when I follow the link to the Data School page, the upper left panel, where I presume the subscription form is supposed to appear, just contains a continually rotating circle of dots. No form ever appears.

[@dataschool]

I'm sorry to hear that! Unfortunately, sometimes that widget does not load. Please try again: www.dataschool.io/ If that doesn't work for you, please email me at [e-mail] and I'll add you manually. Thanks for your interest, and sorry for the trouble! :)

[@AI-Health-posts]

Fantastic Tutorial. thanks for that

[@dataschool]

You're very welcome!

[@kishanastro]

Thanks man, was really helpful!

[@dataschool]

You're very welcome!

[@mukulsn1698]

your videos are very comprehensive and insightful... thankyou ! can you also upload more videos on pandas python and machine learning advanced level?

[@dataschool]

Thanks for your suggestion! This course may interest you: www.dataschool.io/learn/

[@PankajMishra-ey3yh]

Hello,kevin.This video is really amazing.I would like to suggest you something.Why don't you make a full series on the tasks that ever been to kaggle(Like titanic-> Facial Keypoints Detection etc ) .I mean learning with task would be so great.People will actually learn how to apply all the things.What are your views on this ? Can we expect something like this in near future ?

[@dataschool]

Thanks for the suggestion! I will see what I can do :)

[@mohammadabulhasnat4387]

when we fit the data with classifier, do we pass dataframe/series or numpy array? logmodel=LogisticRegression() logmodel.fit(X_train, y_train) X_train --> whether dataframe or np array?

[@dataschool]

scikit-learn understands both pandas objects and NumPy arrays.

[@AliElDousCS]

Hello :) When I run new_pred_class= logreg.predict(x_new) it gives an error "ValueError: Input contains NaN, infinity or a value too large for dtype('float64')." Not sure why? Can you help? Thanks

[@dataschool]

Did you run the exact same code as me? You can double check your code here: nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb Just click on number 22 in the table of contents to jump to the relevant code.

[@zny918951]

because have some null value in data, should do data scrubbing

[@sudaksina]

Sir kindly give me site name where I get diabetic data set CSV file with column name HBA1c or LDL and HDL,. Pima datasets are not having these column. With Thanks

[@dataschool]

I don't know, sorry!

[@MrNagios]

please, show me the best way to import XML, I struggled to find it out

[@dataschool]

I don't have a tutorial on that topic, sorry!

[@sodapopinski9922]

how does the the new_pred_class = logreg.predict(X_new) know your looking for Survived without specifying it, in the training set it was logreg.fit(X,y) but in the test class it was new_pred_class = logreg.predict(X_new) shouldn't it be new_pred_class = logreg.predict(X_new, y). y being the survived???

[@dataschool]

With the fit method, the model is learning the relationship between X and y. With the predict method, you are making predictions for unknown target values using X_new. There is no "y" when you are making predictions. Does that help?

[@sodapopinski9922]

why yes it does sir,, but shouldn't there be at least an empty variable to store the prediction in, im probably over thinking it, I did the same thing getting my degree in nursing, I over thought all of the problems.

[@dataschool]

It returns the predictions into whatever variable is on the left side of the equals sign.

[@artemiofilho]

how can i use the "sex" atribute as a feature in my X_train matrix? can i transform it to 1 and 0, 1 = male and 0 = female?

[@dataschool]

Exactly!

[@kritikeshsinha1716]

By using different feature we get different output..how could I select which one is right

[@dataschool]

My ML videos series might be helpful to you: kzbin.info/aero/PL5-da3qGB5ICeMbQuqbbCOQWcS6OYBr5A

[@injypal]

Your videos are awesome. Can you please explain where should I use .filter vs .loc method? I'm new to pandas and want to know which method is recommended practice.

[@dataschool]

I'm not familiar with filter, sorry! loc is always useful though, see here: kzbin.info/www/bejne/rqfTf3Rtl6hrmdU

[@kofteci408]

Thank you, this is very nicely done and useful. A quick question. Say that I wish to use another classifier, like RidgeClassifier. Its ".fit" method also takes a third parameter "sample_weight". Can you please indicate what that might look like in your example, with the two predictor classes you are using? I can extrapolate from there. Thanks again.

[@dataschool]

I'm sorry, I not familiar with how to use sample_weight. Let me know if you find a good resource that explains it... thanks!

[@robindong3802]

Thank you so much Kavin. great video. I have different output as (891,) (not 891L) after y.shape, can you let me know why. thank you so much.

[@dataschool]

"L" stands for "long integer". I think it used to be displayed in Python 2, but is no longer displayed in Python 3. It's an implementation detail, and nothing to worry about!

[@LonglongFeng]

Hi Kevin, How did you select the 'feature columns', based on your experience or intuition?

[@dataschool]

There are a few main ways to do feature selection: (1) domain expertise, (2) data exploration, (3) experimentation, (4) automated approaches, (5) model-specific approaches. That's the best I can summarize it in a KZbin comment! :)

[@dataschool]

I recently released a video about feature selection, that might be useful to you: kzbin.info/www/bejne/j5Kufph3oa2ap7M

[@joescanlon7502]

Very nice!

[@dataschool]

Thanks!

[@nikitazakharov1046]

Why I have to use pickle if i can do the same things with to_csv and read_csv? Thanks

[@dataschool]

Great question! I think the main reason is that pickle files are smaller than plain text files.

[@nikitazakharov1046]

Thanks for answer :)

[@GabrielBecerraAyala]

Pickle files also let you preserve objects and class structures. They are a powerful tool to do marshalling.

[@ArunKumar-ir5tp]

Hi i have one doubt. When i wish to implement k-means clustering to my dataset. but i had a problem on plotting date cloumn and floating column in 2D array. pls explain

[@dataschool]

I won't be able to help, good luck!

[@injypal]

Would you please create videos for .pivot, .pivot_table, merge, concat?

[@dataschool]

Thanks for your suggestion!

[@sezan92]

Hello. I am eager to learn about feature engineering for ml. can you make a video about it ?

[@dataschool]

Thanks for the suggestion! I'll definitely consider it for the future :)

[@dataschool]

You might enjoy my recent blog post about feature engineering: www.dataschool.io/introduction-to-feature-engineering/

[@guysensei1152]

wish i found this video sooner

[@dataschool]

Better late than never!

[@samarthsaraswat6892]

I'm getting a future warning while using logistic regression

[@dataschool]

That's because the API has changed over time. It's not a big deal, but you should do whatever the warning says.

[@anonify88]

You may use the following: from sklearn.linear_model import LogisticRegression logreg = LogisticRegression(solver='lbfgs') logreg.fit(X, y)

[@Natalia-sh6ck]

Hi Kevin, I'm going through all of your pandas videos and I wanted to thank you because I am enjoying the learning process with your detailed explanations. I am writing here a comment because I am stuck with this one. I have installed scikit-learn (version 1.1.3), but when I do 'from sklearn.linear_model import LogisticRegression', it says 'Import "sklearn.linear_model" could not be resolved'. I have tried this: from sklearn.linear_model import LogisticRegression logreg = LogisticRegression(solver='lbfgs') logreg.fit(X, y) But it did not work either. Do you know what could I be missing? I am running all the commands in 'visual Studio Code', not in Jupyter. Thank you!

[@dataschool]

Sounds like maybe there is a problem with the installation?

[@Natalia-sh6ck]

@@dataschool Hi Kevin. Thank you for answering. I uninstalled and installed again and now it works. Thanks!

[@dataschool]

Great to hear!

[@seandhuynh]

Thank you for the great tutorial.

[@dataschool]

You're very welcome!