Implementing a Spam classifier in python| Natural Language Processing

[@priyasinha2251]

I am not a girl who generally comments on you tube videos but I am learning from your videos and this is my genuine comment that you are amazing and your concept in data science is very clear and to the point. I am very happy that the teacher like you is present here. Superb job Sir !

[@amankukar7586]

Who asked you if you generally comments or not?

[@unknownfacts3716]

@@amankukar7586 good one bro

[@unknownfacts3716]

pehli fursat mei nikal yahan zyaada formality mat kar

[@moindalvs]

"I am not a girl" okay can't say these days "who generally comments on youtube videos" first of all youtube doesn't have any comment history data to prove this second How dare you call this another youtube video? How dare you generalised an educational video that free of cost while people pay an hefty amount of price for such contents? shame on you!

[@vipinbansal8886]

I was trying to understand NLP concepts referring to various books and videos from last two months but concepts were not clear for me.But this explaination is really awesome .Explained in very easy way .Thanks Krish

[@yonasbabulet3836]

i have seen a lot of youtube tutorials , but i cant find tutorial like you which are clear and more precise. keep going.

[@piyushaneja7168]

You are great sir, its very difficult to find a good channel that explains the code line by line ❤💥👏

[@navrozlamba]

I would say to prevent leakage we should split our data before we fit_transform on the corpus. So in other words, we are teaching vocabulary to our model on the whole dataset which defeats the purpose of splitting into train and test after. The whole purpose of the test set is to test our model on unique data that our model has never seen before. Please correct me if I am wrong! Cheers!!

[@cristianovivk4935]

i agree should split before fit_transform to prevent leakage.,.....

[@iEntertainmentFunShorts]

+1

[@tejashshah5202]

Agree, split before getting BOW.

[@КаратэПацан-я6б]

Hi. The CountVectorizer is not a ML model, it just converts to vectors(matrix of numbers)

[@utkar1]

Thank you, the whole NLP playlist is very helpful!

[@arjyabasu1311]

Exactly

[@dipakwaghmare1228]

Sir meri tapshya hi puri ho gae ye apka lecture dekhake ❤️thank you so so so so so much sir ❤️❤️❤️❤️

[@mansoorbaig9232]

Great work Krish. You have this knack of explaining the things in pretty simple manner.

[@billyerickson353]

🎯 Key Takeaways for quick navigation: 00:00 📚 *Introduction to Spam Classifier Project* - Creating a spam classifier using natural language processing. - Overview of the dataset from UCI's SMS Spam Collection. - Reading and understanding the dataset structure. 01:47 📂 *Exploring the Dataset and Data Preprocessing* - Explanation of the SMS spam collection dataset. - Reading the dataset using pandas and handling tab-separated values. - Data cleaning and preprocessing steps using regular expressions and NLTK. 05:46 🧹 *Text Cleaning and Preprocessing* - Using regular expressions to remove unnecessary characters. - Lowercasing all words to avoid duplicates. - Tokenizing sentences, removing stop words, and applying stemming. 13:52 🎒Creating *the Bag of Words* - Introduction to bag-of-words representation. - Implementation of count vectorization using sklearn's CountVectorizer. - Selecting the top 5,000 most frequent words as features. 17:27 📊 *Preparing the Output Data* - Converting the categorical labels (ham and spam) into dummy variables. - Finalizing the output data with one column representing the spam category. - Overview of the preprocessed data for training the machine learning model. 21:04 📊 *Data Preparation for Spam Classification* - Data preparation involves creating independent (X) and dependent (Y) features. - Explanation of dummy variable trap in categorical features. - Introduction to the train-test split for model training. 22:30 🛠️ *Addressing Class Imbalance and Train Spam Classifier* - Discussion on class imbalance issue in the data. - Introduction to Naive Bayes classification technique. - Implementation of the Naive Bayes classifier using multinomial Naive Bayes. 24:22 📈 *Evaluating Spam Classifier Performance* - Explanation of the prediction process using the trained model. - Introduction to confusion matrix for model evaluation. - Calculation of accuracy score for the spam classifier (98% accuracy). 27:50 🔄 *Improving Spam Classifier Accuracy* - Suggestions for improving accuracy, including the use of lemmatization. - Mention of addressing class imbalance for better performance. - Recommendation to explore TF-IDF model as an alternative to count vectorization. Made with HARPA AI

[@matanakhni]

Best NLP videos of all time . A complete gist , mind you not for the faint hearted . Execllent job Krish. Initially ibhad given up NLP completely but now have renewed vigour after such exemplary teaching

[@niksvp93]

The best possible tutorial on Data Science/Machine Learning on KZbin. Cheers to you brother! :D

[@javiermarti_author]

You are an excellent teacher. Thanks for making/uploading these videos

[@sivabalaram4962]

You are genius in explanation krish Naik Ji, your the best 👍👌👌👌

[@dheerajkumar9857]

Excellent , very happy to see such type of explanation @Krissh Naik, we will definitely do good.

[@sauravkumar-cw5bm]

I used Lemmatization and TF-IDF in text preprocessing and got an accuracy score of 0.971.

[@ashishn.c.7913]

I am getting these accuracy values for different combinations: Stemming and CountVectorizer accuracy=98.5650% Lemmatization and CountVectorizer accuracy=98.29596% Lemmatization and TfidfVectorizer accuracy=97.9372197309417% Stemming and TfidfVectorizer accuracy=97.9372197309417%(same as Lemmatization and TfidfVectorizer)

[@sanandapodder5027]

Thank you very much sir,your videos are really very helpful i am learning NLP from your channel first time . I don't know machine learning thats why facing little problem

[@mohammedsohilshaikh6831]

I am so much addicted to his videos, sometimes even forget to like the video.😂

[@ABHINAVARYA]

Best playlist to learn NLP. Thank you Krish.. 🙂

[@ushirranjan6713]

Its really a fantastic video sir. Your really explained the many things which can be understand in very easy manner. Thanks a lots sir!!!

[@lifebytesss]

Just amazing sir , cant comment you too usefull sessions thankyou

[@DhananjayKumar-oh2hh]

you are really great sir. each and every topic u have explained very well. Hats off to u.

[@mbmathematicsacademic7038]

I used logistic regression ,multiclass was specified and I achieved 94.3% accuracy on test data and 95.7% accuracy on test data

[@vinimator]

Hi Krish, I am the newest subscriber of your channel and I hope your this video will help me to complete a project of mine own. Thank you so much. Will continue to learn

[@mandeep8696]

Thank You Krish for sharing the knowledge.

[@rahuljaiswal9379]

u r awesome teacher, it really helpful for me...... god bless u

[@debatradas1597]

Thank you so much Krish Sir...!!!

[@Thebeautyoftheworld1111]

keep up the good work.Thanks

[@tarung7088]

Here the dataset is highly imbalenced (i.e ham : 4825, spam : 747) so got the high accuray

[@gauravpardeshi6056]

very good video sir...thank you

[@ManiKandan-ol9gm]

Really no words to represent you.....lottttttttttts of love sir❤️tq so much sir means alot

[@Skandawin78]

Good job Krish with the NLP playlist

[@chandrakanthshalivahana1417]

hello,sir i am very happy that u r making videos..please make more videos on kaggle competitions...

[@AdityaKumar-cr9mc]

You are simply amazing

[@usaikiran96]

How to decide when to use count vectorizer, or tfidf? How to decide whether/when to use Stemming or Lemmatization? Like in this example why didnt you use tfidf instead of bag of words? And why lemmatization was not used instead of stemming?

[@mujeebrahman5282]

Sometimes the error is good for health😂

[@nehasrivastava8927]

Thanku sir...for the wonderful explanation

[@AltafAnsari-tf9nl]

Thank you so much for sharing your knowledge with us

[@jinks6887]

You are bhagwaan for me Sir

[@nehamanpreet1044]

Please make videos on word embedding like word2vec/GloVe/BERT/Elmo/GPT/XLNet etc

[@farnazfarhand5957]

it was so clear and helpful, thank you so much

[@sandipansarkar9211]

Thanks Krish .Superb explanation once again.All my concepts about NLP is very crystal clear.I know career in NLP is superb.But can you explain what is its exact value in terms of data science carrer. Please guide and feel free to reply as I am eagerly waiting. Thanks once again.

[@ranjanjena2996]

i have created the model and saved the same using joblib. I am not getting how to use the model for prediction? Is there anyway where i can pass the email text to the body and model can detect spam or ham. I am newbie plz help. Thanks

[@aleenajames7609]

Have you got how to do? If yes please let me know also

[@soumyadev100]

Hi Krish, good session. I have one comment. For getting test corpus, better practice may be to use transform. Fit transform on train and only transform test. And train test split to be done before we build corpus. Let me know what you think.

[@shahariarsarkar3433]

Brother you are making helpful content for us. Can you tell me how to remove the stopwords of other languages like Bangla or Hindi etc?

[@suvarnadeore8810]

Thank you krish sir

[@afaqueumer7968]

Hello Sir...can you please make video on Topic Analysis - LDA. There isn't any clear cut videos on utube yet like yours.

[@sathishk8685]

Hi Krish, Excellent explanation

[@nehamanpreet1044]

Sir please make videos on LDA, NMF, SVD and Word2Vec Models

[@amruthasankar3453]

Thankyou sir❤️🔥

[@roshankumarsharma8725]

Sir in this model why we have used MultinomialNB and not BernoulliNB ? and can we use BernoulliNB this instead of MultinomialNB

[@emajhugroo109]

Hello sir, I would like to know how to calssify a new message as ham or spam after building the NB model

[@yogeshprajapati7107]

You can do it like this. df=pd.DataFrame(['this message is a spam'],columns=['message']) corpus=[] for i in range(0,len(df)): review=re.sub('[^a-zA-Z]',' ',df['message'][i]) review=review.lower() review=review.split() review=[ps.stem(word) for word in review if word not in stopwords.words('english')] review=' '.join(review) corpus.append(review) df=cv.transform(corpus).toarray() pred=spam_detect_model.predict(df) label=pred[0] if label==1: print('Spam') else: print('Ham')

[@joelkhaung]

@@yogeshprajapati7107 how does model handle for features 2500 when doing predict? I believe there will mismatch between number of features from new message and number of features from trained model. can share how to overcome this?

[@indian-inshorts5786]

Sir u r too good

[@saratht8223]

Hi Kris, supposing we need to implement a functionality for identifying spam afresh, how can we come up with a solution. The sample data used here already have something tagged as spam and ham by someone, sometime, somewhere. In practice, do we need to have a sample data upfront? Can you please advice?

[@puttacse]

Hi Krish, Why are we hard-coding Max_features=5000, What if this code is Migrated to Production as-is and face more Tokens/Features in Live Data(Ex: if live data has 0.1 Million(1 Lakh) features)? In this scenario, Do our Model fails?

[@rajarshidgp2003]

instead of pd.get_dummies , we can sklearn.preprocessing.LabelEncoder can be used

[@tapabratacse]

why didnt u use label encoder for terget column spam/ham

[@nikhilsharma6218]

i have 2 questions first : Why only multinomialNB, is there specific reason, cant we use bernoulliNB or gaussianNB ?? second : if dataset is imbalanced we have used complimentNB, but how do we know that dataset is balanced or imbalanced??

[@manikhindwan6790]

BinomialNB - when spam classification is being done with a two step decision approach i.e if 'X' is present, then 'spam' else 'not spam' GaussianNB - used when the values are present and are continuous MultinomialNB - counts the presence of words and the frequency of occurrence to decide the decision boundary

[@babyyoda5140]

Boss please also include sentiment analysis and topic modelling to your already wonderful repertoire!

[@juanelnino]

I have a ERROR it is saying unhashable type of list even if all the steps are same

[@gowrisancts]

Good one... actually u may need to use bernoulis naive bayes model as it deals with binary values 0 and 1...correct me if am wrong

[@arjyabasu1311]

Awesome work sir !!

[@aninditadas832]

hello sir, why have we not used lemmatization here? Stemming may or may not give meaningful words but we need meaningful words here right?

[@avinashsingh7698]

Sir, can you please make a video on 'Generate paraphrase from the text using NLP'.

[@yogeshprajapati7107]

To predict whether the new message is spam or ham.write this code. df=pd.DataFrame(['this message is a spam'],columns=['message']) corpus=[] for i in range(0,len(df)): review=re.sub('[^a-zA-Z]',' ',df['message'][i]) review=review.lower() review=review.split() review=[ps.stem(word) for word in review if word not in stopwords.words('english')] review=' '.join(review) corpus.append(review) df=cv.transform(corpus).toarray() pred=spam_detect_model.predict(df) label=pred[0] if label==1: print('Spam') else: print('Ham')

[@maYYidtS]

excellent........ sir instead of taking max_feature parameter at 16:43.....wt if we apply PCA or LDA on that total columns...

[@Anurag_077]

Wonderful

[@ashishgeorge2766]

can we apply label encoder instead of one hot encoding at label column

[@awaisniaz5300]

yes we can apply but when feature have two category

[@JoshDenesly]

Hi Krish, Please make a project relating to Bigram , unigram also . Thank you

[@krishnaik06]

Sure I will do that

[@tarunsubramanian9792]

TypeError: cannot use a string pattern on a bytes-like object Error shown when line 17-20 is executed..... How do I rectify it.... someone help please

[@efefmichelle]

I get the same error!! What did you do?

[@yogeshwarshendye4857]

How do you deploy sklearn models?

[@AarushiMishra-x3w]

can't we make this code work in jupyter notebook instead of spyder because i cant really see any output for spyder

[@Lijoperumpuzhackal]

I had gone through the 7 videos in the playlist . Well explained in every videos . Can you please tell me how can implement this program in real scenario ? Everyone has completing their videos by making only the models . So pls try to explain how we can use this model ? If I have text message. Then how to find whether it is spam or not using this model ..

[@krishnaik06]

Check my deployment playlist u will get to know

[@pradeepvaranasi]

Can we just use an if-else condition on the label column to derive the 0-1 (spam-ham) column? What is the purpose of using the get_dummies function for a binary class column?

[@dushyanthande1556]

sir i have tried running the code but the shape of x function and y is not the same so train test split is not working its saying Found input variables with inconsistent numbers of samples: [11144, 5572]

[@aishwaryabhargava6554]

how can we check the model on user provided input?

[@pinkalshah5237]

With lemmatization and max_features accuracy is 97%

[@akashr9973]

Hi sir, please correct me if I'm wrong. In the line number 30 you are applying the transform function for the whole data , won't it be data leakage? The transform has to be applied after splitting the data right? Thank you.

[@КаратэПацан-я6б]

Hi. The CountVectorizer is not a ML model, it just converts to vectors(matrix of numbers)

[@abhishekpurohit3442]

Sir, is Deep learning necessary to be learned before coming to this playlist (as I see Keras and LSTM being there in the last videos)??

[@ashwinbj]

practically how to check weather the message is spam or ham.? ie how to pass the message in the mode.

[@premranjan4440]

We could have used drop_first in get_dummies label instead of iterating the whole array.

[@abhishekpurohit3442]

Sir, why did we go for Bag of Words and not for TF-IDF? Is TF-IDF only used for sentiment analysis?

[@salmankhan-vq7pc]

Hi nice lecture. I have a dataset with 1.3 million rows. I used your code When I perform bagging of words my Google Collab get crashed. Any solution.

[@gemrose455]

I don't have a dependent variable like "spam" in my imported document. How will I train the dataset

[@sardar92]

very nice kindly post new videos

[@techbenchers69]

Sir, What is the reason behind choosing navie bays classifier.why not other classifier

[@thunuguntlaruparani2058]

There wouldn't be a data leakage problem if we use fit_transform on entire data?

[@Sonu-bc7sm]

Hi Krish, Nice video, Just had a question. What if i put the model in production and the new message have a word which are not part of my training dataset then the features won't match and the model will give error?

[@yashwanthsrinivas4590]

Hello Krish,How can we handle mulitple label classificaton problems?

[@suhailhafizkhan9800]

How can we visualize at the actual result for a clarification?Thanks

[@deepakjoshi4699]

I tried with Tf-IDF but my score is better with bag of words ? is it possible or am I making some mistakes?

[@monicameduri9692]

Thanks a lot!

[@kanishkapatel9077]

How to make GUI for this project ? any idea about it? It would be of great help !

[@Rohan-cw9gn]

U can use stream lit framework without knowledge of html, css u can make beatiful web apps

[@Anandasys]

hello sir if we have different number of labels or category such as business,sports, entertainment,category,politics,tech,history.then how can we get the dummy variables and bag of words and how to find which present are the which labels.

[@furkhanmehdi6405]

Legend ❤️

[@praddhumnasoni3364]

sir, how to predict on real world text ( means text from gmail or something).

[@parimalbhoyar8579]

very helpful...!!!

[@karthikelangovan5006]

i have typed code which you explained in this topic "Implementing a Spam classifier in python| Natural Language Processing" but i am not getting the corpus list... i got empty corpus ie.. [' ', ' ',' ', ............]

[@datascience3008]

Awesome

[@insightworld9910]

By using lemmatization method we get accuracy of97.6