Stemming and Lemmatization: NLP Tutorial For Beginners

Stemming and Lemmatization: NLP Tutorial For Beginners - S1 E10

Рет қаралды 67,205

Күн бұрын

Пікірлер: 42

@codebasics 2 жыл бұрын

Check out our premium machine learning course with 2 Industry projects: codebasics.io/courses/machine-learning-for-data-science-beginners-to-advanced

@Breaking_Bold Жыл бұрын

I love the way you explain - other NLP concepts - customizing the pipeline for example !!!

@ayushgupta80 9 ай бұрын

Stemming (removing something) vs Lemmatization ( mapped with base word) 4:50 Note : Spacy don't have support of stemming . Code : stemming import nltk import spacy from nltk.stem import PorterStemmer stemmer = PorterStemmer() words = ["eating","eats","eat","ate","adjustable","rafting","ability","meeting"] for word in words: print(word,"|",stemmer.stem(word)) -------------------------------------------------------------------------------- Code : lemmatization nlp = spacy.load("en_core_web_sm") doc = nlp("eating eats eat ate adjustable rafting ability meeting better") for token in doc: print(token,"|",token.lemma_,"|",token.lemma) ----------------------------------------------------------------------------------------- Custom lemmatization Code : ar = nlp.get_pipe('attribute_ruler') ar.add([[{"TEXT":"Bro"}],[{"TEXT":"Brah"}]],{"LEMMA":"Brother"}) doc =nlp("Bro, you wanna go ? Brah , don't say no ! I am exhausted") for token in doc: print(token.text,"|",token.lemma_)

@jatinnandwani6678 Жыл бұрын

Thanks so much

@amandaahringer7466 2 жыл бұрын

Very helpful! Looking forward to the rest of the series! Thank you!

@belfloretkoriciza5279 2 жыл бұрын

you are my teacher and i am proud of you

@codebasics 2 жыл бұрын

Thanks 🙏

@pphantom5037 3 ай бұрын

There is a quiz now!! thank your for your awsome work♥♥♥

@Breaking_Bold Жыл бұрын

Fantastic ...you make complex NLP topics simple. !!!

@codebasics 2 жыл бұрын

Do you want to learn technology from me? codebasics.io is my website for video courses. First course going live in the last week of May, 2022

@amandaahringer7466 2 жыл бұрын

8:36 I noticed that the prebuilt language pipelines return an unexpected lemma for "ate". I assumed that lg and trf pipelines would produce ate -> eat while the sm and md pipelines would produce ate -> ate, but that doesn't seem to be the case. def eat_lemma(lang_pipeline): nlp = spacy.load(lang_pipeline) doc = nlp("ate") print(lang_pipeline, '|', doc[0].lemma_) lp = ["en_core_web_sm", "en_core_web_md", "en_core_web_lg", "en_core_web_trf"] for lang_pipeline in lp: eat_lemma(lang_pipeline) en_core_web_sm | ['eat'] en_core_web_md | ['ate'] en_core_web_lg | ['eat'] en_core_web_trf | ['ate'] Update: I see that when "ate" is used in the context of a sentence each pipeline produces a lemma of "eat". doc = nlp("The person ate an apple.") en_core_web_sm | ['the', 'person', 'eat', 'an', 'apple', '.'] en_core_web_md | ['the', 'person', 'eat', 'an', 'apple', '.'] en_core_web_lg | ['the', 'person', 'eat', 'an', 'apple', '.'] en_core_web_trf | ['the', 'person', 'eat', 'an', 'apple', '.']

@aintgonhappen 2 жыл бұрын

This is some quality content. Thank you!

@arnavverma8622 2 жыл бұрын

Excellent Series👌👌🔥🔥

@sandeepnaik6437 2 жыл бұрын

What is Behavioural data science?

@apurav363 2 ай бұрын

Very helpful

@rajiv7 5 ай бұрын

You are the excellent. Fullstop.

@Kaafirpeado54-6ayesha Ай бұрын

Thanks a bunch ❤

@MuhammadIBRAHIM-iy3rg 8 ай бұрын

amazing videos

@aashishmalhotra 2 жыл бұрын

If possible try to come with live sessions it would be helpful

@berkayates6254 9 ай бұрын

Hey Guys when we used stemming and lemmatizing before training the data we just change the words. After training the model model could generate words that are different from lemmatized words. I mean we teach the model `eat` however the model learn also `ate` how?

@muzaffariqbalraja6464 Жыл бұрын

very nice

@raphayzia9214 2 жыл бұрын

Sir it will be very helpful if you make a NLP project like a Chatbot at the end of the series and thanks for making this series

@codebasics 2 жыл бұрын

Yes I will be making few projects

@omarsalam7586 Жыл бұрын

thank you, sir

@firdospathan3700 2 жыл бұрын

I could not unable to install Ai4bharat package in PC. Is there solution. For that error

@zaytech528 2 жыл бұрын

hello sir, if i want to stem and lemmatize my string at the same time, how'd i do that? as spacy doesn't allow stemming. and nltk doesn't allow lemmatization. pls answer asap

@JayShah-m1v Жыл бұрын

Hey! Firstly, this is a very good series. But for the exercise, in the last part using lemmatization, some of my words such as cooking were converted into cook and playing to play while running stayed as it is. Do you know what could be the issue? Or do you have any explanation to this? Thank you.

@agastyabose1645 10 ай бұрын

it just might be how that specific model of nlp you used, performs. maybe idk