Natural Language Processing - Tokenization (NLP Zero to Hero - Part 1)

[@jawadmansoor6064]

Such a great lecture on NLP wow. I wish I had found it when it was uploaded, saving me two years.

[@abdulbasitnisar]

What did you did for two years? I mean which course?

[@Code4You1]

Simple and straight to the point I love it!

[@chiomaanyiam1138]

Wow! Thank you for breaking this down in such an easy way.

[@TheAdamSmithh]

Thank you so much! This is so informative, so quickly, in well structured lessons. I'm using a TensorFlow package for R and this helps me understand my project so much better!

[@muhammadsamy4081]

why are using r instead of python

[@chowadagod]

I've always been discouraged learning NLP ..But you've just made it a whole lot easier

[@laurencemoroney655]

It's a huge field, and I'm just scratching the surface. I hope it's useful! :)

[@muhammadhananasghar4326]

Best Explanation Ever. Best Sir I had ever listened

[@asadanees781]

Thanks Laurence Moroney are blessing for us! Awesome information

[@rabadaba7]

I love your videos! They are very professional and concepts are very clearly explained.

[@18lan]

If you are confused like I was why love receive index 1 then go to the end of that video where it's explained: Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Processing

[@nishalk781]

Thanks for making it clear waiting for the next one

[@laurencemoroney655]

Thanks!

[@Idontknowcode512]

Thanks you made it so easy for me to understand nlp 🙏

[@TensorFlow]

We're happy to hear that the video was helpful. If you'd like to learn more about NLP, check out the NLP Zero to Hero playlist → goo.gle/nlp-z2h

[@Idontknowcode512]

@@TensorFlow I have checked it. But I have one request can we build a model like chatgpt using tensorflow 🤔

[@coded6799]

This is a godsend. No other definition is possible.

[@rishibhatia5056]

Thanks for making.clear

[@kelvinsmith4894]

Lol, you explained this so well that it made me want to implement my own library for tokenization

[@rishavpaudel7591]

loooool I too had this feeling :D :D

[@singhanubhav]

For NLP freshers - this video is more about encoding than being about tokenization itself. Read about both topics separately before going through this video to better understand it.

[@srikrithibharadwaj6779]

Thank u so much 🙏🏻 such a great information.

[@LaurenceMoroney]

Welcome!

[@akshayshah483]

Yeah. Zero to hero back

[@laurencemoroney655]

For 3 episodes, and I'm working on another 3 for text generation to come out in the not-too-distant future. I hope!

[@biswanthpinnika7149]

we can also use tokenization for converting sentences to words

[@mattymallz4207]

Fantastic video! Very informative. Thank you for sharing TensorFlow!

[@laurencemoroney655]

Thanks, Matty!

[@mattymallz4207]

Laurence Moroney, I have a specific tensor flow question regarding beautiful soup and specifically gathering text from an html output. Is there anyway we could start a dialogue?

[@sharawyabdul6222]

Thank u so much , This is very well explained.

[@sunildingankar8657]

I was working in Marathi (Indian regional language) language for last 20 odd years. Since last 8 years I am working as a writer-translator. If I learn NLP, will I be able to combine Marathi linguistic skills and NLP skills in practical use? If yes, how it will be and where can I use it?

[@quadraticlife8314]

Incredibly amazing!

[@ronnierendel9503]

Amazingly well said

[@harikrishnanb7273]

Tokenizer is deprecated now

[@louiebeatty1872]

what's used now?

[@arpanghoshal6910]

He's a Tensorflow guru!

[@balachkhan1578]

Its great. Waiting for the next.

[@laurencemoroney655]

Glad you enjoyed!

[@georgesteele4838]

Excellent presentation.

[@819rajiv]

think you so much sir for grate videos

[@fahemhamou6170]

Thank you very much

[@benjaminkimmang1962]

quite informative. thanks.

[@TensorFlow]

Glad it was helpful!

[@BeGreatttt]

Great explanation, thanks a lot!!!

[@ashimkarki9652]

The legend is back

[@laurencemoroney655]

But you got me instead ;)

[@sharjeelzubair4106]

sentences = [ 'كم سعر الراجحي', 'ما هي قيمة الراجحي؟', 'هل تعرف سعر أرامكو؟' ] {'سعر': 1, 'كم': 2, 'الراجحي': 3, 'ما': 4, 'هي': 5, 'قيمة': 6, 'الراجحي؟': 7, 'هل': 8, 'تعرف': 9, 'أرامكو؟': 10} its putting الراجحي and الراجحي? as two tokens, is that becuase of arabic?

[@yousefsharrab1093]

Great introduction

[@muhammadyaqoob9129]

I need little more help; can you please mention the books you have followed? or Reseach papers? Basically, I am asking for References, so I read them by myself.

[@rahulbhardwaj4568]

Great, thanks for the info!

[@dannerrera]

Thanks for the breakdown! I have a quick question: My OKX wallet holds some USDT, and I have the seed phrase. (air carpet target dish off jeans toilet sweet piano spoil fruit essay). How should I go about transferring them to Binance?

[@PaulineLepre]

I appreciate your efforts! 🙏 I’ve got a question: 🤨 I have these words 🤨. (behave today finger ski upon boy assault summer exhaust beauty stereo over). What should I do with this? 🤷‍♂️

[@Promptgeek2]

Better explanation [imposible].

[@muskanjain1256]

@lmoroney I have come across the chatbot deployments recently. It is said that there is a problem with the continued conversation in the case of chatbots. But I have a query that why can't we add a lstm on a lstm model? I mean that if suppose we are able to provide a memory on sentences too along with memory on particular sentence then it may able to store the essentials of the previous conversations. Please help me with this query actually I am new to nlp and lot more excited to know.

[@eyasulencha5136]

amazing presentation.thanks dear for the info

[@narendrapratapsinghparmar91]

Thanks

[@ouissemmouheb5283]

Thank you so much!

[@harmitchhabra989]

So, its like markov lempel compression?

[@M_Zaroug]

🤩😍😍🤩 Very informative, waiting for the rest

[@laurencemoroney655]

Thanks, Mohamed!

[@oumelkheirofficial5216]

What an amazing and simple way of explication, Thank you

[@amaltej9372]

THANKS 😇

[@fakrulislam3140]

Amazing presentation

[@cloudlover9186]

Good one. Sir i want to know if nlp(" I have III years of exp"), if i check for ,_.ISNUM is not working, do we have any work around for this, is ROMAN letters will not be detected ?

[@ubaydullo_a757]

thank you, it was helpful :)

[@yami6499]

great video

[@theobellash6440]

Nice video

[@WassupCarlton]

perhaps this is coming in a later video, but is there any rhyme/reason to the integers that get assigned to the words? or is it PURELY arbitrary?

[@WassupCarlton]

ope -- looks like the more frequent you are, the smaller your assigned integer. Correct?

[@lencazero4712]

Awesome

[@meg33333]

Hello everyone So I am new to the ML NLP world. I need some tips my team is working on a project in which we want to convert text ( especially Hindi or Sanskrit) to a set of specific images. Which algorithm or model we go for or form where we should start we have made the data set but now what?

[@HuyNguyen-kd5vz]

Thiis is awesome

[@Ricocase]

Excellence! How do I leverage kMeans clustering to find similarities or segment sentences from one another?

[@_petrok]

Great introduction which is easy to understand. Can't wait for the next videos of this series! But is there any way to group words ignoring some grammar? Like: "He plays piano - I play piano" where "plays" != "play", but it basically is the same word and tempus. The part of ignoring the "!" in "dog!" is fascinating.

[@laurencemoroney655]

Yeah...that's a little more difficult in preprocessing text. I won't be covering that...sorry!

[@NelsonYalta]

Those are sub-words, and a different tools can be used for obtaining them, such as sentencepiece (github.com/google/sentencepiece). In this case the model searches for common sub words such as play and in case of plays it tokenizes as . It is also possible to tokenize as the character and as a sub-word.

[@mayankdewli1010]

yup ofcourse. you can lemmatize these words or stem these words

[@samrasoli]

useful

[@ipekbar]

Thank you for the video. Sometimes exclamation mark could be informative for tasks such as sentiment classification. But the tokenizer filters out. Is there way for preventing this?

[@vishnurajyadav8917]

did you got answer for this from any other source ?

[@ipekbar]

@@vishnurajyadav8917 yes, we can control by changing filters parameter www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text/Tokenizer

[@deepakdakhore]

Very nice

[@LaurenceMoroney]

Thanks!

[@harsh.vision]

print(Hashing == Tokenization ) whats the output??

[@shivibhatia1613]

Too good

[@ishanghutake1566]

Suppose if you have 30 textfile in one folder how do you tokenize the word?

[@mohithshivu5475]

Sir my name is mohith I am final year BE student can you help me out some doubt on nlp I am working on data generalization and data sanitization our task is identifying given text weather it is sanitized or not generalized or not how it work in python can you help out sir please.... it is helpfull to me

[@rameshsrivastavachandra]

This code has apache license, so can it be reused?

[@actu_r]

What are the advantages of using TF framework instead of other preprocessing method such as thoose spacy or nltk provides for example ? :) thank you

[@laurencemoroney655]

I can't compare with the others...but this way they're in a unified framework that makes it less code when I get around to training a NN with them (in episode 3)

[@yunishuseynzade5630]

Thanks you so much. But I have a question. How can I use words in other language than English. For example building a NLP in Azerbaijani.

[@தமிழோன்]

You need to find and download Azerbaijani corpus from the Internet. You can then prepare the word index using Tensorflow. The rest of the steps should be the same as the English example shown in the video. I don't know about the Azerbaijani language but some languages, like Tamil, don't have separate grammatical words like English. You need to make heaps of preprocessing before you prepare the word index. This is something you need to be aware of. Also, if you can't find a corpus for your language, use something called "hashing trick" (or "feature hashing") to hash the individual words in your language. Luckily, Tensorflow supports hashing trick.

[@oliverli9630]

Can this be called hacked? Or are there reasons that Keras doesn't include this? (Notice: "'you're", the left quote is still there, and it's got "'": 11 recognized as a word. and num_words=4 doesn't really limit the word count down to 4.) from tensorflow.keras.preprocessing.text import Tokenizer sentences = [ 'i love my dog', 'I, love my cat', 'You love my dog!', "Jack said, 'You're gonna love my cat!'" ] tokenizer = Tokenizer(num_words = 4) tokenizer.fit_on_texts(sentences) word_index = tokenizer.word_index print(word_index) {'love': 1, 'my': 2, 'i': 3, 'dog': 4, 'cat': 5, 'you': 6, 'jack': 7, 'said': 8, "'you're": 9, 'gonna': 10, "'": 11}

[@rajareivan2417]

i'm also wondering why this is the case, especially when i set num_words to be 1 or even 0 it still tokenizes all the provided words. have u got the answer for this?

[@sunanthakrishnan]

Could you help me with python 3.8.2 compatible version of Tensorflow and Keras.

[@JefferyCampos-r7z]

Ashleigh Park

[@aravindravindranatha4260]

I need your advise on finding the text similarity

[@xuantungnguyen9719]

Hi. What happens if I set nums_words to 0? I tried and it still prints all the words

[@renderdreality]

Does NLP only process english? Could it do another language? My question is really if it could be used to learn a different language as basis and go from there.

[@jinzo1171]

It can be used for any language:)

[@Acumen928]

Thanks. Where is the next episode?

[@LaurenceMoroney]

next week!

[@actu_r]

Should we keep only nouns when topic modelling ? I am quite new with NLP and it seems there is no clear universal thumb rule for extracting topics information, what would you advise ?

[@Aoitetsugakusha]

You can probably do decently well using just nouns, but you will probably also lose a lot of information if you filter out non-nouns at the Tokenization or pre-processing step. For example, if you only use nouns, you could very well pick up on a topic like "machine learning" in your dataset, but you might miss separate discussions of "deep learning," because "deep" is an adjective that would get filtered out and you would be left with just general "learning." An ultra-crude way you might augment this a bit is to instead do topic modeling on n-grams and keep only those n-grams that contain at least one noun, but I haven't tried this, so I can't assert it will actually work.

[@laurencemoroney655]

@@Aoitetsugakusha +1 Great answer

[@kumarvikas_134]

Depends what your objective is. If the end result is only centered around identifying entities then keeping NN/NNP may make sense(given your POS tagger is not making errors). It all depends upon the objective, for my use case I remembered I have extracted chunks of SVO phrases(Subject-Verb-Object) and then performed topic modeling, that had worked well for me, but I had made adjustments to my POS tagger to do this task well.

[@TallRiderX]

The colab is labeld as Course 3 - Week 1 - Lesson 1.ipynb - where can I sign up for the full course? Thank you!

[@laurencemoroney655]

The colab was adapted for one I wrote at Coursera, where it was course 3 in teh TensorFlow:In Practice specialization. There's more there. Otherwise, this is a 3 part series, with part 2 now on the YT channel :)

[@RS-vu5um]

Is the link to Part 2: Sequencing - Turning Sentences into Data available?

[@laurencemoroney655]

Yep, came out yesterday, check yt.com/tf for details

[@carlossegura403]

Love Tokenizers ❤️

[@LaurenceMoroney]

:)

[@danylobaibak317]

A question of ignoring the "!". It seems, the Tokenizer doesn't include "!" because it was filtered as punctuation. Let's assume, that we want to use punctuation and set `filters=''` for Tokenizer. In this case, Tokenizer is not smart enough to separate the token "dog" from the token "!" Here's the example in Colab colab.research.google.com/drive/1M6Nf-WQxorf_X9z2jFnCSJ_QjrY3i5BJ

[@LearnWithMilind]

How many languages are supported? Or only English is supporting.

[@LaurenceMoroney]

I've only tried English, but this technique should work with most languages. Try the notebook linked, and change the language and see what happens?

[@hajar2629]

thank you how can make the same example in my raspberry?

[@abail7010]

Medogb Medo exact the same way when you’ve installed python and TensorFlow

[@rupeshmalpani]

can 1679 15223 2 153692 be a word?

[@VibhootiKishor]

Cool

[@laurencemoroney655]

Thanks!

[@sujeeshsvalath]

How to detect difference between "I love my dog" and "l love not my dog"?

[@Metalocif]

Beyond the obvious "it has one more word", there are several approaches. One that is fairly easy is to have a list of all words in a language with their connotations (this can be found online), one possible connotation being negation. Then, you can write code that inverts the connotation of a word if there is a word that implies negation near it.

[@LaurenceMoroney]

If you have lots of sentences that are similar except for the word 'not', and label them accordingly. Then train a classifier like we do here, the 'not' would become a really strong signal towards the negative. Give it a try, instead of using the sarcasm dataset. The code would be very similar to this video.

[@vi.kran.t]

I want TensorFlow track jacket that you have wear

[@tcidude]

Шансон

[@tingnews7273]

Anyone can tell me what is first princeple method teach

[@LaurenceMoroney]

"From first principles" means teaching with zero (or at least very few) assumptions

[@tingnews7273]

@@LaurenceMoroney sank u , I hope I can figure it out

[@felixakwerh5189]

‘From first principle’ could also mean from the smallest to the biggest:from the known to the unknown basically it’s a way of breaking concepts down to the simplest form

[@douggale5962]

What? You end it when I was expecting you to at least say to put the 1's in the input layer. This is how you tokenize in general, nothing to do with AI.

[@HealthyFoodBae_]

Yay😅

[@alexanderpohl1949]

04:15 is really misleading for anyone watching this as their entry to nlp. There are too many steps missing that need to be talked about it in a 'Zero to Hero' tutorial series after this point, instead of jumping into sequenzing. Even steps before this point. I see why these aren't included (because these are not included in tensorflow). But at the same time, this is just setting an unrealistic standard. In machine learning terms, I'd say... This video is just mislabeled

[@laurencemoroney655]

...and what are these steps? With these videos and the codelabs, we'll have everything we need to build a simple text classifier, the beginnings of NLP.

[@தமிழோன்]

@@laurencemoroney655 Maybe he's referring to the clean up required for the grammar (like someone pointed out: play vs plays)? However, Tensorflow cannot include that in the library as he's suggesting. Because Tensorflow is not an English-only library rather a more generic one.

[@rawnakfreak3539]

Exam after 9 hours (⁠T⁠T⁠)

[@cr0wzzz]

The answer was simple all along. It's just dog

[@siddvideos]

Too late to the party Tensorflow!! It’s not 2010. Love the video though, thanks😎

[@LaurenceMoroney]

Ha! I can only produce so many....

[@masternobody1896]

This is so complicated

[@laurencemoroney655]

Even with these instructions? I've tried to make it as simple as possible and provided a colab to step through the code yourself. Don't know if it can be simplified any more.

[@balachkhan1578]

@@laurencemoroney655 Its great. When we will get the next tutorial?

[@laurencemoroney655]

@@balachkhan1578 We're releasing them weekly

[@balachkhan1578]

@@laurencemoroney655 TensorFlow can't be installed with Python 3.8. Will the issue be solved or i should switch to Python 3.7?

[@LaurenceMoroney]

@@balachkhan1578 It's constantly being updated...so keep an eye on www.tensorflow.org/install. Right now it's up to 3.7 on there.

[@Wanderlens197]

Very Difficult to learn

[@laurencemoroney655]

Even with these instructions? I've tried to make it as simple as possible and provided a colab to step through the code yourself. Don't know if it can be simplified any more.

[@tanvipurwar6048]

@@laurencemoroney655 It is a bit complicated the first time. But taking up a small dataset/project for nlp and then revisiting the video again makes everything a lot more clearer. Plus you pick up on things that slipped your mind the first time :)

[@khlatarbaatarchuluun787]

Thank you very much