What is TF-IDF for Beginners (Topic Modeling in Python for DH 02.01)

In this video, we explore TF-IDF, or Term Frequency-Inverse Document Frequency.
If you enjoy this video, please subscribe. I provide all my content at no cost. If you want to support my channel, please donate via
PayPal: www.paypal.com/cgi-bin/webscr...
Patreon: / wjbmattingly (its my www.themedievalworld.com account as well).
If there's a specific video you would like to see or a tutorial series, let me know in the comments and I will try and make it.
If you liked this video, check out www.PythonHumanities.com, where I have Coding Exercises, Lessons, on-site Python shells where you can experiment with code, and a text version of the material discussed here.
You can follow me at:
/ wjb_mattingly

[@olucasharp]

What a treasure this is! ⚡Many thanks! So interesting and I've even managed to use some of the ideas at work already 😀

[@python-programming]

I am so happy to hear that!

[@pritamsarkar2075]

this channel is a beauty

[@amirrahimi6979]

This is really useful. Thank you.

[@python-programming]

No problem!

[@nazmusas]

You are the best. You are so cool.

[@stevedavis3813]

This is great! A++

[@python-programming]

Thanks!

[@mehmetkaya4330]

Thank you so much!!

[@python-programming]

No problem!

[@oliviern.2095]

very clear sir

[@python-programming]

Thanks!

[@olucasharp]

Hi, I have a question if you will (still trying to figure out what are the best ways to use all there different methods): I got some data re requirements for the data analytics in finance from job postings website and wanted to get the sense of what are the most wanted requirements (skills, knowledge) are among those. Now I'm on my way to explore all the methods you explain based on this corpus but it seems that probably for the purpose of summarizing a bunch of similar job requirements' descriptions it's better to use something like key words (mostly threegramms) extraction. So would KeyBert be your choice? Sorry for the long question )

[@python-programming]

I think KeyBert may be a great option. Out of the box, it will do a lot. It really dependa on the data, though. No two corpora are exactly the same. It will require a bit of experimentation.

[@olucasharp]

@@python-programming huge thanks for your comment, indeed, from the result I get I can better understand where to go further) Looking forward to hearing more from you on this channel on these existing topics and the ways to use them in different contexts. Thanks!

[@ANUbhav918]

Good

[@python-programming]

Thanks!

[@ry2743]

if i have tweets is it the best to use it for?

[@dwisetyoaji5007]

sir how to access the website?I wanna read some more of it thanks

[@feroncia]

if we only have one document that is compiled all our text, will TF-IDF useful?

[@python-programming]

Yea it can still tell you the most common words within that document, but for that I would use KeyBERT

[@ayanjain3106]

Wouldn't the IDF score be same for all documents, why do we need to multiply every time with TF score if we just want comparisons?

[@python-programming]

Great question. Not all docs in a corpus will have a word. The IDF places a poportional assessment on that word relative to density in a single document against all relevant docs in a corpus. If you just compare TF alone, you would not get a sense of the docs larger place.

[@ayanjain3106]

@@python-programming Got it! Thank You!

[@python-programming]

@@ayanjain3106 no problem!

[@ANUbhav918]

You can say that you are comparing after normalizing

[@khadimhussainmalik3284]

The corpus may contain various types of documents, such as newspapers, which will enable us to understand the extent to which the term varies across different kinds of documents.

[@mehmetkaya4330]

Thank you so much!!

[@python-programming]

No problem!!