I think you should use hard clustering algorithm like k-means or hierarchical clustering for strict topics because topic modelling is a. Soft clustering approach

Thank You Manish for the reply but could you further elaborate on what is meant by soft and hard techniques

@@HarpreetKaur-qq8rx to my understanding, a hard clustering would assume all the documents in a corpus have the same probably of showing each of the topics. Each document is assumed to only show one of the topics and all the words in this document are assumed to show this topic. A soft clustering assumes each document has different probabilities of showing each of the topics. And a document shows all the topics rather than one. A word in a document shows one of the topics and the words in a document may show different topics.

its good enough if you have to deal with hundreds of documents containing thousands of words each

Sure; but what is it good at ? What is the semantic value of the (let's call it a cartesian distance) between two LDA signatures ? I know what I'm talking about. I worked for a couple of years on an LDA-based classification project and the semantic value of the topics extracted from the documents was too general to be truly useful. I think Blei et all have found an interesting statistical method and a cool idea, but what they fail to express in this entire approach is precisely in what way their metric and the methods by which they choose words yields any meaningful insights on the analyzed texts. I find this whole thing very superficial. Without connecting your word net to some semantic ontology, you are doing nothing but an arbitrary match; arbitrary in the sense that meaning in language occurs in more complex ways than with individual nouns, vebs and adjectives.

I'm a noob on this, few weeks into NLP and i'm trying to solve a usecase and i'm hitting exactly this issue. Ultimately LDA just gives me a bunch of topic ids with words that dont mean anything together. I read that i have to name the topics myself ! And i landed here looking for a 'solution'. hmm .. i'm not the only one. Meanwhile i found something interesting ..dont know its worth, ieeexplore.ieee.org/document/6405699/. This introduces the term 'concept' between topic and word - Could not find any implementations as yet.

Pritish N I applied lda to public speeches and was able to compare results to manual results (i.e. people read the speeches and distinguished the main topics) and lda performed rather well and discovered 12 out 15 distinct topics. For instance health care topic had such words as health care afford insurance cost at the top, so u won't confuse it with anything else. I also have a few topics that are hard to interpret, but it gave me the main topics I needed across 6000 documents. I need to mention that in addition to stopwords I had to exclude about 30 other words that were frequent but noninformative, such as year state always because etc, these will depend on your area, of course, but they pollute the results, and the exclusion helped a lot.

The problem is that the whole concept of the "topic" is grossly inflated. It has very shallow semantic value. A topic is a broad and ambiguous category.

Hi Shiya, please find an Introduction to Design Informatics here: kzbin.info/www/bejne/qWrHnI2phdmWbJI Hope this gives you the information you need!

Thanks!! I'll check it out!! And i have another question.. Is DI (MSc) a STEM subject? I mean I'm not sure what fields MSc DI focuses on mainly.. human factors? design principles? CS? or AI? cause i found that the compulsory courses are not about tech at all.. but this programme is at the School of Informatics right? I know students who major in DI have option courses at Informatics but they can only select 40 credits.. compared with required courses at ECA, that's too less I think.. if this programme is a STEM subject.. I'm really a little confused about this so I'd super appreciate it if you could help me figure this out but it's also okay if you do not have time to reply me😆 Anyways thank you for your lovely reply again ~

Hi Shiya, thank you for this! Did you have the opportunity to have a look at the Design Informatics website? www.designinformatics.org/postgraduate There is plenty of information which will point you to the right places (such as the degree description www.ed.ac.uk/studying/postgraduate/degrees/index.php?r=site/view&id=803) and the right people to contact (please see at the bottom of the page).

Yay I've seen these before and I want more information about this programme.. anyways thanks a lot for your time and patience ~