Silhouette (clustering)- Validating Clustering Models- Unsupervised Machine Learning

[@arjundev4908]

So to brief about clustering algorithm.. 1. its a unsupervised Machine learning algorithm. 2. To find out number of clusters we use "Elbow method". 3. Silhouette(si-lo-wet) scores helps in finding whether the data points in a Clusters formed belongs to respective cluster or not. formula is b-a/max(b,a) where 'a' is distance within the cluster and 'b' is distance between the cluster... if ab and the silhouette scores are negative and highly unlikely that clustering formed is incorrect.. Please add points if missed or correct me if i am wrong..!!

[@shubhamthapa7586]

what do you think of DBSCAN and HDBSCAN silhoutte coefficient wont work there cuz we dont have no of clusters as a parameter ?

[@sampadkar19]

@@shubhamthapa7586 do we actually need k ?? beacuse hiearchical and DBSCAN both form clusters so we can easily do it using the formula

[@NashatJumaah]

Thanks for the easy to follow tutorials...Big Love from Iraq

[@chintansoni6370]

Thanks Krish. Much better understanding. Really appericiate your efforts to provide knowledge by creating videos.

[@ramendrachaudhary9784]

pretty much simple and pretty much amazing explaination. Thanks Krish

[@kushagrak4903]

Sir i have doubt like , in your example the silhouette value of 2 greater than 4 , so how can we decide whether k = 4 is good than k=2 without implementing any diagram .

[@matheusgoes640]

Thank you so much for this video! Great explained!👏

[@sihammohamed7480]

Thanks Krish Awesome explanation :)

[@mprasad3661]

Awesome explanation bro

[@stevemungai3542]

excellent explanation

[@DeepGamingAI]

Loved it, very nicely explained :)

[@DineshBabu-gn8cm]

Excellent explanation Krish

[@naufalsiregar9662]

I just saw your video, this is great... Thanksss

[@amitgupta-tb9td]

Sir plss make more validation techniques for unsupervised learning...!

[@kamilc9286]

Sir, if you'll run the number of negative_samples k=4 has 1 (not visible on the chart), so k=2 has score 0,74 and 0 -ve and k=4 has score 0,65 and 1 -ve. Only elbow method suggests k=4.

[@1potdish271]

Very nice explanation. (y)

[@DevanshKhandekar]

Can be better explained with terms like ' Intracluster distance ' and 'Intercluster distance'

[@krishnaik06]

Yes absolutely right

[@iqbalsaviola6052]

Something that im looking for

[@ashulohar8948]

Plz make vedios on performance metrics of regression algorithm

[@पंकजकुलड़िया]

at 18:02 there is a negative value for Cluster no.-2, so I think k should be 2, Please clarify sir, Also thanks a lot for making such kind of video and Virtual Interview sessions...

[@mohammedameen3249]

but as elbow method used which sow k =4 is the best

[@sajidchoudhary1165]

Sir Please make video on xgboost math intuition for Regression and Classification Please Sir Please

[@sajidchoudhary1165]

@Karthik Vishwanath yes i was watching but i couldn't understand some hyperparameters like gamma, lambda, cover

[@prasadshiva3538]

thats great lecture Krish, but what if ,we got k=5 in Silhouette but k=4 in Elbow, how to conclude this the correct k value

[@sandipansarkar9211]

finished watching

[@thankyouthankyou1172]

8:18: why there's no mention about the min()?

[@krishnag5734]

The min is used to find the nearest neighboring cluster to the current cluster Ci. To get it, compute b(i) for all the clusters from a point in the current cluster and get the min value. Whichever cluster holds that min value, is the nearest neighbor to the current cluster. Let's say you have 5 different clusters, and you are computing b(i) to the rest of the 4 clusters from the 1st. you will have 4 different b(i) values. Whichever value is minimum, the corresponding cluster is closest to the first cluster.

[@sandipansarkar9211]

Superb explanation. Need to get my hands dirty with Jupyter notebook.Thanks

[@padmanabh7031]

Bhudde

[@divyamadhuri126]

That was super helpful. My doubt is that if my clusters are overlapping, what should I interpret? Are my data points poorly clustered?

[@vignesh7687]

Thank you Krish, But I could see a small negative value for K = 4 in the plot.

[@noobgamerstamilyt9963]

yes bro there is some small negative value in it bro

[@rhiothelab5251]

Thanks, Sir for all the content, Sir suppose I am using DBSCAN and it gives only one cluster then how to measure the correctness of cluster

[@MrChudhi]

Please can you explain how we can use adjusted RAND score for a K-Means models.

[@sakshamshivhare2474]

Hi krish , i wanted to ask in an interview i was asked how to interpret the results of K means clustering and how to label the results. can you or anyone help me out with this question

[@rishisingh5581]

Sir community class aaj launch hogi ki nahi?

[@sukshithshetty4847]

has this code been shown in some other video from scratch ?

[@rezamohammadi3096]

Hi @Krish Thanks for amazing tutorial. I'm using k-prototyps library (for mixed numerical and numinal data type) and I want to calculate Silhouette Index to compare my clustering results with previous studies (e.g. k-medoid). Could you please give me a clue to calculate Silhouette Index in my case?

[@saisidhartha2855]

This is the same case with, me kindly let me know whether it is possible. I have used elbow method for k-prototype to determine the K value. Looking forward for shilloute method also

[@zama-sarib]

@@saisidhartha2855 We can use gower distance with gower_distance as precomputer metric in silhoutte sklearn. Gower distance typically works well with mixed data like numerical and categorical data types.

[@adotac]

Can someone tell me what software he used to draw on screen?

[@anbarasanpm3295]

How the point is chosen in Cluster 1?

[@rohitkamra1628]

What can we expect on discord server?

[@yashsethi2402]

What is the name of the plot that you created?

[@sagemaker]

Pronunciation: Sil-who-at

[@varnikareshma1873]

sir, instead of using Euclidean or Manhattan Distance can we use cosine based distance. If it is possible can u please hint me how to use it.

[@ashulohar8948]

Cosine is for text data

[@soniakashyap001]

Hi Krish, when i used to search to find any method by which we could check validity of cluster i used to get only elbow methods in search. I never came across links related to silhouettes. How did you find this method?

[@mranaljadhav8259]

Try to check all the link provided by google, i think you opened only one link. I search like you "validity of cluster" , I got the right methods like silhouette, dunn index etc

[@aimem246]

thx

[@hicodeguru]

X has 500 features, but KMeans is expecting 1000 features as input. ??

[@sambit123sahu]

what is the value of |C| ?

[@ayushgoel9584]

community classes ka kya hua ??

[@krishnaik06]

Will be live in some time

[@chandinisaikumar2736]

Can we use this evaluation method for DBSCAN

[@shubhamthapa7586]

no i dont think so

[@chandinisaikumar2736]

@@shubhamthapa7586 Can you please let me know the evaluation methid for DBSCAN Thanks in advance

[@shubhamthapa7586]

@@chandinisaikumar2736 in current scenario there is no best evaluation metric available for DBSCAN however you can use silhoutte coeffecient for a refrence but you need to optimize the parameters of DBSCAN first which is hard as compare to KMeans clustering , honeynet.github.io/cuckooml/2016/07/19/clustering-evaluation/ here they make a good use of silhoutte coeff in DBSCAN

[@manikprabhug412]

Hi sir

[@oscarfamousdarteh189]

Sir , please could you show another video with an uploaded csv data ? I'm finding some difficulties

[@raviyadav2552]

si-low -et

[@TJ-wo1xt]

dont pay wikipedia, not everything that is there is correct.

[@diosmorbodiosmorbo9547]

It is indeed. That thing of "in wikipedia anyone can writte anything" is false. You have a very estrict and dedicated community validating every time someone writtes smh. And with new topics like ML is even more. So stfu