HDBSCAN, Fast Density Based Clustering, the How and the Why - John Healy

[@benhurrodriguez1807]

Presentation Skills: 100000/10

[@RajatSaxena35]

Presentation Skills: 10/10

[@stanleykurniawan3053]

+ 1000 aura

[@reocam8918]

Nice presentation, I see 200% confidence and eloquence

[@alexanderdevaux661]

this is exactly what I have been looking for! great presentation.

[@-beee-]

Wow, what a great talk! Love the intuitive explanations and visuals. Super helpful. Thank you!

[@21rufus21]

Absolutely fantastic presentation, thank you

[@MrRaisin56]

Wow I love the enthusiasm! It really makes it so much nicer to watch. Very insightful as well thank you very much!

[@vunder8737]

This truly was a wonderful presenter, would love to listen to him on other presentations

[@hannahnelson4569]

A very impressive presentation and algorithm! Thank you for teaching all this!

[@alaaelhadba7310]

Thank you so much. It was exactly what I was looking for 🎉🎉

[@valeryzuev3957]

15:30 there might be a misprint in the formula: d(X_i, X_j), not d(X_j, X_j)

[@jiayangcheng]

Love the presentation. Great work!

[@opelfrost]

thanks a lot, learn a lot from this presentation

[@pankajgoikar4158]

Awesome presentation.

[@honey-py9pj]

what an amazing speaker!

[@maximillianweil2672]

Thank you for the super interesting talk! I was wondering if you have worked with the new HDBSCAN integrated in sklearn 1.3.0? Is it possible to draw the cluster tree with this implementation?

[@RoulDukeGonzo]

Any luck?

[@vampierkill]

Sorry has to comment because of the kiiiiiiick ass animation! Brilliant.

[@danaizenberg2402]

great talk

[@TrixieFromSanFran]

The coloring of the tree at 14:00 is needlessly confusing. See figure 3a in their paper McInnes & Healy 2017 to clarify things

[@sushilkhadka-iu3gf]

that was a great talk!

[@edwardmalthouse973]

Thank you for your presentation. It was very helpful. I'm not sure about the claim that k-means requires small amounts of data. I believe K-means is O(n) (assuming a small number of dimensions and iterations) and I have used on very large data sets without problems. I would also like to respectfully push back on the spherical cow comment. While it certainly depends on the domain, in social science and business applications with large, noisy data sets, the spherical, or at least elliptical, assumption often works very well, and produces better assumptions than the more nonparametric algorithms. It's easy to construct mathematical examples with odd-shaped clusters, but I've not encountered them in practice, although it could just be due to the domains I work in.

[@daisyondwari9795]

👀

[@nihshrey]

Amazing

[@ahmedayman2380]

can someone tell me about his linkedin or his full name please or how to connect to him

[@RoulDukeGonzo]

0:24 name and email

[@RoulDukeGonzo]

Any idea why the GPU version of this method can't take a pre-computed distance matrix?

[@scatteredvideos1]

There is a RAPIDS version of HDBScan. I'm personally struggling to get dependencies working together but it does exist

[@RoulDukeGonzo]

@@scatteredvideos1 I think that's what I used... Anyway, I'll give it another go.

[@scatteredvideos1]

To be honest the speed up really isn't even that great, it's only partially parallelized with GPUs. It's better just to reduce the dimensionality of your data, PCA to 95% of explained variance, and then UMAP to 10 or so dims, then cluster using HDBSCAN. I've found doing a grid search over a bunch of different HDBscan parameters can be helpful if you aren't getting perfect clustering.

[@scatteredvideos1]

With 10 UMAP dims and 184k data points my cluster is done in about 7 s on a Google colab high ram CPU instance

[@RoulDukeGonzo]

@@scatteredvideos1 I haven't tried GPU accelerated HDBSCAN, but for other clustering algorithms, the difference between CPU and GPU is night and day (so I was expecting it to be so here). I'm clustering embedding data from LLMs so it's extremely dense and uncorrelated, so PCA hasn't been much use (at least in my hands).

[@pahulhallan]

27:50 Installation

[@0MVR_0]

clustering is highly driven by the formatting of how the data relates to itself and is near impossible to accomplish using a single method of approach.

[@RoulDukeGonzo]

Agree, but in practical terms, where do you start?

[@0MVR_0]

@@RoulDukeGonzo An intimate descriptive knowledge of the data is recommended.

[@laughingsaeed]

I don't why he's talking so fast! Is someone after him and he needs to run away?!