K-Means Clustering Algorithm with Python Tutorial

[@chromepunk-mm5up]

criminally underrated channel! Your explanations are superb

[@abdoulazizmahamadouhamidou2244]

Thanks ! I am geoscientist just starting my data sciences journey and I find your videos very helpful

[@JosuéEstevane]

Please can you help me I want to know more about data sciences applying in geosciences

[@letsjoinhands]

Your fluency and skill, simply superb! Keep it up!

[@AndyMcDonald42]

Thank you! 😃

[@beyzamutlu7379]

That was the best explanation what i watch for KClustering thank you 😊

[@JeanLouisKali]

Great presentation. The clearest I've seen on KZbin, to date. 👍

[@moaiedbetamour6078]

Very nice, simple, clear and to the point. Thank you for sharing.

[@AaromGuillaume-er8pe]

Explained this better than my professor. Big W

[@ahmetatasever8315]

Hi, I have one question about scettering in 13:21. Why were 'NHPI' and 'RHOB' written in 'plt.scatter()' when all calculations were done according to scaled data (I mean 'NHPI_T' and 'RHOB_T')? I am just trying to learn it. Could you please help me?

[@AndyMcDonald42]

Using the scaled data within certain algorithms can reduce the effect of different data ranges (e.g feature1 ranges from 0 to 1, and feature2 ranges from 0.1 to 10,000), and scaling can also help speed things up. Some algorithms such as decision trees/random forests don't really need scaling whereas Neural Networks and even clustering can benefit from this process. Plotting the data using the original curves allows us to see how the calculated clusters align with the original data. If we were using scaled data, then the numbers on the axes wouldn't make too much sense for petrophysical interpretation. Hope that helps :)

[@ahmetatasever8315]

@@AndyMcDonald42 Yes. It helps. :) Thank you very much. Also I have other question. Is there any way to get information about point in the graph by click using mouse to see which point belongs to which data?

[@AndyMcDonald42]

@@ahmetatasever8315 Yes, there certainly us, The plot shown in this video was done with matplotlib, which is used to create a basic and static figure. You could easily swap that out for Plotly, which will have the extra interactivity and give extra info on hover.

[@ahmetatasever8315]

@@AndyMcDonald42 Thank you again :)

[@SouthwestStet]

This was such a fantastic tutorial, thank you for putting quality content out there.

[@AndyMcDonald42]

Glad you liked it!

[@AntonyJudeDMello]

Thank you! The example script is a huge help

[@bb3132]

Andy - Your videos are very helpful and informative! Thank you!

[@AndyMcDonald42]

Glad you like them! Thanks!

[@MultiDrag90]

Excellent tutorial! Thank you very much for your time

[@elife9508]

What an amazing tutorial! Thank you!

[@Leopar525]

You’re a star. Thank you. Subscribed… very well explained

[@tylerpargiter642]

very useful thank you! I'm midway through a data analysis apprenticeship and this helped me alot!

[@AndyMcDonald42]

You're very welcome! I am glad to hear it has been helpful.

[@calfredie0170]

Amazing video you have put together here. I enjoyed how clear you were as well as the pace you took to go through the steps and explain everything. I am new to this kind of thing so does anyone have resources on where I can learn how to interpret cluster graphs

[@guanyilu5498]

hi , thanks for the video, but could you please direct me that which file in your github is the jupyter notebook for this video? I could not find it. thanks

[@allansalles8895]

Thanks again for the content, Andy! You're a great teacher!

[@AndyMcDonald42]

Thanks Allan. Glad to hear you are enjoying the content.

[@nekohanhanrin]

Thanks buddy, your lesson helped me a lot

[@timothysham6409]

Andy, thanks for sharing. I can’t find the notebook for this specific exercise. I am trying to follow along with a different dataset but I am getting an error “name ‘means’ is not defined” when trying to determine the number of clusters.

[@AndyMcDonald42]

Hi Timothy, did you manage to resolve this? If not, I would go back and check you have ran all of the cells before trying to determine the number of clusters.

[@olaal-najjar7391]

Absolutely useful. Thank you Andy

[@AndyMcDonald42]

Great to hear!

[@kkamalpha]

Thanks! I have been doing this on resistivity and seismic values on different profiles in a catchment. However, everytime I get same trend but clusters change in their places. Would like to know about this issue...

[@craigsmith941]

Hi Andy, this was a great tutorial as it's something I would like to try on a csv file with various metrics in the design of a pharmaceutical. I have one question though: I will be wanting to use 5-7 columns on the csv file for clustering - how do you go about visually representing this? I can't think of a good way to do it. Thanks!

[@789Moonlight]

hi. thank you for this wonderful tutorial. where do you recommend choosing data sets from?

[@josedavidbastoaguirre2099]

pretty cool. I have used K-means and DBSCAN to identify electrofacies, but I am still working on a way to optimize this task. It would be grade to see the Well Plots (depth Vs logs) with each point identified by its own cluster.

[@AndyMcDonald42]

Thanks Jose. I did have a section of code for displaying the facies data on a log plot but I did not include it in the video. The full plotting code can be found here: towardsdatascience.com/how-to-use-unsupervised-learning-to-cluster-well-log-data-using-python-a552713748b5

[@laveshagrawal4241]

Excellent presentation and explanation is there a place from where I see the code you have written for this as that would help me in learning. Thanks

[@mohammadkeshtkar9655]

Hi Andy I think you start machine learning topic and it's my favorite topic thank you 🙏🙏

[@AndyMcDonald42]

I will be jumping between some Python topics and machine learning topics over the future episodes. Is there any particular algorithms you would like to see covered?

[@aboodfal4780]

I’ve searched for this file in the github repository and I didn’t find this tutorial’s code file

[@yesicamagnoli651]

Thank you Andy! I just want to ask you where can I find this notebook to download and work with it? Thanks again!

[@AndyMcDonald42]

Sorry for the late reply. I realised I hadn't uploaded the file to the repo. You can find it here: github.com/andymcdgeo/Petrophysics-Python-Series It is Notebook 18.

[@yesicamagnoli651]

@@AndyMcDonald42 thank you!! Please, keep on doing videos like this, I've been learning a lot!

[@TeeFat]

Thank you so much for this video. I downloaded the data you used and found a negative relationship between RHOB and NPHI. Can tell me how your scatterplot shows a positive relationship between them? Thank you.

[@AndyMcDonald42]

No problem. You are correct that NPHI and RHOB are usually anti-correlated. In petrophysics, we normally display RHOB on an inverted scale, often on the Y-axis. As RHOB values get lower, we likely have a higher porosity, and the values will plot higher up on the y-axis. For higher NPHI (neutron porosity) values, the points will plot further to the right. If we have a case where both NPHI and RHOB are high, they will then plot in the top right. It's a nice and easy way to visualise and identify potential reservoir intervals.

[@TeeFat]

@@AndyMcDonald42 Thank you so much. I am using it to cluster customer data, but I wanted to make sure I could replicate yours before trying. Thank you again for the explanation and such an awesome tutorial.

[@TheWraithkiller]

Has anyone had an issue with In[9] when running from Jupyter Lab? Have fully checked for any spelling errors. The assignment and new columns seems to try to access. KeyError: "None of [Index(['RHOB_T', 'GR_T'. 'NPHI_T', 'PEF_T', 'DTC_T'], dtype='object')] are in the [columns]"

[@tanishqrastogi1011]

do we only use 2 features of a data while using k means clustering or did you do it for visualization purposes?

[@shahzaibkhan7215]

Precise and clear👍👍plz explain naive based, Support vector machine & decision tree as well

[@alopix5468]

Hey! great video, only one question. What if I want to set my own centroids?

[@jialicai6096]

Thank you Andy, great video! What if I want to cluster more than 2 variables?

[@AndyMcDonald42]

In the .fit() call at 12:00 you would pass in more variables. I have just used 2 for this example to illustrate what the output is like. Hope that helps :)

[@cocoshih2948]

I have trouble using kemans.labels_ at the end it keeps showing this error: 'numpy.ndarray' object has no attribute 'labels_' can someone help me with this? Thank you!

[@akande88]

great tutorial, thank you.

[@abdolkarimmehrparvar6583]

I cannot find notebook file of this video in your git

[@압둘하미드이드리스]

Could you please share the link to get the dataset?

[@tsarm___]

I have problem when trying calculate using excel, the result is different with code, what can i do to fix it?

[@pattylu8568]

Thank you so much, Andy! I really find your video helpful. I am just wondering whether it would be possible for us to draw the scatter plot in multi-dimensions? Cuz I followed all of your steps but could not continue the step after the elbow plot when using my 500 columns dataframe.

[@AndyMcDonald42]

Thanks Patty. You would only be able to draw the scatter plot up to 3 dimensions (X, Y and Z). However, you could look at using Seaborn's Pairplot to view 2d scatter plots of each of the variables versus the others: kzbin.info/www/bejne/emangY2vnMpmZ8k I would be wary though of using 500 features with this plot as it will become unwieldy. I would be asking myself the following in your situation: - Do I require all 500 columns? - Are all of the columns relevant? - Can I reduce them manually or look at algorithms such as PCA to reduce the dimensionality of the dataset.

[@mostafakhalid8332]

An error is raised after writing (kmeans_3) while plotting (NPHI vs. RHOB)

[@mominabdlhamed2098]

What a great tutorial, thanks a lot🥰🥰

[@AndyMcDonald42]

Glad you like it!

[@ajithkhan7314]

Okay. So how to draw conclusion from these clusters ? I mean, what are your insights from this model ?

[@dragster100]

Can I say that at the end of the day, the way of interpreting the clusters is kind of subjective especially when the dataset gets more complex? Since the results could vary quite a lot as you apply different clustering algorithms or tuning some of their parameters. So it could be quite subjective, no?

[@AndyMcDonald42]

Yes. That is very true. It is down to you or the person doing the interpretation to understand what the cluster may represent. If another person does there own interpretation they may have their own understanding of what the clusters represent

[@youkendoit123]

Amazing video, thank you Sir

[@robikurniawan8507]

thank you andy for your sharing 🙏🙏

[@AndyMcDonald42]

My pleasure

[@abdullah.montasheri]

Thank you, Andy, I could not find the notebook in your github.

[@AndyMcDonald42]

I believe this may have been my original notebook. It contains much more detail than what I covered in the video. I hope this helps. github.com/andymcdgeo/Petrophysics-Python-Series/blob/master/18%20-%20Unsupervised%20Clustering%20for%20Lithofacies.ipynb

[@Joud3011]

I'm using k-means for the first time. my dataset has more than 400,000 rows and 11 columns, I run the k-means for k= 3, 5, 7, 9, and 10. it took more than 3 hours and still no output. is that normal?

[@lorenzos785]

I'm working on clustering energy consumption profiles of a group of households, how should the starting dataset be structured? For each apartment I'm given the annual energy consumption profile (15 minutes frequency for 1 year), the number of appliances and the number of rooms

[@AndyMcDonald42]

Sounds like an interesting task 🙂 If I understand correctly, you have a continuous variable for the energy consumption and then fixed variables for the rest? Have you considered clustering based on the profiles alone and grouping them into something like high energy users and low energy users or early birds and night owls? After that you could then try to use the other properties to gain more insights

[@AndyMcDonald42]

Maybe have a look at time series clustering techniques for grouping the profiles

[@hieunguyenminh1558]

how to create input and output lines? pls help

[@chottomtaki]

hello Andy, thanks for well-explained session,but on the final part can you assist to explain as to which features or measures differentiate one cluster from other,Thanks again

[@AndyMcDonald42]

Thanks Dominic. One way would be to use a facet grid plot from seaborn and split by the clusters. You could then view the data by histograms, scatter plots and other plot types. That way you can see how the data features vary per cluster

[@chottomtaki]

@@AndyMcDonald42 thank Andy,this is useful,I real appriciate

[@luisnazareth9193]

Andy, i get some NaN value on the datasets.. and then when i try to run the "df.dropna(inplace = True)", all of the datasets become empty (zero). How to handle this? Thankyou

[@AndyMcDonald42]

I would check if one or more columns are entirely nan.

[@rokaskrisciunas6015]

Why are they useful? Do we know what qualities does these clusters have? Are they meaningful if we have lots of variables?

[@DIRECTORZ-BOX]

i liked it, had to hit that belllll

[@vitorcastro42]

Solid video :) Btw, where is your accent from?

[@fiqihnurhadi1266]

sir, how to clustering data 2d with size(512,512), please help me sir tq

[@youorgan2361]

You are a hero!

[@Kittys_life0]

Thanks alot for your helpful videos..

[@AndyMcDonald42]

Glad you like them!

[@mafaldanunes774]

THANKS YOUUUUU AHHHHH SO HAPPY I DID IT

[@syifasyuhaidahazman2384]

very helpful . If you could use example that can be easily understandable for non-science community would be extra helpful!!!

[@katieweir4166]

Yasss! A fellow Scot!!!

[@katieweir4166]

It keeps saying name means not defined :(

[@eyo3303]

great content

[@thirteen174]

Thank you so much !!

[@stephenmackenzie9016]

Excellent thanks

[@AndyMcDonald42]

You are welcome

[@deividbejarano2386]

Where is the meaning the columns of Data?

[@dayansaynes6691]

Thanks a lot!

[@AndyMcDonald42]

No problem 👍

[@caothuydung]

thanks a lot

[@pixelkeckleon1171]

Too good

[@luckyramadhan346]

finally, a non-indian accent speaker