SVM (Support Vector Machine) in Python - Machine Learning From Scratch 07 - Python Tutorial

[@mantidream8179]

Finally a straightforward guide. Thank you

[@Vote4U]

It is very usefull for teachers as well as students. Thank you sir

[@ozysjahputera7669]

Do you plan to offer a tutorial video on non-linear SVM with the kernel trick to extend the linear SVM to handle non-linearly separable data?

[@nikolayandcards]

As always, great video. Everything is getting clearer and I feel I am getting a better grasp of it. However, this SVM implementation considers linear separation of the data, am I right? For non-linenar separation, the math would be different, or am I wrong? I didn't see Kernel functions and Lagrange Multipliers being used in this implementation. Could you give me a bit more insight?

[@patloeber]

Thanks for watching. You are correct. This is the simplest of all implementations and works only for linearly separable datasets. For non linear boundaries you should introduce slack variables or apply the kernel trick. Here is a good explanation of all cases www.cs.toronto.edu/~mbrubake/teaching/C11/Handouts/SupportVectorMachines.pdf

[@nikolayandcards]

@@patloeber Thanks a lot! Appreciate it.

[@aadya3933]

Explained properly. Thank you for this.

[@patloeber]

thanks for watching!

[@ashiqhussainkumar1391]

I've already done linear and logistic regression Mathematics in detailed fashion,here the life becomes far easier ... Exact same concepts only hypothesis function differs

[@patloeber]

glad you like it!

[@shadabshd8328]

so helpful and clear

[@hirenkakkad3747]

Superb. Can you please share code or videos for other kernels like this? It will be a great help.

[@patloeber]

maybe in the future :)

[@mahmoodhonarvar3520]

it was Perfect man! I really enjoy

[@patloeber]

Thanks, happy to hear that

[@thecros1076]

can you please guide me with maths needed fro this any papers to read please i really need to get the cruks of the maths behind

[@huojinchowdhury3933]

Why do we multiply class labels with linear functions?

[@mangilipallylaxminarayana3757]

Thanks a lot man, this video is very beginner friendly. Liked and subscribed.

[@patloeber]

Thank you :)

[@MyChannel-vw9cy]

great work please keep it on...............

[@patloeber]

Thank you :)

[@dwardster]

You can add this code to the training loop to print the hinge loss cost every 25 iterations: if i % 25 == 0: cost = self.lambda_param * np.linalg.norm(self.w)**2 + 1 / n_samples * \ np.max(np.c_[np.zeros(n_samples), 1 - y_ * (np.dot(X, self.w) - self.b)]) # hinge loss function print(f"{i :

[@blackpearl2245]

what if you want to add a csv dataset sir

[@andrianarutyunov7532]

8:02 Just to make sure, that I understand it properly: Doesn't it have to be the k-th component of x_i instead of just x_i ?

[@patloeber]

Yes you are correct, thanks for the hint! This should be x_ik in this formula since it was only for one component here...Later we compute it with numpy in the vectorized form for all components in one operation, so only then it's x_i

[@andrianarutyunov7532]

Python Engineer thanks for such a quick response!

[@estebanbraganza1067]

Amazing content thanks!!!

[@pinakidasgupta3227]

could you make it for more than two classes?

[@tigrayrimey6418]

That is my question too.

[@abhinava3302]

Nice video sir. Can you make videos on Fuzzy classifiers like FCM, PCM, etc?

[@vigneshvicky6720]

Nice very nice💖

[@patloeber]

Thanks 🤗

[@LIMLIMLIM111]

When I test this model with sklearn iris dataset, I get accuracy of around 0.3. While using the sklearn svm.svc gives very high as in 0.9. What is fundamentally different from sklearn model from this one? Thank you for the great tutorials

[@patloeber]

Iris dataset has 3 classes. My implementation only works for binary problems (2 classes), so the third class will always be classified incorrectly. Hence the low accuracy. I guess that the sklearn svm can cope with multiclass problems.

[@LIMLIMLIM111]

@@patloeber I tried doing it with sklearn breast cancer for that reason, the performance is about 0.5 accuracy when compared to 0.98 accuracy from sklearn model.

[@LIMLIMLIM111]

@@patloeber BTW I love your videos. One of the biggest challenges so far for me to really understand what's going on behind the scene. Your videos are so helpful in that regard. Can't thank you enough!

[@patloeber]

Thanks for the feedback! sklearn has a far better and more optimized solution. I tried to implement the easiest equations without any optimization, to understand the concepts. But the gradient descent can be stuck easily in a local minimum instead of the best global minimum. I recommend to apply a scaling to the data: from sklearn.preprocessing import StandardScaler...And also play around with the learnng rate and the lambda parameter to improve the accuracy.

[@LIMLIMLIM111]

@@patloeber Indeed, applying feature scaling gave more steady test accuracy report! and it has increased about +0.1.

[@batuhansozer8337]

great video. can you please share the notebook ?

[@patloeber]

Thanks! They are not yet published but I will consider this for the future

[@justAdancer]

Is it applicable for testing dataset with multiple class instead of two?

[@patloeber]

No this implementation is not suitable for multiclass problems. You can however try to use the "one-against-one" or "one-agains-rest" approach

[@fiqihfathorrachim2359]

if i input csv file using pandas, which parts need to be changed in the fit function? because when I run it always produces weight = [0 0]

[@patloeber]

Try feature scaling with sklearn.preprocessing.StandardScaler, and try to play around with the learning rate...

[@fiqihfathorrachim2359]

@@patloeber thank you, i will try it

[@mehdibouhamidi4675]

Try first to transform the data frame to an array using "df.values then you can use this program

[@alexanderperegrinaochoa7491]

hi, how can this algorithm be extend to a multi class problem?

[@patloeber]

Maybe this can help: nlp.stanford.edu/IR-book/html/htmledition/multiclass-svms-1.html

[@nadabu.274]

Great work. Thank you. I have a question regards equation f(x). Is f(x) = w*x+b or w*x-b. I checked many resources equation of a line is w*x + b

[@patloeber]

Both formulas are fine, if you optimize for w*x-b instead of w*x+b, then your final b just has the sign flipped. For example you solve with wx+b, and your b will be 5. and when you use wx-b, then b will be -5.

[@nadabu.274]

@@patloeber Thank you

[@10bhavankumarv47]

@@patloeber So using W.X + b = 0 also correct right?

[@babaabba9348]

Hello what if we are dealing with more than 2 classes? how shall I approach the problem

[@fearlessgoat2564]

then you have to perform ovo or ovr

[@patloeber]

this is done with a one-versus-all or one-versus-one technique. You can have a look here: nlp.stanford.edu/IR-book/html/htmledition/multiclass-svms-1.html

[@babaabba9348]

@@fearlessgoat2564 Thank you mate

[@babaabba9348]

@@patloeber Thank you for your response and thanks to your valuable videos

[@tigrayrimey6418]

Great work. Please keep it on. I really enjoyed it. Could you pls make a video for the Multiclass classification case from the scratch in python implementation? No video yet on this case I think. Or anyone can share it pls? Thank you so much, Sir!

[@patloeber]

Thanks! Will have a look at this

[@sotirigeorgiou3205]

Hey cool video, just to be slightly pedantic you're referring to the partial derivatives with respect to w_i so the d/dw needs to be different 'd' en.wikipedia.org/wiki/Partial_derivative

[@patloeber]

You are correct! sorry about this slight inaccuracy

[@sotirigeorgiou3205]

@@patloeber No problem. Fantastic set of tutorial you have. I am a practicing data scientist and working through your tutorial provides a great recap. Quick question, I'm looking to get into the deep learning space, out of the 3 popular frameworks (Tensorflow, Keras, Pytorch) which do you recommend to get started with?

[@patloeber]

All are great, but I prefer PyTorch. The syntax is a little bit more intuitive. You can find beginner PyTorch Tutorials on my channel as well :)

[@mariav1234]

Can you recommend a paper or book that teaches the math of SVM? That is, that follows the steps you describe in the video?

[@LIMLIMLIM111]

This is part 3 and has previous parts as well, but the main mathematics involved in this video is the Cost Function so I recommend following link: towardsdatascience.com/optimization-loss-function-under-the-hood-part-iii-5dff33fa015d

[@patloeber]

This one: towardsdatascience.com/support-vector-machine-introduction-to-machine-learning-algorithms-934a444fca47 Or if you want a book I can recommend Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow, or Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow

[@mariav1234]

@@patloeber Thank you for your replay.

[@jaivratsingh9966]

Thanks for the great work! Shouldn't the greadiet be \frac{\partial J_i}{\partial w_K} &= 2 \lambda w_k - y_i x_{ik} instead of \frac{\partial J_i}{\partial w_K} &= 2 \lambda w_k - y_i x_{i} (View in a latex viewer)

[@huichen7338]

No. As far as from what I saw, the math in the video (that part) is correct because it is differentiation with respect to weights, not x_i.

[@yimintong9963]

Hi, very nice video !! can you do K-means clustering from scratch ?

[@patloeber]

Thank you! Yes I will definitely do k-means in a few weeks!

[@yimintong9963]

@@patloeber thank you very much! can you do Artificial Neural Net also? Thanks, hope you get more subs, you explain clearest here and dont use library !

[@patloeber]

@@yimintong9963 Thanks! I will add it to my list!

[@SS-xt5ul]

vielen Dank :*

[@patloeber]

Kein Problem :)

[@balasuryam2758]

Awesome...

[@patloeber]

thanks!

[@moaz5160]

Can you do rbf kernel?

[@patloeber]

thanks for the suggestion. I will have a look into this...

[@nadabu.274]

Can you please recommend an article that helps to understand how to draw hyperplane line and support vector lines. why you choose this equation -w[0] * x - b + offset) / w[1]? thank you

[@Daniel_Zhu_a6f]

The data, visualised in this video is 2d, so the 'hyperplane' is, in fact, a line with equasion x0*w0 + x1*w1 -b =0. it can be rewritten like x1 = (-w0 * x + b) / w1, which is a preferred form for numeric calculations, and the one you see in demonstrated code (when offset is 0). For the propper visualization of multiple-dimension data: dot(x, w) - b =0 is an equasion for your target plane in n-dimensional space, n = len(w), x is an variable vector of length n, w is your learned weights. To visualize it smartly you want your hyperplane appear like a line, ie you need to project your data space in a 2d space (your drawing) that is perpendicular to the hyperplane. any vector x_fixed, that satisfies 1st equasion can be taken, to construct this 2d space. after that you will have 2 orthogonal vectors x_fixed and your weights w. From them you can construct a basis for the desired 2d space (just normalise them). Then you can calculate a projection of your data points to that space (their 2d coordinates will be a dot product of their n-dimensional vectors with a normalized x_fixed and w). The separation line will appear on your drawing as the axis corresponding to normalized w. To quickly find x_fixed, you can use Gauss method of linear equasion solving. I think, it sould be implemented in python somewhere. You will get a solution, that depends on one constant. by changing that constant you can turn the drawing plane around w axis (your separation line). Although, this rotation takes time to compute. To visualize an offset, you need to draw 2 lines, parallel to the w axes, with offset distance from it.

[@patloeber]

That's an accurate answer, thanks! Yes the important thing is the line equation in the so called normal form x0*w0 + x1*w1 = b (see sites.math.washington.edu/~king/coursedir/m445w04/notes/vector/equations.html). We want to solve this for x1 (probably the variable names are confusing here: x0 is our x axis, and x1, is our y axis). Then just take some points on the x axis, and use the equation to get the corresponding y points, and pass these points to matplotlib...

[@nadabu.274]

@@Daniel_Zhu_a6f Thank you very much for your detailed explanation.

[@nadabu.274]

@@patloeber Thank you.

[@rajeshmahor9488]

Getting SVM object has no attribute n_iters. Can help some one

[@patloeber]

Maybe you have a typo? you can compare the code with the one on GitHub

[@terminatorkhan3248]

Is this scratch? Then why you impot svm, dislike

[@balasuryam2758]

He imported the SVM class that he wrote..🤦‍♂️