How to implement Logistic Regression from scratch with Python

In the third lesson of the Machine Learning from Scratch course, we will learn how to implement the Logistic Regression algorithm. It is quite similar to the Linear Regression implementation, just with an extra twist at the end.
You can find the code here: github.com/Ass...
Previous lesson: • How to implement Linea...
Next lesson: • How to implement Decis...
Welcome to the Machine Learning from Scratch course by AssemblyAI.
Thanks to libraries like Scikit-learn we can use most ML algorithms with a couple of lines of code. But knowing how these algorithms work inside is very important. Implementing them hands-on is a great way to achieve this.
And mostly, they are easier than you’d think to implement.
In this course, we will learn how to implement these 10 algorithms.
We will quickly go through how the algorithms work and then implement them in Python using the help of NumPy.
▬▬▬▬▬▬▬▬▬▬▬▬ CONNECT ▬▬▬▬▬▬▬▬▬▬▬▬
🖥️ Website: www.assemblyai...
🐦 Twitter: / assemblyai
🦾 Discord: / discord
▶️ Subscribe: www.youtube.co...
🔥 We're hiring! Check our open roles: www.assemblyai...
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
#MachineLearning #DeepLearning

[@josiahtettey6315]

Best concise video on logistic regression I have seen so far

[@AssemblyAI]

That's great to hear, thanks Josiah!

[@sreehari.s6515]

went through when I first started video editing, now it's taking a whole new switch and learning soft will only boost my courage for the

[@MOTIVAO]

This is amazing, thank you for this video.

[@OmarAmil]

need more algorithms , you are the best

[@AssemblyAI]

Thank you!

[@akhan344]

superb video! I am saying that because coding from scratch is important for me.

[@sarvariabhinav]

def sigmoid(x): x = np.clip(x, -500, 500) return (1/(1+np.exp(-x))) To avoid overflow runtime error as the return statement can reach large values

[@ricardoprietoalvarez1825]

Great video, but my only dubt comes when J'() is calculated as the derivate of MSE and not as the derivate of the Cross Entropy, which is the loss function that we are using

[@rbk5812]

Found the answer yet? Please let us know if you do!

[@GeorgeZoto]

That's what I noticed too.

[@zahraaskarzadeh5618]

Derivation of Cross Entropy looks like derivation of MSE but y^ is calculated differently.

[@upranayak]

log loss

[@HamidNourashraf]

We should start with Maximum Likelihood. Take the log. Then take derivative with respect to B (or W). Not sure how she took derivative. I guess she used MSE for classification problem instead of using Binary Cross Entropy. 🤔. Log L(B) = Sigma_{i=1}{N}((y_i*B*X_i - log(1 + exp(B*X_)).

[@ronakverma7070]

Great work from @AssemblyAI 👍✨thank you from India.

[@prodipsarker7884]

Studying CSE in GUB from Bangladesh , Love the way you teach the explanation & everything ; )

[@Shubham_IITR]

AWESOME EXPLANATION THANKS A LOT !!

[@AssemblyAI]

You're very welcome!

[@DanielRamBeats]

Thanks for sharing this, I am doing something similar in JavaScript. The part about calculating the gradients for backpropagation is very helpful!

[@rizzbod]

Wow , what a great video, very helpful

[@AssemblyAI]

Glad it was helpful!

[@lebesgue-integral]

Very good! Thanks for your videos!

[@salonigandhi4807]

how is the derivative of loss function w.r.t weights same for cross entropy loss and MSE loss ?

[@carloquinto9736]

Great work! Thank you :)

[@luis96xd]

Amazing video, I'm liking so much this free course, I'm learning a lot, thanks! 😁💯😊🤗

[@AssemblyAI]

Happy to hear that!

[@purplefan204]

Excellent video. Thanks.

[@CarlosRedman3]

Love it. Keep up the good work

[@jaredwilliam7306]

This was a great video, will there be one in the future that covers how to do this for multiple classes?

[@justcodeitbro1312]

wow you are an amazing teacher thanks alot god l love youtube !!!!!

[@dasjoyabrata1990]

love your code 👍

[@AssemblyAI]

Thank you!

[@OmarKhaled-dw7oi]

Awesome work, also, great English !

[@lamluuuc9384]

The same problem of missing the 2 multiplication in calculating dw, db in the .fit() method: dw = (1/n_samples) * np.dot(X.T, (y_pred-y)) * 2 db = (1/n_samples) * np.sum(y_pred-y) * 2 It does not affect too much, but we follow the slide to not be confusing

[@LouisDuran]

I noticed this as well. But adding in those *2 reduced the accuracy of my predictor from 92.98% to 88.59%

[@armaanzshaikh1958]

But in bias why we are not using mean coz the formula says summation then it has 1/N too ?

[@karimshow777]

We should maximize likelihood or minimize minus likelihood, I think the cost function is missing a minus , Am i right ?

7:51 Why you didn't multiplied by 2 the derivatives?

[@stanvanillo9831]

Technically you would have to but in the end it does not make a difference, you only effectively half your learning rate if you don't multiply by two.

[@armaanzshaikh1958]

Same question arises for me too why it wasnt multiplied by 2 and for bias it took sum instead of mean ?

[@MOTIVAO]

@@armaanzshaikh1958yeah she missed the 2s but it’s the sum not the mean.

[@xhtml-xe7zg]

wish one day i will be in this level of coding

[@1000marcelo1000]

Amazing video! Can you add the plot of it?

[@khaledsrrr]

Thanks 🙏 ❤

[@anatoliyzavdoveev4252]

Super 👏👏

[@WilliamDye-willdye]

I haven't used enough Python yet to accept the soul-crushing inevitability that there's going to be "self." everywhere. I guess you could call it "self." hatred. Maybe ligatures could come to the rescue, replacing every instance with a small symbol. While we're at it, put in ligatures for double underscores and "numpy." (or in the case of this video, "np."). Yes, it's an aesthetic rant that is ultimately not a big deal, but gradient descent is a beautifully simple concept. The presenter does a great job of matching that simplicity with clean, easy to follow code. Maybe it's not such a bad thing to be irritated at the parts of her code which are inelegant only because the language doesn't give her better options.

[@mohammadnaweedmohammadi5936]

I didn't know exactly why you imported Matplot Library

[@md.alamintalukder3261]

You are superb

[@subzero4579]

Amazing

[@noname-anonymous-v7c]

7:09 The gradient should not have the coefficient 2 . 7:43 linear_pred there should be a minus sign before np.dot.

[@noname-anonymous-v7c]

The above mentioned issue does not have much effect on the predicting result though.

[@LouisDuran]

I have seen other videos where people use a ReLU function instead of sigmoid. Would this logistic regression algorithm be an appropriate place to use ReLU instead of Sigmoid? If not, why not?

[@mertsukrupehlivan]

Logistic regression typically uses the sigmoid activation function because it provides a probability interpretation and is mathematically suited for binary classification. ReLU is more commonly used in deep neural networks with hidden layers, not in logistic regression.

[@MrBellrick]

with the partial derivatives, where did the multiple 2 go?

[@tiwaritejaswo]

Why doesn't this algorithm work on the Diabetes dataset? I'm getting an accuracy of 0

[@mayankkathane553]

why you have not used summation for dw for calculating error

[@emanuelebernacchi4352]

When you write the dw and db, shouldn't be (2 / n_samples)? There is the 2 in the derivative that you can take outside of the summation

[@hitstar_official]

That's what I am thinking from previous video

[@prigithjoseph7018]

I have a question, why can't you include accuracy function in class module?

[@sagarlokare5269]

What if you have categorical data? Like if different scale

[@ayhanardal]

how can find presentation file.

[@gajendrasinghdhaked]

make for multiclass also

[@Semih-nd3sq]

Where is the backward propagation? In conclusion logistic regression is also a neural network

[@sagarlokare5269]

Relevant feature selection not shown

[@am0x01]

Hello @AssemblyAI, where can I find the slides?

[@atenouhin]

But why are we using numpy here it's supposed to be from scratch

[@سعیداحمدی-ل1ث]

تو عالی هستی❤

[@abdulazizibrahim2032]

Please I want to build a personality assessment through cv analysis using this model, could you please help me?

[@md.alamintalukder3261]

Like it most

[@AssemblyAI]

Awesome!

[@moonlight-td8ed]

A M A Z I N G

[@nooneknows6274]

Does this use regularization?

[@ffXDevon33]

Is there a way to visualize this?

[@compilation_exe3821]

NICEEE ♥💙💙💚💚

[@nishchaybn1654]

Can we make this work for multiclass classification?

[@nehamarne356]

yes you can use logistic regression for problems like digit recognition as well

[@luisdiego4355]

Can it be used to predict house prices?

[@AssemblyAI]

Why not!

[@pythoncoding9227]

Hi, does an increase in sample size increase the prediction accuracy?

[@robzeng8691]

Look for statquest's Logistic regression playlist.

[@qammaruzzamman9621]

My accuracy came to be around 40, even after tweaking with n_iters, lr couple of times, is that okay ?

[@prateekcaire4193]

same

[@prateekcaire4193]

I think we made the same stupid mistake where we did not loop through for n_iters. for _ in range(self.n_iters):. silly me

[@dang2395]

What is this witchcraft? I thought ML was supposed to be too hard to know wth is going on!

[@mohammedamirjaved8418]

♥♥♥♥♥

[@waisyousofi9139]

-1/N

[@Lucianull31]

Amazing video but you're going over too fast

[@AssemblyAI]

Thanks for the feedback Lucian!

[@krrsh]

In predict(), is it X.T or just X for the linear_pred dot product?

[@hoseinhabibi2385]

i got 0.94 accuracy with below code: import numpy as np from sklearn.model_selection import train_test_split from sklearn import datasets import matplotlib.pyplot as plt from sklearn.linear_model import LogisticRegression bc = datasets.load_breast_cancer() X, y = bc.data, bc.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1234) clf = LogisticRegression(penalty='l2',max_iter=10000) clf.fit(X_train,y_train) y_pred = clf.predict(X_test) def accuracy(y_pred, y_test): return np.sum(y_pred==y_test)/len(y_test) acc = accuracy(y_pred, y_test) print(acc)