Bias Variance Tradeoff Explained!

[@Raj-gc2rc]

Do we have any mathematical proof for when we increase the model complexity the bias of the estimator decreses ? Like expected value of the estimator tends to the population parameter as complexity increases.. Can you explain this in the context of deep learning?

[@CodeEmporium]

Lemme preface this by saying I am not a math expert and that this is a very good question that has gotten me thinking. To me, Statements like “increased model complexity decreases bias” stem more from intuition (strengthend by say, a simulation) than mathematical rigor. I hesitate to say “prove” since I am not sure if we can say model complexity _always_ will decrease the bias of an estimator. This statement is dependent on data after all. I would instead say “we can build intuition to support this statement for the most part”. “Complexity” can mean many things. But let’s assume it is “increasing the number of parameters in a model”. The way I would think of proving your statement (or rather building intuition) is to create some known distribution; sample data from it; train a base model like y = constant + epsilon; calculate bias of the output y by using many rounds of training and eventually testing each model on the same input X. Then do the same for a slightly more complex model like y = theta1 X + theta0 + epsilon. Keep adding parameters to add complexity. You can then just build intuition to see how the bias (and variance ) change with this complexity.

[@CodeEmporium]

To your last question, I wouldn’t think about bias and variance much more different for a linear regression than I do for a neural network. Internally, yes they work differently. Neural networks can have many tunable parameters (as can other regression models). But on the surface, a linear regressor and a neural network take the same inputs, try to learn some parameters to predict some output. They do so sometimes with even the same loss functions like mean squared error which decompose into a bias and variance term in similar ways. I could be wrong. But this is my intuition. Happy to hear disputes, other perspectives too. Great question!

[@Raj-gc2rc]

@@CodeEmporium tysm for answering ... What's the best approach to learn machine learning for jobs vs for reaserch? ... I have habit of getting into mathematical formalism for everything ...I don't know if it is the right way of learning... Can I mail you for querues regarding data science ?

[@CodeEmporium]

@@Raj-gc2rc Definitely no silver bullet. I think just spending time learning about the concepts, then delving into some application code, then the mathematics helps for a start. With this, you can build general intuition. I find myself always revising concepts but greater detail every time. Every time I touch a concept, I learn some things I didn’t know before For jobs, I would try working on some machine learning projects. With these projects, you can test your intuition. Doing so you might realize that some concepts your learned as “hard truths” actually are not (as you and I alluded to). In interviews, it’s common to talk about case studies. So not only would you need to know core concepts, you need to relate them to a real world problem. This is a nice next step

[@kinanradaideh5479]

The best video on this topic, and ive been looking all day for vids on this topic! Never stop please, your videos are excellent!

[@vitoriacoghi]

Great explanation. Thank you ☺

[@sneha_more]

Thank you for the detailed explanation.

[@CodeEmporium]

You are very welcome!!

[@mdsharaf95]

That was awesome. Please keep posting.

[@CodeEmporium]

That’s the plan! Thank you :)

[@ahmedgabr8009]

Great Video

[@CodeEmporium]

Thank you!

[@seekerpro]

Amazing work!

[@CodeEmporium]

Thank you so much!

[@bdennyw1]

I usually see Bias Variance tradeoff as the idea that at some point in training that you can't make one better with out making the other worse.

[@ajaytaneja111]

Hi Ajay, excellent video. Can you advise of the reference that explains about the MSE as you did? Thanks again

[@CodeEmporium]

Ajay! Thank you! In the description, there are 5 resources that I read to surmise my thoughts. Some of them are wiki articles that I thought were well written. About MSE specifically, I do think the Introduction to Statistical Learning text book has great insight. Let me know if you needed more specifics, but I hope this is a good start for you!

[@jameszhang9628]

Amazing!

[@CodeEmporium]

Thank you:) I’ll

[@ouroboros7388]

Thanos snap hehe