Reconciling modern machine learning and the bias-variance trade-off

[@PeterJMPuyneers]

I struggled with understanding this paper due to lack of knowledge (conceptually spoken), but after seeing your explanation, everything is clear. thank you very much

[@AntonPanchishin]

Mind blown. Super cool! I have so many tests to rerun with higher parameter count now

[@danielbigham]

Fantastic video -- thank you! Fascinating...

[@MLDawn]

you did a great job. This just left me speechless!!!

[@kristoferkrus]

Mind blown. Very interesting paper! Does this mean that if you are in the regime where the test loss has started to decrease (as a function of parameters) again and you add more training examples, your test accuracy will get worse because it makes it harder for the optimizer to find a simple function that perfectly mahces the training data? In theory, this could make it beneficial to reduce the number of training examples, but intuitively, that feels wrong.

[@YannicKilcher]

That's a very interesting point. Technically yes, but I agree it seems strange.

[@YannicKilcher]

I think it all comes down to the inductive bias given implicitly by the network architecture and the optimizer. In this framework, adding training data will take capacity away from the inductive bias and potentially worsen your result.

[@andreg5206]

I know this is 10 months old, but at the end of 2019 OpenAI published a paper that suggests exactly what you imply here: openai.com/blog/deep-double-descent/

[@kristoferkrus]

@@andreg5206 Yes, I saw that; that's so bizarre! Thanks for reminding me about it :)

[@gyeonghokim]

Thanks a lot!

[@995Fede]

I started to read this paper during the last days and I confirm that it is really interesting! However, I have some doubts on the way they evaluate the MSE (how do they deal with the fact the function h(x) is complex?) and the zero-one loss/norm of coefficients (since it is a multi-class classification problem, they probably use one-hot encoding, but again how do they deal with the complex h(x)? Moreover, if they use one-hot encoding, the regressor is a 2D matrix, thus what norm are they plotting? L2 norm for matrices?). Did you try to reproduce their plots with the MNIST database? Are these technical passages clear to you? Thank you again for the video!

[@DasGrosseFressen]

A high-complexity solution be like "Braaaah! Brrraah!" 😂👍

[@DrAhdol]

This is an interesting paper; I wonder if this applies to boosting/bagging with models that don't have many parameter options like multinomial naive bayes. Would parameter optimization on ensemble models have the same effect when the baseline model within are linear? Interesting option for some testing here.

[@YannicKilcher]

Seems worth a try :) don't even know if boosting models can overfit in the classic sense...

[@herp_derpingson]

Can you elaborate on the Hilbert space thing? What does Hilbert space to do with neural networks?

[@YannicKilcher]

That's a bit too much for a YT comment, but the concept is usually well explained in introductory ML classes in the advanced section of kernelized SVMs.

[@singhay_mle]

Lookup 3BlueBrown's video on it

[@herp_derpingson]

@@singhay_mle That does not explain what that has to do with neural networks.

[@singhay_mle]

@@herp_derpingson Sure, try this users.umiacs.umd.edu/~hal/docs/daume04rkhs.pdf , also it have more to do with kernel used by SVM/SVC than NN

[@agusavior_channel]

Very clear

[@sayakpaul3152]

This is such an amazing study. So many synergies with the Deep Double Descent paper.

[@ujjwalkar1886]

Is complexity of H means no of features here ?