Lecture 13 - Validation

Validation - Taking a peek out of sample. Model selection and data contamination. Cross validation. Lecture 13 of 18 of Caltech's Machine Learning Course - CS 156 by Professor Yaser Abu-Mostafa. View course materials in iTunes U Course App - itunes.apple.c... and on the course website - work.caltech.ed...
Produced in association with Caltech Academic Media Technologies under the Attribution-NonCommercial-NoDerivs Creative Commons License (CC BY-NC-ND). To learn more about this license, creativecommons...
This lecture was recorded on May 15, 2012, in Hameetman Auditorium at Caltech, Pasadena, CA, USA.

[@saraghavendra]

Great teaching style. I liked the part where professor paused and asked 2 questions about error curves. Just giving those few seconds to think helped me in absorbing the concept.

[@AlexanderPolomodov]

Amazing lecture. I like most of all its part about cross validation.

[@sepidet6970]

"Customer, this is the system and I am very sure this is terrible " this part was hilarious :)))))))))))))

[@theofilospapadopoulos2476]

Wonderful lecture. Gives insight into validation and cross validation. Two strong points: the optimistic bias of the validation error and the tracking of the out-of-sample error nevertheless

[@carlosmspk]

I feel bad for the fact that no one seems to laugh at his jokes in the lecture :(

[@yastradamus]

Punch line: you can fool yourself into any pattern you want! :D ... I love how explains this!

[@markh1462]

Yeah, that's my favorite part of his lectures too. This is a very subtle concept and nailed it.

[@lamnguyenthanh7242]

Love his lectures.

[@ishanprasad910]

On slide 21, we discuss results based on a 20 parameter variable vs. 6 or 7 parameter variable, and select a model based on validation. Did we use the regularization method in the 20 parameter variable?

[@movax20h]

So, basically validation and cross validation is a bootstraping and jack-knife method known from statistics. Similarly, model selection is meta training with meta parameters. And again, the E_val, is essentially a bootstraped estimate. What is remarkable, is that even if meta parameters give you infinite familiy of models, each only increases your "d_vc" by 1 essentially. Wouldn't something like K = \sqrt(N), or K=log(N)/N be little more accurate for cross validation?

[@gcgrabodan]

+movax20h Im not sure that you are right. Boostrap means drawing new samples from one orignial. Validation means putting some part aside and then use it as a "test". But there is no resampling, which for me is the essence of bootstrapping... Edit: I hadnt watched the entire lecture yet, seems like you are right.

[@Bing.W]

There is no "better" size of K for N/K-fold cross-validation.

[@Bing.W]

At time 43:15, professor Yaser states the Eval as "out-of-sample error", which should be "in-sample error".

[@kmshihabuddin]

it's neither. But it's an estimate of out of sample error.

[@nosh3019]

Great teacher!

[@AndyLee-xq8wq]

[@muhammadanwarhussain4194]

In the cross validation case with leave one out, the out of sample error will be reported by averaging out. But which model will be chosen? The one with minimum error. Finding out of sample error with only 1 sample and select the model which gives minimum of that error seems not optimal to me. I think I am missing something. I will really appreciate an answer.

[@insomniacnomis]

You wuold take the model with better performance and then train it with the whole data set

[@ajayram198]

Could someone please explain the optimistic bias that the professor discusses about? Another Qn - at 43:38 the professor mentions choosing between infinite hypothesis. How is an infinite hypothesis set even possible? Not understanding that.

[@exmachina767]

For example, the set could contain all possible realizations of a neural network with regularization (infinite number of lambdas). Right after 43:38, he explains this is still not an issue and invokes an argument made in earlier lectures regarding the VC dimension.

[@fahimhossain165]

Anyone found the voice of the TA/RA annoying?

[@taroice8442]

very nice! thank you!

[@Nestorghh]

Muy bueno!!!

[@ehsansslman9436]

the expected value of one point is the value of that point, so why we use it?

[@MrSnowmobilefreak]

in case you were still wondering: it has to do with the balance between small and large K (good Eval approx vs good g- approx). The key thing to remember is it isn't just one point. Its one validation point over and over. So by using one point, you can be sure you have a good g- that is close to g, and by doing it over and over, you get a picture for how Eval approximates Eout on average, aka expected. Hope this helps

[@ledinhngoc1102]

@@MrSnowmobilefreak Hello, can you explain me the variance formula? idk why we divide to K square, and don't get what he means when he mentions co-variance?

[@brainstormingsharing1309]

👍👍👍👍👍

[@tempvariable]

thank you :)

[@tempvariable]

Ehsan Sslman You asked your question as a reply to me. If you ask it as a comment instead overall more people can see