Data Mining with Weka (2.6: Cross-validation results)

[@yogeshsabnis8507]

Ian, I like the idea about your classical tune and enjoy your videos as well. I have a question - I am building a 2-class model using Naive Bayes for a dataset of 1200 data points. I find that my ROC area with CV is marginally higher than non-CV. Is this even possible? Also, should sensitivity and specificity correlate with ROC area? Many thanks

[@swapnilgaikwad3773]

What are the different main parameters for classifiers like Random Forest and J48? How the result of cross-validation on it changes and why? Can you answer those questions.

[@arashhabibilashkari5609]

Really useful and nice. Thanks.

[@tamaravasylenko8218]

How to do Jackknife cross validation in Weka for 2-class model? Using, say, J48?

[@jeviounipers]

Nice tutorial. Thanks

[@StevenJonWest]

Hi Ian, I do not think the comparison of 10-fold cross validation to the 10% holdout method is very fair, as cross-validation will by definition assess the whole set of 10 segments 10 times (using each segment in turn as the test set) to get the accuracy measure, whereas with 10% holdout only one 10% test set is used once to get the accuracy. To get a fair comparison of holdout and cross validation, you would need to run the holdout 10 times for every one run of cross validation. This would show that cross-validation is probably marginally more accurate on average, as the variance in the holdout method (due to the RANDOM SAMPLING) is reduced by using a more systematic sampling method as in stratified cross validation. I think the point you're making is valid, but the way you're making it, for me, is not a fair comparison...

[@ClarinetAndCocktails]

Yes, I agree with you; you are exactly correct. And very well put - Thanks! I guess the point I was trying to make was to encourage people to use cross-validation rather than holdout in Weka.

[@Sina-Amn]

That's a great point! But, I think the focus of this video is more on the standard deviation rather than the mean.

[@wasd3108]

Who came here only for the music?