Sergey Feldman: You Should Probably Be Doing Nested Cross-Validation | PyData Miami 2019

[@iancherabier5920]

Thanks a lot, an extremely clear explanation of nested CV!

[@BeckCaesar-r8l]

Davis Timothy Anderson Michael White Donna

[@QIQIWU-fd1xz]

This is really helpful! Thanks for sharing. One question, at 10:55, when running the 5-fold CV, shouldn't we use X_train_val instead of X_train? Because the splitting is done by sklearn, thus we don't need to hold out a validation set.

[@sergey_of_fields]

Yes! Sorry that was a bug in the code.

[@bryanparis7779]

THANK YOU so helpful! so interesting so so so :)

[@BulkySplash169]

Nice, thx!

[@nespereira]

Very useful! One question: in many medical datasets, especially in single-group research settings, the sample sizes are more around 100 or less (being in the thousands is rare). With this number of subjects, one worry is that putting away subjects for testing removes samples in a context where there really is not much data to begin with. Then you need to think about how many features you can afford etc... Don't get me wrong, I'm all in for nested cross-validation, but I am curious to hear your thoughts on this type of scenario, where getting data is really expensive.