The Wrong Batch Size Will Ruin Your Model

How do different batch sizes influence the training process of neural networks using gradient descent?
Colab notebook: colab.research...
🔔 Subscribe for more stories: www.youtube.co...
📚 My 3 favorite Machine Learning books:
• Deep Learning With Python, Second Edition - amzn.to/3xA3bVI
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow - amzn.to/3BOX3LP
• Machine Learning with PyTorch and Scikit-Learn - amzn.to/3f7dAC8
Twitter: / svpino
Disclaimer: Some of the links included in this description are affiliate links where I'll earn a small commission if you purchase something. There's no cost to you.

[@ErlendDavidson]

If you scale the batch size by the learning rate (i.e. lr=(batch_size/32.)*0.01) then the stochastic gradient descent looks sort of okay here.

[@underfitted]

Interesting :)

[@jasdeepsinghgrover2470]

I completely agree ... Because the number of updates happening depend on batch size and even the size of the update. So if the learning rate is scaled according to batch size linearly the model can perform very well even with much smaller batches.

[@Metryk]

Hi! Maybe you can help me with this one: if I want to test an already pre-trained image classifier, how do I proceed regarding the amount of images used? The set containing test images has 100k images, I guess it wouldn't make any sense to load them all at once, so how do I proceed? Thanks!

[@ErlendDavidson]

What do you think of (artificially) adding noise to the learning rate. I feel like it used to be more popular to do that, but almost never see it these days.

[@underfitted]

Yeah… never seen that honestly. I’ve used schedules to decrease the learning rate over time, but never read about adding noise to it.

[@johnmoustakas8897]

Good work, hope your channel gets more attention

[@underfitted]

Thanks, John! It takes time and work but I’ll make it happen.

[@Agrover112]

Hey love this video! Was losing touch of the basics !

[@underfitted]

Glad it was helpful!

[@OliverHennhoefer]

Really like the videos. However, I want to warn against the general statement that a batch size of one is not recommended. It really depends on the problem/data. So don't simply dismiss stochastic gradient descent, try it!

[@underfitted]

I think that’s fair. I’ve never used it in any of the problems I’ve worked on, but you are right.

[@user-zr5mg2wc2i]

i didnt see a helpful video like this one in the entire internet, thank you ♥

[@underfitted]

Glad it was helpful!

[@muhammadtalmeez3276]

Your videos are amazing. Thank you so much for this great knowledge and beautiful videos.

[@underfitted]

Glad you like them!

[@ziquaftynny9285]

I love your presentation style! Very energetic :)

[@underfitted]

Thanks

[@lakeguy65616]

so, what is the optimal batch size?

[@underfitted]

It depends. Start with 32 and experiment from there.

[@lakeguy65616]

@@underfitted Does the amount of main memory Ram or GPU ram make a difference? (great videos!)

[@underfitted]

It does! Your batch has to fit in memory, or it won't work. When you are working with images, for example, you'll quickly find that your batch size can't be too large if you want to fit it in the GPU's memory.

[@edmundfreeman7203]

This is the kind of thing that I hate about deep learning. A single parameter in the optimization method can completely change the results. Batches should be small but not too small. How small? That's for heuristics but will change on different data sets.

[@Levy957]

Amazing!! Did u know why the batch size os always 32, 64, 128?

[@underfitted]

I read somewhere about the ability to fit batches in a GPU... can't remember where exactly. That being said, I've seen experiments that show that it really doesn't matter much (if at all.)

[@MrAleksander59]

It's better for memory usage. GPU, CPU, hard drives, SSD and other in the current 2-bit logic uses memory blocks with sizes of power 2. 2^5 = 32, 2^6=64, 2^7=128 etc. You always want maximum usage of memory. For example you have array with floats, each float will take 32 bits. So, at least it divisible by 32.

[@axelanderson2030]

If you generate a dummy dataset and set a static learning rate, then smaller batch sizes work better? wtf?

[@akshay0072]

Good content. Try improving ur way of teaching. Learning should in relaxed tone

[@underfitted]

Thanks! This was an old video. I’ve tried to improve in the latest few.

[@michaelsprinzl9045]

A new cat video. Cute.

[@sarahpeterson2702]

the question is whether if you use a batch and reach the global minimum is your model functionally equivalent to one that didn't batch? Are the weights identical... no they aren't . if your model is generative you don't have equivalence with batch/non batch.