DeepSpeed: All the tricks to scale to gigantic models

References
github.com/microsoft/DeepSpeed
github.com/NVIDIA/Megatron-LM
github.com/cybertronai/gradie...
github.com/msr-fiddle/pipedream
Zero: arxiv.org/abs/1910.02054

[@mekaneeky]

Thanks Mark! Quite a thorough and useful explanation.

[@darrenbrien]

Thanks Mark great vid. Good update on SOTA in distributed training since horovod

[@randolphzeng6051]

Thanks for such an inspiring and insightful video. What a knowledge feast to enjoy !

[@sandraviknander7898]

If you just add a pair of aviator sunglasses then this is a Yannic Kilcher video. Instant 100k sub upgrade. Jokes aside, this was a great explanation of a great library!

[@user-vl2fp9ye5n]

Thanks mark!. You have been helping me understand concepts better.

[@saratbhargavachinni5544]

Great Video Mark! A few corrections, A100 is available in 40 GB and 80 GB variants.

[@limitlesslife7536]

amazing!

[@user-wp8yx]

Nice explanation, but how to do in ooba?

[@adriangabriel3219]

Hi Mark, great vid. Could you make a video on how to fine-tune large transformer models (e.g. T5 B-11) without running into CUDA errors?

[@marksaroufim]

Great suggestion! Yes I’ll do it

[@adriangabriel3219]

@@marksaroufim great! There is a lot information about fine-tuning T-5 base , but not about fine-tuning models above T-5 base

[@JordanArsenaultYT]

@@adriangabriel3219 Did you ever get t5-11b working?

[@Georgesbarsukov]

You're looking at RAM, not vRAM btw.

[@AndersOland]

A 2080ti with 30 gigs? 🤭 If only my 4090 had that much RAM 😅