Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code

A complete tutorial on how to train a model on multiple GPUs or multiple servers.
I first describe the difference between Data Parallelism and Model Parallelism. Later, I explain the concept of gradient accumulation (including all the maths behind it). Then, we get to the practical tutorial: first we create a cluster on Paperspace with two servers (each having two GPUs) and then training a model in a distributed manner on the cluster.
We will explore collective communication primitives: Broadcast, Reduce and All-Reduce and the algorithm behind them.
I also provide a template on how to integrate DistributedDataParallel in your existing training loop.
In the last part of the video we review advanced topics, like bucketing and computation-communication overlap during backpropagation.
Code: github.com/hkproj/pytorch-tra...
PDF slides: github.com/hkproj/pytorch-tra...
Chapters
00:00:00 - Introduction
00:02:43 - What is distributed training?
00:04:44 - Data Parallelism vs Model Parallelism
00:06:25 - Gradient accumulation
00:19:38 - Distributed Data Parallel
00:26:24 - Collective Communication Primitives
00:28:39 - Broadcast operator
00:30:28 - Reduce operator
00:32:39 - All-Reduce
00:33:20 - Failover
00:36:14 - Creating the cluster (Paperspace)
00:49:00 - Distributed Training with TorchRun
00:54:57 - LOCAL RANK vs GLOBAL RANK
00:56:05 - Code walkthrough
01:06:47 - No_Sync context
01:08:48 - Computation-Communication overlap
01:10:50 - Bucketing
01:12:11 - Conclusion

[@tharunbhaskar6795]

Dang. Never thought learning DDP would be this easy. Another great content from Umar. Looking forward for FSDP

[@chiragjn101]

Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model. Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP

[@amishasomaiya9891]

Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.

[@karanacharya18]

Super high quality lecture. You have a gift of teaching, man. Thank you!

[@oliverhitchcock8436]

Another great video, Umar. Nice work

[@abdallahbashir8738]

I really love your vidoes. you have a natural talent on simplifying logic and code. in same capacity as Andrej

[@user-td8vz8cn1h]

This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.

[@vasoyarutvik2897]

this channel is hidden gem

[@631kw]

Amazing content! Thanks for your sharing

[@prajolshrestha9686]

Thankyou so much for this amazing video. It is really informative.

[@user-wm5xv5ei8o]

very nice and informative video. Thanks

[@810602jay]

Amazing learning stuff ! Very Thanks !~ 🥰🥰🥰

[@user-jf6li8mn3l]

The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.

[@nova2577]

You deserve many more likes and subscribers!

[@riyajatar6859]

In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s. Could anyone explain? I am bit confused. Also what happens if I am having odd numbers of nodes

[@svkchaitanya]

You rock always 😂

[@user-od3ig9qt6h]

Thank you very much for your wonderful video. Can you teach a video on how to use the accelerate library with dpp?

[@Engrbilal143]

Awesome video. Please make tutorial on FSDP as well

[@felipemello1151]

I wish i could like it twice

[@umarjamilai]

You can share it on social media. That's the best way to thank me 😇

[@felipemello1151]

@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!

[@d.s.7857]

Thank you so much for this

[@loong6127]

Great video

[@manishsharma2211]

you teach soooooooo good

[@rohollahhosseyni8564]

great video

[@madhusudhanreddy9157]

If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno . please create a playlist for pytorch for beginners and intermediates. Thanks for reading.

[@mandarinboy]

Great intro video. Do you have any plans to also cover other parallelism: Model, Pipeline, Tensor, etc.

[@user-el4uh3uk2k]

fantastic

[@mdbayazid6837]

Federated learning basics please.❤

[@ramprasath6424]

please do some thing related to audio large models like conformers,quartznet ,etc

[@Yo-rw7mq]

Great!

[@hellochli]

Thanks!

[@umarjamilai]

谢谢你！我们在领英connect吧

[@waynelau3256]

Working with fsdp and megatron now and I really want to figure this out from scratch haha, it sounds fun but a big headache

[@user-fw5sg5mx4m]

could provide another videos with respect to model parallel and pipeline parallel ? thanks..

[@sounishnath513]

SUUUPERRRR

[@Allen-TAN]

Always great to watch your video, excellent work

[@tryit-wv8ui]

another banger

[@ai__76]

How to do in Kubernetes? Please explain it.

[@Erosis]

Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?

[@umarjamilai]

Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum". One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size. But remember that mathematically you don't have to

[@khoapham7303]

I'm always confused with DP and DDP. Can you please tell me the difference between them? While both of them belong to data parallelism method.

[@umarjamilai]

DP only works on a single machine, while DDP can work on multiple machines. However, PyTorch now recommends using DDP also for single-machine setup.

[@khoapham7303]

@@umarjamilai thank you for your reply

[@madhusudhanreddy9157]

Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.

[@umarjamilai]

Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.

[@madhusudhanreddy9157]

@@umarjamilaithanks for the info.

[@user-ze3ok8hh6c]

do you have a discord channel?

[@milonbhattacharya4097]

shouldnt loss be accumulated ? loss += (y_pred - y_actual)^0.5

[@user-pt7gs2ei1r]

In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.