How Fully Sharded Data Parallel (FSDP) works?

[@tinajia2958]

This is the best video I’ve watched on distributed training

[@abhirajkanse6418]

That makes things very clear! Thanks a lot!!

[@yixiaoli6786]

The best video of FSDP. Very clear and helpful!

[@chenqian3404]

To me this is by far the best video explaining how FSDP works, thanks a lot!

[@dhanvinmehta3294]

Thank you very much for making such a knowledge-dense, yet self-contained video!

[@mahmoudelhage6996]

As a Machine learning Research Engineer working on fine-tuning LLMs, I normally use DDP, or Deepspeed, and wanted to understand more about how FSDP works. This video is well structured and provides a detailed explanation about FSDP, I totally recommend it. Thanks Ahmed for your effort :)

[@phrasedparasail9685]

You and I both friend

[@MrLalafamily]

Thank you so much for investing your time in creating this tutorial. I am not an ML engineer, but I wanted to build intuition around parallelizing computation across GPUs and your video was very helpful. I especially liked that you provided multiple examples for parts that were a bit more nuanced. I paused the video many times to think things over. Again, gratitude as a learner

[@yuxulin1322]

Thank you so much for such detailed explanations.

[@xxxiu13]

A great explanation of FSDP indeed. Thanks for the video!

[@AntiochSanders]

Wow this is super good explanation, cleared up a lot of misconceptions I had about fsdp.

[@lazycomedy9358]

This is really clear and help me understand a lot of details in FSDP!! Thanks

[@pankajvermacr7]

thanks for this, im having trouble undertanding FSDP, even i read a research paper but hard to understand, i really appreciate your effort, please make more such videos.

[@bharadwajchivukula2945]

crisp and amazing explanation so far

[@saurabhpawar2682]

Excellent explanation. Thank you so much for putting this out!

[@amansinghal5908]

great video - one recommendation. make 3 videos, one like this, one that goes deeper into the implementation e.g. FSDP code and finally how to use it e.g. case studies

[@mandeepthebest]

amazing video! very well articulated.

[@tharunbhaskar6795]

The best explanation so far

[@NachodeGregorio]

Amazing explanation, well done.

[@yuvalkirstain7190]

Fantastic presentation, thank you!

[@ElijahTang-t1y]

well explained, great job!

[@dan1ar]

Great video!

[@RaviTeja-zk4lb]

I was struggling to understand how FSDP works and your video helped me a lot in understanding. Thank you. After understanding what are this backends. I see that FSDP definetly requires 'GPU'. For 'CPU" we use 'gloo' as backend and it doesn't support reduce-scatter. It would be great if you also cover Paramater serving training using RPC framework.

[@gostaforsum6141]

Great explanation!

[@amirakhlaghi8143]

Excellent presentation

[@coolguy69235]

very good video ! seriously keep up the good work !

[@phrasedparasail9685]

This is amazing

[@Veekshan95]

Amazing video with great visual aids and even better explanation. I just had one question - At 24:45 you mentioned that FSDP layer 0 is never freed until the end. So does this mean, the GPUs will have layer0 all the time and in addition to that they will consider other layers as needed?

[@ahmedtaha8848]

Yes, Unit 0 (layer 0 + layer 3) -- which is the outermost FSDP unit -- will be available across all nodes (GPUs) during an entire training iteration (forward + backward). Quoting from arxiv.org/pdf/2304.11277 (Page #6), "Note that the backward pass excludes the AG0 All-Gather because FSDP intentionally keeps the outermost FSDP unit’s parameters in memory to avoid redundantly freeing at the end of forward and then re-All-Gathering to begin backward."

[@yatin-arora]

well explained 👏

[@p0w3rFloW]

Awesome video! Thanks for sharing

[@TrelisResearch]

Great video, congrats

[@hannibal0466]

Awesome Bro! One short question: in the example shown (24:06), why there are two consecutive AG2 stages?

[@ahmedtaha8848]

Thanks! One for the forward pass and another for the backward pass. I suppose you can write a special handler for the last FSDP unit to avoid freeing the parameters then re-gathering them. Yet, imagine if FSDP unit#0 have another layer (layer#6) after FSDP unit#2, i.e., so total (layer#0, layer#3, layer#6). The aforementioned special handler won't look wise then.

[@clarechen1590]

great video!

[@ManishPrajapati-o4x]

TY!

[@louiswang538]

how is FSDP different from gradient accumulation? seems both have a mini-batch to get 'local gradients' and sum up to get global gradient for model update.

[@adamlin120]

Amazing explanation 🎉🎉🎉

[@ahmedtaha8848]

Thanks!

[@AIwithAniket]

it helped a lot. thank you so much

[@ahmedtaha8848]

Thanks!

[@bennykoren212]

Excellent !

[@mohammadsalah2307]

Thanks for sharing! 19:19 The first FSDP unit to compute forward process is FWD0; however this FSDP unit contains layer 0 and layer 3; How could we compute the result of layer3 without compute the result of layer 1 and 2 first?

[@ahmedtaha8848]

Layer3 is computed only after computing layer 1 and layer 2. Please note that there are two 'FWD0': the first one computes layer 0; the second one computes layer 3 after FWD1 (layer 1 and 2) finishes.

[@aflah7572]

Thank You!

[@santiagoruaperez7394]

Hi. I want to ask you something. In the 3:01 you also include de optimizer state in the multiplication for each parameter. I want to ask if the optimizer state is not just one for the whole model? What I mean is: if I have a 13B model in comparisson with a 7B model, the gradients are going to be more. But in the case of the optimizer state is going to depend also from the number of parameters?

[@ahmedtaha8848]

The optimizer state is not just one for the whole model. A 13B model has both more gradients and more optimizer state compared to 7B model. Yes, optimizer state depends on the number of parameters. For Adam optimizer, the optimizer state (Slide 4 & 5) includes both momentum (first moment) and variance (second moment) for each gradient, i.e., for each parameter.

[@santiagoruaperez7394]

Amazing video, super clear@@ahmedtaha8848

[@maxxu8818]

Hello Ahmed, If it's a 4 way FSDP in a node, does it mean there is only 4 GPU used for that node? Usually there are 8 GPUs in a node? how the other 4 GPUs are used? Thanks!

[@dhineshkumarr3182]

Thanks man!

[@DarnellFie]

Thanks for the interesting content! 😍 I’ve got a question: 🤨 I found these words 😅. (behave today finger ski upon boy assault summer exhaust beauty stereo over). Can someone explain what this is? 😅

[@richeshc]

Namaste, a doubt. For Pipeline parallelism (mins 10 to 12) you mentioned while we send weights from 1st gpu of minibatch 1 training to gpu 2, we start training gpu 1 feedforward network on minibatch 2. Doubt is isnt it suppose to be feedforward followed by backward propogation and updation of weights, then training on batch 2? Words are giving an interpretation that we are straight away starting feedforward training on gpu1 with minimatch 2 soon we transfer weights from minibatch 1 training of gpu 1 to gpu 2.

[@ahmedtaha8848]

For minibatch 1, we can't do back-propogation till we compute the loss, i.e, till mini-batch 1 passes through all layers/blocks. Same for mini-batch 2. After computing the loss for mini-batch 1, we can back-propogate one layer/block at a time on different GPUs -- and of course update the gradient. Yet, again other GPUs will remain idle if we are processing (forward/backward) a single mini-batch. Thus, it is better to work with multiple mini-batches, each with a different loss value. These mini-batches will be forwarding/backwarding on different GPUs in parallel.

[@amaleki]

there is a bit of discrepancy in what is defined as model parallelism with Nvidia literature. Name, in nvidia literature, Model parallelism is overarching idea of sharing the model and it can take two forms: i) Tensor MP (splitting layers between GPUs, each GPU get a portion of each layer) ii) Pipeline Parallelism (each GPU responsible for computing some layers entirely).

[@RedOne-t6w]

Awesome

[@DevelopersHutt]

TY

[@piotr780]

how whole gradient is calculated if weights does not fit single GPU memory ???????

[@parasetamol6261]

That Great vedio.

[@ahmedtaha8848]

Thanks!

[@hyunhoyeo4287]

Great explanation!

[@adityashah3751]

Great video!