Tips and tricks for distributed large model training

Discover several different distribution strategies and related concepts for data and model parallel training. Walk through an example of training a 39 billion parameter language model on TPUs, and conclude with the challenges and best practices of orchestrating large scale language model training.
Resource:
TensorFlow website → goo.gle/3KejoUZ
Speakers: Nikita Namjoshi, Vaibhav Singh
Watch more:
All Google I/O 2022 Sessions → goo.gle/IO22_AllSessions
ML/AI at I/O 2022 playlist → goo.gle/IO22_ML-AI
All Google I/O 2022 technical sessions → goo.gle/IO22_Sessions
Subscribe to TensorFlow → goo.gle/TensorFlow
#GoogleIO

[@XxXx-sc3xu]

Great presentation! Thank you.

[@roy11883]

Thanks, could you also share any link of data parallelism implementation as well?

[@svkchaitanya]

What a great tut, thanks dude ....

[@fahemhamou6170]

تحياتي

[@user-wr4yl7tx3w]

This video especially the second half was a bit difficult to follow given too much assumed knowledge.