Accelerate Transformer training with AWS Trainium

In this video, I show you how to accelerate Transformer training with AWS Trainium, a new custom chip designed by AWS.
First, I walk you through the setup of an Amazon EC2 trn1.32xlarge instance, equipped with 16 Trainium chips. Then, I run a natural language processing job where I adapt existing Transformer training code for Trainium, accelerating a BERT model to classify the Yelp review datatset. Finally, I run the job on 1, 8, and 32 Neuron cores.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos ⭐️⭐️⭐️
⭐️⭐️⭐️ Want to buy me a coffee? I can always use more :) www.buymeacoff... ⭐️⭐️⭐️
AWS Trainium: aws.amazon.com...
AWS Neuron SDK documentation: awsdocs-neuron...
AWS Neuron SDK samples: github.com/aws...
Hugging Face tutorial: huggingface.co...
Setup steps and code: gitlab.com/jul...
Interested in hardware acceleration? Check out my other videos :
Habana Gaudi: • Accelerate Transformer...
Graphcore: • Accelerate Transformer...
Trainium on SageMaker: • Accelerate Transformer...

[@junkmail75034]

Hey Julien thanks for this helpful video. In 16:23, you saved the checkpoint in pt. Per your example, this is how I load checkpoint back: model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=5) ckpt = torch.load("checkpoints/checkpoint.pt") model_weights = ckpt['state_dict'] model.load_state_dict(model_weights) model.to('xla') Hope this is right.