EfficientML.ai Lecture 21 - On-device Training (MIT 6.5940, Fall 2024)

EfficientML.ai Lecture 21 - On-device Training and Transfer Learning (MIT 6.5940, Fall 2024)
Lecture 21: On-device Training and Transfer Learning
Instructor: Prof. Song Han
Slides: efficientml.ai
Outline:
Deep leakage from gradients, gradient is not safe to share
Memory bottleneck of on-device training
Tiny transfer learning (TinyTL)
Sparse back-propagation (SparseBP)
Quantized training with quantization aware scaling (QAS)
PockEngine: system support for sparse back-propagation