Gianluca Ceruti (EPFL-CIS): “Low-rank dynamical training of feed-forward neural networks”

Abstract: Neural networks have recently found tremendous interest in a large variety of applications. However, their memory and computational footprint can make them impractical in settings with limited computational resources. In the present contribution, a brief recapitulation on recent developments for dynamical low-rank approximation is presented. Then, based on the novel rank-adaptive unconventional robust numerical integrator for dynamical low-rank approximation, a novel algorithm(DLRT) for finding and efficiently training feed-forward neural networks having low-rank weight matrices is introduced. It is illustrated that up to a prescribed tolerance parameter, the proposed algorithm dynamically adapts during the training phase the ranks of the weight matrices of the neural network, reducing the overall time and memory resources required by both the training and the evaluating process. Furthermore, up to numerical errors, the DLRT algorithm is shown to preserve the monotonic decrease of the loss-function along the low-rank approximations. The efficiency and accuracy of the proposed method is illustrated through a variety of numerical experiments on fully-connected and convolutional networks.The present contribution is based on a joint work with Steffen Schotthöfer(KIT), Emanuele Zangrando(GSSI), Jonas Kusch(University of Innsbruck), and Francesco Tudisco(GSSI).