Richard Baraniuk - A Spline Tour of Deep Learning: The Scattering Way

Over the past decade, deep (neural) networks trained using massive data sets have enabled remarkable progress on a wide range of challenging computational problems, from pattern recognition and image synthesis to language translation and protein folding. Nevertheless, this progress has been alchemistic and driven largely by empirical observations, hacks, and tricks. Fundamental questions remain, such as: Why do deep learning methods work? When do they work? And how can they be fixed when they don’t work? Intuitions abound, but a coherent framework for understanding, analyzing, and synthesizing deep learning architectures remains elusive. This talk will discuss the implications of this lack of understanding for consumers, practitioners, and researchers of machine learning. We will also briefly overview recent progress towards a theory of deep learning based on rigorous mathematical principles. Of the several promising avenues of research, we will focus on the connection between deep networks and spline approximation that provides a geometric interpretation for how deep networks organize and process data. A particularly interesting special case is provided by the Scattering Network.
Richard Baraniuk (Rice University)