The space of functions computed by deep-learning networks, David Saad | LMS/IMA Joint Meeting 2023

Recent engineering achievements of deep-learning machines have both impressed and intrigued the scientific community due to our limited theoretical understanding of the underlying reasons for their success. I will briefly review some of the challenges to be addressed and then focus on properties of the function space of different types of deep-learning machines, based on the generating functional analysis. This approach facilitates studying the number of solution networks of a given error around a reference multi-layer network. Exploring the function landscape of densely-connected networks, we uncover a general layer-by-layer learning behaviour, while the study of sparsely-connected networks indicates the advantage in having more layers for increasing generalization ability in such models. This framework accommodates other network architectures and computing elements, including networks with correlated weights, convolutional networks and discretised variables. A similar approach also facilitates studying the distribution of Boolean functions computed by recurrent and layer-dependent architectures, which we find to be the same. Depending on the initial conditions and computing elements used, we characterize the space of functions computed at the large depth limit and show that the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with the growing depth.
Part of the LMS/IMA Joint Meeting on 'The Mathematical Foundations' of AI, which took place on Friday 13 October 2023 at De Morgan House, London and online via Zoom.
==========
The London Mathematical Society has, since 1865, been the UK's learned society for the advancement, dissemination and promotion of mathematical knowledge. Our mission is to advance mathematics through our members and the broader scientific community worldwide.
For further information:
► Website: www.lms.ac.uk
► Events: www.lms.ac.uk/events
► Grants and Prizes: www.lms.ac.uk/grants-prizes
► Publications: www.lms.ac.uk/publications
► Membership: www.lms.ac.uk/membership
Follow us:
► Twitter: / londmathsoc
► Facebook: / londonmathematicalsociety
► LinkedIn: / the-london-mathematica...
► KZbin: @LondonMathematicalSociety