Phenomenological Understanding of Neural Networks

Topic: Towards a Phenomenological Understanding of Neural Networks

Abstract:
Neural networks are a powerful tool for an ever-growing list of tasks. However, their enormous complexity often complicates developing theories describing how these networks learn. In our recent work, inspired by the historical development of statistical mechanics, we have studied the use of collective variables to explain how neural networks learn, specifically, the von Neumann entropy and Trace of the empirical neural tangent kernel (NTK). We applied our analysis to the problem of data selection, showing that the entropy and trace of the NTK at the start of training can indicate the diversity of the training data and even predict the quality of the model after training. We further demonstrated that the popular reinforcement learning exploration algorithm, random network distillation, could be used to construct high-trace-entropy datasets. In this talk, I will introduce the theoretical methods behind our collective variable analysis and showcase recent results exploring other aspects of neural network training, including network dynamics, weight initialization, and optimizer algorithms.

Speaker:

Samuel Tovey completed his undergraduate studies at the University of Western Australia where he received a Bachelor of Science with a double Major in Physics and Electronics Engineering. He worked for approximately one year as an electronics engineer before moving to Stuttgart to complete his Masters in Physics where he wrote his thesis on the Development on Machine Learned Potentials for Molten Salts. After the completion of his thesis he applied for and was granted the Landesgraduiertenförderung (LGF) fellowship to undertake his PhD where he studies the role of machine learning in physics. This involves applying various methods in machine learning to problems in computational physics as well as utilizing tools from theoretical physics to better understand machine learning algorithms.