Fisher information flow in neural networks

Abstract

Deep Neural Networks are exceedingly successful at learning complicated, non-linear relations from data sets and a large amount of physical problems can be approached with this technique. However, little is understood about the internal processes that take place during the training. Inspired by light-scattering through complex, diffusive media, we analyse Neural Networks using the estimation theoretic concept of Fisher Information. In this thesis, we train Artificial Neural Networks to estimate continuous parameters from data that is generated from a given probability distribution. It is show that properly trained Deep Neural Networks allow the Fisher Information to pass through its layered structure while staying almost constant. Errors within the network can be interpreted as sinks for the Fisher Information flow.