Investigating Relations between Regularization and Weight Initialization in Artificial Neural Networks

Sammanfattning: L2 regularization is a common method used to prevent overtraining in artificial neural networks. However, an issue with this method is that the regularization strength has to be properly adjusted for it to work as intended. This value is usually found by trial and error which can take some time, especially for larger networks. This process could be alleviated if there was a mathematical relationship that predicted the best strength based on the network's hyperparameters. The aim of this project is to prove part of such a relation, specifically if the optimal regularization strength is proportional to the inverse number of training patterns. This was tested using a network that solves binary classification problems. Several regularization strengths were tested for different amounts of training patterns. The best ones were compared to the proposed relation. Additional tests were performed to check if weight initialization had an effect on said relation, and if it works for L1 regularization as well.