A Label-based Conditional Mutual Information Estimator using Contrastive Loss Functions

Sammanfattning: In the field of machine learning, representation learning is a collection of techniques that transform raw data into a technology that can be effectively developed by machine learning. In recent years, deep neural network-based representation learning technology has been widely used in image learning, recognition, classification and other fields, and one of the representative ones is the mutual information estimator/encoder. In this thesis, a new form of contrastive loss function that can be applied to existing mutual information encoder networks is proposed. Deep learning-based representation learning is very different from traditional machine learning feature extraction algorithms. In general, the features obtained by feature extraction are surface-level and can be understood by humans, while the representation learning learns the underlying structures of data, which are easy to be understood by machines but difficult for humans. Based on the above differences, when the scale of data is small, human’s prior knowledge of the data can play a big role, so the feature extraction algorithm has a greater advantage; when the scale of data increases, the component of the prior knowledge will decline sharply. At this time, the strong computing ability of deep learning is needed to make up for this deficiency, thus the effect of representation learning will be better. The research done in this thesis is mainly aimed at a more special situation, where the scale of training data is small and the scale of test data is large. In this case, there are two issues that need to be considered, one is the distribution representation of the model, and the other is the overfitting problem of the model. The LMIE (label-based mutual information estimator) model proposed in this thesis has certain advantages regarding both issues. The LMIE model mainly contains three parts: (a) a neural network based-mutual information encoder; (b) a loss function calculation module; (c) a linear classifier. Among them, the loss function calculation module is the most important one, as well as the main factor that distinguishes this model from other models.