The effect of model calibration on noisy label detection

Sammanfattning: The advances in deep neural networks in recent years have opened up the possibility of using image classification as a valuable tool in various areas, such as medical diagnosis from x-ray images. However, training deep neural networks requires large amounts of annotated data which has to be labelled manually, by a person. This process always involves a risk of data getting the wrong label, either by mistake or ill will, and training a machine learning model on mislabelled images has a negative impact on accuracy. Studies have shown that deep neural networks are so powerful at memorization that if they train on mislabelled data, they will eventually overfit this data, meaning learning a data representation that does not fully mirror real data. It is therefore vital to filter out these images. Area under the margin is a method that filters out mislabelled images by observing the changes in a network’s predictions during training. This method does however not take into consideration the overconfidence in deep neural networks and the uncertainty of a model can give indications of mislabelled images during training. Calibrating the confidence can be done through label smoothing and this thesis aims to investigate if the performance of Area under the margin can be improved when combined with different smoothing techniques. The goal is to develop a better insight into how different types of label noise affects models in terms of confidence, accuracy and the impact it has depending on the dataset itself. Three different label smoothing techniques will be applied to evaluate how well they can mitigate overconfidence, prevent the model from memorizing the mislabelled samples and if this can improve the filtering process for the Area under the margin method. Results show when training on data with noise present, adding label smoothing improves accuracy, an indication of noise robustness. Label noise is seen to decrease confidence in the model and at the same time reduce the calibration. Adding label smoothing prevents this and allows the model to be more robust as the noise rate increases. In the filtering process, label smoothing was seen to prevent correctly labelled samples to be filtered and received a better accuracy at identifying the noise. This did not improve the classification results on the filtered data, indicating that it is more important to filter out as many mislabelled samples as possible even if this means filtering out correctly labelled images as well. The label smoothing methods used in this work was set up to preserve calibration, a future topic of research could be to adjust the hyperparameters to increase confidence instead, focusing on removing as much noise as possible.