Assessment of Predictive Models for Improving Default Settings in Streaming Services

Sammanfattning: Streaming services provide different settings where customers can choose a sound and video quality based on personal preference. The majority of users never make an active choice; instead, they get a default quality setting which is chosen automatically for them based on some parameters, like internet connection quality. This thesis explores personalising the default audio setting, intending to improve the user experience. It achieves this by leveraging machine learning trained on the fraction of users that have made active choices in changing the quality setting. The assumption that user similarity in users who make an active choice can be leveraged to impact user experience was the idea behind this thesis work. It was issued to study which type of data from different categories: demographic, product and consumption is most predictive of a user's taste in sound quality. A case study was conducted to achieve the goals for this thesis. Five predictive model prototypes were trained, evaluated, compared and analysed using two different algorithms: XGBoost and Logistic Regression, and targeting two regions: Sweden and Brazil. Feature importance analysis was conducted using SHapley Additive exPlanations(SHAP), a unified framework for interpreting predictions with a game theoretic approach, and by measuring coefficient weights to determine the most predictive features. Besides exploring the feature impact, the thesis also answers how reasonable it is to generalise these models to non-selecting users by performing hypothesis testing. The project also covered bias analysis between users with and without active quality settings and how that affects the models. The models with XGBoost had higher performance. The results showed that demographic and product data had a higher impact on model predictions in both regions. Although, different regions did not have the same data features as most predictive, so there were differences observed in feature importance between regions and also between platforms. The results of hypothesis testing did not indicate a valid reason to consider the models to work for non-selective users. However, the method is negatively affected by other factors such as small changes in big datasets that impact the statistical significance. Data bias in some data features was found, which indicated a correlation but not the causation behind the patterns. The results of this thesis additionally show how machine learning can improve user experience in regards to default sound quality settings, by leveraging models on user similarity in users who have changed the sound quality to the most suitable for them.