A Study on the Perception of Feedback with Varying Sentiment Generated Using a Large Language Model

Sammanfattning: Providing high-quality feedback for students is often a time-consuming task that the teaching staff does not have the resources for. Utilizing automatically generated feedback is a potential solution to this problem. When studying automated feedback, it is common to evaluate feedback in regard to students’ performance. However, this research aimed to measure students’ perception of feedback generated with different sentiments in order to see how future feedback should be formatted to be helpful to all students. In this study, feedback has been generated with varying sentiment and length using a Large Language Model, or more specifically using ChatGPT. The feedback that was generated was based on predetermined questions and answers. Furthermore, different prompt engineering techniques were used to craft effective prompts that produced the desired output. Opinions were then gathered from 79 respondents using an online survey, to measure how sentiment in the generated feedback affected recipients’ perception of the feedback. The positive feedback was most popular for all three predetermined questions in the survey and respondents deemed it as most specific to the answer, most constructive as well as most motivating. Despite this, a few respondents viewed the positive feedback as inauthentic due to its AI origin and therefore opted for the negative alternatives. The feedback preferences were also analyzed with respect to respondents’ gender identity, age and years of coding experience with the aim of finding out how feedback can be personalized and helpful to all students. Since the predetermined questions and answers regarded coding and AI, it was particularly important to study how previous coding experience affected feedback preferences among participants. Some differences were observed between feedback preferences of demographic groups including that participants show a growing acceptance of negative feedback as years of coding experience increase. However, the demographic results were not deemed significant enough upon performing statistical testing.