Automation of Formative Assessment : Implementation and Evaluation of an Artificial Teaching Assistant

Sammanfattning: The Swedish government has decided to add programming to the Swedish curriculum to strengthen students’ digital skills. The teachers who will teach programming do not always know programming themselves. Because of that, KTH Royal Institute of Technology researchers are planning to start an initiative of creating a Massive Open Online Course (MOOC) in fundamental programming in Swedish for those teachers. Interpreting error messages given by the compiler is one problem with learning programming. An aim of this study is to lower that threshold. The study seeks to identify common misconceptions about programming among novice programmers in order to design a static analyser that investigates code written by students and provides formative feedback to help students in their own learning process. The study combines the constructivist theory of learning with views about formative assessment in order to automate the process that is usually done by a teacher or teaching assistant when assessing code. A phenomenographic study is done in order to identify teachers’ perceptions of common misconceptions about programming by interviewing five active teachers at KTH. The results of that study are used to construct code examples that correspond to these misconceptions. Those results lead to the design and implementation of a software that detects these problems in code. That software is evaluated using a larger set of test data, consisting of 77 errors divided into five larger programs, inserted by independent individuals. From the initial study, five categories of misconceptions are given. Of the 77 errors, the majority are correctly positioned and almost all are given a good hint about the position. About a quarter of the errors are parse errors, which never reach the analysing part of the software that demands the program to be parsable. The study shows that we have succeeded both in designing and implementing a software that detects the identified misconceptions with good results. In the context of a MOOC, the software might require an extension with a more advanced parser and also dynamic analysis to be able to test the correctness of the students’ programs. The software is limited to handle the language Javascriptish, which is a subset of JavaScript.