Using pose estimation to support video annotation for linguistic use : Semi-automatic tooling to aid researchers

Sammanfattning: Video annotating is a lengthy manual process. A previous research project, MINT, produced a few thousand videos of child-parent interactions in a controlled environment in order to study children’s language development. These videos were filmed across multiple sessions, tracking the same children from the age of 3 months to 7 years. In order to study the gathered material, all these videos have to be annotated with multiple kinds of annotations including transcriptions, gaze of the children, physical distances between parent and child, etc. These annotations are currently far from complete, which is why this project aimed to be a stepping point for the development of semi-automatic tooling in order to aid the process. To do this, state-of-the-art pose estimators were used to process hundreds of videos, creating pseudo-anonymized pose estimations. The pose estimations were then used in order to gauge the distance between the child and parent, and annotate the corresponding frame of the videos. Everything was packaged as a CLI tool. The results of first applying the CLI and then correcting the automatic annotations manually (compared to manually annotating everything) showed a large decrease in overall time taken to complete the annotating of videos. The tool lends itself to further development for more advanced annotations since both the tool and its related libraries are open source.