Investigating Deep Learning algorithms for end-to-end language-based interaction with domestic robots

Sammanfattning: A socially assistive robot capable of helping with domestic work through understanding natural language instructions is still considered a difficult challenge. This work investigates how deep learning algorithms could help us to achieve this goal. Specifically, it focuses on solving the problem of enabling robots to identify objects while navigating in a house environment with language-based interactions. The proposed challenge is solved by implementing three different models. The first model relates the home objects to its typical locations in home regions by solving a classification problem through a neural network architecture. The second model is focused on navigating by understanding language-based commands. This model is solved through a LSTM-based sequence-to-sequence model with an attention mechanism over the language instructions, based on Anderson et al. [1] work. Finally, the third one is centered on identifying the target object by comprehending its associated referring expression. This last model is based on Hatori et al. [2] listener model. Each model is evaluated by using different data-sets suitable to the task. To accomplish the thesis, Matterport3D simulator is used as the main home environment. The purpose of this work is to analyse and study the limitations of the current solutions and the possible problems that we could face when implementing this in a real scenario. Hence, limitations and conclusions from each of the steps are properly stated.