Tools evolving AI systems via experiment management: A survey of machine learning practitioners

Sammanfattning: Artificial intelligence employs machine learning to create intelligent systems. Experiment management tools have been created to support machine learning practitioners in their development efforts relating to the management of artifacts and metadata. Although the technical capabilities of such tools in terms of features have been widely examined, which tools are used as well as the tool´s benefits, limitations and challenges, remain unknown. This paper provides an empirical investigation addressing the questions previously stated. Those interested in gaining a better understanding of the users of these tools, such as tool developers and researchers looking for initial data on this topic, could find the results presented valuable. This was achieved by developing and distributing an online questionnaire to elicit qualitative and quantitative data concerning experiment management tools from 24 machine learning practitioners. Par ticipants reported benefiting from the tools in areas such as reproducibility, time savings, traceability, and result analysis. Reported challenges and limitations of the tools included a lack of features, quality and integration with other systems. Many participants combined tools in order to achieve the desired work flow. The three most commonly used tools were TensorBoard, MLFlow, and SageMaker. The empirical contributions of the survey improved the understanding of experiment management tools from the perspective of machine learning practitioners. The data can be leveraged towards building better supporting tools for AI development and serve as a basis for further research in related areas.