Virtual Assistants and Their Performance In Professional Environments

Sammanfattning: Contributors from the mid 20th century up to now have developed and refined virtual assistants, taking the technology from a set of rules to assistants driven by Artificial Intelligence. Today, virtual assistants can provide value in organisation and support a sustainable society by conducting basic and repetitive tasks, and help reduce inequalities caused by biased advisors on sensitive topics. Despite its prosperity, current research somewhat lack focus on the evaluation of virtual assistants in industrial applications. The purpose of this paper is to evaluate virtual assistants from a technical, economical and organisational perspective, in order to understand their performance and value in an industrial environment. This has been done in collaboration with IBM and a client company which prefers to remain anonymous in this report. In this company, two IBM Watson Assistants are under development; one for the IT Service Desk, and one for the Ethics & Compliance department. To cover all aspects of the virtual assistants’ performance, quantitative and qualitative methods were used by conducting user testings and surveys. In this process, discussions have been conducted with IBM experts and employees of the firm for which the practical implementation has been studied, to gain a general and specific understanding from different perspectives. From this paper, the following can be concluded. First, technological performance can be described using quantitative metrics such as coverage, confidence, precision and helpfulness, and should be complemented using qualitative measures such as user satisfaction and perceived user understanding. Second, specific technological performance is relative and the technical limitations as well as it’s maturity should be used as a complement to the evaluation of the assistants. Third, identified organisational benefits include: • reduced time-to-resolution, • reduced handling time, • all-hour-support, • scalability and • user understanding Conclusions specific for the use cases show that an assistant implemented in a narrower use case, that is the Ethics & Compliance assistant, easier can be implemented and performs relatively well also in less developed environments. A broader use case, such as the IT assistant, requires more effort to perform at a high level but may be even more beneficial than in the narrow use case once sufficiently refined.