How Does Machine Translation of User Interface Affect User Experience? A Study on Android Apps

For global-market-oriented software applications, their user interfaces need to be translated to local languages to facilitate users from different areas in the world. A long-term practice in software industry is to hire professional translators or translation companies to perform the translation. However, due to the large number of user-interface labels and target languages, this is often too expensive for software providers, especially cost-sensitive providers such as personal developers of Android apps. On the other hand, more and more mature machine translation techniques are providing a cheap though imperfect alternative, and the Google Translation service has been widely used for translating websites and apps. However, the effect of translation quality of GUI labels on user experience has not been well studied yet. In this paper, we present a user study on 6 popular Android apps, to have 24 participants perform tasks on app variants with 4 different translation quality levels and 2 target languages: Spanish and Chinese. From our study, we acquire the following 3 major findings, including (1) although sharing only about 30% of GUI labels with the original localized versions, machine translated versions of GUI have similar user experience on most studied aspects, except a slightly lower task completion rate; (2) manually enhanced but still imperfect machine translated versions are able to achieve exact the same user experience in almost all studied aspects; and (3) users are not satisfied with the GUI of machine translated versions and the two major complaints are misleading labels of input boxes, and unclear translation of items in option lists.