What’s (Not) Working in Programmer User Studies?

Date
2023-07-24
Authors
Davis, Matthew C.
Aghayi, Emad
Latoza, Thomas D.
Wang, Xiaoyin
Myers, Brad A.
Sunshine, Joshua
Journal Title
Journal ISSN
Volume Title
Publisher
Association for Computing Machinery
Abstract

A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.

Description
Keywords
empirical software engineering, user study, meta study, human participants, research methodology, human subjects, experiments, human-centered computing
Citation
Davis, M. C., Aghayi, E., Latoza, T. D., Wang, X., Myers, B. A., & Sunshine, J. (2023). What’s (Not) Working in Programmer User Studies? ACM Transactions on Software Engineering and Methodology, 32(5), Article 120. doi:10.1145/3587157
Department
Computer Science