What’s (Not) Working in Programmer User Studies?




Davis, Matthew C.
Aghayi, Emad
Latoza, Thomas D.
Wang, Xiaoyin
Myers, Brad A.
Sunshine, Joshua

Journal Title

Journal ISSN

Volume Title


Association for Computing Machinery


A key goal of software engineering research is to improve the environments, tools, languages, and techniques programmers use to efficiently create quality software. Successfully designing these tools and demonstrating their effectiveness involves engaging with tool users—software engineers. Researchers often want to conduct user studies of software engineers to collect direct evidence. However, running user studies can be difficult, and researchers may lack solution strategies to overcome the barriers, so they may avoid user studies. To understand the challenges researchers face when conducting programmer user studies, we interviewed 26 researchers. Based on the analysis of interview data, we contribute (i) a taxonomy of 18 barriers researchers encounter; (ii) 23 solution strategies some researchers use to address 8 of the 18 barriers in their own studies; and (iii) 4 design ideas, which we adapted from the behavioral science community, that may lower 8 additional barriers. To validate the design ideas, we held an in-person all-day focus group with 16 researchers.



empirical software engineering, user study, meta study, human participants, research methodology, human subjects, experiments, human-centered computing


Davis, M. C., Aghayi, E., Latoza, T. D., Wang, X., Myers, B. A., & Sunshine, J. (2023). What’s (Not) Working in Programmer User Studies? ACM Transactions on Software Engineering and Methodology, 32(5), Article 120. doi:10.1145/3587157


Computer Science