To foster replicable research, many conferences encourage the submission of ‘research artifacts’ along with papers. Artifacts essentially are anything that has contributed to generating the results presented in a research article. The submitted artifacts are evaluated by dedicated artifact evaluation committees (AECs). I’ve had the pleasure to serve on such AECs myself (ISSTA 2016 and ISSTA 2018) and we also received a positive evaluation for our artifact at ISSTA 2019.
One thing that I’ve been struggling with a bit is what makes a good artifact and I’ve had long and interesting discussions with Ben Hermann, who co-chaired the ISSTA 2018 AEC, on this topic. We realized that our views on artifact quality slightly differed in various nuances, which led us to the question if the perceptions of artifact quality generally differ across AEC members and what the causes and possible impact of such different perceptions are.
With the help of Janet Siegmund, who is an expert in qualitative surveys and a fantastic discussion partner when it comes to research methodology in general, we designed a questionnaire and invited all past AEC members from AECs at software engineering and programming language conferences to tell us about their perceptions of artifact purposes and quality.
The paper that discusses the results from this survey has just been accepted at FSE 2020, which is the best venue for the paper that I can imagine. FSE has pioneered artifact evaluations in the software engineering community and has been conducting these evaluations for almost a decade.
We thank all the anonymous participants of our study, the anonymous reviewers of our paper, and the AEC members that currently evaluate our research artifact. We hope our paper contributes to the continuous improvement of artifact evaluations and replicable research in general. A preprint of our paper and its research artifact are available here.