AE Materials

On this page I am providing materials for an artifact evaluation tutorial that Dirk Beyer and I created for a tutorial at FSE 2025 in Trondheim.

In the hands-on session, we suggest to assess the following artifacts, which we have successfully worked with in the past.
Alternatively, participants may pick their favorite artifact from FSE 2025, but we have not have enough time to pre-download these.

Nr Year Title Paper Link Artifact Links Size Docker/VM
4 2018 Large-scale study of substitutability in the presence of effects https://doi.org/10.1145/3236024.3236075 - https://dl.acm.org/do/10.1145/3276998/full
- http://design.cs.iastate.edu/papers/FSE-18		 29M
5 2018 Parameterized model counting for string and numeric constraints https://doi.org/10.1145/3236024.3236064 - https://dl.acm.org/do/10.5281/zenodo.1420217/full
- https://github.com/vlab-cs-ucsb/ABC		 52M
6 2018 Path-based function embedding and its application to error-handling specification mining https://doi.org/10.1145/3236024.3236059 - https://dl.acm.org/do/10.5281/zenodo.1301239/full
- https://github.com/ucd-plse/func2vec-fse2018-artifact		 5.2M Docker
7 2019 A segmented memory model for symbolic execution https://doi.org/10.1145/3338906.3338936 - https://dl.acm.org/do/10.1145/3345840/full 1.4G Docker
8 2019 Empirical review of Java program repair tools: a large-scale experiment on 2,141 bugs and 23,551 repair attempts https://doi.org/10.1145/3338906.3338911 - https://dl.acm.org/do/10.5281/zenodo.3334854/full
- https://github.com/program-repair/RepairThemAll
- https://github.com/program-repair/RepairThemAll_experiment		 282M (Docker)
9 2020 A randomized controlled trial on the effects of embedded computer language switching https://doi.org/10.1145/3368089.3409701 - https://dl.acm.org/do/10.5281/zenodo.3911750/full
- https://doi.org/10.5281/zenodo.3911750		 3.4M
10 2020 Community expectations for research artifacts and evaluation processes https://doi.org/10.1145/3368089.3409767 - https://dl.acm.org/do/10.5281/zenodo.3951724/full
- https://bhermann.github.io/artifact-survey
- https://doi.org/10.5281/zenodo.3951724		 171M (Docker)
11 2020 Detecting numerical bugs in neural network architectures https://doi.org/10.1145/3368089.3409720 - https://dl.acm.org/do/10.5281/zenodo.3843611/full
- https://doi.org/10.5281/zenodo.3843648
- https://github.com/ForeverZyh/DEBAR		 29M (Docker)
12 2020 Domain-independent interprocedural program analysis using block-abstraction memoization https://doi.org/10.1145/3368089.3409718 - https://dl.acm.org/do/10.5281/zenodo.4024268/full
- https://www.sosy-lab.org/research/bam-interprocedural
- https://doi.org/10.5281/zenodo.4024268		 185M
13 2020 Efficient binary-level coverage analysis https://doi.org/10.1145/3368089.3409694 - https://dl.acm.org/do/10.5281/zenodo.3876048/full
- https://doi.org/10.5281/zenodo.3876047		 62M
14 2020 Flexeme: untangling commits using lexical flows https://doi.org/10.1145/3368089.3409693 - https://dl.acm.org/do/10.5281/zenodo.3894559/full
- https://pppi.github.io/Flexeme		 13M
15 2020 Fuzzing: on the exponential cost of vulnerability discovery https://doi.org/10.1145/3368089.3409729 - https://dl.acm.org/do/10.6084/m9.figshare.11911287.v1/full
- https://www.kaggle.com/marcelbhme/fuzzing-on-the-exponential-cost-of-vuln-disc
- https://doi.org/10.6084/m9.figshare.11911287.v1		 702M
16 2020 Making symbolic execution promising by learning aggressive state-pruning strategy https://doi.org/10.1145/3368089.3409755 - https://dl.acm.org/do/10.5281/zenodo.3902978/full
- https://github.com/kupl/HOMI_public		 3.7G VirtualBox
17 2020 Past-sensitive pointer analysis for symbolic execution https://doi.org/10.1145/3368089.3409698 - https://dl.acm.org/do/10.1145/3410246/full
- https://dl.acm.org/action/downloadSupplement?doi=10.1145%2F3368089.3409698&file=fse20main-p237-p-archive.zip&download=true
- https://doi.org/10.6084/m9.figshare.12487679		 521K Docker
18 2020 Recommending stack overflow posts for fixing runtime exceptions using failure scenario matching https://doi.org/10.1145/3368089.3409764 - https://dl.acm.org/do/10.6084/m9.figshare.11948619.v1/full
- https://doi.org/10.6084/m9.figshare.11948619		 106M
19 2020 Robotics software engineering: a perspective from the service robotics domain https://doi.org/10.1145/3368089.3409743 - https://dl.acm.org/do/10.6084/m9.figshare.12376931.v1/full
- https://sites.google.com/view/empirical-study-robotics-se/home
- https://figshare.com/articles/conference_contribution/Robotics_%20Software_Engineering_A_Perspective_from_the_Service_Robotics_Domain/%2012376931		 4.2M
20 2025 Pinning Is Futile: You Need More Than Local Dependency Versioning to Defend against Supply Chain Attacks https://doi.org/10.1145/3715728 - https://doi.org/10.5281/zenodo.15121080 524M
21 2025 ChatDBG: Augmenting Debugging with Large Language Models https://doi.org/10.1145/3729355 - https://doi.org/10.5281/zenodo.15185773 1.3M