Calendar

Generally, we will discuss one research paper each class meeting. Please be sure to read the assigned paper before class. Where possible, I’ve directly linked to PDFs. However, some papers link to the ACM or IEEE library - you can sign in to those services using your Northeastern login (select “Sign in with institutional credentials” and then select Northeastern).

0. Introduction to Program Analysis and Testing Topics

Sep 10: Course Overview; Software Testing Basics, Intro to Program Analysis

1. Test Adequacy

Sep 14: Are mutants a valid substitute for real faults in software testing?, Just et al, FSE 2014
Sep 17: Assessing Oracle Quality with Checked Coverage Schuler & Zeller, ICST 2011

2. Creating Inputs - Some History

Sep 21: An empirical study of the reliability of UNIX utilities Miller et al, CACM 1990
Sep 24: AFL Historical Notes and Technical “Whitepaper” for AFL Michal Zalewski. For additional background: Mutation-Based Fuzzing chapter of The Fuzzing Book by Zeller et al

3. Test Oracles - Differential and Metamorphic

Sep 28: Deep Differential Testing of JVM Implementations Chen et al, ICSE 2019
Oct 1: Automated testing of graphics shader compilers Donaldson et al, OOPSLA 2017

4. Test Oracles - Runtime Property Checking

Oct 5: Dynamically Discovering Likely Program Invariants to Support Program Evolution Ernst et al, ICSE 1999
Oct 8: How Good Are the Specs? A Study of the Bug-Finding Effectiveness of Existing Java API Specifications Legunsen et al, ASE 2016

5. Creating Inputs - Fuzzing

Oct 12: Coverage-based Greybox Fuzzing as Markov Chain Böhme et al, CCS 2016
Oct 15: AFL++ : Combining Incremental Steps of Fuzzing Research Fioraldi et al, WOOT 2020

6. Creating Inputs - Symbolic Execution

Oct 19: KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs Cadar et al, OSDI 2008
Oct 22: On the Techniques We Create, the Tools We Build, and Their Misalignments: A Study of KLEE Rizzi et al, ICSE 2016; Reflection Paper Due 11:59pm

7. Creating Inputs - Structured Fuzzing

Oct 26: Finding and Understanding Bugs in C Compilers Yang et al, PLDI 2011
Oct 29: Semantic Fuzzing with Zest Padhye et al, ISSTA 2019; Project Proposal Due 11:59pm

8. Fuzzing Evaluations

Nov 2: FuzzBench: an open fuzzer benchmarking platform and service Metzman et al, FSE 2021; US Election Day
Nov 5: UNIFUZZ: A Holistic and Pragmatic Metrics-Driven Platform for Evaluating Fuzzers Li et al, USENIX Secuirty 2021

9. Test Oracles - Security

Nov 9: Automatic Creation of SQL Injection and Cross-Site Scripting Attacks, Kieżun et al, ICSE 2009
Nov 12: HotFuzz: Discovering Algorithmic Denial-of-Service Vulnerabilities Through Guided Micro-Fuzzing Blair et al, NDSS 2020

10. Harnesses - Test Generation

Nov 16: Feedback-directed Random Test Generation Pacheco et al, ICSE 2007
Nov 19: Evolutionary Generation of Whole Test Suites Fraser and Arcuri, QISC 2011

11. Once you Find a Bug - Debugging

Nov 23: Debugging Reinvented: Asking and Answering Why and Why Not Questions about Program Behavior Ko and Myers, ICSE 2008 (Prof Bell will join remotely)
Nov 26: No class - Thanksgiving Recess

12. Bonus Fuzzing

Nov 30: Fuzzing the Rust Typechecker Using CLP Dewey et al, ASE 2015
Dec 3: JUSTGen: Effective Test Generation for Unspecified JNI Behaviors on JVMs Hwang et al, ICSE 2021

13. Automated Program Repair

Dec 7: GenProg: A Generic Method for Automatic Software Repair Le Goues et al, TSE 2012
Dec 10: An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems Qi et al, ISSTA 2015

13. Conclusions

Dec 14: Casual project discussions; General course discussion
Dec 17: Project report due, 11:59pm