The structure of this class is designed to provide a broad survey of many topics in software engineering practice and research:
- Software Process
- Modularity and Design
- Mining Software Repositories
- Open Source
- Continuous Integration
- Expertise and knowledge sharing
- Software engineering in specific domains
The goal of the reflection paper is for you to select a topic of particular personal interest, identify five research articles (that are not required readings for this course), read them, and organize your reaction to the topic in a reflection paper. The reflection paper provides you to an opportunity to apply critical reasoning skills in the context of the research papers that we discuss in class, and to dig deeper into a topic of your own particular interest.
This is not a literature review: the goal with this paper is for you to provide some editorial critiques of the work and express your own opinions, rather than to provide a fair and direct summary of the contents of each paper. The expected length of the reflection paper is roughly 2,500 words.
There are three main deliverables for the reflection paper:
- Paper proposal
- The reflection paper
- A short presentation in class of your findings
The objectives for the reflection paper are to:
- Compare and contrast different software testing and analysis approaches
- Critically analyze research projects
- Cultivate new ideas for research directions
Once you have selected the papers that you will respond to, consider one (or more) of the following prompts:
- What are common themes or goals between these papers? Do they share similar assumptions (in terms of how software is designed and tested)?
- How do these works complement each other?
- Based on your own experiences developing software, do you believe that the evaluations in these papers provide sufficient evidence to draw the conclusions that are claimed?
- If you have experience with other related works in software engineering, how do you think those works compare?
- Are there next-steps that you think could be applied to extend the ideas presented in these papers?
- Were there parts of some of these works that you thought were poorly explained, and could have been written in another way to be more accessible?
You should consider these prompts as jumping-off points: feel free to respond to one or more of these questions in your paper, or alternatively, something completely different. The most important part is that your paper should convey your own reactions to the six papers that you have selected — demonstrating that you read the papers and thought about them.
This is an individual assignment. The goal of this assignment is to provide your personal reflection to the topic of your choice. Papers that are substantially similar to each other (or to third-party content) will receive a score of 0, with no opportunity for a revision. If you have any questions or concerns of what constitutes plagiarism, please reach out to the instructor immediately.
Feel free to use whatever word processing environment you prefer. We strongly encourage that you use some editor that will help you manage a bibliography — like LaTeX + BibTex, or Word + Endnote. Please clearly identify the one paper from the required reading list for the course plus the five additional papers that you intend to discuss — one way to do that might be to include them in your bibliography, and cite them in the first paragraph of your paper.
To share your experience with the rest of the class, prepare a lightning talk (8 minutes) that summarizes what you learned about this topic, and your reaction to it. The goal of the lightning talk is to share one or two things that you learned that were interesting to you: an open (or closed) problem in that research area, some methodology or tool that addresses a known problem, etc. The presentation will be graded entirely on a “you did it” or “you didn’t” basis, as long as you show up at the assigned time and talk about the topic.
Your reflection paper will be graded on the scale of (Unacceptable, Check-, Check, Check+). The criteria for each grade are described below.
To receive the grade of Check, the paper/presentation must satisfy all of these criteria:
- The paper is approximately 2,500 words (plus or minus a few hundred is OK; if you are far short of 2,500 you might consider deepening your exploration; if you are far over 2,500 words, you might consider condensing your efforts)
- The paper responds to one of the articles on the course’s required reading list, plus the five articles that were approved by the course staff via the paper proposal
- The paper demonstrates a thoughtful understanding of the papers selected
- The paper uses relevant examples from the papers discussed in class to support and describe your reactions to the works
- The paper uses language that is understandable, and generally does not distract from the content of the work
- The presentation describes a concept related to the paper.
To receive the grade of Check-, the paper must satisfy all of these criteria:
- The paper is between 1,500-2,500 words
- The paper responds to one of the articles on the course’s required reading list, plus the five articles that were approved by the course staff via the paper proposal
- The paper demonstrates a limited understanding of the papers selected, focusing primarily on summarizing the content of the papers without demonstration of higher-level understanding of them
Submissions that do not meet the criteria for “Check-“ will receive the grade of “Unacceptable.” Submissions that exceed the qualities outlined for “Check” (e.g. include additional, insightful connections between the papers discussed in class and other works, uses particularly sophisticated and engaging language, and generally demonstrates an exceptional and thorough understanding of the selected papers) may receive the grade of “Check+”. As per the course policies, note that a grade of “Check” is sufficient to receive an A in the class.
Submit your paper on Canvas. The paper is due on March 14 at 11:00am. No late submissions will be accepted. Presentations will occur in-class on March 14 and 16.
Students who make a good faith effort on the paper (e.g. submit a document on time that is conformant to the general guidelines explained above), and receive a grade below “Check” will receive detailed feedback on their paper, and be allowed to resubmit the paper (within 2 weeks), under the understanding that the highest grade of the submissions will be used. Students who receive a grade of “Check” may not resubmit their work to attempt to achieve a “Check+” (again, note that in this situation, this would have no impact on your course grade).
We have aggregated all of the articlds selected by students for their reflection papers into lists organized by topic (topics with more articles listed reflect topics that more students selected for their reflection papers)
- “Comparative analysis of functional and object-oriented programming”, Alic et al.
- “A comparative study of programming languages in rosetta code”, Nanz and Furia
- “A large-scale study of programming languages and code quality in github”, Ray et al.
- “On the impact of programming languages on code quality”, Berger et al.
- “Code quality metrics for the functional side of the object-oriented language c#”, Zuilhof et al.
- K Fedoseev, N Askarbekuly, A E Uzbekova, and M Mazzara. Application of data-oriented design in game development. Journal of Physics: Conference Series, 1694(1):012035, dec 2020.
- Apostolos Ampatzoglou and Alexander Chatzigeorgiou. Evaluation of object-oriented design patterns in game development. Information and Software Technology, 49(5):445–454, 2007.
- Weishan Zhang, Dong Han, Thomas Kunz, and Klaus Marius Hansen. Mobile game development: Object-orientation or not. In 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), volume 1, pages 601–608, 2007.
- David Wingqvist, Filip Wickström, and Suejb Memeti. Evaluating the performance of object-oriented and data-oriented design with multi-threading in game development. In 2022 IEEE Games, Entertainment, Media Conference (GEM), pages 1–6, 2022.
- Jessica D. Bayliss. The data-oriented design process for game development. Computer, 55(5):31–38, 2022.
- Jessica D. Bayliss. Developing games with data-oriented design. In Proceedings of the 6th International ICSE Workshop on Games and Software Engineering: Engineering Fun, Inspiration, and Motivation, GAS ’22, page 30–36, New York, NY, USA, 2022. Association for Computing Machinery.
- Neamtiu, I., Xie, G., Chen, J. (2013). Towards a better understanding of software evolution: an empirical study on open-source software. Journal of Software: Evolution and Process, 25(3), 193-218.
- Comino, S., Manenti, F. M. (2011). Dual licensing in open source software markets. Information Economics and Policy, 23(3-4), 234-242.
- Capra, E., Francalanci, C., Merlo, F. (2008). An empirical study on the relationship between software design quality, development effort and governance in open source projects. IEEE Transactions on Software Engineering, 34(6), 765-782.
- Tan, X., Zhou, M., Fitzgerald, B. (2020, June). Scaling open source communities: An empirical study of the Linux kernel. ACM/IEEE 42nd International Conference on Software Engineering (pp. 1222-1234).
- Qiu, H. S., Li, Y. L., Padala, S., Sarma, A., Vasilescu, B. (2019). The signals that potential contributors look for when choosing open-source projects. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1-29..
- J. Coelho and M. T. Valente, “Why modern open source projects fail,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, (New York, NY, USA), p. 186–196, Association for Computing Machinery, 2017.
- C. Mendez, H. S. Padala, Z. Steine-Hanson, C. Hilderbrand, A. Horvath, C. Hill, L. Simpson, N. Patil, A. Sarma, and M. Burnett, “Open source barriers to entry, revisited: A sociotechnical perspective,” in Proceedings of the 40th International Conference on Software Engineering, ICSE ’18, (New York, NY, USA), p. 1004–1015, Association for Computing Machinery, 2018.
- X. Tan, M. Zhou, and B. Fitzgerald, “Scaling open source communities: An empirical study of the linux kernel,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ICSE ’20, (New York, NY, USA), p. 1222–1234, Association for Computing Machinery, 2020.
- E. Dias, P. Meirelles, F. Castor, I. Steinmacher, I. Wiese, and G. Pinto, “What makes a great maintainer of open source projects?,” in Proceedings of the 43rd International Conference on Software Engineering, ICSE ’21, p. 982–994, IEEE Press, 2021.
- T. Dey, A. Karnauch, and A. Mockus, “Representation of developer expertise in open source software,” in Proceedings of the 43rd International Conference on Software Engineering, ICSE ’21, p. 995–1007, IEEE Press, 2021.
- D. Riehle, P. Riemer, C. Kolassa, and M. Schmidt, “Paid vs. volunteer work in open source,” in Proceedings of the Annual Hawaii International Conference on System Sciences, 01 2014.
- Dirk Beyer, Adam J Chlipala, Thomas A Henzinger, Ranjit Jhala, and Rupak Majumdar. 2004. Generating tests from counterexamples. In Proceedings. 26th International Conference on Software Engineering. IEEE, 326–335.
- Cristian Cadar, Daniel Dunbar, Dawson R Engler, et al. 2008. Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.. In OSDI, Vol. 8. 209–224.
- Koen Claessen and John Hughes. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming. 268-279.
- George Fink and Matt Bishop. 1997. Property-based testing: a new approach to testing for assurance. ACM SIGSOFT Software Engineering Notes 22, 4 (1997), 74-80.
- Thomas A Henzinger, Ranjit Jhala, Rupak Majumdar, and Gregoire Sutre. 2002. Lazy abstraction. In Proceedings of the 29th ACM SIGPLAN-SIGACT symposium on Principles of programming languages. 58-70.
- Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN conference on Programming language design and implementation. 213–223.
- “Web accessibility evaluation methods: A systematic review. Design, User Experience, and Usability”, Nuñez et al.
- “A brief survey of current software engineering practices in continuous integration and automated accessibility testing”, Sane
- “Aid: An automated detector for gender-inclusivity bugs in OSS Project Pages”, Chatterjee et al.
- “Mobile accessibility guidelines adoption under the perspective of developers and designers”, de Almeida and Gama
- “Evaluating responsive web design’s impact on Blind Users”, Nogueira et al.
- “Accessibility of UI frameworks and libraries for programmers with visual impairments”, Pandey et al.
- Lin, Jun-Wei, Reyhaneh Jabbarvand, and Sam Malek. “Test transfer across mobile apps through semantic mapping.” 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019.
- Behrang, Farnaz, and Alessandro Orso. “Test migration between mobile apps with similar functionality.” 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019.
- Mariani, Leonardo, et al. “Semantic matching of gui events for test reuse: are we there yet?.” Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 2021.
- Bajammal, Mohammad, and Ali Mesbah. “Semantic Web Accessibility Testing via Hierarchical Visual Analysis.” 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021.
- Walsh, Thomas A., Phil McMinn, and Gregory M. Kapfhammer. “Automatic detection of potential layout faults following changes to responsive web pages (N).” 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2015.
- “Automated Model-Based Android GUI Testing Using Multi-Level GUI Comparison Criteria”, Baek and Bae
- “Automated testing of virtual reality application interfaces.”, Bierbaum et al.
- “On the efficiency of automated testing”, Böhme and Paul
- Automated Accessibility Testing of Mobile Apps”, Eler et al.
- Robert Nilsson and Jeff Offutt. 2007. Automated Testing of Timeliness : A Case Study. (2007).
- T.Y Chen, S.C. Cheung, and S.M. Yiu. Metamorphic testing: a new approach for generating next test cases.
- Kyle Dewey, Jared Roesh, and Ben Hardekopf. Language fuzzing using constraint logic programming.
- Kyle Dewey, Jared Roesch, and Ben Hardekopf. Fuzzing the Rust Typechecker UsingCLP.
- Sungjae Hwang, Sungho Lee, Jihoon Kim, and Sukyoung Ryu. JUSTGen: Effective Test Generation for Unspecified JNI Behaviors on JVMs.
- Xuejun Yang, Yang Chen, Eric Eide, and John Regehr. Finding and Understanding Bugs in C Compilers.
- Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, and John Regehr. Swarm testing. In Proc. ISSTA, pages 78–88, July 2012
- Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE Transactions on Software Engineering, vol. 37, no. 5. pp. 649–678, 2011.
- G. Petrović, M. Ivanković, B. Kurtz, P. Ammann, and R. Just, “An industrial application of mutation testing: Lessons, challenges, and research directions,” in Proceedings - 2018 IEEE 11th International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2018, Jul. 2018, pp. 47–53.
- G. Petrovic, M. Ivankovic, G. Fraser, and R. Just, “Practical Mutation Testing at Scale: A view from Google,” IEEE Transactions on Software Engineering, vol. 48, no. 10. Institute of Electrical and Electronics Engineers Inc., pp. 3900–3912, Oct. 01, 2022.
- R. J. Lipton and F. G. Sayward, “Hints on test data selection: Help for the practicing programmer,” Computer (Long Beach Calif), vol. 11, no. 4, pp. 34–41, 1978.
- M. Ivankovi, G. Petrovi, R. Just, and G. Fraser, “Code coverage at Google,” in ESEC/FSE 2019 - Proceedings of the 2019 27th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Aug. 2019, pp. 955–963.
- “Towards Automating the AI Operations Lifecycle”, Arnold et al.
- “Towards ML engineering: a brief history of TensorFlow Extended (TFX)”, Karmarkar et al.
- Leonel Aguilar, David Dao, Shaoduo Gan, Nezihe Merve Gurel, Nora Hollenstein, Jiawei Jiang, Bojan Karlas, Thomas Lemmin, Tian Li, Yang Li, et al. Ease. ml: A lifecycle management system for mldev and mlops. Proc. of Innovative Data Systems Research, 2021.
- Alex Serban, Koen van der Blom, Holger Hoos, and Joost Visser. Adoption and effects of software engineering best practices in machine learning. In Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1–12, 2020.
- Nadia Nahar, Shurui Zhou, Grace Lewis, and Christian Kästner. Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process. In Proceedings of the 44th International Conference on Software Engineering, pages 413–425, 2022.
- Tongfei Chen, Jirı́ Navrátil, Vijay Iyengar, and Karthikeyan Shanmugam. Confidence scoring using whitebox meta-models with linear classifier probes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1467–1475. PMLR, 2019.
- Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
- “Testing Practices for Infrastructure as Code”, Hasan et al.
- Hummer, W., Rosenberg, F., Oliveira, F., & Eilam, T. (2013). Testing Idempotence for Infrastructure as Code. In D. Eyers & K. Schwan (Eds.), Middleware 2013 (pp. 368–388). Berlin, Heidelberg: Springer Berlin Heidelberg.
- “Survey of DevOps Concepts and Challenges”, Leite et al.
- “The Seven Sins: Security Smells in Infrastructure as Code Scripts”, Rahman et al.
- “CI/CD Pipelines Evolution and Restructuring: A Qualitative and Quantitative Study”, Zampetti et al.
- “An Evolutionary Study of Configuration Design and Implementation in Cloud Systems”, Zhang et al.
- R. Kavitha and M. Irfan Ahmed, “A knowledge management framework for agile software development teams,” in 2011 International Conference on Process Automation, Control and Computing, pp. 1–5, 2011.
- S. Mirhosseini and C. Parnin, “Docable: Evaluating the executability of software tutorials,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conferencve and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020, (New York, NY, USA), p. 375–385, Association for Computing Machinery, 2020.
- A. Horvath, M. X. Liu, R. Hendriksen, C. Shannon, E. Paterson, K. Jawad, A. Macvean, and B. A. Myers, “Understanding how programmers can use annotations on documentation,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, (New York, NY, USA), Association for Computing Machinery, 2022.
- T. D. LaToza, M. Arab, D. Loksa, and A. J. Ko, “Explicit programming strategies,” Empirical Software Engineering, vol. 25, no. 4, p. 2416–2449, 2020.
- “Social media and tacit knowledge sharing: Developing a conceptual model”, Panahi et al.
- “An exploratory study of live-streamed programming”, Alaboudi and LaToza
- “How social and communication channels shape and challenge a participatory culture in software development”, Storey et al.
- “Analyzing user comments on YouTube coding tutorial videos”, Poché et al.
- “Influence of social and technical factors for evaluating contribution in GitHub”, Tsay et al.
- S. Baltes and S. Diehl, “Towards a theory of software development expertise,” ESEC/FSE 2018: Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 187-200, 2018.
- F. O. Bjørnson and T. Dingsøyr, “Knowledge management in software engineering: A systematic review of studied concepts, findings and research methods used,” Information and Software Technology, vol. 50, no. 11, pp. 1055-1068, 2008.
- M. Zhou and A. Mockus, “Developer fluency: achieving true mastery in software projects,”FSE ‘10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering, pp. 137-146, 2010.
- M. Caulo, B. Lin, G. Bavota, G. Scanniello and M. Lanza, “Knowledge Transfer in Modern Code Review,” ICPC ‘20: Proceedings of the 28th International Conference on Program
- C. Treude and M.-A. Storey, “Effective communication of software development knowledge through community portals,” ESEC/FSE ‘11: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pp. 91-101, 2011.
- “Backstabber’s Knife Collection: A Review of Open-Source Software Supply Chain Attacks”, Ohm et al.
- “Containing Malicious Package Updates in npm with a Lightweight Permission System”, Ferreira et al.
- “On the impact of security vulnerabilities in the npm package dependency network”, Decan et al.
- “Towards Measuring Supply Chain Attacks on Package Managers for Interpreted Languages”, Duan et al.
- “Managing Messes in Computational Notebooks”, Head et al.
- ” Refactoring in Computational Notebooks”, Liu et al.
- “Understanding and improving the quality and reproducibility of Jupyter notebooks”, Pimentel et al.
- “Error identification strategies for Python Jupyter notebooks”, Robinson et al.
- “A static analysis framework for data science notebooks”, Subotić et al.
- “Novice Reflections on Debugging”, Whalley et al.
- “Towards Effective Foraging by Data Scientists to Find Past Analysis Choices”, Kerry et al.
- “Diff in the Loop: Supporting Data Comparison in Exploratory Data Analysis”, Wang et al.
- Jeremiah W Johnson. Benefits and pitfalls of jupyter notebooks in the classroom. In Proceedings of the 21st Annual Conference on Information Technology Education, pages 32–37, 2020.
- Luigi Quaranta, Fabio Calefato, and Filippo Lanubile. Eliciting best practices for collaboration with computational notebooks. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW1):1–41, 2022.
- A. W. Appel, “Coq’s vibrant ecosystem for verification engineering (invited talk),” in Proceedings of the 11th ACM SIGPLAN international conference on certified programs and proofs, 2022, pp. 2–11.
- F. van Doorn, G. Ebner, and R. Y. Lewis, “Maintaining a library of formal mathematics,” in Intelligent computer mathematics: 13th international conference, CICM 2020, bertinoro, italy, july 26–31, 2020, proceedings, 2020, pp. 251–267.
- K. Roe and S. Smith, “CoqPIE: An IDE aimed at improving proof development productivity: (Rough diamond),” in Interactive theorem proving: 7th international conference, ITP 2016, nancy, france, august 22-25, 2016, proceedings 7, 2016, pp. 491–499.
- A. Celik, K. Palmskog, M. Parovic, E. J. G. Arias, and M. Gligoric, “Mutation analysis for coq,” in 2019 34th IEEE/ACM international conference on automated software engineering (ASE), 2019, pp. 539–551.
- T. Ringer, R. Porter, N. Yazdani, J. Leo, and D. Grossman, “Proof repair across type equivalences,” in Proceedings of the 42nd ACM SIGPLAN international conference on programming language design and implementation, 2021, pp. 112–127.
- Alshayban, A., Ahmed, I., & Malek, S. (2020, June). Accessibility issues in android apps: State of affairs, sentiments, and ways forward. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 1323-1334).
- Perera, H., Hussain, W., Whittle, J., Nurwidyantoro, A., Mougouei, D., Shams, R. A., & Oliver, G. (2020, June). A study on the prevalence of human values in software engineering publications, 2015-2018. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 409-420).
- Beede, E., Baylor, E., Hersch, F., Iurchenko, A., Wilcox, L., Ruamviboonsuk, P., & Vardoulakis, L. M. (2020, April). A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1-12).
- Garde, S., & Knaup, P. (2006). Requirements engineering in health care: the example of chemotherapy planning in paediatric oncology. Requirements Engineering, 11, 265-278.
- Sutcliffe, A., Sawyer, P., Liu, W., & Bencomo, N. (2021, May). Investigating the potential impact of values on requirements and software engineering. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS) (pp. 39-47). IEEE.
- “Stepping OCaml”
- “Stepping lazy programs”
- “Finding latent code errors via machine learning over program executions”
- “Checking Inside the Black Box: Regression Testing Based on Value Spectra Differences”
- “Finding Causes of Program Output with the Java Whyline”