Discussion: Continuous Integration

These notes roughly capture the (mostly unedited) comments from the class:

Summary of opinions and perceptions

Tradeoffs in CI: significance

• Not very: 3
• Unsure: 6
• Very: 10 Are these tradeoffs that could already be known in the small? Tradeoffs in CI: Empirically sound Not very: 0
• Unsure: 11
• Very: 8 Taming Google CI: significance within google? Not very: 0
• Unsure: 4
• Very: 15 Taiming Google CI: significance outside of google? Not very:2
• Unsure: 10
• Very: 7

Tradeoffs in CI

• Methodology
  • Interviews
    • 16 participants
    • Open-ended
    • Coded results
  • Survey
    • Broad population: distribute on twitter, Facebook, reddit
    • Focused population: Target developers at pivotal

Barriers & ideas

• Troubleshooting CI build failures
  • Some may have nothing to do with CI itself
  • Something that makes debugging CI easier: being able to reproduce same environment, and/or connect to it
• Overly long build times
  • Coffee break, lunch break, sleep break, weekend, winter break
  • Developer approach: ignore some unlikely to fail tests
  • “Regression test selection”: Compute for each test: which statements it covers
• Automating the build process
• Lack of support for desired workflow
  • What is organizational culture to build support for these kinds of workflows? Tied to awareness of what CI can perform
  • Today: GitHub actions workflow.yml, action.yml
• Security and access control:
  • Access to “secrets” (aka API key)
    • Vulnerability: secret is publicly readable
    • Vulnerability/attack path: untrusted code could be merged in and have access to that secret

Needs:

• Easier configuration of CI servers or services
  • Bamboo :’(
  • There is usually a big range of skills needed between the technology that is being tested and the technology used for testing
• Better tool integration
  • “There are probably CIs that already do that, but ours doesn’t”
• Better container/virtualization support
• Debugging assistance
• UI for modifying CI configs
• Better notifications
  • “I get 10 emails and I delete them immediately”
• Making it easier to onboard new users to the platform
  • How to extend existing workflows and create new ones
  • If the process is bad enough, then no tool can help you. Bad practices in dev processes get baked into CI processes get baked into CI tools.
  • What is the current state of best-practices in CI?

Discussion

• Any surprising insights, even given past 5 years time?
  • Using CI might be the difference between people actually running the tests and it not happening
• Security restriction in CI tools
  • The workarounds that devs use for running systems and debugging them in CI are not optimal…

Taming Google Scale Continuous Testing

• Bazel
• Monorepo
• Declare dependencies on other modules
• When going to test module A, collect all “affected” modules of module A, run their tests

• Commits -> “changelist”

• “Milestones” optimize when to choose to run which tests
• Frequently modified source files are more often to be involved in test breakages
  • “You are modifying a file that was modified 41 times in the past. The probability that you will cause a breakage is 75%.”
  • Maybe for these files, you go deeper in dependency graph
  • Does separating them into smaller components reduce the impact of this?
  • Maybe you run these more often.
  • More defect-inducing files should be less likely to put into same milestone
• What are the unknown unknowns (the causes of the failures?)
• Impressed by the number of bugs found by downstream test running

---