A couple of weeks before I met Ryan Gerard at CAST this year he was speaking at Google’s Seattle Conference on Scalability. The gist of his talk was that running all your regression tests all the time is inefficient and if you could determine which ones would get you the most bang-for-buck for a particular build then you are much better off. Seems like a good theory to me. Of course, he recently left Symantec so we’re unlikely to get periodic updates on how things are progressing.
- Regression testing
- doesn’t scale
- often tests everything instead of just the things that actually changed
- is people intensive
- So, given a set of source code changes, determine the set of tests that need to be run
- There are (at least) 3 different ways to associate source code deltas to test cases
- requirements – test cases checked-in for this requirement are associated with this chunk of code
- defects – last time this code was checked in, this test case failed
- build correlation – code was checked in for this build failed this set of test cases
- When you start, the requirements type will be the most (only) important type, but over time the build correlation becomes primary coupled with the defects
- The implementation of this seems to have your tests skewed towards the automated side of the scale vs. the manual or exploratory side.
- Some policies that make this possible
- end-to-end traceability for requirements
- in bug fix check-ins, include the bug number in the comments so you can parse the data and determine which files changed
- record the changes that are the diff between the two builds
- I’m pretty sure Ryan suggested that you integrate with your build system first, but if he did not I (also) think that is a smart place to start
Direct link to video here.