Chapter 11 – Software Testing

• Software Testing is defined as the execution of a program to find its fault
• Program testing can be used to show the presence of bugs, buy never the absence – Dijkstra, 1969
• Testing is an inefficient way to find and remove many types of bugs
• Once a program has been tested, the odds that it will work correctly under all conditions are only moderately improved than if you didn’t test. That’s not an excuse though.
• Some definitions
  • Verification – the process of proving the program’s correctness
  • Validation – an attempt to find errors by executing a program in a real environment
  • Debugging – Diagnosing the precise nature of a known error and then correcting it (note: this is a corrective activity, not testing one)
• Properly run unit tests are potentially capable of detecting as many as 70% of the defects in a program. Therefore, unit test the heck out of your program before Test gets it
• Testing, like almost every other activity, must start with objectives
• As the number of detected defects increases, so too does the number of undetected
• Testing should be viewed as a search for symptoms rather than bugs
• The real value of testing is the data obtained
• Error seeding – purposely inject known defects into the system and note what percent are found by testing
• Programmers are inherently incapable of effectuvekt testing their own programs. Not only do they feel guilty about the existence of byg, but they are biased by the creative work they have done. This inherently blinds them to their errors, not because they won’t see them but because they can’t.

Types of Software Tests

• Unit Testing
  • Is essentially path testing
  • The simplist approach is to ensure that every statement is exercised at least once
  • More stringent approach is to require covreage of every path within a program
  • Are written by the programmer who wrote the code, who is unfortunatly the person least likely to recognize a fault
  • Highest error yield of all testing techniques. So unit test already!
• Integration Testing
  • Bottom-Up
    • Modules are individually tested, using specially developed drivers that provide the needed system functions. As more modules are integrated, they replace the drivers
    • Main disadvantage is the need for special drivers
  • Top-down
    • Essentially is prototyping; each new module adds capabilities
    • Disadvantage: The need for program stubs to simulate not yet developed components
    • Disadvantage: Can be difficult (impossible?) to test error handling etc. until entire system is in place
  • Big-Bang
    • Each module is tested independently then assembled together to form the complete system
    • Disadvantage: Problems can be extremely complex to diagnose
    • Keep the critical path of modules in mind as a delay in a single module with delay a Big-Bang integration effort
  • The solution to all these is Continuous Integration. Yes folks, this was the solution presented in a book published in 1989. So why is not everyone doing this if this has been recognized at least 17 years ago
• Functional Testing
  • Exercise a program to it’s external specifications
  • If sufficiently detailed requirements are not available to design functional tests, they likely were not adequate for program design either. Amen
  • Progressive phase testing tests new functions, uncovering problems in newly added or modified module and their interfaces with previously integrated modules
  • Regressive phase testing concerns changes to previously integrated code by newly introduced code
  • Only run full test suite occasionally
  • Run only the relevant tests for that build consistently. In other words, if the certificate verification functionality has not been modified, there is not much added value in running the suite every build. Perhaps every third or fourth depending on the build spread
  • Data from previous test runs can help optimize which tests are currently running
  • If you do not have written objectives for you product, or those objectives are not measurable, you cannot do a system test – Myers
  • System tests should examine as many pathological situations as practical to ensure that the system does not behave pathological as well
  • Eat your own dog food. If you could be running your application is a ‘real’ situation in your company, you should be.
  • Emphasis not mine: As applications become more mission critical, behavior under stressful or anomalous conditions becomes more important. it is at the point of highest system stress when system behavior is generally most critical and when there will be the least tolerance for error. These high-stress periods are precisely when failures are more likely, when system behaviors is least predictable, and when the operators are least able to handle the problems. The modest critical systems should thus be stressed most thoroughly during tests that include performance, recovery and human factors issues
• Types of System Tests
  • Load/Stress – Identify the peak load conditions at which the system fails
  • Volume – Identify the level of continuous heavy load at which the system fails
  • Configuration – legal hardware and software configurations combinations
  • Compatibility – hardware and software interfaces, etc.
  • Security
  • Performance – compare actual operational throughput with performance objectives
  • Installability
  • Reliability/Availability – typical workload
  • Recovery – what happens when something bad happens
  • Serviceability – Identify whether information produced when something goes wrong is sufficient to diagnose the issue
  • Human Factors – Identify those operations which would be inconvenient or difficult for users or operators

Test Development

• Establishing the success criteria is often the hardest part of designing a test
• Things working against you in deciding what to test
  1. Full test coverage is generally impossible
  2. There is no proven comprehensive method for selecting the highest yield test cases
• Test Case design is not simple
• It is essential to hold walkthroughs of Test Plans and inspections of Test Cases
• The more complex the module, the more testing it should receive
• General guidelines for selecting testing paths (in unit tests)
  • Pick defined (in the requirements) paths
  • Pick additional path variations; small over large, simple over large, logical vs. illogical
  • Other paths which have no obvious function meaning (to up your coverage metrics)
• When paths through a program have not been well defined, it is unlikely the program could have been designed properly
• The objective is to test reasonably completely all valid classes for normal operation and to exhaustively test behavior under unusual and illegal conditions
• A way to check the adequacy of a test is to determine the degree to which all the requisite configurations have been covered
• Some Test Case guidelines
  • Do it wrong
  • Use wrong combinations
  • Don’t do enough
  • Do nothings
  • Do too much
  • Don’t forget the normal cases
  • Don’t go overboard
  • Don’t ignore program structure
  • There is always more than one way to test something
• Common Error Types
  • Missing control flow paths
  • Inappropriate path selection
  • Inappropriate or missing action

Test Execution

• Dogged curiosity about strange and unexpected events often produces the most important results
• The amount of detail to include in a test report depends on the audience
• Severity classifications must consider the operational circumstances of the resulting system
• Wise to limit number of severity categories to 4 or 5
• As the number of detected errors in a piece of software increases, the probability of the existence of more undetected errors also increases – Glenford J. Myers
• You are in trouble when
  • The occurrence of problems does not decline monotonically with testing
  • When new test environments or test cases are tried, a new crop of problems shows up
  • Errors generally have their roots in a design problem
  • The code has undergone several changes by several programmers
• Tune regression tests to place particular emphasis on those system areas that have been changed most recently