Metrics to Reduce Risk in Product Ship Decisions – Part V – Case Study: DataFinder – Define Beta Test Entry Criteria
Metrics to Reduce Risk in Product Ship Decisions – Part V – Case Study: DataFinder – Define Beta Test Entry Criteria (#5 in the series Metrics to Reduce Risk in Product Ship Decisions)
By Johanna Rothman
In addition to the system test entry criteria, the DataFinder product development team chose these additional criteria for beta test:
- The code must be branched.
- Minimum 98% regression test pass rate.
- No bugs that cause core dumps.
- Minimum 90% function coverage in regression tests. (Regression tests must enter at least 90% of the modules in the system.)
The product performance criteria were met by entry to system test, and the beta criteria ensured that the customers would not receive software with more defects than they could handle.
A note for the code coverage purists: DataFinder management and technical staff were new to the notion of code coverage. They assumed that their admittedly extensive regression tests would provide them minimum 90% function coverage, and that 90% was somehow “good enough”.
Beta test entry criteria should also be defined as early in the project as possible. Again, management, development, and quality assurance personnel need to agree on the criteria. In addition to deciding on the criteria, they should also estimate the time from the start of system test to the time beta entry criteria are met. That will give them an historical perspective of this part of the project for the next project.
Original article can be found at: http://www.jrothman.com/Papers/QW96.html
Johanna Rothman consults, speaks, and writes on managing high-technology product development. Johanna is the author of Manage It!’Your Guide to Modern Pragmatic Project Management’. She is the coauthor of the pragmatic Behind Closed Doors, Secrets of Great Management, and author of the highly acclaimed Hiring the Best Knowledge Workers, Techies & Nerds: The Secrets and Science of Hiring Technical People. And, Johanna is a host and session leader at the Amplifying Your Effectiveness (AYE) conference (http://www.ayeconference.com). You can see Johanna’s other writings at http://www.jrothman.com.
