This article provides a basic definition and overview of Software Testing Metrics. It will also advise you on how testing metrics can be used to boost the effectiveness of software development project planning, test planning and test execution activities.
What are Testing Metrics?
A definition of software testing metrics - measurements of frequency, quantity or the duration of software testing based events occurring within an observed window of time.
Types of Testing Metrics
The main types of software testing metrics are
Test creation – Metrics relating to test case or test script productivity.
An example of a test definition metric would be a count of how many test cases have been produced to cover a set of business requirements.
Test execution – Metrics covering the progress and status of actual tests performed.
An example of a test execution metric would be the total number of test cases failed during test phase.
Defects – Metrics based upon any defects recorded during the software development life-cycle.
An example of a defects metric would be the count of critical defects raised after user acceptance testing.
Automation – Metrics providing insight into automation effectiveness and coverage.
An example of an automation metric would be, the number of automated tests compared with the total number of test cases
Performance – Metrics collected through the execution of performance, load or benchmarking tests.
An example of a performance metric is, the percentage change between the average transaction rate of a process as measured in software release 1 compared with the same measurement taken in software release 2.
Why Use Testing Metrics?
Software testing metrics, when used intelligently can provide vital information for the improvement of many aspects of software development projects. The main aspects are : Testing Processes, Software Development Processes, Project Planning and overall Software Solution Quality. Getting these elements right is key to the success of any organisation involved with software development.
Further detail on the usefulness to a project of information acquired through the analysis of software testing metrics follows below.
Testing Processes – Metrics collected during test preparation and test execution provide valuable insight into the effectiveness, cost, quality and timing of various testing activities. Testing metrics may indicate that a testing process does not yield outputs which meet the expected levels of quality or timeliness. When this is the case, the testing process may be modified to make improvements to future activities.
Software Development Processes – Metrics collected during test execution may be used by software development teams to adjust their processes to improve the quality of software deliverables.
For example, if testing metrics highlight a high concentration of defects during a particular development phase, analysis can be carried out to compare the conditions and activities carried out during that phase with a phase yielding less defects. A result of this analysis may reveal that less peer review effort was spent during the under-achieving development phase, perhaps due to an underestimation of complexity. The development team may then decide to ensure that the peer review activity should be given a higher priority to avoid such a shortfall in future projects.
Project Planning - Metrics collected during test preparation and test execution can be extremely useful to the project manager in the planning and estimation of projects. Using historical information such as test case creation time test execution time, defect count and defect turnaround time allow the project manager to make a more accurate judgement on resourcing levels and scheduling throughout the course of a similar, current project.
Software Solution Quality – Defect metrics captured during and after test execution offer an indication of the quality or state of the software itself. Trends such as concentrations of defects within certain features, functions or development stages. Another indicator made visible through metrics may be how many defects are reported by end users in the Live system. Metrics therefore offer clues which can be used to investigate why defects appear where they do and how to prevent similar bugs appearing in the future.
Use Software Testing Metrics Carefully
Testing metrics as described above can be really useful. On the flip-side, metrics do have some potential pitfalls and therefore must be used with caution. Here are some pointers on using metrics carefully.
Do not use metrics unless there is value to be had and the solution stakeholders agree and understand what that value means to them.
Select who you publish metrics to. Do not broadcast metrics to the entire planet. Restrict the information to those who wish to make use of them.
Do not over beautify the numbers. It’s all good making pretty looking charts to present the metrics aesthetically. Ask yourself however whether there is any added value in doing so. It’s more likely that you could be spending less time on looks and more time on analysis.
Do not base decision making on metrics alone. Use metrics as part of a well researched pool of information from various sources before making decisions.
Avoid using metrics as evidence of poor staff performance. If metrics show that team member A reports fewer defects than team member B there is a risk that Team member A will start logging additional little minor bugs just to play the numbers game. In the long run this would cause wastage due to the extra admin involved in reviewing unnecessary details.
Software Testing Metrics should be an important part of your testing strategy as much valuable information is locked away in measures that are often untapped. It is important however to ensure that if metrics are being collected, that they are actually being utilised to improve upon the way things were being done beforehand.
If metrics are being recorded and are not being used then perhaps the wrong metrics are being collected. If this is the case then a review is necessary to determine which metrics, if any are of interest and of use to whom and to what end.