There is no standard set of metrics that can be tracked to ensure project success. Agile teams should track a small set of agile software quality metrics initially and then adjust those metrics over time as they learn which are informative and aid with decision-making. It is recommended that agile teams initially track three metrics: test coverage, change failure rate, and deployment frequency.
Rationale for Choosing Each Metric
The following metrics build on top of each other to provide information about the completeness, adequacy, and efficiency of a quality assurance effort, respectively.
Metric 1 – Test Coverage
Test coverage measures the completeness of testing based on the percentage of the system exercised by tests. Coverage of both specific coding constructs and higher-level entities such as user story acceptance criteria should be considered. This measure should be evaluated as part of the DevSecOps pipeline and used to reject builds that do not meet a minimum coverage standard. Project leadership can provide the agile team with direction to improve test coverage for critical parts of the system as a way of reducing the risk of introducing a defect into production.
Metric 2 – Change Failure Rate
Change failure rate is the percentage of changes made to production that fail and can be used to evaluate the quality of the tests. While it is important to achieve sufficient coverage of the system, it is possible to write low-value or even meaningless tests that are ineffective at revealing defects. Even if a “complete” set of tests is executed against the system, defects may be missed, which may cause more failures to occur in production (i.e., the change failure rate will increase). If this happens, the tests should be examined and most likely updated to identify those defects earlier in the pipeline.
Metric 3 – Deployment Frequency
Deployment frequency evaluates how often new or updated features are delivered to end-users. Project leadership can monitor deployment frequency over time to see if it is trending upwards, staying the same, or trending downwards. If deployments start occurring less frequently then the agile team should evaluate the runtime of their automated tests and identify improvements to ensure that a lengthy test cycle is not creating a bottleneck for the rapid delivery of new changes.
How to Choose Additional Metrics
Additional metrics will be needed if project leadership does not have the information that they need to make management decisions or to report the status of the quality assurance effort to corporate leadership. However, it is important to recognize that adding new metrics will incur a cost — both the upfront cost of determining how to collect the metric and also a long-term maintenance cost. Plus, when the team has invested effort into collecting and gets used to seeing the metric, they may find it challenging to stop tracking it, even if at some point it no longer provides any value (i.e., they may fall prey to the sunk cost fallacy). For this reason, the metrics that are being tracked should be revisited on a quarterly basis.
