QA Team Internal Metrics for Improvement

Basic driving force for below metric is to help QA team grow to identify serious valid defects in earliest possible time during the project period


Distribution of severity reported in QA phase test

Goal State: You want your team to find more SEV1 and SEV2 issues than SEV3 and SEV4.

  • If the team is spending more time finding lower severity issues, consider following improvement actions.
    • Ramp up/ Refresh session on business rules & business critical path.
    • Review the tests executed to see if it contains SEV1 /SEV2 scenarios.
    • Review Sev 3 and Sev 4 to understand what is driving the team to focus on lower priority issues.
    • Implement Peer Review
    • Implement QA sign off Demo to BA and PM


Timing of when high severity issues are reported

Goal State:  You want your team to submit high severity issues in earlier part of test execution

·      If the team is submitting serious issues toward the prod live date, consider following improvement actions.

o     Review test execution strategy – are we executing right test in right order?

§         Review the document “Tips for Prioritizing Test Execution”

§         Align the test to high risk items.

o     Review staff – was staffing was there to support early part of testing? 

§         Consider loading up more test execution in earlier parts.

§         Are we controlling PTO/ Time off?  If you have consistent PTO due to epidemic illness spread, add more headcount to come up with true operational head account (ex. If 20% staff is always taking PTO for illness, then you need to up 20% of head count to maintain operationally required head count).  You need to keep track Staff PTO to maintain this.

o     Is unplanned / uncontrolled code deployment is causing this?  Propose code freeze during the project test period.

§         It also suggests a room for improvement in unit test/ code integration test.  Collect metric data and present to Dev Lead.  Organize action plan improvement to prevent codes to be thrown over the wall without good testing.


Test Effort (either # of tests or HRS spent on test) to uncover high risk defects

Goal State:  Over time, you should spend less time to find defects.

·         Check for duplicate tests.  No one should ever test more than necessary.

·         Identify unnecessary tests

·         Reduce rare scenarios when it is not consistently identify issues. 

·         Any test that has never caught issues, should be evaluated for reducing the scope.

·         Increase more alignment with Impact Analysis and test created – review to see if test needs more revision or impact analysis needs more robust.

·         Are we recycling previously created test / data – avoid data to be created from scratch to save more time.


Requirement Coverage

Goal State: We are striving to over 100% of functional requirement provided by BA

·         Understand requirements that were not covered.

o        Environment limitation needs to be documented and addressed

o        If you could not test due to limitation, ask developers how they have done the unit testing.


Defect Discovery Rates

Goal States : Indicates system shows stability if you start to spend more time finding a new issue.

X axis = Time.  Y axis = Submitted Issue Counts

If your team does not see this pattern, consider following improvement actions.

·         Code freeze.  Repeating code push will set the status back to the place where star is.

·         Increase staffing size – obviously your current staffing size is not discovering the defects early enough.

o        If you have automated test that can find major defects, consider running it as soon as code deployment is made.

·         Check effectiveness of the test.  Are you focusing and prioritizing right kind of test?

·         Analyze test method

o        If regression test is more producing issues than function test, consider executing regression test first before running function test.

o        If smoke test should catch issues in earlier stage, review smoke test script and make it more robust to be run a the first time.

·         Increase unit test quality.

·         Move UAT early to begin earlier timing to find more issues earlier points.

o        Understand gap between UAT issues and QA phase issues.  If lack of knowledge in QA team is causing that, consider additional ramp up/ training session with help from BA.

How would you know your test team is doing good quality test?

In addition to above metrics, you can tell teams effectiveness by observing the following

·         Quality of question getting raised in demo, Sharepoint discussion section or defects

·         Quality questions should be more aligned to business user perspectives.