An important part of software quality is the process of testing and validating the software. The purpose of this article is to introduce concepts about, and provide general best practices in the field of, test management. Test management is the practice of organizing and controlling the process and artifacts required for the testing effort.

Introduction

Purpose

Few people can argue against the need for improved quality in software development. Users of technology that utilizes software have come to expect various faults and flaws and, especially in the world of personal computers, we consider frequent problems to be completely normal and expected. However, as software development matures, we are beginning to better understand how to achieve a necessary improvement in quality. The purpose of this article is to introduce concepts about, and provide general best practices in the field of, test management.

What is test management?

An important part of software quality is the process of testing and validating the software. Test management is the practice of organizing and controlling the process and artifacts required for the testing effort. Traditional tools used for test management include:

• Pen and paper
• Word processors
• Spreadsheets

Larger testing efforts may use home-grown software test management solutions, usually built on spreadsheets or databases, or commercial test management applications such as IBM® Rational.

             The general goal of test management is to allow teams to plan, develop, execute, and assess all testing activities within the overall software development effort. This includes coordinating efforts of all those involved in the testing effort, tracking dependencies and relationships among test assets and, most importantly, defining, measuring, and tracking quality goals.

Aspects of test management

    Test management can be broken into different phases: organization, planning, authoring, execution, and reporting. These are described in more detail below.

Test artifact and resource organization is a clearly necessary part of test management. This requires organizing and maintaining an inventory of items to test, along with the various things used to perform the testing. This addresses how teams track dependencies and relationships among test assets. The most common types of test assets that need to be managed are:

• Test scripts
• Test data
• Test software
• Test hardware

Test planning is the overall set of tasks that address the questions of why, what, where, and when to test. The reason why a given test is created is called a test motivator (for example, a specific requirement must be validated). What should be tested is broken down into many test cases for a project. Where to test is answered by determining and documenting the needed software and hardware configurations. When to test is resolved by tracking iterations (or cycles, or time period) to the testing.

Test authoring is a process of capturing the specific steps required to complete a given test. This addresses the question of how something will be tested. This is where somewhat abstract test cases are developed into more detailed test steps, which in turn will become test scripts (either manual or automated).

Test execution entails running the tests by assembling sequences of test scripts into a suite of tests. This is a continuation of answering the question of how something will be tested (more specifically, how the testing will be conducted).

Test reporting is how the various results of the testing effort are analyzed and communicated. This is used to determine the current status of project testing, as well as the overall level of quality of the application or system.

The testing effort will produce a great deal of information. From this information, metrics can be extracted that define, measure, and track quality goals for the project. These quality metrics then need to be passed to whatever communication mechanism is used for the rest of the project metrics.

A very common type of data produced by testing, one which is often a source for quality metrics, is defects. Defects are not static, but change over time. In addition, multiple defects are often related to one another. Effective defect tracking is crucial to both testing and development teams.

Other factors in test management

In addition to the software and hardware test artifacts and resources, the testing team has to be managed. Test management must coordinate the efforts of all project team members involved in the testing effort. This requires controlling user security and permissions for testing members and artifacts. For projects that span more than one site or team (which is rapidly becoming the norm), this also includes organizing site and team coordination.

The particular testing process for a project will have an obvious bearing on test management. For an iterative project, test management will have to provide the foundation and guide the effort to plan, execute, and evaluate testing iteratively. Following this, the testing strategy will also have to follow the test management framework.

Related software development disciplines

While all disciplines in software development have an association to the testing discipline, there are several that have particularly important connections to testing:

• Requirements management
• Change management
• Configuration management

Requirements management is a precursor to the bulk of the testing effort, providing a significant amount of test motivation and validation needs. A project's particular requirements management process can have a huge impact on the test management process. One analogy for this relationship is that of a relay race, where the first runner represents requirements management and the next runner, receiving a baton hand off, represents test management.

Change management affects all parts of software development, but the tracked changes most relevant to the testing effort are defects. Defects are frequently the main communication channel between testing and development. Counts and metrics derived from defects are also often used as measures of quality.

Configuration management is important to test management because tracking which builds to test at what time is crucial to testing. Configuration management controls the builds as well as the environments that are tracked by test management for test execution.

Test management recommendations

The following are general recommendations that can improve software test management.

Start test management activities early

While this may seem like the most obvious suggestion, few software projects truly apply this concept. It is easy and not uncommon to begin identification of test resources at an early stage. However, many test analysis activities (such as the identification of critical, high-priority test cases) can and should begin as soon as possible. As soon as use cases are developed enough to have a flow of events, then test procedures can be derived. If a project is not utilizing use case requirements, then tests can still be derived from the validation of initial requirements. Developing tests as soon as possible helps alleviate the inevitably forthcoming time constraints.

Test iteratively

Software testing should be an iterative process, one that produces valuable testing artifacts and results early on in the overall project lifecycle. This follows the first recommendation of starting the testing process early: an iterative testing process forces early attention to test management. Test management guides this by organizing the various artifacts and resources to iterations. This risk-based approach helps ensure that changes, delays, and other unforeseen obstacles that may come up in the project timeline can be dealt with in the most effective manner.

Reuse test artifacts

Reusing test artifacts within a project, or across projects, can greatly improve the efficiency of a testing team. This can greatly relieve the pressure of limited time and limited resources. Reusable artifacts include not only test automation objects, but also test procedures and other planning information. In order to reuse artifacts efficiently, test management must do a good job of organizing and delineating the various testing-related information used for a given project. Reuse always requires some forethought when creating artifacts, and this principle can be applied in test management generally.

Utilize requirements-based testing

Testing can be broken down into two general approaches:

• Validating that something does what it is supposed to do
• Trying to find out what can cause something to break

While the latter exploratory testing is important to discover hard-to-predict scenarios and situations that lead to errors, validating the base requirements is perhaps the most critical assessment a testing team performs.

Requirements-based testing is the primary way of validating an application or system, and it applies to both traditional and use case requirements. Requirements-based testing tends to be less subjective than exploratory testing, and it can provide other benefits as well. Other parts of the software development team may question or even condemn results from exploratory testing, but they cannot dispute carefully developed tests that directly validate requirements. Another advantage is that it can be easier to calculate the testing effort required (as opposed to exploratory testing, which is often only bounded by available time).

It can also provide various statistics that may be useful quality metrics, such as test coverage. Deriving test cases from requirements, and more importantly tracking those relationships as things change, can be a complex task that should be handled through tooling.

 The constraint to this process is that it depends on good system requirements and a sound requirements management plan to be highly effective. Exploratory testing, on the other hand, can be more ad hoc. It relies less on other parts of the software development team, and this can sometimes lead to testing efforts focusing less on the important tasks of validating requirements. While a superior testing effort should include a mix of different approaches, requirements-based testing should not be ignored.