by Marina Gil-Santamaria

Published Juky 2008

OTN QAZone spoke with Rex Black about Automation, ISTQB certification, Quality Risk Analysis, and more. Rex is the president of the International Software Testing Qualifications Board and the American Software Testing Qualifications Board, published author, and a frequent presenter at testing-centric conferences and workshops around the world. His popular book "Managing the Testing Process", has sold over 25,000 copies around the world, including Japanese, Chinese, and Indian releases

OTN: Rex, please tell us a little about you. What drove you to testing in the first place? Did you start there your professional career? If not, where did you transition from?

Black: I started as a FORTRAN and C programmer and system administrator for CP/M and then Unix systems in 1983. I enjoyed programming --I still do--, but I took an opportunity to create automated compatibility tests for Unix systems in 1987.I found that I enjoyed testing, especially test automation and test management, and that I was good at it. I have been mostly a test professional since then. In 1994, I founded RBCS, a consulting company focusing entirely on testing and quality assurance. Today, RBCS has grown to be an international test consultancy, serving clients on six continents and in a variety of industries. We have partners around the world, and RBCS is a co-owner of two other test-focused consultancies, RBCS NZ in Australasia and PureTesting in India. As you can see, I consider providing testing services to be a good business and one that I intend to focus my energies on for the remainder of my career.

 OTN: What is the future of the testing industry, do you see any clear trends emerging, and how should testing/QA professionals prepare for them?

Black: Since emerging as a profession within software engineering in the 1980s, testing has involved three major skill areas:domain expertise, technical expertise, and testing expertise. While the specific skills in these three areas of expertise continue to evolve, the three fundamental areas remain the same. The important facts to remember are that testing remains a separate profession, not something to be done by programmers or business experts in their spare time, and that, within any given company, the specific mix of skills within those three areas is likely to differ.

OTN: As the president of the International Software Testing Qualifications Board (ISTQB), in your opinion, what is the impact that certification can have in the professional growth of a testing/QA professional?

Black: The impact of the ISTQB program has been huge, and will be even greater in the future. We now have over 60,000 certifications issued under this program. These certifications span the world, covering every industrialized continent and most major software (using or developing) countries. Up to now, much of the testing practice has remained stuck in the 1970s, going in circles and not building on the foundations laid by Glenford Myers and Boris Beizer at that time. By ensuring that the best practices of testing are widely known by practitioners, I expect to see enormous progress in our profession over the next decade, progress which eluded us prior to the adoption of this uniform, international consensus on how to approach testing.

While training is not required to take ISTQB exams, training providers will build on this program to increase the value. For example, the RBCS certification courses not only fully cover the appropriate ISTQB syllabi, but also go beyond them to offer even more hands-on experience with testing in the classroom.

OTN: At what point of the testing career do you advise to go for an ISTQB certification, do you recommend it also for folks without any testing experience that are looking to start a career in this space?

Black: That is one of the great aspects of the ISTQB program.The ISTQB provides a career ladder, or, more accurately, a career tree.For those entering testing, it offers the Foundation certification, to set themselves on the right track with the basic concepts. After five years or so of testing experience, it offers the Advanced certification, to ensure that abilities and knowledge are sufficient for the next steps of their career. At the seven to ten years of experience level, it will offer the Expert certification. These Advanced and Expert certifications branch off from the Foundation, based on career path up the testing career tree. There will be technical, functional, and management focused branches in the career tree defined by the ISTQB. It is really quite unique in that fashion, among all the testing certifications.

OTN: Rex, you mentioned that you enjoyed test automation & test management. What value do you see in automation in general, and what type of testing initiatives do you think that are best candidates for automation?

Black: I have been doing automated testing since before I was a professional tester, when I wrote automated unit tests for my code as a programmer. (Yes, that concept significantly pre-dates the relatively-recently advanced Agile methodologies). My first job as a professional tester was creating automated tests for Unix systems, using shell scripts and C programs. RBCS has used carefully-chosen collections of commercial, freeware, and custom developed test tools to create sophisticated end-to-end automated tests for very complex systems-of-systems. Both RBCS and PureTesting are heavily involved in providing test automation services to our clients. So, based on these successes, I see automated testing as quite valuable. Indeed, for static analysis, dynamic analysis, unit testing, most component integration testing, and in system testing where the risks related to regression, performance, reliability, and load are high, I see good automation techniques as essential. Tools, whether commercial, freeware, or custom-developed, are typically a vital aid to such automation.

Unfortunately, I often see clients making common and entirely avoidable mistakes with automation. People mistake a test automation tool for a test automation strategy. People undertake long-term test automation projects without clearly defined business cases and return-on-investment models. People try to automate tests where frequent human intervention is required to keep the tests running or real-time human judgment is required to interpret the results. They try to automate at interfaces, especially graphical user interfaces, which are changing in ways that undermine test maintainability or which suffer from severe testability problems. Compounding these errors, test managers try to use on-the-job training to teach their teams how to do test automation, ignoring the fact that successful test automation is one of the hardest programming problems there is and that neophyte test automators will learn the classic mistakes of test automation by making them on their test automation project. Having one test automation project fail makes it all the harder for a test team to obtain the funds and the permission to try again. Showing clients how to avoid these classic mistakes and achieve test automation success is one way that professional, test-focused consultancies, like RBCS, PureTesting, and some of our competitors, can play a vital role.

OTN: In your mind, what is the typical ratio of manual testing versus automated testing that folks should strive for?

Black: I don't know that I can put a typical number to this. However, I would say that an organization that is not doing any automated testing is probably not doing enough, particularly in the areas of unit testing and component integration testing. For system testing, as the regression, performance, reliability, and load risks go up, automation should play a larger role in the test effort.

That said, not all tests are automatable.There is a place for some manual testing in all test efforts. Harnessing the skills, knowledge, and creativity of professional tester do carry out some exploratory testing as a complement to more formalized manual and automated testing is a best practice that all test teams should include in their test efforts. Striving to hit an artificial target percentage of automation is likely to lead to some degree of ineffective and inefficient testing.

OTN: Rex, you have written very interesting articles around Quality Risk Analysis, how would you define this concept?

Black: Back in the 1980s, the only really articulated testing strategy was requirements-based testing. In requirements-based testing, you are dependent on the quality of the requirements specification, and you need to have time to cover all the requirements. This makes a purely requirements-based testing approach brittle in the face of poor requirements and tight time constraints, both of which seem to be constant challenges in testing, at least on the projects I have worked on.

So, in the mid-1990s, I started to move to a risk-based testing strategy. Being a pioneer of this strategy, I couldn�t find much written on the topic, but I was able to adapt the concept of failure mode and effect analysis to testing. Failure mode and effect analysis works as a quality risk analysis technique, but it tends to be overly formalized for most projects. So, over time, I have created less formalized but still quite effective techniques for quality risk analysis that have served RBCS and its clients quite well. These techniques are quicker and cheaper than more formalized techniques, but allow RBCS clients to achieve the objectives of risk-based testing. There are a number of articles on the Library page of that can help to explain these quality risk analysis techniques.

Ultimately, the objectives of risk-based testing--the way we do it at RBCS--are simple. First, as the risk in any area of testing goes up, we dedicate more test development and test execution effort to that area. Second, since we try to find the scary bugs as early as possible, we prioritize tests based on risk. Third, if we are confronted by time pressures during testing that force us to reduce the scope of testing, we eliminate tests in a way that drops more of the lower-risk tests than of the higher-risk tests. If risk-based testing is carried out properly, it is a strategy that is more robust and flexible that a purely requirements-based strategy. It is important to be clear about the sources of risk. Too many people, when they talk about risk-based testing, are talking about using the likelihood of finding bugs to focus the test effort. This turns testing into a nothing more than a bug hunt where the goal is to find as many bugs as possible. But testing is also about building confidence and trying to make sure, to the extent possible, that the key areas of the system work properly in the most important scenarios. So, risk-based testing should also consider the potential impact of bugs, both likely and unlikely bugs, when deciding the levels of risk in any given area.

OTN: Keeping this in mind, what is the best approach to tackle estimating testing time and QA resources when planning within a Waterfall-type of project? Do you have some quick guidelines in terms of comparing testing efforts with development efforts, or do you think that this is not feasible at all?

Black: RBCS consultants use risk analysis early on in the project to determine what we are going to test, how much, and in what order. That risk analysis then drives the creation of an estimate, using techniques like work-breakdown-structures, historical metrics, and the experience of the test team. If management will not accept that estimate, then we ask them to work with us to revise and customize the risk analysis so that we can still ensure coverage of the most important risks given the time and resources available. While we use historical metrics from similar past projects, we typically do not use tester-to-developer ratios based on effort or staffing size, unless those ratios have proven reliable for that organization in similar past projects.

OTN: And once you have your testing estimation in place, do you have some tips on best ways to sell them to management, now that globalization is here and corporations are looking to expedite time to market and reduce costs?

Black: Most organizations are not aware of the true return on the testing investment. It's quite typical for an organization to save three, five, ten, or even 20 times what they invest in testing, based only on the comparative cost of bugs found in testing versus those found in production or after release. So, one major element of selling an estimate is making sure management understands the benefits you are offering. Again, there are articles on analyzing the return on the testing investment on the Library page of

When time to market is critical, then early detection and removal of defects becomes all the more important. In these circumstances, test professionals can work with development and management colleagues to ensure that testing is well-integrated into the overall software development lifecycle. In RBCS� assessment work with clients, we often find that they are making sloppy upstream decisions that result in huge numbers of defects being delivered to formalized testing groups at the end of the process, then they are surprised by late-discovered, delaying defects found during this testing. To ensure that formalized testing levels like system test and acceptance test go smoothly, early quality assurance and testing activities like static analysis, code reviews, unit testing, and integration testing are critical.