The Art and Science of Test Estimation

By: Matt Angerer

Estimation techniques play a critical role in not only the proper execution of software testing, but the software development lifecycle as a whole. Poor estimation leads to budget overruns, low quality, and missed deadlines. In this article, we’re going to explore what’s involved in the test estimation process and give some tips on providing a more accurate estimate. Hindsight is certainly 20/20, but with the application of both art and science in test estimation, you can dazzle upper management with incredible and informative foresight. It’s never an exact science, but every piece to the puzzle helps you drive closer and closer to an accurate testing estimate for your project.

 Test Estimation: An Overview

Before delving into the specifics regarding test estimation, it is important to address exactly what it is. Test estimation refers to developing a projected duration and testing cost, which is then agreed upon by both the requesting company and the testers. For the most successful outcome, test estimation must be accurate and realistic. Sticking as closely to the test estimation as possible is vital to ensuring a solid working relationship with your client. Not only does inaccurate test estimation erode your credibility, it can take a project from “Green” to “Yellow” in a matter of weeks. Make no mistake, jobs have been lost due to inaccurate estimates on high-profile projects. Perhaps the Test Manager didn’t lose his or her job, but the Project Sponsors certainly had some difficult questions to answer when a Go-Live date came and went.

In order to develop the most accurate test estimates possible, your testing team must be able to collect and evaluate a wealth of information. Despite these data sets, however, a significant portion of estimation relies solely on experience and expertise. It’s often said that America would be the most difficult country to invade in the entire world. The reason isn’t because we have the largest stock pile of nuclear missiles, tough security, the National Guard, or Seal Team 6. Of course, all of those factors certainly deter other nations from invading the United States, but the reason quoted by military experts is that we have the largest number of combat veterans. These are hardened soldiers with experience who can apply the “art of war” to any situation they encounter. That experience, coupled with the “science of war,” is a synergistic blend that provides a competitive advantage for our nation.

The point of outlining art vs. science in the context of U.S. combat veterans is not to dramatize the role of Software Test Managers. The underlying principle is to make sure your software testing managers are experienced, and to give them the necessary scientific tools and data sets to provide accurate project estimates.

You likely have decades of experience estimating testing cycles for projects of various size, scope, and technology stack. When a Project Manager asks if 1 week will be enough time to regression test a major global template, you laugh silently and politely respond: “Sure, if you can hire me 50 additional manual testers in the next 2 weeks!”  We know that test estimates not only require instinct, but also rely heavily on historical data. This historical data can only be captured by advanced test management tools like HP ALM, Quality Center, and HP Sprinter.

Developing an Accurate Estimation for Testing

One of the first steps of test estimation is determining exactly how long the project is going to take. The firm requesting software testing will need this information to help determine cost. In deciding how long testing will last, the following questions should be considered:

• What tools do you have at your disposal to achieve 100% test coverage?
• Are requirements clearly documented, and is traceability enforced?
• If you weren’t involved in the requirements gathering phase, are you given enough time to analyze the requirements (both functional, technical, and testing) and to interview subject matter experts on the team to understand problem areas?
• If we were to test everything (ideal situation), how long would it take us to get through a testing cycle given resource constraints?
• What value does upper management place on quality? In other words, is this a highly regulated environment where the cost of quality will be higher? Or are you estimating the testing phase for a consumer-facing product that is more forgiving?

Obviously, this short list is not all-inclusive. We advise that you “deep dive” into several different factors as you justify an estimate to the client. For instance, you can always pull data from the “Test Runs” module of HP Quality Center when you’re estimating for UAT or Regression testing. If you look into the “Test Runs”, there is a ‘Duration’ field that captures how long the manual test ran historically. We recommend finding all the ‘runs’ of the same test and computing an average ‘duration’ to determine how long it would take to manually execute the test case. This is an example of applying “science” to the “art” of test estimation.

Testers and the companies requesting their services often take a different approach, because testers will want to test as much as possible. The enterprises requesting testing will only want answers as quickly as possible and at the lowest price. In order to deliver the greatest value, you will need to inquire with the client on what testing is essential for that particular project.

For example, you might ask the client if they want specialist information, or whether they wish to employ a specific methodology in the testing process. It is also important to determine if the client wants your team to test beyond the requirements of the project. Risk-based quality management principles may play an integral role in their engineering process as outlined in this ResultsPositive article. And, of course, you should see if  the client wants your team to test throughout the project’s lifecycle, or only focus on a specific testing layer (e.g., interface/integration testing).

In most instances, you’ll find that test estimation involves at least some negotiation to satisfy all parties and provide the necessary testing depth at the timeline and budget your client needs. At ResultsPositive, we help clients like you negotiate between the people, process, and technology required to arrive at an accurate estimate for your testing project. Whether that be unit, system, integration, user acceptance, or regression testing – we examine the art and science of the process. Our knowledge across test management tools gives us the unique ability to quickly help extract information, apply expert judgement, and ultimately make accurate estimates.

Subscribe for the latest RP Blog Updates: