Elevator Pitch: What Makes a Quality Test Suite?

This final component of my “elevator pitch” on what makes for a quality test suite will describe the techniques, technologies and processes required to deliver on the value propositions described in my previous blog. Based on our work here at Paragon, I could compile a very long list of implementation details then examine and prioritize them based on their impact. Instead I want to focus on a smaller subset of the tools and approaches that have been most effective in delivering the value we were seeking for our environment.

  • Behavior Driven Development (BDD) using Gherkin and “dog fooding” our own Web FASTest technologies

  • Automated system testing with Selenium and WatiN

  • Functional testing based on turnover and design documentation

  • Regression testing based on well defined regression test scripts

  • Smoke testing prior to release

  • Unit testing with nUnit and RhinoMocks

  • Test Driven Development with all tests built in the development cycle committed to the product test repository

I will start with the runner up. The second greatest contributor to delivering on the value proposition was the introduction of single assertion tests. We found that a test suite with single assertion tests, while generating many tests, is actually more efficient than fewer test cases that test multiple system requirements. Each test matches a single system requirement which makes it very easy to change our software and quickly identify tests that no longer match the system behavior. In some cases the failure identifies a flaw in logic and in others it identifies tests that have become obsolete. In either case the information from the test suite is very precise which leads to specific corrective action when needed.

The winner, and by far the single greatest contributor to the value we derived from our quality test suite approach, was the introduction of a Continuous Integration (CI) process. The purpose of a CI process is to create a very quick feedback loop for all work products related to a given software project.

Implementing the CI process forced us to think about the length of our feedback loop. We quickly learned that we took for granted the amount of time we spent manually preparing software and systems for testing. In short, the CI process made us broaden our perspective on testing by making us examine every portion of the process to remove any manual intervention.

Once we removed all manual effort required to setup systems for testing, we started to get traction with our automated test suites. We began to spend more time developing automated tests and less time manually installing software and performing manual test scripts.

Obviously, I can’t just add all THAT into my elevator pitch since I am not sure there is a building tall enough to accommodate all those words. So, I will try to distill it down some.

A quality test suite will simulate any endpoint in a transaction path through a test harness that automatically executes the suite, verifies the results and notifies if variances are found. The end result is an efficient testing process that:
  • Accurately describes the system or component requirements

  • Directs attention to identified defects with laser precision

  • Never wastes valuable resources by drawing attention to false negatives

  • Requires no human interaction to fully test the system or component

It is embedded in a larger Continuous Integration or Continuous Delivery system that ensures test runs begin with the system under test in a known state. Each test in the suite is designed to match a stated requirement of the system and it enforces that single requirement through a single assertion.


Do you agree? Let me know your thoughts.

This is the fourth in a series of blog posts focusing on developing and describing a quality test suite.


« Previous