Using Bottom-Up Functional Testing to Improve Test Efficiency
Allen Zhang, Accusoft Software Engineer II
When talking about test efficiency, we mainly have two topics to address. Our goal when implementing our test coverage is to test as many features as we can in a fixed amount of time. Additionally, we want to write our tests to minimize maintenance by writing them in a modular and easy-to-maintain style.
In the case of Behavior-Driven Development (BDD), which is used within Cucumber, having hundreds of key steps or actions often requires up to tens of thousands of lines of code. If insufficient planning is done ahead of time, the implementation code may end up too tightly coupled with the individual step structure, which can often result in a maintenance nightmare. In the occasion that a test fails, we may need to spend several hours figuring out if the fault lies in the test framework itself or the developed product.
Why bottom-up testing?
There are two major ways of carrying out an integration test; the bottom-up method and the top-down method. Both of them are widely used, but I've found the bottom-up approach is much more suitable for an Agile environment.
Figure 1 below is an example of a test-pyramid diagram. It is a concept developed by Mike Cohn, described in his book, Succeeding with Agile. The fundamental point of the concept is that there should be a greater number of low-level tests than high-level tests.
Low-level tests should be completed first, providing the following benefits:
1. Quicker development & test runtime: In this approach development and testing can be done together. Once the feature/function is done, the corresponding Unit Test or Service Test can be triggered right away.
2. Bug cost: Bugs found late in a product lifecycle have a very high development cost to fix. When QA is able to find bugs at a very early stage, the cost of fixing a bug is greatly reduced.
3. Ease of test coverage: By starting at the lowest level for testing, we avoid the need to pepper our tests with mockups and stubs. This allows us to quickly get down to the metal of product functionality testing.
Wrap up low-level actions to high-level ones
Accordingly, we should design our key test steps (actions) in a bottom-up approach. Here is an example from our Barcode Xpress product testing environment. Barcode Xpress is a powerful product that accurately reads common industry 1D and 2D barcodes, detecting them anywhere on the page, in any orientation. For one aspect of functionality (reading barcodes), three arguments are required for input. Thusly, we need three actions to set up the test and one action to recognize barcodes on the image.
1. Given that I open the file "test image.bmp"
2. Given that I want to detect barcodes in the rectangular area (x, y, width, height)
3. Given that I want to detect barcodes of the type "Code 128"
4. When I recognize barcodes in this image
The above four actions are all low-level actions; every action does only one thing. What sets action #4 apart from the first three is that action #4 has a dependency upon the earlier steps. But in some cases, we want to use a generalized procedure instead of setting the specific parameters to yield a result. To accomplish this, we create a high-level action to represent the above four actions by wrapping the low-level actions into a higher-level one.
For example, we can wrap the four actions into the below actions:
5. When I recognize barcodes in the image "test image.bmp"
6. When I recognize barcodes of the type "Code 128" in the image "test image.bmp"
Breaking down this diagram, we can see that action #5 (recognizing barcodes from an image) is composed of actions #1 (opening the image) and #4 (recognizing from the image). Likewise, we see that action #6 is composed of actions #1 (opening the image), #3 (setting the specific barcode type), and #4 (recognizing from the image). Action #2 may be used elsewhere in the test setup.
By composing our steps, we're able to abstract out more generalized functionality in a modular way that allows us to reuse lower-level steps for more easily maintainable code. If there is any change in a specific feature, we only need to change that specific action; the high-level actions will have their functionality affected accordingly, without us having to rewrite their tests.
As discussed above, conducting functional tests in a bottom-up approach helps us speed up test development, improve early stage test coverage and easily maintain the test framework. Also, by being very responsive to changes it is well-suited for Agile teams.
Allen Zhang is a Software Engineer II with Accusoft. He has over 13 years of testing experience. Allen graduated with a Bachelor of Science in Computer Science from The Nanjing University of Technology in China.