Functional testing in developer's toolbox

Introduction

Everybody knows that one should write tests in order to guarantee good internal and external software quality. In most cases developers focus on structural tests leaving high level functional testing to separate testing expert. However I believe that in many cases it can be useful to include functional tests in the process of writing production code same way as structural tests.

Of course best results can be achieved by combining both approaches but I think that functional tests have many attributes that are usually assigned to only structural tests. In addition they have some extra benefits worth considering when deciding the amount of effort to be invested into on or the other testing style.

Following is a comparison of different aspects of testing in the context of developer's testing. Functional testing benefits/drawbacks are much based on lessons learned while writing functional tests to verify each and every requirement for two game servers in Eget games team.

1. Driving design

Functional tests:
at least for me the main problem when trying TDD has always been that with structural tests it is very hard to get started. I mean how do you test something if you do not have any idea how it should be implemented? It is much easier to think about functionality instead of structure in the beginning hence write high level functional tests first and then lower level structural tests.

Structural tests:
are more useful for helping/guiding you with most design issues.

2. Regression testing

Functional tests:
since functional tests focus on external functionality and usually we are interested mostly in faults that can be externally detected I would say that functional tests are stronger in this area.

Structural tests:
pinpointing the fault that caused failure is easier

3. Tests as documentation

Functional tests:
access high level functionality so they may make it easier to understand different use cases of the system. Downside of functional tests is that they tend to be more complex and hence harder to read.
However I think there are two things that can help to keep functional tests readable:

• avoid exhaustive testing if possible
• implement framework which enables to bring test code closer to domain language.

Structural tests:
help to understand some particular class but not very useful for getting acquainted with the system as a whole.

4. Effectiveness

By this I mean how many potential faults test is able to discover. Code coverage isn't of course ideal measurement since having X% coverage doesn't mean that X% of potential faults are checked. Secondly it is easy to achieve quite high coverage with very few functional tests but this is just because they execute most main paths. Unfortunately these paths are likely to contain very few faults. Based on previous reasons I think that it starts to make sense to compare effectiveness of testing methods when at least ~80% coverage is achieved. I believe that having 80% coverage with functional tests gives much better guarantees for external quality as structural tests.

One lesson we have learned in Eget games is that functional tests are good for verifying that user stories are implemented correctly. However since they ignore internal code structure many low level branches which are not needed for given stories are not tested leaving faults to the system that may be discovered when adding new functionality.

5. Time

In Eget games we have spent about 40% of time on writing functional tests. Manual testing takes us only about 5% of development time. Of course one important factor is that game servers don't include any UI which makes writing and maintaining functional tests much easier. However I believe that with proper framework functional tests may not be that much more expensive to automate than structural tests at least if system under test is relatively simple.

Summary

In summary I think that integrating writing functional tests with implementing production code is something worth considering. It doesn't necessarily have to be so that developers care only about structural tests leaving functional testing solely to test engineers.