Automating Tests: A Suggested Classification

My last Post had described about the our teams experiences in implementing an automated build verification tests. There's been lot of talk and blog post over pros and cons of automated execution of test scripts during past couple of weeks. This post Post in James Bach's blog and Steve Rowe's this post are indicative of where the discussions are heading.


I've found automated execution of test as a significant productivity booster for testers. It relieves testers from the drudgery of re-running tests that were already executed and found to pass.


The current project I'm working on is first version of a business intelligence suite. Being the first version, there's quite of lot of GUI redesign and this is a challenge for development of automated test. However we've been successful in getting the initial level of test automated and successfully executed on daily images vericiations.


We've been fortunate in our project to learn from the mistakes made from other project groups as far as automation approach is concerned.

We've taken the approach for classification of tests to be automated under the following levels. Level-0 or build verification tests, Level-1 or happy path tests, Level-2 or error conditions, Level-3 or detailed user scenarios Level-4 advanced automation strategies.

Level-0 tests are used to check the testability of the product. This is preferably executed on your daily build before this is released for testing. The level-0 tests are often the first to get automated. These tests are also known as smoke tests.

Level-1 tests are used to test the positive paths of the application, mostly focusing your test goals on testing the application features from the user interface. For example, in an banking application, test for correctness of balance', 'check for transfer from one account to another' 'check for clearing of check'. These tests check for your correctness of your GUI and check for the integration of the GUI with the overall system.

Level-2 These will be automated tests for checking the error conditions or conditions that do not occur on normal working of the application. In case of the banking application, this may be be something like 'test for check bouncing', 'test for incorrect signature matches' etc.

Level-3 test are the detailed user cases. For example, create a scenario where a customer opens an accounts and does couple of deposits and withdrawal and closes the account. A suite of these tests can also be used as user acceptance tests.

Level-4 These are tests that typically cannot be carried out without the aid of an tool. High Volume Test Automation is a good example of this type of tests. For a banking application, simulating a typical year end closing procedure or executing hundreds of concurrent accounting transactions.

These classification of test needs to be looked at withing the context of the project in which you are testing. The levels are not really a classification of the order in which the automation project should be implemented. You may very well have tests automated from level-0 and level-2 , skip tests in level 1, level3 and automate tests in level-4