The product we work on is an enterprise-scale solution typically deployed across multiple physical machines and supported on heterogeneous operating systems. The product should work on about 6 types of operating systems and three types of web application server and 3 types of databases. The complete platform support matrix will typically have 12 to 15 unique test-configurations that require to be tested. This is where the need for control over the regression testing comes into picture.
A complete manual approach for covering all these test configurations is not really a feasible plan. This is where the need of automated test executions comes into picture.
The preferred approach we take is to identify the “primary test configurations” which are typically 2 or 3 test configurations. Functional testers typically does exploratory testing on these test configurations. At the same time, they also identify the test they’d like the get included in the regression suite. These tests are documented at a ‘business level’ i.e. in terms of use case that the end user will perform. What’s avoided in the test document is GUI level details. At this time, we have a dedicated tester who develops automation suite. These documented tests are what an “automation tester” will work as input. Since the automation tester will only use the business classes and methods that the automation framework exposes, development of these automation suite is very fast. Our future plan is to have the functional testers take-up the automation suite development. This approach makes sense since the functional testers have functional & domain knowledge of the application that the automation tester do not have. This will also help avoid documenting of test by functional testers for automation.
The important benefit of the object oriented framework is speed of automated test development. Previous to the framework, development of a typical test, automation suite used to take over a week. This has been reduced to less than a day. What’s more, these tests are much more readable and the automation code purely implements the business logic and not the intricacies of the GUI or the automation tool. This leads to efficient maintenance of the automation suite.
The real benefit of an object oriented approach of development is the use of inheritance and polymorphism. With the Object oriented framework, we are in a position to use inheritance concept on test cases. In functional testing testers typically use an existing test scenario and explore further by making focused changes to this existing test scenario. The same testing approach can be extended by us of inheritance in an object oriented approach. We “inherit” an existing automated test with very minimal change in code that this results in new test cases. In our product, one of the important feature is called ‘analysis’. With the object oriented approach we created a single test case class and extended this class to implement over 500 test cases. With a traditional test automation approach we’d have take several months to complete this suite. With the object oriented approach, the entire effort was less than 2 weeks.
The real benefit of this approach is to bring down the time to execute tests on the additional test configurations. Manual execution of the regression tests used to take about 1 week for each configuration. Considering that we have about 12 to 15 configurations this effort was the most time consuming and monotonous part of our testing process. With automation we’re able to bring down the execution time to about 2 days for each configuration. However even with automation suite in place, there are still some manual tests that require to be executed.
There’s a whole class of tests that would have been manually impossible to execute that we’re planning to implement with this framework. With any there are literally millions of combinations of scenarios that a typical user can use in the field. Functional testers only focus on a small subset of these tests. With an automation framework that works at business level, we have the possibility of getting a very high level of coverage. We are now focusing on the approach of high volume test automation with this framework. To give an example, we have 12 types of ‘analysis’ in our application. Each of these analysis can have 12 sub analysis and this can extend to any level. The ‘analysis’ works on data filters that can be of 16 groups. This leads to around 12*12*16 test scenarios. The complexity although is immense if considered to test manually is only a matter of coding 3 nested loops iterating for each type when implemented with framework.
A complete manual approach for covering all these test configurations is not really a feasible plan. This is where the need of automated test executions comes into picture.
The preferred approach we take is to identify the “primary test configurations” which are typically 2 or 3 test configurations. Functional testers typically does exploratory testing on these test configurations. At the same time, they also identify the test they’d like the get included in the regression suite. These tests are documented at a ‘business level’ i.e. in terms of use case that the end user will perform. What’s avoided in the test document is GUI level details. At this time, we have a dedicated tester who develops automation suite. These documented tests are what an “automation tester” will work as input. Since the automation tester will only use the business classes and methods that the automation framework exposes, development of these automation suite is very fast. Our future plan is to have the functional testers take-up the automation suite development. This approach makes sense since the functional testers have functional & domain knowledge of the application that the automation tester do not have. This will also help avoid documenting of test by functional testers for automation.
The important benefit of the object oriented framework is speed of automated test development. Previous to the framework, development of a typical test, automation suite used to take over a week. This has been reduced to less than a day. What’s more, these tests are much more readable and the automation code purely implements the business logic and not the intricacies of the GUI or the automation tool. This leads to efficient maintenance of the automation suite.
The real benefit of an object oriented approach of development is the use of inheritance and polymorphism. With the Object oriented framework, we are in a position to use inheritance concept on test cases. In functional testing testers typically use an existing test scenario and explore further by making focused changes to this existing test scenario. The same testing approach can be extended by us of inheritance in an object oriented approach. We “inherit” an existing automated test with very minimal change in code that this results in new test cases. In our product, one of the important feature is called ‘analysis’. With the object oriented approach we created a single test case class and extended this class to implement over 500 test cases. With a traditional test automation approach we’d have take several months to complete this suite. With the object oriented approach, the entire effort was less than 2 weeks.
The real benefit of this approach is to bring down the time to execute tests on the additional test configurations. Manual execution of the regression tests used to take about 1 week for each configuration. Considering that we have about 12 to 15 configurations this effort was the most time consuming and monotonous part of our testing process. With automation we’re able to bring down the execution time to about 2 days for each configuration. However even with automation suite in place, there are still some manual tests that require to be executed.
There’s a whole class of tests that would have been manually impossible to execute that we’re planning to implement with this framework. With any there are literally millions of combinations of scenarios that a typical user can use in the field. Functional testers only focus on a small subset of these tests. With an automation framework that works at business level, we have the possibility of getting a very high level of coverage. We are now focusing on the approach of high volume test automation with this framework. To give an example, we have 12 types of ‘analysis’ in our application. Each of these analysis can have 12 sub analysis and this can extend to any level. The ‘analysis’ works on data filters that can be of 16 groups. This leads to around 12*12*16 test scenarios. The complexity although is immense if considered to test manually is only a matter of coding 3 nested loops iterating for each type when implemented with framework.
Analysis and evaluation of results is certainly a challenge in any type of testing. The framework is designed to provide a pass/fail result of each of the verification point coded in the test suite. However it completely relying on the automated results is not sufficient on many of the cases. After the automation execution, functional testers furthur does exploratory testing on the results on the execution.
As like any testing effort, this type of automation approach is intended to improve and enhanse the productivity of functional testers.
This is the concluding part of this series of post.
No comments:
Post a Comment