Saturday, December 29, 2007

Lessons Learned Implementing DBVTE

"DBVTE" is an acronym I'd made up about 2 minutes ago when I decided to blog this. It stands for Daily Build Verification Test Executions.

The Background:

My current project has gained much notoriety among testing team for the poor quality builds delivered for testing. Early in the project iterations, the issue was there was no testing build at all. Later on after much delay the daily build process was finally got booted up. But then it came up with its on set of issues. Lack of a coherent integration test ensured the daily builds never work as a single unit. It'd be a miracle if build picked on any particular day will be of minimum quality for testers to start the tests. The situation demanded a consistent way of delivering testable builds on a periodic basis.

The Approach:

The way to go here was for the functional testing team to take-up the integration testing activity. We'd started with documenting the bare minimum use case & scenarios of the product in sufficient detail that a novice computer user could read the document and execute those scenarios. These were written mostly from a GUI perspective like "Step 7. click OK button", "Step 11. Right Click, select Edit From Menu", "Step 23. Verify Graph". These are also called smoke tests or sanity tests.

The important activity here was to identify a dedicated machine for the integration testing purpose. Its very important to have a consistent, reproducible and stable environment for the integration testing machine.

Now that the nightly builds were getting build, we'd developed a series of simple windows batch scripts. The purpose of these scripts were to download the build files ( java jar files, codes developed in SAS scripts, configuration files...) from build machine. The scripts did administrative activities such as stopping the application services, restarting services etc as well lay down the build files like the installer would have performed. The purpose of the script was to eliminate human error in the installation process and to save the time in the install process. A typical install process of the application we test will require 3 hours of install & configuration time. The automated install scripts does this in 10 minutes flat!.

We'd chosen the "SAFS" as our automation tool for automating the smoke tests. Although its not mandatory to automate your smoke tests, we'd taken the approach to automate primarily to ease the drudgery of executing the tests by a human tester. SAFS is an excellent tool and it very neatly took care of incremental changes to the application. Moreover SAFS is developed and supported within our company.

The Process:

Now that we've had the tools in place, its time to work on the process around it. The most important part of any process is to get the buy in from the stakeholders. The good part about it was the development and project mangers were already aware of the problem with integration. Just that they never had given enough thoughts about how to tackle the problem. Our case is that we've the tools in place for integration tests. A quick demo of the automated install scripts and the automated smoke test was a very well accepted. Now was the case to present the case for the process and the development teams commitment for the process.

The process was simple; testing team will download daily builds from the build machine, execute the automated scripts and send in the results of executions to the entire team. In case of defects, encountered in the automated scripts, these will be logged and will be part of the execution results. The development team on its part, should fix the defects immediately or back out the changes in the next build.

The Lessons:
  1. Developing Automation scripts of any GUI tests is hard. We were baffled bySAFS automation tool limitations for which workaround were found. Our GUI was developed on Eclipse framework and support for it was limited in SAFS.
  2. Development of Automation scripts for GUI is incremental. Develop scripts with smaller features, test, stabilize and then operationally.
  3. Concentrate on the breadth of the functionality, not the on the depth while developing the smoke tests. Check all basic features of the application rather than the complex features.
  4. Automate the install and configuration process of daily builds. This will avoid human error in the installation as well as speedup the installation process.
  5. Start with a clean machine for installing the first build on the integration machine. In case of windows, format and reinstall the OS. On Unix, this process is expensive.
  6. Cherish your process. Follow your integration process religiously and advocate it within your entire team.

No comments:

 
Creative Commons License
The Elusive Bug by Rajesh Kazhankodath is licensed under a Creative Commons Attribution-Share Alike 2.5 India License.