Sunday, March 23, 2008

Regression Testing:What is it anyway?

My project is in the 'Regression Testing' phase. So I felt it was appropriate to have a blogpost on regression testing. I did a quick googling and came up with some definitions here and here and here and many more. These definitions do make sense, but the crux of the matter is where do these explanations fit into the context of my project?
To put it in simple terms, regression testing is one step in the of the following sequence of events in the life of a software tester:
Testers find bugs, developers fixed it, testers verify the defect fixes, testers again tests, find new bugs. developer fix them and so on and so on...
The regression testing is "testers again tests, find new bugs".
Testers in my project have been working for the past several months, testing various aspects of the application, finding defects, getting those fixed. Eventually a times comes when the defect finding rate comes down and the application is "relatively stable".

The questions from a person not involved in the project will be as below.

Q) So is the application ready for release?
A) For sure not!

Q) What else is remaining to be done, you've completed all the application features you've planned to test?
A) We've to test for regressions.

Q) Is'nt that what you've been doing all these days?
A) Nope! We were doing the 1st round of testing until now. We were hunting for defects in the application. We've ensured that the defect fixes provided to us fixes the specific problem we've raised. And that's all we've done.

Q) What more you've to do?
A) The defect fixes made in the application may have introduced more defects, we've got to find them. We call them regression testing.

Q) So what's it that you've not done in your 1st round of testing that you've gonna do in regression testing?
A) One thing for sure, test the application again. Rerun the test cases and scenario we've already executed.

Q) Why did'nt you do the regression testing earlier?
A) Because the application had defects, defects were being fixed, lots of code churn happening. Its not the good time to start regression testing. Regression testing should start only when the defects are "under control". Regression testing can start only the changes to the code is so much under control that the sanity test suites can be executed on the entire application to ensure there are no additional problems introduced in the code as a result of the defect fixes.

Q) What's this "under control" thing you keep referring to?
A) When the product was delivered to testing at the begining of the testing phase we were finding on the average 15 defects a day. Out of these lets say on the average 10 defects were getting fixed. With so much defects getting fixed and even when new features were sneaked in into the code nobody kept a watch on how these code changes are doing to affect the application. Testers where busy finding new defects in the application and testing newer and newer areas of the application for the first time. Testers never had the time to check if "a particular" defect fix has broken something that was already working. Developers never had the time to see how their fix in one area of the application is going to integrate with the rest of the application. Now things are different. Testers have completed testing the application as a whole, average number of new defect found per day is very low, maybe one or two. Developers have lesser number of defects to deal with. Developers now can do detailed impact analysis of how their fix will impact the rest of the application. They in turn can pass this information to testers. Testers in turn know where to focus their thier efforts

Q) And when are you doing to release this?

A) As soon as the 1st regression test is complete, we'd go in for a relatively faster 2nd regression test. This should give us an gut feel of the quality of the product. As soon as the 2nd or third regression test do not detect any major new defects we are ready for release.

Creative Commons License
The Elusive Bug by Rajesh Kazhankodath is licensed under a Creative Commons Attribution-Share Alike 2.5 India License.