Saturday, August 9, 2008

Test Cases as security blanket

There's a raging controversy going on, at least in the Indian software testing community about "test cases centric testing" vs "non test cases centric" testing. Pradeep has posted here at length on this subject, also has the imprints of the controversy all over the web, from forum posts to blogs to yahoo groups.

My take on this is to consider a test case as a security blanket. Wiki defines security blanket as "A security blanket is any familiar object whose presence provides comfort or security to its owner, such as the literal blankets often favoured by small children". It gives the sense of comfort for the person using it. "Person" I mean not just testers running the test script but also developers developing the product, product managers or end customers. Everyone derives comfort from the fact that "test cases have been executed and passed"

Are security blankets necessarily bad? certainly not!. For me as a test manager, it gives me comfort ( and thereby the sense of security) to the fact that the software I'm going to sign-off will not fail when used within the confines of the test scenarios that my team has found to be passing. Does it mean that my test cases are foolproof /security blanket is without holes? Certainly not and its my responsibility as a test manager to make it understand to the stakeholders that the test cases are never foolproof.

To find bugs in a software is the most important responsibility of a test team and a set of predefined execution paths ( aka test cases ) will certainly not find many defects. But the test cases has its rightful and important place as a security blanket within the confines of a software development cycle. During the final phases of the release cycle, several rounds of test cases executions provide the necessary confidence to the stakeholders that the product is ready for release.

Again, I've never seen anyone questioning "what's a test case?" all thoughout the controversy. When I mention a test case in this blogpost, I'd need to make it clear that "I" mean by a "test case". This will be a topic for another blogpost.

Equate test cases to comfort object ; certainly not harmful in anyway and certainly provide the psychological strength. To conclude it all, software development is all about human behavior and interactions!


6 comments:

Pradeep Soundararajan said...

@Rajesh,

I'm going to sign-off will not fail when used within the confines of the test scenarios that my team has found to be passing

Those tests appear to have passed at an environment that is so different from your customer environment.

If a sense of security is what you want to gain by getting a software tested then you would ever fear "What probably did we miss that might show up there?"

instead...

you might want to think that a test manager helps the team achieve coverage by translating the senior management's vision to missions that your teams can chew

plus

Software Testing is about questioning a product in order to evaluate it -- James Bach

I understand that your senior management wants you to give that confidence that they want you to give before you sign off BUT they know little about testing than you and hence might need some education.

Some managers stay with the senior management for long enough time to help the management members gain a few important lessons about testing they "must" know in order to work with us.

As I have seen you in action, I am hoping that you are already doing that to some extent but if they appear to ask the same set of questions, you might want to use re-phrase heuristic.

Here is my question: If you start an IT organization ( pumping all money you have earned ) to develop a product as critical as you work on today, would you be fine with a test manager thinking of test cases as a way to help him secure himself from the questions you ask him?

Message to testers and test managers: Here is a quote that I heard in Toronto Workshop in Software Testing and I think its either Jerry Weinberg who said, "Pleasing someone, displeases the purpose of why we are there"

Anonymous said...

I too agree with Rajesh as using test case as a "Security Blanket" when it comes to evaluating a finacial product.

As pradeep said, Yes we need to educate the senior management & customer but eventually when the product fails during UAT or Acceptance testing then probablly the TEST MANAGER is questionable.

Hence i feel that there should be a complete change in thought process as how the customer/senior management thinks.


Rgds,
Raj

Pradeep Soundararajan said...

@Anonymous,

As pradeep said, Yes we need to educate the senior management & customer but eventually when the product fails during UAT or Acceptance testing then probablly the TEST MANAGER is questionable.

By testing we find information about the quality of the product not fix the problems. If we are problem fixers then I can be held responsible.

Hence i feel that there should be a complete change in thought process as how the customer/senior management thinks.

Do you know that people have been saying this for years and nothing has changed? Its because they kept saying it and did nothing to change the situation but there are a few people who did that.

Which one are you?

Anonymous said...

Hi,

We are not 'Problem Fixers' but we are responsible for quality of the application. Hence if customer finds a defect, then surelly this should have been found by the test team.
Assume if the customer finds more then to whom would the blame go? Tester or Test Manager?

Hence for so called "Not Educated" sr. management having a security blanket is no harm right?

Surelly i too would like to change the notion of what the senior management thinks.
Kudos to those who have changed the mindset of the sr. management, but still lot to be done.

And i am one amongst them who is striving to change the mindset.

Rgds,
Raj

Rajesh Kazhankodath said...

@Pradeep , Raj

>>>>>>Here is my question: If you start an IT organization ( pumping all money you have earned ) to develop a product as critical as you work on today, would you be fine with a test manager thinking of test cases as a way to help him secure himself from the questions you ask him?


Certainly not!. But I'd ask him about the steps he's taken to cover the risk of the application failing at customer end.

>>>>> I understand that your senior management wants you to give that confidence that they want you to give before you sign off BUT they know little about testing than you and hence might need some education.

Forunately in my context, senior managerment is quite understanding about the risks involved in testing. They very well understand that the testing process itself is subject to the same set of human error that the development process is subject to. However managements expects the testing team to own the responsibility of NOT finding any significant defects DURING THE TIME SCHEDULED for testing. If customers detects major functional issues in the product and an analysis of "why the issue ended up being detected by the customer" points to the fact that there are shortcomming in the process that the testing team followed, certainly the test manager should be in the line of fire and not the development manager. Of course its not possible to test all the possible scenario the product will be used under. That's why customer reported issues should be analysed and the testing process be continiously be fine-tuned so that similar issues are not found in future. In such a context, testing need to be looked at as a risk avoidance strategy.

Regards
Rajesh.

Anonymous said...

Hi,

Thanks Rajesh/Pradeep for your value added comments.

Rgds,
Raj

 
Creative Commons License
The Elusive Bug by Rajesh Kazhankodath is licensed under a Creative Commons Attribution-Share Alike 2.5 India License.