Test Case!

Here is a basic definition that I think is very relevant and practical while defining what a test case is.

A test case defines a measurement of the software according to currently prevailing business rules and conditions upon which the software is built. A good test case holds the potential to uncover a defect in the software.

To define a set of meaningful test cases is obviously not an easy matter! Proper understandings of the business functionalities often go beyond just specifications and at times even beyond business rules. The knowledge that a tester gains in a particular domain gets easily reflected in his test case work. The test cases document (henceforth referred to as test specification) is the most important testing lifecycle document deliverable that sets the testing standards and also provides direction for future test ideas. If this specification lacks depth and coverage then its ripple effect will be clearly felt on the quality of testing and on the software as this specification forms the primary input for testing.

Besides functional knowledge the author of a test specification also needs to possess the maturity and skill to strike a balance between depth and coverage! On one side, it is often noticed that testers fails to bring in the level of granularity that they normally exercise during manual testing, and on the opposite side they end up having repetitive test cases that hold little value. Between over generalizing and over detailing lies a narrow band that makes the test specification just right! The obvious reason for this is that those who seek excess depth lose sight of the horizons (read as coverage) and vice-versa.

Another important factor that spices up the test specification are the negative test conditions! The value of negative tests is often underestimated to the extent where it reduces the testing to a mere certification activity. “Positive & Negative” tests represent the two pillars of testing “Verification & Validation” at the lowest levels. Negative tests determine if we are building the software right (Boris Beizer) at the atomic levels. However, I must say that most customers don’t care two hoots about the difference between V & V. Simply put, a negative test simulates an error condition and forces the software to react uncharacteristically. It checks if the developer has managed to think out-of-box while designing his algorithm. Thereby it’s needless to say that designing good negative tests needs smarts!

Though there are no specific dos and don’ts for writing test specifications, but as one picks up experience with test definitions one learns to form a few cardinal rules. Here is my set of rules:
- Every test case must be atomic (for a single test condition) and must not contain references to other test cases or documents.
- The test environment and data required by a test case should be completely defined.
- The test steps must contain enough information to rule out assumptions and ambiguities.
- If there are separate test cases for atomic conditions A and B, then there isn’t a need to write another test case combining conditions A & B, unless they have a direct dependency in their code paths. This will help in reducing repetitions.
For example where condition A is directly dependent on condition B:
if (A) {
if (B) {
do X;
}
}
***********WORK IN PROGRESS*****************

Categorizing tests in JUnit4

The JUnit4's annotation based framework doesn't have provisions to selectively pick test methods from test classes and organize them into suites, though it is possible to run the whole test class as a part of the suite. This functionality is available in Junit3.8, but somehow it has got missed in JUnit4.x! I don't see why it had to be omitted specially when frameworks like TestNG and NUnit provide this!

The ability to organize automated test cases into suites is crucial because we can't afford bulky regression suites that seem to run forever. Regressions have to be nimble and effective and are usually directed towards the scope of testing. For example, if there is a security bug fix then it suffice to run the "security" related test cases only and not the entire long regressions, which would be overkill!

However, thanks to David Saff's help we were able to build back this capability into JUnit4. Here is my attempt to describe the procedure to do this.

Below is the outline of a test class & test method that illustrates the use of categories:
package framework.test;
import framework.filters.Category;
import framework.runner.SuiteFilterRunner;
import framework.suite.CategoryFilterSuite;
import org.junit.Test;

public class MathTest{
public double fValue1= 2.0;
public double fValue2= 3.0;
@Test
@Category({"Sanity","SRG","Simple Math"}
public void addition() {
double result = fValue1 + fValue2;
Assert.assertTrue("Double addition error : ",result==5.0);
}

@Test
@Category({"LRG","Advanced Math"}
public void power() {
double result = java.lang.Math.pow(fValue1,fValue2);
Assert.assertTrue("Double addition error : ",result==8.0);
}
}

Test suite to run the above class:
package framework.suite;
import framework.test.MathTest;
import org.junit.runner.RunWith;
import framework.filters.CategoryFilter;
import framework.runner.SuiteFilterRunner;
@RunWith(SuiteFilterRunner.class)
@ SuiteFilterRunner.SuiteProperties (
filterClass = CategoryFilter.class,
filters = {"Simple Math","LRG"},
testClass = { MathTest.class}
)
public class CategoryFilterSuite { }

The testclass "MathTest" is quite self-explanatory, so lets move on to the suite class. The following features can be seen for the suite class:

• The suite is run with a custom runner class called "SuiteFilter.class" instead of the usual JUnit "suite.class".
• SuiteFilter also defines an annotation called "SuiteProperties". This annotation defines a few properties of a test suite.
• filterClass = CategoryFilter.class -> This implies that the custom runner should use the filter defined in the class "CategoryFilter.class" to determine the set of test methods to be run.
• filters = {"Simple Math","LRG"} -> This implies only those test methods must be run which belong to categories "Simple Math" and/or "LRG". In our case, both the addition( ) and power( ) methods would get called.
• testClass = { MathTest.class}) -> This property contains comma separated names of test classes to be run.

Btw, here is the definition of the Category annotation:
package framework.filters;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Category {
String[] value();
}

Now that we have set the expectation right, we can delve into further details. The CategoryFilter class extends the JUnit4 abstract class Filter and it overrides the "shouldRun" and "describe" methods.

This is how the class looks:
package framework.filters;
import java.lang.reflect.Method;
import java.util.StringTokenizer;
import org.junit.runner.Description;
import org.junit.runner.manipulation.Filter;

public class CategoryFilter extends Filter {

public String[] _filters;
public void setCategoryFilters(String[] filters) {
_filters = filters;
}

@Override
public boolean shouldRun(Description description) {
if (description.isTest()) {
Category category = getCategory(description);
if (category != null) {
for(int i=0;i<category.value().length;i++) {
if (runTest(category.value()[i]))
return true;
}
return false;
} else {
return false;
}
}
for (Description each : description.getChildren())
if (shouldRun(each))
return true;
return false;
}

private Category getCategory (Description description) {
try {
StringTokenizer tokenizer = new StringTokenizer(description.getDisplayName(),"(");
String methodName = tokenizer.nextToken();
String className = tokenizer.nextToken(")").substring(1);
Class clazz = Class.forName(className);
Method m = clazz.getMethod(methodName);
return m.getAnnotation(Category.class);
} catch (Exception e) {
e.printStackTrace();
}
return null;
}

@Override
public String describe() {
String categories = "";
for(int i=0;i<getFilters().length;i++)
categories = categories + ", " + getFilters()[i];
return "Running methods beginning with - " + categories;
}

private boolean runTest(String categoryName) {
String[] filters = getFilters();
for(int i=0;i<filters.length;i++) {
if (categoryName.equalsIgnoreCase(filters[i]))
return true;
}
return false;
}

protected String[] getFilters() {
return _filters;
}
}

JUnits runner actually takes care of invoking the “shouldRun” and passing in a Description object that contains the entire test suite (test classes and methods) arranged in a hierarchical manner. A “for” loop iterates through the entire hierarchy and recursively calls “shouldRun” on each child object. If the object turns out to be a test method then it gets the categories assigned to the method and then checks if any of those categories are present in the “filters” list of the test suite (the filters is set by calling the “setCategoryFilters” method). If a match is found then it simply returns “true” and JUnit then takes care of running that test method.

Finally, we need the “SuiteFilterRunner” class to orchestrate a part of the run so that we can pass on the required filters to the CategoryFilter class.

Here is how it looks:
package framework.runner;
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import oracle.search.plugin.framework.category.CategoryFilter;
import org.apache.log4j.Logger;
import org.junit.internal.runners.InitializationError;
import org.junit.runner.Description;
import org.junit.runner.Request;
import org.junit.runner.Runner;
import org.junit.runner.manipulation.Filter;
import org.junit.runner.notification.RunNotifier;

public class SuiteFilterRunner extends Runner
{
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
public @interface SuiteProperties {
Class filterClass();
String[] filters();
Class[] testClass();
}

private final Runner delegate;
private static Logger logger = Logger.getLogger(SuiteFilterRunner.class.getName());

public SuiteFilterRunner(Class klass) throws Exception {
Request filteredRequest = null;
Request classRequest = null;

classRequest = Request.classes("SuiteRun", getAnnotatedClasses(klass));
Filter filter = getFilterClass(klass);
filteredRequest = classRequest.filterWith(filter);
delegate = filteredRequest.getRunner();
}

private Filter getFilterClass(Class klass) throws Exception {
SuiteProperties fKlass = (SuiteProperties) klass.getAnnotation(SuiteProperties.class);
String suite = System.getProperty("suites");
logger.info("value of suites is "+ suite);

if (fKlass.filterClass().getName().endsWith("CategoryFilter")) {
CategoryFilter filter = (CategoryFilter)(fKlass.filterClass().newInstance());
filter.setCategoryFilters(getFilters(klass));
return filter;
} else {
logger.fatal("FILTER CLASS NOT INCLUDED IN CUSTOM RUNNER");
return null;
}
}

private String[] getFilters(Class klass) throws Exception {
return ((SuiteProperties)klass.getAnnotation(SuiteProperties.class)).filters();
}

private Class[] getAnnotatedClasses(Class klass) throws InitializationError {
SuiteProperties annotation = (SuiteProperties)klass.getAnnotation(SuiteProperties.class);
if (annotation == null)
throw new InitializationError(String.format("class '%s' must have a SuiteProperties annotation", klass.getName()));
return annotation.testClass();
}

/**
* @return Description showing the tests to be run by the receiver, JUnit
*/
@Override
public Description getDescription() {
return delegate.getDescription();
}

/**
* @param notifier Runs the tests for this runner, this method gets called by JUnit
*/
@Override
public void run(RunNotifier notifier) {
delegate.run(notifier);
}
}

This class extends the default JUnit Runner class and it overrides the run method, which gets called by JUnit. The constructor of the class takes care of creating a Request object capturing all the test classes to be run and then also applies the category filter using the “filterWith” method on this object. Applying the filters implies calling the "shouldRun" method defined in the filter class (CategoryFilter) in order to decide which test methods would be run. Thereafter the getRunner method is called on the final filteredRequest object and it is delegated to the custom runner. When JUnit calls the overridden run method then this final request is actually run.

Purpose of testing!

I think this topic has been dealt with numerous times before and I am not sure if my treatment of this topic is going to sound any different or revolutionary. In my view, there is nothing complicated to the purpose of testing and it will be my sincere attempt to lay it out to you that way. It astonishes that even after so many years of testing my definition of its purpose hasn't changed one bit. You may be tempted to think that this is because my knowledge on testing hasn’t progressed much :), and honestly I sometimes feel that way when reading other blogs & websites on testing! I always seem to come across a collection of new testing jargons that I am happily ignorant about and I am beginning to think that I am getting retro here! It is heartening to know that so much of thought and effort has gone into testing, but then I often find so little of it to be actually useful to my work and me. Somehow we technical people are good at giving lengthy & complicated explanations for simple jobs! Therefore, I still stubbornly believe that the actual purpose of testing is very basic!

Testing is the art of exercising the software while putting
one self in the customer’s shoe.

I know, this definition sounds very non-technical :). If you had expected more than that then I am sorry to disappointed you! However, I can vouch that it is 100% true and has stood the test of time. Have you ever had the chance to experience an angry customer giving a piece of his mind about that software you tested? Well, why shouldn’t he, after all he has paid good money for it! If you still haven’t gotten a feel for that first definition then lets put that aside and move on to the second one. So, here is my second attempt at defining the purpose of testing, though it isn't too different in spirit when compared to the first one, but I am hoping that it might sound more appealing to the technocrats!

Testing aims at minimizing the risk of failure while delivering perceivably good quality software within budget and on time.

By the way, at this point I must mention that the above definition is somewhat similar to the phase 3 definition of Bezier’s path to testers enlightenment as stated below:

Testing is an activity carried out in order to reduce the failures in the software to within limits that are acceptable to the end user.

In the real world “good quality” is relative to the necessities of the end user. The manner in which he perceives good quality is paramount because he is the final judge. In the software industry the customer’s perception is the point of reference, the perfect. If you fall short of this reference then you sincerely apologize and do whatever it takes to be as close as possible to the perfection or you say that it is technically infeasible! A good tester understands this perception. He anticipates the software to react just like the end user would. This kind of skill is not built in a day or just by thoroughly reading the specifications cover to cover. Good testers knowledge goes far beyond the specification that defines the software. If the opposite were true then people would have built clever tools to glean out all the test cases directly from the functional specifications! Actually, the specifications, just like the customer’s perception is never constant. We all know that specifications themselves are never full and final. Though they can function as a source of truth, but they continually change and evolve along with the knowledge that it is built upon. If you are perceptive enough to give the customer that extra feature that he didn't know he wanted, you are God for him!

There was this one time when a teller at a bank using our retail banking software application was very frustrated because the tabbing order on the demand draft screen was all wrong! In particular, since that branch was situated in a business district, he had to actually process hundreds of demand drafts a day. Every time he had to make entries onto the screen he had to tab back & forth between the various fields, which wasn’t in the same order as it occurred on the physical demand draft. The bank manager escalated this matter to the top and the top sighted this (along with a few other issues) to say that the quality of the release was not up to the mark. On diagnosing the matter it was found that there was a bug filed by a QA engineer for this issue, but it was assigned lowest priority by him because the he didn't think it to be too important. Development had conveniently marked it for a future release! The management obviously cut a sorry figure before the customer and then went about ensuring that the tabbing order on every screen was correct and signed-off by testing. Tabbing sequence became a permanent feature of every test specification thereafter!

Incorrect tabbing order was not that much of a big problem to the tester and he looked at it as an insignificant UI issue and rightly so because he didn't have to process hundreds of demand drafts a day! I don't mean to say that the poor tester must be forced to perform hundreds of transactions, but it is meant to illustrate the point that only the wearer knows where the shoe bites. The more the tester is able to put himself into the shoes of the teller, the more valuable his work will become. He will be able to effectively examine and explore the software from a teller’s perspective and that is great quality! If the developer who triaged the bug had understood the customer, he wouldn't have pushed it to the future release, there again is a failure of perception leading to a lower quality product/software.

Most of the customer bug analyses have led me to one or more of the following conclusions:

• The bug is not addressed in the specifications and therefore it is more an enhancement than a bug.
• The specific environment on which the issue was found is not available on the testing side and hence not reproducible.
• The bug did not occur on the test environment because it does not have the data set required.

We could potentially add more reasons to the above, but all of them will somehow prove that there is a difference between the mindset and the environment of the tester and the end user.

Therefore, testing processes must set goals to bridge the gap between testers and end users in the following manner:

• By learning the mindset of the end user.
• By bringing in forethought in the functional knowledge of the system.
• By learning how to set up a realistic test environment and test database.
• By learning the practical problems faced by the user.

It should be pretty obvious by now that testing is not about finding many bugs or just the most important of bugs, it is about finding the right set of bugs that the customers won't tolerate.

Finally, I come to my third and final attempt at defining the purpose of testing.

Testing gives an idea of how risky the product is when it is being
released.

This is clearly one of the hugest benefits of testing! Theoretically speaking the more time you have to test, the more bugs you can potentially find, but in practice, people have deadlines to meet and customers won’t wait forever. With coding activities eating into most of the time the window of opportunity is slim, but very important. It is important because this is the time when the software is assessed to confirm if the software delivers what it has promised to and in the best possible manner. Without this, no one has a clear picture of the true quality of the software. Once all the test cases are executed and the bugs roll in, then begins its true assessment. It is not enough to just plainly state how many bugs were found in each module and how many of them were severe, rather it is very important to also clearly state if the test coverage for the module was satisfactory and if the module can be considered to be buggy. If we carefully analyze bugs it will be possible to identify patterns of buggy ness, and badly coded areas. Even analyzing the foul bugs aid in detecting issues like incorrectly setup test environments, improper test data, ill-defined functionalities and inadequate specifications. Known issues like known devils are in many ways better than unknown ones!