NB: This article is the english version of my previous article, written in french.

For some reasons, I wanted to categorize my JUnit tests, in order to run only a subset of them.
But this is not as simple as it seems…

Context

In one of my projects, I have many unit tests. It appears that many of them are not unit at all.
Some of them require a real Oracle connection, some others need to load a complex Spring context…

My goal is to split these tests into several categories.

To ease the explanations, I will only consider the two following categories: @FastTest and @SlowTest.

Possible solutions

JUnit introduced recently, in version 4.8, the @Category annotation (see the changelog and also an example here).
The principle is to annotate a class (or a method) with @Category, and then to define the category on which it belongs. For my needs, I could write:

// Interfaces that define my test categories
public interface FastTest { }
public interface SlowTest { }

@Category(SlowTest.class)
public class ASlowTest {
    @Test
    public void foo() { ... }
}

@Category(FastTest.class)
public class ARealUnitTest {
    @Test
    public void foo() { ... }
}

This seems pretty good. But if we go into the details, in order to run all the tests that belong to a specific category, I have to write:

@RunWith(Categories.class)
@IncludeCategory(FastTest.class)
@SuiteClasses( { ARealUnitTest.class, AnotherRealUnitTest.class })
public class FastTestsSuite { }

Here comes the problem: I have to create a Suite (using the annotation @SuiteClasses) to define the classes that should be executed.
That means I have to maintain a list of all the tests for a given Suite. The omissions are therefore likely to be frequent…

Not using JUnit Suite?

How to solve this problem? Here are some ideas:

  • Write the JUnit Suite manually as I just explained before, but I may forget to maintain them…
  • Let a Maven plugin creates the Suite. I wrote such a plugin some years ago, and I can reuse it now. However, I need to define when this plugin should be executed. Indeed, as the purpose of this plugin is to create or modify Java classes, I can’t run it automatically or anytime. Moreover, it is possible that some classes should not be included in these suites. So this solution will not be considered here.
  • Use specific patterns for the filenames of the test classes. For example, the fast tests will be named **FastTest.java, while the other ones will be named **SlowTest.java. Then, I will configure Maven, and some profiles, to use the correct pattern regarding which kind of tests category I want to run. I will not consider this solution neither, as this may also lead to errors.
  • Use TestNG instead of JUnit. TestNG is another test framework, which provides some advanced features. One of them is the definition of groups that would be perfect for our needs, especially since Surefire (the test plugin for Maven) natively supports them. This is maybe the best solution for our current concern, but I want to keep JUnit, so, again, I will ignore this idea.
  • Modify the Surefire plugin (or why not JUnit directly?) in order to manage the @Category without having to define a Suite. However, this is not a good idea to modify these libraries…

So none of these ideas seems to be helpful…

The last idea: Using JUnit Suite!

The solution I will explain now is based on JUnit Suite. However, these suites will define dynamically the test classes that should be run.
This Suite will look into the classpath for all the classes that are annotated with a particular annotation.

During my searches, I found a project called classpathSuite. One of the purpose of this project is to build a Suite by looking into the ClassPath for some specific classes. Exactly what I want to do!

Using this library, I write the following classes:

@RunWith(Categories.class)
@Categories.IncludeCategory(FastTest.class)
@Suite.SuiteClasses( { AllTests.class })
public class FastTestsSuite {
}

@RunWith(ClasspathSuite.class)
public class AllTests {
}

and classpathSuite will find all the classes in the ClassPath that belong to the category FastTest.
Nice! But this seems to work only when I run my tests in Eclipse, not in a Maven context :( Or did I miss something?

So I have to re-implement this feature, but that will work with Maven!

The solution

First step: I create two simple annotations, @FastTest and @SlowTest.
They will be used to annotate my JUnit test classes. Note that my solution will only allow you to categorize an entire class, not a method. These annotations will be a replacement for @Category(FastTest.class) and @Category(SlowTest.class).

Now, I create a utility class that will look into the classpath and find the list of classes that are annotated with a specific annotation (@FastTest or @SlowTest in my case) and that belong to a given package (this condition is not technically mandatory, it is just to limit the searches).

On the Java point of view, I get:

public final class ClasspathClassesFinder {

    /**
     * Get the list of classes of a given package name, and that are annotated by a given annotation.
     *
     * @param packageName The package name of the classes.
     * @param testAnnotation The annotation the class should be annotated with.
     * @return The List of classes that matches the requirements.
     */
    public static Class<?>[] getSuiteClasses(String packageName, Class<? extends Annotation> testAnnotation) {
        try {
            return getClasses(packageName, testAnnotation);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }

    /**
     * Get the list of classes of a given package name, and that are annotated by a given annotation.
     *
     * @param packageName The package name of the classes.
     * @param annotation The annotation the class should be annotated with.
     * @return The List of classes that matches the requirements.
     * @throws ClassNotFoundException If something goes wrong...
     * @throws IOException If something goes wrong...
     */
    private static Class<?>[] getClasses(String packageName, Class<? extends Annotation> annotation) throws ClassNotFoundException, IOException {
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        String path = packageName.replace('.', '/');
        // Get classpath
        Enumeration<URL> resources = classLoader.getResources(path);
        List<File> dirs = new ArrayList<File>();
        while (resources.hasMoreElements()) {
            URL resource = resources.nextElement();
            dirs.add(new File(resource.getFile()));
        }
        // For each classpath, get the classes.
        ArrayList<Class<?>> classes = new ArrayList<Class<?>>();
        for (File directory : dirs) {
            classes.addAll(findClasses(directory, packageName, annotation));
        }
        return classes.toArray(new Class[classes.size()]);
    }

    /**
     * Find classes, in a given directory (recursively), for a given package name, that are annotated by a given annotation.
     *
     * @param directory The directory where to look for.
     * @param packageName The package name of the classes.
     * @param annotation The annotation the class should be annotated with.
     * @return The List of classes that matches the requirements.
     * @throws ClassNotFoundException If something goes wrong...
     */
    private static List<Class<?>> findClasses(File directory, String packageName, Class<? extends Annotation> annotation)
            throws ClassNotFoundException {
        List<Class<?>> classes = new ArrayList<Class<?>>();
        if (!directory.exists()) {
            return classes;
        }
        File[] files = directory.listFiles();
        for (File file : files) {
            if (file.isDirectory()) {
                classes.addAll(findClasses(file, packageName + "." + file.getName(), annotation));
            } else if (file.getName().endsWith(".class")) {
                // We remove the .class at the end of the filename to get the class name...
                Class<?> clazz = Class.forName(packageName + '.' + file.getName().substring(0, file.getName().length() - 6));
                // Does the file is annotated with the given annotation?
                if (clazz.getAnnotation(annotation) != null) {
                    classes.add(clazz);
                }
            }
        }
        return classes;
    }

NB: I used the example given in this post to write this code.

Thus, the code ClasspathClassesFinder.getSuiteClasses("my.company", FastTest.class) will return an array of classes that belong to the package my.company and annotated with @FastTest.

The next step consists in building a Suite that will use my utility class to find all the fast tests:

public class FastTestsSuite extends org.junit.runners.Suite {

    public FastTestsSuite(Class<?> clazz, RunnerBuilder builder) throws InitializationError {
        this(builder, clazz, ClasspathClassesFinder.getSuiteClasses("my.company", FastTest.class));
    }

    public FastTestsSuite(RunnerBuilder builder, Class<?> clazz, Class<?>[] suiteClasses) throws InitializationError {
        super(builder, clazz, suiteClasses);
    }

}

Last class to write: the entry point for the Maven Surefire plugin (this class will only run @FastTest, so I will have to write the same class for @SlowTest):

@RunWith(FastTestsSuite.class)
public class RunAllFastTests {

}

Now, I just need to configure Maven in order to run « only » this test:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <!-- Run only tests annotated with @FastTest. -->
        <includes>
            <include>**/RunAllFastTests.java</include>
        </includes>
        ...
    </configuration>
</plugin>

Please note that if you are working on a project with multi-modules (this is my case), you will have to define this RunAllFastTests.java in each module.

Regarding the slow tests, I just have to define a Maven profile that will launch the RunAllSlowTests class, which calls the Suite SlowTestsSuite. This Suite will use the utility class to find all classes annotated with @SlowTest. My continuous integration server Hudson Jenkins will activate this profile during the nightly-builds.

Conclusion

Finally, this solution is not really complex, and when the mechanism is developed, you just have to annotate your JUnit test class with the adequate annotation. This is simple.

However, I am quite disappointed that JUnit, a widely used test framework, does not provide such a mechanism natively. I hope that it will change in the near future…

Feel free to post your comments if you think that my solution is not the best one to solve this kind of problem.

Sources

Jean-Philippe Briend suggested me on Twitter to provide the sources of the classes. Here they are:

ZIP

This project requires Maven 2 or 3, Java 1.6 and JUnit 4.8.1.

In addition to all the classes required for the mechanism, this project contains three basic unit tests:

  • OneFastTest, annotated with @FastTest ;
  • OneSlowTest, annotated with @SlowTest ;
  • TestWithoutAnnotation, without any annotation. As this test should not be executed, it contains a method that will call the JUnit fail() method. Thus, if this class is executed, the test will fail.

If you run the command mvn clean install, only the class annotated with @FastTest will be executed.
If you run the command mvn clean install -Djenkins=true (or -Pjenkins), both annotated classes will be executed.

Start Slide Show with PicLens Lite PicLens