Ordering unit tests in Eclipse's JUnit view

As Gary said in the comments:

it would be nice if Unit Runner could be told to go ahead and order them by class name. Hmm, maybe I should look into the source code...

I did look but there's no hint of a functionality to sort these names. I would suggest a change request to the JUnit plugin, but I don't think, that there are lot of people using this thing, so: DIY.

I would like to see the solution if you modify the plugin code.


One thing that one might do is using the schema of JUnit 3.x. We used a test suite that was called AllTests where you add the tests to it in a specific order. And for every package we got another AllTests. Giving those test suites a name being the same as the package enables one to easily build a hierarchy that should be valued by the junit plugin.

I really dislike how it is even presenting the test methods inside the Junit viewer. It should be in the very same order as they are specified in the TestCase class. I order those methods in the way of importance and features. So the upmost failing method is to correct first and then the more special one in the later part of the test case.

That is really annoying that the test runner is scrambling those. I will take a look at it myself and if I find a solution I will update this answer.

Update:

My problem with the ordering of method names within a TestCase is related to this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180 (Thanks Oracle!).

So in the end oracle changed the ordering of the methods within a class.getMethods or class.getDeclaredMethods call. Now the methods are random and can change between different runs of the JVM. It seams to be related to optimizations of compare or even is an attempt to compress method name - who knows... .

So whats left. First one can use: @FixMethodOrder (from javacodegeeks.com):

  1. @FixMethodOrder(MethodSorters.DEFAULT) – deterministic order based on an internal comparator
  2. @FixMethodOrder(MethodSorters.NAME_ASCENDING) – ascending order of method names
  3. @FixMethodOrder(MethodSorters.JVM) – pre 4.11 way of depending on reflection based order

Well that is stupid but explains why people start using test1TestName schema.

Update2:

I use ASM since Javassist also produces random sorted methods on getMethods(). They use Maps internally. With ASM I just use a Visitor.

package org.junit.runners.model;

import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;

import org.objectweb.asm.ClassReader;
import org.objectweb.asm.ClassVisitor;
import org.objectweb.asm.MethodVisitor;
import org.objectweb.asm.Opcodes;

import com.flirtbox.ioc.OrderTest;

/**
 * @author Martin Kersten
*/
public class TestClassUtil {
public static class MyClassVisitor extends ClassVisitor {
    private final List<String> names;
    public MyClassVisitor(List<String> names) {
        super(Opcodes.ASM4);
        this.names = names;
    }

    @Override
    public MethodVisitor visitMethod(int access, String name, String desc,
            String signature, String[] exceptions) {
        names.add(name);
        return super.visitMethod(access, name, desc, signature, exceptions);
    }
}

private static List<String> getMethodNamesInCorrectOrder(Class<?> clazz) throws IOException {
    InputStream in = OrderTest.class.getResourceAsStream("/" + clazz.getName().replace('.', '/') + ".class");
    ClassReader classReader=new ClassReader(in);
    List<String> methodNames = new ArrayList<>();
    classReader.accept(new MyClassVisitor(methodNames), 0);
    return methodNames;
}

public static void sort(Class<?> fClass, List<FrameworkMethod> list) {
    try {
        final List<String> names = getMethodNamesInCorrectOrder(fClass);
        Collections.sort(list, new Comparator<FrameworkMethod>() {
            @Override
            public int compare(FrameworkMethod methodA, FrameworkMethod methodB) {
                int indexA = names.indexOf(methodA.getName());
                int indexB = names.indexOf(methodB.getName());
                if(indexA == -1)
                    indexA = names.size();
                if(indexB == -1)
                    indexB = names.size();
                return indexA - indexB;
            }
        });
    } catch (IOException e) {
        throw new RuntimeException("Could not optain the method names of " + fClass.getName() + " in correct order", e);
    }
}
}

Just put this in your src/test/java folder in the package org.junit.runners.model. Now copy the org.junit.runners.model.TestClass of the junit 4.5 lib to the same package and alter its constructor by adding the sorting routine.

 public TestClass(Class<?> klass) {
    fClass= klass;
    if (klass != null && klass.getConstructors().length > 1)
        throw new IllegalArgumentException(
                "Test class can only have one constructor");

    for (Class<?> eachClass : getSuperClasses(fClass))
        for (Method eachMethod : eachClass.getDeclaredMethods())
            addToAnnotationLists(new FrameworkMethod(eachMethod));

            //New Part
    for(List<FrameworkMethod> list : fMethodsForAnnotations.values()) {
        TestClassUtil.sort(fClass, list);
    }

    //Remove once you have verified the class is really picked up
    System.out.println("New TestClass for " + klass.getName());

}

Here you go. Now you have nicely sorted methods in the order they are declared within the java file. If you wonder the class path is usually set that way that everything in your src (target or bin) folder is considered first by the classloader. So while defining the very same package and the same class you can 'override' every class / interface in any library you use. Thats the trick!

Update3 I was able to get a tree view of every package and every class in the right order to.

  • The idea is to subclass ParentRunner and then add all classes to it that you identify as being public and having methods annotated with test.
  • Add a getName() method returning only the package name of the class your suite runner is representing (so you see the tree as a package tree without the suite's class name).
  • Inspect subdirectories if you find a certain suite class (I use AllTests for all suite classes).
  • If you do not find a suite class in a subdirectory check all of its subdirectories, this way you dont miss a package containing tests if the parent directory is not containing a suite.

That was it. The suite class I add everywhere is: @RunWith(MySuiteRunner.class) public class AllTests { }

That's it. It should you give enough to start and extend on this one. The suite runner is only using reflection but I sort the test classes and suits of the subdirectories alphabetically and suits of subdirectories (which represent the packages they are in) are sorted upmost.


mark wrote:

it orders them base on execution time, maybe you should sort your methods? source/sort members

mark is right. But you cannot sort your unit test. It's not allowed to speculate about the order of execution.

Unit tests have to be built independently and it's random, how they are called by the UnitRunner.

In most cases, the test methods are sorted alphabetically. The classes are random. Try to use a TestSuite to order your tests.

Tags:

Java

Eclipse

Junit

Recent Posts