How to write automated unit tests for java annotation processor?

This is an old question, but it seems that the state of annotation processor testing hadn't gotten any better, so we released Compile Testing today. The best docs are in package-info.java, but the general idea is that there is a fluent API for testing compilation output when run with an annotation processor. For example,

ASSERT.about(javaSource())
    .that(JavaFileObjects.forResource("HelloWorld.java"))
    .processedWith(new MyAnnotationProcessor())
    .compilesWithoutError()
    .and().generatesSources(JavaFileObjects.forResource("GeneratedHelloWorld.java"));

tests that the processor generates a file that matches GeneratedHelloWorld.java (golden file on the class path). You can also test that the processor produces error output:

JavaFileObject fileObject = JavaFileObjects.forResource("HelloWorld.java");
ASSERT.about(javaSource())
    .that(fileObject)
    .processedWith(new NoHelloWorld())
    .failsToCompile()
    .withErrorContaining("No types named HelloWorld!").in(fileObject).onLine(23).atColumn(5);

This is obviously a lot simpler than mocking and unlike typical integration tests, all of the output is stored in memory.

jOOR is a small Java reflection library that also provides simplified access to the in-memory Java compilation API in javax.tool.JavaCompiler. We added support for this to unit test jOOQ's annotation processors. You can easily write unit tests like this:

@Test
public void testCompileWithAnnotationProcessors() {
    AProcessor p = new AProcessor();

    try {
        Reflect.compile(
            "org.joor.test.FailAnnotationProcessing",
            "package org.joor.test; " +
            "@A " +
            "public class FailAnnotationProcessing { " +
            "}",
            new CompileOptions().processors(p)
        ).create().get();
        Assert.fail();
    }
    catch (ReflectException expected) {
        assertFalse(p.processed);
    }
}

The above example has been taken from this blog post

You're right mocking the annotation processing API (with a mock library like easymock) is painful. I tried this approach and it broke down pretty rapidly. You have to setup to many method call expectations. The tests become unmaintainable.

A state-based test approach worked for me reasonably well. I had to implement the parts of the javax.lang.model.* API I needed for my tests. (That were only < 350 lines of code.)

This is the part of a test to initiate the javax.lang.model objects. After the setup the model should be in the same state as the Java compiler implementation.

DeclaredType typeArgument = declaredType(classElement("returnTypeName"));
DeclaredType validReturnType = declaredType(interfaceElement(GENERATOR_TYPE_NAME), typeArgument);
TypeParameterElement typeParameter = typeParameterElement();
ExecutableElement methodExecutableElement = Model.methodExecutableElement(name, validReturnType, typeParameter);

The static factory methods are defined in the class Model implementing the javax.lang.model.* classes. For example declaredType. (All unsupported operations will throw exceptions.)

public static DeclaredType declaredType(final Element element, final TypeMirror... argumentTypes) {
    return new DeclaredType(){
        @Override public Element asElement() {
            return element;
        }
        @Override public List<? extends TypeMirror> getTypeArguments() {
            return Arrays.asList(argumentTypes);
        }
        @Override public String toString() {
            return format("DeclareTypeModel[element=%s, argumentTypes=%s]",
                    element, Arrays.toString(argumentTypes));
        }
        @Override public <R, P> R accept(TypeVisitor<R, P> v, P p) {
            return v.visitDeclared(this, p);
        }
        @Override public boolean equals(Object obj) { throw new UnsupportedOperationException(); }
        @Override public int hashCode() { throw new UnsupportedOperationException(); }

        @Override public TypeKind getKind() { throw new UnsupportedOperationException(); }
        @Override public TypeMirror getEnclosingType() { throw new UnsupportedOperationException(); }
    };
}

The rest of the test verifies the behavior of the class under test.

Method actual = new Method(environment(), methodExecutableElement);
Method expected = new Method(..);
assertEquals(expected, actual);

You can have a look at the source code of the Quickcheck @Samples and @Iterables source code generator tests. (The code is not optimal, yet. The Method class has to many parameters and the Parameter class is not tested in its own test but as part of the Method test. It should illustrate the approach nevertheless.)

Viel Glück!