What does the sun.reflect.CallerSensitive annotation mean?

According to the JEP I linked to in the comments (also here),

A caller-sensitive method varies its behavior according to the class of its immediate caller. It discovers its caller’s class by invoking the sun.reflect.Reflection.getCallerClass method.

If you look at the implementation of Class#forName(String)

@CallerSensitive
public static Class<?> forName(String className)
            throws ClassNotFoundException {
    return forName0(className, true,
                    ClassLoader.getClassLoader(Reflection.getCallerClass()));
}

, you notice that it is using Reflection.getCallerClass(). If we look at that method

Returns the class of the caller of the method calling this method, ignoring frames associated with java.lang.reflect.Method.invoke() and its implementation.

@CallerSensitive
public static native Class getCallerClass();

The problem, it seems, before this JEP, was that if the caller sensitive method was called through reflection instead of directly, there had to be a complex process to identify what the actual calling class was. This was problematic if the method was invoked through reflection. A simpler process was proposed (and introduced) with @CallerSensitive.

Basically, the @CallerSensitive annotation is used by the JVM

The JVM will track this annotation and, optionally, enforce the invariant that the sun.reflect.Reflection.getCallerClass method can only report the caller of a method when that method is marked with this annotation.

From jdk.internal.reflect.CallerSensitive

A method annotated @CallerSensitive is sensitive to its calling class, via Reflection.getCallerClass or via some equivalent.

The equivalent would be java.lang.StackWalker.getCallerClass since Java SE 9.

This is effectively the security model of Java versions 1.0 and 1.1, which implemented a sort of pauper's linker checking. It's a coherent approach except for anything related to reflection but is exteremely fragile. The more restrictive Java 2 security model, on the other hand, is magic and doesn't show its workings.

@CallerSensitive methods alter behaviour depending upon the class that called them. This is always surprising, but then so is the Java 2 stack-inspection security model. To make matters worse, these method calls are particularly useful for classes that are working on behalf of their caller, so the context is wrong anyway.

The Secure Coding Guidelines for Java SE covers this area.

• Guideline 9-8 / ACCESS-8: Safely invoke standard APIs that bypass SecurityManager checks depending on the immediate caller's class loader
• Guideline 9-9 / ACCESS-9: Safely invoke standard APIs that perform tasks using the immediate caller's class loader instance
• Guideline 9-10 / ACCESS-10: Be aware of standard APIs that perform Java language access checks against the immediate caller
• Guideline 9-11 / ACCESS-11: Be aware java.lang.reflect.Method.invoke is ignored for checking the immediate caller
• Guideline 9-12 / ACCESS-12: Avoid using caller-sensitive method names in interface classes

The introduction of modules in Java SE 9 has meant that some of the details have changed.

Additionally if the @CallerSensitive method is called with Method.invoke some of the stack frames are ignored by getCallerClass. Internally the JDK uses a "trampoline" ClassLoader to act as a benign dummy caller. java.lang.invoke.MethodHandle copies problems of Method.invoke. The AccessController.doPrivileged methods are also problematic with the consequences of the specification being surprising.

JEP 176: Mechanical Checking of Caller-Sensitive Methods deals with the particular @CallerSensitive jdk-internal annotation. The method lists in the Guidelines were generated by a FindBugs plug-in but then manually updated.