Can you limit annotation target to be subclasses of a certain class?

Yes, this is possible (and was possible when the question was asked).

As a general rule, when working with annotations you need to use an annotation processor. You can write an annotation processor that issues errors whenever an annotation is written in a disallowed location.

If your question is whether this is possible with plain javac and no annotation processor, then the answer is "no".

Not at compile-time; the only restrictions available for annotation placement are by element type (method, class, etc.).