Do enums violate open/closed principle Java?
The answer is no as OCP doesn't apply (and cannot apply) to Enums. Enums should be complete (contain all possible values) and static (static = final, non-mutable). You can see them as a small, finite set of Value Objects.
If you want something that can be extended, you can always use a class or build your own "type-safe enumeration" that can be extended.
About the language. I choose to use the term apply rather than violate, as principles, as many good practices, have a context where it makes sense to apply them. A violation (for me) means that a situation calls to use a principle, but it's not used or it's not implemented properly. It doesn't apply means what it says on the tin, that the principle brings no use in the current context or that it cannot be applied, as otherwise would go against other principles which have stronger forces ... in the context :).
Edit
Another reason why I think Enums don't violate OCP (now I use the word violate :D), is because OCP doesn't mean that all of the classes and methods can be extended, but rather that a conscientious developer makes informed decisions about where to put those extension points. In the case of an Enum a developer understands that there's no need to extend the possible values and further that it would be damaging to add new values, as existing code wouldn't know what to do with the new value. So he/she decides to completely close the class and not provide extension points. The same applies to any other class that is final. A final class doesn't violate OCP, but rather (or ideally) a developer made a decision of not allowing to extend it.
This is similar to what I mentioned before, that Enums are like a small, finite set of Value Objects. VOs are immutable and closed, and as such they don't violate OCP.
And to add something more, I've seen some libraries not implementing OCP correctly (Tapestry 5) and making more than needed final, which made them a PITA to use. Or just not closing anything, which made some devs make mistakes because they didn't understand the finer details of the library, and messed up the invariants.
The answer is yes. All Java enums violate the Open/Closed Principle, because they cannot be extended without modification.
The "better usage of enums" in Java is to have them implement an interface, and have clients depend on that interface rather than the enum implementation, because the interface does not violate the Open/Closed Principle.
Don't jump to the conclusion that if enums violate the OCP, then we shouldn't be allowed to use them. I want to clarify this answer with a reminder that the OCP, by definition, applies only to client-facing code, i.e. code that is public/exposed. The OCP is not a restriction on implementation details.
A module is said to be closed if it is available for use by other modules... At the implementation level, closure for a module also implies that you may compile it, perhaps store it in a library, and make it available for others (its clients) to use.
--Bertrand Meyer, Object-Oriented Software Construction 2nd ed. page 57
So, utilizing enums as part of an application's internal API avoids the OCP altogether. It is only as part of the external API where enums should be abstracted through an interface.
Consider that since an enum is a concrete implementation, exposing it only though an interface serves as much to satisfy the Dependency Inversion Principle as it does the OCP. Even if you believe that directly exposing enums somehow sidesteps the OCP, it still creates a concrete dependency, which violates the DIP.
Also consider the comment from @MarkRotteveel, "Open/closed is a principle, not a dogma that needs to be followed at all times."