:hover but :not on a specific class

You have the option of using the not() selector.

a:not(.active):hover { ... }

However, this may not work in all browsers, as not all browsers implement CSS3 features.

If you are targeting a large audience and want to support older browsers, the best way would be to define a style for the .active:hover and undo whatever you're doing in a:hover.

The functional notation is on :not(), not :hover:

a:not(.active):hover

If you prefer to put :hover first, that's fine:

a:hover:not(.active)

It doesn't matter which pseudo-class comes first or last; either way, the selector works the same. It just happens to be my personal convention to put :hover last as I tend to place user-interaction pseudo-classes behind structural pseudo-classes.