when [:punct:] is too much

You can use [:punct:], when you combine it with a negative lookahead

(?!['-])[[:punct:]]

This way a [:punct:]is only matched, if it is not in ['-]. The negative lookahead assertion (?!['-]) ensures this condition. It failes when the next character is a ' or a - and then the complete expression fails.

Inside a character class you only need to escape the closing square bracket:

Try using '[[\\]]' or '[[\]]' (I am not sure about escaping the backslash as I don't know R.)

See this example.