Closing parenthesis as delimiter not matched when breaklines=true

You might have come across a bug arising from the special treatment of the right parenthesis by the line-breaking algorithm of the listings package. In fact, when the SelectCharTable hook used by the listings package is executed and line breaking is enabled ) is defined as a macro, namely by

\lst@Def{`)}{\lst@breakProcessOther)}

where

\lst@Def=macro:
#1->\lccode `\~=#1\lowercase {\def ~}.

Unfortunately, this means that ) is not recognized as a delimiter any more, which eventually leads to your observation.

I’ve put some effort in finding a fix for this bug, but haven’t succeeded so far. (My TeX programming skills are still limited.) So, the only thing I can provide right now is a workaround in the form of a patch that simply cancels the special treatment of ) by the line-breaking algorithm:

\documentclass{article}

\usepackage{color}

\usepackage{listings}

\lstdefinelanguage{new}{
  moredelim=*[s][\color{red}]{(}{)},
  moredelim=*[s][\color{green}]{[}{]},
}

\lstset{
  language=new,
  breaklines=true
}

\usepackage{etoolbox}

\makeatletter

\patchcmd{\lsthk@SelectCharTable}{%
  \lst@ifbreaklines\lst@Def{`)}{\lst@breakProcessOther)}\fi
}{%
}{
}{
}

\makeatother

\begin{document}

\begin{lstlisting}
outside (in paren) [in brackets (paren+brackets) brackets again] outside again
\end{lstlisting}

\end{document}

The resulting output is:

A simple workaround would be, instead of using closing parenthesis ')', one can use another character e.g ']' and replace it with closing parenthesis ')'.

\begin{lstlisting[literate={(}{{\textbf{(}}}1                          
                           {]}{{\textbf{)}}}1]
(In text I'm closing using]
\end{lstlisting}