Macro: Expand differently depending on pattern?

UPDATE Embrace the power of expl3 and xparse. I choose the delimiter ; to make the macro possible. To be honest v.2: it was quite simple and I totally lied earlier! This is the new macro

\ExplSyntaxOn
\tl_new:N \kdb_temp
\DeclareDocumentCommand{\BrKt}{u;}%
{
    \left.
    \tl_set:Nn \kdb_temp {#1}
    \tl_replace_all:Nnn \kdb_temp{<}{\middle\langle}
    \tl_replace_all:Nnn \kdb_temp{|}{\middle|}
    \tl_replace_all:Nnn \kdb_temp{>}{\middle\rangle}
    \tl_use:N \kdb_temp
    \right.
}
\ExplSyntaxOff

Look how beautiful it is!

You can use the macro as follows: \BrKt<j|\otimes<k|e^{a^\dagger/\hbar}|n>\otimes|m>;, \BrKt|0>|1>|0>|1> = |3>_4; or \BrKt|m>\equiv<\Psi|A|B|\varphi>|n>;$. This allows a much greater variety then originally intended.

---

Old Post To be honest: I don't think you can't perfectly achieve what you want with little effort. It would be possible though. But if you stick to the basics you could use the power of xparse. I worked out the start

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn
\DeclareDocumentCommand{\m}{t< u{|} u>}%
{
    \IfBooleanTF{#1}{}{\GenericWarning{}{Watch out! A missing "<" encountered!}}
    \if\relax\detokenize{#2}\relax
        \if\relax\detokenize{#3}\relax
%           \langle\rangle
        \else
            \left| #3\right\rangle
        \fi
    \else
        \if\relax\detokenize{#3}\relax
            \left\langle #2\right|
        \else
            \left\langle #2 \middle| #3\right\rangle
        \fi
    \fi
}
\ExplSyntaxOff

This command structure strictly demands input of the form \m<input1|input2>, however it checks whether input1 or input2 is empty and processes the input accordingly. But note, this cannot create something like <\Psi\Phi>, without the pipe in the middle. Also note, in this realization the opening < is not mandatory and will only produce a warning if it's missing. I hope you can work with this and go on further.

This is somewhat possible with the suffix package:

\documentclass{article}
\usepackage{suffix}
\begin{document}
\WithSuffix\def\m<#1|{\left\langle #1 \right|}
\WithSuffix\def\m|#1>{\left|       #1 \right\rangle}
\[ \m<x| \quad \m|y>  \]
\end{document}

However, there's a significant limitation with this approach, in that the same "suffix" can only be used once, so your proposed \m<#1> syntax cannot be supported by this package as well as \m<#1|. This no doubt makes the approach a non-starter, but I thought it would be good to add this answer for completeness.

With expl3 the proposed syntax \m{<x|y>}.

\documentclass{article}

\usepackage{mathtools,xparse}
\usepackage{mleftright}

\ExplSyntaxOn
\NewDocumentCommand \m { m } { \kdb_m:n {\begm#1\endm} }
\cs_new_protected:Npn \kdb_m:n #1
 {
  \group_begin:
   \tl_set:Nn \l_tmpa_tl {#1}
   \tl_replace_once:Nnn \l_tmpa_tl { \begm< } { \mleft\langle  }
   \tl_replace_once:Nnn \l_tmpa_tl { \begm| } { \mleft\lvert   }
   \tl_replace_once:Nnn \l_tmpa_tl { >\endm } { \mright\rangle }
   \tl_replace_once:Nnn \l_tmpa_tl { |\endm } { \mright\rvert  }
   \tl_replace_all:Nnn \l_tmpa_tl { | } { \:\middle\vert\: }
   \tl_use:N \l_tmpa_tl
  \group_end:
 }
\ExplSyntaxOff

\begin{document}

\[
  \m{<x>} \quad \m{<x|} \quad \m{|x>} \quad \m{<x|y|z>} \quad \m{<x^{2^{2^{2^{2^{2^2}}}}}|y>}
\]

\end{document}

And with plain LaTeX and a bit different syntax \m<x|>.

\documentclass{article}

\usepackage{mathtools}
\usepackage{mleftright}

\makeatletter
\def\activevert{\@ifnextchar\mlast{\mright\rvert\@gobble}{\:\middle\vert\:}}
{\catcode`\|=\active\gdef|{\activevert}}
\gdef\m<#1>{\begingroup\mathcode`\|="8000
   \@ifnextchar|{\mleft\lvert\@gobble}{\mleft\langle}#1\mlast\endgroup}
\def\mlast{\mright\rangle}
\makeatother

\begin{document}

\[
  \m<x> \quad \m<x|> \quad \m<|x> \quad \m<x|y|z> \quad \m<x^{2^{2^{2^{2^{2^2}}}}}|y>
\]

\end{document}

PS: instead of \: the usual thing is to use \; but they look too big to me, you could use \nonscript\muskip5mu or whatever you want instead.