Smart parentheses?

A package? The following simple macro does it:

\documentclass{article}

\newcount\smcount
\def\smart#1{\ifcase\smcount(\or[\or\{\else TOO DEEP!\fi%
\advance\smcount by1 #1\ifcase\smcount\or)\or]\or\}\else TOO DEEP!\fi%
\advance\smcount by-1 }

\begin{document}

\smart{Ala \smart{ma \smart {kota}}}

\end{document}

(Or, if you wish, not \smart but \enparent). Usage:

\smart{Ala \smart{ma \smart {kota}}}

For comparison I am posting a ConTeXt solution.

Such a macro already exists, albeit for quotes. One of the good things about ConTeXt is that a feature is never defined in a one-off basis. For example, instead of defining a macro for quotations that changes the quote symbol depending on the level of nesting, ConTeXt defines a generic delimitedtext mechanism and quotation is a special case of delimited text. The desired parenthesis macro is also a delimited text where the left and right symbol change depending on the level of nesting. So, use delimited text to define this macro as follows:

\definedelimitedtext[parenthesis]   [location=text]
\setupdelimitedtext [parenthesis:1] [left={(}, right={)}]
\setupdelimitedtext [parenthesis:2] [left={[}, right={]}]
\setupdelimitedtext [parenthesis:3] [left={\{},right={\}}]

This can be used as

\starttext
\parenthesis{My outer layer \parenthesis{my inner layer \parenthesis{my innermost layer}}}
\stoptext

which gives

Here is an approach which is somewhat more modular than alexurba's approach, which accomodates indefinite levels of nesting.

\makeatletter
  \def\@enparen#1{\bgroup\let\enparen\@@enparen(#1)\egroup}
  \def\@@enparen#1{\bgroup\let\enparen\@@@enparen[#1]\egroup}
  \def\@@@enparen#1{\bgroup\let\enparen\@enparen\{#1\}\egroup}
  \let\enparen\@enparen
\makeatother

Sample

\enparen{one \enparen{two \enparen{three \enparen{four} levels} levels} level}

Result