Drawing Markov Decision Processes with Tikz

_{Time spent on learning something will never go a waste!}

As suggested in the comments, you have to use angles and quotes library. Let us say you have 3 points A, B and C. Then the angle ABC can be drawn like this:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{positioning,angles,quotes}
\begin{document}
  \begin{tikzpicture}
    \coordinate (B);
    \coordinate (A) at (3,3);
    \coordinate (C) at (3,0);
    \draw (B) -- (C) (B) -- (A);
    \path pic[draw, angle radius=6mm,"a",angle eccentricity=1.2] {angle = C--B--A};
  \end{tikzpicture}
\end{document}

Here angle radius is the radius of the arc and angle eccentricity is the radial distance of the label from the arc (the value 1 is on the arc)

Applying this to your code:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{positioning,angles,quotes}
\begin{document}
  \begin{tikzpicture}[auto,node distance=8mm,>=latex,font=\small]

    \tikzstyle{round}=[thick,draw=black,circle]

    \node[round] (s0) {$s_0$};
    \node[round,above right=0mm and 20mm of s0] (s1) {$s_1$};
    \node[round,below right=0mm and 20mm of s0] (s2) {$s_2$};

    \draw[->] (s0) -- (s1);
    \draw[->] (s0) -- (s2);
    \draw[->] (s0) [out=40,in=100,loop] to coordinate[pos=0.1](aa) (s0);
    \path pic[draw, angle radius=6mm,"\vphantom{g}a",angle eccentricity=1.2] {angle = s2--s0--aa};
\end{tikzpicture}
\end{document}

angle radius and angle eccentricity need to be carefully adjusted here since one of the lines is a curve.