Can someone explain the algorithm for composition of cycles?

When you read a composition of functions written in the usual notation for permutations, you must remember to read them from right to left. Thus, when you try to compute the composition you must start by looking successively at what does each permutation in the composition do to each integer from $1$ to $5$ (in this case), but from right to left.

So you can start by writing $$\pi \circ \tau \circ \sigma = \begin{pmatrix} 2 &3 &4 \end{pmatrix} \circ \begin{pmatrix} 2 &4 &5 \end{pmatrix} \circ \begin{pmatrix} 1 &3 \end{pmatrix}$$

So you start with $1$. The rightmost permutation sends $1 \mapsto 3 $ and then the middle permutation sends $3 \mapsto 3$ and finally the leftmost permutation sends $3 \mapsto 4$. So in the end the total result is that the composition of the three of them sends $1 \mapsto 4$. In the same way you'll proceed for the other integers and in the end you'll get the answer

$$ \pi \circ \tau \circ \sigma = \begin{pmatrix} 1 &2 &3 &4 &5 \\ 4 &2 &1 &5 &3 \end{pmatrix}$$ or in cycle notation $\pi \circ \tau \circ \sigma = \begin{pmatrix} 1 &4 &5 &3\end{pmatrix}$.

I checked doing the computation from left to right and the "answer" I got from doing that was $\pi \circ \tau \circ \sigma = \begin{pmatrix} 1 &3 &5 &2\end{pmatrix}$ which is what you originally had as your answer.

You're probably composing from the left to the right, instead of from the right to the left.