Permutation group product

In general it is not true that $\varphi(x_{\tau(\sigma(1))},...,x_{\tau(\sigma(p))}) = [\sigma(\tau \varphi)](x_{1},...,x_{p})$ -- this only applies if $\tau$ and $\sigma$ commute in $S_p$. Take for example $p=3$, $\tau = (1\ 3), \sigma = (1\ 2\ 3)$. Then $$\sigma\tau \phi = (1\ 2\ 3)(1\ 3)\phi = \phi(x_1,x_3,x_2) $$ $$\tau\sigma \phi = (1\ 3)(1\ 2\ 3)\phi = \phi(x_2, x_1, x_3)$$

What is true is that when evaluating $\sigma\tau\phi$ (for example), you can first apply $\tau$ to $\phi$ and then finally apply $\sigma$, or you can first apply $\sigma$ to $\tau$ and then apply this combined permutation ($\sigma\tau$) to $\phi$.

There is a mistake when you write

$$[(\tau \sigma)\varphi](x_{1},...,x_{p}) = \varphi(x_{\tau(\sigma(1))},...,x_{\tau(\sigma(p))}) = [\sigma(\tau \varphi)](x_{1},...,x_{p})$$

You have

$$\begin{aligned}(\tau \sigma)\varphi(x_{1},...,x_{p}) &= \varphi(x_{(\tau\sigma)(1)},...,x_{(\tau\sigma)(p))})\\ &= \varphi(x_{(\tau\circ \sigma)(1)},...,x_{(\tau\circ \sigma)(p))})\\ &= \varphi(x_{(\tau(\sigma(1))},...,x_{(\tau(\sigma(p))})\\ &= \tau(\varphi(x_{\sigma(1)},...,x_{\sigma(p)})\\ &= \tau(\sigma\varphi(x_1,...,x_p)\\ \end{aligned}$$

Therefore the equality of your book $$(\tau \sigma)\varphi = \tau(\sigma \varphi)$$