Equivalent definitions of total angular momentum

Consider a smooth function $\psi$ in $L^2(\mathbb{R}^3, d^3x)$ (more precisely $\psi$ is assumed to belong to Schwartz' space) and define for a fixed $n \in \mathbb{S}^2$ and $\phi \in \mathbb{R}$ $$\psi_\phi(x)= \psi(R^{-1}_n(\phi)x)$$ Since $R^{-1}_n(\phi) = e^{-\phi n \cdot S}$, where $S= (S_x,S_y,S_z)$ are the three generators of $SO(3)$: $$(S_j)_{rs}= \epsilon_{jrs}$$ we have $$\frac{d}{d\phi}\psi_\phi(x) = -\sum_{j,r,s=1}^3 n_j \epsilon_{jrs}x_r\frac{\partial }{\partial x_s}\psi_\phi(x)\:.\tag{1}$$ Now consider $$\psi'_\phi(x)= \left(e^{\frac{i}{\hbar}\phi n \cdot J}\psi\right)(x)\:,$$ where, omitting the spin part of $J$ since as far as I unserstand, you are interested in the spatial part of the state only, $$J_j = \sum_{r,s=1}^3\epsilon_{jrs} X_r P_s = -i\hbar \sum_{r,s=1}^3\epsilon_{jrs} X_r \frac{\partial}{\partial x_s} $$ Computing the $\phi$-derivative (using Stone's theorem and some careful analysis about the use of different topologies. I do not want to enter into the details here, I just say that here smoothness of $\psi$ matters), we have $$\frac{d}{d\phi}\psi'_\phi(x) = \frac{i}{\hbar}\sum_{j=1}^3 n \cdot \left(J_j\psi'_\phi\right)(x)= -\sum_{j,r,s=1}^3 n_j \epsilon_{jrs}x_r\frac{\partial }{\partial x_s}\psi'_\phi(x)\:.\tag{2}$$ In summary, for $x$ fixed, $\psi_\phi(x)$ and $\psi'_\phi(x)$ satisfy the same first order differential equation (in normal form with smooth known term) and furthermore they satisfy the same initial condition $$\psi_0(x)=\psi'_0 (x)\:.$$ The uniqueness theorem for the solutions of first-order differential equations implies that $$\psi_\phi(x)=\psi'_\phi (x)\:.$$ In other words $$\left(e^{-\frac{i}{\hbar} \phi n \cdot J}\psi\right)(x) = \psi\left(R_n^{-1}(\phi)x\right)\:.$$ (The result extends to the whole $L^2$ space made als of non-smooth functions exploiting the fact that Schwartz' space is dense therein.) The final step is quite formal, but it can be made rigorous adopting the theory of rigged Hilbert spaces to make rigorous the bra-ket notation for improper eigenvectors of the position operators. In position representation $|y\rangle = \delta(x-y)= \delta_y(x)$, so that, using rotational invariance of the Dirac delta function, $$\left(e^{-\frac{i}{\hbar} \phi n \cdot J}\delta_y\right)(x) = \delta_y\left(R_n^{-1}(\phi)x\right)= \delta\left(R_n^{-1}(\phi)x -y\right)= \delta\left(x -R_n(\phi)y\right)\:.$$ Coming back to the abstract notation, the found identity reads $$e^{-\frac{i}{\hbar} \phi n \cdot J}|y\rangle = |R_n(\phi)y\rangle\:.$$

(A completely rigorous proof can be found in this book of mine.)

The most broad definition of the angular momentum operator ${\bf J}$ is the operator that generates rotation, so the equality you want to prove is just the definiton (assumin that the particle we consider does not have any internal degrees of freedom, which could include spin) and as such does not require a proof.

But what would then needs a proof is the relation between the angular momentum operator and the position operator $$ {\bf J} = {\bf X} \times {\bf P}$$ Let's see that this is satisfied for ${\bf J}_3$, i.e. $$ {\bf J}_3 = {\bf X}_1{\bf P}_2 - {\bf X}_2{\bf P}_1$$ (other components will be analogous).

Let us consider an rotation acting on a state $|\psi\rangle = \int \psi(x) |x\rangle dx$. $$ \exp(- \frac{i}{\hbar}\phi\, {\bf J}_3) \int \psi(x)|x\rangle dx = \int \psi(x)|R(\phi)x\rangle dx = \int \psi(R^{-1}(\phi)x) |x\rangle$$ That means that $$ {\bf J}_3 |\psi\rangle = i\hbar \left.\frac{d}{d\phi}\right|_{\phi=0} \exp(- \frac{i}{\hbar}\phi\, {\bf J}_3) \int \psi(x)|x\rangle dx = i\hbar \int \left.\frac{d}{d\phi}\right|_{\phi=0} \psi(R^{-1}(\phi)x) |x\rangle$$

We have $$\psi(R^{-1}(\phi)x) = \psi(x_1 \cos\phi + x_2 \sin\phi,-x_1\sin\phi+x_2\cos\phi,x_3) $$ so $$ \left.\frac{d}{d\phi}\right|_{\phi=0} \psi(R^{-1}(\phi)x) = x_2 \frac{\partial \psi(x)}{\partial x_1} - x_1 \frac{\partial \psi(x)}{\partial x_2}$$ so \begin{align} {\bf J}_3|\psi\rangle &= i\hbar \int \Big(x_2 \frac{\partial \psi(x)}{\partial x_1} - x_1 \frac{\partial \psi(x)}{\partial x_2}\Big) |x\rangle = \\ &= -{\bf X}_2 \int \Big(-i\hbar\frac{\partial \psi(x)}{\partial x_1}\Big) |x\rangle dx + {\bf X}_1 \int \Big(-i\hbar\frac{\partial \psi(x)}{\partial x_2}\Big) |x\rangle = \\ &= -{\bf X}_2 {\bf P}_1 \int \psi(x) |x\rangle dx + {\bf X}_1 {\bf P_2} \int \psi(x) |x\rangle = \\ &= ({\bf X}_1{\bf P}_2 - {\bf X}_2{\bf P}_1)|\psi\rangle \end{align} so $$ {\bf J}_3 = {\bf X}_1{\bf P}_2 - {\bf X}_2{\bf P}_1$$

If a particle has internal degrees of freedom, then $$ {\bf J} = {\bf X} \times {\bf P} + {\bf S}$$ where ${\bf S}$ is the spin operator acting on these internal degrees of freedom.