What determines whether or not $|jm\rangle$ can be rotated into $|jm'\rangle$?

This isn’t a full answer, but an easy way to see you can’t reach every state with a rotation is that the set of rotations has three real dimensions, while the set of normalized spin $s$ states has real dimensions $2(2s+1) - 2 = 4s$ real dimensions, where the subtraction is due to tossing out a global phase and demanding normalization. This means that for spin $s=1$ and higher, almost all states can’t be reached by a rotation, just on dimensional grounds. The classical intuition doesn’t work, because a “classical” spin always has a two-dimensional state space.

Are there any arguments on physical grounds we could have made to arrive at the above conclusion without going through the math?

Yes. This answer uses some mathematial notation, but it doesn't use any matrices. It uses only geometric intuition and the most basic general principles of quantum physics.

Notation

Consider an irreducible representation of the spin group (the double cover of the rotation group), and use this notation:

• $\hat u$ is a unit vector in 3d space.

• $J(\hat u)$ is the generator of rotations about the $\hat u$-axis. For every axis $\hat u$, the generator $J(\hat u)$ is an observable represented by a self-adjoint operator on the Hilbert space.

• $j$ is the largest eigenvalue of $J(\hat u)$ in the given irreducible representation.

• $\big|\hat u\big\rangle$ is the eigenstate of $J(\hat u)$ with the largest eigenvalue $j$: $$ J(\hat u)\,\big|\hat u\big\rangle = j\,\big|\hat u\big\rangle. $$

We can think of $|\hat u\rangle$ as having an angular momentum with a specific orietnation in 3d space, namely along the $\hat u$-axis, but that interpretation isn't necessary for this answer to work.

Intuition

Rotations affect observables in the obvious way: applying a rotation $\hat u\to R\hat u$ transforms $J(\hat u)\to J(R\hat u)$. This immediately implies two things:

• All of the states $|\hat u\rangle$, for all directions $\hat u$, can be obtained from each other by rotations, up to a physically irrelevant overall phase factor. (A state can be represented by a vector in the Hilbert space, but the representation is not one-to-one: vectors related to each other by an overall nonzero complex constant represent the same state.)

• For any given direction $\hat u$, the only states that can be obtained by rotating $|\hat u\rangle$ are largest-eigenvalue eigenstates of the generators $J(R\hat u)$.

Rotating clockwise about $\hat u$ is the same as rotating counterclockwise about $-\hat u$, so the eigenstate of $J(\hat u)$ with eigenvalue $j$ corresponds to the eigenstate of $J(-\hat u)$ eigenvalue $-j$. This implies:

• The eigenstate of $J(\hat u)$ with eigenvalue $-j$ is a rotation of the eigenstate of $J(\hat u)$ with eigenvalue $j$.

The case $j=1/2$ is special, because then $J(\hat u)$ doesn't have any other eigenstates: the only two eigenstates it has are the ones with eigenvalues $\pm j$.

Now consider a representation with $j>1/2$, and consider an eigenstate $|m\rangle$ of $J(\hat u)$ with eigenvalue $m\neq \pm j$. The operator $J(\hat u)$ is the generator of rotations about the $\hat u$ axis, so $|m\rangle$ must be invariant under rotations about the $\hat u$-axis, up to a physically meaningless overall phase. On the other hand, if $|m\rangle$ could be obtained from $|\hat u\rangle$ by a rotation, then $|m\rangle$ would be an eigenstate of $J(R\hat u)$ for some other direction $R\hat u$ with the largest eigenvalue $j$. But $J(R\hat u)$ is not invariant under rotations about the $\hat u$-axis unless $R\hat u=\pm\hat u$, so that would contradict the fact that $|m\rangle$ must be invariant under rotations about the $\hat u$-axis. Altogether, this implies

• The eigenstate of $J(\hat u)$ with eigenvalue $-j$ is a rotation of the eigenstate of $J(\hat u)$ with eigenvalue $j$, and the other eigenstates of $J(\hat u)$ cannot be obtained by rotating the one with eigenvalue $j$.

By the way

The answer above focused on eigenstates of $J(\hat u)$, but as knzhou's answer pointed out, we can also make a more general statement: in a representation with $j>1/2$, most states are not equal (or proportional) to $|\hat u\rangle$ for any direction $\hat u$. Every state can be expressed as a superposition of these, but in most cases the superposition needs to involve more than one direction in 3d space. Furthermore, the superposition is not unique: the same state may be expressed as different superpositions, involving different sets of directions in 3d space. This follows simply from the fact that the basis $\big\{|\hat u\rangle\big\}$ is overcomplete, which in turn follows from the fact that the parameter $\hat u$ is continuous: an irreducible representation has a basis with a finite number of vectors, so the basis parameterized continuously by $\hat u$ must be overcomplete.