The difference between applying a rotation matrix to a vector (points) and to a matrix (transformation)

Here the matrix $U$ is considered not as a bunch of column vectors, but as a (matrix of the) linear map $F\colon {\Bbb R}^n\to {\Bbb R}^n$ $$ y=F(x)=Ux. $$ What happens if we rotate both $y$ and $x$ by $R$? We get (since $R^TR=I$ for rotations) $$ y=Ux\quad\Rightarrow\quad Ry=RUx\quad\Rightarrow\quad Ry=\underbrace{RUR^T}_{U'}Rx\quad\Rightarrow\quad y'=U'x'. $$ Thus the matrix $U'=RUR^T$ corresponds to the same linear map $F$ in the new coordinates after rotation ($x'\mapsto y'$).

In general, for any change of the basis $x'=Sx$, $y'=Sy$ the corresponding change of the matrix $U$ is $$ Sy=\underbrace{SUS^{-1}}_{U'}Sx\quad\Rightarrow\quad y'=U'x'. $$ It means that the class of all similar matrices $\{SUS^{-1}\colon S\text{ invertible}\}$ is exactly the class of all matrices that describe the same linear map in different bases.

Using your example where $U$ is a stretch matrix in 3D, if you want to "rotate" this matrix, you essentially want this stretch action to occur in a different direction / axis. Suppose you have some shape aligned to this new axis. You want to know what the $U'$ is that stretches the shape parallel to this axis. To do this, you use $R^T$ to rotate everything back to the original orientation. Then you do the original stretch transformation $U$. Then you rotate this back using $R$. So $U'=RUR^T$.

One thing that may be instructive is to recall that every matrix can be represented as the linear combination of a series of dyadic/outer products between two vectors, $U = \sum_i a_i \otimes b_i = \sum_i a_i b_i^T$ where $a_i$ and $b_i$ are a sequence of column vectors.

When changing the basis of the matrix, we are in effect applying the vector rule for changing bases to both sequences of vectors:

$$U’ = \sum_i a_i’ \otimes b_i’ = \sum_i Ra_i (Rb_i)^T = \sum_iRa_ib_i^T R^T = RUR^T$$

Hope this helps!