Proper way to find change-of-basis matrix

We have that$M_{\beta'}^{\beta}(\text{id})=(M_S^{\beta'}(\text{id}))^{-1}M_S^{\beta}(\text{id})$.

But $M_S^{\beta}(\text{id})=\begin{pmatrix}1&-1&0\\1&1&1\\0&1&2\end{pmatrix}$ and $M_S^{\beta'}(\text{id})=\begin{pmatrix}2&0&-1\\1&0&1\\1&1&1\end{pmatrix}$.

I'll leave it to you to compute the inverse and multiply.

I get

$M_{\beta'}^{\beta}(\text{id})=\begin {pmatrix} \frac23&0&\frac13 \\-1&0&1\\\frac13&1&\frac23\end{pmatrix}$.

My go-to solution for almost all linear algebra is to write everything as a bunch of matrix(-vector) products. Let's try that.

The coordinates of some vector $\mathbf u$ with respect to some basis $\beta$ is a column vector $\mathbf v$ such that $\mathbf{M}_{\beta} \mathbf{v} = \mathbf{u}$. Likewise the coordinates with respect to some other basis $\beta'$ is a vector $\mathbf{w}$ such that $\mathbf{M}_{\beta'} \mathbf{w} = \mathbf{u}$.

Thus $\mathbf{M}_{\beta'} \mathbf{w} = \mathbf{u} = \mathbf{M}_{\beta} \mathbf{v}$ and hence $\mathbf{w} = (\mathbf{M}_{\beta'}^{-1} \mathbf{M}_{\beta}) \mathbf{v}$. Evidently the matrix $\mathbf{M}_{\beta'}^{-1} \mathbf{M}_{\beta}$ maps $\beta$-coordinates into $\beta'$-coordinates, and so I'd calculate that.

If I'm not mistaken, basis-conversion of a linear mapping means multiplying the basis-conversion matrix and the linear mapping matrix together. Presumably the particular identity mapping is the one on $\mathbb{R}^3$, represented by the 3-by-3 identity matrix.

I find it hard to decipher your notation, but it appears that you’re making a few fundamental errors.

It’s helpful to think of the notation $M_{\beta'}^\beta$ as specifying the “input“ and “output” bases of the matrix $M$: it eats coordinate tuples expressed relative to the ordered basis $\beta$ and spits out coordinate tuples expressed relative to the ordered basis $\beta'$. In particular, applying it to tuples of coordinates expressed in some other basis is nonsensical, as is interpreting its output in terms of some basis other than $\beta'$.

Now, given $\beta=(v_1,v_2,v_3)$, then it’s certainly true that if $X_\beta(v)=(x_1,x_2,x_3)^T$, then $v=x_1v_1+x_2v_2+x_3v_3$, but that’s just the definition of the coordinates of $v$ relative to $\beta$. However, it makes no sense in principle to multiply this sum by the matrix $A=M_{\beta'}^\beta(\operatorname{id})$: $v$ might not even be an element of $\mathbb R^3$ in the first place. In this exercise it is, which I think contributes to your confusion. Even though $v\in\mathbb R^3$, it still makes no sense to multiply it by $A$ because you’re representing the elements of $\beta$ as coordinate vectors relative to the standard basis $\mathcal E$ (or at least some other, unspecified, basis). Using Lang’s notation, that sum gives you $X_{\mathcal E}(v)$, but the product $$M_{\beta'}^\beta(\operatorname{id})X_{\mathcal E}(v)$$ is nonsensical because the bases don’t match.

The next problem is that the same coordinates $(x_1,x_2,x_3)^T$ appear on both sides of the equation that you’ve formed. That’s tantamount to saying that $X_{\beta'}(v)=X_{\beta}(v)$, that is, that the coordinates of an arbitrary vector $v$ are the same in both bases. That’s quite obviously false if $\beta'\ne\beta$. The left-hand side must use the $\beta'$-coordinates of $v$, which are some other three values $(x_1',x_2',x_3')^T$. The number of unknowns is proliferating quickly.

Going back to the definition of $M_{\beta'}^{\beta}$, what we want here is a matrix $A$ such that $$X_{\beta'}(v_i) = AX_{\beta}(v_i)$$ for every element $v_i$ of $\beta$. However, $X_{\beta}(v_i)=e_i$ and $Ae_i=A_i$, from which $A_i=X_{\beta'}(v_i)$, i.e., the columns of $A$ are the elements of $\beta$ expressed relative to the basis $\beta' = (w_1,w_2,w_3)$. For each column, then, you have a system of linear equations. The nine equations can be expressed as the matrix equation $$\pmatrix{X_{\mathcal E}(w_1)&X_{\mathcal E}(w_2)&X_{\mathcal E}(w_3)}A=\pmatrix{X_{\mathcal E}(v_1)&X_{\mathcal E}(v_2)&X_{\mathcal E}(v_3)},$$ therefore $$A = \pmatrix{X_{\mathcal E}(w_1)&X_{\mathcal E}(w_2)&X_{\mathcal E}(w_3)}^{-1}\pmatrix{X_{\mathcal E}(v_1)&X_{\mathcal E}(v_2)&X_{\mathcal E}(v_3)}.$$ Observe, though, that the first matrix in this product is $M_{\beta'}^{\mathcal E}(\operatorname{id})$ and the second is $M_{\mathcal E}^{\beta}(\operatorname{id})$, so we have the useful identity $$M_{\beta'}^\beta(\operatorname{id}) = M_{\beta'}^{\mathcal E}(\operatorname{id})M_{\mathcal E}^{\beta}(\operatorname{id}) = M_{\mathcal E}^{\beta'}(\operatorname{id})^{-1}M_{\mathcal E}^{\beta}(\operatorname{id}).$$ Formally, the upper and lower $\mathcal E$’s in the product “cancel.”

A fairly convenient way to compute this product by hand is to form the augmented matrix $$\left(\begin{array}{c|c}M_{\mathcal E}^{\beta'}(\operatorname{id}) & M_{\mathcal E}^{\beta}(\operatorname{id}) \end{array}\right) = \left(\begin{array}{ccc|ccc}X_{\mathcal E}(w_1)&X_{\mathcal E}(w_2)&X_{\mathcal E}(w_3)&X_{\mathcal E}(v_1)&X_{\mathcal E}(v_2)&X_{\mathcal E}(v_3)\end{array}\right)$$ and apply Gaussian elimination to it to obtain $$\left(\begin{array}{c|c}I_3 & M_{\mathcal E}^{\beta'}(\operatorname{id})^{-1} M_{\mathcal E}^{\beta}(\operatorname{id}) \end{array}\right).$$