Why does the spectral norm equal the largest singular value?

Put $B=A^*A$ which is a Hermitian matrix.

As a linear transformation of Euclidean vector space $E$ is Hermite iff there exists an orthonormal basis of E consisting of all the eigenvectors of $B$

Let $\lambda_1,...,\lambda_n$ be the eigenvalues of $B$ and $\left \{ e_1,...e_n \right \}$ be an orthonormal basis of $E$

Let $x=a_1e_1+...+a_ne_n$

we have $\left \| x \right \|=\left \langle \sum_{i=1}^{n}a_ie_i,\sum_{i=1}^{n}a_ie_i \right \rangle^{1/2} =\sqrt{\sum_{i=1}^{n}a_i^{2}}$,

$Bx=B\left ( \sum_{i=1}^{n}a_ie_i \right )=\sum_{i=1}^{n}a_iB(e_i)=\sum_{i=1}^{n}\lambda_ia_ie_i$

Denote $\lambda_{j_{0}}$ to be the largest eigenvalue of $B$.

Therefore,

$\left \| Ax \right \|=\sqrt{\left \langle Ax,Ax \right \rangle}=\sqrt{\left \langle x,A^*Ax \right \rangle}=\sqrt{\left \langle x,Bx \right \rangle}=\sqrt{\left \langle \sum_{i=1}^{n}a_ie_i,\sum_{i=1}^{n}\lambda_ia_ie_i \right \rangle}=\sqrt{\sum_{i=1}^{n}a_i\overline{\lambda_ia_i}} \leq \underset{1\leq j\leq n}{\max}\sqrt{\left |\lambda_j \right |} \times (\left \| x \right \|)$

So, if $\left \| A \right \|$ = $\max \left\{ \|Ax\| : \|x\| = 1 \right\}$ then $\left \| A \right \|\leq \underset{1\leq j\leq n}\max\sqrt{\left |\lambda_j \right |}$ (1)

Consider: $x_0=e_{j_{0}}$ $\Rightarrow \left \| x_0 \right \|=1$ so that $\left \| A \right \|^2 \geq \left \langle x_0,Bx_0 \right \rangle=\left \langle e_{j_0},B(e_{j_0}) \right \rangle=\left \langle e_{j_0},\lambda_{j_0} e_{j_0} \right \rangle = \lambda_{j_0}$ (2)

Combining (1) and (2) gives us $\left \| A \right \|= \underset{1\leq j\leq n}{\max}\sqrt{\left | \lambda_{j} \right |}$ where $\lambda_j$ is the eigenvalue of $B=A^*A$

Conclusion: $$\left \| A \right \| _2=\sqrt{\lambda_{\text{max}}(A^{^*}A)}=\sigma_{\text{max}}(A)$$

First of all, $$\begin{align*}\sup_{\|x\|_2 =1}\|Ax\|_2 & = \sup_{\|x\|_2 =1}\|U\Sigma V^Tx\|_2 = \sup_{\|x\|_2 =1}\|\Sigma V^Tx\|_2\end{align*}$$ since $U$ is unitary, that is, $\|Ux_0\|_2^2 = x_0^TU^TUx_0 = x_0^Tx_0 = \|x_0\|_2^2$, for some vector $x_0$.

Then let $y = V^Tx$. By the same argument above, $\|y\|_2 = \|V^Tx\|_2 = \|x\|_2 = 1$ since $V$ is unitary. $$\sup_{\|x\|_2 =1}\|\Sigma V^Tx\|_2 = \sup_{\|y\|_2 =1}\|\Sigma y\|_2$$ Since $\Sigma = \mbox{diag}(\sigma_1, \cdots, \sigma_n)$, where $\sigma_1$ is the largest singular value. The max for the above, $\sigma_1$, is attained when $y = (1,\cdots,0)^T$. You can find the max by, for example, solving the above using a Lagrange Multiplier.