Formal way to evaluate this limit

No, you can't. However, you may note that, as $n\to +\infty$, $$n^2\ln\left(1-\frac{1}{n}\right)=n\ln\left(\left(1-\frac{1}{n}\right)^n\right)\to +\infty \cdot \ln(e^{-1})=-\infty.$$ Hence $$\lim_{n\to\ +\infty}{\left(1-\frac{1}{n}\right)^{n^2}}=\lim_{n\to\ +\infty}{e^{n^2\ln\left(1-\frac{1}{n}\right)}}=e^{-\infty}=0.$$ Another way. Since $\lim_{n\to\ +\infty}{\left(1-\frac{1}{n}\right)^{n}}=e^{-1}\in (0,1/2)$ then, by definition of limit, there is $N$ such that for all $n\geq N$, $\left(1-\frac{1}{n}\right)^{n}<1/2$. Hence, for $n\geq N$, $$0<{\left(1-\frac{1}{n}\right)^{n^2}}=\left(\left(1-\frac{1}{n}\right)^{n}\right)^n<\frac{1}{2^n}.$$ Then use the Squeeze theorem.

One way of doing it is by comparison. For any fixed $k\in \Bbb N$, we have $$ 0\leq \lim_{n\to \infty}\left(1-\frac 1n\right)^{n^2}\leq\lim_{n\to \infty}\left(1-\frac1n\right)^{nk} = e^{-k} $$