Proof of Neumann Lemma

Let's say that we are dealing with matrices in $\text{Mat}_n(\mathbb{C})$. Then, $\|\cdot\|$ induces a topology on $\text{Mat}_n(\mathbb{C})$ which, by the equivalence of norms, is equivalent to the usual topology and so complete (being homeomorphic to $\mathbb{C}^{n^2}$).

Now, note that since $\displaystyle \|\sum_k A^k\|\leqslant \sum_k \|A\|^k$ and $||A\|<1$ it's easy to show that $\displaystyle \left\{\sum_k A^k\right\}$ is a Cauchy sequence in $\text{Mat}_n(\mathbb{C})$ and so by previous discussion, convergent in $\text{Mat}_n(\mathbb{C})$ to some matrix $\displaystyle \sum_{k=1}^{\infty}A^k$.

Now, prove that $(I-A)\left(I+\cdots+A^k\right)=I-A^{k+1}\quad\mathbf{(1)}$ as a formal algebraic identity.

Since $\|A^k\|\leqslant \|A\|^k\to0$ you have that $\|A^k\|\to0$ and so $A^k\to0$. Thus, taking the limit of both sides of $\mathbf{(1)}$ shows that $\displaystyle \sum_{k=1}^{\infty}A^k$ is an inverse for $A$.

I am adding this as a second answer because it's a fundamentally different approach.

There is a neat generalization to the above which is sometimes useful:

Theorem: Let $S\in\text{GL}(\mathbb{R}^n)$ and $T\in\text{End}(\mathbb{R}^n)$ be such that $\|T-S\|_\text{op}\|S^{-1}\|<1$. Then, $T\in\text{GL}(\mathbb{R}^n)$.

It suffices to show that $\ker T$ is trivial. To this end we observe that

$$\begin{aligned}\frac{\|v\|}{\|S^{-1}\|_\text{op}} &=\frac{1}{\|S^{-1}\|_\text{op}}\|S^{-1}(S(v))\|\\ &\leqslant \frac{\|S^{-1}\|_\text{op}}{\|S^{-1}\|_\text{op}}\|S(v)\|\\ &= \|S(v)\|\\ &\leqslant \|(S-T)(v)\|+\|T(v)\|\\ &\leqslant\|S-T\|_\text{op}\|v||+\|T(v)\|\\ &=\|T-S\|_{\text{op}}\|v\|+\|T(v)\|\end{aligned}$$

And thus we obtain that

$$\left(\frac{1}{\|S^{-1}\|_\text{op}}-\|T-S\|_\text{op}\right)\|v\|\leqslant\|T(v)\|\quad\mathbf{(1)}$$

Thus, if $v\ne0$ then $\|v\|>0$ and by assumption we may then conclude that the left side of $\mathbf{(1)}$ is positive, and so $\|T(v)\|$ is positive. Thus, $T(v)\ne0$ and so $T$ has a trivial kernel.

Now, since all matrix norms enjoy all the properties used in the above proof (submultiplicativeness, etc.) the above works if we replaced $\|\cdot\|_\text{op}$ by any norm. Thus, this proves your case taking $S=I$.