Usage of the word "formal(ly)"

As André says in the comments, there is a second meaning of "formally" which means roughly "symbolically." For example when we talk about formal power series we ignore issues of convergence and work only with the symbolic form of various power series. That is, we only look at the form of the things we're manipulating rather than thinking particularly hard about what exactly they are, what spaces they live in, etc.

Regarding your edit, "formal" is more specific than a lack of rigor. It specifically means paying attention to form. Many formal manipulations can be made rigorous but one has to do extra work to do so, and after one has made them rigorous one often finds that there was essentially nothing wrong with the formal manipulations. For example, the Dirac delta function can be treated formally, but it can also be treated rigorously using the theory of distributions. But engineers and physicists didn't need distributions to use the Dirac delta to solve differential equations. In this sense "formal" is related to the notion of moral truth in mathematics.