Latex vs Groff for mathematics formatting
Equations in roff
roff (or groff or any other implementation) doesn't actually support mathematics typesetting. To this end you have to use a preprocessor which transforms equations in your document into something that groff can read. I'm only aware of the eqn preprocessor for equations but there might be others.
One obvious downside of eqn (or groff in general) is that most commands have to appear on a line by themselves, which is very disruptive when writing. That's not really a problem for displayed equations but becomes very annoying for inline math.
LaTeX vs groff
Also the output quality of groff with eqn is not on par with LaTeX (neither is anything else in my opinion).
On top of that LaTeX comes with a lot more features than groff, such as automatic cross-referencing, bibliography management, and multilingual support out-of-the-box. On top of that it is extended by great macro packages, such as amsmath for maths, siunitx for units, beamer for presentation slides, tikz for inline vector graphics, microtype for microtypographical features, and many more.
Other “features” of LaTeX are that scientific journals usually accept submissions in LaTeX and your supervisor might know LaTeX, so you can write your manuscript together. A lot of software can import and export formulas in LaTeX syntax, including Mathematica and even MS Word!
Furthermore, the LaTeX community is much larger than for groff (does it even exist?) and you can enjoy corporate support in the form of online editors such as Overleaf.
I could probably keep going as to why LaTeX is superior to any other non-commercial typesetting system (to any commercial one as well probably). Before going on to comparing output quality, let me answer your two specific questions.
Specific questions
Is it any good for writing maths?
No, the output quality is not on par with LaTeX and the equation syntax (in my opinion) is unintuitive and convoluted. For example spaces matter in some situations, but not in other, sometimes you need to wrap things in {} or "".
Or is it limited like Markdown?
Not really. While Markdown is really a flat static text format, groff at least supports macros but they don't seem to be as powerful as TeX macros (which are famously Turing-complete).
Output quality
LaTeX
\[ s = \sqrt{\frac{\sum_{i=1}^N (x_i - \bar{x})^2}{N - 1}} \]
groff with eqn
.EQ
s = sqrt { { sum from i=1 to N ( x sub i - x bar ) sup 2 } over { N - 1 } }
.EN
Process with
groff -e test.tr > test.pdf
TeX is also used much more frequently than roff or eqn, including on websites and message boards. For example, the math sites here on SX use MathJax, which is based on LaTeX. If you ever want to publish a paper, you’ll also need to know it.