What are these mathematical groups in U.S. universities?

Mathematical research groups are, as you seem to have surmised, just groups of faculty members and students (i.e., graduate students, postdocs, tenure track and tenured professors) whose research interests all involve some common subfield of mathematics. Thus a number theory group will be composed of number theorists and perhaps faculty members whose main interest is in some other area of math (e.g., combinatorics) but involves number theory in a substantial way. Continuing with the example of a number theory group, at some schools all of the members of the group will be involved in the same flavor of number theory (e.g., the group might be biased towards arithmetic geometry), while at other schools the group might have people representing many different area of number theory (algebraic, analytic, arithmetic geometry, etc).

So what is the purposes of these groups? Well, they're just as much for the faculty members as they are for students. As you mentioned in your post, students in a research group are likely to hang out and talk math with one another. (You'll also be taking a lot of topics classes with the other students in your research group.) The same goes for faculty members. If you're a professor working on a project it's very nice to have someone who works down the hall from you that you can ask fairly technical questions to. (Or perhaps "stupid" questions whose answer you don't know but feel you should.)

Research groups will also generally have their own weekly seminars. Sometimes these seminars have a lot of talks by the faculty and students at that school, while other times the speakers are primarily visitors from other schools. This gives graduate students the opportunity to see what sort of problems people are working on at the moment and a bit of perspective about the nature of their research area that goes beyond the particular problems that the local professors are studying.

The groups are a bit more serious than what you suggest. Generally they consist of a few professors and their doctoral students. All are interested in some fairly narrow range of questions. Often they are led by a senior professor. They pose research questions to one another (long term research ideas, not simple exercises) and explore approaches. Often the doctoral students get their research problem ideas from the seminars held by these groups.

Perhaps they discuss papers and work in progress. The currency here is ideas, approaches, insight.

It is a good idea to get attached to one of these relatively early on so that you get some sense about the research that is done and how people approach it. Don't wait until you've passed qualifiers to start narrowing your focus to a professor and a potential topic. These "seminar" groups can be valuable for that.

Just adding to other answers and comments: it'd be unfortunate in several ways if a PhD student didn't talk to anyone other than their own advisor. For one thing, they'll need a PhD committee of 3 or 4 people who're willing/able to endorse the thesis, and also hear the defense.

Oh! And the letters of recommendation to get a job!!! You'll need at least 3, and after your advisor that leaves 2 more. If you've never talked to any other faculty at your home university (or anywhere else) you'll get lukewarm letters. Truly, you'll want some other faculty to be aware of you and what you're working on!!!

And getting a broader education/exposure in your general subject is a good thing... and perhaps the most convivial for beginners is a seminar run by and given by beginners... "from your group".

Finally: as with many human activities, while one may find reason for it, to be "without a group" is generally dangerous... since one has no support, no encouragement, no validation, no sanity checks...