Lack of team support in post-PhD Mathematics career
This model is extremely common in Europe (for example, one of my coauthors in Germany is in a group with her and another senior professor, two junior mathematicians on essentially very long term second postdocs, and about 7 or 8 young postdocs and 7 or 8 graduate students), but there is no institutional framework for it in the US. I know of pairs of tenure-line faculty at the same institution who are close enough to essentially run a joint research group, but outside of a few departments, they are unlikely to have more than one postdoc at a time, so "research group" mostly means them and their graduate students.
Not really. Essentially postdoctoral positions are the only mechanism that exists in the US system for paying people who are not already performing at a pretty high level to do research. You could still apply to these, though as I think you've already guessed, the odds aren't great this long post-Ph.D. I think the other possibility would be obtaining a long term, non-tenure-track lectureship at a research university (for example, my department hired someone to such a job this year) where there are senior faculty who might work with you. There would still be high teaching expectations (though maybe not worse than a TT community college position), and any research involvement would be "on your own time," but presumably that's largely the case now.
I suppose this depends on how you define "career as a research mathematician." The answer is obviously not "absolutely no," look at Yitang Zhang. Of course, that's a very extreme example (Zhang didn't even have a teaching position in mathematics for several years), but I think it does show that if you do something undeniably great, at whatever age, the community is open to it. On the other hand, this is like asking "If I can't get on a Division I college team, does that mean my career as a basketball player is over?" Obviously, no one can stop you from playing basketball, but if you mean doing so professionally, the odds are slim. If you're really that good, then maybe an NBA team will notice and hire you. But you're not going to be in the place that they look for potential new players, and you'll be missing out on how most people develop their skills. So it's a tough row to hoe.
I think in terms of what's realistic: if what you're missing is being around other people doing mathematics, getting ideas, etc. and you're not worried about security then I would look into finding a permanent (or at least renewable) but non-tenure-track teaching position at a larger research department. It will certainly have some downsides, but I think it's the closest feasible approximation to what you're looking for.