Life as a financially independent academic

I think this is completely feasible for a mathematician and maybe several other fields. No academic affiliation is needed to attend academic conferences or to publish papers. What you need is some means of collaboration to help firm up your ideas, nothing more, really. For that you need to meet people and to exchange ideas.

What would be frowned on, however, is replacing a paid faculty member with a volunteer. Most people do need that paycheck and it seems somehow wrong to deny someone that opportunity. You aren't suggesting that, of course, so I see no issue.

The easy way to achieve it, assuming that you live near to some large university (preferably) with a large faculty, is just to go and introduce yourself. You might need an invitation first, arranged by email, campus security being what it is today. If you can interest a member of the faculty (or a few) to talk to you about common interests you could probably get a standing invitation to participate in research seminars and such. If your ideas are sound, people will be happy to work with you.

Since internet communication is easy these days you could even use a conference to meet people and then work remotely. Or use your Alma Mater as a source of contacts.

One possible negative, however, is the level of commitment you are willing to make. If you are working on joint research, you need to do your part so that others aren't let down. I don't see that as a big issue in mathematics, but it might be in some fields.