Can non-PhDs or non-Masters be on an accredited college's faculty?
It depends on what you mean. If you are asking a theoretical question about whether there are rules that forbid it, then the answer is that it is theoretically possible, at least at some universities. For example, Andrew Gleason was a tenured professor of mathematics at Harvard from 1953 through his retirement in 1992, without ever having attended graduate school. (Technically, Harvard awarded him an honorary master's degree when he became a faculty member, but he had no master's degree when he was hired and never received a Ph.D.) The rules vary between universities, but I do not believe Harvard's have changed since Gleason was there. For another example, if you invent the World Wide Web, you can become a professor with no master's degree.
On the other hand, it is impossible in practice. Unless you have received some sort of major academic recognition (a big prize, universities specifically soliciting an application from you despite knowing you have no master's degree, etc.), it's not even worth thinking about, since the chances of being hired are almost indistinguishable from zero. If you are aiming for an academic career, choosing not to go to graduate school means giving up on that career.
By the way, I'm assuming here that you are asking about fields in which there are very few famous practitioners without advanced degrees. I can imagine that in certain fields (perhaps art, business, or politics), there might be more people who would be attractive to universities despite having only a bachelor's degree. But even in those cases, it would require truly impressive achievements.
Yes, it is possible (at least for part-time appointments). I am part-time faculty at the University of Washington, and not only do I have just a bachelor's degree, my degree isn't even in the same field as my faculty appointment. My real world skills and knowledge combined with my mentoring/leadership experience are all that were required.
However, it would be very rare for full-time faculty to not have at least a master's. UW policy requires the master's degree for full-time faculty and a PhD for professorships, and some departments are adopting rules requiring a PhD for any full-time position.
It is not forbidden. Accreditation is based on many factors and percentage of terminal degree holders is one of those statistics that is considered. Even the most prestigious colleges may have a number of MS/MA/MFA on their faculty. In rare instances, even people without degrees might be on a faculty. You may find these rare people on performing or creative arts faculties -- writers, actors, painters, filmmakers, etc. that have outstanding bodies of work or accomplishments.