Thesis Advisor conundrum
You've not really commented on your thesis advisor's role as a research supervisor, which is quite different from being an instructor. Someone can excel in one role but be [almost] incompetent in the other.
Consequently, it's hard to give you solid advice. The basic question is if your advisor's unavailability is impeding your research progress. If so, then you should definitely explore other options, if possible. For instance, look for a co-advisor for your project with A'. If no options with A' pan out, then you should consider switching advisors.
However, if the teaching style doesn't carry over into research issues, then it may be worth your while to wait and see if things improve.
The choice of thesis advisor is extremely important in an academic career, arguably the most important choice you will make. A good advisor will know your talents and set you a great problem, which you will solve in a timely fashion; a good advisor will also be a tremendous asset to your career in writing letters of recommendation and just plain giving advice. A bad advisor can easily send your career to an early grave. Hence this choice is very important, no pressure. :-)
Given all this, I would urge you to select a different advisor. Although A' taught a class you like, there are plenty of warning signs that this will not be a productive relationship for you. A' is busy, A' taught poorly, you didn't learn much in course 2A, A' disrespected your work during your presentation, etc.
You should find an advisor, perhaps in a different specialty, with whom you have a better connection. This may delay your graduation. That's okay; it's much better to do the right thesis in 7 years than the wrong thesis (or drop out) in 5 years.
I agree with both vadim and aeismail: choosing an advisor is important and there's some indication that the chair may not make an ideal advisor for you at present, but there's not enough information to say this person would not be a good advisor for you. Here are some points to keep in mind:
My personal belief is that how well you can work with someone is just as important as their field of expertise in choosing an advisor. While there are some types of math you may not like, probably there are many fields you would be equally happy working in.
Both being chair and being seriously ill (I don't know if this is the case for your professor) can significantly cut down on the availability of advisor. On the other hand, some professors not in these situations are rather hands-off, and some chairs can still make a fair amount of time for their students. It sounds like you already know you want a "hands-on" advisor, so that should help you in finding an advisor. (Suggestion: talk to some of a prospective advisor's other grad students to see how and how much they interact.)
Some students (perhaps you) get more anxious/stressed than others, and some professors are better at putting such students at ease than others. Possibly he was not aware/did not anticipate your reactions to his interruptions and/or maybe he was just cranky because of stress of being the chair and dealing with his own illnesses. It's possible that he can learn to deal with you in a less stressful way (this might involve a conversation, or possibly not).
I don't have personal experience with this, but I hear counseling can help with stress and anxiety issues. Most universities (at least in the US) have services for such issues, so if this is not an isolated experience for your it may be a good idea to see what is available at your school, independent of who you choose for an advisor.