How do teaching buy outs work?
Nearly every school will have a different policy on buying out of teaching.
In the US, many universities have a breakdown of the position's time. For example, you may be hired to do 60% research, 20% teaching, 20% service. At some universities, this means the grants must cover the percentage of your salary that you are buying out of. At others, there is a flat buyout price (presumably the amount it costs to pay an instructor). Although, some also have requirements for a minimum number of courses you must teach per year (can't buy out).
You can also get a reduced teaching load through additional service (e.g, serving as graduate coordinator or department chair).
It is complicated. The amount a grant is charged when a professor "buys out" from a course generally factors in the professor's salary and benefits as well as the replacement teaching costs (salary, benefits, office etc).
The number of courses a professor can buy out of varies. Some departments will let you buy out of all courses, others will require you to teach 1 course a year and some will not let you buy out at all. The exact number can vary year to year depending on the teaching needs of the department, past buy outs, total grant/overhead income, and other factors.
You should not take a TT job expecting to buy out of all teaching and it is not unreasonable to ask during the hiring process what the buy "rules" are.
Normally a professor is paid for eight to nine months each academic year for teaching several courses in the period.
No, professors are paid for doing many things, including teaching. See this question.
However, if a professor gets research grant, he can support himself by the grant in the summer months, or teach less courses.
That’s a misleading statement that’s only approximately correct. Many professors don’t receive summer salary but that doesn’t mean they can’t “support themselves in the summer months” - the base salary is usually adequate for supporting oneself. And it’s not always an option to teach fewer courses - that depends on the nature of the grant and on the department agreeing to a course buyout.
if a professor only teach the half of the normal load of courses, then he will receive half of his normal salary from the department and the rest is covered by the grant, right?
No, again your math shows that you are assuming the incorrect premise that professors are only paid to teach. Even if there is a well-defined percentage X such that X percent of the professor’s salary is given for teaching (there isn’t always such a number), the amount that will be charged to the grant for a course buyout may not directly correspond to a simple arithmetical calculation of the sort you suggested. As others have noted, every institution will have its own policies about such things.
If the grant is large enough, can he choose not teaching any course without loss of pay?
Professors never “choose” how much they teach. They get assigned teaching by their department, and are required to teach the number of courses that they are assigned. But given appropriate grant funding, they can request permission from the department for a course buyout, and if that permission is granted then they can teach less than the normal load. In some places this is a routine matter and effectively professors can assume that permission will always be granted; in others it may not be.
Also note that it’s not just the size of the grant that matters. The grant budget and policies of the funding body have to be compatible with using the money for a course buyout. That won’t always be the case even when the grant is “large enough”.