Find the number of ways in which 6 persons out of 5 men and 5 women can be seated at a round table such that 2 men are never together.

Label the chairs from $1$ to $6$ in increasing order, label the men and the women from $1$ to $5$. Consider three cases:

1) There is 1 man: choose a man in $5$ ways, place the man at chair $1$, and permute the $5$ women in the remaining chairs in $5!$ ways. Therefore the number of ways is $5\cdot 5!=600$.

2) There are 2 men: choose two men in $\binom{5}{2}$ ways, place the man with the smallest label at chair 1 and the other at chair $3$, $4$ or $5$ ($3$ ways), choose four women in $\binom{5}{4}$ ways and arrange them in the remaining chairs in $4!$ ways. Therefore the number of ways is $\binom{5}{2}\cdot 3\cdot \binom{5}{4}\cdot 4!=3600$.

3) There are 3 men: choose three men in $\binom{5}{3}$ ways, place the man with the smallest label at chair 1, and the other at chair $3$ and $5$ ($2$ ways), choose three women in $\binom{5}{3}$ ways and arrange them in the remaining chairs in $3!$ ways. Therefore the number of ways is $\binom{5}{3}\cdot2 \cdot \binom{5}{3}\cdot 3!=1200$.

Hence the total number of ways is $600+3600+1200=5400$.

You forgot to include the number of ways in which the men can be selected. In this case, you get:

$${5 \choose 3}{5 \choose 3}2!3! = 1200$$ $${5 \choose 2}{5 \choose 4}3!\frac{4!}{2!} = 3600$$ $${5 \choose 1}{5 \choose 5}4!\frac{5!}{4!} = 600$$

This results in a total of $1200 + 3600 + 600 = 5400$ possibilities.