Why is this not a valid algorithm for counting poker hands?

The first answer is correct, in this case you're counting unordered sets, however in your solution when you write selecting $B$ and then $C$ separately you're in a sense counting ordered pairs. What I mean by this is consider choosing $3$ Aces($A$), a king ($K$) and a queen ($Q$)as your hand. Then your algorithm is counting the hands $AAAKQ$ and $AAAQK$ as distinct hands when they should be considered the same since you counted your sample space using unordered sets. This also explains why your answer was exactly twice the correct one, it's because you counted every possibility exactly twice with the order flipped.

In your solution, suppose it is for three jacks and two other cards. With your method of counting the other two cards, the pair $(2S,3D)$ is counted again as $(3D,2S).$ So you're double counting the remaining cards, explaining your result being twice the right one.