Probability of picking a uniquely colored ball successively without replacement

If you're picking the balls without replacement, then you can treat this problem with combinations (since it doesn't matter in which order you pick the balls).

There are $_{10}C_3$ ways to pick three balls out.

To pick three different colors, choose the colors ($_5C_3$) and then choose which ball of each color ($8$). This gives probability $8\cdot (_5C_3) / (_{10}C_3)$.

To pick two different colors, pick the color with two balls ($5$) and pick the other ball ($8$). So, probability $5 \cdot 8 / (_{10}C_3).$

This method works for any number of colors and any number of balls per color, as long as the number of balls per color is the same. If you allow for different numbers of balls of particular colors then you'll need to treat those cases differently.

I do not think there is a cooking formula for everything :It is all about counting. First, we count the possible scenarios, and then the ones in our favor.

We have $5$ colors : $C_1, C_2,..,C_5$

We pick $3$ balls, how many ways are they to pick $3$ balls out of $10$ ? $\binom{10}{3}$ possibles scenarios.

Now, here, we count the scenarios in our favor, that is $3$ different colors. First, we must choose $3$ colors in order to create a "good" scenario, for example : one red, one blue , one yellow. Here we have $5$ colors, and there are $\binom{5}{3}$ ways to choose three colors. For each color, we have two choices of balls, that is $2^3$ ways to pick 3 balls among the three chosen colors.

The total is then $\binom{5}{3}*2^3$ , and the probability is the ratio between the number of good scenarios and the number of possible scenarios $$\frac{\binom{5}{3}*2^3}{\binom{10}{3}}$$

Just bear in mind “favorable cases divided by all cases” and you won't be confused. In your example, let's compute the probability of picking three different colors. every ball is good at the first pick, so the probability to be on the right track is $\frac{10}{10}=1$ (certainty). At the second pick, 8 balls are good in overall 9 balls, so the probability to be on the right track becomes $1\cdot\frac{8}{9}$. At the third pick, you have 6 good balls in 8, so the probability of picking three differently colored balls is $\frac{8}{9}\cdot\frac{6}{8}=\frac{6}{9}=\frac{2}{3}$. The other possible case, that is, to have both balls of a color, is then $1-\frac{2}{3}=\frac{1}{3}$. We can also compute it directly: $\frac{1}{9}$ (probability of picking two balls of the same color in the first two picks) and $\frac{8}{9}\cdot\frac{2}{8}=\frac{2}{9}$ (probability of picking two different colors in the first two picks, and a repetition at the third) is $\frac{1}{9}+\frac{2}{9}=\frac{3}{9}=\frac{1}{3}$.

I think this reasoning is much simpler than other answers.