Picking balls from urns.

There will be an urn in which the number $r$ of red balls will satisfy $r\in\left\{ 0,1,2\right\} $.

To be found is an answer to the question: "for which of the options $r=0,1,2$ will $P(S)$ be maximal?"

Let $E$ denote the event that this urn will be chosen at random and let $S$ denote the event that two balls are chosen that have the same color.

If $r=0$ then $S=E$ hence $P\left(S\right)=P\left(E\right)=\frac{1}{2}$.

If $r\in\left\{ 1,2\right\} $ then $$P\left(S\right)=P\left(E\right)P\left(S\mid E\right)+P\left(E^{\complement}\right)P\left(S\mid E^{\complement}\right)=\frac{1}{2}\left[P\left(S\mid E\right)+P\left(S\mid E^{\complement}\right)\right]$$

Here $$P\left(S\mid E\right)=\frac{r}{5}\frac{5-r}{5}+\frac{5-r}{5}\frac{4-r}{4}$$ and $$P\left(S\mid E^{\complement}\right)=\frac{5-r}{5}\frac{r}{5}+\frac{r}{5}\frac{r-1}{4}$$ (do you see why?).

Now substitute $r=1,2$ and draw conclusions.

Let $a$ be the number of red balls in the first urn. Then we have $5-a$ green balls in the first urn and $a$ green balls in the second urn (means that $5-a$ red balls in the second urn). Then, we are trying to maximize the probability of getting $2$ balls with the same color. Here, let $R$ be the event that we pick $2$ red balls and $G$ be the event that we pick $2$ green balls. Then, $$P(R) = \underbrace{\frac{1}{2}\cdot \frac{a}{5}\cdot \frac{5-a}{5}}_{\text{if we pick first urn first}}+\underbrace{\frac{1}{2}\cdot \frac{5-a}{5}\cdot \frac{a}{5}}_{\text{if we pick second urn first}} = \frac{(5-a)a}{25}$$

and

$$P(G) = \underbrace{\frac{1}{2}\cdot\frac{5-a}{5}\cdot \frac{4-a}{4}}_{\text{if we pick first urn first}}+\underbrace{\frac{1}{2}\cdot \frac{a}{5}\cdot \frac{a-1}{4}}_{\text{if we pick second urn first}} = \frac{a^2-5a+10}{20}$$

Then, $P(R \cup G) = P(R)+P(G)$ is the probability we are trying to maximize. $$P(R)+P(G) = \frac{5a-a^2}{25}+\frac{a^2-5a+10}{20} = \frac{a^2-5a+50}{100}$$

Here, we need to find maximum value of the function $f(a) = a^2-5a+50$ in the interval $a \in [0,5]$ and $a \in \mathbb{N}$. Here, you can either try it for $a = 0,1,2$ or you can sketch the graph of this parabola on $[0,5]$ and see that it gets its maximum value at $a = 0$ and $a = 5$ with $50\%$.