Probability and marbles

Assume your brother brings $x+2$ red and $y$ blue marbles, and leaves $2$ marbles, where $0\leq x+y\leq 6$.

Then the probability that those two marbles are both red is: ${\binom{x+2}{2}/\binom{x+2+y}{2}}$, which is claimed to be $1/2$
$$\dfrac{(x+2)(x+1)}{(x+y+2)(x+y+1)}=\dfrac{1}{2}$$

Find the integer solution.

---

Hunt and seek is a viable method.

Note: Increasing the number of red marbles ($x$) always raise the probability closer to one, if not already there, while increasing the number of blue marbles ($y$) will always decrease the probability.   Start at $x=0,y=0$ and searching with this as a guide.

So, indeed your brother brought three red and one blue marble into the room. $$\dfrac{(1+2)(1+1)}{(1+1+2)(1+1+1)}=\dfrac{6}{12}$$

Since you have already found your answer, here are just some follow-up comments.

You are right the key is to solve ${x \over b} {x -1 \over b-1} = \frac12$ or equivalently, $b(b-1) = 2x(x-1)$ for positive integers $b, x$.

This is one equation with two unknowns, so it can have multiple solutions, and indeed it does: the first few pairs are $(b,x) = (4,3), (21, 15), (120, 85), (697, 493), (4060, 2871), (23661, 16731)$. Of course, all of them except the first pair is out of range for this problem (since the brother has $8$ marbles max).

I am not an expert on this kind of equation, but I noticed that, for large values, we have $b^2 \approx 2 x^2$ or $b \approx \sqrt{2} x$, so I just wrote some code to loop through $b$ and search for a few $x$ values near $b / \sqrt{2}$. The above $6$ pairs are all I found for $b\le 100000$.

And lo and behold, the sequence $4, 21, 120, 697, \dots$ is in OEIS where this exact probability puzzle is mentioned and there is a wealth of other information. Enjoy!