How would you prove if $ab|(a+b)(a+b+1)$, then $(a,b) \leq \sqrt{a+b}$ for positive integers $a$ and $b$?

Assume that $a$ and $b$ are positive integers satisfying $ab|(a+b)(a+b+1)$. Set $gcd(a,b)=x$, and set $a_x = a/x$, and $b_x=b/x$. Note that $xa_x = a$ and $xb_x=b$. Note also that $(a_x,b_x)=1.$ Since $$ab|(a+b)(a+b+1)$$ one has $$x^2a_xb_x|(xa_x + xb_x)(xa_x + xb_x +1)$$ which implies that $$xa_xb_x|(a_x + b_x)(xa_x + xb_x +1).$$ Now note that $gcd(x,xa_x + xb_x +1)=1$ so the previous relation forces $x|a_x+b_x$. We have then $$x \leq a_x + b_x = \frac{a}{x} + \frac{b}{x}.$$ One has then from clearing $x$ in the denominator $$x^2 \leq a+b$$ which implies the desired inequality.

Note that from a similar argument you can actually get a lower bound on $x$ and obtain that $$x \geq \sqrt{\frac{ab}{a+b+1}}.$$ So the actual possible range for the gcd is pretty tiny.

I'm highly curious where this problem came from. It isn't one I've seen before.

Let $d=gcd(a,b)$. Then, $d|a+b$ and hence $gcd(d,a+b+1)=1$.

This gives $gcd(d^2,a+b+1)=1$.

Now, you have $$d^2|ab|(a+b)(a+b+1) \, \mbox{ and }\, gcd(d^2, a+b+1)=1 \Rightarrow d^2|a+b$$

This gives you what you want.