Inverse trigonometric function identity doubt: $\tan^{-1}x+\tan^{-1}y =-\pi+\tan^{-1}\left(\frac{x+y}{1-xy}\right)$, when $x<0$, $y<0$, and $xy>1$

From the Article $240,$ Ex$-5$ of Plane Trigonometry(by Loney),

$$\arctan x+\arctan y=\begin{cases} \arctan\frac{x+y}{1-xy} &\mbox{if } xy<1\\ \pi+\arctan\frac{x+y}{1-xy} & \mbox{if } xy>1\\\text{sign}(x)\cdot\dfrac\pi2 & \mbox{if } xy=1\end{cases} $$

Later in this answer, I propose a complete solution to the given problem. But in the first part of this answer, I merely respond to the following doubt:

Now why do we add $\pi$ to the expression when we are already having a positive argument ...

Actually, we subtract $\pi.$ But how are we to understand intuitively that this is something we might want to do?

As you observed, under the given conditions on $x$ and $y,$ we know that $\frac{x+y}{1-xy} > 0$ and therefore $\tan^{-1}\left(\frac{x+y}{1-xy}\right) > 0$ as well.

But the given conditions include $x< 0$ and $y< 0$, from which it follows that $\tan^{-1} x < 0,$ that $\tan^{-1} y < 0,$ and that $\tan^{-1} x + \tan^{-1} y < 0.$

We cannot have an equation with a negative number on the left and a positive number on the right, can we? But we can add or subtract something on the left or right in order to make the two sides equal after all. What is to be shown then is that the thing to add or subtract is a constant for all $x$ and $y$ that satisfy the given conditions, and that subtracting the particular constant $\pi$ from the right side will make the equation satisfied.

---

To actually solve a problem like this, we have to keep in mind that the trigonometric functions are not one-to-one, and therefore the inverse trigonometric functions are not true inverses. For example, $$\tan\left(\frac34\pi\right) = -1,$$ but $$\tan^{-1}(-1) = -\frac14\pi \neq \frac34\pi.$$

As long as we restrict the domain of the tangent to angles in the interval $\left(-\frac12\pi,\frac12\pi\right),$ the function is one-to-one, the inverse tangent is actually an inverse of the tangent, and everything is fine. As soon as any angles go outside that interval, things get more complicated.

For real numbers $x$ and $y$, let $\alpha = \tan^{-1} x$ and $\beta = \tan^{-1} y.$ That is, let $\alpha$ and $\beta$ be the unique angles in the interval $\left(-\frac12\pi,\frac12\pi\right)$ such that $x = \tan\alpha$ and $y = \tan\beta.$ Then $$ \tan(\alpha + \beta) = \frac{\tan\alpha + \tan\beta}{1 - \tan\alpha\tan\beta} = \frac{x + y}{1 - xy}. $$

Now to make things really simple at first, let's restrict the angles even further: let $\alpha$ and $\beta$ both be in the interval $\left[0,\frac14\pi\right),$ which ensures that $0 \leq \alpha + \beta < \frac12\pi.$ That is, all the angles in the formulas above and their tangents are conveniently positive and the inverse tangents of the tangents return the original angles in all three cases: $\alpha$, $\beta$, and $\alpha+\beta$; that is, \begin{align} \tan^{-1}(\tan\alpha) &= \alpha, \tag1\\ \tan^{-1}(\tan\beta) &= \beta, \tag2\\ \tan^{-1}(\tan(\alpha+\beta)) &= \alpha+\beta. \tag3\\ \end{align}

Then from Equations ($1$–$3$) we have $$ \tan^{-1}(\tan\alpha) + \tan^{-1}(\tan\beta) = \tan^{-1}(\tan(\alpha+\beta)) $$ (since both sides of this equation equal $\alpha + \beta$), and replacing the tangents in this equation with the equal expressions in $x$ and $y$ we get $$ \tan^{-1}x + \tan^{-1}y = \tan^{-1}\left(\frac{x+y}{1-xy}\right). \tag4$$

Note, however, that we got this formula under restrictions on the angles $\alpha$ and $\beta$ that guaranteed that $0\leq x < 1$ and $0\leq y < 1,$ which together guarantee that $0 \leq xy < 1,$ which contradicts the conditions on $x$ and $y$ in the question.

So let's consider the case given in the question: $x < 0,$ $y < 0,$ and $xy > 1.$ Setting $\alpha = \tan^{-1} x$ and $\beta = \tan^{-1} y$, we still have $\alpha$ and $\beta$ in the interval $\left(-\frac12\pi,\frac12\pi\right),$ and we have $x = \tan\alpha$ and $y = \tan\beta,$ but in this case $\tan\alpha$ and $\tan\beta$ are negative.

Then $\tan\alpha \tan\beta = xy > 1,$ and multiplying on both sides by $\cot\beta$ (which is negative) reverses the sign, so $$\tan\alpha = \tan\alpha\tan\beta\cot\beta < \cot\beta = \tan\left(\frac12\pi - \beta\right) = \tan\left(-\frac12\pi - \beta\right)$$ (recalling that $\tan(\theta \pm \pi) = \tan\theta$ for any angle $\theta$). Since $\alpha$ and $-\frac12\pi - \beta$ are both in the interval $\left(-\frac12\pi,0\right)$ in this case and since the tangent function is strictly increasing in that interval, $\tan\alpha < \tan\left(-\frac12\pi - \beta\right)$ if and only if $\alpha < -\frac12\pi - \beta,$ which is true if and only if $\alpha + \beta < -\frac12\pi.$ So we see that $\alpha + \beta$ is not in the interval $\left(-\frac12\pi,\frac12\pi\right),$ and we cannot conclude that Equation $(4)$ will be true, but we can observe that $\alpha + \beta > -\pi$ and therefore $0 < \alpha + \beta + \pi < \frac12\pi,$ from which it follows that $$ \tan^{-1}(\tan(\alpha + \beta)) = \tan^{-1}(\tan(\alpha + \beta + \pi)) = \alpha + \beta + \pi. \tag5 $$ Equations $(1)$ and $(2)$ are still true, and together with Equation $(5)$ these yield $$ \tan^{-1}\left(\frac{x+y}{1-xy}\right) = \tan^{-1}(\tan(\alpha + \beta)) = \tan^{-1}x + \tan^{-1}y + \pi, $$ or (with some algebraic rearrangement), $$ \tan^{-1}x + \tan^{-1}y = -\pi + \tan^{-1}\left(\frac{x+y}{1-xy}\right). \tag6 $$

---

That's the answer to the question as asked. But let's consider what $\tan^{-1}x + \tan^{-1}y$ might equal in other cases.

For the case where $x > 0,$ $y > 0,$ and $xy > 1$, simply substitute $-x$ and $-y$ for $x$ and $y$ (respectively) everywhere in the derivation of Equation $(6)$. The result is $$ \tan^{-1}(-x) + \tan^{-1}(-y) = -\pi + \tan^{-1}\left(\frac{-x-y}{1-xy}\right), $$ and using the fact that $\tan^{-1}(-t) = -\tan^{-1} t,$ we can reverse the signs on both sides of the equation to obtain $$ \tan^{-1}x + \tan^{-1}y = \pi + \tan^{-1}\left(\frac{x+y}{1-xy}\right). \tag7 $$

In the case where $x < 0,$ $y < 0,$ and $xy = 1$, we can re-run the derivation of Equation $(6)$, except that we derive equations from $xy = 1$ instead of inequalities, resulting in the conclusion that $\alpha + \beta = -\frac12\pi.$ So in this case $$ \tan^{-1}x + \tan^{-1}y = -\frac12\pi. \tag8 $$

In the case where $x > 0,$ $y > 0,$ and $xy = 1$, we have $$\tan^{-1}x + \tan^{-1}y = -\left(\tan^{-1}(-x) + \tan^{-1}(-y)\right),$$ and the negative numbers $-x$ and $-y$ satisfy the conditions of Equation $(8)$, so $\tan^{-1}x + \tan^{-1}y = \frac12\pi. \tag9$

The only remaining case to consider is the case $xy < 1.$ (This includes all possible cases where $x$ and $y$ have opposite signs or one of the numbers $x$ or $y$ is zero.) In order to prove that Equation $(4)$ is always true in this case, it is sufficient to show that Equations ($1$–$3$) are always true, for which it is sufficient to ensure that $\alpha$, $\beta$, and $\alpha+\beta$ are all in the interval $\left(-\frac12\pi,\frac12\pi\right).$ The conditions on $\alpha$ and $\beta$ are satisfied by $\alpha = \tan^{-1} x$ and $\beta = \tan^{-1} y$ for any real numbers $x$ and $y,$ so all that remains to show is the condition $-\frac12\pi < \alpha + \beta < \frac12\pi.$

First, if $\alpha$ and $\beta$ have opposite signs then this implies that $-\frac12\pi < \alpha + \beta < \frac12\pi$, so Equation $(4)$ is true in that case.

Next, if $\alpha$ and $\beta$ both are positive, then $-\frac12\pi < \alpha + \beta$ due to that fact alone, but $\alpha + \beta < \frac12\pi$ if and only if $\alpha < \frac12\pi - \beta,$ which is equivalent to $\tan\alpha < \tan\left(\frac12\pi - \beta\right) = \cot\beta,$ since both $\alpha$ and $\frac12\pi - \beta$ are in $\left(0,\frac12\pi\right)$ and the tangent function is strictly increasing on that interval. Then since $\tan\beta > 0,$ we can multiply by $\tan\beta$ on both sides to find that $\tan\alpha\tan\beta < \cot\beta\tan\beta = 1$ if and only if $\alpha + \beta < \frac12\pi.$

If $\alpha$ and $\beta$ both are negative, then $\alpha + \beta < \frac12\pi$, but $\alpha + \beta > -\frac12\pi$ if and only if $(-\alpha) + (-\beta) < \frac12\pi,$ which according to the previous paragraph is true if and only if $\tan\alpha \tan\beta = \tan(-\alpha) \tan(-\beta) < 1.$

Finally, in the case where either $\alpha$ or $\beta$ is zero, it follows that $-\frac12\pi < \alpha + \beta < \frac12\pi$ and that $xy = 0 < 1.$

In summary, in all possible cases where $\alpha = \tan^{-1} x$ and $\beta = \tan^{-1} y$ we have $-\frac12\pi < \alpha + \beta < \frac12\pi$ if and only if $xy < 1.$ We can conclude that Equation $(4)$ is true if and only if $xy < 1.$

All of these cases are can be summarized as follows: $$ \tan^{-1}x + \tan^{-1}y = \begin{cases} \tan^{-1}\left(\frac{x+y}{1-xy}\right) & xy < 1, \\ \pi + \tan^{-1}\left(\frac{x+y}{1-xy}\right) & xy > 1, x > 1, \\ -\pi + \tan^{-1}\left(\frac{x+y}{1-xy}\right) & xy > 1, x < 1, \\ \frac12\pi & xy = 1, x > 1, \\ -\frac12\pi & xy = 1, x < 1. \end{cases} $$

We need all these cases because $\tan^{-1}x + \tan^{-1}y$ takes every value in the interval $(-\pi,\pi)$ for some values of $x$ and $y,$ but $\tan^{-1}\left(\frac{x+y}{1-xy}\right)$ is only able to produce values in the interval $\left(-\frac12\pi,\frac12\pi\right).$