circle envelope tangent in another circle

Here is an analytic geometry solution :

We can assume without loss of generality that $R=1$, i.e., we work inside the unit circle. Therefore, we have the following condition : $0 < m <1$.

Let us introduce some notations.

Let $B$ be the second intersection of line $GC$ with the unit circle.

Let $a$ be the polar angle of point $G$ (i.e., oriented angle between positive $x$-axis and $AG$).

Let $b$ be the polar angle of $B$ (i.e., oriented angle between positive $x$-axis and $AB$).

Let $\theta$ be the angle of line $GE$ with line $GB$ (equal to the angle between $GB$ and $GF$).

We will use a certain number of trigonometric formulas. An extensive and well structured list of those can be found here.

Fig. 1: $G(\cos a,\sin a)$, $B(\cos b,\sin b)$, and $C(m,0)$, center of the small circle with radius $r$. The circle to which lines $EF$ are all assumed to be tangent is in red.

Let us look for relationships between angles $a,b$ and $\theta$ and lengths $m$ and $r$.

• a) The fact that $GE$ and $GF$ are tangent to the small circle is expressed by the following relationship :

$$\sin \theta = \dfrac{r}{GC}$$

which is equivalent to :

$$(\sin \theta)^2 = \dfrac{r^2}{GC^2}= \dfrac{r^2}{(\cos a - m)^2+(\sin a - 0)^2}=\dfrac{r^2}{1 - 2m \cos a +m^2}\tag{1}$$

• b) The angle between $AE$ and $AB$ is $2 \theta$ by central angle theorem. The same for the angle between $AB$ and $AF$. Let $I$ be the intersection point of line $AB$ and line $EF$ ; triangle $EAF$ being isosceles, line segment $AI$ is orthogonal to $EF$, implying that its algebraic measure ("signed distance") is $AI=\cos(2 \theta)$ (possibly negative).

Therefore, straight line $EF$ whose normal vector is $\vec{AB} = \binom{\cos b}{\sin b}$ has equation :

$$x \cos b + y \sin b = \cos 2 \theta \tag{2}$$

Using relationship $\cos 2 \theta=1-2 \sin^2 \theta$ with formula (1) :

$$x \cos b + y \sin b = 1-\dfrac{2 r^2}{1 - 2m \cos a +m^2}\tag{3}$$

• c) Let us, finally, express that $G, C$ and $B$ are aligned. This relationship, written under the form (see http://mathworld.wolfram.com/Collinear.html) :

$$\begin{vmatrix}m &\cos a &\cos b\\0 & \sin a&\sin b\\1&1&1\end{vmatrix}=0\tag{4}$$

i.e.,

$$m=\dfrac{\sin(a-b)}{\sin a - \sin b}$$

$$\iff \ \ m=\dfrac{\color{red}{2 \sin(\tfrac12(b-a))}\cos(\tfrac12(b-a))}{\color{red}{2 \sin(\tfrac12(b-a))}\cos(\tfrac12(b+a))}=\dfrac{1+\tan(a/2) \ \tan(b/2)}{1-\tan(a/2) \ \tan(b/2)}$$

yielding the rather unexpected following condition :

$$\tan(a/2) \ \tan(b/2)=k \ \ \text{where} \ \ k:=\dfrac{m-1}{m+1}\tag{5}$$

from which we deduce (using a "tangent half-angle formula" and setting :

$$t:=\tan(b/2)$$

that :

$$\cos a=\dfrac{1-\tan(a/2)^2}{1+\tan(a/2)^2}=\dfrac{1-\left(\tfrac{k}{t}\right)^2}{1+\left(\tfrac{k}{t}\right)^2}=\dfrac{t^2-k^2}{t^2+k^2}\tag{6}$$

Plugging (6) into formula (3), we get for the RHS of (3) after some algebraic transformations :

$$\cos 2 \theta=1-\dfrac{2r^2(t^2(m+1)^2+(m-1)^2)}{(m^2-1)^2(1+t^2)}$$

Using once again tangent half-angle formulas, we can now express the equation of straight line $EF$ under the following parametric form in variable $t$ :

$$x \dfrac{1-t^2}{1+t^2} + y \dfrac{2t}{1+t^2} - 1+\dfrac{2r^2(t^2(m+1)^2+(m-1)^2)}{(m^2-1)^2(1+t^2)}=0.\tag{7}$$

It remains to establish that the distance $d$ of the point with coordinates

$$C'(x_0,y_0)=\left(4m\dfrac{r^2}{(1-m^2)^2},0\right)\tag{8}$$

(would-be center of envelope circle) to straight line $EF$ is constant (i.e., is independent from $t$).

This distance, obtained (see http://mathworld.wolfram.com/Point-LineDistance2-Dimensional.html) by replacing $(x,y)$ in the Left Hand Side of equation (7) by the coordinates $(x_0,y_0)$ of $C'$ given by (8)

$$d=(2r^2(1+m^2) - (m^2 - 1)^2)/(m^2 - 1)^2\tag{9}$$

is indeed independent from parameter $t$ ; moreover it is (up to its sign) the awaited expression (I have used a Computer Algebra System to obtain this expression of $d$).

Remarks :

1) It is possible to characterize the tangent point of $EF$ on the (red) envelope circle, following the method outlined by @Jan-Magnus Økland in comments following this answer.

2) About a possible more (synthetic) geometry solution. I just discovered in this paper dealing with "exponential pencils of conics" that there is so-called "conjugate conic" of the first circle vs. the second circle (see Theorem 2.5 page 4). But I am not sure it will permit a new solution.

---