Maximum area of a triangle

Suppose that the center of the circles did not lie on one of the altitudes of $\triangle ABC$ (as shown in gray). Then by leaving the base of that altitude alone and moving that altitude to the center, we increase the altitude, and thereby the area of the triangle (as shown in black).

$\hspace{3.5cm}$

Therefore, the orthocenter of the triangle must coincide with the center of the circles.

---

Consider the following diagram, in which $O$ is both the orthocenter of $\triangle ABC$ and the center of the circles:

$\hspace{3.5cm}$

$\triangle AOF$ is similar to $\triangle COD$, thus, $|\overline{OC}||\overline{OF}|=|\overline{OA}||\overline{OD}|$. Furthermore, $\triangle AOE$ is similar to $\triangle BOD$, thus, $|\overline{OA}||\overline{OD}|=|\overline{OB}||\overline{OE}|$. Therefore, define $$ p=|\overline{OC}||\overline{OF}|=|\overline{OA}||\overline{OD}|=|\overline{OB}||\overline{OE}|\tag{1} $$ and $$ a=|\overline{OA}|\quad b=|\overline{OB}|\quad c=|\overline{OC}|\tag{2} $$ With these definitions, we get $$ |\overline{OD}|=p/a\quad|\overline{OE}|=p/b\quad|\overline{OF}|=p/c\tag{3} $$ Note that $$ \frac{|\triangle AOB|}{|\triangle ABC|}=\frac{|\overline{OF}|}{|\overline{OF}|+|\overline{OC}|}=\frac{p}{p+c^2}\tag{4} $$ $$ \frac{|\triangle BOC|}{|\triangle ABC|}=\frac{|\overline{OD}|}{|\overline{OD}|+|\overline{OA}|}=\frac{p}{p+a^2}\tag{5} $$ $$ \frac{|\triangle COA|}{|\triangle ABC|}=\frac{|\overline{OE}|}{|\overline{OE}|+|\overline{OB}|}=\frac{p}{p+b^2}\tag{6} $$ and because $|\triangle AOB|+|\triangle BOC|+|\triangle COA|=|\triangle ABC|$, $(4)-(6)$ yield $$ \frac{p}{p+a^2}+\frac{p}{p+b^2}+\frac{p}{p+c^2}=1\tag{7} $$ Using $a=1$, $b=2$, and $c=3$, we can solve $(7)$ to get $p=1.458757077431284$.

Looking at the area of $\triangle AOB$ in two different ways, we get $$ \begin{align} 4|\triangle AOB|^2=|\overline{AB}|^2|\overline{OF}|^2&=(a^2+b^2-2ab\cos(\angle AOB))(p/c)^2\\ &=a^2b^2\sin^2(\angle AOB)\\ &=a^2b^2(1-\cos^2(\angle AOB))\tag{8} \end{align} $$ Solving $(8)$ for $\cos(\angle AOB)$ using the quadratic formula, and similarly for the other angles, yields $$ \cos(\angle AOB)=\frac{p^2-\sqrt{(p^2-c^2a^2)(p^2-b^2c^2)}}{abc^2}\tag{9} $$ $$ \cos(\angle BOC)=\frac{p^2-\sqrt{(p^2-a^2b^2)(p^2-c^2a^2)}}{a^2bc}\tag{10} $$ $$ \cos(\angle COA)=\frac{p^2-\sqrt{(p^2-a^2b^2)(p^2-b^2c^2)}}{ab^2c}\tag{11} $$ Using the equations above, we get

$$\angle BOC=104.071123766006501^\circ$$ $$|\triangle BOC|=2.909984011512956$$

$$\angle COA=119.094556197774592^\circ$$ $$|\triangle COA|=1.310727640381874$$

$$\angle AOB=136.834320036218908^\circ$$ $$|\triangle AOB|=0.684110332666474$$ Therefore, we get $$|\triangle ABC|=4.904821984561304$$

Fix points $A$ and $B$ and maximise the area by varying point $C$. This means maximising the height of the triangle with base $AB$.

With $C$ at the maximum point draw a line parallel to $AB$ through $C$. If it is not a tangent then there is a point on the outer circle which gives a greater area - contradiction. It is therefore a tangent and the perpendicular to $AB$ through $C$ is perpendicular to the tangent at the point it touches the circle (tangent is parallel to $AB$), and therefore passes through the centre $O$.

Some algebraic computation

The accepted answer by robjohn explains nicely why the origin has to be the orthocenter. I want to suggest an alternative approach to the computation that follows. To do so, I want to use coordinates for the points in question:

$$O=(0,0)\qquad A=(1,0)\qquad B=(x,b)\qquad C=(x,c)$$

The use of the same $x$ coordinate for both $B$ and $C$ already ensures that $OA$ is perpendicular to $BC$. You have three conditions to determine these three variables:

\begin{align*} \lVert B\rVert&=2 & x^2+b^2 &= 2^2 \\ \lVert C\rVert&=3 & x^2+c^2 &= 3^2 \\ \langle C-A,B\rangle&=0 & x^2+bc-x &= 0 \end{align*}

The scalar product expresses the condition $(C-A)\perp B$, while the third condition $(C-B)\perp A$ is already implied by these conditions (since the altitudes of any triangle meet in a single point).

You can feed this to Wolfram Alpha to obtain numeric approximations of the relevant coordinates. But you can also eliminate variables, e.g. using resultants, to obtain polynomials which describe your solutions:

\begin{align*} x^3 - 7x^2 + 18 &= 0 \\ b^6 + 37b^4 - 92b^2 + 36 &= 0 \\ c^6 + 22c^4 - 387c^2 + 1296 &= 0 \end{align*}

No ruler and compass solution possible

These are irreducible cubic polynomials in $x,b^2,c^2$ resp. so they cannot be the result of a ruler and compass construction. Every ruler and compass construction which starts from integer coordinates (like the points $O,A$ and integer radii) can only result in coordinates which are sums, differences, products, quotients or square roots of other constructible coordinates. Cubic roots are not possible.

So the main result of this investigation is this:
The triangle of maximal area cannot be constructed using ruler and compass.

Anyone looking for a “fancy geometry” construction to achieve this can stop now, unless he is knowingly looking for a good approximation only.

The value of the maximal area

Using a little bit of computation on the above coorinates using algebraic numbers in sage shows that the area $a$ satisfies the equation

$$16a^6-392a^4+133a^2+900=0$$

so that again is a cubic polynomial in $a^2$. Feeding that polynomial into WA (surprisingly only when omitting commands like “solve”) I get an ugly but exact expression for the solution in question:

$$a=\frac12\sqrt{\frac13\left(98+ \frac{9205}{\sqrt[3]{833939+7974i\sqrt{1329}}}+ \sqrt[3]{833939+7974i\sqrt{1329}}\right)}$$

This is interesting because the post you referred to claims a much simpler formula, but that was based on the incorrect $45°$ assumption robjohn already disproved. The last post by ferret, however, results in the same area I computed myself. So they, too, eventually found an exact solution.